Arguments for "why pulsar is secure?"

Search before asking

[X] I searched in the issues and found nothing similar.

Motivation
When people/companies evaluate open source software to rely and build on
they more and more
do not only look on its functional value alone
but also on topics like

trust giving name/membership (e.g. part of Apache family)
existing community size and growth
development progress and
especially security.

One thing often done, is a quick automated analysis

of used dependencies
their updated state
number of contained unfixed CVEs (total)
risk of these CVEs
number of contained CVEs with available fix (unfixed)

This quick analysis gives a first taste/impression
on two things:

the potential of having a meaningful security problem when using the software
the awareness of the community for the topic security

Often you do not even have to do the analysis on your own,
but you can directly find analysis results using web search.
When doing this for pulsar,
you easily find e.g. on artifacthub.io
an analysis of the latest official helm chart for Apache Pulsar
https://github.com/apache/pulsar-helm-chart
the summarized result of security analysis shows:

992 vulnerabilities (623 fixable) have been detected in this package's
images.

In detail:

there are dependencies with well known security issues (with official CVE numbers)
there is a pretty huge number of known and documented vulnerabilities: 992
including important ones (critical, high rating)
not only in the accompanying software in helm (prometheus, grafana) but in core directly (pulsar)
some were known for 9 years (CVE numbers from 2013)
there are possible fixes for most of them (for 623)

see source https://artifacthub.io/packages/helm/apache/pulsar?modal=security-report
Of course, this is only a first rough impression given by this analysis.
And the chart does not contain the very latest version of pulsar (even so, it's the latetst official helm chart)
When looking into every detail of the reported numbers, you can of course argue why not every counted vulnerabilities is a disaster...
=> But how can you easily argue to anyone having seen this fast result

In general, this software (pulsar) is secure, you can use it without any concerns.
?

Solution

finding/giving some good arguments e.g. in doc

or

if this is really not possible today,
a way/plan to find a solution to make it possible to say in the near future:

Pulsar is secure, you can use it without any concerns.

Alternatives
No response
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

this is how the summary looks like:

some details:

before writing this issue, I had a conversation with Apache Security team,
that it is no problem to contribute this as normal issue:

we'd warmly welcome ....  if you'd contribute this finding as a normal bug
report and/or patch to the project. Since outdated dependencies
without further analysis are quite public, there is no need to use this
private reporting mechanism in that case.

@hpvd  thanks for opening this issue. I think that your point is really valid.
I think that it is appropriate to start this discussion on the mailing lists (dev@pulsar.apache.org), as that's the place we run broad discussions.
Regarding this issue:
this issue is more about the "Apache Pulsar Helm Chart", that is on https://github.com/apache/pulsar-helm-chart
maybe it is better to post your findings there.
Regarding the core Pulsar package we try to keep all the dependencies as up-to-date as possible and to cut releases as soon as problems are discovered and reported to the PMC.
Unfortunately the Pulsar Helm Chart is not very active and this is way the default versions, even for the Pulsar docker image are pretty out-dated and we should upgrade

@eolivelli thanks for your comment.
Just opened an issue also in the repository of pulsars helm chart:
https://github.com/apache/pulsar-helm-chart/issues/294

would be very interesting to see, how an updated helm chart with

latest pulsar image (2.9.3 -> 2.10.1 or even 2.11)
https://github.com/apache/pulsar
latest prometheus image (2.17.2 -> 2.39.1)
https://github.com/prometheus/prometheus
latest streamnative/pulsar grafana image (0.0.16 -> no new release but in repository are some commits with updates)
https://github.com/streamnative/apache-pulsar-grafana-dashboard

would/will perform in the same scan.
=> is the update already the big part of the solution?

just as background info, the security scanner used by artifacthub, providing results shown above
is trivy, so all the finding should be pretty valid.
For details, see:
https://artifacthub.io/docs/topics/security_report/
and trivy
https://github.com/aquasecurity/trivy

there is also an easy to use github action for scanning with trivy

the complete repository,
pull requests,
docker container
IaC
etc.

=> Maybe this is interesting to integrate this directly into the CI pipeline...
See Readme of https://github.com/aquasecurity/trivy-action

number of vulnerabilities is still growing:
Today Pulsars helm chart crosses the 1k:
1005 vulnerabilities (683 fixable) have been detected in this package's images.
see https://artifacthub.io/packages/helm/apache/pulsar?modal=security-report
Good to know there was hard work done during last week and there will be soon a major update
with updated pulsar (2.10.2) and a new monitoring stack
https://github.com/apache/pulsar-helm-chart/issues/326
Details and voting: https://lists.apache.org/thread/b02r3605jvmom23p1ccjbzsfwrq5tkjv

new Helm chart v 3.0 on the base of Pulsar 2.10.2 was just released:
https://github.com/apache/pulsar-helm-chart/releases/tag/pulsar-3.0.0

Latest security analysis show

a stunning step in reducing the number included vulnerabilities:

v2.9.4 with pulsar 2.9.3 ** 1024 vulnerabilities (698 fixable) have been detected in this package's images.**
https://artifacthub.io/packages/helm/apache/pulsar/2.9.4?modal=security-report
v3.0.0 with pulsar 2.10.2 136 vulnerabilities (79 fixable) have been detected in this package's images.
https://artifacthub.io/packages/helm/apache/pulsar?modal=security-report

on the other hand

the number of vulnerabilities with a severity of CRITICAL has risen from 1 to 4 (+300%)


Persistent validation error message on form switch

Prerequisites

I have read the documentation.
I have searched the open/closed issue list
I implemented a workaround for demonstration purposes. check the JSFiddle

There, two similar form views are rendered:

the upper one with the buggy behaviour
the lower one with a workaround (just for demonstrating purposes, its not a proper fix)

Description
I am implementing a more complex form logic, where each submit action on one form triggers the rendering of the next form. the last form is an Overview, that solely displays all previously entered data, thus it merges the individual data sets of each single form view. In the last Overview form, I can choose to submit all data or cancel the submission. On cancel, the first form gets displayed again and the form data of all forms gets cleared (in the JSFiddle, the custom form data is used).
The implementation is done as a simple state machine, logic wise quite similar to the one in the JSFidddle.
In each of my form views, several fields are validated on submit, such as email address, integer fields and so on. If field validation fails, an error message gets displayed for that/those field(s). So far, this is expected behaviour.
When I finally enter the last Overview form, the last error message that occurred in any of the previous forms reappears below the corresponding field, even though the field data is correct (otherwise, the submit of that form would not have been succeeded).
In a similar fashion, if I the re-initiate the process of filling in the forms (by canceling submission in the Overview), and then enter the one form where previously the last validation error occurs, this error message reappears on the corresponding field again, even though the field has been cleared or contains valid default data. Thus I can submit this form anyways, despite the error message.
Searching the issue list, I found a similar bug, but that has already been fixed: #195
Implementation details
Even though I provide custom submit buttons for each form view (as in the JSFiddle, in order to change the button label, and to provide to buttons in the Overview), I provide a onSubmit callback as form property, thus I assume that the internal submission logic of the form library is not altered. My callback gets called on button click anyway, so I assume that onSubmit of the Form class gets triggered correctly on submit-button click.
In Form::onSubmit() I can also see, that the internal "errors" and "errorSchema" state gets reset, but for some reason, this is not reflected in the rendered forms.
Workaround
The workaround that can be seen in the JSFiddle tries to manually reset "error" and "errorSchema" states of the Form class.
In order to do that, I inherit from the Form class and extend my class with a react child context object. My own state machine (the parent of all the rendered forms) "renders" this subclass and provides the parent context (just a simple string that holds the current state name).
Changing the context in the state machine on submission of each single form will trigger a react call to "componentWillReceiveProps". My subclass overwrites this method. There, it calls the parents Form::componentWillReceiveProps first before it resets the error states of its parent.
Steps to Reproduce
In the fiddle, reproduce the following steps in the upper form in order to see the actual buggy behaviour. In the lower form, the same steps should work without error message propagation.
Steps in upper form:

enter an invalid email address (ie. remove the dot)
press next (email validation fails and an error message gets displayed, as expected)
correct email address
press next (email validation succeeds and the next form gets displayed)
press next again (enters the overview form; the email error message reappears but email address format is valid)
press cancel (enters the initial form; the email error message reappears but the default email address format is valid)
press next (next form gets displayed, as expected because email address is valid)
invalidate id field (ie. by appending a character to the number)
press next (id field validation fails; field is expected to be hold an integer. an error message gets displayed, as expected)
correct id field (remove the appended char)
press next (id validation succeeds and the overview form gets displayed. there, the id field error reappears now, while the first email error vanished)
press cancel (enters the initial form again)
press next (to enter the next form. there, the id error message reappears)

Expected behavior
When field data validation errors has been fixed in a form A and its submit succeeded, I do not expect to see any previous errors message in the next form B or when I enter form A again.
Actual behavior
The last validation error message in a form view gets propagated to the next form(s). The message then shows up in any further form, that uses the same schema structure, ie. the same nesting level with the same field key(s). If the form where an validation error has been displayed previously is rendered again, this previous error message reappers.
Version
react-jsonschema-form:
CDN version linked in documentation (probably 0.40.0)
https://unpkg.com/react-jsonschema-form/dist/react-jsonschema-form.js
react: 15.3.1
babel: 6.15.0
babel preset: react, es2015

Thanks for the detailed report. The examples are really complex to follow, also I'm seeing props used never defined anywhere, like this.props.templates.defaultData, so I'm wondering about the actual source of the issue you describe (I'm not saying it's not related, but still).
What would be super helpful is to try to isolate and reduce the problem to its simplest expression; why would the errors be persisted? it may be an issue where these aren't cleared once the form is submitted then reedited, or something like this.
Also, possibly related or possibly not, I'm seeing you use a lot the foo = {...bar} pattern, which may be used to clone objects but keep in mind this doesn't work recursively, so you'll simply clone the first level properties but nested ones will still point at their respective references. I've seen many bugs due to just this in complex form workflows, so you may triple check there's no mutation involved in your object props.
Again, sorry for not being more immediately helpful here, but I'm really lacking time to spend on complicated issues like the one you've kindly reported. Please help me by reducing the cognitive context around the problem and focus on the actual smallest bits of code involved so I can immediately grasp what's in play here.
Thanks for your understanding :)

hi n1k0, txs for your reply.
yes, the example was too complicated. I stripped down the fiddle to the basics (only two forms with one field, no EC6 stuff, no custom buttons, some hardcoded properties).  the code in the new JSFiddle.
I also removed the workaround code, so that only the erroneous form is displayed. The error message behaviour is the same as in my initial description, my local workaround also works for this simplified code.
The reproduction steps are simple:

enter an invalid email address
press submit => validation error display
correct email address
press submit => renders the second form, error message remains, but email address format is correct
press submit => renders the first form, error message remains, but email field is now empty (was set to be empty on submit)

I will also debug this code a bit. my guess would be that the internal error and errorSchema states of the Form class are not reset correctly on form submission. I will let you know about any findings.

I've been experiencing a similar issue with broken form on resubmission lately, we're probably on something here. Will investigate tomorrow.

I have a similar issue, though I'm able to re-submit the form (but validation errors won't disappear). This error only happens when I pass a custom method to the onSubmit prop. I took a look to the source code and it seems like the error could be happening here:
// Form.js
      if (_this.props.onSubmit) {
        _this.props.onSubmit(_this.state);
      }
      _this.setState({ status: "initial", errors: [], errorSchema: {} });

It behaves like if the setState method wasn't being called at all.

I am experiencing the exact same issue.
Got an error, make a change, then onSubmit is executed with { errors.length > 1 & my formData with the data not matching the error (in my case a simple required that is filled in the formData).

Hey there, I was away for some time, but back again and recently started to debug this issue.
After debugging, my assumption is that this issue might be related to _React_component lifecycle.
In order to understand the order of methods being called in/by Form and React I inherited from the Form class, and put logging all over the place.
Observation
I spare you all the logging details but the situation might basically be as follows (for reference, take a look at Form::onSubmit(), Form::componentWillReceiveProps() and Form::getStateFromProps() in Form.js):

I enter a wrongly formated email address and press submit
Form::onSubmit() gets triggered, in there
2.1) validation happens and returns an error object containing the error about wrong email format
2.2) Form::setState() is called, with the new error object
2.3) Form::onSubmit() returns without calling props.onSubmit() callback

Afterwards, at some point of time, React will change the Form's state member and adds the error from Form::onSubmit().

Now I correct the email address and press submit again
Form::onSubmit() gets triggered, in there
4.1) validation happens, this time no error gets returned, as email format is correct
4.2) the external callback props.onSubmit() is called, in there
4.2.1) in there, the state of the parent object (in my case) is changed. That will trigger parent.render() at some point of time...
4.2.2) the external onSubmit() callback returns
4.3) as last action, Form::onSubmit() resets the Form's errors by calling Form::setState() with proper values (empyt list and empty object)
4.4) Form::onSubmit() returns

Conclusion
Basically two actions are happening in parallel, and depending on which action is performed first, the result is as expected or not (as in our case)
During Form::onSubmit():

The Form's state was scheduled to be changed (reseting the errors via Form::setState() )
The parents state was scheduled to be changed (when executing props.onSubmit(), where parent::setState() was invoked)

As React does not set the state of a component immediately in setState(), but at an unknown point of time, we cannot foresee if the setting of the Form's error state will take place before or after the setting of the parents state.
In our case, it seems that the parent component is receiving its state update first and therefore also a call to its render() method. The parents render() method then renders a new Form by passing new properties to the Form.
This results in a call to Form::componentWillreceiveProps(), as defined by the React Components Lifecyle, while the Form's pending error state change has still not been propagated to the Form's state.
Form::componentWillreceiveProps() is implemented as follows:
// Form.js
componentWillReceiveProps(nextProps) {
  this.setState(this.getStateFromProps(nextProps));
}

and in Form::getStateFromProps() we can find the following lines of code:
// Form.js - getStateFromProps()
const {errors, errorSchema} = mustValidate ?
  this.validate(formData, schema) : {
    errors: state.errors || [],
    errorSchema: state.errorSchema || {}
  };

If we assume that the state.errors has not been updated yet, it still holds the previous error. Form::getStateFromProps() is therefore returning a new state object that is partly composed of the newly available properties and partly of the old state (in particular state.errors and state.errorSchema).
A new state update is then scheduled via Form::setState() and adds to the Form's previous state updates, so to say, on top of the pending one performed in Form::onSubmit().
So, the new state holds an error object that will 'overwrite' the previously changed error from Form::onSubmit() on the next occasion, when React merges and propagates all pending state changes to the Form's state.
Therefore, the error persists, even though the error object has been reset.
I hope I could express everything more or less clearly as it is quite complicated to lay out concurrency issues in an understandable way in written text.
What's next?
From my understanding now, I think that it is not important if a Form is a child of a parent React component. It only seems to matter, that the Form's external onSubmit() callback (stored in props.onSubmit) is available and implements a logic that somehow causes a new rendering of a Form, by passing new properties to the Form.
React then triggers the Form::componentwillReceiveProps() method and the situation might be as described above.
Looking at the locations where Form::getStateFromProps() is being called from, it appears that only the Form's constructor() and Form::componentwillReceiveProps() make use of that function.
Would it be correct to assume that  there is no error at all in the constructor and when receiving new properties, the Form can neglect all previous errors, as it is rendered according to those new properties, thus previous errors do not  matter anymore?
In that case, Form::getStateFromProps() could probably just return empty error objects instead of using the state errors.
\\ Form.js - getStateFromProps()
const {errors, errorSchema} = mustValidate ?
      this.validate(formData, schema) : {
        errors: [],
        errorSchema: {}
      };

@quilombo Not sure the solution exposed at the end of your response make sense.
In my case, I found that this "error state reset" end up (I added a callback to see when) after the componentWillReceiveProps, which call getStateFromProps and read the state (outdated) at the beginning of the call (so the validation made is using outdated state and produces false positives...)
Continuing my investigation for a fix...

Found something that works for me:
Replace this:
    if (this.props.onSubmit) {
      this.props.onSubmit(this.state);
    }
    this.setState({status: "initial", errors: [], errorSchema: {}});

By this
    this.setState({status: "initial", errors: [], errorSchema: {}}, () => {
      if (this.props.onSubmit) {
        this.props.onSubmit(this.state);
      }
    });

Tell me how it is for you.

Hi @MoOx Thanks for this solution, this one worked for me :) Hope it will have no weird side effects,

@MoOx are there any plans to update the onSubmit implementation with the edit you suggested above to invoke the user-provided onSubmit handler after the error state has been reset (e.g., to prevent the Form from being re-rendering with stale error messages)?
For now I'm sub-classing Form and overriding the onSubmit method with the workaround from above but it'd be nice if this made it into a release version.  Since the onSubmit property is invoked after validation occurs it seems reasonable to remove errors from the Form state prior to the user onSubmit handler being called right?

@n1k0, could you elaborate on the current Form#onSubmit behavior? It looks like it's trying to set its state twice, and calling the props.onSubmit in between. Is this some React sorcery that I don't know about? Should we be calling the props.onSubmit in callbacks as suggested by the commenters on this issue?

I don't have the context in mind anymore but from what I'm reading what @MoOx suggests makes sense, we should probably open a PR with that patch and see how it goes.

I believe this is fixed by #844.


Cannot nixos-rebuild switch due to cpio segfault

Describe the bug
A clear and concise description of what the bug is.
I upgraded my server to nixos 20.09, but the nixos-rebuild switch command fails:
Traceback (most recent call last):
  File "/nix/store/pgp9z69lscx5vamw0hgn59hb20imsd3s-systemd-boot-builder.py", line 258, in <module>
    main()
  File "/nix/store/pgp9z69lscx5vamw0hgn59hb20imsd3s-systemd-boot-builder.py", line 227, in main
    write_entry(*gen, machine_id)
  File "/nix/store/pgp9z69lscx5vamw0hgn59hb20imsd3s-systemd-boot-builder.py", line 95, in write_entry
    subprocess.check_call([append_initrd_secrets, "/boot%s" % (initrd)])
  File "/nix/store/z65l1jqvxa58zzwwa3bvglb6asj4y8cv-python3-3.8.5/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/nix/store/sssbfdn7m1r5kmnbimiigkj3x222sd3l-append-initrd-secrets/bin/append-initrd-secrets', '/boot/efi/nixos/256xb92fcn9nbmg7w7hliiwjqm
4irmbh-initrd-linux-5.4.68-initrd.efi']' died with <Signals.SIGSEGV: 11>.
warning: error(s) occurred while switching to the new configuration

The segfault comes from the cpio command: just executing cpio or cpio --help is enough to produce a segmentation fault. The weird thing is that the cpio that is installed system-wide is different from the cpio used in the append-initrd-secrets script, and my guess is that it segfaults due to a glibc mismatch:

The runpath from the cpio inside the append-initrd-secrets script: Library runpath:[/nix/store/xg6ilb9g9zhi2zg1dpi4zcp288rhnvns-glibc-2.30/lib]
The runpath from the systemd-wide cpio: Library runpath:[/nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib]

To Reproduce
Steps to reproduce the behavior:
I'm not sure how to reproduce it :-/
Expected behavior
A clear and concise description of what you expected to happen.
No segfault from the cpio used in the append-initrd-secrets script
Additional context
Add any other context about the problem here.
Notify maintainers

@shlevy by git blame? Although I'm not sure this is an append-initrd-secrets issue, but more of an unfortunate consequence of something far more sinister.
Metadata

system: "x86_64-linux"
host os: Linux 5.4.72-hardened, NixOS, 20.09.1500.edb26126d98 (Nightingale)
multi-user?: yes
sandbox: no
version: nix-env (Nix) 2.3.7
channels(root): "home-manager-20.09, nixos-20.09.1500.edb26126d98, nixpkgs-unstable-21.03pre250049.4b30d5601da"
channels(minijackson): ""
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:

Soo, I managed to get out of this situation by executing a nix-collect-garbage -d and then a nixos-rebuild switch, but this was a very stressful experience, because nixos had deleted every boot entry, and failed midway, so for a while, I did not have any boot entry :fearful:
From what I have gathered, every program using the old version of glibc just segfaulted, so when trying to add boot entries of my old generations, it tried to append the initrd secrets, failed, and no boot entry was written.
I'm not sure why the programs using the old glibc segfaulted, but I think that at least we should ensure that we have some boot entries ^^ (either by continuing after an error, or restoring the previous entries)

Sounds to me as if this issue is related to #100799

Thanks for the PR. Unfortunately, my CPU supports both FMA and FMA4. But I am using the hardened profile, so it is true that I am using the scudo allocator. It does sound quite related.

Soo, I managed to get out of this situation by executing a nix-collect-garbage -d and then a nixos-rebuild switch, but this was a very stressful experience, because nixos had deleted every boot entry, and failed midway, so for a while, I did not have any boot entry :fearful:
From what I have gathered, every program using the old version of glibc just segfaulted, so when trying to add boot entries of my old generations, it tried to append the initrd secrets, failed, and no boot entry was written.
I'm not sure why the programs using the old glibc segfaulted, but I think that at least we should ensure that we have some boot entries ^^ (either by continuing after an error, or restoring the previous entries)

So I ran into this and before I realised what had happened rebooted to find no boot entries.  I'm now trying to overcome this from an install media.  Any guidance that you can provide to roughly do the equivalent? Do I need to do a nixos-enter and drop the old generations?
I'll also try dropping the hardened profile and see if that is the cause.

Please disregard my note as more than a me too.  I resolved my situation by doing the following:

Boot recent installation media
Mount existing system
Drop inclusion of the hardened profile
nixos-install

Hopefully if someone else ends up where I was this saves them some time.


modulesync 5.2.0

@ekohl @root-expert could you please have a look at #418?
I have fixed all build/test issues and hope that we could merge and realase this before adding more changes.

@bastelfreak could you please close this PR as all your changes where merged by #418


Add debug coredump instructions

Add how to obtain the debug coredump.
Is there anything you want to mention? (unchecked code, possible bugs, found problems, breaking compatibility, etc.)
Whether should this coredump also be recommended when reporting bugs
Is it ready for merging, or does it need work?
100% working

why do we need 2 methods? the first is simpler and in most cases enough

Yeah, most of the time the first method would be enough.
This pr is just a "just in case you need the debug dump* and you can just point the user to this page instead of teaching them how to do it

make it clearer then that it's "just in case"
Additionally, since most reporters are completely not into debugging, please make additional comments of continuing without paging and enabling debuginfod.

Do you mean this set pagination off?

no, gdb sometimes asks you to "press y to continue without paging"


MS Edge, signinSilent and id_token_hint problem

After upgrading to 1.2.0 (or 1.3.0) we have a problem in Edge when trying to do a signinSilent.  We get a "Frame window timed out". But it works in Chrome/FF/Safari.
This seems to be related to #116, #187 and possibly #311.
After some digging it seems like the problem is the length of the url supplied to the iframe when trying to do signinSilent. When id_token_hint was included by default, in 1.2.0, the length of our url increased to ~4500 characters.
IFrameWindow.navigate does this._frame.src = params.url and apparently Edge ignores all urls longer than 4096 characters when setting the src of an iframe. It doesn't do anything, no error and no request to the server, just a silent ignore. So the timer always timeouts since the frame is never loaded.
If I modify IFrameWindow and remove id_token_hint from the url, since it isn't mandatory, everything works as expected in Edge.
So right now we either have to use a fork of oidc-client-js or trim the token down. Is there any way we could configure or tell oidc-client-js to not send the id_token_hint when doing a signinSilent?

Update: Seems like we had a lot of redundant data in the id token that we could remove that solved the issue for us.
Should I close the issue @brockallen or should I leave it open? It's not an obvious problem to track down if you have a large id token. If you want I could submit a PR that logs warnings on too long urls in the iframe or something similar?

Update: Seems like we had a lot of redundant data in the id token that we could remove that solved the issue for us.

Good to hear. And yes, in general, keeping id_token as small as possible helps many things. Using the user info endpoint should also help.
As for the issue, maybe what's best is to have a flag to toggle if id_token_hint is sent in silent renewal?

@brockallen yes, we removed things from the id token that was already sent by the user info endpoint.
a toggle flag would work great for us!

@crudh just wondering, how did you go about reducing the length of your token/token_hint?
Thanks for any assistance :)

@stuartbloom I don't have any technical info since I'm just on the frontend parts in this current project. But to summarize we were sending out a bunch of user related data, that aren't really claims, both in the token and from the userinfo endpoint. Now we only output that in the userinfo endpoint and only have the required claims in the token.
Does that answer the question or should I ping one of the devs that did the change?

This might sound strange, but to avoid the massive URL, I ended out doing a silent in application redirect to the exact same URL such as mybusiness.com/dashboard. The redirect removes the massive token in the URL and you have a clean SEO experience. Could you possibly show a short snippet of code that you are using.

@GDehnke-Turpin we also update the url (push a new state to the browser history) after we have received the token in the url from the redirect. We have never had any problem with the regular login redirect and url length, it only affected the iframe for the silent renew.

Gents
this is my logout code :s
    private logout() {
        this.appService.userManager.getUser().then((user) => {
            this.appService.userManager.clearStaleState();
            this.appService.userManager.signoutRedirect();
        });
    }

Thanks
Stuart

Hey @crudh @GDehnke-Turpin @brockallen
Just wondering if there is any movement on this enhancement? or a clearer workaround?
Thanks

@stuartbloom not that I know of, I don't think I have the time to work on a PR for this now that it works for us (we are in the last phase in a large project). And sorry for the late answer, just got back from vacation.

Done. Added a includeIdTokenInSilentRenew flag. For now defaults to true to be backwards compat.

I'm having the same issue for signout redirect.


Accessing DefaultTask methods leads to issues in build script

I noticed that my build script failed if I have the following task:
open class ATask : DefaultTask() {

    @TaskAction
    fun xx() {
        println(buildDir)
    }
}

// It does not care if we use `create` or `register`
tasks.create("crash", ATask::class.java)

Gradle reports with this example:

The constructor for class [] should be annotated with @Inject

If I add a @Inject constructor it failed with a ArrayIndexOutOfBoundsException.
The reason is that we try to access the buildDir inside a (Default)Task.
The getBuildDir() method is part of Project not of a DefaultTask. It seems that the IDE can't resolve it correctly but Gradle fails at runtime.
According to @eskatos it is maybe related to the way Gradle instantiate tasks.
He posted a example in Slack which works:
println(Foo())
class Foo { init { println(buildDir) } }

See Slack
Expected Behavior
The best behavior I expect is that the IDE reports me that I can't access top-level members inside a "closed" class.
Or it should work and Gradle does not fail 🙃
Current Behavior
It failed :/
Your Environment
Gradle 4.10-rc-3 (but effects also 4.9)
IntelliJ IDEA 2018.2.2 (Community Edition)
Build #IC-182.4129.33, built on August 21, 2018
JRE: 1.8.0_152-release-1248-b8 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
macOS 10.13.6

Issue moved to gradle/kotlin-dsl #1068 via ZenHub


What about 1 file for all distribution

There are not many styles to make them into a css file. What about move styles into js file as inline styles to simplify of using and distribution? I could send a pull request.

Eh - I'm a proponent for leaving it as a single CSS file and let the user decide what they want to do. I personally can easily import that CSS file as a Less import and have it compile with everything else.

Ok


Add MIUI support and improve error handling for DumpScreen and DumpScreenString

When using DumpScreenString with MIUI OS Andoid 11+ it's throw error before xml:
java.io.FileNotFoundException: /data/system/theme_config/theme_compatibility.xml: open failed: ENOENT (No such file or directory)
...
<?xml version='1.0' encoding='UTF-8' standalone='yes' ?><hierarchy>...</hierarchy>UI hierchary dumped to: /dev/tty

Solution: Filter out the exception from response (from XML parsing error when using MIUI 12 on Pocophone X3 #283 )

If XML contains an error, null is returned
Solution: Throw an exception if the response does not contain valid xml

GitHub crushed🤣

I will merge it If there is nothing to add.


S6: As a player waiting for the start of the round, I want to see a countdown page to get in the mood of playing.

Develpment Tasks

Acceptance criteria

[ ]     Countdown to game start.

[ ]     Display the scores of all players in the lobby (low priority).


Support another (not SQLite) database in Indexer

At present, we save emails index in SQLite, because we can run SQLite in browser (WebAssembly). We can have several databases for saving index, if they bring new values for our users (for example, if we can run Apache Lucene in browser, users could use another type of search queries).
definition of done: email indexes are saved to one more database, and we can query this database from browser (you need to implement simple HTML client to it, like we have for SQLite).
P.S. For example, we can try these projects:

https://leaningtech.com/cheerpj/
https://teavm.org/
Kotlin for WebAssembly

Another options:

https://supabase.com/blog/postgres-wasm
https://github.com/duckdb/duckdb-wasm


Blog Content not loading properly on click 'read more >>'

Description

Steps to reproduce

Create a new Blog Config.
Create a new Blog Category.
Create a new Article for that Config and Category and save it.
In CMS, create a new page and select the plugin 'Latest Blog Articles' and add it to the page.
In the blog preview section on clicking 'read more >>' and navigating to the contents of the individual blog, it does not load the content of the individual blogs.

Versions
djangocms-blog==1.2.3

Expected behaviour
The content of the blogs should be loaded properly.

Additional information
Screenshots

settings.py
PARLER_LANGUAGES = {
    1: (
        {'code': 'en-us'},
    ),
    'default': {
        'fallbacks': ['en-us'],
    }
}

BLOG_AUTHOR_DEFAULT = True
BLOG_AUTO_NAMESPACE = 'Blog'
BLOG_CATEGORY_PLUGIN_NAME = 'Categories'
BLOG_DEFAULT_OBJECT_NAME = 'Article'

META_USE_TWITTER_PROPERTIES = True
META_USE_GOOGLEPLUS_PROPERTIES = True  # django-meta 1.x+
META_USE_SCHEMAORG_PROPERTIES = True  # django-meta 2.x+

Most likely the same problem as in #52 and #85 and #636 and #354 and #270 and #524 and...
To recap:
Default configuration (BLOG_USE_PLACEHOLDER = True, BLOG_USE_ABSTRACT = True):

In the article creation popup, you can only write an abstract (!) for the article. This will only be shown in the article overview.
The actual body of the article must be added via the Django CMS structure menu on the article page (step by step: https://djangocms-blog.readthedocs.io/en/latest/installation.html#attach-the-blog-to-a-page)
Django CMS can be configured to add a text plugin by default, which saves you one step: https://github.com/nephila/djangocms-blog/issues/85#issuecomment-75376614

Simplified configuration (BLOG_USE_PLACEHOLDER = False, BLOG_USE_ABSTRACT = False):

You can write the body directly in the article creation popup.
To display a truncated version of the article body instead of an abstract (if BLOG_USE_ABSTRACT = False) in the article overview, you probably want to modify the djangocms_blog/includes/blog_item.html template like so:

<div class="blog-lead">
      {% if not TRUNCWORDS_COUNT %}
          {% render_model post "post_text" "" "" "safe" %}
      {% else %}
          {% render_model post "post_text" "" "" "truncatewords_html:TRUNCWORDS_COUNT|safe"  %}
      {% endif %}
</div>

Bear in mind that the BLOG_USE_{PLACEHOLDER,ABSTRACT} variables in settings.py seem to take effect only for newly created blogs. For existing blogs, you can change these settings via the admin UI: <site name> -> Administration... -> Blog configuration -> <your blog> -> checkboxes under general settings


TypeError: property is not configurable, js engine: hermes + Invariant Violation: Failed to call into JavaScript module method AppRegistry.runApplication()

Describe the bug
Running npm start and then npm run ios now results in the following error:
Invariant Violation: Failed to call into JavaScript module method AppRegistry.runApplication().
Module has not been registered as callable.
Registered callable JavaScript modules (n = 11):
Systrace, JSTimers, HeapCapture, SamplingProfiler, RCTLog, RCTDeviceEventEmitter,
RCTNativeAppEventEmitter, GlobalPerformanceLogger, JSDevSupportModule, HMRClient, RCTEventEmitter.

A frequent cause of the error is that the application entry file path is incorrect.
This can also happen when the JS bundle is corrupt or there is an
early initialization error when loading React Native., js engine: hermes

My app was working fine before installing nativewind, however I now have this issue. I have tried uninstalling nativewind to get back to where I was before, however the problem persists.
Therefore, this may not be a problem with nativewind itself however the installation process has given me this problem, so hopefully somebody can spot something wrong with my setup or ran into this issue also.
babel.config.js
module.exports = {
  presets: ['module:metro-react-native-babel-preset'],
  plugins: ['nativewind/babel', 'react-native-reanimated/plugin'],
};

package.json
{
  "name": "app",
  "version": "0.0.1",
  "private": true,
  "scripts": {
    "android": "react-native run-android",
    "ios": "react-native run-ios --simulator='iPhone 13'",
    "start": "react-native start",
    "test": "jest",
    "lint": "eslint . --ext .js,.jsx,.ts,.tsx",
    "link": "react-native link"
  },
  "dependencies": {
    "@eva-design/eva": "^2.1.1",
    "@react-native-async-storage/async-storage": "^1.17.10",
    "@react-native-community/blur": "^4.2.0",
    "@react-native-community/masked-view": "^0.1.11",
    "@react-navigation/bottom-tabs": "^6.4.0",
    "@react-navigation/drawer": "^6.5.0",
    "@react-navigation/native": "^6.0.13",
    "@react-navigation/native-stack": "^6.9.0",
    "@stripe/stripe-react-native": "^0.19.0",
    "@supabase/supabase-js": "^2.0.0",
    "@ui-kitten/components": "^5.1.2",
    "@ui-kitten/metro-config": "^5.1.2",
    "base-64": "^1.0.0",
    "base64-arraybuffer": "^1.0.2",
    "moment": "^2.29.4",
    "nativewind": "^2.0.11",
    "postcode": "^5.1.0",
    "react": "18.1.0",
    "react-hook-form": "^7.36.1",
    "react-native": "^0.70.1",
    "react-native-actions-sheet": "^0.8.8",
    "react-native-compressor": "^1.6.1",
    "react-native-draggable-grid": "^2.1.6",
    "react-native-gesture-handler": "^2.1.1",
    "react-native-image-picker": "^4.10.0",
    "react-native-linear-gradient": "^2.6.2",
    "react-native-reanimated": "^2.13.0",
    "react-native-reanimated-carousel": "^3.0.6",
    "react-native-safe-area-context": "^4.4.1",
    "react-native-screens": "^3.17.0",
    "react-native-snap-carousel": "^3.9.1",
    "react-native-svg": "^13.4.0",
    "react-native-toast-message": "^2.1.5",
    "react-native-url-polyfill": "^1.3.0",
    "use-debounce": "^8.0.4",
    "zustand": "^4.1.1"
  },
  "devDependencies": {
    "@babel/core": "^7.12.9",
    "@babel/runtime": "^7.12.5",
    "@react-native-community/eslint-config": "^2.0.0",
    "@tsconfig/react-native": "^2.0.2",
    "@types/jest": "^26.0.23",
    "@types/react-native": "^0.70.0",
    "@types/react-test-renderer": "^18.0.0",
    "@types/react-timeago": "^4.1.3",
    "@typescript-eslint/eslint-plugin": "^5.37.0",
    "@typescript-eslint/parser": "^5.37.0",
    "babel-jest": "^26.6.3",
    "eslint": "^7.32.0",
    "jest": "^26.6.3",
    "metro-react-native-babel-preset": "^0.72.1",
    "react-test-renderer": "18.1.0",
    "tailwindcss": "^3.2.4",
    "typescript": "^4.8.3"
  },
  "jest": {
    "preset": "react-native",
    "moduleFileExtensions": [
      "ts",
      "tsx",
      "js",
      "jsx",
      "json",
      "node"
    ]
  }
}

To Reproduce
Steps to reproduce the behavior:

install nativewind with the above packages
run npm start and then npm run ios
Metro will start up, however will print the above error. The ios simulator will show a white screen.

Expected behavior
The ios app to build successfully
Screenshots

I have tried:

restarting my machine
npm start -- --reset-cache
cd ios pod install
uninstalling nativewind (even when uninstalled, the error now persists, so I can't get back to my working version beforehand)

In RN, you receive that error when there is a runtime crash before your app is registered. Can you please share the original error message.
You may also want to try the next branch (see pinned issue) which fixes a number of issues.

Closing as no reproduction was provided and this doesn't appear to be a NativeWind issue.


Thermodynamic reference states need re-organizing

The way we're diagnosing temperature and pressure from entropy assumes that all massive tracers have an entropy defined with the same reference temperature, but the way that reference states are stored (one for each massive tracer) doesn't enforce that, since each massive tracer has its own reference temperature. (This is something we could address when we add support for massive tracers that aren't non-condensible ideal gases, since this will require some substantial additions to the model thermodynamics.)

Ah this isn't stuff I'm familiar with but would like to learn more about.
If it involves substantial refactoring, then maybe we can draft some API/abstraction ideas and pair program them in together.


Example Does not works

File: compose/wordpress.md

Can you provide more details what is not working?

The database connection is showing error.
On Tue, Apr 28, 2020 at 5:43 PM Sebastiaan van Stijn <
<EMAIL_ADDRESS>wrote:

Can you provide more details what is not working?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker.github.io/issues/10713#issuecomment-620567111,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOPUW67HQ6IAHW44RKFBHIDRO3B6TANCNFSM4MSSP6DQ
.

--
The information contained in this email may be confidential and/or legally
privileged. It has been sent for the sole use of the intended recipient(s).
If the reader of this message is not an intended recipient, you are hereby
notified that any unauthorised review, use, disclosure, dissemination,
distribution, or copying of this communication, or any of its contents, is
strictly prohibited. If you have received this communication in error,
please reply to the sender and destroy all copies of the message. To
contact us directly, email to<EMAIL_ADDRESS><EMAIL_ADDRESS>

Does it work if you change this line;
WORDPRESS_DB_HOST: db:3306

to
WORDPRESS_DB_HOST: db

(so removing the :3306) ?
I see there's an updated version of the example in the new "awesome-compose" repository; https://github.com/docker/awesome-compose/tree/master/wordpress-mysql

It works after exposing the DB Docker ports
On Tue, Apr 28, 2020 at 8:40 PM Sebastiaan van Stijn <
<EMAIL_ADDRESS>wrote:

Does it work if you change this line;
WORDPRESS_DB_HOST: db:3306
to
WORDPRESS_DB_HOST: db
(so removing the :3306) ?
I see there's an updated version of the example in the new
"awesome-compose" repository;
https://github.com/docker/awesome-compose/tree/master/wordpress-mysql
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/docker/docker.github.io/issues/10713#issuecomment-620667514,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOPUW677BSZOC6QJ3STAMWTRO3WXTANCNFSM4MSSP6DQ
.

--
The information contained in this email may be confidential and/or legally
privileged. It has been sent for the sole use of the intended recipient(s).
If the reader of this message is not an intended recipient, you are hereby
notified that any unauthorised review, use, disclosure, dissemination,
distribution, or copying of this communication, or any of its contents, is
strictly prohibited. If you have received this communication in error,
please reply to the sender and destroy all copies of the message. To
contact us directly, email to<EMAIL_ADDRESS><EMAIL_ADDRESS>

Without seeing the logs, the problem was most likely just that the database wasn't initialized before trying to access WordPress.
The first time the docker-compose configuration is run, we need to wait a few seconds for the database to initialize before WordPress can connect to it.


How to normalize

Hi, is it possible to normalize dataset of dynamic ids?
Every day It’ll get new ids 1567, 1568, ... so I can’t get final Id.
I thought to normalize like this  100/1567 but don’t know if this correct way

You can normalize on some very large and non-reachable (in practice) ID as "1" and divide all IDs by this value.
For example, 100000000. So you'll be normalizing your real IDs like 1567/10000000.

You can take some very large and non-reachable (in practice) ID as "1" and divide all IDs by this value.
For example, 100000000. So you'll be normalizing your real IDs like 1567/10000000.

Will Brain js work fine in this way?

Depends on definition of "fine" :-)

If it is an ID if will not be effective in the input. But usually this method works fine. (get large value in data set and divide all values)


Documentation outdated, typedoc fails for chain-ui

The link for the latest SDK documentation currently redirects to v1.3.0:
https://docs.galachain.com/latest  --> https://docs.galachain.com/v1.3.0/
However quite a few newer versions have been published to npm, current latest is 1.4.3:
https://www.npmjs.com/package/@gala-chain/api?activeTab=versions
Attempting to build and serve documentation locally, I see a number of errors caused by the chain-ui package.
e.g.
npm run typedoc-chain-api

The above command outputs a couple hundred lines of error messaging.
It looks like some tsconfig.base.json changes are needed, and/or possibly typedoc needs a vue plugin added.

typedoc_chain-ui_errors.txt
Uploading attachment with terminal output of various chain-ui errors seen when running typedoc.
To replicate, npm install, npm run build from main branch. Then attempt to run any of the npm run typedoc-chain-* commands.

Currently, it's possible to work around this and build/serve the documentation static site locally for development purposes by temporarily deleting the references to chain-ui from tsconfig.base.json in the project root.

We have a lot of documentation to update, it would be nice to get this building again.
This pull request has a basic patch to at least get docs building and serving again:
https://github.com/GalaChain/sdk/pull/376


Photoshop

Icon Request

Icon name: Photoshop
Tags of this icon: Photoshop, PS
Use case: Brand icon
Screenshots of similar icons: https://icons8.com/icon/NeNPFdj7MzXi/adobe-photoshop

👍


feat(pitr): implement func backup

Type of change:

[ ] Bugfix
[x] New feature provided
[ ] Improve performance
[ ] Backport patches

What this PR does / why we need it:

implememt func backup opengauss database
Pre-submission checklist:

[ ] Did you explain what problem does this PR solve? Or what new features have been added?
[ ] Have you added corresponding test cases?
[ ] Have you modified the corresponding document?

@lltgo


fix(functions/console_snippets/http): add error handling and test coverage

This fixes the HTTP console snippet to log an error if invalid JSON is parsed.
It also adds test coverage to handle the various cases.
Fixes #1639

Badly broke this branch, closing will re-open with change.


Autocomplete and type hint using Coc.

Hello! Thank you very much for sharing your Rams-inspired colorscheme - very elegant!
Floating windows from autocomplete and type hint (possibly others) used by Coc uses the same background and foreground color as the rest of the document. Especially type hints are hard to see.
I am not sure if this is design decision, or missing support for a package (perhaps Coc) or if it is my environment and NeoVim-config.
In my init.vim I link Coc-menu-ish things to Pmenu:
hi! link WildMenu Pmenu
hi! link CocMenuSel PmenuSel
hi! link CocHintFloat Pmenu
hi! link CocInfoFloat Pmenu
hi! link CocPumMenu Pmenu
hi! link CocInlayHint Pmenu
hi! link CocPumVirtualText Comment

If you need more information, I will try to provide it.
Example with type hint for ASTNode. In my settings I use K in normal mode.

Example with auto-complete conso from insert mode.

Again, I love this colortheme! So simple, yet elegant and beautiful!

hey @bergsans, thanks for the kind words!
After seeing your examples, I think I understand a bit better, and I'd say it's ... somewhat by design; at least for some popups, I've been setting border = single to differentiate the popup from the rest of the document:

However, I realize that doesn't work that well for all popups / menus. For example, when opening completion with omnifunc, it's really not differentiated from the rest of the UI:

Hmm, I'll have to think about this!

How about inverting the colors in all popus? Or do you think that would break with the rest of the design?

In https://github.com/stefanvanburen/rams/commit/2ee1582b3e9485a78b5835bac21fa4eec5d3b457 I tweaked the settings a little bit - now the popup menu has a light grey background with the currently selected row having a white background, which I like a bit better:

In general I try to avoid using the black as a background because it feels heavy IMO. I'm still undecided, though - feel free to pull it down and try it out!

Looks great on your image, thanks! However, its seems to depend on lua now? I cannot run the latest version. But that's on me, I am a dinosaur, I should change to using lua for NeoVim config. :)

No worries! I don't think it should depend on lua as long as you're using neovim. What's the error you're seeing?

ahh, I see. I think currently, rams requires lush.nvim to also be installed to work. It looks like other colorschemes made with lush will sometimes have a "compatibility mode" to set colors without lush.nvim being installed (see https://github.com/mcchrish/zenbones.nvim/blob/main/colors/zenwritten.vim#L7-L10 - also another great colorscheme!).
I probably won't get around to adding that for awhile as I typically just leave lush.nvim installed for development, so probably your best bet is to install lush.nvim as well for now, sorry 😄.

Will try! And again, an elegant colorscheme! Well done!


[Star Wars Saga Edition] Added Settings Tab

Submission Checklist
Required

[x] The pull request title clearly contains the name of the sheet I am editing.
[x] The pull request title clearly states the type of change I am submitting (New Sheet/New Feature/Bugfix/etc.).
[x] The pull request makes changes to files in only one sub-folder.
[x] The pull request does not contain changes to any json files in the translations folder (translation.json is permitted)

Changes / Description

Added Settings Tab
** Enabled NPC sheet to customize which skills are shown
** "Is Droid?" changed to "No Constitution," and now replaces the Con score and mod with –
** Skill Focus only replaces the individual Skill Focus part of the formula rather than overwriting the entire formula
** Text for DR, SR, and Immunities is hidden as well as the number inputs
** Nonheroic level may be added to the PC tab
** Custom skills may be added to the list of PC skills (same as NPC list)
** Added option for capital ships to increase grappling modifier
** Added option for vehicles to have Con score
Added Defense Notes to NPC sheet
Minor CSS and spacing changes

Character Sheet Info Roll20 Internal Use only.


Added Cross Platform Support for File Sorter

Added .gitignore for a few exceptions
Renamed fileSort.py -> main.py
Changed main.py to now include support for other platforms (Cross-Platform Support).
The README.md now includes instructions to add a function FileSort to bash/zsh as well as powershell.
The requirements.txt is basically the same, it was changed from LF to CRLF.
This is in respect to issue #86

Thanks @AvgBlank for taking this up!


What is the lifetime/scope of ClientInterceptorContext in .NET?

The question is below
First, this is confusing:

and then

Question
I have a .NET gRPC client on .net 4.6.2 using Grpc.Core library.
I've registered gRPC channel as singleton to reuse it following this recommendation
There's how I create an instance of the channel:
public static InvokerAndChannel<T> CreateGrpcCallInvokerAndChannel<T>(
    string endpoint, ChannelCredentials credentials, IServiceProvider serviceProvider)
    where T : class
{
    var licenseCode = serviceProvider.GetRequiredService<ILicenseCodeResolver>().GetLicenseCode();

    var channel = new Channel(endpoint, credentials, new ChannelOption(ChannelOptions.MaxSendMessageLength, "1mb").Yield());
    var callInvoker = channel
        .Intercept(
            new ClientTracingInterceptor(
                new ClientTracingInterceptorOptions { RecordMessageEvents = false }))
        .Intercept(new StaticMetadataInterceptor(licenseCode));

    return new InvokerAndChannel<T>(callInvoker, channel);
}

As you can see the pipeline includes StaticMetadataInterceptor which adds license header to the call. Effectively it's also a singleton.
I have multiple parallel calls to various methods of the same gRPC service. And I see that StaticMetadataInterceptor receives same ClientInterceptorContext. At least context.Options.Headers keeps growing, so after several calls it contains the same header several times. Why?

@veblush , why do you guys have button Ask Question and then you ask not to ask questions? =)


GZIP compression for API responses?

Is it possible for an API implemented with OpenAPI Backend to compress responses using GZIP?
I have such an API, and ran this test (going by advice found here https://davidwalsh.name/check-gzip):
$ curl -H "Accept-Encoding: gzip" -v -f http://localhost:9000/api/openadr/0.1/vens
*   Trying <IP_ADDRESS>:9000...
* Connected to localhost (<IP_ADDRESS>) port 9000 (#0)
> GET /api/openadr/0.1/vens HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/8.1.2
> Accept: */*
> Accept-Encoding: gzip
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 1234
< ETag: W/"4d2-7rJdn+XVdF223rzgKmYSISBaAtc"
< Date: Tue, 20 Jun 2023 20:06:52 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5

Notice that Accept-Encoding is set, but the response was not encoded.

I feel that compression is clearly out of scope for OpenAPI backend. It’s pretty standard for most server implementations to support response compression with Accept Encoding headers out of the box. OpenAPI backend is designed to plug into a server / framework, not be the server.


feat(compile): use repo's root as compile path

Motivation: Because ~/.cache/catppuccin never gets cleaned after uninstall
Note: Tested on windows and vanilla vim

This breaks catppuccin-nvim in NixOS, because the code wants to compile into the immutable /nix/store, which is not read-writable. Results in the following error:
Error detected while processing /home/matthew/.config/nvim/init.lua:
E5113: Error while calling lua chunk: /home/matthew/.config/nvim/init.lua:1: Vim(lua):E5108: Error executing lua Vim:E739: Cannot create directory /nix/store/10qpf4mqj6681xz21fn1qqnw7038kzv7-v
im-pack-dir/pack/myNeovimPackages/start/catppuccin-nvim/compiled: read-only file system
stack traceback:
        [C]: in function 'mkdir'
        ...es/start/catppuccin-nvim/lua/catppuccin/lib/compiler.lua:52: in function 'compile'
        ...imPackages/start/catppuccin-nvim/lua/catppuccin/init.lua:79: in function 'compile'
        ...imPackages/start/catppuccin-nvim/lua/catppuccin/init.lua:149: in function 'setup'
        [string ":lua"]:1: in main chunk
        [C]: in function 'cmd'
        /home/matthew/.config/nvim/init.lua:1: in main chunk
stack traceback:
        [C]: in function 'cmd'
        /home/matthew/.config/nvim/init.lua:1: in main chunk
Press ENTER or type command to continue

Ah sorry, I will fix for nix

@MatthewCroughan https://github.com/catppuccin/nvim/pull/414 Can you review this PR?


[Feature] Callable trees

It might be more intuitive to call an expression like so:
options = Options(binary_operators=(+, -, *))
x1, x2 = Node("x1"), Node("x2")

tree = x1 * x2 - x1
X = randn(Float32, 2, 100)

tree(X)

Rather than the current strategy:
eval_tree_array(tree, X, options)

I think at initialization of Options, this could be eval'd into Main so that calling tree on an input array will automatically use the previously-cached options, and directly call eval_tree_array.

Working example on callable-trees.


chore: new profile hud style

This branch contains the re-design of Profile HUD.
WHAT IS NEW AND HOW TO TEST
[Sign in as Guest]

You should be able to see the Claim name button(click on it to  claim your name), and next to it your alias (click on it to edit it). Both have feedback on mouse hover.
When editing your name you should be able to see characters remaining counter.
Sign In button is visible. Click on it to go to wallets selection modal.
Terms and conditions should open a new taband load https://decentraland.org/terms page. Privacy policy  https://decentraland.org/privacy

[Sign in with a wallet account]

If you don't have a claimed name you should be able to see the Claim name button(click on it to  claim your name), and next to it your alias (click on it to edit it). Both have feedback on mouse hover.
If you have a claimed name, the claim name button and the code should disappear. It won't be posible to edit the name anymore.
Wallet address and mana balance are now visible. Click your address to copy. After clicking it a warning message should appear.
Hover Mana container will display a little tooltip explaining what is mana plus a learn more button.

button is to buy more mana.

Terms and conditions should open a new taband load https://decentraland.org/terms page. Privacy policy  https://decentraland.org/privacy
Log out button should load the landing.

Looks good!


Ensure IAM policies are attached only to groups or roles (Reducing access management complexity may in-turn reduce opportunity for a principal to inadvertently receive or retain excessive privileges.)

Possible check to target the following resource

[ ] aws_iam_user_policy_attachment

closing, tracking in spreadsheet

closing, tracking in spreadsheet


undefined method type_casted_binds

This exception always appears...
NoMethodError (undefined method `type_casted_binds' for Debugbar::ActiveRecordEventSubscriber:Class):

/Users/joelmoss/dev/debugbar/lib/debugbar/subscribers/active_record.rb:26:in `sql'
/Users/joelmoss/dev/debugbar/lib/debugbar/engine.rb:87:in `block (3 levels) in subscribe'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:181:in `block in finish'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:26:in `block in iterate_guarding_exceptions'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:25:in `each'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:25:in `iterate_guarding_exceptions'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:125:in `each'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:180:in `finish'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:262:in `block in finish_with_values'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:26:in `block in iterate_guarding_exceptions'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:25:in `each'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:25:in `iterate_guarding_exceptions'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:261:in `finish_with_values'
activesupport (7.1.3) lib/active_support/notifications/fanout.rb:254:in `finish'
activesupport (7.1.3) lib/active_support/notifications/instrumenter.rb:64:in `instrument'
activerecord (7.1.3) lib/active_record/connection_adapters/abstract_adapter.rb:1143:in `log'
activerecord (7.1.3) lib/active_record/connection_adapters/postgresql_adapter.rb:892:in `exec_no_cache'
activerecord (7.1.3) lib/active_record/connection_adapters/postgresql_adapter.rb:872:in `execute_and_clear'
activerecord (7.1.3) lib/active_record/connection_adapters/postgresql/database_statements.rb:64:in `internal_exec_query'
activerecord (7.1.3) lib/active_record/connection_adapters/abstract/database_statements.rb:630:in `select'
activerecord (7.1.3) lib/active_record/connection_adapters/abstract/database_statements.rb:71:in `select_all'
activerecord (7.1.3) lib/active_record/connection_adapters/abstract/query_cache.rb:112:in `block in select_all'
activerecord (7.1.3) lib/active_record/connection_adapters/abstract/query_cache.rb:152:in `block in cache_sql'
activesupport (7.1.3) lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
activerecord (7.1.3) lib/active_record/connection_adapters/abstract/query_cache.rb:147:in `cache_sql'
activerecord (7.1.3) lib/active_record/connection_adapters/abstract/query_cache.rb:112:in `select_all'
activerecord (7.1.3) lib/active_record/querying.rb:62:in `_query_by_sql'
activerecord (7.1.3) lib/active_record/relation.rb:998:in `block in exec_main_query'
activerecord (7.1.3) lib/active_record/relation.rb:1018:in `skip_query_cache_if_necessary'
activerecord (7.1.3) lib/active_record/relation.rb:984:in `exec_main_query'
activerecord (7.1.3) lib/active_record/relation.rb:962:in `block in exec_queries'
activerecord (7.1.3) lib/active_record/relation.rb:1018:in `skip_query_cache_if_necessary'
activerecord (7.1.3) lib/active_record/relation.rb:956:in `exec_queries'
goldiloader (5.2.0) lib/goldiloader/active_record_patches.rb:40:in `exec_queries'
activerecord (7.1.3) lib/active_record/relation.rb:742:in `load'
activerecord (7.1.3) lib/active_record/relation.rb:264:in `records'
activerecord (7.1.3) lib/active_record/relation/finder_methods.rb:572:in `find_take'
activerecord (7.1.3) lib/active_record/relation/finder_methods.rb:121:in `take'
activerecord (7.1.3) lib/active_record/relation/finder_methods.rb:520:in `find_one'
activerecord (7.1.3) lib/active_record/relation/finder_methods.rb:499:in `find_with_ids'
activerecord (7.1.3) lib/active_record/relation/finder_methods.rb:92:in `find'
/Users/joelmoss/dev/clients/harleytherapy/hue/app/controllers/concerns/admin/auth.rb:14:in `current_administrator'
/Users/joelmoss/dev/clients/harleytherapy/hue/app/controllers/concerns/admin/auth.rb:25:in `authenticate_administrator!'
activesupport (7.1.3) lib/active_support/callbacks.rb:403:in `block in make_lambda'
activesupport (7.1.3) lib/active_support/callbacks.rb:183:in `block (2 levels) in halting_and_conditional'
actionpack (7.1.3) lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>'
activesupport (7.1.3) lib/active_support/callbacks.rb:184:in `block in halting_and_conditional'
activesupport (7.1.3) lib/active_support/callbacks.rb:598:in `block in invoke_before'
activesupport (7.1.3) lib/active_support/callbacks.rb:598:in `each'
activesupport (7.1.3) lib/active_support/callbacks.rb:598:in `invoke_before'
activesupport (7.1.3) lib/active_support/callbacks.rb:119:in `block in run_callbacks'
activesupport (7.1.3) lib/active_support/callbacks.rb:141:in `run_callbacks'
actionpack (7.1.3) lib/abstract_controller/callbacks.rb:258:in `process_action'
actionpack (7.1.3) lib/action_controller/metal/rescue.rb:25:in `process_action'
actionpack (7.1.3) lib/action_controller/metal/instrumentation.rb:74:in `block in process_action'
activesupport (7.1.3) lib/active_support/notifications.rb:206:in `block in instrument'
activesupport (7.1.3) lib/active_support/notifications/instrumenter.rb:58:in `instrument'
activesupport (7.1.3) lib/active_support/notifications.rb:206:in `instrument'
actionpack (7.1.3) lib/action_controller/metal/instrumentation.rb:73:in `process_action'
actionpack (7.1.3) lib/action_controller/metal/params_wrapper.rb:261:in `process_action'
activerecord (7.1.3) lib/active_record/railties/controller_runtime.rb:32:in `process_action'
actionpack (7.1.3) lib/abstract_controller/base.rb:160:in `process'
actionview (7.1.3) lib/action_view/rendering.rb:40:in `process'
actionpack (7.1.3) lib/action_controller/metal.rb:227:in `dispatch'
actionpack (7.1.3) lib/action_controller/metal.rb:309:in `dispatch'
actionpack (7.1.3) lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
actionpack (7.1.3) lib/action_dispatch/routing/route_set.rb:32:in `serve'
actionpack (7.1.3) lib/action_dispatch/routing/mapper.rb:21:in `block in <class:Constraints>'
actionpack (7.1.3) lib/action_dispatch/routing/mapper.rb:51:in `serve'
actionpack (7.1.3) lib/action_dispatch/journey/router.rb:51:in `block in serve'
actionpack (7.1.3) lib/action_dispatch/journey/router.rb:131:in `block in find_routes'
actionpack (7.1.3) lib/action_dispatch/journey/router.rb:124:in `each'
actionpack (7.1.3) lib/action_dispatch/journey/router.rb:124:in `find_routes'
actionpack (7.1.3) lib/action_dispatch/journey/router.rb:32:in `serve'
actionpack (7.1.3) lib/action_dispatch/routing/route_set.rb:882:in `call'
omniauth (2.1.2) lib/omniauth/strategy.rb:202:in `call!'
omniauth (2.1.2) lib/omniauth/strategy.rb:169:in `call'
omniauth (2.1.2) lib/omniauth/builder.rb:44:in `call'
prosopite (1.4.2) lib/prosopite/middleware/rack.rb:10:in `call'
rack (3.0.9) lib/rack/tempfile_reaper.rb:20:in `call'
rack (3.0.9) lib/rack/etag.rb:29:in `call'
rack (3.0.9) lib/rack/conditional_get.rb:31:in `call'
rack (3.0.9) lib/rack/head.rb:15:in `call'
actionpack (7.1.3) lib/action_dispatch/http/permissions_policy.rb:36:in `call'
actionpack (7.1.3) lib/action_dispatch/http/content_security_policy.rb:33:in `call'
rack-session (2.0.0) lib/rack/session/abstract/id.rb:272:in `context'
rack-session (2.0.0) lib/rack/session/abstract/id.rb:266:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/cookies.rb:689:in `call'
activerecord (7.1.3) lib/active_record/migration.rb:655:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
activesupport (7.1.3) lib/active_support/callbacks.rb:101:in `run_callbacks'
actionpack (7.1.3) lib/action_dispatch/middleware/callbacks.rb:28:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/executor.rb:14:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/actionable_exceptions.rb:16:in `call'
sentry-rails (5.16.1) lib/sentry/rails/rescued_exception_interceptor.rb:9:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
web-console (4.2.1) lib/web_console/middleware.rb:132:in `call_app'
web-console (4.2.1) lib/web_console/middleware.rb:28:in `block in call'
web-console (4.2.1) lib/web_console/middleware.rb:17:in `catch'
web-console (4.2.1) lib/web_console/middleware.rb:17:in `call'
sentry-ruby (5.16.1) lib/sentry/rack/capture_exceptions.rb:13:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
railties (7.1.3) lib/rails/rack/logger.rb:37:in `call_app'
railties (7.1.3) lib/rails/rack/logger.rb:24:in `block in call'
activesupport (7.1.3) lib/active_support/tagged_logging.rb:135:in `block in tagged'
activesupport (7.1.3) lib/active_support/tagged_logging.rb:39:in `tagged'
activesupport (7.1.3) lib/active_support/tagged_logging.rb:135:in `tagged'
activesupport (7.1.3) lib/active_support/broadcast_logger.rb:240:in `method_missing'
railties (7.1.3) lib/rails/rack/logger.rb:24:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/remote_ip.rb:92:in `call'
request_store (1.6.0) lib/request_store/middleware.rb:19:in `call'
/Users/joelmoss/dev/debugbar/lib/debugbar/middlewares/track_current_request.rb:14:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/request_id.rb:28:in `call'
rack (3.0.9) lib/rack/method_override.rb:28:in `call'
rack (3.0.9) lib/rack/runtime.rb:24:in `call'
activesupport (7.1.3) lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/server_timing.rb:59:in `block in call'
actionpack (7.1.3) lib/action_dispatch/middleware/server_timing.rb:24:in `collect_events'
actionpack (7.1.3) lib/action_dispatch/middleware/server_timing.rb:58:in `call'
rails_performance (1.2.1) lib/rails_performance/rails/middleware.rb:42:in `call!'
rails_performance (1.2.1) lib/rails_performance/rails/middleware.rb:38:in `call'
rails_performance (1.2.1) lib/rails_performance/rails/middleware.rb:17:in `call!'
rails_performance (1.2.1) lib/rails_performance/rails/middleware.rb:9:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/executor.rb:14:in `call'
proscenium-0.15.0.beta.1-arm64 (darwin) lib/proscenium/middleware.rb:30:in `call'
rack (3.0.9) lib/rack/etag.rb:29:in `call'
rack (3.0.9) lib/rack/conditional_get.rb:31:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/static.rb:25:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/static.rb:25:in `call'
rack (3.0.9) lib/rack/sendfile.rb:114:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/ssl.rb:79:in `call'
actionpack (7.1.3) lib/action_dispatch/middleware/host_authorization.rb:141:in `call'
railties (7.1.3) lib/rails/engine.rb:536:in `call'
puma (6.4.2) lib/puma/configuration.rb:272:in `call'
puma (6.4.2) lib/puma/request.rb:100:in `block in handle_request'
puma (6.4.2) lib/puma/thread_pool.rb:378:in `with_force_shutdown'
puma (6.4.2) lib/puma/request.rb:99:in `handle_request'
puma (6.4.2) lib/puma/server.rb:464:in `process_client'
puma (6.4.2) lib/puma/server.rb:245:in `block in run'
puma (6.4.2) lib/puma/thread_pool.rb:155:in `block in spawn_thread'

I've had the same issue.
That method, type_casted_binds is part of https://apidock.com/rails/v<IP_ADDRESS>/ActiveRecord/ConnectionAdapters/Quoting/type_casted_binds
When I extend that module in the ActiveRecordEventSubscriber, I get another error about a missing method: filter.
NoMethodError: undefined method `filter' for Debugbar::ActiveRecordEventSubscriber:Class (NoMethodError)

            filtered_params = filter(attribute_name, casted_params[i])

Referenced here: https://github.com/julienbourdeau/debugbar/blob/master/lib/debugbar/subscribers/active_record.rb#L38

Sorry about that! I have copied this directly from the log_subscriber in Rails I forgot2 methods.
I have never got :binds in my events so I never noticed. By any chance, do you know how I could reproduce?

Can anyone please try v0.1.3? 🙏
The issue should be resolved.

I just tried 0.1.3 and got that filter error.


Mismatch between parquet statistics and parts silently drops data

What happened:
Hi,
I've had some difficulty reading in (older) parquet data files, due to not Dask not finding statistics for about 2/3 of the files.
I'm running the following simple code:
df = dask.dataframe.read_parquet(urlpath, engine='fastparquet').persist()

Where urlpath is a glob path that resolves to 5 folders, each with a _metadata and a _common_metadata file, and 17 parquet files (one of the folders has 16).
The total number of partitions should be (17*4)+16 = 84, but df.npartitions returns 36; the other files are not included in the df.
After a bunch of trying to trace the problem through the code, I ended up putting a print here:
https://github.com/dask/dask/blob/7446308083e2f284229a11e9b3e330aa9e73cf96/dask/dataframe/io/parquet/core.py#L347
Which gives me:
len(meta)=0, len(statistics)=36, len(parts)=84, len(index)=15

Due to the zip() on this line:
https://github.com/dask/dask/blob/7446308083e2f284229a11e9b3e330aa9e73cf96/dask/dataframe/io/parquet/core.py#L1143-L1150
Only the first 36 of the parts will be included in df.
What you expected to happen:
It is quite probable that the root problem here occurred back when this data was written (although I have no clue how to investigate that further).
I can also circumvent the problem by passing in gather_statistics=False.
However, I can't see how cases where len(statistics) != len(parts) would return reliable results, and would at least expect a warning here.
Does anyone have pointers on how to further investigate and/or improve Dask's handling of this (exceptional?) case?
Or an explanation on why the current behavior is desired?
Minimal Complete Verifiable Example:
Not possible without the data in question.
Anything else we need to know?:
/
Environment:

Dask version: 2021.11.1
Fastparquet version: 0.7.1 (currently, for reading, unclear which version was used for writing)
Python version: 3.9.7
Operating System: Ubuntu 20.04.3 LTS
Install method (conda, pip, source): Conda

Thanks for writing this up @j-r77 if you are able to create an example where all the data is generated, that would definitely help towards a speedy resolution.
But I'll ping @martindurant and @rjzamora anyways in case they have ideas of how to proceed.

At a quick skim of the report, I imagine indeed we don't test for a case where only some of the data contains statistics. The statistics should be aligned with the "parts", so that you only exclude parts when you can be sure there is no good data there.

Where urlpath is a glob path that resolves to 5 folders, each with a _metadata and a _common_metadata file, and 17 parquet files (one of the folders has 16).

I suspect that passing in a glob with multiple directories is causing the issue here. I am fairly confident that the glob-pattern feature was designed to handle a collection of files (not a collection of datasets).

Not possible without the data in question.

Hopefully this is an exaggeration, or the problem will likely never get fixed ;)  I will try to reproduce the problem with a toy directory-glob example (I have alrady confirmed that leaving statistics out of some files in a single directory does not lead to problems).

#8416 doesn't close this, but it should result in a clear UserWarning when the statistics and partitions are misaligned (and automatically bail on statistics processing).
I have not been able to reproduce a real case where misalignment occurs, but I will be happy to work on a fix for the root cause of this issue once a reproducer becomes available.

Thanks for the update @rjzamora . The data in this case comes from one dataset, but is manually partitioned into the 5 separate folders. Columns are the same in every folder, and the glob works for our situation in most cases. These problems were found in older data that we are trying to reprocess.
The metadata files are a black box for me, but I'd like to understand more about what's going on. Is there any way I can check them for peculiarities? For instance, you tested 'leaving statistics out of some files', but I have no clue how you would do that.

Columns are the same in every folder, and the glob works for our situation in most cases

Thanks for the info! Can you clarify the glob pattern you are using? Is it just something like "root_dir/*/*"?  Also, are there multiple _metadata files, or just one within the root directory?  I ask about the _metadata file count, because there can never be more than one "global" metadata file  for proper read_parquet behavior.  If Dask detects the existence of any one of the _metadata files, it will ignore the others, and will assume that this single file contains all footer metadata for the entire dataset.

The metadata files are a black box for me, but I'd like to understand more about what's going on. Is there any way I can check them for peculiarities? For instance, you tested 'leaving statistics out of some files', but I have no clue how you would do that.

You can always inspect the _metadata file with fastparquet or pyarrow. For example:
import fastparquet
# Use _metadata file to generate a `ParquetFile` object
pf = fastparquet.ParquetFile("your-dataset/_metadata")
# Extract a list of row-group information
row_groups = pf.row_groups
# For each row_group, you can get a list of column-chunk information
column_chunks = row_groups[0].columns
# The column-chunk statistics are stored in `meta_data.statistics`
# for each element in `column_chunks`
print(column_chunks[0].meta_data.statistics)

Example output:
<class 'fastparquet.parquet_thrift.parquet.ttypes.Statistics'>
distinct_count: None
max: b'\x03\x00\x00\x00\x00\x00\x00\x00'
max_value: None
min: b'\x03\x00\x00\x00\x00\x00\x00\x00'
min_value: None
null_count: 0

If statistics were not written for this particular column chunk, the printed value would just be None.  I am actually not sure if fastparquet makes the statistics optional, but pyarrow does have a write_statistics option for write_table.
Regardless of these _metadata/statistics  questions, I would be interested to know if #8416 (just merged) avoids the error originally reported here?

You might try a glob pattern like "root_dir/**.parquet" (or whatever the data files are called).

I am actually not sure if fastparquet makes the statistics optional

Yes!
    stats: True|False|list(str)
        Whether to calculate and write summary statistics. If True (default), do it for
        every column; if False, never do; and if a list of str, do it only for those
        specified columns.

Thanks for the fast responses all! Took me some time to test, but I can confirm that #8416 solves our problem.
With regards to the cause of the statistics mismatch @rjzamora :
I'm passing in a path of the form parquet/year=2021/month=12/day=20/brandcode=*/*.parquet
Every brandcode=x folder has its own _metadata and _common_metadata file, containing a dataset written separately but with the same schema as for the other brandcodes
So in this case, would Dask assume there is no metadata file available? Or take the first one it encounters?
In this concrete example, data from 2 days ago, Statistics are found for alphabetically the 1st, 4th, and 5th brand, while 2nd and 3rd are nowhere to be seen.
I've tried checking the _metadata files with fastparquet, but so far I've not found any differences between any of them, and all seem so have statistics for every file.

Our code looks for a file named "_metadata", but it may well be that it's not getting the right directory root ("parquet/year=2021/month=12/day=20/"), where there is no such file in this case. fastparquet has a root= argument to ParquetFile that can be used for such tricky cases, I'm not immediately certain whether this is exposed in the dask call, or if pyarrow has similar.

When you specify "parquet/year=2021/month=12/day=20/brandcode=*/*.parquet", dask will not find a _metadata file in the list of paths that is generated. Therefore, having multiple _metadata files is probably not the problem (although, Dask does assume that there can only be one of these files when it does find it).

@j-r77 closing this issue since it seems your problem was resolved with #8416 and the advice from @rjzamora and @martindurant


[Snyk] Security upgrade django from 3.2.18 to 3.2.19

This PR was automatically created by Snyk using the credentials of a real user.Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.
Changes included in this PR

Changes to the following files to upgrade the vulnerable dependencies to a fixed version:

requirements.txt

Vulnerabilities that will be fixed
By pinning:

Severity
Priority Score (*)
Issue
Upgrade
Breaking Change
Exploit Maturity

551/1000  Why? Recently disclosed, Has a fix available, CVSS 5.3
Arbitrary File Upload SNYK-PYTHON-DJANGO-5496950
django:  3.2.18 -> 3.2.19
No
No Known Exploit

(*) Note that the real score may have changed since the PR was raised.
Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.
Check the changes in this PR to ensure they won't cause issues with your project.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
🛠 Adjust project settings
📚 Read more about Snyk's upgrade and patch logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

Codecov Report

Merging #357 (3b711dd) into develop (7dd8a23) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           develop     #357   +/-   ##
========================================
  Coverage    77.01%   77.01%
========================================
  Files          137      137
  Lines         7466     7466
========================================
  Hits          5750     5750
  Misses        1716     1716

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

Fixed in 8234d5e


Potentially add some way of pre-declaring inactive timers

Not sold on whether we need this but might be worth investigating

We need this for pausing!!!!!!!

[4e376fef016cf7176556b4de4971ad1531715f36]


Batch downloading

i have a suggestion can batch downloading be added. So we can download whole season with a click rather then clicking each eposide maunally

There are two problems in batch downloading
there is now downloading queuing support I have to add that first and also another problem is  how to automate server selection

You just need to add any one provider for batch download like a uhd movie that is not streaming but downloading fast in your app you add uhd movie batch download Or you can add new server like a https://vadapav.mov/


Prevent completion within comments

This matches rust-analyzer's behavior in comments: completions are avoided completely. Ocassionally the completions can be useful (for example completing a parameter while writing docs for a function) but on balance I think the completions are noisier than they are helpful. Maybe there's room in the long run to be smarter about completions in comments (similar to rust-analyzer#4170) but without that I think it's better to avoid completions in comments entirely.

It makes sense to me, thanks for the contribution!


BoundingBox and NavigateTo not work

Hi guys,
I've a series of pins in the map and I want to zoom only on area covered from pins. I use this function to calculate a bundle box 👍
public static BoundingBox ToBoundingBox(this IList pins)
{
var geometries = pins.Select(x => SphericalMercator.FromLonLat(x.Position.Longitude, x.Position.Latitude)).ToList();
        if (geometries.Count == 0) return null;

        var minX = geometries.Min(g => g.BoundingBox.MinX);
        var minY = geometries.Min(g => g.BoundingBox.MinY);
        var maxX = geometries.Max(g => g.BoundingBox.MaxX);
        var maxY = geometries.Max(g => g.BoundingBox.MaxY);

        return new BoundingBox(minX, minY, maxX, maxY);
    }

and  I call in this way:
var boundingBox = mapView.Pins.ToBoundingBox();
mapView.Navigator.NavigateTo(boundingBox, ScaleMethod.Fit);
and I've this result:

but my expected result should  be :

where is my mistake ?

Did you checked the BoundigBox in between? If yes, what values do you get? What are the converted values for the pins?

And the converted values, which are used to calculate the BoundingBox?

var geometries = pins.Select(x => SphericalMercator.FromLonLat(x.Position.Longitude, x.Position.Latitude)).ToList();

Is there some configuration that I've set ?

Do i neeed to set some info in viewport? how do I do ?

I checked it here and found, that it worked like a charm. So it must be something on your side.
For an image as you showed, the width of the BoundingBox had to be around 70 millions. For your pins you should get the following values for the BoundingBox:
Min = {POINT (1033324.65383779 5012688.29449056)}
Max = {POINT (1872547.26679702 5780452.83896506)}
Width =<PHONE_NUMBER>592323
Height =<PHONE_NUMBER>7449371

If you don't get such values, your part before creating the BoundingBox is buggy. If you get this values, then something with the navigation is wrong.

Thank !

@aprilesalvatore It would be nice if you could explain, what solved your problem.
And if it is solved, could you close the issue?

I've created this functions:
public static void ZoomToBox(this INavigator navigator, Mapsui.Geometries.Point beginPoint, Mapsui.Geometries.Point endPoint)
{
var width = Math.Abs(endPoint.X - beginPoint.X);
var height = Math.Abs(endPoint.Y - beginPoint.Y);
if (width <= 0) return;
if (height <= 0) return;
        var info = DeviceDisplay.MainDisplayInfo;

        var xamarinWidth = (info.Width / info.Density) - 50;
        var xamarinHeight = (info.Height / info.Density) - 50;

        Mapsui.Utilities.ZoomHelper.ZoomToBoudingbox(beginPoint.X, beginPoint.Y, endPoint.X, endPoint.Y,
            xamarinWidth, xamarinHeight, out var x, out var y, out var resolution);

        navigator.NavigateTo(new Mapsui.Geometries.Point(x, y).BoundingBox.Centroid, resolution);
    }

I don't understand why such a helper function should be needed. Either there is a bug in our code or you do something wrong in your initial sample (but I don't see how).
Since it is important to me that the Navigation code work correctly I tested it by modifying the AsyncLayer sample and used a version of your initial extension method. It works as expected. This is the whole sample:
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using Mapsui.Geometries;
using Mapsui.Layers;
using Mapsui.Projection;
using Mapsui.Providers;
using Mapsui.Styles;
using Mapsui.UI;
using Mapsui.Utilities;
using Newtonsoft.Json;

// ReSharper disable UnusedAutoPropertyAccessor.Local

namespace Mapsui.Samples.Common.Maps
{
    public class AsyncFetchSample : ISample
    {
        public string Name => "Async fetch";
        public string Category => "Special";

        public void Setup(IMapControl mapControl)
        {
            mapControl.Map = CreateMap();
        }

        public static Map CreateMap()
        {
            var map = new Map();

            map.CRS = "EPSG:3857";
            map.Transformation = new MinimalTransformation();

            map.Layers.Add(OpenStreetMap.CreateTileLayer());
            var cities = GetCitiesFromEmbeddedResource();
            map.Layers.Add(CreateAsyncLayer(cities));
            map.Home = n => n.NavigateTo(ToBoundingBox(cities));

            return map;
        }

        public static BoundingBox ToBoundingBox(IEnumerable<IFeature> pins)
        {
            var geometries = pins.Select(x => SphericalMercator.FromLonLat(x.Geometry.BoundingBox.Centroid.X, x.Geometry.BoundingBox.Centroid.Y)).ToList();

            if (geometries.Count == 0) return null;

            var minX = geometries.Min(g => g.BoundingBox.MinX);
            var minY = geometries.Min(g => g.BoundingBox.MinY);
            var maxX = geometries.Max(g => g.BoundingBox.MaxX);
            var maxY = geometries.Max(g => g.BoundingBox.MaxY);

            return new BoundingBox(minX, minY, maxX, maxY);
        }

        private static Layer CreateAsyncLayer(IEnumerable<IFeature> cities)
        {
            // The Layer type fetches data asynchronous (it should be renamed).
            // This impacts the speed with which data is shown after panning.
            // This Delayer class is used to filter a continuous stream of
            // RefreshData requests.
            return new Layer
            {
                Name = "Async Layer",
                IsMapInfoLayer = true,
                DataSource = new MemoryProvider(cities) { CRS = "EPSG:4326" },

                Style = CreateBitmapStyle()
            };
        }

        private static IEnumerable<IFeature> GetCitiesFromEmbeddedResource()
        {
            var path = "Mapsui.Samples.Common.EmbeddedResources.congo.json";
            var assembly = typeof(PointsSample).GetTypeInfo().Assembly;
            var stream = assembly.GetManifestResourceStream(path);
            var cities = DeserializeFromStream<City>(stream);

            return cities.Select(c =>
            {
                var feature = new Feature();
                var point = SphericalMercator.FromLonLat(c.Lng, c.Lat);
                feature.Geometry = new Point(c.Lng, c.Lat);
                feature["name"] = c.Name;
                feature["country"] = c.Country;
                return feature;
            }).ToList();
        }

        private class City
        {
            public string Country { get; set; }
            public string Name { get; set; }
            public double Lat { get; set; }
            public double Lng { get; set; }
        }

        public static IEnumerable<T> DeserializeFromStream<T>(Stream stream)
        {
            var serializer = new JsonSerializer();

            using (var sr = new StreamReader(stream))
            using (var jsonTextReader = new JsonTextReader(sr))
            {
                return serializer.Deserialize<List<T>>(jsonTextReader);
            }
        }

        private static SymbolStyle CreateBitmapStyle()
        {
            // For this sample we get the bitmap from an embedded resource
            // but you could get the data stream from the web or anywhere
            // else.
            var path = "Mapsui.Samples.Common.Images.home.png"; // Designed by Freepik http://www.freepik.com
            var bitmapId = GetBitmapIdForEmbeddedResource(path);
            var bitmapHeight = 176; // To set the offset correct we need to know the bitmap height
            return new SymbolStyle { BitmapId = bitmapId, SymbolScale = 0.20, SymbolOffset = new Offset(0, bitmapHeight * 0.5) };
        }

        private static int GetBitmapIdForEmbeddedResource(string imagePath)
        {
            var assembly = typeof(PointsSample).GetTypeInfo().Assembly;
            var image = assembly.GetManifestResourceStream(imagePath);
            return BitmapRegistry.Instance.Register(image);
        }
    }
}


[zookeeper] 3.4.9 -> 3.4.10

Hi,
zookeeper 3.4.10 has been released as the latest stable. Many frameworks[0] & datastores[1] using zk has upgraded their client library to 3.4.10, hence it will be good to build an k8szk image of 3.4.10. This PR adds the required changes.
[0] - https://github.com/lagom/lagom/pull/986
[1] - https://github.com/apache/kafka/commit/c2d17c830ca309a4e3989981a47790209955273c

This change is

/assign @enisoc

Hi, I am waiting for using zk-3.4.10, will be thankful if you can review it soon please.

/lgtm

@kow3ns Are you going to build and push the image?

@enisoc @kow3ns i was expecting the new image to be available automatically - is building and pushing the image a manual step ?

It has to be done manually. I went ahead and pushed it just now.

Great !! thank you.


Cloudflare Workers Handler

What is the feature you are proposing?
Hi, @yusukebe
How about the idea of creating a cloudflare worker handlerlike cloudflare pages handler to absorb additional fields except for fetch in an default exported object?
Common Part

import { Hono } from "hono";

type Environment = {
  readonly ERROR_QUEUE: Queue<Error>;
  readonly ERROR_BUCKET: R2Bucket;
};

const app = new Hono<{
  Bindings: Environment;
}>();

app.get("/", (c) => {
  if (Math.random() < 0.5) {
    return c.text("Success!");
  }
  throw new Error("Failed!");
});

Without Handler
export default {
  fetch: app.fetch,
  async queue(batch: MessageBatch<Error>, env: Environment): Promise<void> {
    let file = "";
    for (const message of batch.messages) {
      const error = message.body;
      file += error.stack || error.message || String(error);
      file += "\r\n";
    }
    await env.ERROR_BUCKET.put(`errors/${Date.now()}.log`, file);
  },
};

Proposal 1
import { handle } from "hono/cloudflare-workers";

export default handle(app, {
  queue: async (batch, env) => {
    let file = "";
    for (const message of batch.messages) {
      const error = message.body;
      file += error.stack || error.message || String(error);
      file += "\r\n";
    }
    await env.ERROR_BUCKET.put(`errors/${Date.now()}.log`, file);
  }
});

Proposal 2
import { handle } from "hono/cloudflare-workers";

const handler = handle(app)

handler.queue = async (batch, env) => {
  let file = "";
  for (const message of batch.messages) {
    const error = message.body;
    file += error.stack || error.message || String(error);
    file += "\r\n";
  }
  await env.ERROR_BUCKET.put(`errors/${Date.now()}.log`, file);
};

export default handler

handler takes over env from the Hono instance
Can absorb various default export extensions

https://github.com/honojs/examples/issues/98

Hi @sor4chi
If this purpose only for Type inference, you can use ExportedHandler exported from @cloudflare/workers-types:
import type { ExportedHandler } from '@cloudflare/workers-types'

// ...

export default {
  fetch: app.fetch,
  queue: async (batch, env) => {},
} as ExportedHandler<{}>

Oh, it certainly could be substituted with an existing one. Thank you very much.


showcase how to test operator

Actually, tests are E2E integration tests and require external dependencies (database instance, k8s cluster). In some CI environment, it may not be possible to do so.
We should be able to unit test with mock for those dependencies. This would allow to add more tests as we have more control on the state of those dependencies.
See related works on quarkus extension to unit test mysql schema operator : https://github.com/quarkiverse/quarkus-operator-sdk/pull/201/files#diff-43e4c3f21bc716a010119d458691c0a57aeafe813c6606e805c7d63a62da3e17

I was looking at the provided E2E tests and I found a good foundation for what I was trying to build.
Specifically I'm completely happy to have "real" resources in the cluster (as opposed to mocks) as long as the lifecycle of such resources is managed by the testing infrastructure.
Those are the "missing features" I found:

decide if the resources are created on each test or on each testing class (e.g. beforeEach/beforeAll)
spin additional infrastructure and manage the lifecycle of it
using a simple flag being able to run the operator locally (in-process) or remotely (in-kubernetes)

I just opened: https://github.com/java-operator-sdk/java-operator-sdk/pull/891 to show-case those.

I was looking at the provided E2E tests and I found a good foundation for what I was trying to build. Specifically I'm completely happy to have "real" resources in the cluster (as opposed to mocks) as long as the lifecycle of such resources is managed by the testing infrastructure.

I agree even if I think that it works well for small E2E test campaign. When the number of tests augment, it become longer and longer to run them all. And usually, there are not so independent as unit tests and you can't just run one test method. Also, the arrange part can be tricky, especially for very edge cases.
What I'm not sure is what should we test the framework doesn't already guarantee us. I mean, all the logic of calling reconcile is done by the framework and should be well tested. Does unit testing reconcile method quicker and easier ?
In my case, the operator is actually operating the deployment of an application that need a lot of other external components and operators to be able to observe READY state. Not only, arranging the infrastructure is difficult but waiting to observe the result can take quite a long time. And all of those dependents component are already tested on their own. What should be pretty isolated starts to be a project on its own.
If the operator is just creating a deployment that start a pod in a millisecond, obviously it is pretty easy to test it will eventually succeed. But how do you test the case when the deployment failed due to some infrastructure errors (k8s quota reached, k8s api down, mysql down, etc...). Introducing chaos could be very challenging.
I'm not saying E2E are useless, they definitively need to exists. But what can be offered to a team developping a more complex operator to be able to test it without installing all the components of the real platform.

Those are the "missing features" I found:

decide if the resources are created on each test or on each testing class (e.g. beforeEach/beforeAll)
spin additional infrastructure and manage the lifecycle of it
using a simple flag being able to run the operator locally (in-process) or remotely (in-kubernetes)

I just opened: #891 to show-case those.

I think a way to start/stop the operator between test to be able to arrange the infrastructure while it is not running would be needed as well.

For an exemple of complexity, you can look at https://github.com/nais/naiserator/blob/master/pkg/synchronizer/synchronizer_test.go
It uses the go EnvTest which basically is just k8s api server and etcd, pretty much what fabricio mock server is offering.

I think it can be closed, there is enough samples and the JUnit extension fo integration tests. And dependent resources allow for better unit testing.


Config Prettier

Hi @jiri132,
for the code styling I can config prettier for this project if you want to
Please assign this to me

Hey @N3v1,
I have never used the coding style prettier I have heard of it many times before but I don't really know what it stands for.
I'll assign you to this so we can take a look at this!