Main module blending

First 'steps' towards a main module for a blended nowcasts-NWP forecast following the original STEPS approach as described in Bowler et al. (2006) and Seed et al. (2013).

[x] Working main module
[x] Tests
[x] Examples
[x] Documentation
[x] Probability matching following a different method
[x] Implementation of blending weights as in Bowler et al. (2006)
[x] Implementation of blending weights as in Seed et al. (2013)
[x] Fixes to prability matching method, masking method and noisy output
[x] blending.utils tests

Part of issue #212

Question: the V_models has no time dimension as it represents the velocity field of the NWP precipitation forecast at the valid time = the start of the nowcast (taking into account the last 3 rainfall fields from NWP). I wonder if it would be feasible to have a velocity field per lead time instead. This can be computed ahead of time from the NWP forecast.
By the way, @pulkkins, is there anything we need to take into account when applying an optical flow algorithm on NWP output (5-minute accumulations or longer) instead of on the near-instantaneous precipitation intensities measured by radar?

@loforest and @dnerini, thanks a lot for your inputs so far!
Regarding @loforest's comments, the calculation of r_e is described in the technical report of Bowler et al. (2004):

We have implemented the same, see also my response to @dnerini's comment.
Regarding the implementation of e.g. adjust_lag2_corrcoef2, thanks for mentioning this! It is applied to the extrapolation component, in the same way as in the pysteps.nowcasting.steps code, but not to the NWP component. I'll have to look into that.

@dnerini (and @pulkkins), after some tests, I think I agree with you. Indeed, we are left with the problem of the boundaries when advecting the noise cascade. I do like the idea of the mirroring the boundaries, perhaps worth the try?

Very good, I also thing this currently looks like the most promising approach. I not sure exactly how yet (need to look at the details in the code), but I feel there should be an elegant solution to this, something like wrapping around the indices of the advection routine when these exceed the size of the domain, if you see what I mean? @aperezhortal might have some good inputs in this too

@RubenImhoff I just realized that the routine map_coordinates used in semilagrangian already offers a mode="reflect" option, which sounds very much like what we have in mind... maybe you could have a try at it? If we are lucky, this might be as simple as changing few optional keyword arguments ....

Good to know, I'll have a look at it!

@dnerini, this seems to work well! "grid-wrap" did not work, but "wrap" does. I do not know why the "grid-..." options do not work, but it must be based on the input we're providing (I have to admit that my scipy map_coordinates knowledge is not sufficient for that, haha).
Anyway, the results go from (lead time is 3h; zoomed in on the top of the composite, where no new noise was coming in at the top right corner - movement is south/southwest):

to:

I'll do some more tests to see if everything works the way we want (also for less symmetrical rainfall fields) and then commit the changes here.

Great @RubenImhoff! Does it improve the scores in any way or is the effect too far outside the composite to be verified?

Hi @dnerini and @ladc,
The last addition, mirroring the noise, does not change the skill scores, because it is all outside the radar domain. I can imagine that there where the radar domain is closer to the edges of the extent (or for other radar domains), the effect is more pronounced.
Below an animation of one full forecast.

The radar domain edge, where the blended nowcast changes into a blended NWP-noise forecast, remains visible.  I'm thinking of a slightly smoother transition, maybe a simple linear transition in space from the extrapolation-NWP-noise forecast to NWP-noise forecast? Do you think something is needed for this and do you have any ideas for it?

Thanks @RubenImhoff, fantastic work!
Personally I don't mind too much the discontinuity you mention, it's anyway being smoothed out by the noise itself with time.
Instead, what do you think are the next important steps? Is there any important component that is still missing? And what about the performance of the code? Do you plan to conduct a more in-depth verification analysis with multiple events or is it too early?

Thanks, @dnerini!
It's true that this discontinuity is not too worrying. Plus your region of interest is likely within the radar domain, so you might not even notice it. Anyway, it would be more something from a aesthetics perspective.
The last step left is to implement the weights calculation as in Seed et al. (2013), based on the covariance matrix. I will try to do this the coming days. I think that after that, we're ready for a good review of the code. Once that has finished, we are planning to, indeed, conduct a more in-depth hydrometeorological verification analysis with multiple events for one (or a couple of) catchments.

Question: the V_models has no time dimension as it represents the velocity field of the NWP precipitation forecast at the valid time = the start of the nowcast (taking into account the last 3 rainfall fields from NWP). I wonder if it would be feasible to have a velocity field per lead time instead. This can be computed ahead of time from the NWP forecast. By the way, @pulkkins, is there anything we need to take into account when applying an optical flow algorithm on NWP output (5-minute accumulations or longer) instead of on the near-instantaneous precipitation intensities measured by radar?

I do not see any major issues. However, the default thresholds need to be adjusted because I expect the NWP output to be smoother (and have smaller gradients) than radar measurements.

mmmmh right we would loose the advantage of blending with respect to boundaries! Forget what I said then relaxed
But I still think that the refiltering step might be obsolete in our case. So we are left with the problem of the boundaries when advecting the noise cascade.
Another idea: add a buffer to the noise cascade before advection? By using a larger domain we would at least limit the problem.
Another idea: with noise we could also simply "mirror" the boundaries: what goes out from one side, goes in from the opposite one... In practice this could be implemented as an option in the advection scheme. What do you think @spulkkin ?

When applying mirroring, you assuming continuity when wrapping over the border. I would not do that. Adding a buffer to the noise cascade sounds like a good idea.

@pulkkins, thanks for your comments!
Regarding the adjustment of the thresholds for the NWP models, I'll put that on my list. Makes sense to have a look at that.
Regarding the noise mirroring, do you know if this 'buffer' functionality is in some way already part of scipy map_coordinates?

mmmmh right we would loose the advantage of blending with respect to boundaries! Forget what I said then relaxed
But I still think that the refiltering step might be obsolete in our case. So we are left with the problem of the boundaries when advecting the noise cascade.
Another idea: add a buffer to the noise cascade before advection? By using a larger domain we would at least limit the problem.
Another idea: with noise we could also simply "mirror" the boundaries: what goes out from one side, goes in from the opposite one... In practice this could be implemented as an option in the advection scheme. What do you think @spulkkin ?

When applying mirroring, you assuming continuity when wrapping over the border. I would not do that. Adding a buffer to the noise cascade sounds like a good idea.

I see your point, but I still think that in this particular case the approach would be justified since it applies to noise only and mostly to the edges of the domain...
Adding a buffer would be more expensive computationally and be more involved code-wise.

@pulkkins, thanks for your comments!
Regarding the adjustment of the thresholds for the NWP models, I'll put that on my list. Makes sense to have a look at that.
Regarding the noise mirroring, do you know if this 'buffer' functionality is in some way already part of scipy map_coordinates?

I don't think that the buffer functionality we need is implemented in scipy.map_coordinates. What I was thinking is to generate the noise field in a larger domain and crop it to the original domain after advection. But as @dnerini said, that could be tricky to implement.

@pulkkins, thanks for your comments!
Regarding the adjustment of the thresholds for the NWP models, I'll put that on my list. Makes sense to have a look at that.
Regarding the noise mirroring, do you know if this 'buffer' functionality is in some way already part of scipy map_coordinates?

I don't think that the buffer functionality we need is implemented in scipy.map_coordinates. What I was thinking is to generate the noise field in a larger domain and crop it to the original domain after advection. But as @dnerini said, that could be tricky to implement.

Clear, also thanks for your input, @dnerini. Would it be an idea to stick to the mirroring for now, as this functionality is only for outside the domain (unless you're radar domain covers or almost cover the entire extent..), and make an issue for further improvement of this?

yes, if @pulkkins agress, I think we can leave the current approach and
focus on the remaining issues.
Il giorno gio 11 nov 2021 alle ore 08:21 Ruben Imhoff <
@.***> ha scritto:

@pulkkins https://github.com/pulkkins, thanks for your comments!
Regarding the adjustment of the thresholds for the NWP models, I'll put
that on my list. Makes sense to have a look at that.
Regarding the noise mirroring, do you know if this 'buffer' functionality
is in some way already part of scipy map_coordinates?
I don't think that the buffer functionality we need is implemented in
scipy.map_coordinates. What I was thinking is to generate the noise field
in a larger domain and crop it to the original domain after advection. But
as @dnerini https://github.com/dnerini said, that could be tricky to
implement.
Clear, also thanks for your input, @dnerini https://github.com/dnerini.
Would it be an idea to stick to the mirroring for now, as this
functionality is only for outside the domain (unless you're radar domain
covers or almost cover the entire extent..), and make an issue for further
improvement of this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pySTEPS/pysteps/pull/233#issuecomment-966051607, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AC3J3YYXCDIRSKA6RVHS5BLULNVI5ANCNFSM5CWJDUZA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--
Daniele

After some consultation with @cvelascof and also Alan Seed about how the weights in Seed et al. (2013, WRR) were implemented, I've made a first try to implement it in the blending code as well. So far so good, it seems! Below for two runs (the same ones as you've seen earlier in the conversation) the CRPS.
The implementation of the weights method is as such that the user can choose which one to use (BPS2006 or SPN2013), with for now the SPN2013 method as default.
Note that the implementation does not exactly follow the SPN2013 paper, as the notation in that paper turned out to be partly erroneous. However, we've decided to continue with the normalized variance-covariance matrix, filled with the cross-correlations between the models.

Hi @RubenImhoff , sorry for the late feedback... It's really exciting what you achieved! So this now means that you managed to include all the features needed for this module and we should get started on code review? Or are you planning to run a validation analysis first with some more precipitation cases?

Thanks, @dnerini! Yes, I'm working on some tests (call it a small validation) to also double check that all functionalities work. After that, I would say, it is time for the code review!

I think we're ready for the code review! :)

Very good, @RubenImhoff, let's have a look then !
Just a comment for you and @ladc: I fear that the original plan with pysteps v2 will take some more time than expected, in particular for the refactoring with xarray and the object-oriented approach for forecasting methods.
To avoid postponing for too long the release of the blending module, one option could be to merge it into master and release a new v1.6. Although you already merged into this branch already several changes for v2 (e.g. xarray-utils), it should still be relatively easy to isolate just your contributions for the blending module, perhaps in a clean branch?
What do you think?

That is not a bad idea, Daniele. How much time do you think we still need for v2?
I'm thinking of our idea (at least @ladc and mine - but not limited to) to work towards a blending paper, including the open-source implementation of the STEPS code in pySTEPS and a hydrometeorological analysis. As long as we have a release ready by that time, it should suffice.
@ladc, what do you think?
Only changes I can think of, is that the importers and subsequent testing schemes are (partly) based on the xarray implementation.

How much time do you think we still need for v2?

Difficult to say. Unfortunately I don't have a lot of time to dedicate to this, and the same appears to be true for other core developers. We might have a pre-release ready by end of summer 2022, but this is optimistic and won't be stable.
So yes, I recommend to make this PR suitable for v1 (or cherry-pick from here into a new branch, as you prefer). I will anyway start reviewing your code here.

Sounds like the best decision then! Thanks for reviewing and I'll start thinking about how to nicely put this, together with the steps_blending and linear blending branches, into v1.

pystepsrc
You have included a new path_workdir. I'm not against, but what's its purpose? Perhaps you could include a short comment to help understand its scope?

Good point. We implemented this as a place to store the decomposed NWP fields and NWP velocity fields, but seeing how it used now, we can reach the same with 'just' the path_outputs in pystepsrc. @ladc, what do you think?

In the blended forecast, I have the impression that something is a bit off with the NWP values, as I observed a strong increase of the intensities with leadtime. I anyway still need to have a closer look at your implementation of the masking and probability matching.

I completely forgot to have a new look at the examples forecast after the latest changes. The minor code changes will follow soon. Indeed, something seems to go wrong with either the masking or prob_matching.. I don't see the same effect (at least it is not as clear if it is there) in the test forecasts with the RMI data, so I wonder what goes wrong.

In the blended forecast, I have the impression that something is a bit off with the NWP values, as I observed a strong increase of the intensities with leadtime. I anyway still need to have a closer look at your implementation of the masking and probability matching.

I completely forgot to have a new look at the examples forecast after the latest changes. The minor code changes will follow soon. Indeed, something seems to go wrong with either the masking or prob_matching.. I don't see the same effect (at least it is not as clear if it is there) in the test forecasts with the RMI data, so I wonder what goes wrong.

While working on the tests for pysteps.blending.utils, I came across an error in pysteps.blending.utils. In the function stack_cascades, the mean and sigma could be set to 0.0 and 1.0 (I don't know why), which eventually gives wrong values after blending with NWP. As this only happened with the NWP component(s), the prob_matching was compensating for this and that is why we saw the effect with increasing time in the examples scripts. However, the changes fix the problems (nevertheless, masking and prob_matching still require a critical look)!  Also for the RMI test scripts the results look better (I'll upload a figure soon)!

Very nice @RubenImhoff! The animation is very promising!
I'm having a closer look at the probability matching. Do I understand it right that you basically perform a linear blending of radar extrapolation and NWP based on the weights from the second cascade level in order to build the reference field for the probability matching? This is now a bit of a detail, but the problem with this is that you would loose the peak values, especially when both weights are around 0.5.
An alternative approach would be a simple resampling: for each grid point you sample either the radar or the NWP value (and this could be based on the same weights you are already using). This way you would build a new target distribution that is not smoother than the original radar and NWP fields.
I have some code for this (I used it in the EnKF approach). Should I give it a try to implement it in your code?

An alternative approach would be a simple resampling: for each grid point you sample either the radar or the NWP value (and this could be based on the same weights you are already using). This way you would build a new target distribution that is not smoother than the original radar and NWP fields.
I have some code for this (I used it in the EnKF approach). Should I give it a try to implement it in your code?

Great idea, @dnerini, and thanks if you want to implement that!

Yesterday, @ladc, @cvelascof and me had a nice talk with Alan Seed about the blending code and the weights implementation. He was happy to see that we're making this open source in pysteps and thought that the results so far look promising. Based on our talk, some main suggestions from Alan (and things to work on probably in the coming days):

Have a look at default skill values for cases where the weights implementation either becomes unstable or for when there is hardly any rain in the domain. This should ensure that the noise weights always increases with time and the nowcast weight decreases.
Have a look at the combined weight term of the nowcast and NWP component (prior to the noise weight estimation) and make sure this always decreases with lead time.
For the transformations, perhaps apply a log sine transformation which can include zeroes, thereby making the variance of the individual components more representative than with a threshold set (Alan said it is better not to mask the radar and NWP data with a given threshold prior to the blending). This is something we could implement. I wonder what the optical flow methods (and perhaps also the FFT) need, maybe we should at least keep a threshold for the motion estimations, but not for the remaining part of the blending?

pystepsrc
You have included a new path_workdir. I'm not against, but what's its purpose? Perhaps you could include a short comment to help understand its scope?

Good point. We implemented this as a place to store the decomposed NWP fields and NWP velocity fields, but seeing how it used now, we can reach the same with 'just' the path_outputs in pystepsrc. @ladc, what do you think?

Hi guys, sorry for the super late reply. I think we made the distinction because the one path is aimed at files for internal use, including input and output (also e.g. the weights file, pre-decomposed NWP, ...) and the other is where we store only the output products for users. But I also understand if you would prefer to reduce the number of different directories or parameters in the config file, and also store these in path_outputs.

That is not a bad idea, Daniele. How much time do you think we still need for v2? I'm thinking of our idea (at least @ladc and mine - but not limited to) to work towards a blending paper, including the open-source implementation of the STEPS code in pySTEPS and a hydrometeorological analysis. As long as we have a release ready by that time, it should suffice.
@ladc, what do you think?

The xarray overhaul (among other things) is indeed not a small task, and it takes some manpower to work on pysteps v2. If you think it's doable to integrate the blending into pysteps v1.6, then it might be better to go for that option indeed. I hope this does not create too much duplicate work?

Only changes I can think of, is that the importers and subsequent testing schemes are (partly) based on the xarray implementation.

The xarray overhaul (among other things) is indeed not a small task, and it takes some manpower to work on pysteps v2. If you think it's doable to integrate the blending into pysteps v1.6, then it might be better to go for that option indeed. I hope this does not create too much duplicate work?

It should be doable in my opinion. To simplify the task, I suggest to exclude the nwp importers from the branch and publish them as separate pysteps plugins: https://pysteps.readthedocs.io/en/latest/developer_guide/importer_plugins.html
The rest of the code is already very much in line with v1 I believe? I'd also suggest a minimal refactoring to reorganize the main method in few components (I'd be happy to give it a try).
If we all agree, I'd start refactoring this PR so that it can be merged into master rather than pysteps-v2.
@RubenImhoff very cool that you could exchange with Alan about your work! One comment from my side on your list of improvements: ad this point I'd try to limit adding complexity to this implementation. We want a robust base blending method that works, possibly as close as possible to the reference publication. Further improvements can come incrementally later on (with your publication? ;-)), and as separate modules, to avoid adding too much complexity to the base method. For example: climatological skill. This is an important component for an operational system (it has to run in any situation), but I would not include it in the base method. Instead, we can make sure that the method can accept is as an optional argument, and leave to the user to compute it in his/her implementation.

If we all agree, I'd start refactoring this PR so that it can be merged into master rather than pysteps-v2.
@RubenImhoff very cool that you could exchange with Alan about your work! One comment from my side on your list of improvements: at this point I'd try to limit adding complexity to this implementation. We want a robust base blending method that works, possibly as close as possible to the reference publication. Further improvements can come incrementally later on (with your publication? ;-)), and as separate modules, to avoid adding too much complexity to the base method. For example: climatological skill. This is an important component for an operational system (it has to run in any situation), but I would not include it in the base method. Instead, we can make sure that the method can accept is as an optional argument, and leave to the user to compute it in his/her implementation. Same idea with the log sine transformation: would be nice to experiment with it, but this can be included in a separate PR.

Sounds good, Daniele, and I agree! I'll upload a couple of minor changes and implement your idea to make the decomposed NWP input optional (so, you could also start with a non-decomposed NWP forecast) and then we're ready to go.

I've just merged Daniele's great refactoring work. Should we continue the review here, or should I merge this branch in the steps_blending branch and renew that for pysteps 1.6 (and review from there)?

I've just merged Daniele's great refactoring work. Should we continue the review here, or should I merge this branch in the steps_blending branch and renew that for pysteps 1.6 (and review from there)?

Very good. Yes, please merge into steps_blending and we'll continue from there for the integration into master


boot.initrd.clevis.devices.<name>.secretFile leaks the secret into nix store

Issue description
boot.initrd.clevis.devices.<name>.secretFile has a type of  path in NixOS 24.05. This means that every *.jwe file used with this option ends up in world-readable nix store.
Why this seems disturbing?
To the best of my knowledge, when using a Tang pin, the *.jwe file contains everything needed for an interaction with the Tang server that results in key decryption.
To validate this thesis, I've set up an experiment with 2 machines: S that acted as a Tang server and C that used Clevis to decrypt a LUKS partition on boot. After C performed successful unattended boot, I've logged in as an unprivileged user and run:
find /nix/store -name '*.jwe' -execdir clevis decrypt {} ';'

It printed the LUKS passphrase. I also tried copying the *.jwe file to another machine (that was able to contact the Tang server), run clevis decrypt < *.jwe there, and also recovered LUKS passphrase.
Recommendations
In the Security Considerations subsection of Tang's readme it is said that:

it is of utmost importance that the client protect cJWK from prying eyes. This may include device permissions, filesystem permissions, security frameworks (such as SELinux) or even the use of hardware encryption such as a TPM.

Looking at the above experiment, it seems to me that the cJWK can be read from the *.jwe file; therefore, this file should be treated as a proper secret and not included in the nix store.

You should set boot.initrd.clevis.devices.foobar.secretFile = "/path/to/file" (note the quotes) in order not to put it in the Nix store.


reconfigure slack alerts for Heroku service updates

This was easy to do. I added an email trigger on Zapier and then connected it to the Slack channel. You can test the trigger right away and see confirmation messages if you have to confirm an email subscription, so it should work...even if Zapier has no email inbox to look at I think you can get the email messages via REST.

I'm not sure this is still open. It was moved to "done" on a project board, but that doesn't automate closing the issue.

I'm not sure this is still open. It was moved to "done" on a project board, but that doesn't automate closing the issue.


docs: add a list of (major?) companies using TypeID in production

It would be nice to collect a list of companies using this, so that those of us who are looking for evidence to persuade coworkers of the value of this don't have to spend time doing the research. The list I have right now:

Stripe, e.g. customers are cus_NffrFeUfNV2Hib
Jetify (unsure about examples)
Vercel, e.g. deployments are dpl_4UoGcLSZDZvoik6MvAANu9jVVLsX

GitHub and OpenAI's API keys also reflect this pattern (gha_<secret> and sk_<secret>)

We use this over at Authress, some of them are typeId, some of them are prefix plus base5X or base3X when we want case insensitive. Technically it would be the typeid-js variant, but I don't know if that matters.


Updates README.md with svg badge

I am not sure what problem this PR is solving. Might you please explain why the svg badge is better?

Sure!
Without:

With:


access: fix mail cookie check

Fixes mail cookie check.

Adds a testsuite for mail cookie.

Signed-off-by: Leonardo Rossi<EMAIL_ADDRESS>

ping @jirikuncar

Reviewed-by: me :shipit:

done! :shipit: XD


Taps go through absolutely positioned View on Android when backgroundColor is not provided

Description
I have a simple modal component similar to this:
<View style={styles.background}>
  <View style={styles.box}>
    {/* more stuff here */}
  </View>
</View>

Here are styles for this component:
const styles = StyleSheet.create({
  background: {
    position: 'absolute',
    top: 0,
    left: 0,
    right: 0,
    bottom: 0,
    alignItems: 'center',
    justifyContent: 'center',
    paddingHorizontal: 25,
  },
  box: {
    height: 325,
    alignSelf: 'stretch',
    borderRadius: 5,
    shadowColor: '#000000',
    shadowOffset: {width: 0, height: 2},
    shadowOpacity: 0.2,
    shadowRadius: 4,
    elevation: 5,
  },
});

As you can see the first View tag fills the whole screen with absolute position. Then I have another View which acts like a box (actual modal). I have of course stuff under this component and because the first View is absolutely on top of everything the taps should not go through this view. On iOS the taps do not go through but on Android they DO.
The way I "fixed" it was I added backgroundColor: 'transparent' to the outer View. Then the taps did not go through the View like it should :)
Reproduction
Make modal component with similar structure and do not add backgroundColor to outer View.
Additional Information

React Native version: 0.38
Platform: Android
Operating System: MacOS

We're cutting down on the number of outstanding issues, in order to allow us to focus. I'm closing this issue because it has been open for over 60 days with no activity. If you think it should still be opened let us know why. PRs are always welcome.


Improve Renovate bot configuration on this repository

What
Standardize and improve our Renovate bot setup.
Why
We're not consistent with the setup we use on this repository, compared to the setup we have on the main renovate bot repo.
In any case we're missing these things that I think are important/handy to have:

We're not extending from our global renovatebot/.github Renovate config
Wait 3 days until getting updates for npm packages (can be unpublished within 72 hours)
Pin GitHub Actions to SHA / Digest

We also have some overly large groups, not sure what we want to do with them:
https://github.com/renovatebot/parser-utils/blob/ba1ea2afff2155f86e74b4cdbc1934e4517d9a3b/renovate.json#L10-L40
I also don't like how the bot adds labels, our other Renovate bot PRs do not add any labels, so we might want to strip the labeling part out of the config as well.
How
Discuss what changes we want/need to our Renovate config on this repository.
Make PR to make the changes.

@rarkins @zharinov What do you think you want/need Renovate bot do to on this repository?

This is now using our shared preset, so closing this.

BTW, this repository is superseded by zharinov/good-enough-parser


Raise exceptions instead of printing + os exit

Before submitting the PR, please go through the sections below and fill in what you can. If there are any items that are irrelevant for the current PR, remove the row. If a relevant option is missing, please add it as an item and add a PR comment informing that the new option should be included into this template.

All relevant items should be ticked before the PR is merged

Description
These may also cause issues with the busy status as they are now, and we should anyway never be using os._exit instead of raising so we should change this anyway.

[x] Add a summary of the changes and the related issue
[x] Add motivation and context regarding why the change is needed

Type of change

[x] Bug fix (non-breaking)

Checklist:
General

[x] Code change

[x] Self-review of code done
[x] Comments added, particularly in hard-to-understand areas

Repository / Releases

[x] Rebase / update of branch done

Checks

[x] Formatting: Black & Prettier checks pass
[x] CodeQL passes
[x] Tests

[x] The tests pass

I don't really know how to test it, but I don't see anything strange with the code changes

The way I tested it was by testing before and after without the if statements to see what happened with the messages and the busy status. You don't need to do this though, I've tried, so as long as you don't see anything strange 👍🏻


Reported detected license in system packages #478

This PR properly detects system packages #478

@tdruez This is ready for review and is a fix for #478

Looks good! Thanks for the fix.


CI testing of links and anchors

Internal links are broken from time to time, this change adds CI to check before we add them.

Bit of rubber ducking 🦆 , but this if failing as it appears I'm not moving the docs correctly in .github/workflows/test.yml.
Maybe I'm trying to be overly clever by cloning a repo to its location and then moving the contents? I think I just need to clone elsewhere, to tmp or so.

Closes #315


No response using xgboost after uwsgi integration with flask

When I call xgb.DMatrix for the first time after my service has started, the thread always gets stuck at _LIB.XGDMatrixCreateFromDense and remains unresponsive, and then the second time I request it xgb.DMatrix returns a matrix of all zeros resulting in an exception in my reasoning, so what could be the cause?
xgboost   version is 1.7.6
uwsgi version is 2.0.23

If I set uwsgi to threads=1, it blocks on every request

If I set up 4 processes, 3 threads per process, then the first request gets stuck, the second and third requests are fine, the fourth request gets stuck again, and if I understand uwsgi correctly, the fourth request goes into a new process, which means that every time the uwsgi process executes XGDMatrixCreateFromDense for the first time it gets stuck, and then continuing to execute XGDMatrixCreateFromDense on that process returns a matrix of all 0s!

Have you tried using inplace_predict instead? This function does not require the construction of DMatrix.
XGBoost internally uses OpenMP to use multiple threads when running prediction. I wonder if the use of OpenMP interferes with uwsgi.

Have you tried using inplace_predict instead? This function does not require the construction of DMatrix.
XGBoost internally uses OpenMP to use multiple threads when running prediction. I wonder if the use of OpenMP interferes with uwsgi.

I'm actually more interested in figuring out why this is happening, because it doesn't come back when I switch the program to start without uwsgi or switch to another machine, which means that it's not 100% reproducible, but it's 100% reproducible on the machine that had the problem.

inplace_predict

It's amazing, I don't get stuck and prediction errors when I call predict after my code calls inplace_predict first!

I can be pretty sure that the problem is in the XGDMatrixCreateFromDense method, but after reading the source code I found that this method doesn't have any thread-unsafe operations, it's all about new object operations in each thread, so I'm very confused

I defined clf = joblib.load(filepath) as a public variable in the py file, and even running inplace_predict gets stuck!

I'm not sure why this is happening. Let's hope someone else might have a better insight.

I'm not sure why this is happening. Let's hope someone else might have a better insight.

Under what circumstances would XGDMatrixCreateFromDense return a matrix with all 0. 0.'s?

The stuck places all have one thing in common they all use an ArrayAdapter, if you call a method that doesn't have an ArrayAdapter such as XGBoosterCreate, it won't get stuck!

ArrayAdapter is also used by inplace predict

inplace predict

yes，because  inplace predict is also get stuck

Newly discovered, as long as the first access after uwsgi startup is not to access the interface related to these two methods, the second access will not be a problem.

Maybe it's libgomp running into issues with fork https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58378 .
I don't think it's the array adapter to blame, it's more likely that when an array adapter is used by a public interface function, OpenMP is also used (create DMatrix, inplace predict, etc). Googling "libgomp fork safe" can find a lot of similar issues, (and we run into it before)
If you want to verify whether this is the case, you can try to disable OpenMP or use llvm/clang-based toolchain along with its libomp. Both require recompiling XGBoost though.

Maybe it's libgomp running into issues with fork https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58378 .
I don't think it's the array adapter to blame, it's more likely that when an array adapter is used by a public interface function, OpenMP is also used (create DMatrix, inplace predict, etc). Googling "libgomp fork safe" can find a lot of similar issues, (and we run into it before)
If you want to verify whether this is the case, you can try to disable OpenMP or use llvm/clang-based toolchain along with its libomp (instead of libgomp). Both require recompiling XGBoost though.

Thank you very much for your reply, I'm sure it will be very helpful, I'd like to support how to turn off openMp in xgboost, I already know how to compile it, also I'm very curious, if the blocking is due to a fork, why does a subsequent call to XGDMatrixCreateFromDense from another thread in the same process return a matrix with all zeros!

Maybe it's libgomp running into issues with fork https://gcc.gnu.org/bugzilla/show_bug.cgi?id=58378 .
I don't think it's the array adapter to blame, it's more likely that when an array adapter is used by a public interface function, OpenMP is also used (create DMatrix, inplace predict, etc). Googling "libgomp fork safe" can find a lot of similar issues, (and we run into it before)
If you want to verify whether this is the case, you can try to disable OpenMP or use llvm/clang-based toolchain along with its libomp (instead of libgomp). Both require recompiling XGBoost though.

I'm more certain now that it might be a fork issue, because I configured lazy-apps = true, which would load the Flask app once for each worker instead of forking it from the main staff, and then there was no problem

You can look into the CMake file at the root of project directory, there's a list of options for compiling. Also, we have document for how to compile XGBoost.
As for 0 value, that might be the default value for floating point. However, when there's a concurrency issue (fork), I think there's very little meaning in interpreting any output and can safely assume it's random.

I'm more certain now that it might be a fork issue,

That's good news, we can focus on that. Consider the llvm toolchain or disabling openmp should help. In general, XGBoost is not fork safe since neither CUDA nor libgomp is fork safe, and we don't test such use case.

You can look into the CMake file at the root of project directory, there's a list of options for compiling. Also, we have document for how to compile XGBoost.
As for 0 value, that might be the default value for floating point. However, when there's a concurrency issue (fork), I think there's very little meaning in interpreting any output and can safely assume it's random.

I'm more certain now that it might be a fork issue,

That's good news, we can focus on that. Consider the llvm toolchain or disabling openmp should help. In general, XGBoost is not fork safe since neither CUDA nor libgomp is fork safe, and we don't test such use case.

Thank you very much for your reply, it's very helpful, if I understand it correctly, it's because uwsgi forked a sub-thread, and then this sub-thread called the method of xgboost's so-file, and if libgomp or openMP is used in this method, it may cause the first sub-thread to hang, and that's the reason why my program is blocking, and the only thing that confuses me is that the later forked sub-thread not only doesn't block, but also gets a value of 0 when creating the matrix, but the reasoning part is right again. The only thing I'm confused about is that the subsequent forked subthread not only doesn't block but also gets a value of 0 when the matrix is created, but the reasoning is correct. Can you give me some advice from the code point of view of the XGDMatrixCreateFromDense function that creates the matrix? Where in the code or causing this? I read the code and I suspect it is: collective::Allreducecollective::Operation::kMax(&info_.num_col_, 1);
This line of code causes

I think I've got it all figured out, first of all why some environments have problems and some don't, because the environment that doesn't have problems (running on K8S) limits the number of CPU cores to 0.25, which is less than 1, then openMP will only provide a thread that won't be enabled, which won't trigger the fork problem, and when I limit the number of CPU cores of the pod that has problems to 1, I'm not sure if I can get it to work. When I limit the CPU cores of the problematic pod to 1, the problem does not occur. Secondly why the matrix is, I read the XGDMatrixCreateFromDense code and found that openMP is used in simpleMatrinx for example:
#pragma omp parallel for reduction(+ : num_elements, num_rows) num_threads(ctx_.Threads())
for (int i = 0; i < static_cast(batches.size()); ++i) { // NOLINT
num_elements += batches[i]->Import(missing);
num_rows += batches[i]->Size();
}
If there is a problem with openMP then no looping will take place I guess, that is, when doing matrix operations all values will be


Failed to start the download. Unauthorized

Failed to start the download. Unauthorized
this happens when i send /mirror command

no reply


Fix merge conflicts from PR #24

Resolved all conversation offline. The summary is:

Using another JQuery method doesn't solve the repetition issue
The repetition issue is already happening in prod right now (we're getting a bunch of Summary hardcoded titles)
We'll move this to GraphQL before Apr 7 so having this isn't a breaker


[stable/nginx-ingress] Loops through the yaml instead to generate the configMap rather than …

What this PR does / why we need it:
ConfigMaps only accept key value pairs as strings or null. Objects, integers, booleans etc might be set accidentially in the values.yaml file and not quoted. This will ensure those values are wrapped in quotes and passed to k8s as string values.
Unquoted values below will be accepted by the helm linter as valid YAML and not picked up by a dry-run of kubectl apply -f <configmap.yaml> --dry-run
E.g.
data:
  test: false
  this: 0.01

The template change in this PR will ensure the values provided will be wrapped in double quotes and accepted by k8s:
data:
  test: "false"
  this: "0.01"

Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] Chart Version bumped

@jackzampolin @mgoodness @chancez

This looks related to https://github.com/helm/charts/issues/8845 🤔


[OPENVPN] Cipher 'AES-256-GCM' mode not supported

Hello,
I have a problem on two OPNsense system.
I try to improve speed of existing openvpn connexion between these two OPNsense.
The VPN was using "AES-128-CBC" cipher, I want to use the "AES-256-GCM" cipher.
The first system is based on an Asrack C2550d4i.
The second is based on a  KVM virtual machine, host has also an Atom C2550 processor, and the AES-NI is passed to the guest.
On the two system, I enable the AES-NI :
dmesg | grep AESNI
Features2=0xc3f82203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND,HV>

dmesg | grep AESNI
Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>

Without AES-NI, I can't use the AES-GCM cipher :
/usr/local/bin/openssl speed -elapsed aes-256-gcm
Error: bad option or value

But with, It work, like expected :
/usr/local/bin/openssl speed -elapsed -evp aes-256-gcm
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-gcm for 3s on 16 size blocks: 19482378 aes-256-gcm's in 3.01s
Doing aes-256-gcm for 3s on 64 size blocks: 9111617 aes-256-gcm's in 3.04s
Doing aes-256-gcm for 3s on 256 size blocks: 2900328 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 1024 size blocks: 805390 aes-256-gcm's in 3.00s
Doing aes-256-gcm for 3s on 8192 size blocks: 106650 aes-256-gcm's in 3.02s
OpenSSL 1.0.2m  2 Nov 2017
built on: reproducible build, date unspecified
options:bn(64,64) md2(int) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: cc -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -pthread -D_THREAD_SAFE -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -O3 -Wall -O2 -pipe  -fPIE -fPIC -Werror -Qunused-arguments -fstack-protector -fno-strict-aliasing -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DRC4_ASM -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM -DECP_NISTZ256_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-gcm     103636.13k   191882.69k   247494.66k   274906.45k   289716.66k

So in the Openvpn config, I enable the "Hardware Crypto" to "Intel RDRAND engine - RAND", but OPENVPN said the cipher is not supported :
Nov 26 14:49:57 | openvpn[68563]: Exiting due to fatal error
Nov 26 14:49:57 | openvpn[68563]: Cipher 'AES-256-GCM' mode not supported
Nov 26 14:49:57 | openvpn[68563]: Initializing OpenSSL support for engine 'rdrand'
Nov 26 14:49:57 | openvpn[68563]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Nov 26 14:49:57 | openvpn[68563]: MANAGEMENT: unix domain socket listening on /var/etc/openvpn/server1.sock
Nov 26 14:49:57 | openvpn[68284]: library versions: OpenSSL 1.0.2m 2 Nov 2017, LZO 2.10
Nov 26 14:49:57 | openvpn[68284]: OpenVPN 2.4.4 amd64-portbld-freebsd11.0 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Nov 21 2017
Nov 26 14:49:57 | openvpn[68284]: disabling NCP mode (--ncp-disable) because not in P2MP client or server mode

The Openvpn config file :
cat /var/etc/openvpn/server1.conf
dev ovpns1
verb 3
dev-type tun
tun-ipv6
dev-node /dev/tun1
writepid /var/run/openvpn_server1.pid
#user nobody
#group nobody
script-security 3
daemon
keepalive 10 60
ping-timer-rem
persist-tun
persist-key
proto udp
cipher AES-256-GCM
auth SHA384
up /usr/local/sbin/ovpn-linkup
down /usr/local/sbin/ovpn-linkdown
local x.172.192.x
engine rdrand
ifconfig <IP_ADDRESS> <IP_ADDRESS>
lport 54165
management /var/etc/openvpn/server1.sock unix
route <IP_ADDRESS> <IP_ADDRESS>
secret /var/etc/openvpn/server1.secret
comp-lzo no

Sorry for my English and thank's for the help

If your system supports AES-NI, don't select a crypto engine in OpenSSL and it should work.

Hi, thank's for the reply.
I have test without Hardware Crypto in OpenVPN config and with Cryptographic Hardware Acceleration set to "AES-NI CPU Based Acceleration (aesni)" in System: Settings: Miscellaneous, same problem : "Cipher 'AES-256-GCM' mode not supported".
Vice et versa : with Hardware Crypto to "Intel RDRAND engine - RAND" in OpenVPN config and without Cryptographic Hardware Acceleration in System: Settings: Miscellaneous, same problem : "Cipher 'AES-256-GCM' mode not supported".
Without hardware crypto in both openvpn config, and system settings, same problem.
But in shell, the following command work's :
/usr/local/bin/openssl speed -elapsed -evp aes-256-gcm
System runing up-to-date :
OPNsense 17.7.8-amd64
FreeBSD 11.0-RELEASE-p15
OpenSSL 1.0.2m 2 Nov 2017
Thank's a lot.

I could reproduce. Which server mode did you configure?

I use the "Peer to Peer ( shared key )" server mod.

Yes, this does not work. The cipher is TLS-only, you can even see that from the drop down selection. :)

Oh I never see this mention in the drop down selection ! Thank's.
So it will work if I switch to peer to peer SSL/TLS ?
Thank's a lot, and sorry for this mistake !

From a quick test, all other modes seemed to work ok. No worries. :)

Perfect, it work's !
Thank's

👍


pull request for e2e test at 12/29/2016 12:56:11

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Added support for emojis in Labels

Added it in my private project and thought other might like it as well.
I have added a third-party library called GraphemeSplitter which I have slightly modified to be able to import it with define().
The other change is to LabelCollection where it now splits double characters like emojis correctly.

Thank you so much for the pull request @andreborud! I noticed this is your first pull request and I wanted to say welcome to the Cesium community!
The Pull Request Guidelines is a handy reference for making sure your PR gets accepted quickly, so make sure to skim that.

:x: Missing CONTRIBUTORS.md entry.

Please add yourself to the contributors file!

:x: Missing CLA.

Please send in a Contributor License Agreement (CLA) and comment back here to let us know to check this!

:grey_question: Changes to third party files were made.

Looks like a file in one of our ThirdParty folders (ThirdParty/, Source/ThirdParty/) has been added or modified. Please verify that it has a section in LICENSE.md and that its license information is up to date with this new version.

Reviewers, don't forget to make sure that:

[ ] Cesium Viewer works.
[ ] Works in 2D/CV.
[ ] Works (or fails gracefully) in IE11.

I am a bot who helps you make Cesium awesome! Contributions to my configuration are welcome.
:earth_africa: :earth_americas: :earth_asia:

Thanks @pjcozzi.
I'm realising "Added support for emojis in Labels" might not be the best description for what this pull request adds. We had two problems that got solved simultaneously one was that we needed to use national flags which we used billboards for which reduced the performance for us, but using emoji national flags in a label instead reduced the number of entities we had to draw.
The other problem was that cesium didn't support many non-english characters which we needed. So maybe this pull request should be called "Added full UTF-8 support for labels".
Can't say I'm certain this is the best approach and I haven't made any extensive performance/memory testing.
Could it be a better idea to make this optional for labels or to only apply it if any of the characters in the label has a higher character number than 128 as an example?

Thanks again for your contribution @andreborud!
No one has commented on this pull request in 30 days. Maintainers, can you review, merge or close to keep things tidy?
I'm going to re-bump this in 30 days. If you'd like me to stop, just comment with @cesium-concierge stop. If you want me to start again, just delete the comment.

I am a bot who helps you make Cesium awesome! Contributions to my configuration are welcome.
:earth_africa: :earth_americas: :earth_asia:

Sorry for the delay here.  I'm interested in getting this reviewed and merged, since as you mentioned it's more about UTF-8 support) but I need to take a look at your library and better understand the issue all around.  I'll try to get to it before the next release.

Thanks again for your contribution @andreborud!
No one has commented on this pull request in 30 days. Maintainers, can you review, merge or close to keep things tidy?
I'm going to re-bump this in 30 days. If you'd like me to stop, just comment with @cesium-concierge stop. If you want me to start again, just delete the comment.

@andreborud please merge in master when you have a chance.  @mramato will hopefully be able to look at this for the next release.  Thanks!

Thanks again for your contribution @andreborud!
No one has commented on this pull request in 30 days. Maintainers, can you review, merge or close to keep things tidy?
I'm going to re-bump this in 30 days. If you'd like me to stop, just comment with @cesium-concierge stop. If you want me to start again, just delete the comment.

Thanks again for your contribution @andreborud!
No one has commented on this pull request in 30 days. Maintainers, can you review, merge or close to keep things tidy?
I'm going to re-bump this in 30 days. If you'd like me to stop, just comment with @cesium-concierge stop. If you want me to start again, just delete the comment.

Thanks again for your contribution @andreborud!
No one has commented on this pull request in 30 days. Maintainers, can you review, merge or close to keep things tidy?
I'm going to re-bump this in 30 days. If you'd like me to stop, just comment with @cesium-concierge stop. If you want me to start again, just delete the comment.

Thanks again for your contribution @andreborud!
No one has commented on this pull request in 30 days. Maintainers, can you review, merge or close to keep things tidy?
I'm going to re-bump this in 30 days. If you'd like me to stop, just comment with @cesium-concierge stop. If you want me to start again, just delete the comment.

Thanks again for your contribution @andreborud!
No one has commented on this pull request in 30 days. Maintainers, can you review, merge or close to keep things tidy?
I'm going to re-bump this in 30 days. If you'd like me to stop, just comment with @cesium-concierge stop. If you want me to start again, just delete the comment.

Thanks again for your contribution @andreborud!
No one has commented on this pull request in 30 days. Maintainers, can you review, merge or close to keep things tidy?
I'm going to re-bump this in 30 days. If you'd like me to stop, just comment with @cesium-concierge stop. If you want me to start again, just delete the comment.

Thanks again for your contribution @andreborud!
No one has commented on this pull request in 30 days. Maintainers, can you review, merge or close to keep things tidy?
I'm going to re-bump this in 30 days. If you'd like me to stop, just comment with @cesium-concierge stop. If you want me to start again, just delete the comment.

Thanks again for your contribution @andreborud!
No one has commented on this pull request in 30 days. Maintainers, can you review, merge or close to keep things tidy?
I'm going to re-bump this in 30 days. If you'd like me to stop, just comment with @cesium-concierge stop. If you want me to start again, just delete the comment.

Sorry for the delay. Master branch merged.

@andreborud I apologize again for it taking so long to get this merged. I think "support for emojis" is definitely underselling the improved support for non-latin character sets that this PR provides.  I'll tweak CHANGES in master, but otherwise this looks good to me.  Thanks!
I was slightly worried that this would affect label performance, but since it only happens at label generation time, it has negligible impact in the small amount of testing I did.  Plus if performance does turn out to be an issue, it would be trivial to make this opt-in.


Cleanup and improve Infinispan template

NON_XA  may swallow exceptions depending on where they happen, and we want them to be sent to Keycloak.
Change transaction mode to NON_DURABLE_XA.
Update generated examples in the Keycloak repository.
Generate the example for volatile sessions.
It is easier to point to an existing example when users want to try it.

Follow up: https://github.com/keycloak/keycloak/issues/35451


feat: Manage .env files using bucket

I love how little code changes that needed 👍
Did you check deployment already as well?


Update dependencies

This will become relevant after https://github.com/input-output-hk/cardano-ledger-specs/pull/1536

Maybe you can bump the dependencies again, some things probably have changed in the meantime.

bors merge


No profile to crawl right now

@leonardiwagner thanks for this great stuff i am using your library to crawl but it says no profile to crawl right now can you please fix this

this is my config.json
{
"email"<EMAIL_ADDRESS>"password": "mypass",
"cookiesFile": "./linkedincookie.txt",
"relatedProfilesKeywords": [],
"maxConcurrentCrawlers": 2,
"profileLoadWaitTime": 5000,
"saveDirectory": "./crawledProfiles",
"puppeteerArgs": { "args": ["--no-sandbox"] },
"isHeadless": true,
"hasToLog": true,
"proxyAddress": null,
"rootProfiles": [
"https://www.linkedin.com/in/syed-asher-ahmed-6762b6134/",
"https://www.linkedin.com/in/syed-asher-ahmed-6762b6134/detail/contact-info/"
]
}

answered there: https://github.com/linkedtales/scrapedin-linkedin-crawler/issues/31


S3Router "region" Option Updates and Uses Global aws-sdk Region Option

As currently implemented, the region option in s3Router is passed to aws.config.update(): https://github.com/odysseyscience/react-s3-uploader/blob/6e3c62656c317fa8721762694b230fa41cc76965/s3router.js#L23
The aws.config.update function will update the global config object for the aws-sdk, meaning that any other service objects instantiated after s3Router will default to the region passed in the s3router options.
Also, because of the way that s3router instantiates an s3object on each request, any changes to the global aws.config.region later could result in a different region being used than the one passed to s3router initially.
The AWS docs reference the global nature of the aws.config object specifically just below "Setting The Region" in the "Locking API Versions" section: http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/node-configuring.html#Setting_the_Region
The better option is to pass an object with the region setting directly to the s3 objects created by s3router. PR submitted: https://github.com/odysseyscience/react-s3-uploader/pull/47

Thanks for PR.  Merged and published v1.2.2 with this fix.


Works only when the application running the front

How to make broadcasting to work even when the application running in the background?

Unfortunately, no.
On iOS, apps simply cannot send out standard Bluetooth LE advertisements when they are in the background. They have to be in the foreground to do this.
Here's how Apple describes it:

...you should be aware that advertising while your app is in the background operates differently than when your app is in the foreground. In particular, when your app is advertising while in the background:

The CBAdvertisementDataLocalNameKey advertisement key is ignored, and the local name of peripheral is not advertised.
All service UUIDs contained in the value of the CBAdvertisementDataServiceUUIDsKey advertisement key are placed in a special “overflow” area; they can be discovered only by an iOS device that is explicitly scanning for them.
If all apps that are advertising are in the background, the frequency at which your peripheral device sends advertising packets may decrease.

Read more here: https://developer.apple.com/library/content/documentation/NetworkingInternetWeb/Conceptual/CoreBluetooth_concepts/CoreBluetoothBackgroundProcessingForIOSApps/PerformingTasksWhileYourAppIsInTheBackground.html

@williamtran29  I think there are still ways. Check this. Android unable to detect iOS when iOS app running in the background. I think I'm missing something

Core Bluetooth Background Execution Modes
If your app needs to run in background to perform certain Bluetooth-related tasks, it must declare that it supports a Core Bluetooth background execution mode in its Information property list (Info.plist) file. When your app declares this, the system wakes it up from a suspended state to allow it to handle Bluetooth-related events. This support is important for apps that interact with Bluetooth low energy devices that deliver data at regular intervals, such as a heart rate monitor.
There are two Core Bluetooth background execution modes that an app may declare—one for apps implementing the central role, and another for apps implementing the peripheral role. If your app implements both roles, it may declare that it supports both background execution modes. The Core Bluetooth background execution modes are declared by adding the UIBackgroundModes key to your Info.plist file and setting the key’s value to an array containing one of the following strings:
bluetooth-central—The app communicates with Bluetooth low energy peripherals using the Core Bluetooth framework.
bluetooth-peripheral—The app shares data using the Core Bluetooth framework.
Note: The property list editor in Xcode by default displays human-readable strings for many keys instead of the actual key name. To display the actual key names as they appear in the Info.plist file, Control-click any of the keys in the editor window and enable the Show Raw Keys/Values item in the contextual window.
For information about how to configure the contents of your Info.plist file, see Xcode Help.
The bluetooth-central Background Execution Mode
When an app that implements the central role includes the UIBackgroundModes key with the bluetooth-central value in its Info.plist file, the Core Bluetooth framework allows your app to run in the background to perform certain Bluetooth-related tasks. While your app is in the background you can still discover and connect to peripherals, and explore and interact with peripheral data. In addition, the system wakes up your app when any of the CBCentralManagerDelegate or CBPeripheralDelegate delegate methods are invoked, allowing your app to handle important central role events, such as when a connection is established or torn down, when a peripheral sends updated characteristic values, and when a central manager’s state changes.
Although you can perform many Bluetooth-related tasks while your app is in the background, keep in mind that scanning for peripherals while your app is in the background operates differently than when your app is in the foreground. In particular, when your app is scanning for device while in the background:
The CBCentralManagerScanOptionAllowDuplicatesKey scan option key is ignored, and multiple discoveries of an advertising peripheral are coalesced into a single discovery event.
If all apps that are scanning for peripherals are in the background, the interval at which your central device scans for advertising packets increases. As a result, it may take longer to discover an advertising peripheral.
These changes help minimize radio usage and improve the battery life on your iOS device.
The bluetooth-peripheral Background Execution Mode
To perform certain peripheral role tasks while in the background, you must include the UIBackgroundModes key with the bluetooth-peripheral value in your app’s Info.plist file. When this key-value pair is included in the app’s Info.plist file, the system wakes up your app to process read, write, and subscription events.
In addition to allowing your app to be woken up to handle read, write, and subscription requests from connected centrals, the Core Bluetooth framework allows your app to advertise while in the background state. That said, you should be aware that advertising while your app is in the background operates differently than when your app is in the foreground. In particular, when your app is advertising while in the background:
The CBAdvertisementDataLocalNameKey advertisement key is ignored, and the local name of peripheral is not advertised.
All service UUIDs contained in the value of the CBAdvertisementDataServiceUUIDsKey advertisement key are placed in a special “overflow” area; they can be discovered only by an iOS device that is explicitly scanning for them.
If all apps that are advertising are in the background, the frequency at which your peripheral device sends advertising packets may decrease.

Any solution to this yet? I need iOS advertising in the background state to be detectable by Android scanning in the foreground.
https://stackoverflow.com/questions/30874311/read-advertisement-data-on-android-when-an-ios-device-is-advertising

The advertisement is limited to 31 bytes so there is no room to place several 128bit UUIDS. You can try to pass [CBUUID UUIDWithString:@"16bit uuid"] to the advertising data.

Does this work?

Did anyone find a solution for this package to make it work in the background ? Please guide me on this if you have a solution


Adding an example using the file plugin for additional hosts.

Here is an example Corefile and associated db file to illustrate how to use the file plugin in conjunction with omada to add additional hosts to the dns server. This is the recommended approach from #45

Thanks for this!


1.1.x Helm Charts use release-1.1-latest-daily image

The release binary uploaded to Github uses the correct image tags, but the Helm repo does not.
wget https://storage.googleapis.com/istio-release/releases/1.1.0/charts/istio-1.1.0.tgz
tar xvf istio-1.1.0.tgz
$ grep tag istio/values.yaml
  # Default tag for Istio images.
  tag: release-1.1-latest-daily

This will impact anyone using Helm from the repo (helm install istio.io/istio ... vs helm install install/kubernetes/helm/istio ...)
@utka @duderino @mandarjog

Tested on 1.1rc1, 1.1.0, 1.1.1 draft. All have the same issue. 1.0.6 did not have this issue since we didn't have a Helm repo I think.

What is helm repo? Is this published somewhere outside this repository?

https://storage.googleapis.com/istio-release/releases/1.1.0/charts/
Under "Helm Chart Index" in releases

I've never used helm install. Is this an issue with our CI?

I'm seeing "the specified key does not exist" when I follow that link.

helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.1.0/charts/
helm install istio.io/istio --name istio --namespace istio-system
helm fetch istio/io/istio

https://storage.googleapis.com/istio-release/releases/1.1.0/charts/index.yaml describes what can be downloaded, helm will use this. So it shows the 3 charts we have, istio, istio-cni, istio-init.
https://storage.googleapis.com/istio-release/releases/1.1.0/charts/istio-1.1.0.tgz

Hmm, I've started looking for a place where this is published but so far no luck.

release/gcb/github_publish_release.sh:69 Indicates where we publish it on github - added in https://github.com/istio/istio/pull/9496/files
https://github.com/istio/istio/blob/master/release/gcb/helm_values.sh is what we need to run on https://github.com/istio/istio/blob/master/release/gcb/helm_charts.sh maybe?

This has never worked, all RCs also pointed to the latest daily...
I am fixing 1.1.0 and 1.1.1 charts now.

@mandarjog is this a manual fix? are we using GCB for helm repo?

Yes,  I fixed 1.1.0 release manually to point to the correct tag.
however 1.1.1 charts are not correct including its name, so I am not fixing them since 1.1.1 is not released yet.

I verified 1.1.0 looks good Mandar, thanks for the fix

Actually, think it still has issues. The tag should be 1.1.0 not release-1.1.0

fixed.

@mandarjog this is fixed in 1.1.0 but not 1.1.1

This has been fixed


Add diversity and github account

Requires stackage nightly in order to use the stackage fasta package:
stack install --resolver nightly-2015-09-20

Looks like diversity has an upper bound on MonadRandom

The old version I still had upper bounds, so I removed them from diversity <IP_ADDRESS> which I uploaded right before submitting this request. It's possible that it was too recent and the servers need to update.


Added ability to set a delegate to the adjust configuration

The call to [Adjust appDidLaunch:] should only be called once. Using the adjust provider from ARAnalytics it was not possible to set the delegate of the adjust configuration that is needed when an installation e.g. should be tracked.
This PR adds the possibility to initialise the provider with the delegate for the configuration to get the needed callback.
It only adds an initialiser, so it should not break anything.

assuming this is #237 without the squashed commits? closing

Oh yeah, sorry. Forgot to close this one.


Ghost roles: Outpost Survivor and Syndicate Fugitive

About the PR

Added two new ghost roles to appear on salvage expedition missions. Probability can be adjusted via CCvars.
Outpost Survivor - essentially a passenger role that starts on a salvage planet. Current probability is 10% chance for the role to spawn on each salvage mission with breathable atmosphere. Main reason they are here is to add an element uncertainty for the expedition crew, so they don't automatically assume the survivor is a fugitive.
Syndicate Fugitive - a traitor role with only an objective to escape on the shuttle uncuffed. Current probability is 10% of spawned Outpost Survivors making them pretty damn rare. They start in prisoner jumpsuits with no equipment except an emergency crowbar (in case the power is off on the outpost or they spawn inside a glass garden). To make matters worse for them, as soon as the ghost role is requested centcom will automatically send wanted notices to all fax machines on the station outing them by name as a fugitive. Since there is no direct communication during expeditions, the fugitive has some leeway to bluff their way onto the shuttle. As soon as they FTL to the station however, they either need to hope nobody noticed the faxes or they gotta flee immediately.
Media

[x] I have added screenshots/videos to this PR showcasing its changes ingame, or this PR does not require an ingame showcase

Changelog

:cl:

add: There have been reported sightings of stranded civilians on some salvage expedition missions. Please keep an eye out for them.

I think these 2 roles should look identical from a salvs perspective so they can never be sure if its a survivor or a syndie.

I think these 2 roles should look identical from a salvs perspective so they can never be sure if its a survivor or a syndie.

Is there currently a way to randomize the jumpsuit and shoes someone spawns with? I want the fugitives to start with a prisoner jumpsuit so that the first thing they need to do is hide/destroy it and find other clothes available on the outpost. This was if the fugitive is fairly sloppy and salv with a keen eye could discover the clothes.

I think these 2 roles should look identical from a salvs perspective so they can never be sure if its a survivor or a syndie.

Is there currently a way to randomize the jumpsuit and shoes someone spawns with? I want the fugitives to start with a prisoner jumpsuit so that the first thing they need to do is hide/destroy it and find other clothes available on the outpost. That way if the fugitive is fairly sloppy and salv with a keen eye could discover the clothes.
Randomizing the survivors spawned jumpsuit would help with lowering suspicions.

Eh. Randomizing the jumpsuits is getting dangerously out of scope. I'll do it in another PR later. For now fugitives are starting with Outpost Survivor gear.

I love this, however isn't instantly sending the fax upon ghost spawn way too harsh of a condition? In order to avoid instant arrest the agent would have to square against a heavily armoured salv squad, get a suit and somehow bunker down until shuttle arrives, since no TK for stealthy items. Otherwise he'll just be delivered straight to Sec and get cuffed. (although some people would love to play as prisoners from roundstart, so maybe it's fine)
If it's possible - I think there should be a timer on fax being sent to the station.

I love this, however isn't instantly sending the fax upon ghost spawn way too harsh of a condition? In order to avoid instant arrest the agent would have to square against a heavily armoured salv squad, get a suit and somehow bunker down until shuttle arrives, since no TK for stealthy items. Otherwise he'll just be delivered straight to Sec and get cuffed. (although some people would love to play as prisoners from roundstart, so maybe it's fine)
If it's possible - I think there should be a timer on fax being sent to the station.

Yeah. In response to your comments, I had a quick discussion with a friend of mine and we agree with you. The solution we've come up with is to delay the wanted notice by a randomish time (between 2 - 10 minutes) after the expedition mission has ended. I may still base the timer on when the role is accepted and adjust the timer by a flat 15 minutes (typical mission time) for simplicity.

sec would probably powergame this and just camp salv when they see the fax, id still love to see this added however

This PR has been open for about a month waiting for reviews from maintainers. I get that there's an antag freeze due to code quality issues with antag PRs lately which I'm cool with.
Antagonists. We have had code quality issues with these and the code will be scrutinised much more heavily than normal.
Can I please get some feedback? I don't want to have to repeatedly resolve conflicts if this is never going to get looked at.

Can I please get some feedback? I don't want to have to repeatedly resolve conflicts if this is never going to get looked at.

Reposting my comment from #codebase-changes.
Just a reminder that if your PR gets conflicts you do not need to resolve them immediately. We get 700+ PRs a month and your PR is bound to get conflicts. You only need to resolve them if a maintainer asks you to. Most conflicts are just import conflicts which are trivial and a waste of time to resolve constantly.

The idea is super cool, but the balancing feels a bit off to me tbh. Can’t wait to see how it turns out

@JustinTrotter you coming back to this?

@JustinTrotter you coming back to this?

Yes. Just need to find time.

re-open when you've had a chance to look


Replace prompt with modal for macro import feature.

@Ermad

I would add the autofocus attribute to the textarea, so you can paste right away after clicking the import button. Otherwise looks good to me.

Deployed to beta site.


Driver tests for Gallery's top 5 measured/monitored views

[ ] LazyBlock and Viktor's new app benchmark
[ ] Grid demo with many different images
[ ] Hero transition demo (new)
[ ] Flexible App Bar scrolling
[ ] Transition into the Weather app (and two more views) from a cold start

We've landed a driver test for all transitions, and we're monitoring it. We can reopen if we want more perf related measurements.


bug:

Plugin version:
5.0.0
Platform(s):
Android
Current behavior:
I don't know if it is a bug or if it is the intended behavior.
When requestManageOverlayPermission(), the app opens the settings screen with ALL installed apps listed, instead of prompting the user to ALLOW ou DENY.
This behavior is not good for me because my app relies on moveToForeground() to work, and the way it is, is not natural for user to find my app in the middle of all installed apps and enter the configs to allow.
Expected behavior:
Prompt user to allow/deny permission, instead of open settings.
Steps to reproduce:
Just calling requestManageOverlayPermission()
Related code:
allowPermission(){
requestManageOverlayPermission()
}
Thanks a lot.

This is the intended behavior. I can request the permission, but I can't tell Android how to request the permission. I am sorry that I do not have better news.
For this reason, I am closing this issue.


flash_attn_2_cuda.cpython-311-x86_64-linux-gnu.so: undefined symbol

how do I fix with it?

torch2.10 and cuda12.1, flashattn2.4.2

torch2.1.0 cuda12.1 and flash-attn 2.4.2

I encountered a similar error while attempting to install the package using the command
pip install flash_attn-2.5.5+cu118torch2.2cxx11abiTRUE-cp38-cp38-linux_x86_64.whl
However, the issue was resolved by using the command
pip install flash_attn-2.5.5+cu118torch2.2cxx11abiFALSE-cp38-cp38-linux_x86_64.whl
PyTorch version 2.2.0
CUDA version 11.8
Python version 3.8.18

I also encountered the issue. Could you please advise how to fix the issue? Thanks.
import torch
import flash_attn_2_cuda as flash_attn_cuda
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: /XXXXX/lib/python3.11/site-packages/flash_attn_2_cuda.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi

My environment is as follows:

OS: Red Hat Enterprise Linux 8.4
Python: 3.11.7
PyTorch: 2.3.1+cu121
CUDA: 12.2
flash-attn: 2.5.9.post1

I just found a solution from here: https://github.com/Dao-AILab/flash-attention/issues/620
CUDA needs to be downgraded to version 11, then install this module instead: https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.8/flash_attn-2.5.8+cu118torch2.3cxx11abiFALSE-cp311-cp311-linux_x86_64.whl
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.8/flash_attn-2.5.8+cu118torch2.3cxx11abiFALSE-cp311-cp311-linux_x86_64.whl

I just found a solution from here: #620 CUDA needs to be downgraded to version 11, then install this module instead: https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.8/flash_attn-2.5.8+cu118torch2.3cxx11abiFALSE-cp311-cp311-linux_x86_64.whl
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.8/flash_attn-2.5.8+cu118torch2.3cxx11abiFALSE-cp311-cp311-linux_x86_64.whl

actually you don't have to downgrade cuda, you just need to make sure you install the correct version of flash-attn which matches your cuda and python version


Tiny Toon Adventures - Babs' Big Break (USA, Europe), SolarStriker (World), Donkey Kong Land 2+3, Any Pokemon games?

I would love to be able to play those games with savestates. I just recently bought a GB flash card and missed the fact, that it does not have savestates. It is now pretty much useless to me...Fortunately i found this project :)! Please help <3

if it is any incentive (and not meant disrespectful) i would be willing to send 5$ via paypal for the work on the games :)

Hi there!
I haven't had a chance to look at these ones yet, donations not necessary. Will try to take a look soon :)

Thats is awfully kind of you, thank you very much :)!!

Hi again, I've just uploaded some patches for Bab's Big Break, SolarStriker and DKL 2 and 3. Give it a try and let me know if any issues :)
I'm not sure about the pokemon games. If they require RTC then they likely won't work with my save state patches. I'll have to look into those ones.

Hej man, they all work perfectly well. I havent played through any of the games yet but the initial test worked perfectly :). This is awesome!
I think pokemon red and blue as well as the zelda links awakening DX, oracle of ages and seasons do not use rtc features, so they would be working i assume?
if you find time, i would love to play the games with savestates :). its funny how fast you play the games with savestates, when it took months when i was a kid :D!! if patching of the pokemon games works, maybe you could do the german version <3

Hi again, I've now created a patch for Pokemon Red.
It seems to work ok, but it needed an MBC5 fix and some patching to track SRAM bank changes, so it is quite possible I've missed something. Give it a try and let me know if any issues :)
If the Red patch is working fine then I should be able to do Blue and the German versions fairly easily too.

Dear Mr. Matt Currie,
as you seem to know a lot about MBC1/MBC5 fixing: is there a chance you could create a patch for the German version of Zelda Link's awakening (1993 Version, not the DX one) to run on an MBC5 cart (no save states needed)?
The 27 year old batterie in my childhood Zelda died (I had saved the rom and the save file with a joey-joebag from BennVenn before the batterie died). I tried to clone the game to a MBC5 cart with batterie which works great as long a I do not try to save the game on the MBC5 cart (saving will crash the game and delete the save file). Other people found out that this is due to a difference between the behavior of the MBC5 and the MBC1 when the saving process forces the gameboy to reset and the MBC5 tries to adress Bank 0 while a MBC1 would adress Bank 1. I even found a fix for that works great for the US-version of the game. Trying to aply this patch (or your's for the US 1.2) to the German version of the game will chrash the game right at the start. It looks like "only" a few bytes need to be changed to fix this problem but I have absolutly no idea of how to do that.... Kind regards Dennis

Hi @gsi-man ,
Can you send me the MBC5 patch for the US version of the game? I should be able to adapt it for the German version pretty easily.

Hello Matt,
thanks for your reply.
The Patch (an IPS File) was created by someone called MottZilla and can still be downloaded (along with others) on his homepage for free ( https://thegaminguniverse.org/ninjagaiden4/mottzilla/  - which I have nothing to do with). So I guess it should be O.K. to use this file here as long as MottZilla is given the credit for making it. So thanks MottZilla and thanks in advance to you Matt.
Legend of Zelda, The - Link's Awakening (U) (Fix).zip
.

Hi @gsi-man,
Ok I've modified MottZilla's MBC5 patch for the German version. Give it a try and let me know if any issues :)
Legend of Zelda, The - Link's Awakening (G) (MBC5 Fix).zip

Hi @mattcurrie,
thanks a lot. Works like a charm :-)
Very cool that I can now play my childhood savegames with the "clone cart" instead of having to try to solder a new batterie to the PCB of the original cart.
The original cart was a christmas present by my grandparents back in 1994. I still got it CIB in almost mint condition so it's kind of a nice memorabilia to me.
Thanks again for helping me and for doing it that fast.
Kind regards from Germany
Dennis

@gsi-man hey that's great to hear. Happy I could help :)

Hi again, I've now created a patch for Pokemon Red.
It seems to work ok, but it needed an MBC5 fix and some patching to track SRAM bank changes, so it is quite possible I've missed something. Give it a try and let me know if any issues :)
If the Red patch is working fine then I should be able to do Blue and the German versions fairly easily too.

Hej, thank you for getting back to it :)! I just tried Pokemon red and it workes great so far. I only played for 20 minutes but i abused the save system and no crash or any other problem so far :). If something comes up I will let you know. I would greatly appreciate the patch for the german versions, yeah <3
Greetings!

Hi @1alien1,
Thanks for testing :) I've added a patch for the German version of Pokemon Blue. Can you give that a try?
If that is working fine then I can do the Red version too (it might be the exact same patch if its anything like the English version).

Hi @1alien1,
Thanks for testing :) I've added a patch for the German version of Pokemon Blue. Can you give that a try?
If that is working fine then I can do the Red version too (it might be the exact same patch if its anything like the English version).

Hej matt, as always, seems to work splendidly :), thank you very much! So i can use the same patch for the german red version or do you have to change something?

Hi @1alien1,
Thanks for testing :) I've added a patch for the German version of Pokemon Blue. Can you give that a try?
If that is working fine then I can do the Red version too (it might be the exact same patch if its anything like the English version).

Hej,
so I tried the same patch with the german red version, seems to work flawlessly so far :). Just for the fun of it I tried it also with the german yellow version, but that did not work out. It has a "bootloop" on the first ingame screen :). Again, thank you very much !


BUG: Fix failing to install optional dependencies during selective install

Traceback:
Traceback (most recent call last):
  File "C:\Users\butlej30383\AppData\Local\slicer.org\Slicer 5.6.2\bin\Python\slicer\util.py", line 3255, in tryWithErrorDisplay
    yield
  File "C:/Users/butlej30383/AppData/Local/slicer.org/Slicer 5.6.2/slicer.org/Extensions-32448/TotalSegmentator/lib/Slicer-5.6/qt-scripted-modules/TotalSegmentator.py", line 319, in onPackageUpgrade
    self.logic.setupPythonRequirements(upgrade=True)
  File "C:/Users/butlej30383/AppData/Local/slicer.org/Slicer 5.6.2/slicer.org/Extensions-32448/TotalSegmentator/lib/Slicer-5.6/qt-scripted-modules/TotalSegmentator.py", line 821, in setupPythonRequirements
    self.pipInstallSelective('nnunetv2', nnunetRequirement, packagesToSkip)
  File "C:/Users/butlej30383/AppData/Local/slicer.org/Slicer 5.6.2/slicer.org/Extensions-32448/TotalSegmentator/lib/Slicer-5.6/qt-scripted-modules/TotalSegmentator.py", line 706, in pipInstallSelective
    slicer.util.pip_install(requirement)
  File "C:\Users\butlej30383\AppData\Local\slicer.org\Slicer 5.6.2\bin\Python\slicer\util.py", line 3887, in pip_install
    _executePythonModule("pip", args)
  File "C:\Users\butlej30383\AppData\Local\slicer.org\Slicer 5.6.2\bin\Python\slicer\util.py", line 3848, in _executePythonModule
    logProcessOutput(proc)
  File "C:\Users\butlej30383\AppData\Local\slicer.org\Slicer 5.6.2\bin\Python\slicer\util.py", line 3814, in logProcessOutput
    raise CalledProcessError(retcode, proc.args, output=proc.stdout, stderr=proc.stderr)
subprocess.CalledProcessError: Command '['C:/Users/butlej30383/AppData/Local/slicer.org/Slicer 5.6.2/bin/../bin\\PythonSlicer.EXE', '-m', 'pip', 'install', 'black;extra', '==', 'dev']' returned non-zero exit status 2.

importlib finds the optional dependencies where nnUNet has optional dependencies (see here) for dev usage for enforcing various linting rules. Although these don't need to be installed to run TotalSegmentator successfully, I've changed the code to continue to install all the optional dependencies when using the selective pip install action.
https://github.com/lassoan/SlicerTotalSegmentator/blob/3ca58b17db54b8f52ec5ac193a83d3d03e5ff4a2/TotalSegmentator/TotalSegmentator.py#L679-L681
cc: @lassoan

This was originally brought up by @Keyn34 in https://github.com/Slicer/ExtensionsIndex/pull/2127#issuecomment-2523214057.

I also get an error while installing TotalSegmentator Extension for black;extra==dev. I did not investigate further, though.

Thank you for your contribution!


Enable LTO again

Pull Request
Related issue
Fixes #2718
What does this PR do?
This PR re-enables LTO once again for the project. Previously, LTO was disabled due to a build issue on the aarch64 platform (more details in https://github.com/meilisearch/meilisearch/pull/2717).
I performed build tests on my Apple Macbook M1 Pro (aarch64 platform) with Rust 1.79 (the version from the toolchain file) - cargo build --release completes successfully with both LTO modes: Fat and Thin (lto = true and lto = "thin" respectively).
AFAIU, performance is the main concern for the project, that's why I enabled Fat LTO instead of Thin LTO - from my experience, it allows us to perform better optimizations compared to Thin in most cases.
Below I collected build time benchmarks for the latest Meilisearch version at the moment on my Macbook (each build is a clean build):

Release: 4m 43s
Release + Thin LTO: 4m 32s (yep, it's not an error - build time improvement also can happen from enabling LTO)
Release + Fat LTO: 7m 39s

If Fat LTO build time overhead is too much for the project, we can switch to Thin LTO instead.
Below I collected some binary sizes between different LTO modes (+ current Release as a baseline).
meilisearch binary:

Release: 112 Mib
Release + Thin LTO: 113 Mib
Release + Fat LTO: 108 Mib

meilitool binary:

Release: 2.8 Mib
Release + Thin LTO: 2.8 Mib
Release + Fat LTO: 2.4 Mib

fuzz-indexing binary:

Release: 88 Mib
Release + Thin LTO: 88 Mib
Release + Fat LTO: 85 Mib

I didn't perform a performance comparison of all of these binaries though.
PR checklist
Please check if your PR fulfills the following requirements:

[x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
[x] Have you read the contributing guidelines?
[x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!

Hey @zamazan4ik 👋
Thank you very much for your contribution. We are in a little rush to release the new indexer we have been working on for 3-4 months. We will review your PR in a couple of days. The team thinks enabling thin or fat LTO is a great idea. However, to evaluate the performances, we need to run a bunch (if not many) of benchmarks before merging them.
Have a nice day 💪


[Bug]: Candy World Biomes still spawn after setting config to 0

General Info

[X] I am running a modpack
[X] I can reproduce this issue consistently in single-player
[X] I can reproduce this issue consistently in multi-player
[X] I have searched for this issue previously and it was either (1) not previously reported, or (2) previously fixed and I am having the same problem.
[ ] I am crashing and can provide my crash report(s)
[X] I am using the latest version of the mod

Forge version
36.2.28
Mod version
Latest version
Java version
Java 8 321
Issue Description
Joined an existing world with config set to 0, did /locatebiome and still found the cotton candy biome spawning in overworld
Additional Information
No response

Have you restarted your game after changing the config?

I will do that and get back to you, I changed it at the main menu then joined a world

This issue can be closed, it wont work if you change the configs at the main menu, only works if you restart your game, on an another note, it would be cool to see the chocolate milk and other buckets have a purpose, you cant drink them or use them to make anything, thanks for the hard work


KEP 1618: Optional garbage collection of finished Workloads

What type of PR is this?

/kind documentation
/kind api-change
What this PR does / why we need it:
Which issue(s) this PR fixes:

KEP for #1618
Special notes for your reviewer:
Does this PR introduce a user-facing change?

NONE

/cc @tenzen-y @PBundyra

nit, please say "KEP for https://github.com/kubernetes-sigs/kueue/issues/1618" in the PR description instead of "Fixes". Otherwise the robot will close the issue after merging the PR :)

/assign

/ok-to-test

/lgtm /approve /hold To wait for feedback from @tenzen-y as another pair of eyes.

ACK

/remove-kind api-change

/assign

FYI I just wanted to let everyone know that I didn't abandon the work. I've been on vacation last week and I'll be back in September. Sorry for the delay 🙇

FYI I just wanted to let everyone know that I didn't abandon the work. I've been on vacation last week and I'll be back in September. Sorry for the delay 🙇

No worries. I'm looking forward to moving this discussion after you come back here!

I'm really sorry it's been dragging. I'm coming back to it starting Monday.


Setup https://travis-ci.org

This allows us to run the unit tests (and e2e?) every time the project is pushed.

Done!


Support for Python 3.11

Missing functionality
Missing support for python 3.11
Proposed feature
Develop support for python 3.11, given that 3.11 is on road for some time and delivers some relevant performance upgrades.
Alternatives considered
No response
Additional context
Hello guys,
Any ETC for supporting python 3.11?
I´ve been facing some errors when trying to import pandas profiling (cannot import name 'DataError' from 'pandas.core.base'), and after some research I´ve found that is related to my current python version.

Looking at the error log, the last line that was run prior to getting this error is <a href='file:///---/pandas_profiling/model/correlations.py?line=12'>13
Line 15 in src/pandas_profiling/model/correlations.py is what is causing this error.
Opened pull request: #1239

The PR to support Python 3.11 has been merged (https://github.com/ydataai/ydata-profiling/pull/1273)
The support will be available from the next release (4.1.0).

Hi,
I tried to install the latest version with conda/mamba, but I could not.
The error message shows the following.
$ mamba install -c conda-forge ydata-profiling=4.1.2

Looking for: ['ydata-profiling=4.1.2']

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache

Pinned packages:
  - python 3.11.*

Could not solve for environment specs
The following package could not be installed
└─ ydata-profiling 4.1.2**  is installable and it requires
   └─ python >=3.7,<3.11 , which can be installed.

It is the same with conda.

How to solve this problem which occurred when I was trying to generate a html file using pandas profiling:
cannot import name 'DataError' from 'pandas.core.base'


how can I change the min face size that the detector detects

hey, how can I change the min face size that the detector detects,please

Hi @Jacklee12138
Dlib library detects human face(frontal) at least 80x80.
There are two options

If you want to detect smaller faces, I think you should change the dlib source code.
Enlarge the original image.
Thank you for your interest !


LICENSE change

This app will likely be having parts of it become copyright EeshvarDasIKCM at some point, because no one other than myself is showing any interest in this app.

This is going to happen now. I see no reason to put another hour of work in doing open source code for a while. The open source community is already having a big debt they owe me, and there's no sign that they intend to pay me anytime soon. Another reason to stop the open source contributions is that GitHub Actions and rulesets appear to be severely broken on this repository, and I can't find much fault in myself for this happening. It was nice getting to experience the conclusion that there appears to be little reason for me to trust the GitHub Actions and Rulesets for any serious work. I already have a very useful Ayurveda and Project Management app that I have been using privately and internally that is much superior to GitHub and much more reliable. I didn't know that these GitHub difficulties are what people and corporations have had to deal with, so I am happy that I chose a few years ago to create my own solution. So, you all are very welcome to have this repository here that demonstrates how much I am qualified to be paid like everyone else is in my industry. I understand that some companies encourage their employees to do some open source contributions, but I have yet to receive a paycheck, so I calculate that it is the open source community that needs to pay me if they want to see any more of my work. You're welcome. It has been fun over the past 20-40 hours of work that I have done here on GitHub. Again, let me know if any of you, or if GitHub changes their mind and decides to pay me. Have a nice day. See you next time.

For clarification, I will be using this code myself, but no additional enhancements or changes will be made available to the open source community. All new code I write forking off from this repository will be mine and mine alone. Please make a payment to my sponsor profile to see any new code that you can be free to use. Okay, thank you. Bye for now.

It will be interesting to see how my eeshvardasikcm-workflow continues to provide me some assistance here on GitHub in the future. Hope it was fun for you. I certainly had fun!
😄


rate limit calls to posix_fadvise()

There can be a large kernel overhead involved in POSIX_FADV_DONTNEED.
There is no point in calling this per item logged, so rate limit
to at most once per 2MiB written.
With a simple test program that logs 100K items at WARNING level:
Before:
$ time strace -c -e fadvise64 log.test
-log_dir=/dev/shm -logtofiles=true -logtostderr=false
% time     seconds  usecs/call     calls    errors syscall

100.00   12.522509         125     99957           fadvise64

real    0m52.671s
user    0m2.194s
sys     0m44.022s
After:
$ time strace -c -e fadvise64 log.test
-log_dir=/dev/shm -logtofiles=true -logtostderr=false
% time     seconds  usecs/call     calls    errors syscall

100.00    0.000759         152         5           fadvise64

real    0m4.206s
user    0m1.436s
sys     0m3.153s
Fixes issue #84

Looks great, thanks!


Implement Codacy coverage badge

I've just started using Codacy in preference to Scrutinizer. I noticed Shields had a Codacy grade indicator, but no coverage indicator, so here's the implementation.

The existing grade badge has been changed, but is backwards compatible (so people using the URL without the grade/ part still continue getting the correct grade.
New URLs for Codacy shields identify the grade/ or coverage/.

Tested locally, please let me know if there are any questions.

Would be really useful to start using this on a project soon. Any chance of a merge, @espadrine ?

Thanks for the contribution! I fixed the things myself to avoid you the trouble.

Excellent! Thanks for merging. Always a pleasure to contribute to this project. Is there anything I can do to help speed up PR acceptance?