What are the difference between this repo and NVIDIA/waveglow?

It seems that both of the projects refer to each other and both of them train using  LJ Speech Data. Thanks.

Most text-to-speech system consists of two steps: feature prediction and audio synthesis.
Tacotron performs the feature prediction, given text it produces mel-spectrograms.
WaveGlow is the synthesizer, which given mel-spectrograms produces audio files.

Sorry for the late reply and thanks for your reply.
So, if I want to use the latest technology to do TTS, I should follow the guild and README of NVIDIA/waveglow repo? Am I right?

Take a look at this file and read @ViktorIgeland's message.

@GabrielLin if you want to do TTS, as @ViktorIgeland mentioned, you need two models (I'm gonna use casual words): 1. model that generates mel-spectogram from given text; 2. model that generates audio from given mel-spectogram.
In this case, tacotron2 is the first model, that works on mel-spectogram generation from the text (again, @ViktorIgeland mentioned this).
And, waveglow is vocoder, that produces audio from mel-spectogram.
You'll need both models for end-to-end speech synthesis, so that you pass the text to tacotron2, it generates mel, and waveglow generates audio from mel.
Unless, you're not using Parallel Wavenet, that basically transforms the noise to Wavenet-level speech, and doesn't require the steps mentioned here, but it requires huge work on student-teacher training, and really careful adjustment of hyperparameters.

Closing due to inactivity.


[Suggestion] Mod Name Under The Tab Icon

I like this Idea, but since tabs are so close together, it could get a little cramped.
A tooltip might be a little better in this context.
I'll try to come up with something.

Ok, no hurry! Thank you very much! :)

Done!

Nice! Thank you very much! (1.18.2) :p


⚠ Jenkins Plugins Artifactory has degraded performance

In e856fe6, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1211 ms

Resolved: Jenkins Plugins Artifactory performance has improved in e3f8507 after 12 minutes.


v1.2.57

This repository contains GitHub workflows for managing releases automatically.
Please name this issue what you would like the new release to be called. E.x. "v1.0.0"
The following slash commands are available to use in comments on this issue:

Command
Description
Parameters

/release
Creates a new released based on the current state of the master branch
frontend=<version>backend=<version>engagement-status=<version>gitapi=<version>status=<version>config=<version>activity=<version>artifacts=<version>hosting=<version>participants=<version>engagements=<version>dispatcher=<version>agnosticv=<version>anarchy=<version>poolboy=<version>

/promote
Promotes the created release to the next environment
N/A

/cancel
Cancels the current release (and closes this issue)
N/A

/finish
Completes the current release (and closes this issue)
N/A

/release frontend=v1.2.5 backend=v1.4.5 gitapi=v1.3.0 config=v1.2.1 status=v1.2.0 dispatcher=v1.0.15 activity=v0.3.0 artifacts=v0.1.0 participants=v0.1.0 engagements=v0.0.1

/promote

/cancel


[Feature][Task Plugin] Add jupyter task plugin

Purpose of the pull request

Add jupyter task plugin to enable Dolphin users to scheduler notebooks developed in Jupyter Lab or Jupyter Notebook.
This PR closes: #9816

Brief change log

Encapsulate papermill in Dolphin jupyter task plugin.

Verify this pull request

Verified by unit test.

Codecov Report

Merging #9872 (e50f23c) into dev (f9f9100) will increase coverage by 0.09%.
The diff coverage is 72.22%.

@@             Coverage Diff              @@
##                dev    #9872      +/-   ##
============================================
+ Coverage     40.31%   40.40%   +0.09%
- Complexity     4530     4556      +26
============================================
  Files           835      840       +5
  Lines         33742    33850     +108
  Branches       3725     3736      +11
============================================
+ Hits          13602    13678      +76
- Misses        18855    18875      +20
- Partials       1285     1297      +12

Impacted Files
Coverage Δ

...cheduler/plugin/task/jupyter/JupyterConstants.java
0.00% <0.00%> (ø)

...eduler/plugin/task/jupyter/JupyterTaskChannel.java
0.00% <0.00%> (ø)

...plugin/task/jupyter/JupyterTaskChannelFactory.java
0.00% <0.00%> (ø)

...phinscheduler/plugin/task/jupyter/JupyterTask.java
74.62% <74.62%> (ø)

...heduler/plugin/task/jupyter/JupyterParameters.java
93.33% <93.33%> (ø)

...er/master/dispatch/host/assign/RandomSelector.java
77.77% <0.00%> (-5.56%)
:arrow_down:

...r/plugin/task/sqoop/parameter/SqoopParameters.java
53.33% <0.00%> (-1.34%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9f9100...e50f23c. Read the comment docs.

@caishunfeng I've resolved the comments above in the latest commit. Could you please take another look when you have time? Thx~

BTW, it seems miss UI and doc, another PR?

BTW, it seems miss UI and doc, another PR?

@caishunfeng Yes. As soon as the backend PR is merged, I will work on UI and docs.

BTW, it seems miss UI and doc, another PR?

@caishunfeng I just modified the description of this PR. Issue:#9816 will not be closed by this PR and I will submit another PR of UI and docs.

If you want to submit UI and docs, you could combine them in one single PR(this PR), I think

If you want to submit UI and docs, you could combine them in one single PR(this PR), I think

@zhongjiajie The reason why I planned to submit separate PRs for backend and UI is that it seems people who review the backend part and UI part are usually different and it looks more convenient to review. If there's no more changes needed for the backend part, I can continue to add the UI and docs in this PR : )

If you want to submit UI and docs, you could combine them in one single PR(this PR), I think

@zhongjiajie The reason why I planned to submit separate PRs for backend and UI is that it seems people who review the backend part and UI part are usually different and it looks more convenient to review. If there's no more changes needed for the backend part, I can continue to add the UI and docs in this PR : )

WDYT @caishunfeng @songjianet

FYI, I've added jupyter task plugin UI in the latest commit. Docs is WIP, will add it in the next commit.

This is a great task plugin,  Awesome!

This is a good task plugin, Awesome!

@jieguangzhou Thanks! BTW, I saw the MLops issue #9725  you opened and it looks really cool. I'm quite looking forward to it.

This is a good task plugin, Awesome!

@jieguangzhou Thanks! BTW, I saw the MLops issue #9725 you opened and it looks really cool. I'm quite looking forward to it.

I'm looking forward to adding the jupyter task to the MLops workflow.

I find out @Amy0104 fix the frontend issue, will merge now


Fix for HWI upstream

Qtum fix for HWI upstream is provided.
@icodeface In HWI branch cf/merge-upstream file ledger.py, the following line need to be added to work with the stake apps:
if self.app.getAppName() not in ["Qtum", "Qtum Test", "Qtum Stake", "Qtum Stake Test", "Bitcoin", "Bitcoin Test", "app"]:

ok, will update


Wrong pom version

Hello,
When I try to build my project I got this issue:
[WARNING] Failed to retrieve plugin descriptor for com.gitlab.haynes:libsass-maven-plugin:0.2.23: Invalid plugin descriptor for com.gitlab.haynes:libsass-maven-plugin:0.2.23 (.../repository/com/gitlab/haynes/libsass-maven-plugin/0.2.23/libsass-maven-plugin-0.2.23.jar), Plugin's descriptor contains the wrong version: 0.2.24-SNAPSHOT
Seems like the version in the POM is not corresponding to the version declared in maven repo.

@zemoa I can't reproduce this. I also checked the pom inside the 0.2.23 jar on maven central and it also contains 0.2.23 as the version number.
Can you provide a small example project where you experience this?

Actually I got this artifact through my nexus. And it seems that on this Nexus, the jar contains wrong version.
I don't understand how this is possible.
So I think this is not an issue linked to your library...
Thanks!


Remove purged package reference

For whatever reason this was making my local tests fail. It's been
deployed since it was added in 586a64ff9b70ac41c11837726bd4c60ee8c77f2f
so this should be safe to remove.

:+1: reviewed by @deanwilson over my shoulder.


Azure specific changes

Fixing an issue with the Azure GPT-4 instances where the Azure Models API endpoint '/openai/deployments' responds with a 404 'Resource Not Found' error for newer GPT environments. Providing the ability to change 'deployments' to 'models' to fix this issue with a new environment variable AZURE_MODELS_PATH.
When starting a new chat, the models drop-down can show the model name and ID to help distinguish between multiple models having the same name.
Providing the ability to only show 1 model in the drop-down when the AZURE_DEPLOYMENT_ID environment variable is provided.
If AZURE_DEPLOYMENT_ID is not provided, the models drop-down will show all available models, and use the selected model for the Chat Completion API requests.
Fixed an issue where Azure provides a '[DONE]' response which breaks the JSON.parse function and where the JSON response doesn't contain any 'choices'. Each of these cause the UI to break.
Ignoring '.env' files from being committed
Updated the README.md with the changes mentioned above that affect the environment variables.

@pureit-dev
I had the same issue and this PR solves it. A short term workaround is to build the docker image yourself and run it locally or push it to GitHub or Azure container registry to deploy it. I am using a private registry so I can not share.


kernels should be found in $PREFIX/share/jupyter/kernels

Rather than looking for executables in the various envs and building specs by hand, a more appropriate source of info should be the kernelspecs installed in those envs, walking $env/share/jupyter/kernels in each kernel env. This will allow respecting the argv of each kernelspec, and support all kernels in conda envs, not just IPython and R. It would also remove the need to ship a logos directory, as the resources dir would already be correct.
The effect ought to be:
for each env:
    for each kernel in env/share/jupyter/kernels:
        load kernelspec
        modify kernel name to include env key
        modify display name to indicate env name

additionally, it would probably be nice to activate the env, e.g. launching
bash -c 'source activate $env; exec $argv'

(I'm not sure what the Windows equivalent is)

I'm doing this in #80

Now implemented in master via #80. Turned out to be pretty complicated to get the activate-and-execute part right. I see nb_conda_kernels.runner for details.


[GHSA-5j5w-g665-5m35] Ambiguous OCI manifest parsing

Updates

References

Comments
Adding patch link for v1.4.12: https://github.com/containerd/containerd/commit/db00065a969a983ceb0a409833f93f705f284ea4
Adding patch link for v1.5.8: https://github.com/containerd/containerd/commit/26c76a3014e71af5ad2f396ec76e0e0ecc8e25a3
The GHSA-ID is in the message: "Merge pull request from GHSA-5j5w-g665-5m35"

Hi there @dmcgowan! A community member has suggested an improvement to your security advisory. If approved, this change will affect the global advisory listed at github.com/advisories. It will not affect the version listed in your project repository.
This change will be reviewed by our highly-trained Security Curation Team. If you have thoughts or feedback, please share them in a comment here! If this PR has already been closed, you can start a new community contribution for this advisory

Thank you for the additional reference links!


TODO: Investigate ability to refactor GridField::getCastedValue into GridFieldComponent

Description
Investigate ability to refactor GridField::getCastedValue into GridFieldComponent.
See: https://github.com/silverstripe/silverstripe-framework/blob/cf78a9d785b9959a92dfb52c2a456bea80a4c567/src/Forms/GridField/GridField.php#L345

Closing as I don't see any clear value in making this change.


Short term change

Function to return short term change in a dataframe

@antagomir @TuomasBorman

In holiday currently. I will check this early next week.

Good to finalize this as proposed.

Example shows the following:
# Load time series data
#' data(minimalgut)
#' tse <- minimalgut
#'
#' short_time_labels <- c("74.5h", "173h", "438h", "434h", "390h")
#'
#' # Subset samples by Time_lable and StudyIdentifier
#' tse <- tse[, !(colData(tse)$Time_label %in% short_time_labels)]
#' tse <- tse[, (colData(tse)$StudyIdentifier == "Bioreactor A")]
#'
#' # Plot short term change in abundance
#' shortTermChange(tse, rarefy = TRUE, plot = TRUE) + ggtitle("Bioreactor A")

Is this mean to be an analysis or visualization function?
The example could explain a bit more what this aims to show. I am not sure if I understand the motivation of this function from the manpage description.
It would be good to have a separate function, like getShortTermChange / addShortTermChange and then a separate function for visualization (unless it is easy to do with existing functions, like miaViz::plotSeries).
Is this linked to some miaTime issue?

#'
#' # Plot short term change in abundance
#' shortTermChange(tse, rarefy = TRUE, plot = TRUE) + ggtitle("Bioreactor A")

Is this mean to be an analysis or visualization function?

original reference had plotting as part of the function. It is now removed and the function returns a dataframe with short term changes

Is this linked to some miaTime issue? Issue #7 ?

yes, syncomR consumed phyloseq object.

Moreover given the nature of the data frame results, I am not sure plotSeries is the right visualization method for it.  Suppose we somehow addShortTermChange using one of the tse slots(say maybe reducedDims),  it would still require it's own ggplot wrapper .
I just checked but to be sure @TuomasBorman do we have miaViz function that takes dataframes?
For reference, this is how the plots from syncomR could look like linear or polarized:

and of course we can also discuss how we want to represent our own plots.

Moreover given the nature of the data frame results, I am not sure plotSeries is the right visualization method for it. Suppose we somehow addShortTermChange using one of the tse slots(say maybe reducedDims), it would still require it's own ggplot wrapper . I just checked but to be sure @TuomasBorman do we have miaViz function that takes dataframes? For reference, this is how the plots from syncomR could look like linear or polarized:

and of course we can also discuss how we want to represent our own plots.

There is no that kind of restriction that miaViz cannot have functions that take df as input. In fact, loadings plotting function can take df as input,
This outputs currently data.frame and I think it makes sense. reducedDims is not correct because this does not reduce dimensionality. This could be stored to metadata and then we could have a plotting function for TreeSE and df.

Or can it be part of colData? Could be simpler. This is a measure that is available on a per-sample basis. Except that some samples may have NAs if they are missing the comparison.

Or can it be part of colData? Could be simpler. This is a measure that is available on a per-sample basis. Except that some samples may have NAs if they are missing the comparison.

That could be possible but then columns would denote features. It might be confusing and there might be lots of features to add in the columns

Or can it be part of colData? Could be simpler. This is a measure that is available on a per-sample basis. Except that some samples may have NAs if they are missing the comparison.

That could be possible but then columns would denote features. It might be confusing and there might be lots of features to add in the columns

Yep, not so optimal.

"This function essentially calculates the change in abundance over time for a microbe by abundancet2/ abundancet1."
We already have getStepwiseDivergence?
getStepwiseDivergence calculates x(t + d) - x(t) for abundance x at time points t+d vs. t any user defined interval d along the time series.
getShortTermChange calculates this per taxa. One can generalize and use it for a more general interval x(t+d)/x(t), analogous to getStepwiseDivergence.
getShortTermChange suggests to do x(t+d)/x(t) instead of x(t+d)-x(t). But both are available and the choice may also depend on whether the data is logarithmic. log(a/b) = log(a)-log(b) although zeroes may cause trouble. It could be enough to implement the difference "x-y" and the rest ("x/y") can be always dealt (by the user?) with log/exp operations the data if necessary.


Text based textures generated by MC-TextureGen may differ from official generated textures (larger sample size needed)

I dreaded making this discovery...
This is the version generated by the program:

And this is the texture cropped out from the debug.stitched-(terrain/items).png from 13w02a:

There are just minimal differences in the "xtu" for whatever reason.
Both of these are generated in Windows 10 version 2004. I'm not sure what version the Java bundled with the launcher is (8?).
Here are the two texture atlases generated by 13w02a:

:

Pre-13w02a seems to have the same difference in the x.

Judging by all the screenshots and other info I could dredge up on both pre-13w18a missing textures, the wonky x and co. seem to be the only type that ever existed, rather than there being a whole array of different pixel differences across characters inconsistent across platforms or even sessions, at least from what I saw. So hopefully this news isn't too terrible.
https://www.minecraftforum.net/forums/minecraft-java-edition/discussion/175116-the-secret-text-in-paintings?comment=14

:/ I really thought I covered everything, evidently I didn't. Really sorry, I'll look at this today after I've woken up a bit.

@muzikbike I found another one. Java 1.6.0_65-b14-468 (Apple legacy Java 6 runtime), MacOS 10.14.6.

In game (MultiMC instance, from debug.stitched_items.png)

MC-TextureGen

I've seen both of these variations in bug reports, so I'm not sure which is supposed to be the correct one. I think I'll temporarily note that the generator looks to be inaccurate in the README until I fix whatever is causing this. I'm really sorry about the inconvenience!

On the topic: I'll need to check this more rigorously, but it seems the "unknown" OS I posted earlier might have been Ubuntu according to some recent samples.
What version of Java did you use to generate the original Mac version you posted, anyway?
I also think that this missing texture generator should be moved to the deterministic category, or to a new "system-deterministic" category, since it's much closer to stuff like the newer missing textures than it is to lava and co.

@muzikbike I believe that the cause of this issue is that you're running MC-TextureGen with a different version of Java than the one you used to run Minecraft with!
Minecraft 1.5.2, Java 1.8.0_51:

Cropped:

Minecraft 1.5.2, Java 11.0.10:

Cropped:

Running MC-TextureGen with each respective version of Java produces results identical to the dumped textures from Minecraft 1.5.2. Note that the Java 8 one is identical to the one you generated with Minecraft 13w02a, and the Java 11 one is identical to the one you generated with MC-TextureGen. It looks like newer versions of Java changed the way that fonts render, which is worth documenting.
With note to your recent comment: I agree that I should make a new category for this type of image. I think the first texture I generated was generated with Java 16. Here's a few for the record (all generated on MacOS 10.14.6):

Java 6 (Apple legacy Java runtime 1.6.0_65-b14-468), no anti aliasing (default for non-Retina displays)

Java 6 (Apple legacy Java runtime 1.6.0_65-b14-468), anti aliasing (default for Retina displays)

Java 1.7.0_80-b15

Java 1.8.0_51

Java 1.8.0_301, GraalVM EE 21.2.0

Java 11.0.12, GraalVM EE 21.2.0

Java 16.0.2, GraalVM EE 21.2.0

That is relieving.

Worryingly, WinMerge points out some extremely, extremely subtle differences between your generated Retina antialiased texture and the one listed in https://bugs.mojang.com/browse/MC-7208 - with the differences being a single number (for example 194, 194, 194 instead of 193, 193, 193). I'll document these separately for now and just hope the program is still working as expected and that maybe mspaint introduced weirdness like this while cropping or something.

Worryingly, WinMerge points out some extremely, extremely subtle differences between your generated Retina antialiased texture and the one listed in bugs.mojang.com/browse/MC-7208 - with the differences being a single number (for example 194, 194, 194 instead of 193, 193, 193). I'll document these separately for now and just hope the program is still working as expected and that maybe mspaint introduced weirdness like this while cropping or something.

...@muzikbike you're entirely right, that's really subtle. Here's the output of ImageMagick's -compare command (with -metric AE):

It's currently 12:30 at night in my timezone, so I'll have to look at this more tomorrow, but thank you for finding this. I hope it's just a Java version difference (the bug report says "Powermac 10.5.8", suggesting Apple Java 5 on a G4 / G5).

Just to make sure, can you send the terrain atlases generated by those versions as well?
I don't have any atlases to compare any of the other smoothed textures you posted, so I can't be sure on that front.
I also found some stuff through google images that might imply some sort of setting influences these, or that it might be an OS version difference again:

Just to make sure, can you send the terrain atlases generated by those versions as well?
I don't have any atlases to compare any of the other smoothed textures you posted, so I can't be sure on that front.
I also found some stuff through google images that might imply some sort of setting influences these, or that it might be an OS version difference again:

@muzikbike Here's the texture atlases generated with Java 6 (Apple legacy Java runtime 1.6.0_65-b14-468) on MacOS 10.14.6:

Here's the cropped "missing texture" texture:

This actually seems to match the texture generated with MC-TextureGen perfectly, meaning that it appears to be accurate. Tomorrow I'll boot up an old iMac G4 to see if I can generate the same texture that the bug report had.

I'll also check to see if I can influence anything with font smoothing settings. Different versions of MacOS have various amounts of control over text anti aliasing settings. I have it turned off right now. I remember testing to see if it had any influence on the output textures, which it did not, but I did not test many versions of Java with it.

@muzikbike On a whim, I tried ticking "use font smoothing" with Java 6 (Apple legacy Java runtime 1.6.0_65-b14-468) on MacOS 10.14.6, and it did actually influence the generated texture (uses polychromatic smoothing instead of monochromatic smoothing)!
Texture atlases from 1.5.2:

Cropped:

MC-TextureGen:

The texture generated by MC-TextureGen is an exact match to the one generated by Minecraft 1.5.2.
As it's currently 1:30 in the morning, I'll probably not reply to any further messages until I get some sleep.

I've tried comparing these generated textures with the ones from MC-7267, and they don't match.
Output of ImageMagick -compare command (with -metric AE):

I think this might caused by differences in the font smoothing on different types of screens. RenderingHints defines many types of font anti aliasing, all of which may produce different results. I might add a command line switch to generate a texture for each type of font anti aliasing.

@muzikbike Here's some results for Java 1.6.0_65 on MacOS 10.6.8, running on an Intel iMac.
Command: java -jar MCTextureGenerator.jar -platformTextures true
Notes: standard texture generation

Command: java -Dawt.useSystemAAFontSettings=on -Dapple.awt.textantialiasing=on -jar MCTextureGenerator.jar -platformTextures true
Notes: -Dawt.useSystemAAFontSettings=on -Dapple.awt.textantialiasing=on are passed to force text anti aliasing to be on.

Command: java -Dawt.useSystemAAFontSettings=on -Dapple.awt.textantialiasing=on -Dapple.awt.graphics.UseQuartz=true -jar MCTextureGenerator.jar -platformTextures true
Notes: same as above, but -Dapple.awt.graphics.UseQuartz=true is passed to simulate how Java 5 might render the text. apple.awt.graphics.UseQuartz would normally be set to "true" for Java 5 and below.

The texture generated when apple.awt.graphics.UseQuartz is set to "true" is unique, making it the third unique variation of a polychromatic text texture generated by an Apple Java runtime. Given that there's so much variation in these textures, I think it's likely that MC-TextureGen is accurate, and there's just a lot of variations on this texture. I'll keep this issue open for now, as I may have failed to cover some edge cases.

Worryingly, WinMerge points out some extremely, extremely subtle differences between your generated Retina antialiased texture and the one listed in bugs.mojang.com/browse/MC-7208 - with the differences being a single number (for example 194, 194, 194 instead of 193, 193, 193). I'll document these separately for now and just hope the program is still working as expected and that maybe mspaint introduced weirdness like this while cropping or something.

@muzikbike Here's an exact match for MC-7208:

MC-TextureGen on an iMac G4, running MacOS 10.3.9, on Java 4, with -Dawt.useSystemAAFontSettings=on -Dapple.awt.textantialiasing=on

MC-7208

I'll see if I can replicate this in more natural conditions.

I'm pretty sure if we included linux (unless you already do) the text on missing texture could be controlled in almost every possible way, so I don't think it's even worth covering all cases and simply generate it same way minecraft does?
Also there may have been user error when checking for inconsistency between generated in-game texture and mctexturegen one, such as java on %PATH% being a different one from launching the game.

Oh yeah, and considering this project is also made in java, an accurate port should give the same results. No edge cases

Documenting some Discord discussions for future reference:

Getting back into documenting these. Can you provide a b1.4-13w01b version as well?

My iMac G4 has sadly died, so I was unable to provide those textures.

Probably late on relaying this information, but there was a particularly confusing case last August by a user who no longer appears to be a member of the Omniarchive discord server:

The in-game textures were likely generated with the JRE currently bundled with Minecraft on macOS. Here are the results from using it to generate the text-based missing textures (macOS 12.7, M1):

It's possible the user was using a different version of Java to run MC-TextureGen, but it would be good to investigate any possible inaccuracies.

Oh yeah, and considering this project is also made in java, an accurate port should give the same results. No edge cases

The differences caused by this particular generator are due to how each version of Java handles text rendering. The texture is generated by calls to Graphics2D.drawString, without a specific font. This may cause differences based on the default font Java uses on each version and operating system, as well as differences from the used text rendering library (may use platform native font rendering, different versions of libraries etc), as well as AWT state (i.e. with the legacy Apple JRE) and operating system state (different anti-aliasing font settings) etc.


Apply colors Inputs (CheckBoxes, Radio buttons) ?

Hi
I try to set red color to a radio button but the color doesn't apply to the first element in the list, i'm doing something wrong or it's not possible?
Thanks

Yup, I was able to replicate this. Looks like you stumbled onto a strange bug, I'll take a look at it!


fix(semantic-release): provide semantic-release automation

use semantic-release to automate releases
remove obsolete change log template
update package.json dependencies
update eslint rules
update aem-spa-component-mapping and aem-spa-component-mapping  dependencies

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@34be794). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #14   +/-   ##
=========================================
  Coverage          ?   88.88%
=========================================
  Files             ?       16
  Lines             ?      333
  Branches          ?       44
=========================================
  Hits              ?      296
  Misses            ?       37
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34be794...3c5aee7. Read the comment docs.


Allow to disable tests through BUILD_TESTING=OFF

Hi!
I am creating a package for this awesome library in vcpkg: https://github.com/microsoft/vcpkg/pull/6426
It would be nice to have an option to disable tests.
By including CTest.cmake module, we got a BUILD_TESTING option which has a default value: ON.
CTest.cmake module will also call enable_testing() for us if BUILD_TESTING is ON.

Thanks!


Upload IPA to TestFlight success but job failed

New Issue Checklist

[x] Updated fastlane to the latest version
[x] I read the Contribution Guidelines
[x] I read docs.fastlane.tools
[x] I searched for existing GitHub issues

Issue Description
It happens After fastlane uploading IPA successful and app connect finished processing it.
CI Console
+------+-------------------------------------+-------------+
|                     fastlane summary                     |
+------+-------------------------------------+-------------+
| Step | Action                              | Time (in s) |
+------+-------------------------------------+-------------+
| 1    | default_platform                    | 0           |
| 2    | last_git_commit                     | 0           |
| 3    | setup_circle_ci                     | 0           |
| 4    | Switch to ios generate_api_key lane | 0           |
| 5    | app_store_connect_api_key           | 0           |
| 6    | Switch to ios certificates lane     | 0           |
| 7    | match                               | 2           |
| 8    | match                               | 1           |
| 9    | update_project_provisioning         | 0           |
| 10   | update_project_provisioning         | 0           |
| 11   | update_code_signing_settings        | 0           |
| 12   | build_app                           | 650         |
| 13   | Switch to ios upload_IPA lane       | 0           |
| 💥   | upload_to_testflight                | 1942        |
+------+-------------------------------------+-------------+

[04:41:51]: fastlane finished with errors

[!] Could not set changelog: This request is forbidden for security reasons - The API key in use does not allow this request

I guess it occurred due to token has been expired when fastlane assign changelog to TestFlight.
Because whole process cost over than 20 min.
Now I assign changelog to TestFlight manually after job failure.Is there any other solution for this?
Command executed
upload_to_testflight(
      api_key: api_key,
      changelog: changelog,
      ipa: ipa_path
)

Complete output when running fastlane, including the stack trace and command used

 [REPLACE THIS WITH YOUR INFORMATION]

Environment

 [REPLACE THIS WITH YOUR INFORMATION]

Upload successful for me too, but error is as below:
Details - Verbose logs

 ...
2021-04-12T14:42:08.3205660Z INFO [2021-04-12 14:42:08.32]: -----------------------------------------------------
2021-04-12T14:42:08.3206950Z INFO [2021-04-12 14:42:08.32]: Successfully uploaded package to App Store Connect. It might take a few minutes until it's visible online.
2021-04-12T14:42:08.3207980Z INFO [2021-04-12 14:42:08.32]: -----------------------------------------------------
2021-04-12T14:42:08.3234810Z INFO [2021-04-12 14:42:08.32]: Successfully uploaded the new binary to App Store Connect
2021-04-12T14:42:08.3236070Z INFO [2021-04-12 14:42:08.32]: Creating authorization token for App Store Connect API
...
...
2021-04-12T14:44:13.5121160Z INFO [2021-04-12 14:44:13.51]: Waiting for App Store Connect to finish processing the new build (1.4.0 - 20210412.1628) for IOS
2021-04-12T14:44:13.5122380Z WARN [2021-04-12 14:44:13.51]: Build did include information for app, build beta detail and pre release version
2021-04-12T14:44:13.5123420Z WARN [2021-04-12 14:44:13.51]: Fetching a new build with all the information needed
2021-04-12T14:44:14.3321550Z Timeout received: 'Spaceship::AccessForbiddenError', 'This request is forbidden for security reasons - The API key in use does not allow this request'. Retrying after 3 seconds (remaining: 4)...
2021-04-12T14:44:17.5401670Z Timeout received: 'Spaceship::AccessForbiddenError', 'This request is forbidden for security reasons - The API key in use does not allow this request'. Retrying after 3 seconds (remaining: 3)...
2021-04-12T14:44:20.7108740Z Timeout received: 'Spaceship::AccessForbiddenError', 'This request is forbidden for security reasons - The API key in use does not allow this request'. Retrying after 3 seconds (remaining: 2)...
2021-04-12T14:44:23.8177130Z Timeout received: 'Spaceship::AccessForbiddenError', 'This request is forbidden for security reasons - The API key in use does not allow this request'. Retrying after 3 seconds (remaining: 1)...

I'm using Azure DevOps


Use the presence service for presence-related things

As the title says

 Use new presence endpoint in JS SDK


Module concatenation

What problem does this feature solve?
It would be nice to be able to enable module concatenation the same as webpack does with https://webpack.js.org/plugins/module-concatenation-plugin/#root
What does the proposed API of configuration look like?
	optimization: {
		concatenateModules: true,
                 ......
	},

yeah it is on the roadmap

Oh sorry, didn't found any reference. Feel free to close in that case

bump

closed in favor of https://github.com/web-infra-dev/rspack/pull/5559


scan_files method in SecretsCollection class with parallel processing issue

If the scan_files method is called with multiple files using scan_files("file1", "file 2") blueprint, it has serious infinite looping issue. The exact error comes from parallel processing but the sub process spawning continues -
untimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module

One can stage the issue by using scan_files() method with multiple file name as arguments.

Hi @KBiru thanks for reporting this. I haven't been able to reproduce this issue on either macOS or Windows with detect-secrets 1.2.0. Can you confirm what OS you're using and the version of detect-secrets? Any other info you can provide to reproduce this issue would also be helpful. Thanks!

Hi, I am using v1.2.0 of detect-secrets and the OS is windows 10.
So, the problem occurs when we pass more than 1 file name as argument to the scan_files function of SecretsCollection class.

Thanks for the additional information; I've been able to reproduce this issue on Windows 10 as well.
This issue is specific to Windows and is mentioned in the Python multiprocessing documentation (see the Safe importing of main module section). You'll need to make sure any Python scripts on Windows use the if __name__ == '__main__': entry point and then the scan_files function should work as expected.
Rewriting the simple example from the detect-secrets README for Windows would look like this:
from detect_secrets import SecretsCollection
from detect_secrets.settings import default_settings

def main():
    secrets = SecretsCollection()
    with default_settings():
        secrets.scan_file('test_data/config.ini')

    import json
    print(json.dumps(secrets.json(), indent=2))

if __name__ == "__main__":
    main()

Please let me know if that fix works for you and if so we can make a note in the README file.

Hi @ttibbel ,
Thank you for the resolution, this seems about right. I got it to work properly now.


Fix inconsistent javax.validation constraints on interface implementations

Unfortunately there was an inconsistency where one implementation marked a field @NotEmpty instead of @NotBlank. This caused runtime exception:
javax.validation.ConstraintDeclarationException: HV000151: A method overriding another method must not redefine the parameter constraint configuration, but method ScriptCommandSelectorImpl#select(Set, JobRequest, String) redefines the configuration of ResourceSelector#select(Set, JobRequest, String)
Because integration tests don't yet use this path it wasn't caught in testing.

Coverage increased (+0.05%) to 92.72% when pulling 99e6994595152b3507ddfb64be51dd84276aad83 on tgianos:fixInconsistentValidationConstratints into 5fea909008524874c11bee4d0349d92c397e6913 on Netflix:master.


parse percentages as float

This one could be called "personal preference" or even "breaking compatability for no good reason", but since it's a change I will be running locally I thought I should at least throw it out there:
All values returned by the API are parsed as floats (or ints), with only three exceptions: urls, achievements (which are boolean) and percentage value. Since mathematically, "50%" == 0.5, it seems more consistent to me to parse percentage values as float. This leads to entire sections, like the hero stats, to become a collection of float values.
As a concrete example, this change changes
"stats": {
  "competitive": {
    "d.va": {
      "general_stats": {
        "teleporter_pads_destroyed": 3.0,
        "time_played": 3.0,
        "weapon_accuracy_best_in_game": "66%",
        "win_percentage": "41%"

to
"stats": {
  "competitive": {
    "d.va": {
      "general_stats": {
        "teleporter_pads_destroyed": 3.0,
        "time_played": 3.0,
        "weapon_accuracy_best_in_game": 0.66,
        "win_percentage": 0.41

Note that I have left the manually computed overall quickplay/competitive win_rate alone in an effort to break less stuff, meaning that it is still reported as an integer between 0 and 100. But that value was never formatted consistently with other pecentage values like a hero's win_percentage, and is currently broken for quickplay anyways.

Right, this is the one change I'm going to wait to merge as it breaks for both v2 and v3 APIs.
I will make an issue with the warning about the backwards incompatible change then merge it after a week.

That sounds like a good call.

@SunDwarf it's been 2 weeks now.


issue while using "unique: true" for the sequence data type

Environment

Python version: 3.5.2
PyKwalify version: 1.8.0

Steps to Reproduce

create input YML file as below
test.yml

mydata:
name: 123
age: ABCD
hobbies: [cooking, painting, cooking]

create YML schema as below

type: map
mapping:
 mydata:
  type: map
  mapping:
    name:
      type: str
      range:
       min: 1
       max: 10
    age:
       type: int
       range:
          min: 21
          max: 100
    hobbies:
       type: seq
       sequence:
       -  type: str
          unique: true
          required: true

Write Python test case for validation as below
validate_yml.py

#! /usr/bin/python3

@author: Minal Deshmukh

import pykwalify
from pykwalify.core import Core, SchemaError
from pykwalify.errors import RuleError

yaml_file = "./test.yml"
schema_file = "./schema.yml"
yaml_schema = Core(source_file=yaml_file, schema_files=[schema_file])
try:
    pykwalify.init_logging(0)
    yaml_schema.validate(raise_exception=True)
except (SchemaError, RuleError):
    for error in yaml_schema.errors:
        print("error -->",error)
        print ("Data Type of error is -->",type(error))

Execute test case as below
python3 validate_yml.py

OUTPUT
error --> Value 'ABCD' is not of type 'int'. Path: '/mydata/age'
Data Type of error is --> <class 'pykwalify.errors.SchemaError.SchemaErrorEntry'>
error --> Value '123' is not of type 'str'. Path: '/mydata/name'
Data Type of error is --> <class 'pykwalify.errors.SchemaError.SchemaErrorEntry'>
error --> Value 'cooking' is not unique. Previous path: '/mydata/hobbies/0'. Path: '/mydata/hobbies/2'
Data Type of error is --> <class 'str'>

Schema
type: map
mapping:
 mydata:
  type: map
  mapping:
    name:
      type: str
      range:
       min: 1
       max: 10
    age:
       type: int
       range:
          min: 21
          max: 100
    hobbies:
       type: seq
       sequence:
       -  type: str
          unique: true
          required: true

Data
mydata:
    name: 123
    age: ABCD
    hobbies: [cooking, painting, cooking]

Expected Behavior
Expected data type of the error below is 'pykwalify.errors.SchemaError.SchemaErrorEntry'
error --> Value 'cooking' is not unique. Previous path: '/mydata/hobbies/0'. Path: '/mydata/hobbies/2'
Data Type of error is --> <class 'str'>

Observed Behavior
Data type is different than the other error data types
error --> Value 'cooking' is not unique. Previous path: '/mydata/hobbies/0'. Path: '/mydata/hobbies/2'
Data Type of error is --> <class 'str'>

Mkay, good catch. I think the error stems from 2 different places
https://github.com/Grokzen/pykwalify/blob/master/pykwalify/core.py#L449 is where the errors is stored during parsing to later be added to the errors list. But when the iterator runs over this temporary lits here https://github.com/Grokzen/pykwalify/blob/master/pykwalify/core.py#L468 we run iteration over a dict and i guess you get out the keys only and in this case that would be the __repr__ part of the key and not the exception object itself.

Mkay, good catch. I think the error stems from 2 different places
https://github.com/Grokzen/pykwalify/blob/master/pykwalify/core.py#L449 is where the errors is stored during parsing to later be added to the errors list. But when the iterator runs over this temporary lits here https://github.com/Grokzen/pykwalify/blob/master/pykwalify/core.py#L468 we run iteration over a dict and i guess you get out the keys only and in this case that would be the __repr__ part of the key and not the exception object itself.

Thank you for looking into it & your immediate response.
When can we expect fix for this issue?

@MinalRajendra I can't give any estimate as i can't give this project to much of my free time right now. If you want a fix, the absolute fastest way is for you to test to modify the parts above i mentioned and submit a PR. Otherwise, maybe in Dec, most likley in Jan at this point


Misfunction on crontab

Hi team,
I really love this tool, so hope you can help me with the error I have.
I use this tool for many of my task, including activating/deactivating crontab jobs on the start/stop of the SAS services, backups and archival processes.
Recently there is the need to keep a constant monitoring on the SAS services across the SAS Server. To solve this I added a crontab job to check the status of the SAS services using LSM utility. For this I created a copy of my config file and added the following line to the crontab:
*/5 * * * * bash /apps/thirdparty/lsm/SAS_lsm -s -c /apps/thirdparty/lsm/sas_ci_cron.cfg  >/dev/null 2>&1
At the beginning this was working like a charm, however and apparently without any reason I started receiving emails regarding the failure on the status check using the tool, so when the team took a look into the services all was working properly, also if we executed the "status" functionality manually it was working properly, so the decision was to remove the crontab job until we figure out what happened and to make sure we can avoid false positives cases as this one.
The false positive alarms were running from Tue Jul 12 23:00:13 EDT 2022 and until Wed Jul 13 00:30:12 EDT 2022 , at that time there were no additional reports, there was no any outage, all servers were accessible, but the tool triggered by crontab was not working properly, also manual executions returned the correct status.

Hi acamacho 85,
In this case the error is not a false positive.  The error message indicates that the SASlsm utility does not understand the output from the sas.servers status command on 3 of the tiers.  Essentially it is not claiming that anything is down, just that it can't determine if it is up or down.
Can you share/paste the config file that the cronjob was using?  And the one you use for manual checks.

Hi Greg,
Just to clarify, is not failing every time, it was set and was working properly but some hours after that, it began failing.
What I want to avoid is that this behavior happen especially in my non-working hours alerting users with these alarms when services are available.
This is the config file we use for manual validations:
cat sas_ci.cfg
##########################################################################################################################################
BEGIN MISCELLANEOUS VARIABLES
REQUIRED: CONFIGURATION DIRECTORY (E.G. /opt/SAS/94/production/config/Lev1)
if you have unique CONFIGURATION directories for the TIERs in your deployment, you can define the CONFIGDIR value before each TIER definition
CONFIGDIR=/apps/config/Lev1
REQUIRED: if you have unique INSTANCE OWNER userids for the TIERS in your deployment, you can define the INSTID value before each TIER definition
INSTID=svc_npebidsasadm
REQUIRED: NUMBER OF TIERS IN THIS CONFIGURATION FILE
MAXTIERS=4
REQUIRED/CONFIGURABLE: SSH OPTIONS
SSHOPTIONS="-q -o StrictHostKeyChecking=no -o PasswordAuthentication=no"
REQUIRED/CONFIGURABLE NUMBER OF RETRIES
MAXRETRY=10
REQUIRED/CONFIGURABLE: SLEEP INTERVAL IN SECONDS PER RETRY
SLEEP=6
REQUIRED/CONFIGURABLE: STATUS REPORT ROOT DIRECTORY
STATUSROOT=/apps/thirdparty/lsm/logs
REQUIRED/CONFIGURABLE: offset in hours that is used to trim centralized log files
TRIMOFFSET=1
OPTIONAL: SAS ADMINISTRATOR EMAIL ADDRESS(ES)
REMOVE # in COLUMN 1 OF THE VARIABLE TO ACTIVATE
#SASADMIN="ADDRESS1 ADDRESS2 ADDRESSN"
<EMAIL_ADDRESS>END MISCELLANEOUS VARIABLES
##########################################################################################################################################
##########################################################################################################################################
BEGIN OPTIONAL sas.servers services Tier Descriptor Template
REMOVE # in COLUMN 1 OF TIER VARIABLES AND REPLACE ARRAY ELEMENT VALUE "N" WITH SEQUENTIAL NUMERIC VALUE TO ACTIVATE
TIERNAME[1]='Metadata'
TIERINST[1]=${INSTID}
TIERHOST[1]=sasnldurdev01
TIERSTAR[1]="${CONFIGDIR}/sas.servers start"
TIERSTOP[1]="${CONFIGDIR}/sas.servers stop"
TIERREST[1]="${CONFIGDIR}/sas.servers restart"
TIERSTAT[1]="${CONFIGDIR}/sas.servers status"
TIERSTAS[1]='is NOT up'
TIERSTOS[1]='is UP'
TIERLOGD[1]=${CONFIGDIR}
TIERTYPE[1]='standard'
TIEREXIT[1]=/apps/thirdparty/lsm/user_scripts
TIERNAME[2]='MA Compute'
TIERINST[2]=${INSTID}
TIERHOST[2]=sasnldurdev02
TIERSTAR[2]="${CONFIGDIR}/sas.servers start"
TIERSTOP[2]="${CONFIGDIR}/sas.servers stop"
TIERREST[2]="${CONFIGDIR}/sas.servers restart"
TIERSTAT[2]="${CONFIGDIR}/sas.servers status"
TIERSTAS[2]='is NOT up'
TIERSTOS[2]='is UP'
TIERLOGD[2]=${CONFIGDIR}
TIERTYPE[2]='standard'
TIEREXIT[2]=/apps/thirdparty/lsm/user_scripts
TIERNAME[3]='MO Compute'
TIERINST[3]=${INSTID}
TIERHOST[3]=sasnldurdev03
TIERSTAR[3]="${CONFIGDIR}/sas.servers start"
TIERSTOP[3]="${CONFIGDIR}/sas.servers stop"
TIERREST[3]="${CONFIGDIR}/sas.servers restart"
TIERSTAT[3]="${CONFIGDIR}/sas.servers status"
TIERSTAS[3]='is NOT up'
TIERSTOS[3]='is UP'
TIERLOGD[3]=${CONFIGDIR}
TIERTYPE[3]='standard'
TIEREXIT[3]=/apps/thirdparty/lsm/user_scripts
TIERNAME[4]='Midtier'
TIERINST[4]=${INSTID}
TIERHOST[4]=sasnldurdev04
TIERSTAR[4]="${CONFIGDIR}/sas.servers start"
TIERSTOP[4]="${CONFIGDIR}/sas.servers stop"
TIERREST[4]="${CONFIGDIR}/sas.servers restart"
TIERSTAT[4]="${CONFIGDIR}/sas.servers status"
TIERSTAS[4]='is NOT up'
TIERSTOS[4]='is UP'
TIERLOGD[4]=${CONFIGDIR}
TIERTYPE[4]='standard'
TIEREXIT[4]=/apps/thirdparty/lsm/user_scripts
END OPTIONAL sas.servers services Tier Descriptor Template
##########################################################################################################################################
This is the one for the crontab, as you can see there is no difference between files, just created 2 separated files as per the recommendations to avoid any conflict while running the same command multiple times on the servers.
cat sas_ci_cron.cfg
##########################################################################################################################################
BEGIN MISCELLANEOUS VARIABLES
REQUIRED: CONFIGURATION DIRECTORY (E.G. /opt/SAS/94/production/config/Lev1)
if you have unique CONFIGURATION directories for the TIERs in your deployment, you can define the CONFIGDIR value before each TIER definition
CONFIGDIR=/apps/config/Lev1
REQUIRED: if you have unique INSTANCE OWNER userids for the TIERS in your deployment, you can define the INSTID value before each TIER definition
INSTID=svc_npebidsasadm
REQUIRED: NUMBER OF TIERS IN THIS CONFIGURATION FILE
MAXTIERS=4
REQUIRED/CONFIGURABLE: SSH OPTIONS
SSHOPTIONS="-q -o StrictHostKeyChecking=no -o PasswordAuthentication=no"
REQUIRED/CONFIGURABLE NUMBER OF RETRIES
MAXRETRY=10
REQUIRED/CONFIGURABLE: SLEEP INTERVAL IN SECONDS PER RETRY
SLEEP=6
REQUIRED/CONFIGURABLE: STATUS REPORT ROOT DIRECTORY
STATUSROOT=/apps/thirdparty/lsm/logs
REQUIRED/CONFIGURABLE: offset in hours that is used to trim centralized log files
TRIMOFFSET=1
OPTIONAL: SAS ADMINISTRATOR EMAIL ADDRESS(ES)
REMOVE # in COLUMN 1 OF THE VARIABLE TO ACTIVATE
#SASADMIN="ADDRESS1 ADDRESS2 ADDRESSN"
<EMAIL_ADDRESS>END MISCELLANEOUS VARIABLES
##########################################################################################################################################
##########################################################################################################################################
BEGIN OPTIONAL sas.servers services Tier Descriptor Template
REMOVE # in COLUMN 1 OF TIER VARIABLES AND REPLACE ARRAY ELEMENT VALUE "N" WITH SEQUENTIAL NUMERIC VALUE TO ACTIVATE
TIERNAME[1]='Metadata'
TIERINST[1]=${INSTID}
TIERHOST[1]=sasnldurdev01
TIERSTAR[1]="${CONFIGDIR}/sas.servers start"
TIERSTOP[1]="${CONFIGDIR}/sas.servers stop"
TIERREST[1]="${CONFIGDIR}/sas.servers restart"
TIERSTAT[1]="${CONFIGDIR}/sas.servers status"
TIERSTAS[1]='is NOT up'
TIERSTOS[1]='is UP'
TIERLOGD[1]=${CONFIGDIR}
TIERTYPE[1]='standard'
TIEREXIT[1]=/apps/thirdparty/lsm/user_scripts
TIERNAME[2]='MA Compute'
TIERINST[2]=${INSTID}
TIERHOST[2]=sasnldurdev02
TIERSTAR[2]="${CONFIGDIR}/sas.servers start"
TIERSTOP[2]="${CONFIGDIR}/sas.servers stop"
TIERREST[2]="${CONFIGDIR}/sas.servers restart"
TIERSTAT[2]="${CONFIGDIR}/sas.servers status"
TIERSTAS[2]='is NOT up'
TIERSTOS[2]='is UP'
TIERLOGD[2]=${CONFIGDIR}
TIERTYPE[2]='standard'
TIEREXIT[2]=/apps/thirdparty/lsm/user_scripts
TIERNAME[3]='MO Compute'
TIERINST[3]=${INSTID}
TIERHOST[3]=sasnldurdev03
TIERSTAR[3]="${CONFIGDIR}/sas.servers start"
TIERSTOP[3]="${CONFIGDIR}/sas.servers stop"
TIERREST[3]="${CONFIGDIR}/sas.servers restart"
TIERSTAT[3]="${CONFIGDIR}/sas.servers status"
TIERSTAS[3]='is NOT up'
TIERSTOS[3]='is UP'
TIERLOGD[3]=${CONFIGDIR}
TIERTYPE[3]='standard'
TIEREXIT[3]=/apps/thirdparty/lsm/user_scripts
TIERNAME[4]='Midtier'
TIERINST[4]=${INSTID}
TIERHOST[4]=sasnldurdev04
TIERSTAR[4]="${CONFIGDIR}/sas.servers start"
TIERSTOP[4]="${CONFIGDIR}/sas.servers stop"
TIERREST[4]="${CONFIGDIR}/sas.servers restart"
TIERSTAT[4]="${CONFIGDIR}/sas.servers status"
TIERSTAS[4]='is NOT up'
TIERSTOS[4]='is UP'
TIERLOGD[4]=${CONFIGDIR}
TIERTYPE[4]='standard'
TIEREXIT[4]=/apps/thirdparty/lsm/user_scripts
END OPTIONAL sas.servers services Tier Descriptor Template
##########################################################################################################################################

In your config file for the crontab status runs, try adding "-o ConnectTimeout=30" in the SSHOPTIONS variable so that it looks like this:
SSHOPTIONS="-q -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=30"
Let me know if the behavior is any different.

Hi Greg,
I've updated the config files but I got the same behavior after one day.
COMMAND: SAS_lsm -s -c /apps/thirdparty/lsm/sas_ci_cron.cfg
<< INFO >> Checking SAS_lsm version...
<< INFO >> Current SAS_lsm version 4.1.0 was successfully verified as up-to-date.

The behavior seems to me like ssh either fails or takes too long to return (even after the 30 second timeout was added), but it would be odd if that happened in cron but not in a manual execution of the SAS-lsm script. Both of those should be doing the exact same thing and making the same ssh call. Even more odd is that it sounds like it is inconsistent, which would make me think a network or sshd-related issue.
If you're able to catch it when it happens, try to manually run the ssh command it shows after << EXECUTING >> and see what it returns. Swap the -q (quiet) for -vvv for maximum verbosity (example:  ssh -vvv -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=30 svc_npebidsasadm@sasnldurdev02 /apps/config/Lev1/sas.servers status ).
Alternatively, update your SAS-lsm configuration file and replace the -q with -vvv in SSHOPTIONS (example: SSHOPTIONS="-vvv -o StrictHostKeyChecking=no -o PasswordAuthentication=no -o ConnectTimeout=30" ). This will make SAS-lsm's output look like a mess, but it doesn't seem to prevent SAS-lsm from understanding any of the command responses that it's reading, at least for status... just tested on my sandbox and it was still able to determine deployment state.
If we can see ssh returning some error message when the behavior occurs, that might explain what's going on.

Hi Andy,
Usually when we receive the notification we check in both ways, this is with the LSM (using a different .cfg file) and with the sas.servers scripts. Also we have tested using the cfg file used for the crontab job and we get good response (weird). Anyway I'm adding the verbose (-vvv) options and will let you know what happen next time.

Hi Andy / Greg,
Using the verbose I was able to identify the root cause of the issue. This was caused due to our accounts are authenticated by GSSAPIAuthentication. When GSSAPIAuthentication is enabled , SSH uses Kerberos ticket to authenticate the user.
If a cron job expects access to something in your home directory, (like /.ssh/authorized_keys) you need to make sure you have a valid kerberos ticket for the cronjob to be able to access.
In my case the ticket expired after 10 hours hence cron tab jobs start failing. To solve the issue along with the execution of the LSM tool I submit the kinit command to create a new or renew ticket-granting every 8 hours.


Tabbing to searchable select input adds arrow text

Package
filament/filament
Package Version
v2.17.17
Laravel Version
v10.4.1
Livewire Version
No response
PHP Version
PHP 8.2.2
Problem description
https://user-images.githubusercontent.com/888834/226725314-5a4b74ba-2d18-4a4f-b265-e6191e5d4906.mov
Apologies for the poorly named title. I've got a select input that is searchable. Tabbing from a previous field into the select and hitting the up or down arrow adds "arrowdown"/"arrowup" respectively. If you hit ESC and then press an arrow key again, it'll append the previous text.
Expected behavior
Not to display the text or not to duplicate the text each time.
Steps to reproduce

Login<EMAIL_ADDRESS>/ password)
Visit admin/users/create
Focus name input
Press tab
Press up or down arrow (see text)
Press ESC
Press up or down arrow (see repeated text).

Reproduction repository
https://github.com/jameshine/filament-searchable-select-bug
Relevant log output
No response

Very weird. Sure this isn't a Choices.js bug?

Duplicate of https://github.com/filamentphp/filament/issues/5924?

Don't think so

Pretty sure it is. Check https://filament-bug.jordanwelch.com.

Fair, it can also be replicated there. Not sure if the issue author originally intended it to be used for this problem too though

To me it seemed like the original issue report was this exact problem.

The original issue is to point out that the search doesn't work immediately after focusing. Not the arrowright text stuff

Actually, this particular issue is present in Choices.js itself, it's not a Filament issue.
https://choices-js.github.io/Choices/
Tab to the "Default" "Single select input" and then press the right arrow

You were right, @danharrin.
I mistakenly thought this was the exact same as linked issue.
However, they do all seem to be related to a bug in Choices.js.


Adding CorDA as an optional initialization method of LoRA

Existing PEFT methods are mostly agnostic of the context of a task of concern, e.g., a downstraem task to learn or some pre-trained world knowledge to maintain. In our paper (https://openreview.net/pdf?id=Gi00NVru6n) accepted by NeurIPS 2024, we propose a PEFT method named Corda, context-oriented decomposition adaptation, which builds task-aware LoRA adapters from weight decomposition oriented by the context of the task concerned for task-aware PEFT.
Concretely, CorDA randomly collects a few (by default 256) data samples from a target task, e.g. questions from a QA dataset or instructions to write a code or solve a math problem, and feeds these samples into a pre-trained LLM. We can obtain the covariance matrix of the input activation of each linear layer, i.e., $C=XX^T\in\mathcal{R}^{d_{in}\times d_{in}}$, where $X$ is the input of this linear layer. We then perform singular value decomposition (SVD) for the weight $W\in \mathcal{R}^{d_{out}\times d_{in}}$ multiplied by the covariance matrix, i.e., $\verb|SVD|(WC) = U\Sigma V^T$, where $U$ and $V$ are singular vectors and $\Sigma$ is the diagonal matrix with the singular values arranged in descending order. In this way, the context expressed by these representative covariance matrices is able to orientate the decomposition, such that the principal components are most associated with the task of concern. To ensure the same inference result with the pre-trained model at the start of adaptation, we multiply the inverse of these covariance matrices with the decomposed components, $\hat{W}=U\Sigma V^T C^{-1}$, where $\hat{W}$ is the weight after decomposition and reconstruction.
Thanks to the task-awareness, CorDA enables two optional modes, knowledge-preserving adaptation mode (KPM) and instruction-previewed adaptation mode (IPM). In KPM, we use questions from question-answering dataset whose knowledge needs to be preserved, such as TriviaQA and NQopen, to obtain the covariance matrices. After our context-oriented decomposition, we use the components with the smallest $r$ singular values, $U_{[:,-r:]}$, $\Sigma_{[-r:]}$, and $(V^TC^{-1})_{\left[-r:,:\right]}$ to initialize the learnable LoRA adapters $A=\sqrt{\Sigma}_{[-r:]}(V^TC^{-1})_{[-r:,:]}$ and $B=U_{[:,-r:]}\sqrt{\Sigma}_{[-r:]}$. The other components that compact the question-answering ability are frozen during adaptation. KPM enables to learn new tasks effectively while keeping the world knowledge associated with $C$ as sound as possible. Alternatively, when one only aims to achieve performance as high as possible on the finetuning task without concern for world knowledge maintenance, our IPM will be favored. In this mode, CorDA uses the instruction and response from the fine-tuning task (e.g., Math or Code) to produce the covariance matrices. The principal components with large singular values capturing the characteristics of the finetuning task in advance can better accommodate the new ability. So we initialize adapters as $A= \sqrt{\Sigma}_{[:r]} (V^T C^{-1})_{[:r,:]}$ and $B =U_{[:,:r]} \sqrt{\Sigma}_{[:r]}$, and freeze the remaining components. The implementations of KPM and IPM are compared as follows:

Mode
Collect covariance from
LoRA $A$
LoRA $B$

KPM
questions from a knowledge benchmark to maintain
$A=\sqrt{\Sigma}_{[-r:]}(V^T C^{-1})_{[-r:,:]}$
$B=U_{[:,-r:]}\sqrt{\Sigma}_{[-r:]}$

IPM
instructions and responses from a downstream task to learn
$A= \sqrt{\Sigma}_{[:r]} (V^T C^{-1})_{[:r,:]}$
$B =U_{[:,:r]} \sqrt{\Sigma}_{[:r]}$

Similar to PiSSA, our method is also an initialization method for LoRA adapter with the same LoRA structure, and our IPM also uses the principal components to initialize $A$ and $B$. But PiSSA adopts the naive SVD and does not consider task awareness. Our method builds task-aware adapters, enabling two optional modes to satisfy customized needs:

KPM not only achieves better fine-tuning performance than LoRA but also effectively preserves pre-trained knowledge.
IPM can further accelerate convergence and enhance the fine-tuning performance on downstream tasks, surpassing PiSSA in our experiments.

Some results are shown below.
1. The superiority of our decomposition method.

Plain SVD is the normal SVD on pre-trained weights, as adopted by PiSSA. This experiment discards the smallest $r$ components after different decomposition methods and tests the perplexity on Wiki and PTB. It indicates that our decomposition better compacts the associated task abilities into the principal components, and thus can better maintain the corresponding knowledge by freezing them in KPM, or better learn new abilities by adapting them in IPM.
2.  Adaptation performance with instruction-previewed mode (sample from Math, fine-tune on Math)

Method
Model
GSM8k
Math

LoRA
Llama-2-7b
42.68
5.88

PiSSA
Llama-2-7b
51.63
7.32

CorDA (IPM)
Llama-2-7b
53.45
8.64

LoRA
Llama-2-13b
57.24
8.92

PiSSA
Llama-2-13b
60.88
11.08

CorDA (IPM)
Llama-2-13b
62.47
11.54

LoRA
Gemma-2-9b
83.47
42.30

PiSSA
Gemma-2-9b
84.23
43.52

CorDA (IPM)
Gemma-2-9b
84.45
43.88

@BenjaminBossan Thanks for your review! As @iboing 's collaborator, I've revised the code according to your comments.

@BenjaminBossan  Thank you for your review!

Thanks a lot for the updates. The week has been quite busy so far, but I'm hopeful I can give this a proper review tomorrow. Just from reading the description of the method and skimming the code, I have some questions/comments:

The description of the method mentions that it depends on the input data. I could not find in the code where this happens, for instance preprocess_corda does not take any data as input and in the corda_finetuning.py script I also don't see how CorDA incorporates the dataset. Could you please point me to where this happens?
You're using logs throughout the code. In general, we don't do this in PEFT. Could you please remove those calls.
Purely from the description (but I haven't studied the paper yet), this method reminded me of EVA, which recently added to PEFT (#2142, https://arxiv.org/abs/2410.07170). Do you know if the two methods are similar? AFAICT, the respective papers don't mention another.

@BenjaminBossan Thanks for your comments! Let me clarify:

I also don't see how CorDA incorporates the dataset. Could you please point me to where this happens?

In corda_finetuning/preprocess.py#L38-L59
 calib_loader is the dataset iterator, and we're passing config.corda_config.run_model = lambda: run_model(model, calib_loader), this way CorDA gets a closure that captures a reference to the dataset.
This does not happen in corda_finetuning/corda_finetuning.py because it's the file to do the actual finetuning, but as we've preprocessed CorDA initial weights and residual model, we can train it like standard LoRA models, thus there's no need for dataset.

You're using logs throughout the code. In general, we don't do this in PEFT. Could you please remove those calls.

Sure, I've updated the PR so that logs are removed.

@BenjaminBossan Thanks for your detailed review!
I've removed run_model and hooked_model from CordaConfig, user should pass the model runner to preprocess_corda before calling get_peft_model as described in updated documents. This is different from EVA where the model weights are initialized after calling get_peft_model, but these two methods should all address the config serialization problem you've mentioned. Example usages are updated synchronously.
Also, I've gone through all your comments, please check my code edits and replies.

@5eqn Thanks so much for the updates. LMK if you want to get a review now or if I should wait until you finish the remaining changes. Also, could you please fix the small merge conflict?

@BenjaminBossan Remaining changes are pushed, I've added the ability to convert CorDA weights to LoRA, and added supplementary tests for this feature. The code is ready for review.

@BenjaminBossan Thank you so much for your review!

I added a suggestion to completely get rid of CordaInitConfig, please check if that makes sense.

It absolutely makes sense, I've changed the code so that preprocess_corda accepts run_model and hooked_model separately as optional arguments.

I could not run the included example as is, because it assumes that the data is already downloaded. Ideally, the scripts should run for the users when they follow the instructions. One solution would be to load the data from HF Hub, another to add an instruction on how and where to download the data.

Sure, I've changed the code so that it directly loads dataset from @iboing's CorDA collection.

I have added some argument explanation for the advanced usage in Readme, and will complement the command for convert corda to lora in readme soon. It is ready for re-review.

The command for converting CorDA to LoRA is added in the example README.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @BenjaminBossan  , regarding the memory consumption, the covariance is collected in torch.float32 by default. To improve memory usage, users can specify use_float16_for_covariance=True in CordaConfig to collect covariance matrices in torch.float16. But this may cause numerical instability only in a few cases, such that the initialized model does not ensure the exact same inference result as the original model. Users are suggested to check, e.g., comparing the inference result of Wiki/PTB perplexity before and after preprocessing, if they choose to perform in torch.float16.
I have added this note about memory in the doc in this update .

I've pushed a commit that calculates sample_size automatically in preprocess_corda, so that the user no longer needs to specify it in preprocess_corda. The document of preprocess_corda has updated accordingly.

Sorry for the delay in reviewing, we had a release past Friday and are fighting the CI today, hopefully I'll have time tomorrow.

@BenjaminBossan Thanks for your review! I've read your comments, the code will be updated later.

Hi @BenjaminBossan , we've updated the code according your comments. The code is now ready for review.

Hi @iboing @5eqn, this is very interesting work!
I am a first author of the EVA paper and we noticed that in the table in your README you specified EVA as not being able to maintain knowledge of the base model.
We (tagging @sirluk here) think that this might be a mistake, as with EVA we can simply recover the base model performance by unloading the adapters. This is not the case for PiSSA or CorDA as they alter the base model's weights during initialization so they are subject to change during fine-tuning, i.e. you cannot restore the base model weights as they change during fine-tuning. By initializing with the components that contain the least amount of information this can be mitigated as shown in your work. However, we intentionally followed the design of LoRA (B = 0), which allows us to fully recover the base model by unloading adapter weights, i.e. full knowledge maintenance of the base model. Therefore, we believe the table should be altered to say "yes" for EVA in the knowledge maintenance column. What are your thoughts @iboing @5eqn?
We also updated our manuscript on arXiv and added a reference to CorDA :)
Also many thanks to @BenjaminBossan as he made us aware of this pull request.

Hi @paischer101 , many thanks for your attention. I think I have made some explanations when you just came by my poster at NeurIPS'24.
Actually PiSSA, CorDA, OLoRA have the ability of unloading the adapters and keeping the base weights unchanged. Concretely for PiSSA and CorDA, the weight change is $\Delta W=AB-A_0 B_0$, and we can keep the base weight unchanged and set the lora adapter as [ $A$, $A_0$] and [ $B$, $-B_0$] , which doubles the rank. So, all these methods can unload the adapters and keep base weights unchanged.
Here "maintain knowledge" does not refer to the ability to keep base weights unchanged. LoRA and PiSSA are task-agnostic. Your method EVA is data-dependent considering the task for finetuning, but seems to not include the task awareness of pre-trained knowledge to maintain. I do not find any results in your paper about preserving knowledge. Our method explicitly associate a knowledge benchmark with the pricipal components after our decomposition and keep them frozen in KPM. This is what "maintain knowledge" means in our context. So, we mark "no" in that column.
We can remove this table but it is up to the suggestion of @BenjaminBossan

Thanks for the constructive discussion @paischer101 and @iboing. From my point of view, this looks like a question of terminology. If we can keep the table, but add a sentence to clarify what knowledge preservation means, that should be enough. I think the idea is already explained well elsewhere, but for readers who directly jump to the table, it could make sense to add a short description below that explains what is meant be "Support knowledge maintenance" (which btw. should be "Supports").

Hi @BenjaminBossan  , thanks for your suggestion. I have added a sentence of explanation below the table. Please let us know if there is further necessary change for this pr.

Sorry for the delayed response on my end.
@iboing thanks for clarofication on unloading the weights, you are correct, this is possible for the other inits as well.
Also, I agree with @BenjaminBossan, it is a question of terminology. KPM in CorDA is simply a different use case we did not include in our paper. The terminology of "knowledge preservation" led to this confusion.
In any case, thanks @iboing for adding the clarifying sentence to clear up such confusions.

@iboing Could you please run make style? The doc builder is complaining, probably because of the long line.

@githubnemo @BenjaminBossan Thank you for your review! I've added handle removal as proposed by @githubnemo , and also fixed the logic of automatically dividing sample_count after collecting covariance. Corresponding tests are appended.

I think this is good to go :)
Thank you for the great work everyone!


User/mlu/faq reklamation für kassenhändler

Hi @MLU0408,
thanks for the PR! I added one comment about the filename - when this is fixed, we'll merge it :+1:

Okay perfect. Then I will communicate that this FAQ is online :)


Bug: cannot create or drop database with custom config

i cannot create or drop my database with custom configuration provided in your documentation
this is my code :
import { DataSourceOptions } from 'typeorm'
import { dropDatabase } from 'typeorm-extension'

void (async () => {
  const dataSourceOptions: DataSourceOptions = {
    type: 'postgres',
    host: 'localhost',
    username: 'masb0ymas',
    password: 'mypassword',
    database: 'express-typeorm',
    port: 5432,
  }

  // Create the database with specification of the DataSource options
  await dropDatabase({ options: dataSourceOptions })
})()

run script :
npx ts-node ./src/database/dropDB.ts

the error message says: error: database "masb0ymas" does not exist
Please help me, find solution with the problem.
Versions

Node: 14.18.1
OS: macOS 11.6

Reproduction

Additional Details

Steps to reproduce
What is Expected?
What is actually happening?

The problem here is, that postgres requires an initialDatabase to connect to.
By default the name is equal to the name of the database user (in your case: masb0ymas)

the createDatabase function is working normally, but the dropDatabase threw error error: database "masb0ymas" does not exist.
is initialDatabase parameter exist in dropDatabase function ?
dropDatabase({ options: dataSourceOptions, initialDatabase: '<db-name>' })

Actually no 🙈. Good catch! I will fix this today and publish a new release.

Sorry, did not made it in time yesterday.
Issue should be fixed in e571418bf6d1598aae92e02d1b3409ad20f45328

Thanks 👍


Relax substrate licensing scheme

As promised a while back, ahead of the release of 2.0 this PR relaxes the Licenses substrate is being released under. In particular this changes the license as follows:

Everything in client stays GPL-3, but with the additional classpath linking extension
frame, primitives, the utilities and binaries are now released under "Apache-2".

This PR:

[x] updates the headers accordingly (while keeping the copyright years in tact)
[x] updates the Cargo.tomls with the new SPDX term
[x] adds the full Apache-2 License full-text in LICENSE-APACHE2
[x] renames the existing LICENSE to LICENSE-GPL3, adding the linking exception
[x] Updates the Readme to explain the licensing scheme and link to the proper licenses.

Please note: this is not the place discuss the why or pro and cons of this license or any other scheme. Please contact @bjornwgnr if you want to understand better why this schema has been chosen.

bot merge


Implement MS.AAD.7.2v1 Policy to check Secure Score

🗣Implement MS.AAD.7.2v1 - Secure Score for Least Privilege Policy##

Implements MS.AAD.7.2v1
Closes #375

💭 Motivation and context

Change allows for automated check eliminating previous manual check

Checks the Secure Score for "Least Privileged Administrative Roles" is 100%

🧪 Testing

Added Rego unit test and checked all unit tests pass
Ran against E5, G5, and G3 tenants and all passed

scubag3forthee:

✅ Pre-approval checklist

[x] This PR has an informative and human-readable title.
[x] Changes are limited to a single goal - eschew scope creep!
[x] All future TODOs are captured in issues, which are referenced
in code comments.
[x] All relevant type-of-change labels have been added.
[x] I have read the CONTRIBUTING document.
[x] These code changes follow cisagov code standards.
[x] All relevant repo and/or project documentation has been updated
to reflect the changes in this PR.
[x] Tests have been added and/or modified to cover the changes in this PR.
[x] All new and existing tests pass.

✅ Pre-merge checklist

[ ] Revert dependencies to default branches.
[ ] Finalize version.

✅ Post-merge checklist

[ ] Create a release.

There are some OPA issue with "Testing/Unit/Rego/AAD/AADConfig_07_test.rego:3: rego_compile_error: import data.report.utils.NotCheckedDetails unused" Can you try to fix the unit test to remove the line

@ssatyapal123, Can you please resolve the conflicts for this to be merged into Emerald. thanks.


Wording Embedding

Hi, thanks for sharing codes. I would ask whether I need to re-build a new word embedding if I change an image dataset (not ImageNet).  Thanks

Sorry to ask you one more question. Do you have the codes for ConSE and SynC in python version? Thanks


Updated README.md

User description

fixed links
Updated UI for Readme.md
Created "Table of contents" for better navigation on page

PR Type
Documentation

Description

Enhanced the README.md with a new user interface, including badges for tests, styling, code coverage, version, downloads, and license.
Added a "Table of Contents" to improve navigation and accessibility of the document.
Updated installation and usage instructions with detailed steps and examples.
Improved overall formatting and alignment for better readability and presentation.

Changes walkthrough 📝
Relevant filesDocumentation

README.mdRevamped README with badges, TOC, and detailed instructions

README.md
Added badges for tests, styling, code coverage, version, downloads, and license.  Introduced a "Table of Contents" for better navigation.  Enhanced sections with detailed instructions for installation, usage, and configuration.  Improved formatting and alignment for better readability.

+227/-167

💡 PR-Agent usage: Comment /help "your question" on any pull request to receive relevant information

@SURYANSH-RAI Please use the correct coding style.


memory

Our pipeline can sometimes use a lot of memory (especially when running the GTEX samples). This often results in issues where some jobs will get killed because we use all the memory on the cluster. Last I checked, this was happening when I used STAR to map reads (the second time). It also occurred when I was using WASP's get_as_counts.
We can fix get_as_counts by implementing a lighter hdf5 version of it (see my fork of WASP), but I'm not sure about what to do with STAR. My guess is that it has something to do with loading all of the VCF data all at once but you should generally look into how much memory all of the rules use and whether there is anything you can do about it (besides just scaling down the number of jobs that snakemake will run at any given moment of time).

this article in the snakemake docs about resource allocation may be useful

It seems like things are less on fire now, especially after submitting this get_as_counts pull request into WASP. I'm going to close the issue for now


Druid builds do not run properly on Java 7

Druid builds need to be done with JDK8 compilers, because one of the extensions (druid-caffeine-cache) targets JDK8. But the other modules are supposed to target JDK7 and so we pass -source 1.7 -target 1.7 to javac. But, this is not enough to properly target JDK7, due to method signature changes the Java API. One example is that ConcurrentHashMap.keySet's return type changed from Set to KeySetView, leading to errors like the one below when Druid is run on JDK7.
We use this method in 4 places:

io.druid.indexing.overlord.supervisor.SupervisorManager
io.druid.indexing.kafka.supervisor.KafkaSupervisor
io.druid.firehose.rocketmq.RocketMQFirehoseFactory
io.druid.server.router.CoordinatorRuleManager

Presumably if one avoids the features corresponding to these classes (supervised ingestion, rocketmq extension, and the router; none of them are required) then things should work fine on JDK7. But still, this is not what we intended. Possible fixes:

Cast ConcurrentHashMap to Map before calling keySet
Pass the JDK7 runtime to bootclasspath when compiling with javac

If #3746 is agreed on then this issue becomes moot. See also https://bugs.openjdk.java.net/browse/JDK-8151366 for more details.
java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_121]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_121]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_121]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_121]
    at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.stop(Lifecycle.java:362) [java-util-0.27.10.jar:?]
    at com.metamx.common.lifecycle.Lifecycle.stop(Lifecycle.java:280) [java-util-0.27.10.jar:?]
    at io.druid.indexing.overlord.TaskMaster.stopLeading(TaskMaster.java:227) [druid-indexing-service-0.9.2.jar:0.9.2]
    at io.druid.indexing.overlord.TaskMaster.stop(TaskMaster.java:207) [druid-indexing-service-0.9.2.jar:0.9.2]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_121]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_121]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.7.0_121]
    at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_121]
    at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.stop(Lifecycle.java:362) [java-util-0.27.10.jar:?]
    at com.metamx.common.lifecycle.Lifecycle.stop(Lifecycle.java:280) [java-util-0.27.10.jar:?]
    at com.metamx.common.lifecycle.Lifecycle$1.run(Lifecycle.java:306) [java-util-0.27.10.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.7.0_121]
Caused by: java.lang.NoSuchMethodError: java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
    at io.druid.indexing.overlord.supervisor.SupervisorManager.stop(SupervisorManager.java:116) ~[druid-indexing-service-0.9.2.jar:0.9.2]
    ... 16 more

How did travis CI java7 build not catch this?

ohhh, because it compiles in java 7 for java 7

I bet the travis CI java7 build actually uses a java7 compiler.

You could use Animal Sniffer and Collections.newSetFromMap()

can you explain, "Pass the JDK7 runtime to bootclasspath when compiling with javac" a bit more?
will that be equivalent of compiling with java7 compiler instead of java8 compiler targeting version-7 ? If yes, I would like that because that will also take care of other similar (undiscovered) issues.
and, let us keep marching on java-8 only goal if tranquility and spark work fine but not let that be a blocker for the issue at hand.

The suggested work around from the maintainers is to cast to Map before calling keySet()
https://bugs.openjdk.java.net/browse/JDK-8151366?focusedCommentId=13973374&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13973374 Which sounds good to me

@himanshug "Pass the JDK7 runtime to bootclsaspath" was referring to this comment https://bugs.openjdk.java.net/browse/JDK-8151366?focusedCommentId=13909482&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13909482

Both of those two workarounds were suggested by different people in the JDK-8151366 ticket.

there is not way to quantitatively select either of those. let us just go with "casting to Map" workaround . if we find similar issues in future then we can switch to "pass the jdk7 runtime to bootclasspath" later.

That Animal Sniffer thing @ben-manes mentioned looks like it has a maven plugin (http://www.mojohaus.org/animal-sniffer/animal-sniffer-maven-plugin/), maybe that would help verify.

PR in #3796.

We use Java 8 to build and run druid code and facing this issue in our internal deployment. The way it manifests is that thread gets stuck at point where JavaCompatUtils.keySet(<Map>) is used. Using entrySet for iteration or explicitly casting to a Map like ((Map)m).keySet as suggested in https://bugs.openjdk.java.net/browse/JDK-8151366 works. Explicit cast however leads to loss of Generic type for key and values in the map.
Any thoughts ?

What do you mean by "the thread gets stuck"? We use jdk 8 to build and run Druid code too and haven't seen anything like that.

I mean the code beyond that point is not executed. For example, neither this for loop gets executed nor the control goes beyond the loop. I took a thread dump but that suggests that the supervisor thread is waiting on take method. So I am not entirely sure what's happening.
"KafkaSupervisor-v2_metrics_cluster-Scheduler-0" #196 daemon prio=5 os_prio=0 tid=0x00007fe780180000 nid=0x74b0 waiting on condition [0x00007fe6f9da2000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000004e1c30bc8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"KafkaSupervisor-v2_metrics_cluster" #195 daemon prio=5 os_prio=0 tid=0x00007fe780173800 nid=0x74af waiting on condition [0x00007fe6f9ea3000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000004e1c30df8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I added log statements to check how far the code is being executed.

Hmm, that seems like some sort of "Throwable gets thrown but not caught and thread dies" thing. Do you see interesting errors if you put a try/catch Throwable around whatever is calling keySet?

I just tried that and do not see any exceptions

Actually there is an Error thrown, I was just catching Exception and seems like I have an old version of java-util jar that does not have the JavaCompatUtils sorry for bothering. My mistake, I think it will work after I put in the new jar. Thanks


[read-write-set] Update diem transaction analyzer

Motivation
This is a PR consisted of multiple commits:

diem-tools/diem-read-write-set got moved to diem-vm. The main reason for this is for cyclic dependency: diem-read-write-set uses the system_module_names for analyzing transactions, whereas the crate will be used by diem-vm in the future.
Change the API to return read&write set at the same time. This is done to reduce the overhead of having to do substitution twice.
Implement the read-write-set analysis for block metadata transactions and script transaction.

Test Plan
TBD

LEC dependency change summary:
target packages:
  A read-write-set-dynamic 0.1.0 (workspace, path 'language/tools/read-write-set/dynamic')
    * features: [none]

/land


NTR: optogenetic evidence

optogenetic evidence: "Evidence arising from experiment in which neuronal activity is manipulated using genetically encoded, optically activated neuronal actuators."
subClassOf: 'experimental phenotypic evidence'

Closing as per discussion in #121


libclang_rt.builtins-wasm32.a not found ?

$ clang++ --target=wasm32 -Wl,--export-all -Wl,--no-entry -o main.wasm main.cpp

wasm-ld: error: cannot open crt1.o: No such file or directory
wasm-ld: error: unable to find library -lc++
wasm-ld: error: unable to find library -lc++abi
wasm-ld: error: unable to find library -lc
wasm-ld: error: cannot open /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/lib/clang/17/lib/libclang_rt.builtins-wasm32.a: No such file or directory
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

$ ls /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/lib/clang/17/lib/wasi/libclang_rt.builtins-wasm32.a
/home/nyue/systems/wasi-sdk/wasi-sdk-21.0/lib/clang/17/lib/wasi/libclang_rt.builtins-wasm32.a

$ which wasm-ld
/home/nyue/systems/wasi-sdk/wasi-sdk-21.0/bin/wasm-ld

I noticed that the actual location of the library is in a subdirectory call wasm so it seems wasm-ld is looking for library in the wrong place.
I am using the latest version 21 build of wasi-sdk and using it on Ubuntu
Kind regards

Are you running clang++ from the wasi-sdk?   What does which clang++ say?
The clang that ships with wasi-sdk I believe sets up a default sysroot, but if you use your system clang you would need to specify --sysroot (and also install libclang_rt.builtins-wasm32.a)

(BTW wasm-ld doesn't know how to construct library paths, those paths are all constructed by the clang driver.  You can see a little more about what clang is doing if you add -v to your command line)

Are you running clang++ from the wasi-sdk? What does which clang++ say?

$ which clang++
/home/nyue/systems/wasi-sdk/wasi-sdk-21.0/bin/clang++

(BTW wasm-ld doesn't know how to construct library paths, those paths are all constructed by the clang driver. You can see a little more about what clang is doing if you add -v to your command line)

$ clang++ -v --target=wasm32 -Wl,--export-all -Wl,--no-entry -o main.wasm main.cpp
clang version 17.0.6
Target: wasm32
Thread model: posix
InstalledDir: /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/bin
 "/home/nyue/systems/wasi-sdk/wasi-sdk-21.0/bin/clang-17" -cc1 -triple wasm32 -emit-obj -mrelax-all -dumpdir main.wasm- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name main.cpp -mrelocation-model static -mframe-pointer=none -ffp-contract=on -fno-rounding-math -mconstructor-aliases -target-cpu generic -fvisibility=hidden -debugger-tuning=gdb -v -fcoverage-compilation-dir=/data2/nyue/projects/vii/vii_git/programming/wasm/via_cpp -resource-dir /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/lib/clang/17 -isysroot /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/bin/../share/wasi-sysroot -internal-isystem /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/bin/../share/wasi-sysroot/include/c++/v1 -internal-isystem /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/lib/clang/17/include -internal-isystem /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/bin/../share/wasi-sysroot/include -fdeprecated-macro -fdebug-compilation-dir=/data2/nyue/projects/vii/vii_git/programming/wasm/via_cpp -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -o /tmp/main-424572.o -x c++ main.cpp
clang -cc1 version 17.0.6 based upon LLVM 17.0.6 default target wasm32-wasi
#include "..." search starts here:
#include <...> search starts here:
 /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/bin/../share/wasi-sysroot/include/c++/v1
 /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/lib/clang/17/include
 /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/bin/../share/wasi-sysroot/include
End of search list.
 "/home/nyue/systems/wasi-sdk/wasi-sdk-21.0/bin/wasm-ld" -m wasm32 -L/home/nyue/systems/wasi-sdk/wasi-sdk-21.0/bin/../share/wasi-sysroot/lib crt1.o --export-all --no-entry /tmp/main-424572.o -lc++ -lc++abi -lc /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/lib/clang/17/lib/libclang_rt.builtins-wasm32.a -o main.wasm
wasm-ld: error: cannot open crt1.o: No such file or directory
wasm-ld: error: unable to find library -lc++
wasm-ld: error: unable to find library -lc++abi
wasm-ld: error: unable to find library -lc
wasm-ld: error: cannot open /home/nyue/systems/wasi-sdk/wasi-sdk-21.0/lib/clang/17/lib/libclang_rt.builtins-wasm32.a: No such file or directory
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Can you try removing the --target flag or setting it to wasm32-wasi (which is the default)?
if that doesn't work can you include the output of the command with -v included?

Thank you @sbc100 , removing --target fixed it so my error was specifying the wrong target, if I specify the correct/default target (wasm-wasi) it compiles file.

Awesome!  Closing this out


oath token issue

Hi , How you pass oath token to the shopping API ?
I was looking for something like
public function setEBayAuthToken(?string $eBayAuthToken = null): self
which has been implemented for that purpose in src_trading/StructType  for trading API ,
any hints would be much appreciated ,
Thanks for looking at it
Emil

hi,
this SDK does not support the oauth token, sorry.
it only supports the 'auth'n'auth token, without the (unnecessary) overhead that comes with oauth.
are you working for a seller (or more) or is your something that provides a "login with ebay" button ? 😏

ah sorry, i forgot 😁 ebay changed that..... shopping only works with oauth.... that sucks.... let me see.....

i guess, somewhere at this point:
https://github.com/michabbb/php-ebay-sdk/blob/php8/base.php#L118
you could just try to add something like, you already see here:
https://github.com/michabbb/php-ebay-sdk/blob/php8/base.php#L102
so in our case, it would be:
if ((string)$this->Service==='macropage\ebaysdk\shopping\ServiceType\Service') {
    $this->Service->setHttpHeader('X-EBAY-API-IAF-TOKEN', $this->appid);
}

"appid" would be your "application access token".
"appid" is the wrong wording, but I keep it here, because of this line:
https://github.com/michabbb/php-ebay-sdk/blob/php8/test.php#L25
does this help ? 😏

Thanks mate , it helps indeed ,
no error anymore but response now looks like :
object(macropage\ebaysdk\shopping\StructType\GetSingleItemRequestType)#115 (6) {
["ItemID":protected]=>
string(12) "334664496876"
["VariationSKU":protected]=>
string(7) "Details"
["VariationSpecifics":protected]=>
NULL
["IncludeSelector":protected]=>
NULL
["MessageID":protected]=>
NULL
["any":protected]=>
NULL
}
not receiving any details from API despite item ID

and when as such object being used to call $item = $ShoppingService->GetSingleItem($SingleItemRequestType);
Exception: Bad Request
EbayAPIs/php-ebay-sdk-php7/src_shopping/ServiceType/Service.php', line 139.

Thats the request :
[2023-01-24T12:16:49.256360+00:00] name.DEBUG: Request {"url":"","name":"GetSingleItem","headers_request":"POST /shopping?callname=GetSingleItem&siteid=3&appid=################## token #################&version=1199&routing=new HTTP/1.1\r\nHost: open.api.ebay.com\r\nConnection: Keep-Alive\r\nUser-Agent: PHP-SOAP/7.4.29\r\nContent-Type: text/xml; charset=utf-8\r\nSOAPAction: ""\r\nContent-Length: 365\r\nX-EBAY-API-IAF-TOKEN: ################## token #################\r\n\r\n","headers_response":"HTTP/1.1 400 Bad Request\r\ndate: Tue, 24 Jan 2023 12:16:49 GMT\r\nserver: ebay-proxy-server\r\nconnection: close\r\ncontent-length: 0\r\n","request_xml":"\n<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/\" xmlns:ns1="urn:ebay:apis:eBLBaseComponents">\n  SOAP-ENV:Header/\n  SOAP-ENV:Body\n    ns1:GetSingleItemRequest\n      ns1:ItemID334664496876</ns1:ItemID>\n      ns1:VariationSKUDetails</ns1:VariationSKU>\n    </ns1:GetSingleItemRequest>\n  </SOAP-ENV:Body>\n</SOAP-ENV:Envelope>\n","response_xml":null,"last_error":{"macropage\ebaysdk\shopping\ServiceType\Service::GetSingleItem":"[object] (SoapFault(code: 0 faultcode: HTTP):
Bad Request at /EbayAPIs/php-ebay-sdk-php7/src_shopping/ServiceType/Service.php:139)"},"faultstring":"Bad Request","faultstring_detail":null} []

oh sorry... i forgot that the "app-id" is used in the URL as you can see in your logs:
https://github.com/michabbb/php-ebay-sdk/blob/6bed0ea37aa3d7ae0fb1dbd0567305c776fef5a5/base.php#L127
🤔 I don´t use the shopping API anymore, so can you please remove '&appid=' . $this->appid  in that line from the URL and make a test?
the trading API still needs that parameter, but only for testing the shopping API, please remove it and let me know.

Hi mate , after removing appid from url :
object(macropage\ebaysdk\shopping\StructType\GetSingleItemResponseType)#124 (7) {
["Item":protected]=>
NULL
["Timestamp":protected]=>
string(24) "2023-01-24T19:24:10.754Z"
["Ack":protected]=>
string(7) "Failure"
["Errors":protected]=>
array(1) {
[0]=>
object(macropage\ebaysdk\shopping\StructType\ErrorType)#111 (8) {
["ShortMessage":protected]=>
string(25) "Variation does not exist."
["LongMessage":protected]=>
string(102) "The item contains no variation matching variation label or VariationSpecifics included in the request."
["ErrorCode":protected]=>
string(5) "10.89"
["UserDisplayHint":protected]=>
NULL
["SeverityCode":protected]=>
string(5) "Error"
["ErrorParameters":protected]=>
array(0) {
}
["ErrorClassification":protected]=>
string(12) "RequestError"
["any":protected]=>
NULL
}
}
["Build":protected]=>
string(28) "E1199_CORE_APILW_19146596_R1"
["Version":protected]=>
string(4) "1199"
["CorrelationID":protected]=>
NULL
}

what you use instead of shopping API to get listing details / pics gallery ect ?

hi, first of all, it looks like your authentification is now working, because you get a normal error that is related to your request, not to your way of authorization: "Variation does not exist.". so the error is about your request, please show it to me 😏
to get all item info, why don´t you use the "GetItem" call of the trading API ? 🤔


geant4: 10.6.1 -> 10.6.2

Motivation for this change
Updating to the newest version.  Added TENDL environment variable support.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build g4py


When I create and deb and try install it, it says no data found?? what am I doing wrong?

When I create and deb and try install it via sideloady, I get an error saying : failed to prepare to inject dylibs, no data found in deb

I resolved this by using the command: make package FINALPACKAGE=1 instead of: make package


MakeCode.com thumbnail has BETA in it.

Repro steps:

Make a tweet and link to makecode.com
See thumb nail in the tweet showing BETA

This still repros.

This has been fixed. You could see the preview of them all here https://www.opengraph.xyz/url/https%3A%2F%2Fwww.microsoft.com%2Fen-us%2Fmakecode%3Frtc%3D1


construct dataset

Hi Wei,
I ran the example prediction code and get error when using the 'affinity_pred_list = torch.cat(affinity_pred_list)', I don't know what caused error and how to fix it.
Thanks!

please use torchdrug v0.1.2