{tabulizer} got archived on CRAN on 2021-10-31

Are there short-term plans to bring it back? :)

:wave: @tpaskhalis, do you need any help?

The release of Java 17 caused some issues in the way R interacts with it through rJava. More specifically from Brian Ripley's email:
On the most common form of failure (all but jsr223, also seen in rJava),
Simon Urbanek writes

"There was a major change in JDK-17 that now disallows reflection to use
non-public access.  In short rJava can no longer pretend it is part of a
class when accessing methods/fields which allowed us to call methods
without explicitly subclassing a class.  This may have serious
repercussions, because code that could be before written in R now has to
be written in Java, compiled and then called from R."

It's not the best of timing for me, as it all occurred at the beginning of a new teaching semester. And also my Java is not that strong, so it will take me some time to figure this issue. But it's definitely fixable. If anybody wants to take a shot at pull request, I'd be more than happy.
@maelle, thanks, yes, I'd be happy with any help on this issue. But it's certainly on my to-do list, if nobody else attends to it. Seems like it still works with GH installation for some people.

@tpaskhalis Thanks for your answer! As it's not something I would personally be able to help with, I've added the "help wanted" label (we're working on making such issues more easily discoverable cc @steffilazerte), and have posted in rOpenSci slack package-maintenance channel.

So right now there is no way to use it, right?
As I am a novice user, I don't know a lot about programming to help, but a lot of encouragement!

@DavidPerea You can install it from GitHub, which should work: remotes::install_github("ropensci/tabulizer").

@DavidPerea You can install it from GitHub, which should work: remotes::install_github("ropensci/tabulizer").

Before installing "tabulizer", I see that I must first have 'tabulizerjars' installed. But I get the following error when using this command: remotes :: install_github ("ropensci / tabulizerjars")
Downloading GitHub repo ropensci/tabulizerjars@HEAD
Running `R CMD build`...
* checking for file 'C:\Users\david\AppData\Local\Temp\Rtmp8ee6dr\remotesf0472d62281\ropensci-tabulizerjars-d1924e0/DESCRIPTION' ... OK
* preparing 'tabulizerjars':
* checking DESCRIPTION meta-information ... OK
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
* building 'tabulizerjars_1.0.1.tar.gz'
* installing *source* package 'tabulizerjars' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
  converting help for package 'tabulizerjars'
    finding HTML links ... done
    tabulizerjars                           html
** building package indices
** testing if installed package can be loaded from temporary location
*** arch - i386
*** arch - x64
Error: package or namespace load failed for 'tabulizerjars':
 .onLoad failed in loadNamespace() for 'tabulizerjars', details:
  call: NULL
  error: .onLoad failed in loadNamespace() for 'rJava', details:
  call: fun(libname, pkgname)
  error: JAVA_HOME cannot be determined from the Registry
Error: loading failed
Ejecución interrumpida
ERROR: loading failed for 'x64'
* removing 'C:/Users/david/OneDrive/Documentos/R/win-library/R-4.1.2/library/tabulizerjars'
Warning message:
In i.p(...) :
  installation of package ‘C:/Users/david/AppData/Local/Temp/Rtmp8ee6dr/filef042d3e148b/tabulizerjars_1.0.1.tar.gz’ had non-zero exit status

How is it solved?

@DavidPerea make sure that you can install and load rJava before installing tabulizerjars.

@DavidPerea make sure that you can install and load rJava before installing tabulizerjars.

Solved, thank you very much!

Hi, I'm in same boat. Would like to use this package, but the remotes :: install_github ("ropensci / tabulizerjars") fails like this:
remotes::install_github("ropensci/tabulizer")
Downloading GitHub repo ropensci/tabulizer@HEAD
Skipping 1 packages not available: tabulizerjars
✓  checking for file ‘/private/var/folders/55/2g40bwx56rvg3bnchwhvqkdr0000gn/T/Rtmp3jLMsJ/remotes6afbece498f/ropensci-tabulizer-9f45f91/DESCRIPTION’ ...
─  preparing ‘tabulizer’:
✓  checking DESCRIPTION meta-information
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
Removed empty directory ‘tabulizer/docs’
─  building ‘tabulizer_0.2.2.tar.gz’
ERROR: dependency ‘tabulizerjars’ is not available for package ‘tabulizer’

removing ‘/Library/Frameworks/R.framework/Versions/4.0/Resources/library/tabulizer’
Warning message:
In i.p(...) :
installation of package ‘/var/folders/55/2g40bwx56rvg3bnchwhvqkdr0000gn/T//Rtmp3jLMsJ/file6afb93f52e1/tabulizer_0.2.2.tar.gz’ had non-zero exit status

rJava is installed and loaded - and I don't get any obvious errors from it.
This is a Mac. Happy to move to a PC/Ubuntu if that is an easier path.
Any clues? Really, really could use this package right now!
thanks,

I got this working.  It was simple.
You need this:
remotes::install_github(c("ropensci/tabulizerjars", "ropensci/tabulizer")

For macOS >= to Catalina, it won’t work b/c the relevant legacy Java runtime (SE 6) is not compatible, downgrade OS, switch OS systems or find an alternative application.

Hi!

@DavidPerea You can install it from GitHub, which should work: remotes::install_github("ropensci/tabulizer").

I tried this but I had this error
Downloading GitHub repo ropensci/tabulizer@HEAD Error: Failed to install 'tabulizer' from GitHub:   invalid octal digit
Help, please! i need tu use this package

@araceliramirezg This seems to be a separate issue of your local R installation. I suggest to open a new issue and provide more information on your local R installation. You might use the reprex package to provide this information.

@araceliramirezg This seems to be a separate issue of your local R installation. I suggest to open a new issue and provide more information on your local R installation. You might use the reprex package to provide this information.

@pat-s I tried it again and i have this message:
`remotes::install_github("ropensci/tabulizer")
Downloading GitHub repo ropensci/tabulizer@HEAD
Downloading GitHub repo ropensci/tabulizerjars@HEAD
WARNING: Rtools is required to build R packages, but is not currently installed.
Please download and install Rtools 4.0 from https://cran.r-project.org/bin/windows/Rtools/.
√  checking for file 'C:\Users\pc\AppData\Local\Temp\RtmpY7EkSu\remotes9e414fe2c0a\ropensci-tabulizerjars-d1924e0/DESCRIPTION' (773ms)

preparing 'tabulizerjars':
√  checking DESCRIPTION meta-information ...
checking for LF line-endings in source and make files and shell scripts
checking for empty or unneeded directories
building 'tabulizerjars_1.0.1.tar.gz'

Installing package into ‘C:/Users/pc/Documents/R/win-library/4.0’
(as ‘lib’ is unspecified)

installing source package 'tabulizerjars' ...
** using staged installation
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
converting help for package 'tabulizerjars'
finding HTML links ... done
tabulizerjars                           html
** building package indices
** testing if installed package can be loaded from temporary location
*** arch - i386
*** arch - x64
Error: package or namespace load failed for 'tabulizerjars':
.onLoad failed in loadNamespace() for 'tabulizerjars', details:
call: NULL
error: .onLoad failed in loadNamespace() for 'rJava', details:
call: fun(libname, pkgname)
error: No CurrentVersion entry in Software/JavaSoft registry! Try re-installing Java and make sure R and Java have matching architectures.
Error: loading failed
Ejecución interrumpida
ERROR: loading failed for 'x64'
removing 'C:/Users/pc/Documents/R/win-library/4.0/tabulizerjars'
Downloading GitHub repo ropensci/tabulizerjars@HEAD
Skipping tabulizerjars, it is already being installed
WARNING: Rtools is required to build R packages, but is not currently installed.

Please download and install Rtools 4.0 from https://cran.r-project.org/bin/windows/Rtools/.
√  checking for file 'C:\Users\pc\AppData\Local\Temp\RtmpY7EkSu\remotes9e46a0f279d\ropensci-tabulizer-3049a79/DESCRIPTION' (822ms)

preparing 'tabulizer': (391ms)
√  checking DESCRIPTION meta-information ...
checking for LF line-endings in source and make files and shell scripts
checking for empty or unneeded directories
Removed empty directory 'tabulizer/docs'
building 'tabulizer_0.2.2.tar.gz'

Installing package into ‘C:/Users/pc/Documents/R/win-library/4.0’
(as ‘lib’ is unspecified)
ERROR: dependency 'tabulizerjars' is not available for package 'tabulizer'

removing 'C:/Users/pc/Documents/R/win-library/4.0/tabulizer'
Warning messages:
1: In i.p(...) :
installation of package ‘C:/Users/pc/AppData/Local/Temp/RtmpY7EkSu/file9e42a277c7b/tabulizerjars_1.0.1.tar.gz’ had non-zero exit status
2: In i.p(...) :
installation of package ‘C:/Users/pc/AppData/Local/Temp/RtmpY7EkSu/file9e4616d358b/tabulizer_0.2.2.tar.gz’ had non-zero exit status`

Help:(

Any update on accepting @AQLT pull request?

Maybe @leeper has time to help here and eventually merge #146?


Custom variable callback doesn't work correctly

I'm trying to use a similar variable definition as defined in the docs, but all it does is print the function as string into the template, which looks like this:
%svg-common {
	background-repeat: no-repeat;
	background-image: url(function (sprite, render) {
					return render(sprite).split('.svg').join('.png');
				});

	.svg & {
		background-image: url(../image/sprite.svg);
	}
}

My configuration looks like this:
{
	variables: {
		png: function() {
			return function(sprite, render) {
				return render(sprite).split('.svg').join('.png');
			}
		}
	}
}

My template is this:
{{#common}}.{{/common}}{{^common}}%{{/common}}{{mixin}} {
	background-repeat: no-repeat;
	background-image: url({{{png}}});

	.svg & {
		background-image: url({{{sprite}}});
	}
}

Haven't found anything why mustache shouldn't run the function and instead print it. Any idea?

You should definitely get familiar with the Mustache syntax! ;)
Please see here for an example of how to use function calls in Mustache. Using {{{whatever}}} (which is the unescaped version of {{whatever}}) does nothing else than printing a variable to the output file. To call a function, you have to use {{#whatever}}argumentForFunctionCall{{/whatever}}. In case the string passed to the callback contains Mustache lingo itself, you will have to explicitly render it inside your callback. This is what the render(sprite) in the example function is about.

Yeah, indeed - should get familiar with the mustache basics. Thank you for responding so quickly!


Hide PM2's Keymetrics ad, it's confusing people

PM2 prints out an ad for the paid Keymetrics service, which isn't useful to any of our users. It's confusing since PM2 is an implementation detail to them, and exp runs locally, not on a server that needs to be monitored anyway.
Creating a marker file looks like the best way to hide the banner: https://github.com/Unitech/pm2/blob/master/lib/CLI.js#L69.

I'm no longer seeing the ad on this fresh machine. Let me know if I just missed it somehow though.


ValueError: the greenlet library is required to use this function. No module named 'greenlet'

Runs
pipx install elia-chat
elia
Gets
ValueError: the greenlet library is required to use this function. No module named 'greenlet'

Thanks - seems like this was an undocumented requirement of SQLModel for using async on M1 Macs.
Should hopefully be fixed in 1.0.1. Could you try pipx upgrade elia-chat to install the latest version, and let me know if it's working for you now?

Tested on my M1 Macbook at work and it seems to fix it. Feel free to re-open if needed!

It works, thank you!


chore: bump examples to use latest configuration changes

Ran ./scripts/update-all-examples-dependencies.sh

Realized there's a workflow for this


split BasePlot.show() into show and savefig to match matplotlib style

@ivirshup Default for stacked violin and dotplot now looks  like this (changes: colormap=Blues for violins and border for both violins and dotplot, title of each plot shows which average measure is used):

Alternatively, the default for stacked_violins could be (not my preference as this takes more effort to interpret but similar to the stable branch output):

That looks great! Thanks!
I also prefer the shaded plot, since I don't think the categorical colors are adding a ton of information. If I wanted category colors I think I'd prefer a "barcode" like what sns.clustermap does, but that isn't necessary right now.
One very minor quibble. Would it be easy to make the outline of the dots to be the same color/ width in the legend and figure?


Add whitelist request with CDC for test.moveit.state.co.us

Requesting whitelist in staging of:
HOST: test.moveit.state.co.us
PORT: 22
for Colorado Department of Public Health and Environment

Ticket CHG0034787 created for the CSPO firewall folks


Doc/update

Please check if the PR fulfills these requirements

[x] The commit message follows our guidelines
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

What kind of change does this PR introduce? (Bug fix, feature, docs update, ...)
What is the current behavior? (You can also link to an open issue here)
What is the new behavior (if this is a feature change)?
Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
Other information:

Codecov Report
Base: 22.12% // Head: 22.12% // No change to project coverage :thumbsup:

Coverage data is based on head (6d226e0) compared to base (8502fb4).
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff            @@
##           dev/1.0     #141   +/-   ##
========================================
  Coverage    22.12%   22.12%
========================================
  Files           11       11
  Lines          479      479
  Branches        70       70
========================================
  Hits           106      106
  Misses         368      368
  Partials         5        5

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Mark down - Position of text with header and line changes after collapsing the composer

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.58-0
Reproducible in staging?: y
Reproducible in production?: n
Issue found when executing PR: https://github.com/Expensify/App/pull/38152
Issue reported by: Applause - Internal Team
Action Performed:

Go to staging.new.expensify.com
Go to chat.
Type # header.
Hit Enter to create a new line.
Note the position of text in the composer.
Expand the composer.
Collapse the composer.

Expected Result:
The position of text in the composer after expanding and collapsing the composer should be the same as the position before expanding the composer.
Actual Result:
After expanding and collapsing the composer, the text in the composer becomes top-aligned.
Workaround:
n/a
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence
https://github.com/Expensify/App/assets/93399543/f792c8fb-90a0-4e3e-9596-1725b2f6581a
View all open jobs on GitHub

We think that this bug might be related to #vip-vsb

I don't believe this is a bug 😄  The added header is the first line of the text. It should remain so even when the composer is expanded.


pull request for e2e test at 12/29/2016 01:56:10

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Modules without a remote master branch trigger "[2020-12-31 14:43:37.337] [ERROR]  fatal: your current branch 'master' does not have any commits yet"

I found a bug in MagicMirror
Any module without a "master" branch will give the fatal error
[ERROR]  fatal: your current branch 'master' does not have any commits yet
Platform:
Raspberrypi 3b+
Node Version:
v10.23.0
MagicMirror Version:
2.13.0
Description:
Any module without a "master" branch will give the fatal error
[ERROR]  fatal: your current branch 'master' does not have any commits yet
Steps to Reproduce:

Install a module without the master branch (i.e. mine: https://github.com/0m4r/MMM-covid19, it has a "main" branch as per the default github config for new repositories)
start magic mirror
check the server logs

Expected Results:
No fatal error is shown. Main is used as a reference branch
Actual Results:
A fatal error is shown
Configuration:
Not relevant
Additional Notes:
All seems to work well even with this error, and I know I could rename my main branch to master, but I think it should not be the case.

This works for me on the develop branch, looking in simple-git changelog this was fixed with v2.23.0.
So the new release v2.14.0 of MagicMirror (coming tomorrow) should work with your module.

So we can say it was fixed with PR https://github.com/MichMich/MagicMirror/pull/2222

is this the same as the cause for #2383 ?

I have just switched to develop, run "npm i" and this error is gone, but now I do get another one:
[31.12.2020 15:51.58.377] [ERROR] (node:1763) UnhandledPromiseRejectionWarning: TypeError: Object.fromEntries is not a function
    at userOptions (/home/pi/MagicMirror/node_modules/simple-git/src/lib/tasks/log.js:32:19)
    at parseLogOptions (/home/pi/MagicMirror/node_modules/simple-git/src/lib/tasks/log.js:62:31)
    at Git.log (/home/pi/MagicMirror/node_modules/simple-git/src/git.js:871:26)
    at sg.git.fetch.status (/home/pi/MagicMirror/modules/default/updatenotification/node_helper.js:78:13)
    at onSuccess (/home/pi/MagicMirror/node_modules/simple-git/src/lib/task-callback.js:8:9)
    at process._tickCallback (internal/process/next_tick.js:68:7)
[31.12.2020 15:51.58.378] [ERROR] (node:1763) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)

simple git is 2.26.0
pi@raspberrypi3:~/MagicMirror $ npm ls simple-git
<EMAIL_ADDRESS>/home/pi/MagicMirror
└──<EMAIL_ADDRESS>

@0m4r yeh, reported in #2383
I can't reproduce it
i tested with<EMAIL_ADDRESS>and didn't see it, but can't cause it again on 2.26
if u can repeat, install<EMAIL_ADDRESS>and try again

as per #2383, upgrading simple-git to 2.31.0 solves the issue.
As far as I am concerned this one can be closed. thanks for the quick support!
Let me know if I can be of any help!

at the moment simple-git v2.26.0 is in package.json. So should this be updatet to v2.31.0 before the release tomorrow?

I guess that would be a good idea. But I do not know what it takes in terms of effort to update it.
I think you may want/need to go through all your QA process and stuff??

just submitted pr

the problem is gone with the new release.
I close the issue.

the problem is gone with the new release.
I close the issue.


Appstudio update mvp-test-component

Pipelines as Code configuration proposal

Pipelines as Code CI/mvp-test-component-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
12 seconds
init

✅ Succeeded
27 seconds
clone-repository

✅ Succeeded
2 minutes
build-container

✅ Succeeded
17 seconds
inspect-image

✅ Succeeded
13 seconds
deprecated-base-image-check

✅ Succeeded
16 seconds
clair-scan

✅ Succeeded
59 seconds
clamav-scan

✅ Succeeded
14 seconds
sbom-json-check

✅ Succeeded
16 seconds
optional-label-check

✅ Succeeded
16 seconds
label-check

✅ Succeeded
6 seconds
show-summary


Replace eval in pb generated file to avoid CSP error

Resolve #43

Thanks @losnappas ! Merged your PR.


Build failure: opentelemetry-collector-contrib

Steps To Reproduce
Steps to reproduce the behavior:

build opentelemetry-collector-contrib
view contents of result

Build log

Build Log
Running phase: unpackPhase
@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking source archive /nix/store/g3kmgm4z39khdfm18lr924xbkpgsh1g4-source
source root is source/cmd/otelcontribcol
Running phase: patchPhase
@nix { "action": "setPhase", "phase": "patchPhase" }
Running phase: updateAutotoolsGnuConfigScriptsPhase
@nix { "action": "setPhase", "phase": "updateAutotoolsGnuConfigScriptsPhase" }
Running phase: configurePhase
@nix { "action": "setPhase", "phase": "configurePhase" }
Running phase: buildPhase
@nix { "action": "setPhase", "phase": "buildPhase" }
Running phase: installPhase
@nix { "action": "setPhase", "phase": "installPhase" }
Running phase: fixupPhase
@nix { "action": "setPhase", "phase": "fixupPhase" }
shrinking RPATHs of ELF executables and libraries in /nix/store/vlqk2b12bak2ajjkc4kmamdqpqr8wil9-opentelemetry-collector-contrib-0.112.0
checking for references to /build/ in /nix/store/vlqk2b12bak2ajjkc4kmamdqpqr8wil9-opentelemetry-collector-contrib-0.112.0...
patching script interpreter paths in /nix/store/vlqk2b12bak2ajjkc4kmamdqpqr8wil9-opentelemetry-collector-contrib-0.112.0

Additional context
While the package "builds", it produces no output. I bisected this and its last package bump caused this: c65eb72d9587a416ac083d76f1d862217cb912f2
Metadata

Notify maintainers

@06kellyjac @uri-canva

Note for maintainers: Please tag this issue in your PR.

Add a :+1: reaction to issues you find important.

It's caused by https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/34844. Generated code is removed.

@domenkozar I saw your comment on the other PR about a broken MacOS build. Is the idea to close this issue and make a fresh one for the MacOS build failure or does it all work fine now?

Let's keep this open to track the issue.

It's not super clear to me why obc doesn't work on macOS

@sandydoo  has a fix

the current v0.112 in nixos 24.11 is broken.
Could it be possible to update the latest version to nixos 24.11 ?
Thanks


Tests failing silently

Which packages are impacted by your issue?
@graphprotocol/graph-cli
Describe the issue
Under macOS Sonoma, tests can fail silently (no error is printed). This seems to be an issue with matchstick. See:

https://github.com/LimeChain/matchstick-as/issues/66
https://github.com/LimeChain/matchstick/issues/421

Reproduction
https://github.com/OlympusDAO/olympus-protocol-metrics-subgraph/blob/bea228c830478218ca0c291ca941fb5a9d250111/subgraphs/ethereum/tests/OhmCalculations.test.ts#L250-L266
Steps to Reproduce the Bug or Issue

Run yarn subgraph test ethereum
The "closed auction/after bond expiry/all burned" test case in OhmCalculations should fail. No error is printed, even the name of the test case doesn't show up. (Search for the previous test case, "closed auction/after bond expiry", and it will show up.)

Expected behavior
The "closed auction/after bond expiry/all burned" test case should have failed and thrown an error.
Screenshots or Videos
No response
Platform

OS: macOS Sonoma 14.3
NodeJS: v20.10.0
<EMAIL_ADDRESS>
Subgraph Manifest
No response
Subgraph GraphQL Schema
No response
Additional context
No response

Is it really not a priority to fix this??

Not at the moment. However, if you'd like to contribute a patch, we'd be more than happy to accept it!

Facing the same issue and just started using matchstick. Very frustrating. Makes me not want to use it


Fix svelte package name

Fix svelte package name in documentation
Description
angular-svelte -> agnostic-svelte
Checklist:

[x] Have you updated the docs? These live in site/docs

It wasn't difficult :)


dCnwuq3t: Add special text for UC users on IDP disconnection

See https://trello.com/c/dCnwuq3t/112-use-cookie-to-remind-sign-in-users-that-those-rm-cs-will-no-longer-work
Co-authored-by: John Watts<EMAIL_ADDRESS>Co-authored-by: Alex Newton<EMAIL_ADDRESS>

This is good to merge and release once hub and hub-fed-conf releases have happened (@phantas)


Add a README.md to the template

WHY are these changes introduced?
Currently, the template for a Checkout UI extension doesn't contain a basic README.md file that helps guiding developers into what the project contains and how can they work with it. This was reflected in the following Issue:
Fixes https://github.com/Shopify/shopify-cli-extensions/issues/300
The Checkout UI GA Launch will take place next 3rd October, which will make the project template to become more exposed. Therefore, providing a README before the date will improve the developer experience when it becomes more used.
WHAT is this pull request doing?
This PR adds a README.md file to the template so that every generated extension in an app contains this guide.
How to test your changes?

In this branch, generate a scaffold for Checkout-UI extension
If the extension created contains the README, it's done

Measuring impact
How do we know this change was effective? Please choose one:

[X] n/a - this doesn't need measurement, e.g. a linting rule or a bug-fix
[ ] Existing analytics will cater for this addition
[ ] PR includes analytics changes to measure impact

Checklist

[ X] I've considered possible cross-platform impacts (Mac, Linux, Windows)
[X] I've considered possible documentation changes
[X] I've made sure that any changes to dev or deploy have been reflected in the internal flowchart.

I'm still working on tophatting this, but feel free to review the README

LGTM, clear and informative. It would be good to get a review from a technical writer + someone with more context than me!

@lemonmade Thanks for the feedback, I appreciate a final check before merging


Inconsistency between pretrain.sh and finetune script

In the pretrain script, the mm_projector_type is unset and is defaulted into "linear". However, in the finetune script this value is set to "mlp2x_gelu", which causes a conflict in loading the corresponding weights. Which configuration is used for the actual model?

Thank you for your attention. For the chat model, we used “mlp2x_gelu”, and for fine-tuning on downstream tasks, we used a simple linear projector.

Sorry, I didn't quite understand your reply. Shouldn't the chat model have the same multi-modal MLP projector for pretraining and finetuning? How can they be different according to different tasks?

也可以用中文说，麻烦作者了，就是那个mmprojector我发现pretrain的config用的是一个线性层，但是finetune里面的mmprojector用的是两个线性层，这就导致了之前pretrain训练的权重无法被正常加载到finetune的script里面，会报错

可能语言习惯上有歧义导致有误解。我这里的finetune指的是在sqa等下游任务上直接进行微调的意思，对应于scripts中的sqa，而chat模型指的就是sft后得到的模型，对应于scripts中的sft。
对于前者我们简单的使用一层线性层作为projector，对于后者，我们跟随llava-1.5的范式使用的是mlp2x_gelu。如果你是想做sft，应该要把pretrain.sh中的projector改为对应的mlp2x_gelu类型。

好的，谢谢作者


clean knowledge integration project

Codecov Report

Merging #382 into master will decrease coverage by 1.65%.
The diff coverage is 6%.

@@            Coverage Diff             @@
##           master     #382      +/-   ##
==========================================
- Coverage   49.34%   47.69%   -1.66%
==========================================
  Files          91       91
  Lines        5087     5261     +174
==========================================
- Hits         2510     2509       -1
- Misses       2577     2752     +175

Impacted Files
Coverage Δ

jack/torch_util/rnn.py
0% <0%> (ø)
:arrow_up:

jack/torch_util/misc.py
0% <0%> (ø)
:arrow_up:

jack/torch_util/embedding.py
0% <0%> (ø)
:arrow_up:

jack/io/load.py
58.13% <10%> (-41.87%)
:arrow_down:

jack/io/SNLI2jtr.py
34.09% <100%> (-6.73%)
:arrow_down:

jack/core/model_module.py
75.75% <100%> (ø)
:arrow_up:

jack/core/data_structures.py
100% <100%> (ø)
:arrow_up:

jack/util/preprocessing.py
65.54% <20%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 46ac384...9c7ceee. Read the comment docs.


Will not render animation

Describe the bug
States that I must click "Render animation" button to render animation when rendering animation.
To reproduce
Create basic animation, set prompts, render animation.
Error log
AI Render Error: To render an animation with AI Render, use the "Render Animation" button in the Animation panel below
Environment

Blender version (upper right corner of splash screen): 4.1
AI Render version (find in Preferences > Add-ons): 0.9.1
Operating system (Windows/Mac/Linux): Windows 10

Screenshots/video
https://github.com/benrugg/AI-Render/assets/128502425/d234c6ca-9920-432c-9f2c-a3fa0e6f8768
Additional information
No response

Yes, unfortunately Blender gives add-on developers very limited access in a lot of different areas. I can't hook into the standard animation rendering pipeline and I had to create my own. So instead of doing the standard "render animation" button in the Blender menu, you need to click the button in the AI Render panel, under the "Animation" heading.
I just realized that the wording of my error message doesn't make that very clear, though 🤦♂

Just in case I wasn't clear enough - find the "Render Animation" button in the AI Render panels!

Ah, I tried looking there but saw nothing because it was greyed out. Maybe set the default path to C:\tmp?

Ah yes, just enter a path first. That's probably another thing I could improve in the UI!


Where to find Scatter-specific error code

Now I just meet two Scatter-specific error: 423 and 402. I want to know whether there are other error codes.
Maybe I miss something, I try my best but can not find any error information in document.
{
	type: "locked",
	message: "The user's Scatter is locked. They have been notified and should unlock before continuing.",
	code: 423,
	isError: true
}

{
	type: "identity_rejected",
	message: "User rejected the provision of an Identity",
	code: 402,
	isError: true
}

The first one is an old extension error just meaning that the Scatter Extension is locked and needs to be opened.
The second one might be related to the first one, or just a rejection of the Login/GetIdentity API

FYI the identity_rejected error is created here: https://github.com/GetScatter/ScatterDesktop/blob/ef2248e1e6ac75fa653d82968474bfaf39456ac0/src/services/ApiService.js#L152


OpenVPN interfaces doesn't show ip addresses

In the Interfaces page all the interfaces tied to tun/tap devices doesn't show any ip address. This was not the case with CC 15.05.

As you can see in the first image there is nothing, instead in the second there are ip (the second is CC 15.05).
 I created my own netifd helpers so that its possible to ifup/ifdown openvpn interface. I find this approach much more compliant and comfortable.

@bol-van Have you got them published (and what license?) anywhere?  It would be good to get openvpn into the modern age from the point of view of openwrt.

I can share my work but I dont pretend it to be the best quality and decisions made was perfect. I done it for myself.

@bol-van Are you still willing to share your netifd helpers for openvpn? The lack of netifd integration is causing additional issues (openwrt/openwrt#220, openwrt/packages#3486).

No problem. I'll share my work. But I will not take part in making it clean and compliant, i've finished with programming and do it only for myself and dont share my code because it can be dirty and does not meet clean standards. Take it as is and do whatever you want.
https://www.sendspace.com/file/9vmqh4
My code relies on standard uci openvpn config in /etc/config/openvpn  but requires to make section "enabled 0" to stop it from launch by /etc/init.d/openvpn.
This approach leaves luci-app-openvpn working
Add proto section to /etc/config/network as in example

not a LuCI issue at all.


Improve clarity in docs around use of no_coverage and toolchain on Windows

Hey there, thanks so much for your incredible work on this tool.
Just noticed two things as a new user of this tool on Windows:

I couldn't clearly see in the README that the tool is only supported on MSVC on Windows, I couldn't get it working on the GNU toolchain
I understand that no_coverage is an unstable feature, but perhaps it should be stated that it will only work on nightly Rust and be ignored on the stable channel

And as two follow-ups to these points:

Is it possible to get this tool working on the GNU toolchain for Windows?
Is there a way to specify no_coverage on stable Rust?  The tool itself works perfectly on stable Rust but the no_coverage attribute is simply ignored and not taken into account

Thanks heaps
Fotis

Is it possible to get this tool working on the GNU toolchain for Windows?

It might work with the proper linker and linker args (e.g., rust-lang/rust#79555 (comment)), but I have not tested it.

Is there a way to specify no_coverage on stable Rust?  The tool itself works perfectly on stable Rust but the no_coverage attribute is simply ignored and not taken into account

The reason it is ignored on stable is that the code generated by coverage-helper uses the cfg_attr(coverage_nightly, ..) attribute and cargo-llvm-cov sets coverage_nightly cfg only on nightly. However, you can manually set it.
RUSTC_BOOTSTRAP=1 RUSTFLAGS='--cfg coverage_nightly' cargo llvm-cov

RUSTC_BOOTSTRAP is something like a hack that disables stability checks, so we do not officially recommend using it.

Awesome stuff, thank you so much for the reply! 😄

(upstream issue about bug with windows-gnu toolchain: https://github.com/rust-lang/rust/issues/111098)


Fix unused setter argument warning

Fixes:
Sources/HotKey/KeyCombo.swift:26:22: warning: setter argument 'newValue' was never used, but the property was accessed
                        carbonModifiers = modifiers.carbonFlags
                                          ^
Sources/HotKey/KeyCombo.swift:26:22: note: did you mean to use 'newValue' instead of accessing the property's current value?
                        carbonModifiers = modifiers.carbonFlags
                                          ^~~~~~~~~
                                          newValue

I'm seeing the same warning. Would love to see this to make it in.

I was about to create a PR and found yours 😁.


UTF-8 Encoding for AppInfo.xml in export

UTF-8 Encoding
When the AppInfo.xml file is encoded in UTF-16 and you include the package in another install package. It fails because it tries to change to unicode. When it's UTF-8 it doesn't fail.

Should be fixed in version <IP_ADDRESS> or install package 1.0.1-CU10. See releases.


[Docs] Update Explanations: manipulator explanation

In this PR I try to update the manipulator explanation, under explanations.
I document

the change in coordinate systems: manipulators use world coordinates now
the added radius and handle_size properties to change the size in world coord
try to update the implementation description

@kevinyamauchi @alisterburt
Pretty unsure of this one, but thought it would be useful to open a PR so y'all can make suggestions or provide input to get the facts straight.
❤

thanks @psobolewskiPhD ! any further changes can happen in new PRs of course :)


Tp2000 762  virus checker

TP2000-762 Loading Report Virus Checker
Why
For file uploads we need to check the files for viruses before uploading to s3.
This PR uses the chunk handler library to call our Clam AV shared service to check files.
What
adding a validator to the loading report file field
Checklist

Requires migrations? yes
Requires dependency updates? yes

Codecov Report
Patch coverage: 93.92% and project coverage change: +0.08 :tada:

Comparison is base (d7b02e1) 92.32% compared to head (fbd4dfa) 92.40%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #888      +/-   ##
==========================================
+ Coverage   92.32%   92.40%   +0.08%
==========================================
  Files         413      413
  Lines       27807    28098     +291
  Branches     2092     2117      +25
==========================================
+ Hits        25672    25965     +293
+ Misses       1736     1735       -1
+ Partials      399      398       -1

Impacted Files
Coverage Δ

measures/models.py
98.17% <ø> (ø)

quotas/forms.py
100.00% <ø> (ø)

workbaskets/views/ui.py
84.24% <0.00%> (-0.63%)
:arrow_down:

geo_areas/validators.py
70.00% <50.00%> (-30.00%)
:arrow_down:

common/views.py
83.64% <72.72%> (-0.60%)
:arrow_down:

certificates/tests/test_views.py
97.36% <80.00%> (-2.64%)
:arrow_down:

footnotes/tests/test_views.py
96.82% <80.00%> (-3.18%)
:arrow_down:

publishing/tests/test_tasks.py
97.56% <80.00%> (-2.44%)
:arrow_down:

geo_areas/tests/test_views.py
98.59% <95.45%> (-1.41%)
:arrow_down:

geo_areas/forms.py
96.93% <96.15%> (+12.31%)
:arrow_up:

... and 17 more

... and 8 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


[FLINK-14355] [docs] Example code in state processor API docs doesn't compile

Fix compiling errors in examples of state processor api.

@sjwiesman @klion26 @KarmaGYZ  I've addressed your comments. Could you take a look again?


[Feature] widget - A must have

widget on the homescreenn to view a selected notes would complete the app @maltaisn

Duplicate of #33.


Added utils from broadinstitute/vital-ngs, added simple unit tests to verify Travis CI

Changes Unknown when pulling<PHONE_NUMBER>6e6b046808e788c880ba4046310556 on ct-cms into ** on master**.

Changes Unknown when pulling<PHONE_NUMBER>6e6b046808e788c880ba4046310556 on ct-cms into ** on master**.

Changes Unknown when pulling<PHONE_NUMBER>6e6b046808e788c880ba4046310556 on ct-cms into ** on master**.

The only downside to this commit is that coveralls will keep complaining as our coverage has no where to go but down!


fix #125 npm test failed

use this line to test
jest --coverage=false --colors FormSchemaInput -t 'should successfully emit the click event'

This project has been moved to https://gitlab.com/formschema/native


[To rel/0.12] [ISSUE-3786] Data file is deleted while file handle is not released

#3786
This problem is caused by the reference number of query reader is not decreased corrcetly, which leads to the reference number is not 0 and the object can not be closed.

Jira https://issues.apache.org/jira/browse/IOTDB-1574
Don't forget to update RELEASE_NOTES.md.


Versioning complies with PEP 440

And stops breaking pip

Sorry about this. Was trying to submit a PR to a local fork.


star: 2.6.0c -> 2.6.1a

Semi-automatic update generated by https://github.com/ryantm/nixpkgs-update tools. This update was made based on information from star.
meta.description for star is: '"Spliced Transcripts Alignment to a Reference"'.
Release on GitHub
Compare changes on GitHub

Checks done (click to expand)

built on NixOS
/nix/store/028micl7wrv4c8a7mk21r42g3iby8325-star-2.6.1a/bin/STAR passed the binary check.
/nix/store/028micl7wrv4c8a7mk21r42g3iby8325-star-2.6.1a/bin/STARlong passed the binary check.
2 of 2 passed binary check by having a zero exit code.
2 of 2 passed binary check by having the new version present in output.
found 2.6.1a with grep in /nix/store/028micl7wrv4c8a7mk21r42g3iby8325-star-2.6.1a
directory tree listing: https://gist.github.com/7da2491ba82e61071dc4053853240838
du listing: https://gist.github.com/b7eafe4f97d6f63745f97ca52ee5a312

Instructions to test this update (click to expand)

One-time optional setup to skip building using Cachix:

Install cachix from https://cachix.org/
Use r-ryantm's cache:cachix use r-ryantm

Test this update by entering a nix shell, seeing what is inside the
result, and if applicable, running some binaries:
nix-shell --pure -I nixpkgs=https://github.com/r-ryantm/nixpkgs/archive/d7a21fba103b2688224ca18ba6f1df71cfd99439.tar.gz -p star
ls -la /nix/store/028micl7wrv4c8a7mk21r42g3iby8325-star-2.6.1a

cc @arcadio for testing.

No attempt on x86_64-darwin (full log)
The following builds were skipped because they don't evaluate on x86_64-darwin: star
Partial log (click to expand)

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnsupportedSystem = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnsupportedSystem = true; }
to ~/.config/nixpkgs/config.nix.

Success on x86_64-linux (full log)
Attempted: star
Partial log (click to expand)
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/028micl7wrv4c8a7mk21r42g3iby8325-star-2.6.1a
shrinking /nix/store/028micl7wrv4c8a7mk21r42g3iby8325-star-2.6.1a/bin/STARlong
shrinking /nix/store/028micl7wrv4c8a7mk21r42g3iby8325-star-2.6.1a/bin/STAR
strip is /nix/store/h0lbngpv6ln56hjj59i6l77vxq25flbz-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/028micl7wrv4c8a7mk21r42g3iby8325-star-2.6.1a/bin
patching script interpreter paths in /nix/store/028micl7wrv4c8a7mk21r42g3iby8325-star-2.6.1a
checking for references to /build in /nix/store/028micl7wrv4c8a7mk21r42g3iby8325-star-2.6.1a...
/nix/store/028micl7wrv4c8a7mk21r42g3iby8325-star-2.6.1a

Success on aarch64-linux (full log)
Attempted: star
Partial log (click to expand)
installing
post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/cfb6b8cb7ipj6r5ww3jd1yjscn0xvq6i-star-2.6.1a
shrinking /nix/store/cfb6b8cb7ipj6r5ww3jd1yjscn0xvq6i-star-2.6.1a/bin/STARlong
shrinking /nix/store/cfb6b8cb7ipj6r5ww3jd1yjscn0xvq6i-star-2.6.1a/bin/STAR
strip is /nix/store/y4ymnvgxygpq05h03kyzbj572zmh6zla-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/cfb6b8cb7ipj6r5ww3jd1yjscn0xvq6i-star-2.6.1a/bin
patching script interpreter paths in /nix/store/cfb6b8cb7ipj6r5ww3jd1yjscn0xvq6i-star-2.6.1a
checking for references to /build in /nix/store/cfb6b8cb7ipj6r5ww3jd1yjscn0xvq6i-star-2.6.1a...
/nix/store/cfb6b8cb7ipj6r5ww3jd1yjscn0xvq6i-star-2.6.1a


Red Alert 2 Yuri's Revenge with Phobos, Renderer does not support icon changes.

When the CnC-DDraw renderer is used for Yuri's Revenge, the -icon change command line argument from Phobos does not change the gamemd instance icon to the new icon as specified. Other renderers such as TS-DDraw properly display the change of icon.
Mirror Issue on Phobos's GitHub

should work now

should work now

should work now

should work now


Go CLI argument parity with Python CLI

The following command arguments are not consistent across the Go and Python wsk CLI.

wsk action list - The Go CLI has a --full argument, the Python CLI does not
wsk package list - The Go CLI has --full and --shared s arguments, the Python CLI does not
wsk rule get - The Python CLI has a --summary argument, the Go CLI does not
wsk trigger get - The Python CLI has a --summary argument, the Go CLI does not
wsk package create - The Go CLI has a --service_guid argument, the Python CLI does not
wsk package update - The Go CLI has a --service_guid argument, the Python CLI does not

The following commands take an additional non-flag argument which acts as a property selector.  The Python CLI supports this argument, the Go CLI does not.

wsk action get ACTION_NAME [PROPERTY_NAME]
wsk activation get ACTION_NAME [PROPERTY_NAME]
wsk package get ACTION_NAME [PROPERTY_NAME]
wsk rule get ACTION_NAME [PROPERTY_NAME]
wsk trigger get ACTION_NAME [PROPERTY_NAME]

Package (and action summaries) are not working correctly in the go cli (another test gap):
openwhisk> wsk package update p -a parameters '[ {"name":"someP", "required":false}, {"name":"anotherP", "required":false} ]'
ok: updated package p
openwhisk> ./bin/go-cli/wsk -i --apihost <IP_ADDRESS> package get p --summary
package /guest/p
openwhisk> wsk package get p --summary
package /guest/p
   (params: someP anotherP)

For bookkeeping sake, the above mentioned --full, and ---serviceguid issues have been fixed.

wsk package create - The Go CLI has a --service_guid argument, the Python CLI does not
wsk package update - The Go CLI has a --service_guid argument, the Python CLI does not

These should not be in the go-cli, they are entirely Bluemix specific.

@dubeejw @mdeuser
What work is left here to implement?   Just the get a specific property from the entity?

The following commands take an additional non-flag argument which acts as a property selector.  The Python CLI supports this argument, the Go CLI does not.

wsk action get ACTION_NAME [PROPERTY_NAME]
wsk activation get ACTION_NAME [PROPERTY_NAME]
wsk package get ACTION_NAME [PROPERTY_NAME]
wsk rule get ACTION_NAME [PROPERTY_NAME]
wsk trigger get ACTION_NAME [PROPERTY_NAME]

I was going over the docs, and we document this feature for the CLI, in python is used to work.
Now with Go CLI, it doesn't I would consider this a regression from the customer point a view.
From the docs packages.md#creating-a-package
From the docs we document the following three commands that are broken

package get PACKAGE_NAME parameters

$ wsk package get custom parameters

action get ACTION_NAME parameters

$ wsk action get custom/identity parameters

package get PACKAGE_NAME publish

$ wsk package get custom publish

This needs to be high on our priority list
@dubeejw Can you comment on how difficult would be to fix this?

@csantanapr, it should not be too difficult after we get https://github.com/openwhisk/openwhisk/pull/1175 merged.

Linking to PR https://github.com/openwhisk/openwhisk/pull/1376 so we revert back the docs when this is fix in CLI


Refactor/GitHub provider file

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

126 of 140 (90.0%)  changed or added relevant lines in 4 files are covered.
54 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-5.6%) to 91.739%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/github-provider.ts
53
55
96.36%

src/types.ts
0
12
0.0%

Files with Coverage Reduction
New Missed Lines
%

src/types.ts
54
0%

Totals

Change from base Build<PHONE_NUMBER>:
-5.6%

Covered Lines:
1119

Relevant Lines:
1214

💛  - Coveralls


[Fun] Successfully managed to hose my Windows file system

I was being greedy about my time and the constant arch updates so I figured that I will have a "single" /var/cache/pacman/pkg for both my "Arch in WSL' and regular Arch.
I moved files from /var/cache/pacman/pkg to /mnt/F/SuperShare/pkg from inside regular Linux installation. Then I did a sudo pacman --sync --refresh --sysupgrade from WSL Linux installation. Now WSL fails to understand some of those files. And so does Powershell. By "understand" I mean ls fails from inside Powershell and WSL for those files. e.g. ls gnome* in WSL:arch gives: /bin/ls: cannot access 'gnome-keyring-1:3.20.0+16+gbf8aa97-1-x86_64.pkg.tar.xz': No such file or directory
I did do, in all my folly, a chmod 700 on files inside /mnt/F/Supershare/pkg despite knowing that ext4fs != ntfs.

Upon further inspection this is not to my credit, ntfs doesn't allow ':' in filename.


Update glam to 0.13.0.

See https://github.com/bitshifter/glam-rs/blob/master/CHANGELOG.md for details on changes.

Don't forget to cargo fmt.

Looks like there were more deprecations in this update:
error: use of deprecated associated function `bevy_math::Mat4::identity`: use Mat4::IDENTITY instead
   --> crates/bevy_core/src/bytes.rs:234:25
    |
234 |         test_round_trip(Mat4::identity());
    |                         ^^^^^^^^^^^^^^
    |
    = note: `-D deprecated` implied by `-D warnings`

what do you think about replacing VecN::default() with VecN::ZERO to benefit there also from the new consts?

@mockersf  I agree, I think VecN::ZERO would be a bit clearer intent wise than VecN::default(). Other than that, it might remove a function call in unoptimised builds but I think it's probably much of a muchness otherwise.

Please resolve warnings locally first?

@Ratysz I did build locally but I did not see these warnings. They appear to be from iOS examples, perhaps they aren't part of the standard build?

The one I linked runs cargo clippy --workspace --all-targets --all-features -- -D warnings -A clippy::type_complexity -A clippy::manual-strip on Linux, and the warning comes from an example.

Does that require nightly? failed to build for me at all on Rust stable 1.50 on Windows:
error: use of a blacklisted/placeholder name `foo`
   --> crates\bevy_tasks\src\task_pool.rs:283:13
    |
283 |         let foo = Box::new(42);
    |             ^^^
    |
    = note: `-D clippy::blacklisted-name` implied by `-D warnings`

etc.

The file that is failing now was added at some point after I made this PR, which is why I didn't see it. I need to rebase with main.

That's odd. I know this sounds silly, but it works on my machine 😄

It worked after rebasing.

Thanks for keeping us up to date! Looks good to me.

We will need to update hexasphere before merging to avoid duplicate deps: https://github.com/OptimisticPeach/hexasphere
@OptimisticPeach down to do an update?

Of course! I'll have it ready in an hour or so.

Half an hour, actually, hexasphere 3.2.0 is up.
Related: how would you feel if I were to rewrite/refactor hexasphere to be generic over the vector used in a separate crate (and then use that in bevy), so updating glam is seamless?

That sounds good to me!

bors r+


General speed optimization

This 'issue' is a place to collect thoughts about general speed and efficiency improvements.
There's a pretty minimal sampling java profiler at http://code.google.com/p/jsamp/wiki/Documentation that you can just attach to a running JVM on a compute node, leave it for a few hours, come back and kill it, and analyze the collected samples.
I sampled every 100ms for ~4 hours (5 enlargement iterations) on a job of mine with pdep, a large seed, and 32 reaction systems, but no pruning. Stats when I started sampling were:

The model core has 20524 reactions and 281 species.
The model edge has 314641 reactions and 43038 species.
There are 48093 partial pressure-dependent networks containing 50250 path and 13346 net reactions.
The chem.inp file has 281 species (excluding inert gases).
The chem.inp file has 20149 reactions (excluding duplicates).
Running time: 2248.988 min
Memory used: 1778.66 MB / 3585.67 MB (49.60%)

It spends roughly:

90% solving the reaction systems jing.rxnSys.JDASSL.solve
60% waiting for the unix proces (presumably mostly DASSL)  java.lang.UNIXProcess.waitForProcessExit
26% checking if reaction structures are equal jing.rxn.Structure.equals
22% in java.util.LinkedList.clone (possibly in the above Structure.equals call?)
only 5% enlarging the model jing.rxnSys.ReactionModelGenerator.enlargeReactionModel

First observation: solving all the reaction systems each time this way is incredibly wasteful - especially as 30 of my 32 reaction systems are valid and only 2 are being used to enlarge the model!

A follow-up analysis on Richard's work, that will systematically expanded to more features, such as Pdep, QMTP, restart,etc...
Introduction:
RMG-Java, at commit 8574cb8a6cbb5a250648de79921c552e2d3d4158 is used. The GUI plugin for Eclipse called “JVM Monitor” is used for the CPU Profiling. CPU Sampling technique is used.
Settings:
A simulation of JP10 (C10H16) pyrolysis for 1 reaction system (1 set of conditions) without QMTP, without P-Dep activated, with one small seed mechanism, without pruning, without reading restart. A Intel Core  2 Duo CPU P9600 @ 2.66Ghz is used for this purpose.
This is the state of the RMG generated model:
The model core has 172 reactions and 73 species.
The model edge has 43345 reactions and 4459 species.
The chem.inp file has 73 species (excluding inert gases).
The chem.inp file has 172 reactions (excluding duplicates).
Running time: 35.795 min
Memory used: 251.22 MB / 259.52 MB (96.80%)
Results
After 3.5 K seconds of sampling at 50ms of all the RMG core packages, the following order of total method invocation time is observed at ReactionModelGenerator.modelGeneration as base method:
•	78% : ReactionModelGenerator.enlargeReactionModel
•	3% : ReactionSystem.solveReactionSystem
•	13% : ReactionModelGenerator.writeEdgeReactions
•	5% : ReactionModelGenerator.writeEdgeSpecies
In ReactionModelGenerator.enlargeReactionModel the majority (69% ) of the time is spent in ReactionTemplate.reactTwoReactants. The majority in that method is spent to species comparison during the creation of a new species: ChemGraph.equals(): 21% (SpeciesDictionary.getSpecies() + 5% (SpeciesDictionary.putSpecies()). Another great portion is the initialization of a chemgraph (ChemGraph.init()) ( 20 %) in the ChemGraph.make() method. 13% abs. is consumed by the verification to check for a forbidden structure.
Conclusion:
for large molecules for which many possible reaction possibilities exist, writing the edge reactions/species already becomes an important bottleneck and consumes roughly 20% of the CPU.
ps: all % are absolute, compared to the total CPU time (3.5K sec)
Check the snapshot i took in Eclipse from the JVM monitor:
http://cloud.github.com/downloads/nickvandewiele/RMG-Java/CPU profiling.bmp

On second thoughts, the observations I made are significantly different from what @rwest saw during his trial. The main differences at Richard's run are the larger profiling time (> 4h compared to my < 1h), Pdep on and the large seed. I started with Leeds methane combustion. I am not sure if the profiling application influences these results, but that might be a culprit too.

Very interesting, thanks Nick. I had some ideas for potential opportunities for speedup, which I put on a new branch: https://github.com/GreenGroup/RMG-Java/commit/a75660378885f95a1a9ec55e2d0dc164b881db5a Haven't gotten around to testing them yet, but I'll update this issue if the potential changes are useful/safe.

Another possible difference: I had 32 reaction systems (this is partly why it spent a long time simulating them)


[Table] sticky header issue

With the last changes introduced by #274 there are some issues when table has sticky header and tableLayout "auto" as seen in the screen recording:
https://user-images.githubusercontent.com/2563389/203015636-d341384e-9774-4dcc-8c30-68e18fbe6496.mov

Hi @alessandrotilli, I'm going to work today on some minor changes. If you agree I'd like to solve it pragmatically: if tableLayout is set to auto then you have sticky set to false by default.
Allowing scroll in both directions is not so compatible with sticky headers.
What do you think? Can I proceed in this way?

Hi @luciob, i've just onpened a this PR with a possible way to fix this.
I know it uses some "magic number" trickery to adjust some sizes but it seems to work fine in different browsers.
If you think it's safer to enable scroll in one direction only it's ok for me!

:tada: This issue has been resolved in version 7.3.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


add tests which test smaller parts of the code

the two tests currently set up test essentially the full code, by loading dat files and processing profiles and loading them into xarrays. More useful tests could just test individual parts of the code, for example, listing all the dat files in a directory and testing that it finds the correct number.
This would be useful for testing because a failed test would be more specific to why something failed (although you do get a traceback).

#20 added a few more tests. We now have five and some of them do test smaller parts of the code.
Could definitely add more as we go along though.

I think we have quite a few new tests, which I think are doing a reasonable job. We need more for the velocity and strain rate calcs though. (#59)


Release 1.5 preparation

There are many minor changes to make to prepare release 1.5.
This task does not have concrete feature to implement. It is created to make notes for small changes to release areg-sdk 1.5.

release 1.5 prepared


java.lang.NumberFormatException: For input string: "20090811081019"

Checklist

[ ] I verified that the task ./gradlew :dependencyUpdates works correctly. If that's not the case, it's better to submit a bug-report to https://github.com/ben-manes/gradle-versions-plugin
there is no task dependencyUpdates ?
[x] I tried to reproduce the bug within sample-groovy, sample-kotlin or sample-versionsOnlyMode
[x] I have run with $ ./gradlew --scan buildSrcVersions or $./gradlew --scan refreshVersions to provide a Build Scan URL

Describe the bug
scan: https://scans.gradle.com/s/bvm2k6uha4tpi
when i add this dependency
implementation(group = "org.scala-lang", name = "scala-library", version = "_")
// same thing: implementation("org.scala-lang:scala-library:_")

the gradle refresh (or refreshVersions task) crashes with this log
FAILURE: Build failed with an exception.

* What went wrong:
For input string: "20090811081019"

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
com.intellij.openapi.externalSystem.model.ExternalSystemException: For input string: "20090811081019"
	at org.jetbrains.plugins.gradle.model.ProjectImportAction.addBuildModels(ProjectImportAction.java:272)
	at org.jetbrains.plugins.gradle.model.ProjectImportAction.execute(ProjectImportAction.java:130)
	at org.jetbrains.plugins.gradle.model.ProjectImportAction.execute(ProjectImportAction.java:56)
	at org.gradle.tooling.internal.consumer.connection.InternalBuildActionAdapter.execute(InternalBuildActionAdapter.java:79)
	at org.gradle.tooling.internal.provider.runner.ClientProvidedPhasedActionRunner$ActionRunningListener.runAction(ClientProvidedPhasedActionRunner.java:120)
	at org.gradle.tooling.internal.provider.runner.ClientProvidedPhasedActionRunner$ActionRunningListener.run(ClientProvidedPhasedActionRunner.java:110)
	at org.gradle.tooling.internal.provider.runner.ClientProvidedPhasedActionRunner$ActionRunningListener.buildFinished(ClientProvidedPhasedActionRunner.java:104)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:382)
	at org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:364)
	at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:58)
	at org.gradle.internal.event.DefaultListenerManager$EventBroadcast$ListenerDispatch.dispatch(DefaultListenerManager.java:352)
	at org.gradle.internal.event.DefaultListenerManager$EventBroadcast$ListenerDispatch.dispatch(DefaultListenerManager.java:339)
	at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:42)
	at org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:231)
	at org.gradle.internal.event.BroadcastDispatch$SingletonDispatch.dispatch(BroadcastDispatch.java:150)
	at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:58)
	at org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:325)
	at org.gradle.internal.event.BroadcastDispatch$CompositeDispatch.dispatch(BroadcastDispatch.java:235)
	at org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:141)
	at org.gradle.internal.event.ListenerBroadcast.dispatch(ListenerBroadcast.java:37)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy14.buildFinished(Unknown Source)
	at org.gradle.initialization.DefaultGradleLauncher.finishBuild(DefaultGradleLauncher.java:167)
	at org.gradle.initialization.DefaultGradleLauncher.finishBuild(DefaultGradleLauncher.java:113)
	at org.gradle.internal.invocation.GradleBuildController$3.create(GradleBuildController.java:86)
	at org.gradle.internal.invocation.GradleBuildController$3.create(GradleBuildController.java:78)
	at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:189)
	at org.gradle.internal.work.StopShieldingWorkerLeaseService.withLocks(StopShieldingWorkerLeaseService.java:40)
	at org.gradle.internal.invocation.GradleBuildController.doBuild(GradleBuildController.java:78)
	at org.gradle.internal.invocation.GradleBuildController.run(GradleBuildController.java:57)
	at org.gradle.tooling.internal.provider.runner.ClientProvidedPhasedActionRunner.run(ClientProvidedPhasedActionRunner.java:60)
	at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
	at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
	at org.gradle.launcher.exec.BuildOutcomeReportingBuildActionRunner.run(BuildOutcomeReportingBuildActionRunner.java:63)
	at org.gradle.tooling.internal.provider.ValidatingBuildActionRunner.run(ValidatingBuildActionRunner.java:32)
	at org.gradle.launcher.exec.BuildCompletionNotifyingBuildActionRunner.run(BuildCompletionNotifyingBuildActionRunner.java:39)
	at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner$3.call(RunAsBuildOperationBuildActionRunner.java:51)
	at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner$3.call(RunAsBuildOperationBuildActionRunner.java:45)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:416)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$CallableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:406)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
	at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:102)
	at org.gradle.internal.operations.DelegatingBuildOperationExecutor.call(DelegatingBuildOperationExecutor.java:36)
	at org.gradle.launcher.exec.RunAsBuildOperationBuildActionRunner.run(RunAsBuildOperationBuildActionRunner.java:45)
	at org.gradle.launcher.exec.InProcessBuildActionExecuter$1.transform(InProcessBuildActionExecuter.java:50)
	at org.gradle.launcher.exec.InProcessBuildActionExecuter$1.transform(InProcessBuildActionExecuter.java:47)
	at org.gradle.composite.internal.DefaultRootBuildState.run(DefaultRootBuildState.java:78)
	at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:47)
	at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:31)
	at org.gradle.launcher.exec.BuildTreeScopeBuildActionExecuter.execute(BuildTreeScopeBuildActionExecuter.java:42)
	at org.gradle.launcher.exec.BuildTreeScopeBuildActionExecuter.execute(BuildTreeScopeBuildActionExecuter.java:28)
	at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:78)
	at org.gradle.tooling.internal.provider.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:52)
	at org.gradle.tooling.internal.provider.SubscribableBuildActionExecuter.execute(SubscribableBuildActionExecuter.java:59)
	at org.gradle.tooling.internal.provider.SubscribableBuildActionExecuter.execute(SubscribableBuildActionExecuter.java:36)
	at org.gradle.tooling.internal.provider.SessionScopeBuildActionExecuter.execute(SessionScopeBuildActionExecuter.java:68)
	at org.gradle.tooling.internal.provider.SessionScopeBuildActionExecuter.execute(SessionScopeBuildActionExecuter.java:38)
	at org.gradle.tooling.internal.provider.GradleThreadBuildActionExecuter.execute(GradleThreadBuildActionExecuter.java:37)
	at org.gradle.tooling.internal.provider.GradleThreadBuildActionExecuter.execute(GradleThreadBuildActionExecuter.java:26)
	at org.gradle.tooling.internal.provider.ParallelismConfigurationBuildActionExecuter.execute(ParallelismConfigurationBuildActionExecuter.java:43)
	at org.gradle.tooling.internal.provider.ParallelismConfigurationBuildActionExecuter.execute(ParallelismConfigurationBuildActionExecuter.java:29)
	at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:60)
	at org.gradle.tooling.internal.provider.StartParamsValidatingActionExecuter.execute(StartParamsValidatingActionExecuter.java:32)
	at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:55)
	at org.gradle.tooling.internal.provider.SessionFailureReportingActionExecuter.execute(SessionFailureReportingActionExecuter.java:41)
	at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:48)
	at org.gradle.tooling.internal.provider.SetupLoggingActionExecuter.execute(SetupLoggingActionExecuter.java:32)
	at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:68)
	at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:39)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:27)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:35)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:78)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.create(ForwardClientInput.java:75)
	at org.gradle.util.Swapper.swap(Swapper.java:38)
	at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:75)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.LogAndCheckHealth.execute(LogAndCheckHealth.java:55)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:63)
	at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:82)
	at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:37)
	at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:104)
	at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:52)
	at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:297)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
	at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
java.lang.NumberFormatException: For input string: "20090811081019"
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
	at java.lang.Integer.parseInt(Integer.java:583)
	at java.lang.Integer.parseInt(Integer.java:615)
	at de.fayard.versions.extensions.VersionExtensionsKt.toComparableList-F2_lFBY(VersionExtensions.kt:116)
	at de.fayard.versions.extensions.VersionExtensionsKt.access$toComparableList-F2_lFBY(VersionExtensions.kt:1)
	at de.fayard.versions.extensions.VersionExtensionsKt$createVersionComparator$1.compare-3RWntRo(VersionExtensions.kt:57)
	at de.fayard.versions.extensions.VersionExtensionsKt$createVersionComparator$1.compare(VersionExtensions.kt:45)
	at java.util.Collections$ReverseComparator2.compare(Collections.java:5232)
	at java.util.TimSort.countRunAndMakeAscending(TimSort.java:356)
	at java.util.TimSort.sort(TimSort.java:234)
	at java.util.Arrays.sort(Arrays.java:1438)
	at kotlin.collections.ArraysKt___ArraysJvmKt.sortWith(_ArraysJvm.kt:1864)
	at kotlin.collections.CollectionsKt___CollectionsKt.sortedWith(_Collections.kt:949)
	at de.fayard.versions.internal.LatestVersionsRetrievalKt$getDependencyVersionsCandidates$2.invokeSuspend(LatestVersionsRetrieval.kt:35)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.UndispatchedCoroutine.afterResume(Builders.common.kt:207)
	at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:113)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:561)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:727)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:667)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:655)

* Get more help at https://help.gradle.org

BUILD FAILED in 8s

To Reproduce
Build scan URL: https://scans.gradle.com/s/xxxxxxxxxx

buildscript to repoduce build.gradle.kts:
plugins {
    id("scala")
    id("de.fayard.dependencies")
}

repositories {
    jcenter()
    mavenCentral()
}

dependencies {
    implementation(group = "org.scala-lang", name = "scala-library", version = "_")
}

Expected behavior

no crash ?
I thik the scala plugin adds some repositories (ivy?) and refreshVersions seems to not support those

The versions sorter will eventually get removed as part of #153, which will fix this issue.
When it comes to ivy repositories, refreshVersions doesn't support them for now.
If you need it, please open a separate issue, and I'll be grateful if you also link some info as to where to get the versions lists ordered as maven-metadata.xml files do.

looking at this more i don't think ivy is actually involved...
the scala artifact is on mavencentral here: https://repo.maven.apache.org/maven2/org/scala-lang/scala-library/maven-metadata.xml
it seems like the parsing freaks out on this tag: <version>2.8.0.r18462-b20090811081019</version>
which is reasonable because the number is too big
this line is responsible: https://github.com/jmfayard/refreshVersions/blob/develop/plugins/core/src/main/kotlin/de/fayard/versions/extensions/VersionExtensions.kt#L116
how should this be handled ?
it could be replaced with toLong() to handle larger numbers
purely alphanumerical comparison between the different components should work as well right ?
as long as they are split on ., -, +, etc
so #176 already found this (i suck at searching for duplicate issues)
and #153 will fix it
the workaround of hardcoding a single thing is not too bad

It should be using BigInteger I believe rather than long, or we're just moving the problem some steps away.

@NikkyAI Please, use permalinks next time (press the "y" key before copying from the address bar), because as I moved code, it's now pointing to 404.
Regardless, I think I see what can cause the failure, and this version parsing should go away as part of #153.
I'll try to finally work on it this weekend.

Do you know what the "20090811081019" number is supposed to represent?

@LouisCAD my educated guess would be that it represents the build date: august 11th, 2009, at 08:10:19.
A similarly mistaken version is available for commons-io.

I see. It fits in a Long, just like a timestamp would. I think if this code keeps on being used, I'll just support Long in place of Int, and not use BigInteger for the sake of performance. Plus reading the longest long is already insane, I don't see how it would land in the version name of a legit library.

@LouisCAD not using BigInteger "for the sake of performance" in a Gradle plugin, without any profiling in place indicating a possible problem (there will be none) is a gargantuan mistake that will eventually backfire as soon as someone includes a longer date format.
It's not an intensive computation, I warmly recommend to favor logic and design over performance, as there is not even and indication that it might be an actual issue. Premature optimization is the root of all evil.

Premature optimization is the root of all evil.

@DanySK I don't like this speech, because it's often an excuse to not care about efficiency at all, and that is the root of other evils, that can end up being worse.
Anyway, I don't know yet if this code will still be used, but I know that in the current form, this can be a hot path, hence my will to make it light. Now it's very possible that the difference would be insignificant. I still think it's very unlikely that one would invent a date format made of digits only that would span over 19 digits or more (Long.MAX_VALUE is 19 digits in decimal), and put it in a version number.
I'll take my decision after work on #153 is started.

@LouisCAD my point is: what do you gain in using Long rather than BigInteger? An insignificant performance/memory gain (the second more important than the former despite overall irrelevance, probably). Plus, I guess there's networking involved in version lookup, which I guess easily cannibalize the execution time.
On the other hand, what do you lose? The ability to support arbitrary-length versions.
I believe I got a decent experience in performance-critical JVM software (and related profiling/sampling/targeted optimization). I warmly recommend to begin by capturing the abstraction you actually need (in this case, an arbitrarily long sequence of digits), and only if you see that the whole abstraction is unused in practice and it is an actual issue for performance, then optimizing. Abstraction-first is, in my opinion, a good engineering practice.
I agree that it's rare to see a number larger than Long.MAX_VALUE, but both mistakes and silly versioning schemes exist (e.g., every TeX version adds a digit, and the version number converges to π). The thing is that you can't control what others do upstream, the only thing under your control is if and how to fail.
Ultimately, you are the maintainer of the software, so it's up to you to weigh the performance gain of Long over BigInteger vs. the possibility that someone comes up (by will or mistake) with a version number longer than 19 digits.
Cheers.

Thanks for your thoughts, and sharing that fact regarding TeX versioning scheme @DanySK.
I agree with your abstraction-first principles, although you always need to beware to not over-abstract, or wrongly-abstract (which are things as well), it makes sense in this context.
If I end up using Long, I'll make sure to catch that NumberFormatException and provide an error message that can be understanded. But I'll consider BigInteger, it might be a good fit after all.

You are welcome, thanks for your efforts


Problem: Can't see the status of the build on Github

You have to download the project and then run the tests yourself. This means we have no way of checking the status of the build other than manually.
Solution: Integrate with a test system that will test the code on every commit to the repo.
SuccessCriteria: There is a way to view the status of the build on the repo homepage.

Travis CI might be a nice solution to this
https://travis-ci.org/

Travis CI may have issues with the NodeJS integration (for parser generation). It seems to allow only testing of PHP code. This suggests that we split our project into two, the parser generator as a NodeJS service, which the PHP codebase hooks into via an interface. Then in the PHP tests, it should use a mock for the interface.
I will create both these issues and reference this issue.

Merged fine, no longer getting composer issues, instead we're seeing PHPunit tests failing. Issue seems to be related to this.
http://stackoverflow.com/questions/35062953/i-keep-getting-this-error-when-i-try-input-phpunit
Create new issues for these bugs, will link them to this issue.

Fix for issue #33

We have a working build. Closing issue.


[Feature/Extensions] Principal identity for Opensearch requests to extensions

Description
Adds a basic identifiers to determine owner for each request from OpenSearch to extensions.
Issues Resolved

Resolves https://github.com/opensearch-project/opensearch-sdk-java/issues/37

Check List

[x] New functionality includes testing.

[x] All tests pass

[x] New functionality has been documented.

[x] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff
[ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Hey @DarshitChanpura thanks for this draft PR.   Some quick feedback that should hopefully help you iterate:

Don't use the RegisterRestActionsRequest.  That's inbound to OpenSearch from an extension that just registers the action on the RestController.
Take a look at #4282.  In the RestActionsRequestHandler class we are registering the RestController to respond to a User's REST request by forwarding it to the extension.  This forwarding process is where I think you'll need to insert identity information

The RestSendToExtensionAction directly handles the RestRequest. You may have to back up and see where that Request comes from and how you can attach a principal to it.  Then we could send it along to the extension with the method and URI, just add a third String parameter which is a key to an internal map where the user is the principal.  (Eventually we'll actually want to send a JWT containing an Access Token.)

Continuing on the above comment, the RestRequest does include the headers which would have the user's authorization info in it.  So I'd suggest using that class as a basis (it's not merged yet but you can copy it and start with it for now) and:

parse the RestRequest headers to get the user info (also see #4247 where it looks like @peternied has done this for basic auth) and turn that into a Principal
store the Principal in a map with some unique key.  For now I think a random UUID would work.
send the key as a String along with the Method and URI in the RestExecuteOnExtensionRequest class.

Later the extension will send this key back with a different request or transport, and you can look up the Principal with it.

As far as storing in the map, if you go with Apache Shiro, it looks like the RememberMeManager is probably built for this purpose.  Not sure if the token it produces is an "access token" but worth looking into.

Finally comment for today: where to put the map?  I'd suggest a brand new object somewhere.  Look in Node.java where a lot of individual objects are instantiated, such as the RestController and ExtensionsOrchestrator and others.  Find a nice home in org.opensearch.identity and build a new object to your liking and instantiate it in Node and pass it around where it's needed.

I believe it should be "Username" and "Metadata" vs. "UserName" and "MetaData".
https://english.stackexchange.com/questions/43436/username-user-name-or-user-name

will re-base this branch once feature/extensions branch gets rebased after this PR gets merged.

I did some brainstorming today with @dbwiddis, attempting to capture that:
What do these identifiers need to do?

They should be consistent over time (years).
The identifier should be usable to lookup the principal of an identity.
If an extension compares two identifiers together, they should be equivalent to one another.
They should not provide the extension with any additional information.

Potential solutions:
Use the principal as the identifier
If we control the Idp's, such as the in memory user store, we can made the identifiers effectively random UUID, but we are proxying identity such as through ActiveDirectory, SAML, or OpenId we can be divulging information that the extension shouldn't have - a first/last name or an email address. If we 'wrap' all Idp's, the solution is the same as below.
Registration of a random UUID
This requires building a system to vendor and store UUID <-> principal mappings. Depending on the size of the cluster or the size of the principals this can create scaling limits, support for one million identifiers and/or synchronizing UUID across 1000 node cluster.
Obfuscate the identifier (current)
Using a transparent encoding process, in this case base64, its really easy to break and we are expecting extension writers not to inspect our open source code to figure out what we are doing. Seems like we would be providing a false sense of security.
Encrypt the principal (proposed)
If we used a cryptographic processed, with a key from the OpenSearch service + a key per extension, we can create decryptable principals that are not useful if leaked. While storage and management of the key inputs are needed, they scale based on the number of extensions.
I know there has been much considerable in this space, but I think this last option will provide the integrity and trade a small amount of computation that prevents scaling overhead. What do you think of this last proposal?
Note; I think we could naïvely pass the principal through a principal identifier and add the cryptographic process in a follow up PR.

I completely agree with the proposed solution of sending encrypted information to extensions. However for this initial PR are we to pass the name string as is or something other than this should be passed?

I completely agree with the proposed solution of sending encrypted information to extensions. However for this initial PR are we to pass the name string as is or something other than this should be passed?

I like the proposal. @DarshitChanpura lets make it iterative (a.k.a be scrappy). We can start with a String for now and lets get the initial change across OpenSearch <-> Extension working and then we can start working on encryption.

I was able to get by the error by adding PrincipalIdentifierToken to the registry in NetworkModule. https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/common/network/NetworkModule.java#L128

Pre-commit failed with  unable to resolve netty dependencies. Unable to reproduce it locally.

Build failed with this:
FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':distribution:bwc:staged:buildBwcLinuxTar'.
> Building 2.3.0 didn't generate expected file /var/jenkins/workspace/gradle-check/search/distribution/bwc/staged/build/bwc/checkout-2.3/distribution/archives/linux-tar/build/distributions/opensearch-min-2.3.0-SNAPSHOT-linux-x64.tar.gz

Probably need to re-trigger the check

Codecov Report

Merging #4299 (803c044) into feature/extensions (edde6a4) will increase coverage by 0.14%.
The diff coverage is 73.46%.

@@                   Coverage Diff                    @@
##             feature/extensions    #4299      +/-   ##
========================================================
+ Coverage                 70.57%   70.72%   +0.14%
- Complexity                57370    57527     +157
========================================================
  Files                      4642     4644       +2
  Lines                    276302   276347      +45
  Branches                  40389    40394       +5
========================================================
+ Hits                     195012   195446     +434
+ Misses                    64936    64557     -379
+ Partials                  16354    16344      -10

Impacted Files
Coverage Δ

...rch/extensions/rest/RestSendToExtensionAction.java
27.63% <25.00%> (-0.15%)
:arrow_down:

...extensions/rest/RestExecuteOnExtensionRequest.java
63.33% <45.45%> (-1.89%)
:arrow_down:

.../opensearch/identity/PrincipalIdentifierToken.java
86.66% <86.66%> (ø)

...g/opensearch/identity/ExtensionTokenProcessor.java
88.88% <88.88%> (ø)

...a/org/opensearch/common/network/NetworkModule.java
92.10% <100.00%> (+0.10%)
:arrow_up:

...cluster/coordination/PendingClusterStateStats.java
20.00% <0.00%> (-48.00%)
:arrow_down:

...ain/java/org/opensearch/search/sort/MinAndMax.java
63.15% <0.00%> (-36.85%)
:arrow_down:

...tions/InternalSingleBucketAggregationTestCase.java
59.74% <0.00%> (-28.58%)
:arrow_down:

.../java/org/opensearch/index/reindex/RemoteInfo.java
50.45% <0.00%> (-28.45%)
:arrow_down:

...opensearch/index/reindex/BulkByScrollResponse.java
48.38% <0.00%> (-28.23%)
:arrow_down:

... and 514 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


[flutter_tools] don't compute hashes of well known artifacts

Description
Currently the tool uses the file hashes of all  artifacts in order to ensure that any updates cause the corresponding targets to be re-run. Since the cache directory is versioned, this is strictly necessary on known release branches, and can be short-circuited with the engine revision.
The relies on the fact that the bin/cache engine rolls together, and that the cache is correctly up to date. The result is a measurable speedup from calculating fewer hashes.
Local engine builds still use the artifacts to compute the target.
Now:
jonahwilliams-macbookpro2:hello_world jonahwilliams$ time flutter build web
Running "flutter pub get" in hello_world...                         0.5s
Compiling lib/main.dart for the Web...                             10.9s

real	0m12.223s
user	0m15.644s
sys	0m1.804s
jonahwilliams-macbookpro2:hello_world jonahwilliams$ time flutter build web
Compiling lib/main.dart for the Web...                              0.4s

real	0m1.193s
user	0m1.261s
sys	0m0.299s

Then
Running "flutter pub get" in hello_world...                         0.5s
Compiling lib/main.dart for the Web...                             12.2s

real	0m13.500s
user	0m16.909s
sys	0m1.973s
jonahwilliams-macbookpro2:hello_world jonahwilliams$ time flutter build web
Compiling lib/main.dart for the Web...                              1.9s

real	0m2.704s
user	0m2.671s
sys	0m0.537s

For web builds which only compute the hash for dart, this saves about a full second off off all compiles, which is especially nice for no-ops.

Adding @zanderso  for thoughts too


fail to produce 2.3KB wasm binary from example

Hi, I am building example/hellovecty. While everything works, the binary go compiler produce is 2.3M instead of 2.3KB in the readme.
Is it a typo in readme or there is some steps I need to follow to reduce the binary size? Thank you.
go version:

go version go1.12.4 linux/amd64

Yeah, it looks like this was a typo. @slimsag fixed and explained it in this PR.
The correct bundle size of hellovecty w/ WASM is 2.3MB (0.5MB compressed).

@Andoryuuta is correct, see https://github.com/gopherjs/vecty/pull/242 for an explanation. Apologies for the confusion!


Bug: Horizontal scroll bar is hidden when 8 or less rows are on the screen

SQL Operations Studio Version: 0.30.6

Horizontal scroll bar is hidden when 8 or less rows are on the screen:

the issue is fixed.


Format error for ES mobile range

Hello Support,
File: PhoneNumberMetadata.xml
Country: ES
Example number(s) and/or range(s):<PHONE_NUMBER>
Number type ("fixed-line", "mobile", "short code", etc.): Mobile
In "ES" territory for  one of your comments reads "Added 78 range based on user bug reports" (issue #1450).
When you added range 78 I think that you made a mistake. Right now the number pattern you have configured is:
(?:6\d{6}|7[1-48]\d{5}|9(?:6906(?:09|10)|7390\d{2}))\d{2}
But this includes this range: from 710000000 to<PHONE_NUMBER>.
Number 710000000 is correct, but<PHONE_NUMBER> includes 10 digit numbers that are not supported for that territory ().
You should change from:
7[1-48]\d{5}
to:
7([1-4]|8)\d{5}
Correct full number patter would be:
(?:6\d{6}|7([1-4]|8)\d{5}|9(?:6906(?:09|10)|7390\d{2}))\d{2}
That includes this ranges for numbers starting in 7:
From 710000000 to 749999999
From 780000000 to 789999999
Best Regards,
Jorge

Thanks for the report!
I think actually it's fine. You'll notice on our demo that http://libphonenumber.appspot.com/phonenumberparser?number=7489999999&country=ES is not valid (too long).
7[1-48]\d{5} means 7 followed by 1, 2, 3, 4 or 8, followed by 5 digits.
7([1-4]|8)\d{5} is the same.
Did you have any particular example number that we are validating as valid that we shouldn't be?

No particular example. Just a wrong parser. Thank you for the tip, you can close or delete this thread

@Jsuareza ,
Thanks for the confirmations. Closing this bug as working as intended.


Check input schemas for reserved types on deploy

Certain operations fail when a subgraph's GraphQL input schema contains types that we reserve for the API schema, such as Subscription or Query. Using such types leads to obscure and hard to track down error messages. Instead, we should check for these types when a subgraph is deployed and reject the deploy with a clear error message.
To find the types affected by that, search the code for APISchemaError::TypeExists

We could use https://github.com/dotansimha/graphql-eslint and also lint in graph-cli even before deploying to the graph-node.

You can also run into the same issue if you set signature as an entity field name


snprintf formatting does not honour AVR '%S' (i.e. not '%s') for printing F() flash strings

My sample failing test (that works on an Arduino AVR Nano) is
test(T1272_GoldStandard_Test_snprintf_9) {
  char buff128[128] = "?";
  auto str1 = "Resolved";
  auto str2 = F("Hello9");
  auto str3 = F("not this");
  str3 = F("World");
  snprintf(buff128, ARRAY_SIZE(buff128), "<%s><%S><%S>", str1, str2, str3);
  assertEqual("<Resolved><Hello9><World>", buff128);
}

Note that traditionally (whatever that means), '%S' is used for UNICODE(wchar_t) strings, but AVR uses it for ASCII strings in Flash(Programme) memory..
When compiling my test, it gives the warning:
TestGetFlashString.ino: In member function ‘virtual void test_T1272_GoldStandard_Test_snprintf_9::once()’:
TestGetFlashString.ino:34:42: warning: format ‘%S’ expects argument of type ‘wchar_t*’, but argument 5 has type ‘const __FlashStringHelper*’ [-Wformat=]
   34 |   snprintf(buff128, ARRAY_SIZE(buff128), "<%s><%S><%S>", str1, str2, str3);
      |                                          ^~~~~~~~~~~~~~        ~~~~
      |                                                                |
      |                                                                const __FlashStringHelper*

You are calling the snprintf() on the native host (Linux, MacOS, FreeBSD?) so of course it does not know about %S. (In fact, I didn't even know about %S, I cannot find any documentation after a few minutes of internet searching). Unfortunately, there is little EpoxyDuino can do to fix this. The only workaround I can suggest is to use some conditional code:
test(T1272_GoldStandard_Test_snprintf_9) {
  char buff128[128] = "?";
  auto str1 = "Resolved";
  auto str2 = F("Hello9");
  auto str3 = F("not this");
  str3 = F("World");
#if defined(EPOXY_DUINO)
  snprintf(buff128, ARRAY_SIZE(buff128), "<%s><%s><%s>", str1, str2, str3);
#else
  snprintf(buff128, ARRAY_SIZE(buff128), "<%s><%S><%S>", str1, str2, str3);
#endif
  assertEqual("<Resolved><Hello9><World>", buff128);
}

I don't see any reasonable way to fix this under POSIX environment. Closing.


Configuration example for GitLab CI multirunner environment

Currently, no documentation exists on how to use Protractor with Angular e2e tests on GitLab CI. It's an impossible mix yet it is one of the popular CI platforms.
I'm calling for addition of the separate block in documentation that would showcase GitLab CI integration as the whole topic is heavily underdocumented on all sides from Selenium to Protractor to Angular.
To illustrate the hardship, please also see discussion on Stackoverflow.

I think we can add label PR's are welcome! ;)

I worked it out and I do have a working solution.
I'll take a look into it and see how it'd make sense for the Protractor.

I'm facing this same exact problem  So @ain  if you have any clues or advices on a proper config (protractor settings, package.json, .gitlab-ci.yml samples for example). Your help would be very appreciated...

I'll do my best to publish the setup this week, yes.

The key of solving this impasse was to use Xvfb.
I've posted my .gitlab-ci.yml in How to run AngularJS end-to-end tests on GitLab CI.


Bourbon releases

any chance we can let us know what versions work with other version of neat, bitters and refills
thanks

@weareREASON Hello!
You can find requirements in each project’s readme:

https://github.com/thoughtbot/bourbon#requirements
https://github.com/thoughtbot/neat#requirements
https://github.com/thoughtbot/bitters#requirements
https://github.com/thoughtbot/refills#requirements


Refactoring count

boolean(count = 1) {
    if (count < 1) throw new Error("count must be greater than 0");

    const arr = [];

    for (let i = 0; i < count; i++) {
      const oneOrTwo = this.to(2);

      if (oneOrTwo === 1) {
        arr.push(true);
      } else arr.push(false);
    }

    return count === 1 ? arr[0] : arr;
  }

I found a neat way to remove a lot of duplicate code. It creates an array no matter what the value of count is. In the return statement you'll see how it handles what to output. Need this in all of the other methods that take count as an argument.

I'll knock this out while handling #29

@Gridonyx is this done?

Yep, all of them use the ternary for the return. shuffle and string don't currently have the option for returning an array so those don't have it of course. Those two are on the todo list though.

Ok I'll close this whenever that's complete. Just make mention of it in the PR

ETA on this? I don't mind taking a stab at it if you're too busy

I've been on vacation this week which is why I haven't worked on it haha, I'll be getting back home this weekend


pull request for e2e test at 12/10/2016 14:45:02

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Use GetHessians in RHS

Hi,
I am trying to solve the following equation for f
Δf = Phi_xx Phi_yy -Phi_xy^2 - J^2 - (J_x Psi_x + J_y Psi_y)
where Phi, Psi, and J are all already solved GridFunctions.
I am trying to define RHS as a coefficient and then call DomainLFIntegrator like this
    double RHSCoefficient::Eval(DenseMatrix &M, ElementTransformation &T,
                     const IntegrationRule &ir)
    {
       ...
        Psi->GetGradients(T, ir, gradPsi);
          J->GetGradients(T, ir, gradJ);
          J->GetValues(T, ir, Jvalue);
        Phi->GetHessians(T, ir, Hessians);

        for (int i=0; i<ir.GetNPoints(); i++)
        {
            M(0,j)= Hessians(i, 0)*Hessians(i,2) - Hessians(i,1)*Hessians(i,1)
                  -Jvalue(i)*Jvalue(i)-gradJ(0,i)*gradPsi(0,i)-gradJ(1,i)*gradPsi(1,i);
        }
    }

But this is not how GetHessians work. Any suggestions?
Qi
Correction: I think to use DomainLFIntegrator, I need to evaluate the RHS at the integration point instead of evaluate at each element. So I am not sure how to use GetHessians. Is there any other solution? It is only in post-processing, so it is fine if the solution is very expensive.

If I understand correctly, the rhs Phi_xx Phi_yy -Phi_xy^2 - J^2 - (J_x Psi_x + J_y Psi_y) is a scalar, so you need to derive your RHSCoefficient from class Coefficient and override the virtual method: https://github.com/mfem/mfem/blob/0f239c22e03b8e3b7d8e3311597ca6d809e8f5c2/fem/coefficient.hpp#L53-L59 In that case you only have a single IntegrationPoint and not IntegrationRule (which is an array of IntegrationPoint with some additional methods). In order to call the method GetHessians you can add an IntegrationRule ir with one point to your class RHSCoefficient, initialize it with the constructor: https://github.com/mfem/mfem/blob/0f239c22e03b8e3b7d8e3311597ca6d809e8f5c2/fem/intrules.hpp#L217-L218 and then inside the definition of Eval simply copy the IntegrationPoint &ip to ir with ir.IntPoint(0) = ip;.
Does this make sense?
One thing to remember is that Phi has discontinuous first derivatives across element boundaries, so computing the Hessian locally on each element may not be always sufficient -- e.g. for Phi in a linear FE space on triangles or tets, the Hessian will be 0. Therefore, one may need to reconstruct a global approximation of the Hessian first to take into account the jumps of the first derivatives of Phi across faces.

Thanks @v-dobrev for the suggestion. It makes sense and I am working on that.
I have one question on your point of reconstructing Hessian. I am working on a model that needs a lot second-order derivative terms in my nonlinear residual, for instance,
\partial_{RZ}\Psi ( (\partial_R \Psi)^2 - (\partial_Z \Psi)^2 ) + \partial_R \Psi \partial_Z \Psi(  \partial_{ZZ}\Psi-  \partial_{RR}\Psi)
I am using higher order bases, so none of them will be 0 but the direct evaluation of Hessian will have kink or discontinuity along the element interface.
In such a case, will you suggest to reconstruct Hessian first and then evaluate the nonlinear term in the residual? Is there any other approach to evaluate this nonlinear term? Unfortunately, I cannot really do integration by parts.

In such a case, will you suggest to reconstruct Hessian first and then evaluate the nonlinear term in the residual?

Generally, I expect that reconstructing a Hessian approximation will be more accurate, however there may be cases where it is not essential.

Is there any other approach to evaluate this nonlinear term?

The only thing that comes to mind is if the term you have can be simplified by introducing some auxiliary field derived from \Psi. However, I suspect you would have done that already if it were possible.

Thanks a lot. @v-dobrev It was very helpful.
I got this working in post processing. I will open a new issue if I run into more questions regarding the new model.