Add Mean and Median to std.algorithm

This covers issue 14034.
I couldn't find a way to make a median function for input ranges that didn't use the GC, so I just left it to random access ranges.

Fixed mentioned issues and rebased.

BTW, it is possible to implement median for (sorted) forward ranges without allocating. If length is defined, you just drop the first r.length/2 elements of the range and compute the median.
If length is not defined, save the forward range at the beginning, then iterate over the original range and the saved range at half the speed, i.e., for every 2 elements popped from the original range, pop 1 element from the saved range. When the original range is exhausted, the saved range will be either on the median element, or on the element right before the median, and can therefore be computed easily.
For bidirectional ranges, you can just call popFront and popBack until the range is down to just 1 or 2 elements. (If you save the values of the last front and back elements, you can just pop both ends until empty then return their average.)
Note, though, that for anything other than a random access range with length, the complexity of median will be O(n).  This should be, of course, documented clearly. Also, in all cases median should take a SortedRange, so that the onus is on the caller to sort his data (or ensure it arrives in sorted order) before feeding it to median.

@quickfur Addressed all concerns and rebased.

as the median is only a fancy word for the 0.5 quantil why not add a generic quantil functions like:
auto quantil(R)(R r, double q) { // q <= 0.0 && q <= 1.0

?

And then add a convenience function for the median? Yeah that sounds reasonable.

Changed code to match @burner's requests.
I'm not happy with the cast's everywhere, but I don't think there is another way to do it.

The submitted code is incorrect as long cannot be used as array indexes on 32 bit systems. Other than using casts, I see no way to round a real to an int because all of the rounding functions give long.

Updated code and auto tester is green.

version(debug) {
} else {
}
but having test behavior change between debug and release IMO is wron

The test behavior needs to change for the empty range test as the contract will fail in debug mode but in release mode the function will return double.nan.

@burner your solution doesn't compile
std/algorithm/iteration.d(4358): Error: identifier or integer expected, not debug

debug mode but in release mode the function will return double.nan.

Just always return nan and in debug mode it will assert before that?

Just always return nan and in debug mode it will assert before that?

Did you accidentally a word?

quantile should be pure, but is isn't because of its use of modf. I submitted a pull at https://github.com/D-Programming-Language/druntime/pull/1373 which would automatically mark quantile as pure.

Adressed all comments and rebased.

Adressed all concerns and rebased. Also changed ddoc to use proper big O notation.

Also, probably should also unittest the corner case of median with a single-element range. Does it do the right thing in that case?

Adressed all concerns.
Not only is this function not pure because of modf, but it's also not CTFE-able, which significantly reduces it's usefulness IMO. Let's hope Iain Buclaw's math refactorings come sooner rather than later.

Adressed all concerns and rebased.

Ping? I have another PR waiting on this.
Also, for some reason druntime unit tests on 32bit linux are failing. I doubt that has anything to do with this PR.

@JackStouffer not everything has been addressed.
please recheck

Changed the return types of the two functions to always return reals, as having the mean of [1, 2] as 1 makes no sense.

Druntime failed again :(

Please test with an empty range, then it LGTM

Done.

LGTM

What happened to the latest comments from myself and @yebblies ? Thought I saw them earlier but can't find them now...

Other than the comments made by yebblies five days ago on the SortedRange issue, I never received email notifications about any comments.

@quickfur @DmitryOlshansky This got an OK from @burner, can I get your opinions.

All tests are green and no one seems to have any more objections.

ping @DmitryOlshansky @JakobOvrum @quickfur

@John-Colvin can I get your opinion on this, seeing as you're the unofficial D statistics guy.

Ok, now that's three people who have given this PR an ok. Ready to merge.

Ping @DmitryOlshansky @JakobOvrum @quickfur @schveiguy
It has been 11 days since this has been ready to merge (6 if you count very minor doc updates). As I said before I have another PR waiting on this, it would be great to get this merged soon.

Overall it LGTM, it's just that I'm not really qualified to spot issues in the quantile algorithm.

@JakobOvrum Yes, but burner and John-Colvin, who seem confident in their knowledge of these algorithms, gave it an ok.

@JackStouffer That's not really a fair representation of what I've said. To be clear: I think mean is ok. I haven't reviewed quantile but will do so now.

@John-Colvin Sorry, I took your silence as acceptance.

@JackStouffer on a meta note.
I think you pushing to hard. D is a "open source/no one is paid to work" project. Asking people to do work for you drives people away, and they do less. This comes from personal experience.
What makes me really itchy though is:

Ok, now that's three people who have given this PR an ok. Ready to merge.
when ctrl-f "LGTM" only today gives me two people actually saving LGTM. when D has as signed-off-by then it is LGTM.

This functionally will get in eventually, but not as fast as you like

Sorry, you're right.
I was frustrated that people had given an ok but it was just sitting here. I'm happy to fix things if people tell me they're broken, I just didn't want to see this die on the vine.

it will not die. just use more of the number one computer science skill:
frustration tolerance

Quantiles from data aren't really a trivial, done-and-dusted thing. See https://stat.ethz.ch/R-manual/R-devel/library/stats/html/quantile.html for a serious quantile function and the wikipedia article for an example of a simple one.
A simple quantile, like is shown in the wikipedia example can be done something like this: auto idx = size_t(round(r.length * q)); r.topN(idx); return i[idx]; (although obviously you wouldn't want to do that sort for every call to quantile, I just wanted to illustrate how easy it can be).
Overall, I'm not sure something like quantile belongs in std.algorithm, especially if it is going to do any interpolation (i.e. anything other than "choose the nearest index"). It's too much messy statistics decisions, not enough generic manipulation of arbitrary data. If your not doing any interpolation though, it's no longer really much of an algorithm, it's just "read the value at the index closest to q*length, assuming sortedness".
Could I suggest that if you're interested in doing a more comprehensive treatment of quantiles, you could make a pull request to https://github.com/DlangScience/dstats with it?
mean is a safer proposition, but it'd be great to hear from @andralex about whether he sees this as within the remit of std.algorithm or whether it should go elsewhere. The thing is, it's not very generic, seeing as it only works (by definition), with types that support addition and division.

Quantiles from data aren't really a trivial, done-and-dusted thing. See https://stat.ethz.ch/R-manual/R-devel/library/stats/html/quantile.html for a serious quantile function and the wikipedia article for an example of a simple one.

Thanks for the link.

The thing is, it's not very generic, seeing as it only works (by definition), with types that emulate real numbers.

I'll make the functions more generic by allowing any user defined type that defines + and / and test with BigInt.

but it'd be great to hear from @andralex about whether he sees this as within the remit of std.algorithm or whether it should go elsewhere (e.g. std.math?).

I don't think std.math is a good fit, as there are no other range based functions in there. I think the only other module that fits is std.numeric, but that seems to be an attempt at a strait one to one implementation of the STL.

Could I suggest that if you're interested in doing a more comprehensive treatment of quantiles, you could make a pull request to https://github.com/DlangScience/dstats with it?

I keep quantile in there for now, until more people in this PR tell me it's no good for Phobos. If that's true, I'll open a PR on that repo.

After doing more research, I have concluded that the quantile is not nearly complex enough to be useful and requires a much more through look. So, I removed it from this PR, I added the mean overloads back in, and I added support for user defined types to both functions that don't implicitly convert to real.
I'll open a WIP PR on dstats and if it's good it can always be added back into phobos later.

I think mean is ready to go (has been ready for a while now). Perhaps the thing to do here is to split this PR into one for mean and one for median/quantile/etc.. The first one can probably be merged already, it's the second one that's been holding this up.
Generally, it's a good idea to split things up into multiple PRs when there are various pieces that don't directly depend on each other, otherwise perfectly good changes will get held up unnecessarily just because another, mostly-unrelated piece.

@quickfur ok, but before I do that there was one thing that I have been thinking about with this PR.
I also want to add various other statistical methods to Phobos, such as standard deviation, which fit in std.algorithm, and things such as rolling mean, which don't fit into std.algorithm. I was thinking that something like a rolling mean would best fit in std.numeric, but, would it be best if all of the statistical methods were in the same module, or is it ok to have them separate?

@JackStouffer this is relevant to discussions about rolling means etc. https://github.com/D-Programming-Language/phobos/pull/2991

@John-Colvin looks interesting. I'll wait on the rolling mean/median stuff until a decision is reached on that PR. In the mean (ha) time, mean and median need to have a definite home.

https://github.com/D-Programming-Language/phobos/pull/3679
https://github.com/D-Programming-Language/phobos/pull/3680


Update example in the numpy cheatsheet to use the xt namespace

Trivial change to include xt:: for consistency.

Thanks!


Automate building of qiskit-tutorials to make doc

Summary
This commit changes the statically generated markdown from the
qiskit-tutorials repo to use automatically generated output on each run
of make doc. It will clone the qiskit-tutorials copy the relevant
notebooks to the proper place in the docs tree and then leverage the
nbsphinx extension to render those as part of the sphinx build. This
means we'll automatically get updates to the tutorials in the docs as we
start to use them for more guides in terra and we don't have to manually
update them.
Details and comments
This depends on having nbsphinx and pandoc installed to run.

can i see a link to the documentation to see what it looks like?

@jaygambetta I hosted the output of a docs build with this patch here: http://<IP_ADDRESS>/docs/

can we remove the IN and out

I'm not sure, I'll have to read through the nbsphinx docs to see if there is a method to do that in there. Although it's just calling nbconvert internally so we probably can just leverage a template file and put it in the right place. I'll try to come up with something

I've found a way to remove the In and Out prompts from the code blocks. It turns out there are multiple open issues and PRs on the nbsphinx project trying to add a direct config flag for this (and exposing other nbconvert features), but none have merged and the maintainer seems to be blocking them. But in the meantime I've found a workaround, just setting the plot width to 0 will remove those.
As for treating the output as not a code block that will probably be a bit more difficult, I'll come back to that when I have some time I can dedicate to working on the problem

It seems we might be able to play with css - some classes input_area and output_area are appended to the divs.

can you post another link

I don't know I think the nbconvert still does a better job (less spaces etc).

nbsphinx is using nbconvert internally. It just wraps it so it adds the notebooks as a document in a sphinx build. Is there a particular way in which you were running nbconvert locally? nbsphinx could be changing the formatting or something and I can look into adjusting the output more.

for example
jupyter nbconvert --to rst the_ibmq_provider.ipynb


Synonyms max length revision

Synonyms max. length is 255 characters. This is not enough for some of the synonyms of chemical compounds. Suggestion is to raise the max. length to 400 char.

@mehdiabbasi please go ahead and increase the max. length for Synonyms to 400 char.

@Osanna123 this task is done.


add missing openapi 2.0 operationid

What this PR does / why we need it:
Add openapi 2.0 specs' missing operationid field.
See spec docs: https://swagger.io/docs/specification/paths-and-operations/ (search operationId) here.
Which issue(s) this PR is related to (optional, link to 3rd issue(s)):

Special notes for your reviewer:

Release note:

NONE

Ping @iawia002


IAM permissions documentation may be missing glue:CreateDatabase

The IAM permissions documentation lists the following required permissions for Glue:

Glue IAM permissions
dbt-athena uses the AWS Glue API to fetch metadata. You will need to set these permissions on the Glue databases you are reading from:
"glue:GetDatabase",
"glue:GetDatabases",
"glue:GetTable",
"glue:GetTables",
"glue:GetPartition",
"glue:GetPartitions",

You will need these permissions on the glue databases you are writing to:
"glue:GetDatabase",
"glue:GetDatabases",
"glue:GetTable",
"glue:GetTables",
"glue:GetPartition",
"glue:GetPartitions",
"glue:BatchCreatePartition",
"glue:BatchUpdatePartition",
"glue:BatchDeletePartition",
"glue:BatchDeleteTable",
"glue:BatchDeleteTableVersion",
"glue:CreatePartition",
"glue:UpdatePartition",
"glue:DeletePartition",
"glue:CreateTable",
"glue:UpdateTable",
"glue:DeleteTable",
"glue:DeleteTableVersion",

However, with these permissions in place for the role that runs our CI, we're seeing a permissions error message on dbt run when dbt-athena attempts to create a schema that does not exist:
$ dbt --debug run --target ci
20:46:11  Creating schema "database: "awsdatacatalog"
schema: "ci_jeancochrane31_default"

20:46:11  Using athena connection "create_awsdatacatalog_ci_jeancochrane31_default"
20:46:11  On create_awsdatacatalog_ci_jeancochrane31_default: -- /* {"app": "dbt", "dbt_version": "1.5.4", "profile_name": "athena", "target_name": "ci", "connection_name": "create_awsdatacatalog_ci_jeancochrane31_default"} */
create schema if not exists `awsdatacatalog`.`ci_jeancochrane31_default`

20:46:11  Opening a new connection, currently in state closed
20:46:12  Athena adapter: Query state is: QUEUED. Sleeping for 1.0...
20:46:13  Athena adapter: Query state is: RUNNING. Sleeping for 1.0...

Athena adapter: Error running SQL: macro create_schema
20:47:35  On create_awsdatacatalog_ci_jeancochrane31_default: Close
20:47:36  Athena adapter: Error running SQL: -- /* {"app": "dbt", "dbt_version": "1.5.4", "profile_name": "athena", "target_name": "ci", "connection_name": "create_awsdatacatalog_ci_jeancochrane31_default"} */
create schema if not exists `awsdatacatalog`.`ci_jeancochrane31_default`

20:47:36  Connection 'create_awsdatacatalog_ci_jeancochrane31_default' was properly closed.
20:47:36
20:47:36  Finished running  in 0 hours 1 minutes and 25.29 seconds (85.29s).
20:47:36  Encountered an error:
Runtime Error
  Runtime Error
    [ErrorCategory:USER_ERROR, ErrorCode:PERMISSION_ERROR], Detail:Amazon Athena experienced a permission error. Please provide proper permission and submitting the query again. If the issue reoccurs, contact AWS support for further assistance. You will not be charged for this query. We apologize for the inconvenience., Message:Amazon Athena experienced a permission error. Please provide proper permission and submitting the query again. If the issue reoccurs, contact AWS support for further assistance. You will not be charged for this query. We apologize for the inconvenience.
20:47:36  Command `dbt run` failed at 20:47:36.221700 after 87.46 seconds

Adding glue:CreateDatabase permissions to the IAM role policy resolved the error.
Perhaps the documentation assumes that we will not need to create a database, which could make sense given that the AWS resources page lists creating a database as a prerequisite. But since dbt-athena has the power to create a database and will do so if one does not exist, it seems confusing to me that it isn't part of the recommended permissions, especially as someone who is not an expert on Glue permissions.
What do you think? Would you be open to a patch that tweaks the documentation to add this permission, or would you prefer that the recommended permissions be locked down as tightly as possible given that the prerequisites imply that a database should already exist?

In dev environments and PR's we also let dbt take care of creating databases 👍 (e.g. dev_jessedobbelaere_* or dev_<branch_name>) so I guess it makes sense to document this permission indeed


MINOR: Add maybeThrow method to ZooKeeperClient AsyncResponse

Add maybeThrow method to ZooKeeperClient AsyncResponse
Update KafkaZkClient to use maybeThrow method

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@junrao @ijuma pinging for review


WebExtensions question

Will this extension be supported by Firefox 57 + versions?
https://blog.mozilla.org/addons/2016/11/23/add-ons-in-2017/

The web-extension will be a separate extension. The repository is here. At the moment I have implemented the Copy Tab URLs function and will be submitting it to AMO within two weeks.


Migrate BrowserStore to redux

Related to the Github Issue: mattermost/mattermost-server#7236
Checklist

[x] Added or updated unit tests (required for all new features)
[x] Has redux changes (please link)

Fixed the packages.json and rebased to master :)


what is the algorithm used to draw protein 2D representation?

Hello. What is the algorithm used in pdb-topology-viewer? Is that based on HERA? Was it published somewhere?
Thank you very much in advance.

Hi, We use the PDBsum package maintained by Roman Laskowski. Here is the PDBsum publication FYR https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2686501/

Thank you!


update cowboy_http_rest:variances/2

Fix pattern in case statement that was intended
to strip away the first comma separating the
values in the variance header.
Update generation of variance list to use more
idiomatic erlang. Pattern match on list structure
over using erlang:length/1 to compute length.

Merged, thanks!


Lock signature row before verifying

There exists a race condition where a verification link is used twice at the same time can result in a double signature count increment. To fix this we need to obtain a row level lock on the signature record and then check to see if the signature is pending.
This change shouldn't affect performance significantly as concurrent access to the signature record will only occur when the background job that validates signature counts runs which is every 30 minutes or the verification link is used twice which is what we want to prevent.
The tests needed to be tweaked because with_lock reloads the record from the database so in memory changes will have been lost.

This looks fine to me. I've done some testing at the SQL level and it looks good.


[skyr-url] build failure

Host Environment

OS: Windows 10 Enterprise
Compiler: MSVC Enterprise 2017 (version 15.9.28)

To Reproduce
Steps to reproduce the behavior:
./vcpkg install skyr-url
Failure logs
Computing installation plan...
The following packages will be built and installed:
skyr-url[core]:x86-windows
Detecting compiler hash for triplet x86-windows...
Starting package 1/1: skyr-url:x86-windows
Building package skyr-url[core]:x86-windows...
Could not locate cached archive: C:\Users\M. F. Mokbel\AppData\Local\vcpkg\archives\17\1708472a16f3aebb76d29b50113fadd73e25d85e.zip
-- Note: skyr-url only supports static library linkage. Building static library.
-- Using cached C:/vcpkg/downloads/cpp-netlib-url-v1.13.0.tar.gz
-- Cleaning sources at C:/vcpkg/buildtrees/skyr-url/src/v1.13.0-654c69c924.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source C:/vcpkg/downloads/cpp-netlib-url-v1.13.0.tar.gz
-- Using source at C:/vcpkg/buildtrees/skyr-url/src/v1.13.0-654c69c924.clean
-- Configuring x86-windows
-- Building x86-windows-dbg
CMake Error at scripts/cmake/vcpkg_execute_build_process.cmake:141 (message):
Command failed: C:/vcpkg/downloads/tools/cmake-3.18.4-windows/cmake-3.18.4-win32-x86/bin/cmake.exe --build . --config Debug --target install -- -v -j9
Working Directory: C:/vcpkg/buildtrees/skyr-url/x86-windows-dbg
See logs for more information:
C:\vcpkg\buildtrees\skyr-url\install-x86-windows-dbg-out.log
Call Stack (most recent call first):
scripts/cmake/vcpkg_build_cmake.cmake:92 (vcpkg_execute_build_process)
scripts/cmake/vcpkg_install_cmake.cmake:24 (vcpkg_build_cmake)
ports/skyr-url/portfile.cmake:21 (vcpkg_install_cmake)
scripts/ports.cmake:79 (include)
Error: Building package skyr-url:x86-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
Package: skyr-url:x86-windows
Vcpkg version: 2020.06.15-nohash
Additionally, attach any relevant sections from the log files above.
Additional context
The issue seems to be related to cmake.
No issues building dependencies though!
Attached is the build log.
install-x86-windows-dbg-out.log

Seems from log output, range-v3 library requires VS2019 and you're using VS2017 on this case. I'd recommended you to update your build environment.

This is weird, since I'm able to build range-v3 independently with no errors.

I confirm a successful build with Visual Studio 2019.

Closing case since there is nothing that can be done at the vcpkg level.


Does this only work with GitHub repos?

I'm trying to sync a remote repo from BitBucket to GitHub, and I get this error:
load pubkey "/root/.ssh/id_rsa": invalid format

I'm using the SSH_PRIVATE_KEY env var to set my private key.

It should work please fix your key format.

It should work please fix your key format.

What should the format be?

Something like this:
-----BEGIN PRIVATE KEY-----
MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgiYydo27aNGO9DBUW
eGEPD8oNi1LZDqfxPmQlieLBjVShRANCAAQhPVJYvGxpw+ITlnXqOSikCfz/7zms
yODIKiSueMN+3pj9icDgDnTJl7sKcWyp4Nymc9u5s/pyliJVyd680hjK
-----END PRIVATE KEY-----

Please also make sure the source_repo and env are set correctly so the Action can access it.
   - name: repo-sync
      uses: repo-sync/github-sync@v2
      with:
        source_repo: "git@bitbucket.org:name/repo.git"
        source_branch: ""
        destination_branch: ""
        github_token: ${{ secrets.PAT }}
      env:
        SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}

Another easy way per README is to use the https clone url with access token to prevent issues in SSH.

If you have any further questions, please kindly paste or link your workflows and workflow runs so we can better assist you.

Here is my workflow job:
jobs:
  repo-sync:
    runs-on: self-hosted
    steps:
    - uses: actions/checkout@v2
      with:
        persist-credentials: false
        fetch-depth: 0
    - name: sync stash repo
      uses: repo-sync/github-sync@v2
      with:
        source_repo<EMAIL_ADDRESS>        source_branch: master
        destination_branch: master
        github_token: ${{ secrets.PAT }}
      env:
        SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}

Here is the error I'm getting:
Pushing changings from tmp_upstream to origin
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/current-org/current-repo/'

I thought this wasn't using user/pass auth and was using PAT instead?

Try your PAT locally to make sure it works

Try your PAT locally to make sure it works

PAT seems to work locally, is this all the permission it needs?

In the actions run, I get the message:
Saving SSH_PRIVATE_KEY
UPSTREAM_REPO=ssh://git@stash.my-bit-bucket:x/my-org/my-repo.git
BRANCHES=master:master
Resetting origin to: ***github.com/my-org/my-repo
Adding tmp_upstream ssh://git@stash.my-bit-bucket:x/my-org/my-repo.git
Fetching tmp_upstream
Warning: Permanently added '[stash.my-bit-bucket]:x,[x]:x' (RSA) to the list of known hosts.
origin	***github.com/my-org/my-repo (fetch)
origin	***github.com/my-org/my-repo (push)
tmp_upstream	ssh://git@stash.my-bit-bucket:x/my-org/my-repo.git (fetch)
tmp_upstream	ssh://git@stash.my-bit-bucket:x/my-org/my-repo.git (push)
Pushing changings from tmp_upstream to origin
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/my-org/my-repo/'

looks like these folks hard coded the public url
https://github.com/repo-sync/github-sync/blob/35f2d0ad0b44816d1ad7fccd250596bd257b0fd7/github-sync.sh#L22

Can we add a setting to this action in order to change the URL? We're using self-hosted GitHub Enterprise at a different URL.

This line is culprit.
https://github.com/repo-sync/github-sync/blob/3832fe8e2be32372e1b3970bbae8e7079edeec88/github-sync.sh#L32
Instead of using this action, please use https://github.com/wei/git-sync which gives you more flexibility in what you want.

Please read README in full. It doesn't if you specify the full clone url.


Add Transfer Disabled Level

Adds a new Security Level that disables all transfers. This effectively make a token soul bound while in this level.
I've also added StakerConstraints.Disabled for parity.
Commits are separate and can be cherry-picked if only one change is desired.
Note!
This should be the maximum level. If more security requirements (and levels) are introduced, it would make sense to bump this up. However, that would affect any tokens that use an upgradable mechanism.
I considered using a high value like type(uint256).max, however using level 7 felt a lot cleaner.

We have already added Soulbound mode to the Creator Token Transfer Validator in V3.  Closing this PR as already completed.

Please note that the creator-token-contracts repo has been replaced by creator-token-standards and this is the repo that should be used going forward.
https://github.com/limitbreakinc/creator-token-standards


Ref added to all links without considering existing query attributes

As you can see on the attached image, the ref=frontendchecklist is being added to all the links, which results in breaking various links.

I got this too. First link I clicked was a 404. When I removed the ?ref=frontendchecklist it came up.
Link was https://marketplace.visualstudio.com/items?itemName=aws-scripting-guy.cform?ref=frontendchecklist from Authoring->Essential DevOps Tools->CloudFormation support for Visual Studio Code

Hey @falnyr @natefox . Appreciate your input; this slipped under my radar. I just fixed an issue where the javascript file was not being updated. Please try it out and let me know (you may have to clear your browser cache)


Can we make the policy to periodically check for compliance and automatically create remediation tasks?

[Enter feedback here]
Hello,
Can we make the policy to periodically check for compliance and automatically create remediation tasks? As far as I can see, we can use portal, PowerShell, or CLI to manually create remediation tasks. Can we configure this once and forget about it? So the policy will automatically check the subscription, and create/run remediation tasks when needed?
Thanks

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 99d433a1-4e81-e226-e835-639daa90dab4
Version Independent ID: ab3aa1ad-f438-e2a1-5556-9f9b6269ba76
Content: Remediate non-compliant resources - Azure Policy
Content Source: articles/governance/policy/how-to/remediate-resources.md
Service: azure-policy
GitHub Login: @timwarner-msft
Microsoft Alias: timwarner

@zaalion  As there are no further documentation changes, we will close the issue for now. If there are further questions , please revert and we will be glad to assist you.


Go to previous/ next error, warning, problem does not read the content to the screen reader

turn on a screen reader
run Go to previous/next error, warning, problem
expectation: the error, warning, or problem content will be read to me
🐛 nothing is read

Discovered while trying to get aria-live to work for presenting the inline ghost text to a screen reader. It appears that aria-live does not work even when it's set to assertive. Focusing the tooltip reads it, so maybe that's what we should do here
https://github.com/microsoft/vscode/blob/1028dd25f857c6b735a1870bb63653845786a5d5/src/vs/editor/contrib/gotoError/browser/gotoErrorWidget.ts#L62-L63

@isidorn would be great if you could pls test and confirm this expectation

/duplicate of https://github.com/microsoft/vscode/issues/148634

Thanks for creating this issue! We figured it's covering the same as another one we already have. Thus, we closed this one as a duplicate. You can search for similar existing issues. See also our issue reporting guidelines.
Happy Coding!


Potential concurrency issues AbstractMappedCommitter

Documents from other simultanious running jobs are added as ICommitOperation to the jobs commiter when using a commiter based on AbstractMappedCommitter.
Using a simpel Commiter like this will log out URLs from all simultanious running jobs.
public class LoggingCommiter extends AbstractMappedCommitter {
    private static final Logger LOG = LogManager.getLogger(LoggingCommiter.class);

...

    @Override
    protected void commitBatch(List<ICommitOperation> batch) {
        for (ICommitOperation op : batch) {
            if (op instanceof IAddOperation) {
                LOG.info(((IAddOperation) op).getReference());
            }
    }

}

Is this how it should word? or should it be possible to have a AbstractMappedCommitter while run several jobs simultaneously?

Not sure I understand your question.  The intent is that multiple thread can invoke the committer and that committer will queue requests and process them in batch, regardless which thread added the documents.  Does that answer the question?

If I create the following two job files:
<!-- crawljob1.xml -->
<httpcollector id="crawl1">
  <crawlerDefaults>
...
    <committer class="... LoggingCommiter" />
...
  </crawlerDefaults>
  <crawlers>
    <crawler id="crawl1">
      <startURLs>
       <url>http://coffee.someurl.com/</url>
        </startURLs>
      <referenceFilters>
        <filter class="$urlRegex" onMatch="include">^https?://coffee.someurl.com/.*$</filter>
      </referenceFilters>
    </crawler>
  </crawlers>

</httpcollector>

<!-- crawljob2.xml -->
<httpcollector id="crawl2">
  <crawlerDefaults>
...
    <committer class="... LoggingCommiter" />
...
  </crawlerDefaults>
  <crawlers>
    <crawler id="crawl2">
      <startURLs>
       <url>http://tea.someurl.com/</url>
        </startURLs>
      <referenceFilters>
        <filter class="$urlRegex" onMatch="include">^https?://tea.someurl.com/.*$</filter>
      </referenceFilters>
    </crawler>
  </crawlers>

</httpcollector>

Now if I start both jobs at the same time:
nohup ./collector-http.sh -a start -c crawljob1.xml &
nohup ./collector-http.sh -a start -c crawljob2.xml &

This would result in URLs from crawljob2 is printed out in the log file for crawljob1 and vice versa.
I would only expect URLs from crawljob1 in the log file for crawljob1 and the same for crawljob2.

I agree with you.  The behavior your describe does not make much sense.  Did you make sure to specify different log directories in your config?  Maybe they are pointing to the same location?  Even then, the log file names bear the name of the collector followed by the name of the crawler, which are unique in your example.   What are the log names generated by each?

I can reproduce the behaviour with the following two configurations:
https://github.com/leonardsaers/SimpleLoggingCommitter/blob/master/sample-norconex-config/a-config.xml
https://github.com/leonardsaers/SimpleLoggingCommitter/blob/master/sample-norconex-config/b-config.xml
The configurations both uses the following simple committer:
https://github.com/leonardsaers/SimpleLoggingCommitter/blob/master/src/main/java/com/norconex/collector/committer/SimpleLoggingCommitter.java
Try run both a-config and b-config from the same folder at the same time.
The committer will be given IAddOperation from both jobs:
INFO  [SimpleLoggingCommitter] Commit IAddOperation: http://www.jabbo.se/stores/210
INFO  [SimpleLoggingCommitter] Commit IAddOperation: http://www.jabbo.se/stores/190
INFO  [SimpleLoggingCommitter] Commit IAddOperation: http://www.norconex.com/search-ui-showcase-the-search-box/
INFO  [SimpleLoggingCommitter] Commit IAddOperation: http://www.norconex.com/enterprise-search-pitfalls-believing-in-magic/
INFO  [SimpleLoggingCommitter] Commit IAddOperation: http://www.jabbo.se/stores/162
INFO  [SimpleLoggingCommitter] Commit IAddOperation: http://www.jabbo.se/stores/173
INFO  [SimpleLoggingCommitter] Commit IAddOperation: http://www.jabbo.se/stores/154
INFO  [SimpleLoggingCommitter] Commit IAddOperation: http://www.jabbo.se/stores/90
INFO  [SimpleLoggingCommitter] Commit IAddOperation: http://www.norconex.com/google-search-appliance-gsa-a-journey-into-an-accessible-responsive-web-design/

Hello !
I was able to reproduce your issue. I was able to fix it by adding the parameter  <queueDir>queue-B</queueDir>
Give it a try and let met know how it works for you.
<committer class="com.norconex.collector.committer.SimpleLoggingCommitter">
 <queueDir>queue-B</queueDir>
</committer>

Thanks, it works fine after adding queueDir.
I would suggest that the default queueDir includes the httpcollector id. Than this would be handled with default configuration.

We thought about your suggestion, but the challenge is that the Committers are designed so they can be used by anything, not just the Collectors.  This means in a context where it is used without a collector, we would have to request an arbitrary prefix, and nothing would prevent someone from storing the same.
Still, we should figure out something when used with Collectors, so I will create a feature-request to that effect under the Norconex Collector Core project.

Since you have a viable solution for now and a new ticket has been created to address this better in the future (https://github.com/Norconex/committer-core/issues/9), I am closing this issue.  Feel free to re-open or create a new one if something else pops up.  For follow-ups on general Committer issues, use this link.


Implement Object Print

It would be helpful for the cle::Object to have its << operator implemented.
This is helpful to view states/use with in pipline processes

Thanks for the feedback,
This has been added with the incoming version.

https://github.com/clEsperanto/CLIc_prototype/pull/94/commits/41bdf312a5e31b17266f2cdde23a3175a7206d54

All data class and kernel operation class will have now both ToString() and <<operator() implemented

Thanks!
Looking forwart to testing it out


Watch doesn't trigger while processing scss with Gulp

Describe the bug
I am using Gulp to compile several .scss files into my main theme.scss.css file (which is in the assets directory). This works fine, but Theme Watch doesn't recognise when the theme.scss.css file has been updated when recompiled. However, if I manually update the theme.scss.css file in my IDE it will recognise the change and push up to the remote server.
Working with other normal section, asset and snippet files works fine - theme watch detects the change and pushes them up.
To Reproduce
Steps to reproduce the behavior:

Install Gulp and create the following Gulpfile.js:

const gulp = require('gulp');
const sass = require('gulp-sass');
const rename = require("gulp-rename");

gulp.task('sass', function(){
  return gulp.src('scss/main.scss')
	.pipe(sass())
	.pipe(gulp.dest('css'))
});

gulp.task('rename', function(){
return gulp.src("css/main.css")
  .pipe(rename("theme.scss.css"))
  .pipe(gulp.dest("assets"));
});

gulp.task('do-sass', gulp.series('sass', 'rename'))`

Run theme watch
Compile scss with the gulp do-sass command
theme watch may push up the first set of changes to theme.scss.css, but not any subsequent changes

Expected behavior
A clear and concise description of what you expected to happen.
Environment (please complete the following information):

OS [e.g. iOS]: MacOS 10
Themekit version [e.g. v0.7.5]: 1.0.0
Editor [e.g. atom, sublime]: Sublime

Additional context
Add any other context about the problem here.

Thanks trev-dev, you rock...

I couldn't get Trev-Dev's solution working for me, so came up with an alternative. Here's a stripped-down version. I essentially run a seperate Gulp Watch on the generated CSS file, and perform a 'deploy' on that specific file.
const themeKit = require('@shopify/themekit');
function uploadStyle(){
return themeKit.command('deploy', {
config: params.config_live_yml,
files: [params.styles.dest+'/styles-theme.min.css.liquid'],
env: "live"
}).catch(err => console.error('Error', err));
}
function processStyles(){
return gulp.src(params.styles.src)
.pipe(autoprefixer({
browsers: ['last 2 versions'],
cascade: false
}))
.pipe(cleanCSS())
.pipe(rename({
basename: 'styles-theme',
suffix: '.min',
extname: '.css.liquid'
}))
.pipe(gulp.dest(params.styles.dest));
}
function shopify(){
gulp.watch(params.styles.watch, processStyles);
gulp.watch(params.styles.dest+'/styles-theme.min.css.liquid', uploadStyle);
themeKit.command('watch', {
    config: params.config_live_yml,
    notify: 'notify.tmp',
    env: "live"
}).catch(err => console.error('Error', err));

}
gulp.task('default', function() {
shopify();
});

I just ran across this.. As @tanema notes above, it seems to because the timestamp does not change. There are a few options, like gulp-touch (or gulp-touch-fd better, for use with Gulp 4). I liked this one from stackoverflow:
.pipe(through2.obj(function(file,enc,cb){ // change mod date, for qs watch
  let date = new Date();
  file.stat.atime = date;
  file.stat.mtime = date;
  cb(null,file);
}))

@rickydazla and @trev-dev I'm glad you found my little pipe function useful!
I've updated it to be better and more reusable:
import through2 from 'through2';
const touch = () => through2.obj( function( file, enc, cb ) {
  if ( file.stat ) {
    file.stat.atime = file.stat.mtime = file.stat.ctime = new Date();
  }
  cb( null, file );
});

So you can use .pipe( touch() ) like so
function sassDev() {
  return gulp.src( files )
    .pipe( sourcemaps.init() )
    .pipe( pipeSass() )
    .pipe( postcss() )
    .pipe( sourcemaps.write() )
    .pipe( touch() )
    .pipe( gulp.dest( sassDest ) )
    .pipe( server.stream() );
}

Currently, this is a feature of gulp 4 due to a limitation of node's fs.Stat object. If you need mtime to change - you need to modify the mtime on the files in your pipeline.

So it seems like will not be fixed in Gulp, we will have to find ways of handling it downstream and it sounds like they even suggest something like my gulp-touch solution. I will see if there is something I can do within themekit to handle this better without effecting other tools
Edit: It seems like even in the other issues linked in the gulp issue, other developers have solved this in a similar way.

gulp-touch (and similar gulp-touch-xxx) npm packages are 2 years out of date and don't appear to be working for gulp 4, FYI. I was able to achieve results using through2 which will accept a Writeable stream and allow me to change the atime/mtime.
const { task, watch, src, dest, series } = require('gulp');
const sass = require('gulp-sass');
const autoprefix = require('gulp-autoprefixer');
const t2 = require('through2'); // Get through2 as t2

sass.compiler = require('node-sass');

function scss(cb) {
  src('./src/scss/**/*.scss')
    .pipe(sass({outputStyle: 'expanded'}))
    .on('error', sass.logError)
    .pipe(autoprefix({
      browsers: ['> 2%', 'ie >= 10'],
      grid: true
    }))
    .pipe(t2.obj((chunk, enc, cb) => { // Execute through2
      let date = new Date();
      chunk.stat.atime = date;
      chunk.stat.mtime = date;
      cb(null, chunk);
    }))
    .pipe(dest('./assets/'))

    cb();
}

function watcher(cb) {
  watch('./src/scss/**/*.scss', series(scss))
  cb()
}

module.default = task('default', series([scss, watcher]));

This is working like a charm for me.

Brilliant! Thanks so much! Works!

Not sure if it's a gulp problem or not but since updating to ThemeKit 1.1.6 linux/amd64  I now have to use @WraithKenny's touch hack for everything that gets moved from ./src to my local Shopify theme directory. The only exception is my webpack bundled js.


Memory leak on Raspberrypi Zero W

SerialPort Version: 6.2.0

NodeJS Version:  v6.11.4  also v8.11.1

Operating System and Hardware Platform: Raspbian GNU/Linux 9.3 (stretch)

Have you checked the right version of the api docs?: Yes

Are you having trouble installing and you checked the Installation Special Cases docs? No issues installing

Are you using Electron and have you checked the Electron Docs?: N/A

Summary of Problem
I am getting a memory leak of about 2MB per second and CPU thrashing.
Steps and Code to Reproduce the Issue
var SerialPort = require('serialport');
var port = new SerialPort('/dev/ttyAMA0', {
  baudRate: 115200
});

// Switches the port into "flowing mode"
port.on('data', function (data) {
  console.log('Data:', data.toString() );
});

I thought it might be #1572 so I waited for the fix there, but I still have the issue using 6.2.0 . The weird thing is that I don't have the memory leak on a raspberry pi zero (no wifi). Other assorted details:
root@raspberrypi:# node -v
v6.11.4
root@raspberrypi:# grep serialport package.json
    "serialport": "^6.2.0",
root@raspberrypi:~# lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 9.3 (stretch)
Release:    9.3
Codename:   stretch
root@raspberrypi:~# cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=tty1 root=PARTUUID=74121984-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait quiet splash

I'm guessing it is environment related, but not sure where to look.

Confirmed on :
RPi Zero W v1.1
NodeJS : v8.11.3
Raspbian 9.4 (stretch)
rss usage grow by ~2MB/s and cpu load is almost 100%
Cause
On RPi Zero W (and probably RPi 3) port serial1 is linked to ttyAMA0 and used by Bluetooth
pi@zero:~ $ ls -la /dev/ttyAMA0
lrwxrwxrwx 1 root root 7 Jun 25 13:53 /dev/serial1 -> ttyAMA0

pi@zero:~ $ systemctl status hciuart.service -l
● hciuart.service - Configure Bluetooth Modems connected by UART
   Loaded: loaded (/lib/systemd/system/hciuart.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-06-25 13:53:17 CEST; 17s ago
  Process: 227 ExecStart=/usr/bin/btuart (code=exited, status=0/SUCCESS)
 Main PID: 357 (hciattach)
   CGroup: /system.slice/hciuart.service
           └─357 /usr/bin/hciattach /dev/serial1 bcm43xx 3000000 flow - b8:27:eb:35:a6:a0

Solution 1
Don't use /dev/serial1 or /dev/ttyAMA0.
Instead, add enable_uart=1 to /boot/config.txt and reboot.
Device /dev/serial0 will appear. It is wired to GPIO14 and 15.
This serial port is miniUART and have serious limitations (as one can find in net)
Solution 2 (maybe better)
If Bluetooth is not needed, add dtoverlay=pi3-disable-bt to /boot/config.txt and reboot. Now full UART /dev/ttyAMA0 is usable on GPIO14 and 15.

Woo! This did it:

If Bluetooth is not needed, add dtoverlay=pi3-disable-bt to /boot/config.txt and reboot. Now full UART /dev/ttyAMA0 is usable on GPIO14 and 15.

Thanks!


[5.7] Adds missing logging options to slack log driver

Usecase
Hide some fields from the Slack message (eg. PII), when Slack is used as a Log driver.
More info
While you can define a property excludeFields in your logging.php config file, this is not taken into account when Slack is used as a log driver. The method of LogManager that creates the Slack driver does not pass this information to SlackWebhookHandler even though the later can support it. This PR addresses this by adding the missing fields.
Note
The default values have been set to match the ones found at the constructor of SlackWebhookHandler.

@taylorotwell Thanks, updated. Can you also guide me on where I could possibly document this feature?

Probably laravel/docs notifications.md


Added endpoint to show smart contract logs

Is it ok that this is in the full node? @fassadlr

Also, I don't think this is the right place to put this endpoint.
Each feature will have its own UI, so dumping a smart contract related thing in the Stratis.Bitcoin is probably not the right thing to do. @majikandy @fassadlr

@bokobza this only a merge to our sc-alpha branch ;) but you are 100% correct the controller should go in the Smart Contract Feature. @majikandy can you please move it there and see that it works?


[SoftDeletable] Deleted entry and unique validator

I have a case where users are entering unique IDs into a table directly (rather than autogenerating) and I am using the @UniqueEntity validator to prevent constraint violations.
However it seems if an entry is "soft deleted" the validator is ignored resulting in a error/exception being executed due to unique constraint violation.  Steps to reproduce would be:

Create an entry with id "test"
Delete entry (soft delete)
Create a new entry with "test" again.

Is there a way to continue to use this validator even with soft deleted entries?
I know I could work around this by using an auto generating ID column and keeping this as a unique secondary identifier however I was hoping for a cleaner solution.

extend the validator and make your own which disables softdeleteable when the check is done. the symfony framework cannot interfere if extensions and visa versa. you have to adapt to whatever you use.

Thanks.  I found a thread here which outlines how someone approached it:
https://github.com/symfony/symfony/issues/6955
Definitely not a quick and simple solution.  Here's hoping something more straight forward come along to help us newbs in the future.


Sign In API - Internal Server Error

Do you have the latest update on branch develop? It works on my end.

Then check if the file app/views/api/v1/auth/sign_in.json.jbuilder exists.

Fixed this by forcing all request on API endpoints as json.
api/v1/sign_in.json, api/v1/sign_in, api/v1/sign_in.html` and
others are now treated as json requests


Support both ORC and Parquet

So we used both and recently switched to ORC, but I'm noticing that Athena will read ALL the bytes when hitting just one column, so lets support both and have Parquet be the default.

Okay, put in support for both ORC and Parquet, also put up a small wiki page: https://github.com/SuperCowPowers/sageworks/wiki/Parquet-vs-ORC


Added the AI Anime Avatar Generator model with with documentation (README, requirements, LICENSE)

Related Issues or bug

Added the AI Anime Avatar Generator folder with documentation, including README.md, requirements.txt, and LICENSE.md files. Fixed issue #495

Fixes: #495
Proposed Changes

Implemented an AI Anime Avatar Generation feature that transforms input images into anime-style avatars using OpenCV and Matplotlib.
Developed functions to load images, apply Gaussian blurring, adaptive thresholding, and bilateral filtering for achieving smooth edges and vibrant colors.
Enhanced the output images' vibrancy and smooth shading transitions to closely mimic traditional anime art styles.
Included functionality to display the original and transformed images side by side for easy comparison.
Added documentation files, including README.md, requirements.txt, and LICENSE.md, to provide project details, dependencies, and licensing information.

Screenshots

Original
Updated

Screenshot of model implementation :

@UppuluriKalyani kindly add level2 as its fixing level2 issue #495 and this matches with that difficulty level.


Report an error when js-waku is running over http and hence cannot connect via WSS

This is a change request
Problem
When a js-waku app is ran on HTTP (not HTTPS) and also not localhost connection-manager will silently fail to connect to any node.
This is visible when debug:waku* is enabled, but not in "production" setting
waku:error:connection-manager Error dialing peer 16Uiu2HAmSve7tR5YZugpskMv2dmJAsMUKmfWYEKRXNUxRaTCnsXV - Error occurred during XX handshake: Error occurred while verifying signed payload: TypeError: Cannot read properties of undefined (reading 'digest') +0ms

Proposed Solutions
Surface an error/warning about not using HTTPS and hence not being able to connect to any peers to the dev/user
Notes

Let's confirm that full nodes advertise ws connection and not only wss.
In nwaku-compose - https://github.com/waku-org/nwaku-compose/blob/8abb2a474d9bf9f9acc5c3a8bc03c4eda015551d/run_node.sh#L36
Todo:

make sure local development over HTTP works;
double check that ws connection can be established if advertised by node;
if problem persists with establishing connection from http to ws we should add warning;


Helm Chart update: make executor ClusterRole name unique

Summary
Executor ClusterRole name should be dynamic instead of hard-coded. Without it, you cannot have several namespaced releases of k8up, as it triggers a collision on the executor ClusterRole name and fails.
Checklist
For Helm Chart changes

[ ] Categorize the PR by setting a good title and adding one of the labels:
bug, enhancement, documentation, change, breaking, dependency
as they show up in the changelog
[ ] PR contains the label area:chart
[ ] PR contains the chart label, e.g. chart:k8up
[ ] Variables are documented in the values.yaml using the format required by Helm-Docs.
[ ] Chart Version bumped if immediate release after merging is planned
[ ] I have run make chart-docs
[ ] Link this PR to related code release or other issues.

Hi @bgounon
Thank you for the PR!
I like the idea, but just making the name dynamically based on the fullname could also break single installations, if a non-default name is specified.
Instead, you could add a new value under rbac like disableClusterRole. Then the first k8up install can install the clusterrole and all subsequent ones don't.
Another approach could be to disable the rbac completely and deploy the ClusterRoles separately, via the value rbac.create.


[BUG] Azure Static Web Apps Deploy Previews for Dependabot (or forks) updates are not working

Azure Static Web Apps Deploy Previews for Dependabot updates are not working:
Similar issue: https://github.com/Azure/static-web-apps/issues/788
To Reproduce
Steps to reproduce the behavior:

Go to https://github.com/trimble-oss/modus-web-components/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-desc+author%3Aapp%2Fdependabot
see Azure GitHub Action is failing with

Expected behavior
It should complete without error
Screenshots

hmm, I'm seeing this in all PRs

hmm, I'm seeing this in all PRs - it may have to do with it being on a forked repo

Yep, I’ll look into this some more. I always make PRs on a branch from the repo (not a fork) and they work fine that way. Honestly, Netlify was much easier :-(

@coliff okay sounds good, let me know what you find!

@coliff have you gotten a change to look into this again? Do we still need the Azure preview if we can preview with Netlify?

Hi, I had this same issue and adding the deployment token as a Dependabot-specific secret fixed it (Settings -> Security -> Secrets -> Dependabot).

Hi, I had this same issue and adding the deployment token as a Dependabot-specific secret fixed it (Settings -> Security -> Secrets -> Dependabot).

awesome tip! thanks! I'll give that a try

thanks @danielwoodhead - got it working for Dependabot.

@coliff can you please close the issue if it is fixed ?

@coliff @enowak1031 should we keep this issue to find an alternative for using Christian's Netlify license?

I think this issue should be kept open. It's possible that a contributor could find this issue and know of a fix.

We've migrated all Deploy Previews to Netlify now so I can close this.


added JAVA_HOME environment variable

Fixes #27

I think this should be /usr/lib/jvm/java-VERSION-openjdk-amd64/jre for all cases.
I just ran this test program against all the combinations:
public class Test {
    public static void main(String... argz) {
        System.out.println(System.getProperty("java.home"));
    }
}

With that file as Test.java, I did the following:
docker build --rm -v "$(pwd):/work" -w /work java:6-jdk javac Test.java
for version in 6 7 8; do for flavor in jdk jre; do \
    docker run --rm -v $(pwd):/work -w /work java:$version-$flavor java -cp . Test; done; done

This was the output:
/usr/lib/jvm/java-6-openjdk-amd64/jre
/usr/lib/jvm/java-6-openjdk-amd64/jre
/usr/lib/jvm/java-7-openjdk-amd64/jre
/usr/lib/jvm/java-7-openjdk-amd64/jre
/usr/lib/jvm/java-8-openjdk-amd64/jre
/usr/lib/jvm/java-8-openjdk-amd64/jre

Based on this the path is different for JDK and JRE

Perhaps you're right, though I'm not sure that SO link counts for much.
Looking at Tomcat's catalina.sh as an example, it actually looks for both JAVA_HOME and JRE_HOME, preferring JRE_HOME (cf. https://github.com/apache/tomcat/blob/7cf92c21bde40bc9d3b9145e7a6781ab6b591d53/bin/catalina.sh#L47-L52 and https://github.com/apache/tomcat/blob/7cf92c21bde40bc9d3b9145e7a6781ab6b591d53/bin/setclasspath.sh)

ok so there is a difference between JAVA_HOME and JRE_HOME interesting.
I'm also not very familiar with JAVA so I'm not sure about the differences.
IMHO relying on JRE_HOME seems better. What do you think?

I don't think that JRE_HOME is something that belongs in this image. It's not anywhere near as common as JAVA_HOME.

As I said I'm not very familiar with JAVA so I'm not sure what is more common or where the differences are.
So should I change the PR to use /jre in all cases?

I'm actually not sure.
What was the use case you were trying to address by adding JAVA_HOME? I know it's pretty common, but did you actually have an app that wouldn't work without it?

I described my my situation in #27
I had to edit my default Java keystore to add a certificate.
And everything I found was just referencing to $JAVA_HOME for the location and this was not defined.
So not that it was not working without the variable but I had to go manually into the container to figure out the location.

I just noticed that openjdk-N-jre-headless includes this file a file called JAVA_HOME that looks like this:

                         JAVA_HOME

Legacy use of JAVA_HOME
As of the latest versions of java-common there really should
be no need for JAVA_HOME as Java(TM) should be completely
integrated with your Debian GNU/Linux system.

Selecting OpenJDK 7 as the default Java(TM) implementation
Please see README.alternatives for instructions on how
to make OpenJDK 7 executables and man pages the default on your system.

cf. docker run --rm java:8 find -name JAVA_HOME -exec cat {} \;

So so should I update my PR or should we consider JAVA_HOME as legacy?

I think you should update your PR, but I'm personally +0 on this change.
Perhaps just documenting JAVA_HOME would be enough.

I think having JAVA_HOME will help; it can't really hurt anything.  I agree that the /jre is not needed; when I tested single mode hadoop, it only needed the /usr/lib/jvm/java-7-openjdk-amd64 even when using the jre image.

ok I've updated the PR

LGTM, @tianon?

I'm also totally +0 on whether this should be a code change or a docs change, especially since it's a legacy thing.
What I'm definitely -1 on is having it be hard-coded in each Dockerfile like this with no verification that it's correct.
I think what I'd be more comfortable with is having some way to verify that the value is correct after our install, and having the whole build bail if it's not:
ENV JAVA_VERSION 6b34
ENV JAVA_DEBIAN_VERSION 6b34-1.13.6-1~deb7u1

RUN apt-get update && apt-get install -y openjdk-6-jdk="$JAVA_DEBIAN_VERSION" && rm -rf /var/lib/apt/lists/*

ENV JAVA_HOME /usr/lib/jvm/java-6-openjdk-amd64
RUN set -x && [ "$JAVA_HOME" = "$(readlink -f "$(dirname "$(readlink -f "$(which java)")")/../..")" ]

Example:
$ for v in 6 7 8; do for a in $v-jre $v-jdk; do { echo "FROM java:$a"; echo "ENV JAVA_HOME /usr/lib/jvm/java-$v-openjdk-amd64"; echo 'RUN set -x && [ "$JAVA_HOME" = "$(readlink -f "$(dirname "$(readlink -f "$(which java)")")/../..")" ]'; } | docker build -; done; done
Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon
Step 0 : FROM java:6-jre
 ---> bf4469de640a
Step 1 : ENV JAVA_HOME /usr/lib/jvm/java-6-openjdk-amd64
 ---> Running in 1a03bea755dc
 ---> 124dbafcd548
Removing intermediate container 1a03bea755dc
Step 2 : RUN set -x && [ "$JAVA_HOME" = "$(readlink -f "$(dirname "$(readlink -f "$(which java)")")/../..")" ]
 ---> Running in f92e985520a5
+ which java
+ readlink -f /usr/bin/java
+ dirname /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java
+ readlink -f /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/../..
+ [ /usr/lib/jvm/java-6-openjdk-amd64 = /usr/lib/jvm/java-6-openjdk-amd64 ]
 ---> a3c08eeff205
Removing intermediate container f92e985520a5
Successfully built a3c08eeff205
Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon
Step 0 : FROM java:6-jdk
 ---> ac9411624983
Step 1 : ENV JAVA_HOME /usr/lib/jvm/java-6-openjdk-amd64
 ---> Running in 875657e67ecd
 ---> 1af4b405173f
Removing intermediate container 875657e67ecd
Step 2 : RUN set -x && [ "$JAVA_HOME" = "$(readlink -f "$(dirname "$(readlink -f "$(which java)")")/../..")" ]
 ---> Running in f98017de94cc
+ which java
+ readlink -f /usr/bin/java
+ dirname /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java
+ readlink -f /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/../..
+ [ /usr/lib/jvm/java-6-openjdk-amd64 = /usr/lib/jvm/java-6-openjdk-amd64 ]
 ---> 5b46b59fa73d
Removing intermediate container f98017de94cc
Successfully built 5b46b59fa73d
Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon
Step 0 : FROM java:7-jre
 ---> d0b904b99306
Step 1 : ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64
 ---> Running in 185d27736972
 ---> d0d84a069215
Removing intermediate container 185d27736972
Step 2 : RUN set -x && [ "$JAVA_HOME" = "$(readlink -f "$(dirname "$(readlink -f "$(which java)")")/../..")" ]
 ---> Running in 08526a5df2bb
+ which java
+ readlink -f /usr/bin/java
+ dirname /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
+ readlink -f /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/../..
+ [ /usr/lib/jvm/java-7-openjdk-amd64 = /usr/lib/jvm/java-7-openjdk-amd64 ]
 ---> d28718162603
Removing intermediate container 08526a5df2bb
Successfully built d28718162603
Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon
Step 0 : FROM java:7-jdk
 ---> 513c90950366
Step 1 : ENV JAVA_HOME /usr/lib/jvm/java-7-openjdk-amd64
 ---> Running in ad67dbc6013b
 ---> e893fcd7d191
Removing intermediate container ad67dbc6013b
Step 2 : RUN set -x && [ "$JAVA_HOME" = "$(readlink -f "$(dirname "$(readlink -f "$(which java)")")/../..")" ]
 ---> Running in 4cdb109cf11f
+ which java
+ readlink -f /usr/bin/java
+ dirname /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
+ readlink -f /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/../..
+ [ /usr/lib/jvm/java-7-openjdk-amd64 = /usr/lib/jvm/java-7-openjdk-amd64 ]
 ---> b906c66c2670
Removing intermediate container 4cdb109cf11f
Successfully built b906c66c2670
Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon
Step 0 : FROM java:8-jre
 ---> 7aaf9421464d
Step 1 : ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
 ---> Running in 9547bbd478ef
 ---> 43cca47c38c6
Removing intermediate container 9547bbd478ef
Step 2 : RUN set -x && [ "$JAVA_HOME" = "$(readlink -f "$(dirname "$(readlink -f "$(which java)")")/../..")" ]
 ---> Running in 326a591e149a
+ which java
+ readlink -f /usr/bin/java
+ dirname /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
+ readlink -f /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/../..
+ [ /usr/lib/jvm/java-8-openjdk-amd64 = /usr/lib/jvm/java-8-openjdk-amd64 ]
 ---> 8e6183c64018
Removing intermediate container 326a591e149a
Successfully built 8e6183c64018
Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon
Step 0 : FROM java:8-jdk
 ---> ed632caf877b
Step 1 : ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
 ---> Running in 4f291a67a9b0
 ---> a6193c9085a5
Removing intermediate container 4f291a67a9b0
Step 2 : RUN set -x && [ "$JAVA_HOME" = "$(readlink -f "$(dirname "$(readlink -f "$(which java)")")/../..")" ]
 ---> Running in b552769a249a
+ which java
+ readlink -f /usr/bin/java
+ dirname /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
+ readlink -f /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/../..
+ [ /usr/lib/jvm/java-8-openjdk-amd64 = /usr/lib/jvm/java-8-openjdk-amd64 ]
 ---> 24cfc8d56faa
Removing intermediate container b552769a249a
Successfully built 24cfc8d56faa

@tianon I also thought about setting the env variable via the command you used but as the Java version itself is hardcoded it is, in my opinion, it is also ok to hardcode the path to JAVA_HOME and it would be less cryptic.
But I can't give any real strong arguments beside my own opinion, so I would be ok with both solutions.

@soupdiver, it seems @tianon was just adding a RUN line that would fail if the JAVA_HOME did not match the installed java, so that there would be a failsafe protecting us from ourselves.

Would it be sufficient for that check to be in update.sh?

Or in a test in official-images?

@yosifkit Oh, sorry totally overlooked that

I am not sure where we would want it.

update.sh: it could autofill the line in the Dockerfile, but would spin up more containers.
tests: nice to have more tests, but it is late in the process to find this possible "bug"

in the Dockerfile, easy, but adds unnecessary layer for users


llvm-4: small improvements on Darwin and elsewhere

Split outputs because there's no point in keeping a reference to Python and it causes trouble during the Darwin stdenv bootstrap. There's also an unnecessary dependency on LLVM in libc++ which causes us to rebuild LLVM several more times than necessary during bootstrap, and an awkward
dependency on XPC in the TSAN that we turn off.
I'm getting an internal g++ error when I try to compile clang 4 with this on NixOS. Can someone give it a go on their machine and see if it's just me? I can't see how I'd cause the internal error with the changes I made... Turns out the internal error was due to being memory-constrained. It worked fine on another machine with more memory.
Motivation for this change
This is in preparation for using LLVM 4 in the Darwin stdenv and by default across nixpkgs.
Things done

[ ] Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)

[ ] NixOS
[x] macOS
[ ] Linux

[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Trying this on NixOS :
$ nix-shell -p nox --run "nox-review -k pr 24014"
[...]
error: while querying the derivation named ‘freecell-solver-4.8.0’:
while evaluating the attribute ‘nativeBuildInputs’ of the derivation ‘freecell-solver-4.8.0’ at /home/demo/.nox/nixpkgs/pkgs/games/freecell-solver/default.nix:8:3:
while evaluating the attribute ‘propagatedNativeBuildInputs’ of the derivation ‘perl-Task-FreecellSolver-Testing-v0.0.10’ at /home/demo/.nox/nixpkgs/pkgs/development/perl-modules/generic/default.nix:28:5:
while evaluating the attribute ‘propagatedNativeBuildInputs’ of the derivation ‘perl-Task-Test-Run-AllPlugins-0.0105’ at /home/demo/.nox/nixpkgs/pkgs/development/perl-modules/generic/default.nix:28:5:
while evaluating the attribute ‘propagatedNativeBuildInputs’ of the derivation ‘perl-Test-Run-Plugin-ColorFileVerdicts-0.0124’ at /home/demo/.nox/nixpkgs/pkgs/development/perl-modules/generic/default.nix:28:5:
while evaluating ‘chooseDevOutputs’ at /home/demo/.nox/nixpkgs/lib/attrsets.nix:463:22, called from /home/demo/.nox/nixpkgs/pkgs/stdenv/generic/default.nix:232:32:
attempt to call something which is not a function but a list, at /home/demo/.nox/nixpkgs/pkgs/top-level/perl-packages.nix:13239:29
Traceback (most recent call last):
  File "/nix/store/rivajj7qdr1hqrhxpyvz10m52mav8z33-nox-0.0.5/bin/.nox-review-wrapped", line 12, in <module>
    sys.exit(cli())
  File "/nix/store/qqmy5wgjcpywa59hp7div73hqz7s1yqh-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/nix/store/qqmy5wgjcpywa59hp7div73hqz7s1yqh-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/nix/store/qqmy5wgjcpywa59hp7div73hqz7s1yqh-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/nix/store/qqmy5wgjcpywa59hp7div73hqz7s1yqh-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/nix/store/qqmy5wgjcpywa59hp7div73hqz7s1yqh-python3.5-click-6.6/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/nix/store/qqmy5wgjcpywa59hp7div73hqz7s1yqh-python3.5-click-6.6/lib/python3.5/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/nix/store/rivajj7qdr1hqrhxpyvz10m52mav8z33-nox-0.0.5/lib/python3.5/site-packages/nox/review.py", line 139, in review_pr
    attrs = differences(packages_for_sha(base),
  File "/nix/store/yd7z2nrjn11q5ym654528k98hi3j47qc-python3.5-dogpile.cache-0.5.4/lib/python3.5/site-packages/dogpile/cache/region.py", line 1013, in decorate
    should_cache_fn)
  File "/nix/store/yd7z2nrjn11q5ym654528k98hi3j47qc-python3.5-dogpile.cache-0.5.4/lib/python3.5/site-packages/dogpile/cache/region.py", line 640, in get_or_create
    async_creator) as value:
  File "/nix/store/an52v4cpr7l1cxwwzh69vwyzlpzmj6g4-python3.5-dogpile.core-0.4.1/lib/python3.5/site-packages/dogpile/core/dogpile.py", line 158, in __enter__
    return self._enter()
  File "/nix/store/an52v4cpr7l1cxwwzh69vwyzlpzmj6g4-python3.5-dogpile.core-0.4.1/lib/python3.5/site-packages/dogpile/core/dogpile.py", line 98, in _enter
    generated = self._enter_create(createdtime)
  File "/nix/store/an52v4cpr7l1cxwwzh69vwyzlpzmj6g4-python3.5-dogpile.core-0.4.1/lib/python3.5/site-packages/dogpile/core/dogpile.py", line 149, in _enter_create
    created = self.creator()
  File "/nix/store/yd7z2nrjn11q5ym654528k98hi3j47qc-python3.5-dogpile.cache-0.5.4/lib/python3.5/site-packages/dogpile/cache/region.py", line 612, in gen_value
    created_value = creator()
  File "/nix/store/yd7z2nrjn11q5ym654528k98hi3j47qc-python3.5-dogpile.cache-0.5.4/lib/python3.5/site-packages/dogpile/cache/region.py", line 1009, in creator
    return fn(*arg, **kw)
  File "/nix/store/rivajj7qdr1hqrhxpyvz10m52mav8z33-nox-0.0.5/lib/python3.5/site-packages/nox/nixpkgs_repo.py", line 90, in packages_for_sha
    return packages(repo.path)
  File "/nix/store/rivajj7qdr1hqrhxpyvz10m52mav8z33-nox-0.0.5/lib/python3.5/site-packages/nox/nixpkgs_repo.py", line 81, in packages
    universal_newlines=True)
  File "/nix/store/wkl9lz79bdq503b3a22hpjblxj61r081-python3-3.5.3/lib/python3.5/subprocess.py", line 316, in check_output
    **kwargs).stdout
  File "/nix/store/wkl9lz79bdq503b3a22hpjblxj61r081-python3-3.5.3/lib/python3.5/subprocess.py", line 398, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['nix-env', '-f', '/home/demo/.nox/nixpkgs', '-qaP', '--out-path', '--show-trace']' returned non-zero exit status 1

@c0bw3b that sounds like someone broke evaluation somewhere?

Yes. I get the same error when running nox-review on PR #23161 actually. :/

Yeah, looks like @7c6f434c fixed it in https://github.com/NixOS/nixpkgs/commit/d221e40fd7f3beb8295b628122cd7e59beb09505, thanks!

Rerun nox-review here and it completed successfully.
/nix/store/m9pnn0bnaaigjln132d2kh8xsyhp10vl-clang-wrapper-4.0.0
Result in /run/user/1000/nox-review-tj77hy_q
total 0
lrwxrwxrwx 1 demo users 63 18 mars  18:48 result -> /nix/store/m9pnn0bnaaigjln132d2kh8xsyhp10vl-clang-wrapper-4.0.0

Checking the resulting derivation :
[demo@nixos:~]$ tree -ah /nix/store/m9pnn0bnaaigjln132d2kh8xsyhp10vl-clang-wrapper-4.0.0
/nix/store/m9pnn0bnaaigjln132d2kh8xsyhp10vl-clang-wrapper-4.0.0
├── [4.0K]  bin
│   ├── [  64]  as -> /nix/store/k8c23rwbrril65rmm7z1036h2makk012-binutils-2.28/bin/as
│   ├── [   7]  c++ -> clang++
│   ├── [   5]  cc -> clang
│   ├── [5.7K]  clang
│   ├── [5.8K]  clang++
│   ├── [5.7K]  cpp
│   ├── [5.5K]  ld
│   ├── [5.6K]  ld.bfd
│   └── [5.6K]  ld.gold
└── [4.0K]  nix-support
    ├── [1.7K]  add-flags.sh
    ├── [2.2K]  add-hardening.sh
    ├── [  63]  cc-cflags
    ├── [  63]  cc-ldflags
    ├── [  80]  dynamic-linker
    ├── [ 233]  libc-cflags
    ├── [  61]  libc-ldflags
    ├── [  96]  libc-ldflags-before
    ├── [  56]  orig-cc
    ├── [  55]  orig-libc
    ├── [  59]  orig-libc-dev
    ├── [   1]  propagated-native-build-inputs
    ├── [ 173]  propagated-user-env-packages
    ├── [1.4K]  setup-hook
    └── [1.2K]  utils.sh

2 directories, 24 files

[demo@nixos:~]$ /nix/store/m9pnn0bnaaigjln132d2kh8xsyhp10vl-clang-wrapper-4.0.0/bin/clang --version
clang version 4.0.0 (tags/RELEASE_400/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /nix/store/m0wld6jq7xw2cnchrxijyri0xw44svpq-clang-4.0.0/bin

@c0bw3b thanks for checking! The build also ran fine on my other Linux machine, so I'll merge.

LLVM 4 as default Darwin stdenv :+1:
In my research branch I've nuked the same to reduce runtime deps on python and libxml2 (c-index-test).  I think you gave me the idea O:).
Probably should move $out/share/scan-view as well? Contains various python scripts....

Not runtime dependency (as discussed on IRC), but perhaps the supporting python bits for scan-view should be moved as well?
$ git rev-parse HEAD
4707ac31f1a8406aaa6b8d892512fe59388349a5
$ nix-build -A clang_4.cc
/nix/store/m0wld6jq7xw2cnchrxijyri0xw44svpq-clang-4.0.0
$ ls -l result
lrwxrwxrwx 1 will users 55 Mar 18 14:19 result -> /nix/store/m0wld6jq7xw2cnchrxijyri0xw44svpq-clang-4.0.0
$ tree -ah ./result/share
./result/share
├── [4.0K]  clang
│   ├── [1.4K]  clang-format-bbedit.applescript
│   ├── [7.0K]  clang-format.el
│   ├── [ 17K]  clang-include-fixer.el
│   └── [3.1K]  clang-rename.el
├── [4.0K]  man
│   └── [4.0K]  man1
│       └── [3.7K]  scan-build.1.gz
├── [4.0K]  scan-build
│   ├── [1.3K]  scanview.css
│   └── [ 16K]  sorttable.js
└── [4.0K]  scan-view
    ├── [ 318]  bugcatcher.ico
    ├── [ 18K]  FileRadar.scpt
    ├── [   0]  GetRadarVersion.scpt
    ├── [8.0K]  Reporter.py
    ├── [ 25K]  ScanView.py
    └── [5.9K]  startfile.py

5 directories, 13 files


Half turn as 180 degree single rotation

Currently double layer moves like F2, L2, M2 are performed in two single 90 degree rotations. This can be done in one 180 degree rotation.

I'm trying to keep with original terminology. F2 is a half turn.


exported GCT does not contain all metadata tracks

I'm using write_gct() to export a GCT object with 8 tracks of column metadata.  When I open the exported file, only one metadata track is displayed (in additional to column names), even though the dimensions in the file denote 8.
Inspecting the GCT object itself shows that all 8 tracks are there in cdesc.  Any idea why this might be happening?  Thanks!

Hi @mhk658, could you please include some code to indicate how your GCT object is being constructed? That will help me debug.
Thanks a lot,
Ted

Hi Ted -- I'm using the following code, from a data.frame filtered.data and a annotation.matrix tab-delimited table with the first column named "id" containing sample names :
mat <- data.matrix(filtered.data[,(metadata.cols+1):(ncol(filtered.data))]) #data only
cdesc <- read.delim(annotation.matrix, header = TRUE, sep = '\t', colClasses = 'character')
rdesc <- data.frame(filtered.data[,1:metadata.cols]) #metadata only

colnames(mat) <- cdesc[,1]
rownames(mat) <- filtered.data[,1]

for(col in c(1:ncol(rdesc))){
  rdesc[,col] <- as.character(rdesc[,col])
}

output.GCT <- new("GCT", mat=mat, rdesc=rdesc, cdesc=cdesc)
write_gct(output.GCT, paste(output.file.name, '.gct', sep = ''), appenddim = FALSE)

I've coerced everything in cdesc and rdesc to characters in hopes that it would help, but it didn't.  If I print(output.GCT), it appears to look like it says it should at https://github.com/cmap/cmapR/blob/master/cmapR_tutorial.ipynb.  The issue only appears in the exported GCT, and only in the column metadata (row metadata is fine) Please let me know if any other info would be helpful!
Thanks,
Harry

Thanks Harry. Are there any NA values in cdesc? Would it be possible to share the underlying table?

There were some NA values, but I've changed them and the issue still persists.  I've attached the table used to create cdesc.
annotation_matrix.txt

Hi @mhk658, thanks for sending this file. I was able to reproduce the issue. Turns out there was a small bug in the write_gct code causing the problem. I pushed a fix with this commit: 18293e6
Could you please try pulling the updated code and seeing that it works for you?
Thanks a lot,
Ted

Hi Ted -- the issue no longer occurs with the updated code.  Thanks for the help!


Support manager address changes

One of the things necessary to unblock full support for swarm mode in Docker for Mac/Windows is support for dynamic IP addresses on managers. Users of Docker for Mac/Windows are much more likely to be running in an environment like a laptop with WiFi. Of course, this will also be useful in other setups.
This PR begins to explore allowing address changes at the raft level. Each manager keeps track of the last source address it saw in RPCs from each other manager. When a send fails, and the most recently seen source address differs from the address we're sending to, and the observed address passes a health check, we switch to that address. Also, if this happens on the leader, the new address will be written to raft. Nodes are protected from election loops during an address change by ignoring vote requests unless the sender passes a health check.
One nice property of this approach is that address changes are recognized automatically, and don't need the admin or the manager whose address changed to specifically request an address change.
I haven't tested this yet, so the next step will be to write a unit test that confirms it works.
There will probably need to be libnetwork changes to fully support dynamic addresses.
Also, this does not address the case where quorum is lost (i.e. all nodes restarting and changing their addresses at the same time). We will probably need some kind of variant of --force-new-cluster to deal with that.
cc @justincormack @LK4D4

Addressed review comments. Tried working on a unit test for this, but I don't see a good way. Both the bind address and source address for outgoing connections have to change. The unit tests just use <IP_ADDRESS>.

Feedback addressed. Also squashed the commits, because new changes made it hard to maintain the old commit structure.
I tested this with 3 VMs. I set up a 3-manager swarm, then I restarted one of the follower VMs, which caused it to get a new IP address. When it came back up, it was able to participate in the swarm. I also restarted the leader's Docker engine a few times to trigger leader changes, and confirmed that things work when the node with the changed address becomes the leader. I also confirmed that restarting all engines at once works after the address change.
I'm also going to test adding new nodes to the cluster after the address change, to make sure they pick up the correct address and not the original one.

Adding a new node after the address change works. No problems when the new node becomes the leader, or when the one whose address changed becomes the leader.

Docker integration tests are passing with this - except for TestSwarmTaskListFilter, which is also failing on docker master for me (strange, CI seems fine).

LGTM
ping @aluzzardi @dongluochen @nishanttotla

Going to merge this, because nobody appeared to review it.


getting coverage data out of the database

Once I've loaded coverage data into the database, is there a way to get it out by a certain region? Or should I roll my own SQL queries?

Yeah, chanjo itself isn't terribly helpful in getting data out of the database once imported. I've intended to build in some basic queries into the CLI but haven't gotten around to it.
If you figure out some interesting SQL queries I would be super happy if you shared them in an issue or blog post or whatever!

Closing this in favour of already covered in #118


Add github disc

Okay, i make change and it isn't in docs.
discussions on github are cool and i added these!

There is a typo in this which prevents us from accepting it in its current state, but I’m leaning against accepting it even if the typo is fixed. This isn’t that impactful an improvement, and you’ve been asked a couple times to try contributing a fix for an existing, known issue, rather than continuing to try to change something else, and you seem to be disregarding that request. Could you please focus on getting your first fix for an existing issue accepted, before submitting anymore of these?

Thank you @jab!
It took me a long time to formulate an answer for you.
  Now I will act smarter in open-source projects and I will not be annoying so that people do not think how to get rid of me. Sorry and forgive me again.


UPF becomes unresponsive/sluggish when adding a fair bit of text to a text box that has wrapping.

I have a read only textbox in UPF that is created via
<TextBox Name="chatTextBox" TextWrapping="Wrap" Grid.Row="0" Grid.Column="0" VerticalScrollBarVisibility="Visible" IsReadOnly="True"></TextBox>

I add to it using the code
private void addChatText(string text)
            {
                bool scrolledToBottom = chatTextBox?.VerticalOffset > chatTextBox?.ExtentHeight - chatTextBox?.ViewportHeight - 50;
                chatTextBox?.AppendText(text + "\n");
                if (scrolledToBottom)
                {
                    chatTextBox?.ScrollToEnd();
                }
            }

For my system info,
Manjaro Linux running KDE
Dotnet Core 3.1
When I add a bunch of lines that need wrapped things become VERY slow and problematic entering text, If i destroy the textbox everything becomes fast again, and this doesn't happen without the wrapping.  This is pretty hard to capture in video, but I have made one.  My current workaround is to not use text wrapping, which is fine.
https://www.youtube.com/watch?v=FzdgWf5BoyM&feature=youtu.be

I think TextBox doesn't support virtualization.
What is Virtualization?
Virtualization technique in WPF improves the rendering performance of UI elements. By applying virtualization, the layout system ensures that only the visible items of a container are rendered on the screen. For example, a list control may have thousands of items but virtualization will reduce the rendering to the visible items only.
The Solution to this is to use the ListBox control.

I think in the somewhat short run you're probably right and I should convert it, it also would make certain styling far easier, I'll throw it on my todo list.

What is happening here is that the ListBox control has an ItemSource property you can see i created a dummy list named List in the ViewModel class and assigned it. The second necessary property is the  ViewModelType it needs two parameters. The first one is the  namespace (the namespace plus the class name) the second parameter is the actual solution name or assembly name call you as you like it.

If you have a simple string array like this

You can leave out the ItemTemplate part and end up with this.

.

I would expect appending text to get a bit slow before very long, since the entirety of the TextBox's content is laid out as a single unit. Ultraviolet isn't a text editor, so the reality is that its multi-line TextBox control is probably never going to work well with massive strings of dynamically changing text. However, there's probably room for further optimization here, so I'll look into it.
What does surprise me is that it gets so slow when just rendering the text. I suspect I know what the problem is, though, so I'll look into speeding that up for you.
I agree that for your use case, something other than TextBox is probably the correct choice, potentially even something custom.

Commit c9d93652c502449e0512fde09149cb983cb61038 changes how text inside of a TextEditor control is clipped, which should improve rendering performance for very long strings.

I've committed some micro-optimizations for text layout, but now that I've spent some time looking into this, I think that properly supporting very large strings of text is going to require fundamentally rearchitecting the text layout engine. Reworking fonts and text layout is something I've wanted to do for a while, but it's not currently a priority, I'm afraid.
For now, my recommendation is to use TextBox and PasswordBox only for relatively short strings, and to prefer custom solutions for use cases which require large amounts of text or text which changes frequently.


Fix permission error in mnist.py

When I ran dataset/mnist.py to just download mnist data set, I got an error which said permission error.
~/Downloads/deep-learning-from-scratch-master/dataset
✔ python3 mnist.py
Downloading train-images-idx3-ubyte.gz ...
Traceback (most recent call last):
  File "mnist.py", line 128, in <module>
    init_mnist()
  File "mnist.py", line 75, in init_mnist
    download_mnist()
  File "mnist.py", line 42, in download_mnist
    _download(v)
  File "mnist.py", line 37, in _download
    urllib.request.urlretrieve(url_base + file_name, file_path)
  File "/usr/local/Cellar/python3/3.4.0/Frameworks/Python.framework/Versions/3.4/lib/python3.4/urllib/request.py", line 188, in urlretrieve
    tfp = open(filename, 'wb')
PermissionError: [Errno 13] Permission denied: '/train-images-idx3-ubyte.gz'

When dataset/mnist.py is used from other file like sample code in the book, this error never happened.
But when I ran this file,  os.path.dirname(__file__) returns empty and it makes dataset_dir empty. This causes a permission error for most of uses because save_file will be created at root path.
To avoid this, we have to convert current file path as absolute path then take its parent directory path.
This file seems to be written to be ran by itself (from if __name__ == '__main__': code), I'm thinking this change can help others who want to just download mnist data set.

Cool, thanks!


Question: How/where does this compare to EconML

I've been experimenting with DoWhy recently and really enjoy the structure. I noticed some other libraries out there, such as EconML (mentioned here) and Uber's CausalML. I'll focus on EconML for this discussion, particularly because I see some recent PRs that brought in EconML's CATE estimator.
I'd just like to confirm the differences and overlap between DoWhy and EconML. Please let me know if I understand this correctly.
My Attempt at Comparisons
Let's start with DoWhy's structure:

model (make assumptions),
identify (find what to estimate given the assumptions),
estimate
refute (sensitivity and robustness checks).

1. Model

DoWhy: Provides ability to explicitly define complex causal graphs. Or alternately, (though not preferred?), define common confounders to assess.
EconML: I didn't see a means to define the causal graph other than through variable definition Y, T, X, W, Z

2. Identify

DoWhy: Hunts down causal effects using graph analysis and do-calculus
EconML: Not sure I saw this explicitly in the library?

3. Estimate

DoWhy: Backdoor, instrumental variables, and most recently do-sampling (Which is aweeesoome!)
EconML: Seems to be where EconML currently shines. There's a whole slew of approaches, many implementing approaches very recent ML research papers

4. Refute

DoWhy: Heavy focus on model validation with several methods
EconML: I didn't notice anything explicit.

Where they overlap
It seems to me they overlap most heavily in the Estimation section. That's where I saw some references here on the roadmap in bringing in EconML calls. Is that correct?
Question on estimators and terminology
I see EconML has the following called out for estimators:

Potential Outcomes
Structural Equations
CATE

What is the difference between the estimators listed above and the ones built into DoWhy? I was just struggling to connect the dots there.
Thank you.

thanks @tszumowski for starting this discussion. Yes, you are right---DoWhy and Econml overlap only in the estimate section. DoWhy implements the full process of causal reasoning including model, identify, estimate and refute. In comparison, Econml implements only the estimate step.
In designing DoWhy, we kept a focus on the "ideal" process of doing a causal analysis, which includes identification and more importantly, refutation so that modeling assumptions can be tested. The estimators in DoWhy currently are the standard estimators for causal inference. As you rightly point out, EconML has much more advanced estimators for estimating the conditional average treatment effect (CATE). This is why we are implementing an interface in DoWhy so that you can call Econml methods directly from dowhy's estimate function. Here's an experimental Jupyter notebook to see it in action.
For your second question, there are actually two considerations when designing a causal analysis. One is about the modeling framework, and the other is about the target estimand.
Potential outcomes and structural equations are ways to construct a causal model. Another way is to use structural causal model which is based on a graphical model. The differences between these frameworks are often a matter of detail (and of big academic debate). But in practice, both econml and dowhy are compatible with these different ways of expressing a causal model. In DoWhy specifically, we use the structural causal model framework in the identify step, and rely heavily on methods derived from the potential outcomes framework in the estimate step.
The other consideration is the target estimand: do you want an effect for the full population (average treatment effect, ATE) or for a specific population, e.g., conditioned on "Gender=Female", (conditional average treatment effect, CATE). EconML methods are designed to estimate CATE, which is a subject of active research. Most of DoWhy's methods focus on estimating the ATE so far, although we are extending some of the methods to also estimate CATE.
Hope this helps.

Wow. @amit-sharma thank you for the fantastic summary.  I'm only recently trying to ramp up on causal analysis, coming from a stats/ML/bayesian background. I am noticing what you mean by "big academic debate" as I was getting a bit lost in the various different methods. This clarified a lot for me.
Regarding the EconML integration, I ran into an issue when trying to run that notebook. I'll create a separate issue for that. Otherwise, this ticket can be closed. Thank you.

we use the structural causal model framework in the identify step

What does it mean by this? Is the identify step used as input for the estimate step? How are we integrating the prior into the estimation?


Excluding vulnerability sonatype-2022-6438 {patch}

Excluding sonatype-2022-6438 as DoS should not be an issue while using the library as intended.


Server PvP is still 1.8 after disabling plugin

Title says it, I deleted the plugin and restarted. The hit delay is still removed?

Were you able to resolve your issue?

Yes I reset the playerdata for all players, luckily it was only kitpvp that I was testing it on.


Support for error handlers registered using decorators

It would be interesting to support the same way to register error handlers using decorators, like in Flask:

https://twitter.com/testdrivenio/status/1384398245648994307

@RobertoPrevato Hey,
Sounds interesting, if no one doesn't work on this, I'll happy to take it, once I've got a free time.

Hi @myusko, yes please, I would appreciate that very much.
The application class has a property, exceptions_handlers used to define exception handlers by status or by exception class.
Inside the file test_application.py you can find some examples of exception handlers defined by status code (test_application_http_exception_handlers) and  by exception class (e.g. test_application_user_defined_exception_handlers)

@RobertoPrevato Thank you for the information!


Developer tutorial on building api-driven features with SilverStripe 4

Problem Description
Modern web sites and apps face increasing expectations among their stakeholders to be fast, robust, and cross-platform. This has lead to the idea of Progressive Web Applications (PWAs). As a collection of technical and design solutions, their aim is to feel like native software. Critical to this effort is a scalable API layer.SilverStripe 4 provides all the tooling necessary to develop a powerful API, and deliver PWAs on the web, mobile, or as a service. However, supporting developer documentation is lacking. This proposal aims to encourage developers to build their applications using leading-edge technologies and best practices, through step-by-step tutorials and technical documentation.
Key features to solve the problem

Develop a demo PWA to provide hands-on examples for developers.
Document high-level approach and guide through implementation through code examples
Create and publish a new recipe for a PWA setup. This could include a headless CMS architecture with predefined routing to help developers get started, a simple GraphQL API, frontend tooling using React/Apollo/Redux, with build chain preconfigured

Key benefits

Decoupled frontends (“headless”) allow agencies to work with vendors that don’t have SilverStripe knowledge
More responsive user interfaces on SilverStripe sites benefit users, both in websites with interactive features and web application

Special considerations

This proposal is well-suited for iterative delivery, with room for more advanced concepts depending on developer interest
Excludes video tutorials.

Some developer feedback on what would be useful: https://groups.google.com/forum/#!topic/silverstripe-dev/Qm3mSE9Z5aQ


[Sidenav] Sidenav does not complete state changes when animations are disabled.

Bug, feature request, or proposal:
Bug
What is the expected behavior?
Sidenav makes state changes without animations.
What is the current behavior?
Sidenav does not complete state changes and some animations still occur.
What are the steps to reproduce?
Add<EMAIL_ADDRESS>to the element containing a sidenav container and attempt to use sidenav.
Providing a Plunker (or similar) is the best way to get the team to see your issue.
https://plnkr.co/edit/v29ARj2yt3vQbeBsEqVf?p=preview
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
@angular/angular 4.3.6
@angular/material 2.0.0-beta.10
At least chrome and firefox
Is there anything else we should know?
The average set of human lungs weigh 840g (1.85lbs) source

updated plunker: https://plnkr.co/edit/k4PlA4rL7G2nqGe0Fxwk?p=preview


PowerCLI sample

We should make a sample that uses a containerized PowerCLI to script against the API.
This article makes me think we should have our binding setting up the equivalent of: %APPDATA%\VMware\credstore\vicredentials.xml

Looks like this is a containerized PowerCLI image: https://hub.docker.com/r/vmware/powerclicore/

So I managed to get something relatively simple working using a hacked up version of the govc sample:
apiVersion: batch/v1
kind: Job
metadata:
  name: vsphere-script
  labels:
    # Make this Job match the binding!
    role: vsphere-job
spec:
  template:
    metadata:
      labels:
        # So it's easier to list the Pods
        role: vsphere-job
    spec:
      restartPolicy: Never
      containers:
      - name: create-tag
        image: vmware/powerclicore
        command: ["pwsh", "-Command"]
        args:
        - |
          # Log into the VI Server
          Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false | Out-Null
          Connect-VIServer -Server ([System.Uri]$env:GOVC_URL).Host -User $env:GOVC_USERNAME -Password $env:GOVC_PASSWORD

          # Get Events and write them out.
          Get-VIEvent | Write-Host

Right now it needs to explicitly connect with the auth because I haven't found a clever way to have this pulled from the environment.  Connect-VIServer seems to support a -SaveCredentials option, which hints at a file we could precreate, but this isn't yet supported on PowerCLI Core :(
One thing we could consider doing it mounting our own Setup-VIBinding.ps1 with a little script that does the above auto-magically.  However, at a minimum I think we should consider cleaning up the environment variables to better align with this use-case, e.g.
          # Log into the VI Server
          Set-PowerCLIConfiguration -InvalidCertificateAction $env:VI_CERT_ACTION -Confirm:$false | Out-Null
          Connect-VIServer -Server $env:VI_SERVER -User $env:VI_USERNAME -Password $env:VI_PASSWORD

... or something.

Here's what we get against vcsim:
Connect-VIServer : Object reference not set to an instance of an object.
At line:3 char:1
+ Connect-VIServer -Server ([System.Uri]$env:GOVC_URL).Host -User $env: ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [Connect-VIServer], NullReferenceException
+ FullyQualifiedErrorId : System.NullReferenceException,VMware.VimAutomation.ViCore.Cmdlets.Commands.ConnectVIServer

Been quite a while since I've tried PowerCLI against vcsim, but one requirement for PowerCLI is that vcsim is listening on port 443: https://github.com/vmware/govmomi/issues/1533#issuecomment-514654925

Yeah, it's on 443 with TLS validation disabled.

ok, I can reproduce and fix Object reference not set to an instance of an object by reverting: https://github.com/vmware/govmomi/pull/1740/files#diff-520495bbef31a0ad52f7d74b057a7568L75
That's something we can fix in vcsim by adding support back for the obsolete service type. But after doing that I see a different error and don't know why yet:
PS /root> Connect-VIServer -Server <IP_ADDRESS> -User user -Password pass
Connect-VIServer : Value cannot be null.
Parameter name: source
At line:1 char:1
+ Connect-VIServer -Server <IP_ADDRESS> -User user -Password pass
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [Connect-VIServer], ArgumentNullException
+ FullyQualifiedErrorId : System.ArgumentNullException,VMware.VimAutomation.ViCore.Cmdlets.Commands.ConnectVIServer

Wasn't obvious from the PowerCLI error, but found the 2nd issue.  Can you review the fixes?  https://github.com/vmware/govmomi/pull/1930

That gets us past the Connect-VIServer 💪
However, the next thing we're currently doing in our sample is Get-VIEvent | Write-Host which gets the following error from vcsim:
Get-VIEvent : 04/13/2020 22:07:55       Get-VIEvent             EventHistoryCollector:session[92e093e6-6a2e-40f4-aeb1-f5a67b8d2720]ed4050db-e2d0-484a-bf71-2aa67d187cd1 does not implement: ResetCollector
At line:6 char:1
+ Get-VIEvent | Write-Host
+ ~~~~~~~~~~~
+ CategoryInfo          : NotSpecified: (:) [Get-VIEvent], MethodNotFound
+ FullyQualifiedErrorId : Client20_InventoryServiceImpl_GetEvent_ViError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetEvent

To save some scrolling: does not implement: ResetCollector

ok, both fixed, I've opened https://github.com/vmware-tanzu/sources-for-knative/pull/46 to add an e2e test and update the docs.  thanks @dougm!!!


Define API

Textrous could benefit a lot from exposing a public REST API and factoring out the web frontend app.
In order to do this it would be nice to have some sort of software diagram or overview of how textrous works and what the requirements of the web app are

Doodled a small "software diagram" on the train

Currently the main components are "Textrous", starting from "index.html" in the root folder and submitting form data to the client*.cgi scripts which connect to gene2Word through a socket on localhost.
The request contains a header int that determines the request type and what kind of operation gene2Word will perform on the rest of the data.
Based on this information I think we xna make a very simple REST API:
POST /api/<version>/geneset/
Where version will be v1. The POST body should contain JSON like this:
{
  "genes": ["A", "B"],
  "operation": "word_vector"
}

Where operation is one of the following:

word_vector
word_vector_p
word_vector_z
word_matrix
phrase_vector
hier
excluded

I don't know yet what these operations do so as we move along we might discover that some of these have to be merged, split or altered.


pull request for e2e test at 11/13/2016 15:15:09

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

optest/docs/updated_at.md
:white_check_mark:Succeeded

Details

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Add Opcodes PUSHT and PUSHF and change the generation of boolean literal values

Summary or solution description
Included two opcodes added in Neo 3.5 to handle boolean values and refactored the opcode generation to use those instead of generating integer value and converting it.
Tests
Rerun all the unit tests

Task linked: CU-864dpjg3u Add Opcodes PUSHT and PUSHF and change the generation of boolean literal values