feat: Add release scripts for streamlined release process (#27)

This PR provides a set of scripts and configuration to simplify the version bumping and
CHANGELOG generation process. The process works for both open-source projects hosted on
Github, or closed-source projects hosted on a private hosting instance such as Gitlab.
This lengthy comment is broken into two major sections: 1) An overview of the current process
and an outline of the problems that it has, some of which are solved by this PR. This is
to remind the reviewer of the current process so they can compare it to the new process.
2) Integration and usage instructions for the new scripts.
Current Process
Initial Release Candidate
Here is an outline of the current version-bumping process, starting with the process for
creating the first release candidate for a new version. This process assumes that the
project has a dependency on standard-version and an NPM script called "release" defined
as: "release": "standard-version".

In the project's root folder, check out a new branch called
prep-v${WHAT_YOU_THINK_THE_NEW_VERSION_WILL_BE}
Run npm run release -- --skip.commit --skip.tag --prerelease=rc

This command calculates the next recommended version and CHANGELOG entries from the
git log's commit messages and outputs them to CHANGELOG.md.
It also bumps the version in package.json and package-lock.json. We don't want to
do that just yet, but there's no way to tell standard-version to skip that step
without breaking CHANGELOG generation. (I can provide more details on why this is
so, as needed. If you're curious, ping me directly.)
Note that this command adds the CHANGELOG entries under a release candidate heading,
e.g. ### 1.3.0-rc.0. We do not include release candidate headings in the final
version, but we do need them for release candidate versions for two reasons: 1) They
are convenient to refer to while the project is in its release candidate phase. 2)
standard-version does not provide a way to generate a list of CHANGELOG entries using
a specific range of commits or tags. It will only ever generate the list from the
previous tagged version to HEAD, so when it comes time to create the CHANGELOG
for the final version, we need to already have the entries from the previous release
candidates.

Get rid of the unwanted changes to package.json and package-lock.json. (I use git checkout package*)
Open CHANGELOG.md and review the new CHANGELOG entries to make sure they look correct.
Commit the new entries with the message: chore: Update CHANGELOG for v${NEW_VERSION}
Run npm run release -- --skip.changelog --prerelease=rc.

This command bumps the version in package.json and package-lock.json, commits
them, and creates a new v${NEW_VERSION} tag.

Push the branch to a remote and open a PR/MR.
Once the PR/MR is reviewed and merged, push the new v${NEW_VERSION} tag.

Subsequent Release Candidates
After the initial release candidate, we may need to create more release candidates before
the final version. If the changes in the new release candidate are only fixes for features
that are in the current version, we may skip CHANGELOG generation for the sake of
simplicity, since we do not want to include CHANGELOG entries for fixes for features
introduced in the current version and follow an abbreviated process:

Check out a new branch, e.g. git checkout -b prep-v1.3.0-rc.1 if the the previous
release candidate was  v1.3.0-rc.0.
npm run release -- --skip.changelog --prerelease=rc
Push the branch and open a PR/MR.
Once the PR/MR is reviewed and merged, push the new v${NEW_VERSION} tag.

In practice, however, we sometimes add new bug fixes that should be added to the CHANGELOG
(not just bug fixes for features in the current version). When that happens, we must
follow the longer process in the "Initial Release Candidate" section again.
Final Version
Follow the process in the "Initial Release Candidate" section, but with these differences:

Omit the --prerelease flag from all commands
After generating the final CHANGELOG heading, delete the previous release candidate
headings and merge/re-organize the entries under their appropriate "Feature", "Bug
Fix", etc. headings. This process is described in more detail in the "Creating the
Final Version" subheading near the end of this comment.

Problems With the Current Process

When you check out the prep-v${NEW_VERSION} branch, you have to already know what the
next version will be if you want to include the version in the branch name. To get the
next version, you either have to look at the git log and figure it out or do a
dry-run of an npm release command to see what the next version is before creating
your branch. Or create a temporary branch name and rename the branch after the version
bumping process is finished.
The command that generates the CHANGELOG also modifies package.json and
package-lock.json before we need it and you have to manually back those changes out
before continuing.
There are a number of CLI flags that you have to pass to each step and it's easy to
forget to pass one and what they're called. It's certainly not beginner-friendly.
If the reviewer catches something you missed in the CHANGELOG changes, you have to
delete the tag and version bump commit, fixup the CHANGELOG, and then re-run the
npm run release -- --skip.changelog --prerelease=rc command to regenerate the version
bump commit and tag.
When creating the final version, you have to manually merge the CHANGELOG entries from
previous release candidate versions into a single final version.

The reviewer will notice later that the scripts introduced in this PR do not solve this
problem. Doing so would have required switching from standard-version to some other
CLI tool, which was out of scope for this PR.

New Process
Goals

Simplify the release process and commands without writing a new/complex release tool right now.
Allow someone to run the release process without having to be familiar with the project itself and
what entries should appear in the CHANGELOG.

The MR that's submitted during this new process is for the CHANGELOG changes only. The MR
should be assigned to the developer who is most familiar with the project and the changes in the
new version.

Here is a set of instructions for adding these scripts to one of our repos, creating a
release candidate version, and creating a final version.
If this process is approved, these instructions will be the basis for another PR to the
silvermine-info repo to document the new release process.
Adding These Release Scripts to a Project

npm i -DE @silvermine/standardization
Ensure that your package.json file has a repository.url field with the
URL to the canonical repo for this project in your Git hosting solution, e.g.
https://github.com/silvermine/event-emitter.git for the @silvermine/event-emitter
project in Github.

This is necessary because standard-version needs to know the URL to your Git hosting
solution so that it can make links to "compare URLs" in the CHANGELOG (e.g. the "9.4.0" link here).
standard-version looks at package.json's repository.url field first, and then
falls back to the origin git remote's URL. Until recently, we've been relying on the origin
remote URL, but since we've started naming our origin remote canonical in some
instances, standard-version can't find a URL to use. standard-version does not allow
you to configure the name of remote it uses to get the URL. Therefore, I suggest we
consistently use the repository.url field in our projects.

Create a symlink to<EMAIL_ADDRESS>in your
project's root folder.
Add these NPM scripts to your package.json file:

"release:prep-changelog"<EMAIL_ADDRESS>"release:finalize"<EMAIL_ADDRESS>

(Optional) If you are using an issue tracking system other than what your Git hosting
solution provides (e.g. Azure DevOps instead of Github Issues), add this config to
package.json:

"standard-version": {
   "issueUrlFormat": "https://${YOUR_ADO_INSTANCE}/tickets/{{id}}"
}

Creating a Release Candidate

Check out the master branch (or a release_* branch if the release you are
creating is not based off of master ). Fetch and rebase the latest changes from
the corresponding remote branch. For example, if you are working off of master: git fetch --all; git rebase origin/master
Run npm release:prep-changelog -- --prerelease=rc. You should now have a git
branch called changelog-v${NEW_VERSION} checked out with the CHANGELOG updates
committed.
If the CHANGELOG output results in a heading with no new CHANGELOG entries, delete
the branch and continue to step 5. Otherwise, push the branch to the correct remote
repo, open a PR/MR, and continue with step 4.
After the CHANGELOG PR/MR is reviewed and merged, fetch and rebase again so that you
have the CHANGELOG updates in your local repo.
Run npm run release:finalize -- --prerelease=rc. You should now have a git branch
called prep-v${NEW_VERSION} checked out, with version bumps in
package.json/package-lock.json committed along with a new tag called
v${NEW_VERSION}.
Push the branch to the correct remote repo and open a PR/MR.
Once the PR/MR is reviewed and merged and the build pipelines finish, push the
v${NEW_VERSION} tag to origin or canonical.

Creating the Final Version

Check out the master branch (or a release_* branch if the release you are
creating is not based off of master ). Fetch and rebase the latest changes from
the corresponding remote branch. For example, if you are working off of master: git fetch --all; git rebase origin/master
Run npm release:prep-changelog. You should now have a git branch called
changelog-v${NEW_VERSION} checked out with the CHANGELOG updates committed.
We do not want to include individual headings for the previous release candidates in
the final version's CHANGELOG. So, we need to remove the headings and move any
CHANGELOG entries under them to the final version's heading. For example, consider a
CHANGELOG that looks like this:

### [1.0.0](https://${REMOTE_URL}/compare/v1.0.0-rc.1...v1.0.0) (2021-11-22)

### [1.0.0-rc.1](https://${REMOTE_URL}/compare/v1.0.0-rc.0...v1.0.0-rc.1) (2021-11-19)

### Bug Fixes

* A bugfix that we decided to add to this release ([4fff161](https://${REMOTE_URL}/commit/4fff161bee343d3caaaf74d4d576456437cfaa8f))

### [1.0.0-rc.0](https://${REMOTE_URL}/compare/v0.9.0...v1.0.0-rc.0) (2021-11-09)

### Features

* Remove flux capacitor from DeLorean ([3db39af](https://${REMOTE_URL}/commit/3db39af31ea515da9f74c0ad3f163965f41a0f59))

### Bug Fixes

* Ensure flux capacitor can reach 1.21 gigawatts when needed ([9fd45e2](https://${REMOTE_URL}/commit/9fd45e29ded483ae1b065a1bc78f5506401c20b3))

The headings in the CHANGELOG for the final version should look like this:
### [1.0.0](https://${REMOTE_URL}/compare/v1.0.0-rc.1...v1.0.0) (2021-11-22)

### Features

* Remove flux capacitor from DeLorean ([3db39af](https://${REMOTE_URL}/commit/3db39af31ea515da9f74c0ad3f163965f41a0f59))

### Bug Fixes

* A bugfix that we decided to add to this release ([4fff161](https://${REMOTE_URL}/commit/4fff161bee343d3caaaf74d4d576456437cfaa8f))
* Ensure flux capacitor can reach 1.21 gigawatts when needed ([9fd45e2](https://${REMOTE_URL}/commit/9fd45e29ded483ae1b065a1bc78f5506401c20b3))

Update the diff link in the final version's CHANGELOG heading so that all commits in
the new version are included. In the example above, the CHANGELOG heading looks like
this:

### [1.0.0](https://${REMOTE_URL}/compare/v1.0.0-rc.1...v1.0.0) (2021-11-22)

Note that the version range in the URL is v1.0.0-rc.1...v1.0.0. We want to include
all of the changes in v1.0.0, not just the ones since the last release candidate.
If, for example, the last released version was v0.9.0, update the URL to:
### [1.0.0](https://${REMOTE_URL}/compare/v0.9.0...v1.0.0) (2021-11-22)

Push the branch to the correct remote repo and open a PR/MR.
After the CHANGELOG PR/MR is reviewed and merged, fetch and rebase again so that you
have the CHANGELOG updates in your local repo.
Run npm run release:finalize. You should now have a commit with version bumps in
package.json/package-lock.json along with a new tag called v${NEW_VERSION}.
Push the changes to the branch that you are on (either master or a release_*
branch) to the canonical remote (either origin or canonical).
Push the tag to the canonical remote: git push ${ORIGIN_OR_CANONICAL} v${NEW_VERSION}

@jthomerson could I have your concept review on these changes? I tried to give enough context for you to do so, but let me know if you have any questions.
Please note that these are bash scripts and will only work on macOS, Linux, or WSL.
@pbredenberg once @jthomerson is done, could I get your review on the code itself?

In talking with @pbredenberg, we might be able to simplify the release process even more. As such, I'd like to propose the changes found in #32. The full process is documented in the README changes in that PR. But in short the differences between the process described in this PR are:

When creating a final version, the npm run release:prep-changelog script will automatically remove the changelog entries for the release candidates and regenerate a single changelog for all the commits in the release. This reduces the manual editing that needs to take place to merge the headings.

In the event any manual edits were made to one of the RC changelog entries, these won't have been preserved in the regenerated changelog. However, the edits will still be in the git diff and should be easy enough to get back.

Drops the use of standard-version, but still uses conventional-changelog and friends under the hood. One of our biggest hurdles has been "standard-version doesn't allow for X". Rather than build our opinionated release tooling, working around the opinions of standard-version, it appears simpler just to use its underlying changelog/versioning libraries.
Will theoretically work on Windows

In talking with @pbredenberg, we might be able to simplify the release process even more. As such, I'd like to propose the changes found in #32. The full process is documented in the README changes in that PR. But in short the differences between the process described in this PR are:

When creating a final version, the npm run release:prep-changelog script will automatically remove the changelog entries for the release candidates and regenerate a single changelog for all the commits in the release. This reduces the manual editing that needs to take place to merge the headings.

In the event any manual edits were made to one of the RC changelog entries, these won't have been preserved in the regenerated changelog. However, the edits will still be in the git diff and should be easy enough to get back.

Drops the use of standard-version, but still uses conventional-changelog and friends under the hood. One of our biggest hurdles has been "standard-version doesn't allow for X". Rather than build our opinionated release tooling, working around the opinions of standard-version, it appears simpler just to use its underlying changelog/versioning libraries.

Will theoretically work on Windows

Thanks for taking a look at this PR @onebytegone / @pbredenberg. About the points above:

It would be great not to have to manually merge the release candidate CHANGELOG entires for final versions. However, I'm a little on the fence about this change because, particularly for our highly-used libraries, we edit the CHANGELOG entries more often than not to add clarification and to fix grammar. Losing those changes and then having to re-create them from the git history every time offsets the gains from this change a bit.
There are a few scenarios that standard-version (and this PR) supports that #32 does not (as best I can tell). I'll describe those below.
Those that we work with that use Windows are already having to use WSL, which I think these scripts are compatible with.

From what I can tell (please correct me if I'm wrong), #32 doesn't support a few scenarios that this PR does (and that we've had using standard-version directly until now):

alpha/beta prerelease tags (or any tag other than rc) - It looks like #32 assumes that we will only ever use an rc release tag. We've used alpha and beta in the past to publish versions that aren't ready for a full release candidate bump: https://github.com/silvermine/eslint-config-silvermine/releases/tag/v3.1.0-beta.0 How would we support those?
Overriding the auto-calculated version - There are two scenarios where we need to be able to specify the version number that we want to publish instead of letting our release script calculate the version. We do this using standard-version's --release-as flag.

When performing a major version bump (e.g. 1.2.1 to 2.0.0). It's been difficult for developers to consistently use the BREAKING CHANGE: tag in a commit message footer so that conventional-* calculates a major version bump instead of a minor one. It's easy to forget, and sometimes there's not a single commit to point to as a "breaking change" but rather a series of commits that implement a breaking change. So, we almost always have to tell standard-release to override its calculated version with a major version bump.
Developers sometimes use conventional-commit tags incorrectly. A handful of times developers have used feat when they should've used fix, refactor, docs, or ci. When it slips past code review and ends up in master, the release scripts will auto-calculate the wrong version and we need to be able to override it. True, we should strive not to do this and train others to use commit tags correctly, but our release process should realistically acknowledge that we will make mistakes and allow us to correct them.
Breaking changes are acceptable between 0.x versions, but the script in #32 bumps to v1.x when a commit includes the BREAKING CHANGE tag. Let's say that we are on v0.8.0, create a breaking change in the API and use the BREAKING CHANGE tag but want the next version to be v0.9.0 because we are not ready for a v1.0.0 yet. standard-version accounts for this scenario and bumps you to v0.9.0 (you supply the --release-as flag to take it to a v1.0.0 when you're ready).

Minor issues:

We lose the ability to skip version bumping for the first release - standard-version has a --first-release flag that we use when generating the CHANGELOG and tag for the first release. That flag skips the version bump when performing the initial release because we usually don't want to bump the version in package.json for the first release.
We lose the ability to do dry-runs - standard-version has a --dry-run flag that you can use to check the results of a command without actually performing the command. This is useful when you want to preview what the next version number will be without actually updating the changelog/committing. For example, if you know that the next version should be a major version bump but you'd rather not scroll through the list of commits to check whether or not the developers included the BREAKING CHANGE footer tag, you can just run the CHANGELOG generation command with the --dry-run flag to understand what version you're going to get.

Overall, I think the logging output and programming language is better in #32. However, while standard-version isn't ideal for our use case, ultimately I stuck with it in this PR because:

The flexibility it offers accounts for these scenarios
Overall it's fewer LOC than stitching together different libs ourselves

Perhaps there's a way to adjust #32 so that we can at least support the first two "must-haves"? Or would it be simpler to stick with standard-version and this PR? What do @onebytegone and @pbredenberg think about these points and use cases?

Great notes @yokuze! It's good we're finally getting the requirements documented. I think most of the changes noted are simple to resolve. The changes have been pushed to #32. Comments inline.

It would be great not to have to manually merge the release candidate CHANGELOG entries for final versions. However, I'm a little on the fence about this change because, particularly for our highly-used libraries, we edit the CHANGELOG entries more often than not to add clarification and to fix grammar. Losing those changes and then having to re-create them from the git history every time offsets the gains from this change a bit.

In my testing, pulling back the manual edits for these cases is basically:
npm run release:prep-changelog
git reset HEAD^
git add -p # and stage the changes you want
git commit -m "chore: update changelog for v1.0.0"

As example based on your prior example, the RC-based changelog of:
## [1.0.0-rc.2](https://github.com/onebytegone/package-versioning-test-repo/compare/v1.0.0-rc.1...v1.0.0-rc.2) (2022-03-02)

### Features

* install Mr. Fusion ([9370f17](https://github.com/onebytegone/package-versioning-test-repo/commit/9370f171554ef0b3c4d07aebfa294432102af3ab))

### Bug Fixes

* walk past the almanac ([992a4c7](https://github.com/onebytegone/package-versioning-test-repo/commit/992a4c742ac1af3a58cf182308327a743de2ad35))

## [1.0.0-rc.1](https://github.com/onebytegone/package-versioning-test-repo/compare/v1.0.0-rc.0...v1.0.0-rc.1) (2022-03-02)

### Bug Fixes

* remove need for roads ([09d3174](https://github.com/onebytegone/package-versioning-test-repo/commit/09d317461f246a291844f964418067fdcfa99ff9))

## [1.0.0-rc.0](https://github.com/onebytegone/package-versioning-test-repo/compare/v0.1.0...v1.0.0-rc.0) (2022-03-02)

### Features

* add flux capacitor to the DeLorean OUTATIME ([6adb1f9](https://github.com/onebytegone/package-versioning-test-repo/commit/6adb1f9eb28c32eae4407eda222ffeb0f8cc7263))
   * If you're gonna build a time machine into a car, why not do it with some style?

### Bug Fixes

* ensure flux capacitor can reach 1.21 gigawatts when needed ([8e1c5f6](https://github.com/onebytegone/package-versioning-test-repo/commit/8e1c5f6ea7fc3fcd70f9373e0a000772526e4b63))
   * ONE POINT TWENTY-ONE GIGAWATTS!!!

will be regenerated and committed as:
@@ -3,39 +3,20 @@
 All notable changes to this project will be documented in this file.
 See [our coding standards][commit-messages] for commit guidelines.

-## [1.0.0-rc.2](https://github.com/onebytegone/package-versioning-test-repo/compare/v1.0.0-rc.1...v1.0.0-rc.2) (2022-03-02)
+## [1.0.0](https://github.com/onebytegone/package-versioning-test-repo/compare/v0.1.0...v1.0.0) (2022-03-02)

 ### Features

+* add flux capacitor to DeLorean ([6adb1f9](https://github.com/onebytegone/package-versioning-test-repo/commit/6adb1f9eb28c32eae4407eda222ffeb0f8cc7263))
 * install Mr. Fusion ([9370f17](https://github.com/onebytegone/package-versioning-test-repo/commit/9370f171554ef0b3c4d07aebfa294432102af3ab))

-### Bug Fixes
-
-* walk past the almanac ([992a4c7](https://github.com/onebytegone/package-versioning-test-repo/commit/992a4c742ac1af3a58cf182308327a743de2ad35))
-
-
-## [1.0.0-rc.1](https://github.com/onebytegone/package-versioning-test-repo/compare/v1.0.0-rc.0...v1.0.0-rc.1) (2022-03-02)
-
-
-### Bug Fixes
-
-* remove need for roads ([09d3174](https://github.com/onebytegone/package-versioning-test-repo/commit/09d317461f246a291844f964418067fdcfa99ff9))
-
-
-## [1.0.0-rc.0](https://github.com/onebytegone/package-versioning-test-repo/compare/v0.1.0...v1.0.0-rc.0) (2022-03-02)
-
-
-### Features
-
-* add flux capacitor to the DeLorean OUTATIME ([6adb1f9](https://github.com/onebytegone/package-versioning-test-repo/commit/6adb1f9eb28c32eae4407eda222ffeb0f8cc7263))
-   * If you're gonna build a time machine into a car, why not do it with some style?
-
 ### Bug Fixes

 * ensure flux capacitor can reach 1.21 gigawatts when needed ([8e1c5f6](https://github.com/onebytegone/package-versioning-test-repo/commit/8e1c5f6ea7fc3fcd70f9373e0a000772526e4b63))
-   * ONE POINT TWENTY-ONE GIGAWATTS!!!
+* remove need for roads ([09d3174](https://github.com/onebytegone/package-versioning-test-repo/commit/09d317461f246a291844f964418067fdcfa99ff9))
+* walk past the almanac ([992a4c7](https://github.com/onebytegone/package-versioning-test-repo/commit/992a4c742ac1af3a58cf182308327a743de2ad35))

After resetting that commit, it's just a process of picking the changes that you want to keep (i.e. dropping mention of the almanac, it never happened).
 @@ -3,39 +3,20 @@
 All notable changes to this project will be documented in this file.
 See [our coding standards][commit-messages] for commit guidelines.

-## [1.0.0-rc.2](https://github.com/onebytegone/package-versioning-test-repo/compare/v1.0.0-rc.1...v1.0.0-rc.2) (2022-03-02)
-
-
-### Features
-
-* install Mr. Fusion ([9370f17](https://github.com/onebytegone/package-versioning-test-repo/commit/9370f171554ef0b3c4d07aebfa294432102af3ab))
-
-
-### Bug Fixes
-
-* walk past the almanac ([992a4c7](https://github.com/onebytegone/package-versioning-test-repo/commit/992a4c742ac1af3a58cf182308327a743de2ad35))
-
-
-## [1.0.0-rc.1](https://github.com/onebytegone/package-versioning-test-repo/compare/v1.0.0-rc.0...v1.0.0-rc.1) (2022-03-02)
-
-
-### Bug Fixes
-
-* remove need for roads ([09d3174](https://github.com/onebytegone/package-versioning-test-repo/commit/09d317461f246a291844f964418067fdcfa99ff9))
-
-
-## [1.0.0-rc.0](https://github.com/onebytegone/package-versioning-test-repo/compare/v0.1.0...v1.0.0-rc.0) (2022-03-02)
+## [1.0.0](https://github.com/onebytegone/package-versioning-test-repo/compare/v0.1.0...v1.0.0) (2022-03-02)

 ### Features

 * add flux capacitor to the DeLorean OUTATIME ([6adb1f9](https://github.com/onebytegone/package-versioning-test-repo/commit/6adb1f9eb28c32eae4407eda222ffeb0f8cc7263)
    * If you're gonna build a time machine into a car, why not do it with some style?
+* install Mr. Fusion ([9370f17](https://github.com/onebytegone/package-versioning-test-repo/commit/9370f171554ef0b3c4d07aebfa294432102af3ab))

 ### Bug Fixes

 * ensure flux capacitor can reach 1.21 gigawatts when needed ([8e1c5f6](https://github.com/onebytegone/package-versioning-test-repo/commit/8e1c5f6ea7fc3fcd70f9373e0a
    * ONE POINT TWENTY-ONE GIGAWATTS!!!
+* remove need for roads ([09d3174](https://github.com/onebytegone/package-versioning-test-repo/commit/09d317461f246a291844f964418067fdcfa99ff9))

Worse comes to worse, the releaser can manually update the changelog as described earlier. However, I think the number of repos that we manually the edit the changelog for is far surpassed by those we don't need to edit. I'd rather have the default cover the 80% case for changelog generation.

alpha/beta prerelease tags (or any tag other than rc) - It looks like feat: add script to simplify the package release process (#27) #32 assumes that we will only ever use an rc release tag. We've used alpha and beta in the past to publish versions that aren't ready for a full release candidate bump: https://github.com/silvermine/eslint-config-silvermine/releases/tag/v3.1.0-beta.0 How would we support those?

Yeah, @pbredenberg and I didn't have this requirement in our notes. On #32, the --rc flag has been replaced with --prerelease rc. Wanted to keep the interface simple, but it doesn't seem possible to get around this one.

Overriding the auto-calculated version - There are three scenarios where we need to be able to specify the version number that we want to publish instead of letting our release script calculate the version. We do this using standard-version's --release-as flag.

When performing a major version bump (e.g. 1.2.1 to 2.0.0). It's been difficult for developers to consistently use the BREAKING CHANGE: tag in a commit message footer so that conventional-* calculates a major version bump instead of a minor one. It's easy to forget, and sometimes there's not a single commit to point to as a "breaking change" but rather a series of commits that implement a breaking change. So, we almost always have to tell standard-release to override its calculated version with a major version bump.
Developers sometimes use conventional-commit tags incorrectly. A handful of times developers have used feat when they should've used fix, refactor, docs, or ci. When it slips past code review and ends up in master, the release scripts will auto-calculate the wrong version and we need to be able to override it. True, we should strive not to do this and train others to use commit tags correctly, but our release process should realistically acknowledge that we will make mistakes and allow us to correct them.

These has been remedied by adding a --version <version> option.

Breaking changes are acceptable between 0.x versions, but the script in feat: add script to simplify the package release process (#27) #32 bumps to v1.x when a commit includes the BREAKING CHANGE tag. Let's say that we are on v0.8.0, create a breaking change in the API and use the BREAKING CHANGE tag but want the next version to be v0.9.0 because we are not ready for a v1.0.0 yet. standard-version accounts for this scenario and bumps you to v0.9.0 (you supply the --release-as flag to take it to a v1.0.0 when you're ready).

There's a preMajor flag on conventional-recommended-bump that made this simple enough to fix.

We lose the ability to skip version bumping for the first release - standard-version has a --first-release flag that we use when generating the CHANGELOG and tag for the first release. That flag skips the version bump when performing the initial release because we usually don't want to bump the version in package.json for the first release.

With the --version <version> option, it's now possible to run npm run release:prep-changelog -- --version 0.1.0 and  npm run release:finalize -- --version 0.1.0 for this case.

We lose the ability to do dry-runs - standard-version has a --dry-run flag that you can use to check the results of a command without actually performing the command. This is useful when you want to preview what the next version number will be without actually updating the changelog/committing. For example, if you know that the next version should be a major version bump but you'd rather not scroll through the list of commits to check whether or not the developers included the BREAKING CHANGE footer tag, you can just run the CHANGELOG generation command with the --dry-run flag to understand what version you're going to get.

Added npm run release:preview. This also makes it so developers can see what their commits will do to the changelog and version if they want.

Overall, I think the logging output and programming language is better in #32. However, while standard-version isn't ideal for our use case, ultimately I stuck with it in this PR because:

The flexibility it offers accounts for these scenarios
Overall it's fewer LOC than stitching together different libs ourselves

I don't like the increase in LoC either. However, I think it's worth the cost.

Our goal has been to make releases less painful. The cost is ~300 lines of "glue" and logging code to remove most of the pain points. Based on other scripts of a similar nature and size, I don't expect there to be a large maintenance cost.
Does what we need and no more. standard-version can do a lot of things, but knowing which we need isn't always clear (as we've found in the many conversations about releases that the team's had 😆 ).
Clarifies what lib is doing what part of the process. We've long run into the issue "what's doing this part of the release process?" There's value in having one layer of abstraction removed.

Closed in favor of #32


Custom docker config does not work with buildah container backend

Before proceeding

[X] I didn't find a similar issue

Version
v1.2.174
How to reproduce
werf cr login -u $USERNAME -p $PASSWORD $WERF_REPO --docker-config=/tmp/test
werf render --docker-config=/tmp/test

Result
Error: UNAUTHORIZED: unauthorized to access repository

Expected result
Command runs without error.
Additional information
No response

fixed by https://github.com/werf/werf/issues/4957


fix for tinyrpc dep dispatch three args

Fix for the following error running the pyeth hive client:
[9c3a9f8c] INFO:jsonrpc starting JSONRPCServer port=8545
[9c3a9f8c] INFO:p2p.peermgr     waiting for bootstrap
DBUG[09-25|17:14:38] client container online                  client=pyethereum:develop simulator=ethereum/consensus req-id=4966 id=9c3a9f8c time=19.354373188s
DBUG[09-25|17:14:38] new simulator request                    client=pyethereum:develop simulator=ethereum/consensus req-id=4967 from=<IP_ADDRESS>:42584 method=GET endpoint=
/nodes/9c3a9f8c
[9c3a9f8c]
[9c3a9f8c] TypeErrorTraceback (most recent call last)
[9c3a9f8c] /usr/lib/python2.7/site-packages/gevent-1.1.0-py2.7-linux-x86_64.egg/gevent/greenlet.pyc in run(self=<Greenlet at 0x7f56bef5c690: handle_message(<Que...params":
 [], "jsonrpc": "2.0", "method": "web3_)>)
[9c3a9f8c]     532
[9c3a9f8c]     533             try:
[9c3a9f8c] --> 534                 result = self._run(*self.args, **self.kwargs)
[9c3a9f8c]         result = undefined
[9c3a9f8c]         self._run = <function handle_message at 0x7f56bef9b398>
[9c3a9f8c]         self.args = (<Queue at 0x7f56bef8fd50 getters[1]>, '{"params": [], "jsonrpc": "2.0", "method": "web3_clientVersion", "id": 1}')
[9c3a9f8c]         self.kwargs = {}
[9c3a9f8c]     535             except:
[9c3a9f8c]     536                 self._report_error(sys.exc_info())
[9c3a9f8c]
[9c3a9f8c] /usr/lib/python2.7/site-packages/tinyrpc-0.6-py2.7.egg/tinyrpc/server/__init__.pyc in handle_message(context=<Queue at 0x7f56bef8fd50 getters[1]>, message='{"params": [], "jsonrpc": "2.0", "method": "web3_clientVersion", "id": 1}')
[9c3a9f8c]      81                 response = self.dispatcher.dispatch(
[9c3a9f8c]      82                     request,
[9c3a9f8c] ---> 83                     getattr(self.protocol, '_caller', None)
[9c3a9f8c]      84                 )
[9c3a9f8c]      85
[9c3a9f8c]
[9c3a9f8c] TypeError: dispatch() takes exactly 2 arguments (3 given)
[9c3a9f8c] > /build/bdist.linux-x86_64/egg/tinyrpc/server/__init__.py(83)handle_message()
[9c3a9f8c]
[9c3a9f8c] ipdb>
[9c3a9f8c] Debugger exited. Do you want to quit pyethapp? [Y/n]
[9c3a9f8c] EOFErrorTraceback (most recent call last)
[9c3a9f8c] /usr/lib/python2.7/site-packages/gevent-1.1.0-py2.7-linux-x86_64.egg/gevent/greenlet.pyc in run(self=<Greenlet at 0x7f56bef5c690: handle_message(<Que...params": [], "jsonrpc": "2.0", "method": "web3_)>)
[9c3a9f8c]     534                 result = self._run(*self.args, **self.kwargs)
[9c3a9f8c]     535             except:
[9c3a9f8c] --> 536                 self._report_error(sys.exc_info())
[9c3a9f8c]         self._report_error = <bound method Greenlet._report_error of <Greenlet at 0x7f56bef5c690: handle_message(<Queue at 0x7f56bef8fd50 getters[1]>, '{"params": [], "jsonrpc": "2.0", "method": "web3_)>>
[9c3a9f8c]         global sys.exc_info = <built-in function exc_info>
[9c3a9f8c]     537                 return
[9c3a9f8c]     538             self._report_result(result)
[9c3a9f8c]
[9c3a9f8c] /usr/lib/python2.7/site-packages/gevent-1.1.0-py2.7-linux-x86_64.egg/gevent/greenlet.pyc in _report_error(***failed resolving arguments***)
[9c3a9f8c]     522
[9c3a9f8c]     523         try:
[9c3a9f8c] --> 524             self.parent.handle_error(self, *exc_info)
[9c3a9f8c]         self.parent.handle_error = <bound method Hub.handle_error of <Hub at 0x7f56bf441af0 epoll default pending=0 ref=5 fileno=3 resolver=<gevent.resolver_thread.Resolver at 0x7f56bef6b0d0 pool=<ThreadPool at 0x7f56bef73290 0/1/10>> threadpool=<ThreadPool at 0x7f56bef73290 0/1/10>>>
[9c3a9f8c]         self = <Greenlet at 0x7f56bef5c690: handle_message(<Queue at 0x7f56bef8fd50 getters[1]>, '{"params": [], "jsonrpc": "2.0", "method": "web3_)>
[9c3a9f8c]         exc_info = undefined

depends on https://github.com/ethereum/pyethapp/pull/255

#255 has been merged.  does this need to be updated?

No update, its good.

Coverage decreased (-0.2%) to 49.318% when pulling 488a7723052192d35672f65546f3038a9ebdddd8 on cdetrio:fix-hive-rpc into 1f97f98b332f2a03cb58bd6bc78daa45181a3fb1 on ethereum:develop.


[Apple Reminders] Natural language does not set recurring & location

Extension
https://www.raycast.com/raycast/apple-reminders
Raycast Version
1.81.2
macOS Version
No response
Description
When i add a quick reminder, such as the one in the example "Send the weekly project status report Friday at 2 PM in the Work list", it creates the reminder of "Send weekly project status report in Work list" in the default task list.
Additionally, the "Remind me to buy milk when I'm leaving the office." will create a "Remind me to buy milk when I'm leaving the office." reminder with no time or location.
it also does not work for repeating tasks.
Steps To Reproduce

raycast mac
nothing extra, just apple reminders extention
run "quick add reminder"
see that it creates the reminder without the natural language

Current Behaviour
Creates a task without natural language based on location and repeating tasks in correct list
Expected Behaviour
create a reminder with location set correctly, extra words scraped off (such as "remind me to get milk" modified to "get milk"), repeated tasks being repeated correctly, lists selected correctly.

 These features work only if your reminder is parsed by Raycast AI, which is a feature limited to Pro users.


Update e2e usage guide with benchmark test execution instructions

Problem
The current documentation for e2e tests does not provide clear instructions for running e2e benchmark tests. Adding instructions for these tests not only clarifies their existence but also assists users in executing and interacting with them.
Acceptance Criteria

[ ] Update the e2e tests usage guide with a section detailing how to execute benchmark tests.

is this still being worked on @staheri14


Nullable types when requested non-nullable

For input List<String> -> String Inkuire found firstOrNull: <T> Iterable<T>.() -> T? which is wrong, because we requested T, not T?

It's actually a feature


Separate type for aggregated committee attestation

Description
Introducing separate container for aggregated attestation in a committee. The new container replaces the unnecessary committee bits with committee index. It also has 63x shorter max length than Attestation on aggregation_bits which yields max size going from 16.23kb to 0.48kb. This ensures aggregates with 63 committees worth of zeros in aggregation bits with not be deserialized.
The current Attestation represents aggregated attestation both on-chain and in a committee. Having a dedicated container for committee attestation improves readability and removes the need to maintain a one-bit-set committee bit list.

Referencing https://github.com/ethereum/consensus-specs/pull/3900#issuecomment-2383952335 and https://github.com/ethereum/consensus-specs/pull/3787 which have kind of similar ideas.


[chores] Minor typographical error fix

Related openwisp/openwisp-users#173

Coverage remained the same at 99.06% when pulling ba0d6fb28b4e2e16ed3601dae4ac3b680c91676d on devkapilbansal:typo-error into 3d6b8767af728e0d82d04c98b38846260cdd7183 on openwisp:master.

Coverage remained the same at 99.06% when pulling ba0d6fb28b4e2e16ed3601dae4ac3b680c91676d on devkapilbansal:typo-error into 3d6b8767af728e0d82d04c98b38846260cdd7183 on openwisp:master.


implement asyncGenerator

Would something like this speed up TTFB in certain scenarios?
I'll refactor the tests

My god, I'll add label or something for the benchmark, this is untenable 😄

But I'm thinking, instead of for awaiting in the inner loops, maybe we can batch them in an array
We'd still yield one by one in order as they get resolved but it could maybe improve performance


feat(icons) - ma-823 - add new arrow up/down icons to be used by vitals metric cards

Summary
https://konghq.atlassian.net/browse/MA-823

Vue 3
Did you create a corresponding PR to add this change to the beta branch? (required)

[ ] Yes, here is a link to the PR: [link to beta branch PR]
[ ] No, I did not create a PR for the beta branch and I have added the port to beta branch tag to this PR.
[ ] No, because [type your reasons]

If you have questions, tag @adamdehaven or @kaiarrowood.

PR Checklist

[ ] Does not introduce dependencies
[ ] Functional: all changes do not break existing APIs and if so, bump major version.
[ ] Tests pass: check the output of yarn test packages/
[ ] Naming: the files and the method and prop variables use the same naming conventions as other Kongponents
[ ] Framework style: abides by the essential rules in Vue's style guide
[ ] Cleanliness: does not have formatting issues, unused code (e.g., console.logs), or leftover comments
[ ] Docs: includes a technically accurate README, uses JSDOC where appropriate

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Vite: package is not published?

Describe the bug
The dependencies showing in packages/vite/package.json is requiring NX 15+
https://github.com/nxext/nx-extensions/blob/main/packages/vite/package.json
But the actual version installed via NPM is ^14.8.5
{
  "name": "@nxext/vite",
  "version": "14.2.0",
  "license": "MIT",
  "author": "Dominik Pieper",
  "description": "Nx plugin for ViteJs",
  "homepage": "https://nxext.dev/",
  "keywords": [
    "nx",
    "nx-plugin",
    "viteJs",
    "vite"
  ],
  "repository": {
    "type": "github",
    "url": "https://github.com/nxext/nx-extensions"
  },
  "bugs": {
    "url": "https://github.com/nxext/nx-extensions/issues"
  },
  "main": "./src/index.js",
  "typings": "./src/index.d.ts",
  "executors": "./executors.json",
  "generators": "./generators.json",
  "dependencies": {
    "@nrwl/devkit": "14.8.5",
    "rxjs": "^7.0.0",
    "rxjs-for-await": "1.0.0",
    "tsconfig-paths": "^3.14.1",
    "globrex": "^0.1.2"
  },
  "peerDependencies": {
    "@nrwl/workspace": "^14.8.5",
    "vite": "^3.0.5"
  },
  "nx-migrations": {
    "migrations": "./migrations.json"
  },
  "types": "./src/index.d.ts"
}

Which causing NPM unable to perform fresh install.

Same thing here

Hey, we are removing our vite package in favour of NX vite. Migration will be provided soon

@Jordan-Hall The stencil package on the repo is also on 15 but on NPM there is no version on this range. Is this expected because of the vite package removal?

@ianvieira  this is cause there’s an open issue with the Storybook integration on stencil. Once this is closed the v15 package will be published

@ianvieira the new Stencil package is released for v15

@DominikPieper Thank you very much 🙇🏻

I'll close this here :-) We are working on migrations to the new official packages.


Bugfix/2345 Character issues from previous applications are not appearing

What's included?
Closes #2345

Fix Firebase modular syntax.

Who should test?
✅ Product owner
✅ Developers
How to test?
Example exercise: https://jac-admin-develop--pr2358-bugfix-2345-characte-6zg4bd2g.web.app/exercise/aDxEqK36807d0Nhpugo3/reports/character-issues

Go to "Character Issues" report page of an exercise.
Click "View all character issues from previous applications" and check if the historic character issues display correctly.

Risk - how likely is this to impact other areas?
🟢 No risk - this is a self-contained piece of work
Additional context
Demo:
https://github.com/jac-uk/admin/assets/79906532/26763ceb-47e1-44c3-a99c-7cac2f116137
Related permissions
Have permissions been considered for this functionality?

No permission changes required
Permissions have been added / updated. Details:

PREVIEW:DEVELOP
can be OFF, DEVELOP or STAGING

@nickaddy I have fixed the test data and the historical character issues for JAC00685 and JAC00636 are appearing. "Data not requested" is triggered when there are no character issues from the application.

@HalcyonJAC Looking at a live exercise, A***** A****** declared the same issues on his application for JAC00077, but they do not appear when you click on View all character issues from previous applications. Even if they are the same ones, anything that has been declared on a previous exercise should appear when View all character issues from previous applications is clicked.


BUG: Signal.butter in analog mode has all the gain in the first sos

Describe your issue.
Filter coefficients in analog mode should be normalized to section gains of one (1). Practically they are not realizable and plotting the section frequency responses without normalization is also not too nice. Plot figures attached, Y in dB. Top default, bottom normalized.

Reproducing Code Example
>>> signal.butter(2, 200, analog=True, output='sos')
array([[0.00000000e+00, 0.00000000e+00, 4.00000000e+04, 1.00000000e+00,
        2.82842712e+02, 4.00000000e+04]])
>>> signal.butter(4, 200, analog=True, output='sos')
array([[0.00000000e+00, 0.00000000e+00, 1.60000000e+09, 1.00000000e+00,
        3.69551813e+02, 4.00000000e+04],
       [0.00000000e+00, 0.00000000e+00, 1.00000000e+00, 1.00000000e+00,
        1.53073373e+02, 4.00000000e+04]])
>>> signal.butter(6, 200, analog=True, output='sos')
array([[0.00000000e+00, 0.00000000e+00, 6.40000000e+13, 1.00000000e+00,
        3.86370331e+02, 4.00000000e+04],
       [0.00000000e+00, 0.00000000e+00, 1.00000000e+00, 1.00000000e+00,
        2.82842712e+02, 4.00000000e+04],
       [0.00000000e+00, 0.00000000e+00, 1.00000000e+00, 1.00000000e+00,
        1.03527618e+02, 4.00000000e+04]])

Error message
None

SciPy/NumPy/Python version information
1.8.0 1.21.5 sys.version_info(major=3, minor=10, micro=6, releaselevel='final', serial=0)

There is more in this bug than already told. Gains in lowpass zpk format are off completely and lowpass returns an empty array which is not fine. Highpass works and the original bug is not there. Something else is broken as freqs picks about 1000 times too low frequency range. Plot shows default behavior. All this is present in SciPy 1.10.0 as well.
>>> butter(2, 1050, btype='low', analog=True, output='zpk')
(array([], dtype=float64), array([-742.46212025+742.46212025j, -742.46212025-742.46212025j]), 1102500.0)
>>> butter(2, 1050, btype='high', analog=True, output='zpk')
(array([0., 0.]), array([-742.46212025-742.46212025j, -742.46212025+742.46212025j]), 1.0)

All analog gains should be one including Chebyshev and Elliptic...
>>> butter(2, 200, btype='high', analog=True, output='sos')
array([[1.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00,
        2.82842712e+02, 4.00000000e+04]])
>>> butter(4, 200, btype='high', analog=True, output='sos')
array([[1.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00,
        3.69551813e+02, 4.00000000e+04],
       [1.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00,
        1.53073373e+02, 4.00000000e+04]])
>>> butter(6, 200, btype='high', analog=True, output='sos')
array([[1.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00,
        3.86370331e+02, 4.00000000e+04],
       [1.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00,
        2.82842712e+02, 4.00000000e+04],
       [1.00000000e+00, 0.00000000e+00, 0.00000000e+00, 1.00000000e+00,
        1.03527618e+02, 4.00000000e+04]])

There is more in this bug than already told. Gains in lowpass zpk format are off completely. Highpass works and the original bug is not there. Something else is broken as freqs picks about 1000 times too low frequency range.

I don't agree: that non-unity gain is how the zpk representation works, i.e.:
$$
g(s) = k\frac{\prod_i s-z_i}{\prod_j s-p_j}
$$
so, to get unity gain at DC (where s=0), we need $k = \prod_j -p_j$, as it is in your example:
In [1]: import scipy

In [2]: from scipy import signal

In [3]: scipy.__version__
Out[3]: '1.10.0'

In [4]: signal.tf2zpk([10],[1,10])
Out[4]: (array([], dtype=float64), array([-10.]), 10.0)

In [6]: zpk = signal.butter(2, 1050, btype='low', analog=True, output='zpk')

In [8]: signal.zpk2tf(*zpk)
Out[8]: (array([1102500.]), array([1.00000000e+00, 1.48492424e+03, 1.10250000e+06]

$$
One might wonder why we can't have a convention of unity gain at DC, but then we wouldn't be able to have poles or zeros at the origin.

"All the gain in the first second-order section" is true for all combinations of (analog, digital) and (butter, cheby, cheby2, ellip) (I'm not sure about bessel, but likely: think they all use zpk2sos).
Consider, e.g., the numerator of magnitude $10^{-14}$ in this elliptical filter:
In [18]: signal.iirdesign(0.09, 0.11, 0.1, 300, ftype='ellip', output='sos')
Out[18]:
array([[ 1.64825222e-14,  6.03310059e-15,  1.64825222e-14,
         1.00000000e+00, -9.67090642e-01,  0.00000000e+00],
       [ 1.00000000e+00,  1.00000000e+00,  0.00000000e+00,
         1.00000000e+00, -1.93371066e+00,  9.36370425e-01],
       [ 1.00000000e+00, -9.10214404e-01,  1.00000000e+00,
         1.00000000e+00, -1.93236515e+00,  9.39544725e-01],
       [ 1.00000000e+00, -1.40194610e+00,  1.00000000e+00,
         1.00000000e+00, -1.93032634e+00,  9.44394208e-01],
       [ 1.00000000e+00, -1.61526902e+00,  1.00000000e+00,
         1.00000000e+00, -1.92784655e+00,  9.50377038e-01],
       [ 1.00000000e+00, -1.72276085e+00,  1.00000000e+00,
         1.00000000e+00, -1.92519540e+00,  9.56922819e-01],
       [ 1.00000000e+00, -1.78326738e+00,  1.00000000e+00,
         1.00000000e+00, -1.92261599e+00,  9.63532375e-01],
       [ 1.00000000e+00, -1.82008832e+00,  1.00000000e+00,
         1.00000000e+00, -1.92030038e+00,  9.69834981e-01],
       [ 1.00000000e+00, -1.84374256e+00,  1.00000000e+00,
         1.00000000e+00, -1.91838485e+00,  9.75602650e-01],
       [ 1.00000000e+00, -1.85949227e+00,  1.00000000e+00,
         1.00000000e+00, -1.91695883e+00,  9.80734540e-01],
       [ 1.00000000e+00, -1.87018494e+00,  1.00000000e+00,
         1.00000000e+00, -1.91608075e+00,  9.85227466e-01],
       [ 1.00000000e+00, -1.87745292e+00,  1.00000000e+00,
         1.00000000e+00, -1.91579498e+00,  9.89144839e-01],
       [ 1.00000000e+00, -1.88226630e+00,  1.00000000e+00,
         1.00000000e+00, -1.91614707e+00,  9.92590829e-01],
       [ 1.00000000e+00, -1.88520530e+00,  1.00000000e+00,
         1.00000000e+00, -1.91719603e+00,  9.95692024e-01],
       [ 1.00000000e+00, -1.88659980e+00,  1.00000000e+00,
         1.00000000e+00, -1.91902361e+00,  9.98586109e-01]])

It would be interesting to know if this is a problem in application of the filter; my first, unexamined guess is no, at least for digital filters, since it applies to all elements of the numerator, so the "down-scaling" is uniform for the first second-order section.
One case the scaling could be a real problem is in cascaded second-order electronic filters: in that case one would probably want to keep the pass-band gain close to 1 to avoid hitting power-supply limits on the one hand, and descending below the noise floor on the other hand.
I think the numerators could be exactly rescaled using numpy.frexp and numpy.ldexp  to distribute the exponent between the sections ("exactly" here is from a floating-point point-of-view.)
Here's a proof-of-concept:
import numpy as np

def rescalesos(sos):
    # Strictly proof-of-concept
    #  - doesn't handle general analog (analog 1st order section has leading 0 in numerator)
    #    - maybe check for largest absolute exponent in whole numerator?
    #  - assumes SOS is of form
    #     [k, .... ],
    #     [1, .... ],
    #     [1, .... ],
    #     [..., .... ],
    #     [1, .... ],
    #     - fix this by summing over the numerators
    sos = sos.copy()

    mnum, enum = np.frexp(sos[:,:3])
    q, r = divmod(enum[0,0], sos.shape[0])

    enum[0] -= enum[0,0]
    enum[:r] += q+1
    enum[r:] += q

    sos[:,:3] = np.ldexp(mnum, enum)

    return sos

if __name__ == '__main__':
    from scipy import signal
    sos = signal.iirdesign(0.09, 0.11, 0.1, 300, ftype='ellip', output='sos')
    resos = rescalesos(sos)

    z, p, k = signal.sos2zpk(sos)
    z2, p2, k2 = signal.sos2zpk(resos)

    assert np.all(z == z2)
    assert np.all(p == p2)
    assert np.all(k == k2)

    with np.printoptions(precision=3):
        print(resos)

which on my system gives
[[ 0.072  0.027  0.072  1.    -0.967  0.   ]
 [ 0.125  0.125  0.     1.    -1.934  0.936]
 [ 0.125 -0.114  0.125  1.    -1.932  0.94 ]
 [ 0.125 -0.175  0.125  1.    -1.93   0.944]
 [ 0.125 -0.202  0.125  1.    -1.928  0.95 ]
 [ 0.125 -0.215  0.125  1.    -1.925  0.957]
 [ 0.125 -0.223  0.125  1.    -1.923  0.964]
 [ 0.125 -0.228  0.125  1.    -1.92   0.97 ]
 [ 0.125 -0.23   0.125  1.    -1.918  0.976]
 [ 0.125 -0.232  0.125  1.    -1.917  0.981]
 [ 0.125 -0.234  0.125  1.    -1.916  0.985]
 [ 0.125 -0.235  0.125  1.    -1.916  0.989]
 [ 0.125 -0.235  0.125  1.    -1.916  0.993]
 [ 0.125 -0.236  0.125  1.    -1.917  0.996]
 [ 0.125 -0.236  0.125  1.    -1.919  0.999]]

FWIW, to whoever can triage this, I see it as more of an enhancement than a defect.

To be fair I am not an expert of any kind in this, just trying to get some real world designs done for amateur radio. I have obviously stated some things without complete understanding of the subject.
Regarding the bug, why the behavior is different for highpass (that seems proper to me) and lowpass?
I am using Scipy for cascaded electronic filter design, so it would make sense to have default section (DC) gains of one.
To save some amplifiers I also combine second order sections to fourth order ones, with combined sections being optimized for dynamic range. E.g. for 8th order filter with two 4th order sections, one with sos 2, 3 and the other with 1, 4. Actual component values calculated by Differential Evolution optimizer for the transfer function (that is from the two combined sos). So far the results have been amazing and totally not possible without Scipy, for me at least. Thanks for everyone involved!

Regarding the bug, why the behavior is different for highpass (that seems proper to me) and lowpass?

The IIR filter designs all start by computing zeros, poles, and a gain k.  This is converted to SOS if requested by the user, using sos2zpk, and sos2zpk always puts the gain in the first second-order section.  For a high-pass filter $g(s)$, we expect $\lim_{\omega\rightarrow\infty} g(jw) \approx 1$, and given the definition of SOS I gave before, for a system where len(z)=len(p), which we expect for a HPF, $\lim_{\omega\rightarrow\infty} g(jw) = k$, so we expect k to be about 1.  We can check that:
In [343]: for ftype in ['butter', 'cheby1', 'cheby2', 'ellip']: print(signal.iirdesign(10, 9, 0.1, 30, ftype=ftype, output='zpk', analog=True)[2])
1.0000000000000004
1.0
1.0
0.988553094656939

compare to the complementary low-pass filters, which have wildly varying gains between the types:
In [344]: for ftype in ['butter', 'cheby1', 'cheby2', 'ellip']: print(signal.iirdesign(9, 10, 0.1, 30, ftype=ftype, output='zpk', analog=True)[2])
3.039172424287326e+49
4066118519.7896285
4.106622065499582
0.03162277660168413

I am using Scipy for cascaded electronic filter design, so it would make sense to have default section (DC) gains of one.

This is a good idea, but it doesn't work in general; in particular, not all LPFs have a DC gain of 1 (ellip and cheby1 have gain <1 for even orders).  And it definitely won't work for filters that aren't LPFs.  However, you can do this normalization yourself after calling iirdesign, etc.
We could special-case butter and cheby2 LPFs to do this normalization, but I think the complexity cost for testing and maintenance would not justify it.  I think there is a case to be made to distribute the gains between the second-order sections automatically, especially for analog filters.  Support for analog SOS is relatively recent, so it's not too surprising that we're missing a few features.  (I see, e.g., that there's no sos_freqs.)
FWIW, the method I posted 3 weeks ago is almost certainly overkill; one could just geometrically divided k between the sections using an nth-root, something like this (untested):
    k = sos[0,2]
    sos[0,:2] /= k
    sos[:,:2] *= k**(1/sos.shape[0])

Regarding DC gains I meant 0 Hz for LP, ∞Hz for HP, center gain for BP, etc.
Implementation in analog electronics makes use of e.g. amplifiers with (voltage) gain of one, an example being Sallen–Key architecture [1]. With such, the LP 0Hz gain is one no matter the function, so any ripples will be on the above one gain side. Depending on the application the section gains may be scaled, but quite often a compromise can be found by rearranging the order, especially if the interfering signal properties are somewhat known. Quite often the DC gain is most important e.g. for measurements near 0 Hz (I consider DC a theoretical signal).
For digital implementation many issues of the analog world disappear or at least change markedly. I guess that is why the gain in first stage is maximized. I still do not get how it would not interfere with filtering as it will limit the input dynamic range greatly. Is it expected all input is below one? Overflows will result otherwise. In the end just the same problems as in analog side, just with stable coefficients ;^)
[1] https://en.wikipedia.org/wiki/Sallen–Key_topology

Anyone interested to work on this or providing some guidance how to proceed for a fix/patch? For example GNU Octave provides the sos and gain separately from zpk2sos... Also one more question, is there interest for SciPy application level code within SciPy distribution? E.g. a filter design tool?

Rtype ellip provides section gains in inverted order compared to butter.

For digital implementation many issues of the analog world disappear or at least change markedly.

I mostly use the signal processing tools for post-processing data, in which case here I use double precision.  The range of values is about $10^{-300}$ to $10^{300}$, so analog issues like absolute noise floor or clipping are mostly irrelevant.

guess that is why the gain in first stage is maximized.

It's not maximized - it just is the "accumulated" gain (see my first comment in this thread).  For digital filters, it's often very small (see my second comment); my guess is that for a very frequency low-pass filter ( << 1 rad/time unit), analog filters would also have a small gain.

Anyone interested to work on this or providing some guidance how to proceed for a fix/patch?

I'm not a core dev, but I think commits making scipy.signal better for analog filter design would be welcome.  The change to zpk2sos to allow for analog filters at all was relatively recent, and I think it would be good to build on that.
I like your idea of normalizing the gain of each section of a SOS at a chosen frequency; it should be straightforward to  write a post-processing function to do exactly that.  I don't know if it would be accepted in scipy, but the SOS format isn't going to change anytime, so you could keep it as part of your own toolbox.

is there interest for SciPy application level code within SciPy distribution? E.g. a filter design tool?

I don't think Scipy packages any applications.
For discrete time filters there is https://github.com/chipmuenk/pyFDA which builds on scipy and looks very nice, though I've never used it.  It looks like it also handles fixed-point filters, and here I think there is a need to distribute gains to avoid overflow and underflow, so maybe it has the tools needed for this sort of thing.  It might also be a good place to start with for a more complicated analog filter design tool.

Rtype ellip provides section gains in inverted order compared to butter.

Could you give some detail, e.g., example code and results?  All SOS designs use zpk2sos under the hood, so should all have the same behaviour.

Thanks for patience in explaining and advise.
I have prepared an example case for mentioned gain distribution bug and the difference of elliptic case. One can clearly see that all high pass cases are roughly fine, but the only lp filter that is at 1 rad/s works as expected. It also shows elliptic has negative gain at first SOS. It still seems like a bug somewhere. Is there a way to discuss with the author or the maintainer for the background? I would like to have an idea if the analog side has seen much real design usage or has it mostly been a numerical exercise tool, where these issues do not matter at all.
There are two easily expected gain normalization points for analog filters, 0/inf frequency and peak response. Those two would nicely cover most of the cases. Every analog filter designer expects the normalization at 0/inf frequency.
For the filter designer a parametric pole-zero fitter would be a nice addition. The one that allows free number of poles and zeros, both complex and real to be fitted optimally for a specified response mask. Without the limitations of the classical filter functions.
I knew pyFDA, but unfortunately it seems there is no interest for the analog side.
from scipy.signal import butter, ellip, sos2tf, freqs
from scipy import __version__
import matplotlib.pyplot as plt
import numpy as np

if __version__ != '1.10.1':
	print('Warning: SciPy version '+__version__+' does not match the original bug report.')

n = 4 # Plots for orders of 3, 4, 5 and 6 are very informative

fig, axs = plt.subplots(2,4)
axs[0,0].title.set_text('Elliptic LP')
axs[0,1].title.set_text('Butterworth LP')
axs[0,2].title.set_text('Elliptic HP')
axs[0,3].title.set_text('Butterworth HP')

w0 = 100

sos=ellip(n, .1, 60, w0, 'lp', analog=True, output='sos')

for s in sos:
	w, h = freqs(sos2tf([s])[0], sos2tf([s])[1]);axs[0,0].semilogx(w, 20 * np.log10(abs(h)));

sos=butter(n, w0, 'lp', analog=True, output='sos')
for s in sos:
	w, h = freqs(sos2tf([s])[0], sos2tf([s])[1]);axs[0,1].semilogx(w, 20 * np.log10(abs(h)));

sos=ellip(n, .1, 60, w0, 'hp', analog=True, output='sos')
for s in sos:
	w, h = freqs(sos2tf([s])[0], sos2tf([s])[1]);axs[0,2].semilogx(w, 20 * np.log10(abs(h)));

sos=butter(n, w0, 'hp', analog=True, output='sos')
for s in sos:
	w, h = freqs(sos2tf([s])[0], sos2tf([s])[1]);axs[0,3].semilogx(w, 20 * np.log10(abs(h)));

w0 = 1

sos=ellip(n, .1, 60, w0, 'lp', analog=True, output='sos')
for s in sos:
	w, h = freqs(sos2tf([s])[0], sos2tf([s])[1]);axs[1,0].semilogx(w, 20 * np.log10(abs(h)));

sos=butter(n, w0, 'lp', analog=True, output='sos')
for s in sos:
	w, h = freqs(sos2tf([s])[0], sos2tf([s])[1]);axs[1,1].semilogx(w, 20 * np.log10(abs(h)));

sos=ellip(n, .1, 60, w0, 'hp', analog=True, output='sos')
for s in sos:
	w, h = freqs(sos2tf([s])[0], sos2tf([s])[1]);axs[1,2].semilogx(w, 20 * np.log10(abs(h)));

sos=butter(n, w0, 'hp', analog=True, output='sos')
for s in sos:
	w, h = freqs(sos2tf([s])[0], sos2tf([s])[1]);axs[1,3].semilogx(w, 20 * np.log10(abs(h)));

axs[0,0].grid();axs[0,1].grid();axs[0,2].grid();axs[0,3].grid()
axs[1,0].grid();axs[1,1].grid();axs[1,2].grid();axs[1,3].grid()

plt.show()

# ?? Conjugate pole-pair ordering...
# iirdesign output zpk has different conjugate pair ordering than tf2zpk that has neat ordered pairing

I think your solution, @roryyorke, does work at least for a Butterworth LP filter.

FWIW, the method I posted 3 weeks ago is almost certainly overkill; one could just geometrically divided k between the sections using an nth-root, something like this (untested):
    k = sos[0,2]
    sos[0,:2] /= k
    sos[:,:2] *= k**(1/sos.shape[0])

Another way would be to go via zpk notation:
z, p, k = signal.sos2zpk(sos)
sos = signal.zpk2sos(z, p, 1.)
G_per_stage = k**(1/sos.shape[0])

Thanks for all the ideas in this very informative thread! Imho, some of the problems could be mitigated with a few remarks in the docs. For example, it may be clear that coefficients are ordered by index, but it's only ever explicitly written down in rather random places (e.g. in tf2zpk). Could this be a small additional section in the User's Guide?


Merge/release v1.6.1 to release v1.7.0

Rationale
Merging release/v1.6.1 into release/v1.7.0 and bumping the version numbers.
Changes

Version Numbers
Other changes added to the v1.6.1 branch

docker compose default variables (#134)

Checklist

[x] This PR maintains parity between Docker Compose and Helm

Testing
unit tests pass.
To Do
After this is merged, I will delete the release/v1.6.1 branch.

make test-integration-compose seems to fail on this branch
=== RUN   TestDockerComposeUpInternalAuth/TestDockerComposeUp/composeDedicatedPlugins
TestDockerComposeUpInternalAuth/TestDockerComposeUp/composeDedicatedPlugins 2024-05-07T18:33:11-04:00 logger.go:66: Running command docker with args [compose --project-name fiftyone-fagjeg -f compose.dedicated-plugins.yaml -f ../../tests/fixtures/docker/compose.override.yaml -f ../../tests/fixtures/docker/compose.override.mongodb.yaml -f ../../tests/fixtures/docker/compose.override.mongodb_plugins.yaml -f ../../tests/fixtures/docker/compose.override.darwin.yaml -f ../../tests/fixtures/docker/compose.override.darwin_plugins.yaml config]
TestDockerComposeUpInternalAuth/TestDockerComposeUp/composeDedicatedPlugins 2024-05-07T18:33:11-04:00 logger.go:66: stat /Users/topher/workspace/fiftyone-teams-app-deploy/release-v1.6.1-to-release-v1.7.0/tests/fixtures/docker/compose.override.mongodb.yaml: no such file or directory
    docker_compose.go:31: error while running command: exit status 14; stat /Users/topher/workspace/fiftyone-teams-app-deploy/release-v1.6.1-to-release-v1.7.0/tests/fixtures/docker/compose.override.mongodb.yaml: no such file or directory
--- FAIL: TestDockerComposeUpInternalAuth (0.76s)
    --- FAIL: TestDockerComposeUpInternalAuth/TestDockerComposeUp (0.76s)
        --- FAIL: TestDockerComposeUpInternalAuth/TestDockerComposeUp/compose (0.46s)
        --- FAIL: TestDockerComposeUpInternalAuth/TestDockerComposeUp/composePlugins (0.15s)
        --- FAIL: TestDockerComposeUpInternalAuth/TestDockerComposeUp/composeDedicatedPlugins (0.15s)
FAIL
exit status 1
FAIL    github.com/voxel51/fiftyone-teams-app-deploy/integration/compose        1.026s
make: *** [test-integration-compose-internal] Error 1

so does test-integration-helm
--- FAIL: TestHelmInternalAuth (295.61s)
    --- FAIL: TestHelmInternalAuth/TestHelmInstall (295.61s)
        --- FAIL: TestHelmInternalAuth/TestHelmInstall/builtinPlugins (96.79s)
        --- FAIL: TestHelmInternalAuth/TestHelmInstall/sharedPlugins (99.87s)
        --- FAIL: TestHelmInternalAuth/TestHelmInstall/dedicatedPlugins (98.95s)
FAIL
exit status 1
FAIL    github.com/voxel51/fiftyone-teams-app-deploy/integration/helm   296.280s
make: *** [test-integration-helm-internal] Error 1

Maybe I have these configured incorrectly, but I followed the directions (the helm integration directions seem to do duplicate things in step two... but maybe I'm wrong)

make test-integration-compose seems to fail on this branch

The log output includes:

tests/fixtures/docker/compose.override.mongodb.yaml: no such file or directory

I had to force add that file. See https://github.com/voxel51/fiftyone-teams-app-deploy/pull/138/commits/cd59b768cc57186a35d372e85d1b1f4d38c10177.
I don't expect the integration tests to pass yet.  We blindly replace/update version numbers and set container image tags to versions of non-existent artifacts:
From the kubectl describe fiftyone-app ...:

Back-off pulling image "us-central1-docker.pkg.dev/computer-vision-team/dev-docker/fiftyone-app:v1.7.0rc2"

After https://github.com/voxel51/fiftyone-teams-app-deploy/pull/138/commits/d6493bfe376a8b41d3137ac1f60b6dc51d8dcaf9 integration tests now run and pass

Deleted the branch release/v1.6.1.


need hotkey for 'continue'

I really need hotkey for  'continue' when debugging

I completely agree. What could be the hotkey?

f8 would be good

F8 is already used to "Stop the scene". Perhaps something like Ctrl+F5.

how about f4 or f12

and then they were victims of a terrible fate as, two months later, no one has heard of them again

I like ctrl-f5, will add that

no, wait a second, pause button actually works as resume.. so this is not needed anymore, just use F7.


Pop is not defined

Hi,
After trying to list the properties I get a message: pop is not defined. Account is slected as instructed.

Same here,  ReferenceError: pop is not defined , doesn't matter if I select 1 or many properties. Hence the GA4 property details sheet doesn't get filled.

This issue has been resolved in the code in the template sheet. An update to the github repo code will be forthcoming.


Make Visual Studio 2019 a prerequisite to building this repo

We're preparing to consume some changes soon which will require Visual Studio 2019 (C# 8, updated .NET Core 3 SDKs, etc.)
Changes:

Updated documentation in aspnet/AspNetCore
Updated InstallVisualStudio.ps1 scripts
Updated .sln files to require VS >= 16
Updated CI config to use agents with VS 2019 installed

Blocked on https://github.com/dotnet/core-eng/issues/5029

We need a plan for https://github.com/aspnet/AspNetCore-Internal/issues/1764 as part of this.

/AzurePipelines run AspNetCore-ci

We need a plan for aspnet/AspNetCore-Internal#1764 as part of this.

Agreed. I believe the plan is to keep running tests on TeamCity until Helix comes online (wip right now -> #6728 ). I'm updating this PR to remove the ANCM build jobs for now.
I believe the core-eng team is rolling out the updated VS2019 agents today, so we should be unblocked on this now. I'd like to merge soon, so PTAL, reviewers.

Build fails with an new error. I can't tell for sure, but my guess is it's a problem building the Blazor VSIX. @SteveSandersonMS @rynowak @ajaybhargavb  have you seen this before?
2019-02-05T23:03:00.8270212Z Unhandled Exception: System.Runtime.Serialization.SerializationException: Unable to find assembly 'Microsoft.VisualStudio.CodeAnalysis, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
2019-02-05T23:03:00.8272220Z    at System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly()
2019-02-05T23:03:00.8273711Z    at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(BinaryAssemblyInfo assemblyInfo, String name)
2019-02-05T23:03:00.8280961Z    at System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor(String objectName, String[] memberNames, BinaryTypeEnum[] binaryTypeEnumA, Object[] typeInformationA, Int32[] memberAssemIds, ObjectReader objectReader, Int32 objectId, BinaryAssemblyInfo assemblyInfo, SizedArray assemIdToAssemblyTable)
2019-02-05T23:03:00.8288479Z    at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped record)

https://dev.azure.com/dnceng/public/_build/results?buildId=88221&view=logs&jobId=f31c9f97-4411-58e7-49ac-fc73f645e6b6&taskId=6eab48b9-b4ce-58d4-7c0f-2d2ef2bbc0bd&lineStart=2937&lineEnd=2943&colStart=1&colEnd=103

Blocked again on https://github.com/Microsoft/msbuild/issues/4141.
@SteveSandersonMS @rynowak @ajaybhargavb - the error is a red herring. Something in MSBuild itself is failing and hiding the real error.

I've updated this PR to use .NET Core SDK 3.0 Preview 2 and still getting this error.

Unhandled Exception: System.Runtime.Serialization.SerializationException: Unable to find assembly 'Microsoft.VisualStudio.CodeAnalysis, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

On Linux/macOS, there is a new error: @ajaybhargavb @NTaylorMullen @pranavkm  - what is your recommendation for fixing this? My guess is happens as a result of having only some projects in the repo using Microsoft.NET.Sdk.Razor as a PackageReference, and others using the one that ships with the Web SDK. Could we remove Microsoft.NET.Sdk.Razor  as a package reference completely?

Microsoft.NET.Sdk.Razor.CodeGeneration.targets(99,5): error MSB4062: The "Microsoft.AspNetCore.Razor.Tasks.RazorTagHelper" task could not be loaded from the assembly /Users/vsts/agent/2.146.0/work/1/s/.nuget/packages/microsoft.net.sdk.razor/3.0.0-preview-19104-04/build/netstandard2.0/../../tasks/netstandard2.0/Microsoft.NET.Sdk.Razor.Tasks.dll. Assembly with same name is already loaded Confirm that the  declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask. [/Users/vsts/agent/2.146.0/work/1/s/src/Security/samples/ClaimsTransformation/ClaimsTransformation.csproj]

Could we remove Microsoft.NET.Sdk.Razor as a package reference completely?

No, that becomes problematic if there are breaking changes in the SDK with corresponding runtime changes (and we've had a few rounds of these in the past). It makes sense to treat it as just as any other partner dependency and uniformly. I'll get the Security project sorted.

Welp, next layer of the upgrade onion -- https://github.com/NuGet/Home/issues/7706

/AzurePipelines run AspNetCore-ci (Build: Windows x64/x86)

Ok, found a workaround for MSBuild crashing (manually apply this change to msbuild.exe.config https://github.com/Microsoft/msbuild/pull/4139)
Next error: there are 3 compiler errors in C++ projects with VS2019. Not sure if this is the result of updating the platform toolset to v142 to workaround this bug, or the result of other changes in VS2019.
Signalr Client. @BrennanConroy - any ideas why this header is missing?
C:\src\aspnet\AspNetCore\src\SignalR\clients\cpp\src\signalrclient\stdafx.h(21): fatal error C1083: Cannot open include
 file: 'cpprest/details/basic_types.h': No such file or directory [C:\src\aspnet\AspNetCore\src\SignalR\clients\cpp\src
\signalrclientdll\Build\VS\signalrclientdll.vcxproj]
C:\src\aspnet\AspNetCore\src\SignalR\clients\cpp\src\signalrclient\stdafx.h(21): fatal error C1083: Cannot open include
 file: 'cpprest/details/basic_types.h': No such file or directory [C:\src\aspnet\AspNetCore\src\SignalR\clients\cpp\src
\signalrclient\Build\VS\signalrclient.vcxproj]

ANCM - @jkotalik is this a legit bug in our code that we should fix?
c:\src\aspnet\aspnetcore\src\servers\iis\aspnetcoremodulev2\aspnetcore\servererrorapplication.h(32): error C26487: Don'
t return a pointer '(*pRequestHandler)' that may be invalid (lifetime.4). [C:\src\aspnet\AspNetCore\src\Servers\IIS\Asp
NetCoreModuleV2\AspNetCore\AspNetCore.vcxproj]

Yeah, v142 will not work with the cpprestsdk and I don't see anything in there issue tracker for adding support for it.

@BrennanConroy is there a related issue you can add this comment to? If not, shouldn't we file one? Let's not stay blocked for long without an issue tracking it.

Ok, I've opened https://github.com/Microsoft/cpprestsdk/issues/1039. In the meantime, I'm going to try reseting back to v141 and attempt to workaround https://developercommunity.visualstudio.com/content/problem/434385/vs2019-preview-2-targetframeworkversion-or-platfor.html

Next error:

C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\MSBuild\Current\Bin\Microsoft.CSharp.CurrentVersion.targets(321,5): error MSB4019: The imported project "C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\MSBuild\Current\Bin\Roslyn\Microsoft.CSharp.Core.targets" was not found. Confirm that the path in the  declaration is correct, and that the file exists on disk.

Seems like the installation of VS on our CI machine is busted.

There is no good workaround for https://developercommunity.visualstudio.com/content/problem/434385/vs2019-preview-2-targetframeworkversion-or-platfor.html, but according to the internal thread, this should be resolved in VS 2019 Preview 3. Let's wait until we have Preview 3 instead of trying to hack our way around this bug in the C++ tooling.

Do we have a rough ETA for that? I would almost recommend temporarily removing the C++ client from our build until then, as I'm the only one who will be really working on it.

It's not just SignalR. ANCM stuff fails to compile as well.
See the schedule for an ETA...not sure if I can disclose this on GitHub.

But ANCM was fine updating to 142 right?

ANCM fails with v142

c:\src\aspnet\aspnetcore\src\servers\iis\aspnetcoremodulev2\aspnetcore\servererrorapplication.h(32): error C26487: Don't return a pointer '(*pRequestHandler)' that may be invalid (lifetime.4). [C:\src\aspnet\AspNetCore\src\Servers\IIS\AspNetCoreModuleV2\AspNetCore\AspNetCore.vcxproj]

Oh, that should be an easy fix, either turn off the code analysis rules or fix the rule break. @pakrym @jkotalik

I've opened https://github.com/aspnet/AspNetCore/issues/7548 and https://github.com/aspnet/AspNetCore/issues/7549 to track updating the platform toolset in a separate PR. The CI agents have been updated to include a workaround for the C++ tooling issue.

Next problem: when building with VS2019, the C++ projects fail due to a header which cannot be found. @jkotalik @pakrym - any ideas?
e:\a\_work\1\s\src\servers\iis\aspnetcoremodulev2\iislib\precomp.h(10): fatal error C1083: Cannot open include file: 'atlcomcli.h': No such file or directory (compiling source file acache.cpp) [E:\A\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\IISLib\IISLib.vcxproj]
e:\a\_work\1\s\src\servers\iis\aspnetcoremodulev2\iislib\precomp.h(10): fatal error C1083: Cannot open include file: 'atlcomcli.h': No such file or directory (compiling source file ahutil.cpp) [E:\A\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\IISLib\IISLib.vcxproj]
e:\a\_work\1\s\src\servers\iis\aspnetcoremodulev2\iislib\precomp.h(10): fatal error C1083: Cannot open include file: 'atlcomcli.h': No such file or directory (compiling source file base64.cpp) [E:\A\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\IISLib\IISLib.vcxproj]
e:\a\_work\1\s\src\servers\iis\aspnetcoremodulev2\iislib\precomp.h(10): fatal error C1083: Cannot open include file: 'atlcomcli.h': No such file or directory (compiling source file multisz.cpp) [E:\A\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\IISLib\IISLib.vcxproj]
e:\a\_work\1\s\src\servers\iis\aspnetcoremodulev2\iislib\precomp.h(10): fatal error C1083: Cannot open include file: 'atlcomcli.h': No such file or directory (compiling source file multisza.cpp) [E:\A\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\IISLib\IISLib.vcxproj]
e:\a\_work\1\s\src\servers\iis\aspnetcoremodulev2\iislib\precomp.h(10): fatal error C1083: Cannot open include file: 'atlcomcli.h': No such file or directory (compiling source file stringa.cpp) [E:\A\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\IISLib\IISLib.vcxproj]
e:\a\_work\1\s\src\servers\iis\aspnetcoremodulev2\iislib\precomp.h(10): fatal error C1083: Cannot open include file: 'atlcomcli.h': No such file or directory (compiling source file stringu.cpp) [E:\A\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\IISLib\IISLib.vcxproj]
e:\a\_work\1\s\src\servers\iis\aspnetcoremodulev2\iislib\precomp.h(10): fatal error C1083: Cannot open include file: 'atlcomcli.h': No such file or directory (compiling source file util.cpp) [E:\A\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\IISLib\IISLib.vcxproj]

@natemcmaster I'll see if we can drop it.

In the meantime, I'm adding "Microsoft.VisualStudio.Component.VC.v141.ATL" as a required workload.

Nope, we use quite a bit of types from it. No easy way to drop.

/azp run aspnetcore-ci

Status update: we're waiting until we have more VS2019 agents on CI. We currently have 1 shared with the whole dotnet team, which is not going to be sufficient.
In the meantime, I see the code-check step still consistently fails. For some unclear reason, PowerShell is treating output to stderr as a fatal error, but only on the VS2019 agent machines. This script works just find locally (Win 10, PowerShell 5.1) and on the Hosted VS2017 pool, but fails consistently in the PR check.
Run git diff to check for pending changes

Summary:

   0 error(s)

##[error]git : warning: LF will be replaced by CRLF in eng/ProjectReferences.props.
##[error]At E:\A\_work\5\s\eng\scripts\CodeCheck.ps1:132 char:21
+ ... ngedFiles = git --no-pager diff --ignore-space-at-eol --name-only 2>  ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (warning: LF wil...ferences.props.:String) [], RemoteException

##[error]    + FullyQualifiedErrorId : NativeCommandError

Resolved the git.exe problem by redirecting stderr to stdout and filtering the warning messages.
I believe this is ready for final review and ready to merge (pending CI checks, of course).

 It looks like they fall into 3 categories:

HttpSys response caching is broken (cc @Tratcher any ideas?)
MusicStore - running dotnet publish fails, but not error is present in logs
MusicStore - some tests timeout trying to update the database

@natemcmaster - HttpSys - No, nothing should have changed here. The thing most likely to destabilize those tests it is changing the OS version.

Yeah, that might be it. I've updated to make windows RIDs in test consistent across all projects.

All required checks passed.

"The version of the selected solution is higher than the solution version supported by this Visual Studio release. Please try downloading the latest service pack for this Visual Studio release, or opening in a more recent version of Visual Studio, if available."
Hosting.sln
Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 16
VisualStudioVersion = <IP_ADDRESS>
MinimumVisualStudioVersion = 16.0.28608.199

Microsoft Visual Studio Enterprise 2019 Preview
Version 16.0.0 Preview 3.0
VisualStudio.16.Preview/16.0.0-pre.3.0+28608.199
Microsoft .NET Framework
Version 4.7.03190
Installed Version: Enterprise

Hmm, setting a specific build number of VS worked in P2. I wonder if something changed in P3. I'll reset the .sln files to <IP_ADDRESS>.

MinimumVisualStudioVersion = 16.0.28606.0 seems to be the highest value that works.

Hi. What is the status here? I dont see any cpprestsdk.v142 nuget package.

@jkotalik - Can you answer @petke's question?

@Pilchie. Sorry, I commented on the wrong project. So no need to answer me. Thanks

No worries.


Issues with Admission Method - default code clash

ADMISSION_METHOD:
Not sure what has happened with the last row:

Admission via Mental Health Crisis Resolution Team is code 25
There is no Gate-Kept Mental Health Admission 41 in the DD
The codes are now on the Data Element, but the note is still on the Attribute
This is the same for others, e.g. ADMISSION METHOD
This needs addressing as the notes may be missed.

21/07/20 - Checked on ontoserver and beta site (June 2020 release)
code 25 is correct
No code 41 found in code system
Notes are being adapted/added by DD team
However - default code clash noted for 99

Default code 99 is wrong on ontoserver. There is a default code clash between the two elements for this attribute...so should NOT be getting sent to term server.
ADMISSION METHOD CODE (MOTHER LABOUR AND DELIVERY HOSPITAL PROVIDER SPELL)
99 - Not Known (Not Recorded)
ADMISSION METHOD CODE (HOSPITAL PROVIDER SPELL)
99 - Not known: a validation error

We can't fix this now as both items are used in different data sets.

Likin - Please investigate why default codes are not getting sent in knitted txa for items where there is no ontoserver content.

Text on the default codes will be fixed in the October release

Can see that default codes now match on v3. site. However, not picked up by FHIR generation - old default codes on new site for Oct release. Also, content not on ontoserver.

These items are now getting sent to Ontoserver correctly - From Jan 21 release onwards. Current day looks correct.
Business decision needed on if the gap in the series of items (March 2019 to Jan 2021) due to the default code clash needs a fix to correct history. If we don't - helpdesk will have a record of these dates if any queries come in.

This is only an issue for the Terminology Server. Cath will talk to Maxine about this.

Maxine said "am waiting for final run and error file in a digestible format so that we can agree whether or not we need to something clever (manually) to plug the historic gap."

the default code clash is no longer present. Hence the marking as fixed


Very Slow Launch and High CPU Usage

Storage Explorer Version: 1.3.1
Platform/OS Version:  Windows
Architecture: ia32
Regression From: It was fine but I don't know where the old downloads are to tell you. I was going to try a previous version before posting here. Also happened with 1.0.0. Where are the previous versions?
Steps to Reproduce:

Download and install this program from https://azure.microsoft.com/en-us/features/storage-explorer/
Do a clean boot https://support.microsoft.com/en-in/help/929135/how-to-perform-a-clean-boot-in-windows
Launch Task Manager
Launch this application

Expected Experience:
To launch faster and not consume so much CPU.
Actual Experience:
Watch as 60+ processes are created and CPU spikes to 99% for 10+ seconds if doing a clean boot. Increases a lot more if not doing a clean boot.
This is on an i7-3740QM, if that helps and on an SSD.
Original issue: https://social.msdn.microsoft.com/Forums/en-US/ad029871-462a-4815-b97b-29fd5d361068/azure-storage-explorer-high-cpu-usage?forum=windowsazuredata

Marking this as duplicate of #166. The bug does not appear to be restricted to Windows 10.


[IDE] cannot import src/main/scala & src/test/scala into eclipse as sour...

...ce folder
When import the whole project into eclipse as maven project, found that the
src/main/scala & src/test/scala can not be set as source folder as default
behavior, so add a "add-source" goal in scala-maven-plugin to let this work.

Can one of the admins verify this patch?

Hm, I wonder why? src/main/scala and src/test/scala should be the default locations picked up by the Maven compiler / scala plugins. You shouldn't have to configure anything else. Has anyone else reported this behavior? I'd be surprised if Eclipse has a particular problem since IJ (and Maven) don't.

I'm not sure why it has this behavior, but the eclipse surely act as that. I've tested in  Eclipse 3.8 and 4.3 with scala-plugin and scala-maven-plugin in both linux and windows platform, and got the same results. And after add this add-source goal, the src/main/scala will be imported as source folder which is reasonable.

Can anyone else using Eclipse comment? I know people have been using Spark in Eclipse successfully already.

ok to test

Test build #27396 has started for   PR 4531 at commit 4e4db4c.

This patch merges cleanly.

Test build #27396 has finished for   PR 4531 at commit 4e4db4c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27396/
Test PASSed.

OK. Can't really hurt as the effect should just be to establish src/main/scala as a source root, which it already should be. This is small, but could you make a quick JIRA for this one and update the title with SPARK-xxxx [BUILD] ...?


Theme support

How do we apply themes as described here?
Using:
<style type="text/sass" global>
@import "node_modules/carbon-components/scss/globals/scss/vendor/@carbon/themes/scss/themes";

$carbon--theme: $carbon--theme--g100;
@include carbon--theme();

@import 'node_modules/carbon-components/scss/globals/scss/styles';
</style>

Seems to have no effect at all.

Nevermind, this DOES work, it was a problem on how i was handling my rollup.
The snippet bellow works like charm if you wan't use the Gray-100 theme:
@import "carbon-components/scss/globals/scss/vendor/@carbon/themes/scss/themes";

$carbon--theme: $carbon--theme--g100;
@include carbon--theme();

@import 'carbon-components/scss/globals/scss/styles';

@NickStarlight I'm having this same issue, but it's not clear what I need to do in my rollup to fix this. Do you have any information on how to do this?

Aha! I see that the path you used is different than the path recommended on the main documentation. Adjusting the path fixes things.


Determining "primary" asset

When consuming a STAC Item without detailed knowledge of vendor-specific properties or filename conventions, I'd like to know which asset should be treated as the "primary" one.
Specifically, when pointing STAC Browser at a catalog, I'd like to make an informed choice about which asset to provide an interactive preview for. Since assets is an object, there's no "first" item to use as a default.
We have the thumbnail key (which should also be rel=thumbnail?). Should we also introduce a primary key (and rel) for this purpose?
(Additional keys / rels are already relegated to extension / vendor-specific use, which should handle use-cases like "analytic")

In the case of datasets with bands split into individual files (Landsat 8), multiple assets may be "primary".

What I don't really get: Is primary just a special thumbnail or could it also be used for other purposes? I'm wondering whether we could come up with a more intelligent way of determining the "primary" asset. We would need it anyway for cases without a primary in the assets - or should primary be required? Not sure whether there is always a primary asset though. I am also wondering how we would name the keys for multiple thumbnails. thumbnail, thumbnail2, etc? I agree that asset rel seems very useful (#309) as the asset keys are somewhat restricted. I think we should wait for the asset discussions in 0.7.0 (dict vs. list, asset schemas, etc). We may find solutions there (e.g. using arrays).

"primary" would be the primary dataset(s) (not thumbnail) associated with an Item--all other items would be derivatives / sidecars, e.g. QA band, etc. Maybe "visual" is actually what I'm looking for here.
"analytic" would be another rel, at least from the Planet perspective.

I definitely see a need for this. I see it as a 'hint' that a data provider gives to help clients determine what to 'display'. It is more than just a thumbnail, ideally it's the full Cloud Optimized GeoTIFF that can be fully browsed. Indeed having STAC refer to COG's to me is one of the most compelling features, as data can be used right there. But you may have an Unusable Data Mask and a 4-band analytic product both as COG's, and you need a way to hint which should be used in a COG slippy map. So I wouldn't see primary as being required, indeed Items with just one asset wouldn't really need it.
I've also wondered about making the dictionary names useful - like defining some common ones.
But I agree we need to have a fuller discussion for 0.7.0 on asset definitions in general, and keep this bit in mind as something we need to do.
@mojodna - I do think we can experiment with this with STAC browser and a couple of the repositories to show how it would work.

It's certainly a valid use case, but I think you're on the right track that 'primary' isn't the right term here. A 'primary' asset completely depends on the use case, and it's really a vague term.
'visual', 'true-color', 'rgb' are all terms that I think more accurately describes this. You are looking for which asset or assets are needed to display as a byte-scaled 3 color image.
This very much fits in with the asset definition talk. Perhaps part of the asset definition is specifying certain combinations of assets that give a desired set of data:
'usecase': {
'visual': [ 'B4', 'B3', 'B2'],
'ard': [ 'B1', 'B2', 'B3', 'B4', 'B5', 'B6', 'B7', 'B8'],
'qa': ['BQA']
}
In the first use case it tells me that assets with those keys form a 3 band color image, in that order.
In the second use case it tells me that assets with those keys are part of a 3d raster set of data, and it excludes asset keys that represent non-raster sources of data, or QA bands.
In the third use case it tells me which assets are used for quality information.
In reality, the situation is even more complicated than this. In fact, I could have the same assets be used for a visual usecase, as for an ARD use case, but supplying different scales and offsets to the data.
Which is why...we need to discuss this with asset definitions.

See #316 for proposed changes.
(Agreed that "primary" isn't the right term.)

Since multiple assets serve a single role, the order of those need to be captured somewhere. It is not good enough to just state that 3 bands are used to make a visual image.

They don't necessarily need to be ordered; the DG images don't overlap (and are each 3 band), so they can be rendered in any order.
Moving assets back to a list would provide a preliminary ordering to draw from.

In Earth Engine, we sort of denote the primary "assets" when we specify the preferred visualization for rasters. In most cases this is just the list of bands, plus maybe gain/bias or min/max, plus maybe a palette. Also a lat/lon/zoom with an interesting viewport. This is done at the collection level - we don't have cases where different items in a collection need different primary assets.

I also wondered this morning where/how we could place a collection level thumbnail.

Rebooting this conversation in light of the new asset-level "role" attribute.
Here's an example of how I'd see this working, with roles "overview", "thumbnail", "analytic", and "visual".  (open question - should roles be an array rather than a string?)
I feel a key part here is that I should be able to find assets by their description, without needing to know the implementation-specific, arbitrary object keys that have been assigned to them.  I think of it it like a decision tree, that I might first look for all the assets with "eo:common_name == green", then select by "role == analysis" (not visual), then select by "gsd == 10" (not 60), then by media type "type == image/jp2" (not cog).
      "assets": {
        "TCI_60m": {
          "eo:gsd": 60.0,
          "href": "s3://sentinel-s2-l2a/tiles/46/V/CQ/2019/5/27/0/R60m/TCI.jp2",
          "size": 616584,
          "title": "True Color Composite",
          "type": "image/jp2",
          "role": "overview"
        },
        "TCI_60m_jpg": {
          "eo:gsd": 60.0,
          "href": "s3://sentinel-s2-l2a/tiles/46/V/CQ/2019/5/27/0/R60m/TCI.jpg",
          "size": 616584,
          "title": "True Color Composite",
          "type": "image/jpg",
          "role": "overview"
        },
        "TCI_thumb": {
          "eo:gsd": 60.0,
          "href": "s3://sentinel-s2-l2a/tiles/46/V/CQ/2019/5/27/0/preview.jpg",
          "size": 10584,
          "title": "True Color Composite",
          "type": "image/jpg",
          "role": "thumbnail"
        },
        "B03_60m": {
          "eo:common_name": "green",
          "eo:gsd": 60.0,
          "href": "s3://sentinel-s2-l2a/tiles/46/V/CQ/2019/5/27/0/R60m/B03.jp2",
          "size": 616584,
          "title": "Band 3 - Green 0.560μm (60m)",
          "type": "image/jp2",
          "role": "analytic"
        },
      "B03_10m": {
          "eo:common_name": "green",
          "eo:gsd": 10.0,
          "href": "s3://sentinel-s2-l2a/tiles/46/V/CQ/2019/5/27/0/R10m/B03.jp2",
          "size": 8016584,
          "title": "Band 3 - Green 0.560μm (60m)",
          "type": "image/jp2",
          "role": "analytic"
        },
      "B03_10m_cog": {
          "eo:common_name": "green",
          "eo:gsd": 10.0,
          "href": "s3://sentinel-s2-l2a/tiles/46/V/CQ/2019/5/27/0/R10m/B03.TIF",
          "size": 8016584,
          "title": "Band 3 - Green 0.560μm (60m)",
          "type": "image/tiff; application=geotiff",
          "role": "analytic"
        },
        "B03_preview": {
          "eo:common_name": "green",
          "eo:gsd": 60.0,
          "href": "s3://sentinel-s2-l2a/tiles/46/V/CQ/2019/5/27/0/preview/B03.jpg",
          "size": 6584,
          "title": "Band 3 - Green 0.560μm (60m)",
          "type": "image/jpeg",
          "role": "visual"
        },
}

open question - should roles be an array rather than a string?

Yes, maybe it should. Similarly to the roles for the providers.

Should be solved with #651


Broken in Chrome 49.0.2620.0 canary (64-bit)

My chrome updated during the night and now my stuff doesn't work anymore, fun times, I get to debug!
The first thing was actually already fixed by 5f5478a34e8ed7b9642407b062e8c5edc6799510, I just had to update to the latest version. @feross Was that commit intentionally fixing a Chrome bug?
The next problem is that .subarray(start, end) seems broken when the __proto__ is tampered with.

The following code fixes the problem but I'm not sure that it's the right approach:
diff --git a/index.js b/index.js
index 15d7c50..cbb1516 100644
--- a/index.js
+++ b/index.js
@@ -782,7 +782,9 @@ Buffer.prototype.slice = function slice (start, end) {

   var newBuf
   if (Buffer.TYPED_ARRAY_SUPPORT) {
+    this.__proto__ = Uint8Array.prototype
     newBuf = this.subarray(start, end)
+    this.__proto__ = Buffer.prototype
     newBuf.__proto__ = Buffer.prototype
   } else {
     var sliceLen = end - start

No, https://github.com/feross/buffer/commit/5f5478a34e8ed7b9642407b062e8c5edc6799510 was just changing Safari 5-7 to use the typed array implementation.
It sounds like Chrome broke something in Canary. I opened an issue on the V8 tracker so we can fix this before this makes it farther in the release cycle. https://bugs.chromium.org/p/v8/issues/detail?id=4665

Hehe, that is some timing though :laughing:
The output of this script in the above stated version of Chrome alerts false and true. In Chrome Stable, Version 47.0.2526.106 (64-bit), it alerts true and true.
function typedArraySupportBroken () {
  function Bar () {}
  try {
    var arr = new Uint8Array(1)
    arr.foo = function () { return 42 }
    arr.constructor = Bar
    return arr.foo() === 42 && // typed array instances can be augmented
        arr.constructor === Bar && // constructor can be set
        typeof arr.subarray === 'function' && // chrome 9-10 lack `subarray`
        arr.subarray(1, 1).byteLength === 0 // ie10 has broken `subarray`
  } catch (e) {
    return false
  }
}

function typedArraySupportWorks () {
  try {
    var arr = new Uint8Array(1)
    arr.foo = function () { return 42 }
    return arr.foo() === 42 && // typed array instances can be augmented
        typeof arr.subarray === 'function' && // chrome 9-10 lack `subarray`
        arr.subarray(1, 1).byteLength === 0 // ie10 has broken `subarray`
  } catch (e) {
    return false
  }
}

alert(typedArraySupportBroken())
alert(typedArraySupportWorks())

Conclusion: The constructor cannot be set in Chrome 49.

Conclusion: The constructor cannot be set in Chrome 49.

It's actually throwing the exception on arr.subarray(1, 1). But it only does that because the constructor property was set.
Crazy timing.

Actually, the tests aren't failing because arr.subarray is throwing an exception. They're failing because it's returning the full array!
function Buffer (arg) {
  var arr = new Uint8Array(arg)
  arr.__proto__ = Buffer.prototype
  return arr
}
Buffer.prototype.__proto__ = Uint8Array.prototype
Buffer.__proto__ = Uint8Array

var buf = new Buffer(10)

var buf2 = buf.subarray(2)

console.log(buf2.length) // returns 10 in Canary, instead of 8 as expected

That's true, I must have missed typing out the end of that sentence :)
Hopefully this will get fixed sooner rather than later in Chrome and than we won't need to do anything at all here...

Yeah, I think let's wait a couple days. If there's no indication they plan to fix in Chrome, we can do a new release with your PR and add a big TODO to the code to remove that as soon as possible.

This has already been fixed in Chrome Canary. Thanks for the bug report.

Closing based on https://github.com/feross/buffer/pull/95#issuecomment-172240690


added missing , in README.md

found it through js2.coffee, which refused to parse it without it, gotta love coffee :)

Duplicate.


Adds CI build for the mbed example

Adds a CI build for the mbed example

 Development Examples: remove hard-coded PRODUCT_UIDs from sample apps and example repos.