bitrate limit with -hq

Hello, I have a question. Is there a way to leave the -hq argument, but slightly reduce the bitrate for it? It's just that at moments where there is almost no movement, the bitrate is about 4-7 thousand. I would like to somehow limit it for low-activity scenes, but do not touch it for active ones, it suits me. Well, or at least reduce it a little for all the scenes

There are two ways I can see to go about this. First is that you could manually specify a maximum bitrate. Alternatively, you change the target crf (constant rate factor) that you tell ffmpeg to target when you encode. From your description, it sounds like you want to keep the ability of the encoder to go to high bitrate when needed, but try to reduce the overall bitrate if possible. So, with that in mind, I would recommend adjusting your crf.
I should mention up top that diving into video encoder tuning is a deep, deep well. For a primer, ffmpeg itself has an h264 encoding guide that covers the basics. Everything I am writing is about h264. I don't really have any experience with h265, but that isn't used by default by PySceneDetect.
The -hq flag
From the documentation, the -hq flag passes --rate-factor=17 --preset=slow to ffmpeg. So, by specifying -hq, we are simply changing the crf to 17 (lower numbers mean higher quality) and using the slow preset in ffmpeg. You can compare this to the defaults used by PySceneDetect below:

mode
without -hq
with -hq

crf
22
17

preset
slow
veryfast

The -a flag
If you want to tune your target crf outside of the options above, then you will need to manually specify the arguments you pass to the ffmpeg encoder. As you can see, the default is a crf of 22 while hq has a crf of 17. So, you will want to choose a value between those two numbers. Also, choosing a slower preset can help with compression at the cost of taking a lot longer to encode. So, an example might be specifying a crf of 20 with a slow preset.
To specify the custom ffmpeg arguments, let's take the default arguments and simply modify them to our needs. The documentation lists the default parameters as

Default ffmpeg parameters:

-map 0:v:0 -map 0:a? -map 0:s? -c:v libx264 -preset veryfast -crf 22 -c:a aac

Setting crf to 20 and using a slow preset:

-map 0:v:0 -map 0:a? -map 0:s? -c:v libx264 -preset slow -crf 20 -c:a aac

Capping the bitrate
If you wanted to manually specify a maximum bitrate for the video, you would use the -maxrate and -bufsize ffmpeg arguments. So, let's say that you wanted to place a hard cap of 4000 kbps on the example listed above, then you would modify it like this:

Default ffmpeg parameters:

-map 0:v:0 -map 0:a? -map 0:s? -c:v libx264 -preset veryfast -crf 22 -c:a aac

Setting crf to 20 and using a slow preset:

-map 0:v:0 -map 0:a? -map 0:s? -c:v libx264 -preset slow -crf 20 -c:a aac

Additionally capping bitrate at 4000 kbps:

-map 0:v:0 -map 0:a? -map 0:s? -c:v libx264 -preset slow -crf 20 -maxrate 4000k -bufsize 4000k -c:a aac

What the proper crf should be is entirely subjective and depends a lot on the type of content that is being encoded. Usually, content that is largely static or has large, uniform areas of color can get away with a higher crf (news content, animation, etc.). Content that involves more motion, rapid changes, or fine details would demand a lower crf (sports, action, etc.). I am far from an expert on this topic and, if you search around, there is a lot of advice and opinions on the internet about this topic. Hopefully I have armed you with the keywords you need to track down more reading that is relevant to your type of content.

thx very much


Creating a new ng project in a tmp folder, clashes with existing files at the root (different location)

From @sebawita on March 9, 2018 17:17
Bug Report or Feature Request (mark with an x)
- [x] bug report -> please search issues before submitting
- [ ] feature request

Area
- [ ] devkit
- [x] schematics

Versions
"@angular-devkit/core": "^0.4.5",
"@angular-devkit/schematics": "^0.4.5",
"@schematics/angular": "^0.4.5",

Repro steps
Run the below rule, in a folder that already contains any of the files (i.e. /src/app/environments/environment.prod.ts), which the application schematic generates.
This will throw an Error, even though the schematic should create the files in a tmp folder.
const createNgProject = () => {
  const options: ApplicationSchema = {
      name: 'tmp-name',
      directory: 'tmp',
      sourceDir: 'src',
      skipInstall: true
  }
  return externalSchematic('@schematics/angular', 'application', options);
}

The log given by the failure
Error: Path "/app/environments/environment.prod.ts" already exist.
Path "/app/environments/environment.prod.ts" already exist.

Desired functionality
Ideally the application schematic run for a different directory (i.e. tmp), shouldn't clash with the existing files, which are at a different location.
Mention any other details that might be useful
When I provide different value for sourceDir, then the application files get generated without any problems.
However, in my use case I need to use the same sourceDir, as the current project, as I need to extract .angular-cli.json settings, where some of the settings point to sourceDir.
Copied from original issue: angular/devkit#493

Closing as the current application schematic no longer has a directory option.
The applyToSubtree rule can be used to execute a schematic (or any rule) in a particular path.


Typo --ti and missing my_strace_docker_image

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "patch-1" git@github.com:elgalu/docker.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

LGTM
ping @thaJeztah


question: multiphysics in mfem

Hello
I have seen that MFEM have different application areas (fluid, electromagnetics)? It is possible to couple fluid mechanics and electromagnetics? It is certainly possible because  there are projects based on MFEM that must use that coupling (https://www.rfscidac4.org/), but I do not know if this possibility is already included in MFEM or it must be coded by the user.
Thank you.

Generally speaking this is something that would be done at the application level, combining the lower level, more general abstractions of the MFEM library in the desired way.

To add to Bob's answer -- we provide the building blocks and it is up to apps to build whatever multiphysics infrastructure suits them best. One example that could be useful to check is the Joule miniapp in the miniapps/electromagnetics directory.


New data sources: api.energy-charts.info

In my opinion the most complete EPEXSpot API you can found at this time and really easy to integrate.
Provided by:
https://api.energy-charts.info/
curl -X 'GET' \
  'https://api.energy-charts.info/price?bzn=DE-LU' \
  -H 'accept: application/json'

{
  "license_info": "CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de",
  "unix_seconds": [
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>,
<PHONE_NUMBER>
  ],
  "price": [
    93.91,
    86.83,
    86.73,
    83.95,
    84.44,
    91.13,
    91.25,
    91.29,
    100.84,
    109.4,
    110.84,
    117.39,
    118.36,
    116.74,
    111.91,
    121.38,
    105.94,
    113.61,
    98.55,
    97.69,
    91.29,
    90.56,
    88.6,
    75.01
  ],
  "unit": "No Unit",
  "deprecated": false
}

Available bidding zones:
AT - Austria
BE - Belgium
BG - Bulgaria
CH - Switzerland
CZ - Czech Republic
DE-LU - Germany, Luxembourg
DE-AT-LU - Germany, Austria, Luxembourg
DK1 - Denmark 1
DK2 - Denmark 2
EE - Estionia
ES - Spain
FI - Finland
FR - France
GR - Greece
HR - Croatia
HU - Hungary
IT-Calabria - Italy Calabria
IT-Centre-North - Italy Centre North
IT-Centre-South - Italy Centre South
IT-North - Italy North
IT-SACOAC - Italy Sardinia Corsica AC
IT-SACODC - Italy Sardinia Corsica DC
IT-Sardinia - Italy Sardinia
IT-Sicily - Italy Sicily
IT-South - Italy South
LT - Lithuania
LV - Latvia
ME - Montenegro
NL - Netherlands
NO1 - Norway 1
NO2 - Norway 2
NO2NSL - Norway North Sea Link
NO3 - Norway 3
NO4 - Norway 4
NO5 - Norway 5
PL - Poland
PT - Portugal
RO - Romania
RS - Serbia
SE1 - Sweden 1
SE2 - Sweden 2
SE3 - Sweden 3
SE4 - Sweden 4
SI - Slovenia
SK - Slovakia

License info:

"CC BY 4.0 (creativecommons.org/licenses/by/4.0) from Bundesnetzagentur | SMARD.de"
"The data provided herein is for private and internal use only. The utilization of any data, whether in its raw or derived form, for external or commercial purposes is expressly prohibited. Should you require licensing for market-related data, please direct your inquiries to the original data providers, including but not limited to EPEX SPOT SE."

Publishing Notes:
https://energy-charts.info/publishing-notes.html?l=en&c=DE
Fraunhofer Institute for Solar Energy Systems ISE
Heidenhofstr. 2
79110 Freiburg
Tel +49 761 4588-0
Fax +49 761 4588-9000
www.ise.fraunhofer.de
is a constituent entity of the Fraunhofer-Gesellschaft, and as such has no separate legal status.
Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
Hansastraße 27 c
80686 München
Phone: +49 89 1205- 0
Fax: +49 89 1205-7531
www.fraunhofer.de
Responsible Editor:
Prof. Dr. Bruno Burger
<EMAIL_ADDRESS>Additional information according to the Tele Media Act (TMG)
VAT Identification Number in accordance with §27 a VAT Tax Act
DE 129515865
Court of jurisdiction
Amtsgericht München (district court)
Registered nonprofit association
Registration no. VR 4461

Interesting German article about energy-charts.info:
https://www.energiezukunft.eu/erneuerbare-energien/stromnetze-speicher/fakten-statt-fake-news-zur-energiewende


Error after upgrading from Talos OS v1.7.6 -> v1.8.0

Spegel version
v0.0.24 & v0.0.23
Kubernetes distribution
Talos OS
Kubernetes version
v1.30.4
CNI
Cilium
Describe the bug
Spegel has been running perfectly under version 0.0.23 in my cluster for months.
It was installed using helm with the command:

helm upgrade --create-namespace --namespace spegel --install --version v0.0.23 spegel oci://ghcr.io/spegel-org/helm-charts/speg
el   --set spegel.containerdRegistryConfigPath="/etc/cri/conf.d/hosts"

I also set the namespace to privileged with:

kubectl label namespace spegel pod-security.kubernetes.io/enforce=privileged

I have a 3 node Talos OS cluster, and have just upgraded two of the nodes from Talos OS v1.7.6 to v1.8.0.
Spegel has ceased working on those two nodes, I am holding off the upgrade of the third node until I resolve this issue.
Pod logs for the failed pods show:

{"time":"2024-10-04T09:06:53.393977199Z","level":"ERROR","source":{"function":"main.main","file":"/build/main.go","line":89},"msg":"run exit with error","err":"Containerd registry config path needs to be set for mirror configuration to take effect"}

But I am setting the containerd registry config path.
I attempted an upgrade of Spegel to v0.0.24 using the command:

helm upgrade --create-namespace --namespace spegel --install --version v0.0.24 spegel oci://ghcr.io/spegel-org/helm-charts/speg
el   --set spegel.containerdRegistryConfigPath="/etc/cri/conf.d/hosts"

Which completed successfully. However only the Spegel install on Talos OS v1.7.6 is working.
I can't work out what TalosOS has changed to break this. Any ideas?

Oh, I see this is a duplicate of https://github.com/spegel-org/spegel/issues/579 (That title didn't lead me to believe it was the same issue) - Will close this.


Item transformer in Item Application recipe type [1.19.2]

When using the script
var brushTransformer = creosotebrush.transformCustom("brush/uses", (item) => {
if (item.damage < 4) {
return item.withDamage(item.damage - 1);
}
return brush;
});
craftingTable.addShapeless("test", treatedplanks, [brushTransformer]);
The item transformer will work, reducing durability until there is more than 4 damage, at which point it replaces it with a different item. I was wondering if it could be made compatible with the item_application recipetype. I ask because while it works in a crafting table, if you look at the script below, it will just keep damaging the item and never return a normal brush when a certain amount of damage is reached.
recipetype:create:item_application.addRecipe("wood/planks/treating", [treatedplanks], tag:items:minecraft:planks, brushTransformer);
I can't say I'm exactly sure why this happens or how it would be changed, but just wondering if the mod could be changed to allow this.

Unfortunately this is not possible as create does not call the methods that item transformers require to be done.


LTR for MI tests

History Notes:
(Fill in the following template if multiple notes are needed, otherwise PR title will be used for history note.)
[Component Name 1] (BREAKING CHANGE:) (az command:) make some customer-facing change.
[Component Name 2] (BREAKING CHANGE:) (az command:) make some customer-facing change.

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

add to S167

I would love if the length of short-term-retention-policy can be aggressively reduced in length. Try typing this 10 times in a row and I'm sure you would not want to be a customer using this command repeatedly :) Ergonomics is more important than clarify here - abbreviations can be easily explained with help text, as has been done for sql mi, sql midb, etc. My first idea is just short-term, but there are more possibilities.

I would love if the length of short-term-retention-policy can be aggressively reduced in length. Try typing this 10 times in a row and I'm sure you would not want to be a customer using this command repeatedly :) Ergonomics is more important than clarify here - abbreviations can be easily explained with help text, as has been done for sql mi, sql midb, etc. My first idea is just short-term, but there are more possibilities.

@jaredmoo I will talk to the code owners (I'm just doing long term retention) and see if we can get that cleaned up!  I think I'm going to be doing another PR with LTR for sterling DB's so I can fix up long-term-retention-policy then

/azp run

@Juliehzl

I have several suggestions for command design:

Can we consider shorter name for commands? currently the name is too long.
It looks long-term-retention-policy/backup/ltr are related command groups. It will be better if we can group them together.

I am curious about restore command. It should be a general command for midb with different options, why do you only add it in long term retention part?

Yes - I'm changing all instances of 'long-term-retention-' to 'ltr-' (@jaredmoo)
They are indeed all related; however I think it makes most sense to keep ltr policy and ltr backup separate still (to make it clear what the show, set, list-by-* commands are doing)
I will move restore into the midb restore.  I originally made a new function just for ltr restore because currently, the managed db restore command is hard-coded to be point in time restore.

Previously, I was recommended to add az sql midb ltr restore to the existing az sql midb restore command (which currently uses point in time restore).  However, after review from @jaredmoo, we decided not to go this route since the two restore modes have different required parameters, and introducing LTR restore into the existing command would be a breaking change.
I wanted to use az sql midb ltr-backup restore, but the restore command belongs to a different operations group (managed_databases_operations) from the rest of ltr-backup (long_term_retention_managed_instance_operations), so I was not able to include restore in the ltr-backup group; please let me know if there is indeed a way to achieve this.
Therefore, I am going with my original idea of simply doing az sql midb ltr restore.  This distinguishes from point in time restore, avoids breaking changes, and indicates that the managed database will be restored from a long term retention backup.


Add pipeline for publishing dev builds of typespec-go

@benbp does this pipeline need to be manually created or can we use prepare pipelines?

@jhendrixMSFT we'll need to create one. Can you link me to some of the other publish pipelines?

Here's the one for publishing autorest.go to npmjs.org https://dev.azure.com/azure-sdk/internal/_build?definitionId=2214
I can't find the one for publishing beta builds though. I know we had this at one point as I can see old beta builds (example). IIRC the pipeline for beta builds only published the artifacts to github and not npmjs.org.

@jhendrixMSFT to start https://dev.azure.com/azure-sdk/internal/_build/index?definitionId=6686&_a=completed

Thanks!


Removes strtestonly from test classes

Critical Changes
Changes
Issues Closed
New Metadata
Deleted Metadata

@dvenor This cleanup PR is the opposite of urgent. My choice of you as reviewer is arbitrary, so let me know if you're busy and I'll see if someone else wants it.

@lmeerkatz No problem.  Happy to do it.


Search for DAOs by display name

Currently it is only possible to search for a DAO by the Safe multisig address. Without access to or knowledge of that wallet address, searching for a DAO on Fractal is extremely difficult.
Can we update the search engine to include results for semantic DAO display names?
ex. I want to search for Decent DAO but do not know the multisig address, so I search "Decent" in the search bar and select an option that displays in a dropdown.

Initial thoughts are that we would need to set up / modify a Subgraph to index DAO Name information, which is stored (emitted?) onchain. If this issue were to move into the Backlog, it would likely require some accompanying (prerequisite) issues / work to be created in the subgraph and/or contract repos.

This is a big effort that would require intentional planning behind it. Closing this issue, we'll pick it back up in the future if deemed necessary.


Update azure-storage-blob to 12.20.1

Updates com.azure:azure-storage-blob from 12.20.0 to 12.20.1.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.azure", artifactId = "azure-storage-blob" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "@monthly" },
  dependency = { groupId = "com.azure", artifactId = "azure-storage-blob" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #820.


from fastapi_users.db import SQLAlchemyBaseUserTableUUID, SQLAlchemyUserDatabase - these 2 classes are not implemented/available

Describe the bug
Hello,
Documentation implies presence of these 2 classes:  SQLAlchemyBaseUserTableUUID, SQLAlchemyUserDatabase
https://fastapi-users.github.io/fastapi-users/latest/configuration/databases/sqlalchemy/
However, when code is executed it shows Error message:
from fastapi_users.db import SQLAlchemyUserDatabase, SQLAlchemyBaseUserTable
ImportError: cannot import name 'SQLAlchemyUserDatabase' from 'fastapi_users.db' (\fastapi\Lib\site-packages\fastapi_users\db_init_.py)
To Reproduce
In source code this 2 classes are just not implemented, either replaced or deleted.
So documentation regarding this issue is not updated.

solved by doing correct installation command: pip install 'fastapi-users[sqlalchemy]'


Unused home directory cleanup feature

While helping openscapes folks cleanup their user access list (https://github.com/2i2c-org/infrastructure/issues/3240), I also helped them cleanup unused home directories. This isn't a service 2i2c offers to users yet, but I wanted to open this to track work to be done here long term - otherwise it just increases costs monotonically for our users (https://github.com/2i2c-org/infrastructure/issues/3240#issuecomment-1751218146). This issue mostly currently tracks the one-off job I helped do for them, but needs more work to be rolled out sustainably to others.
I wrote an impromptu script that helped us archive home directories of a list of users given in the commandline to S3.

python script

#!/usr/bin/env python3
"""
Archive home directories onto object storage (like S3 or GCS).
Designed to be run manually, and takes care to not delete anything without a lot of
confirmation.
"""
import hashlib
import string
import sys
import shutil
import os
import argparse
import boto3
from botocore.exceptions import ClientError
from escapism import escape
from pathlib import Path
from contextlib import contextmanager
import tempfile
import time
import subprocess
from functools import cache
@cache
def get_tar_command() -> str:
"""
Return the tar command to use.
We use `gnu` tar for compressing files, and Mac OS ships with bsd tar by
default. We detect this, and tell users to get gnu tar if needed for local
testing. Should not be an issue when running on containers.
"""
out = subprocess.check_output(["tar", "--version"]).decode()
if out.startswith("tar (GNU tar)"):
    return "tar"
else:
    # We may be on Mac OS, and GNU Tar is not installed by default
    # It can be installed from homebrew with `brew install gnu-tar`,
    # which provides `gtar`
    if shutil.which("gtar"):
        return "gtar"
    else:
        print("Could not find GNU Tar on the system", file=sys.stderr)
        print(
            "If on Mac OS, please install gnu-tar with the following command (if using homebrew) and try again",
            file=sys.stderr,
        )
        print("brew install gnu-tar", file=sys.stderr)
        sys.exit(1)

def validate_homes_exist(basedir: Path, usernames: list[str], ignore_missing: bool):
"""
Validate that all given homedirectories for users exist
"""
errors = []
for username in usernames:
escaped_username = escape(
username, safe=set(string.ascii_lowercase + string.digits), escape_char="-"
).lower()
# We should still protect against directory traversal attacks
user_home = (basedir / escaped_username).absolute()
if basedir not in user_home.parents:
errors.append(
f"{user_home} refers to a directory outside of {basedir}, can not be archived"
)
    if not user_home.exists() and not ignore_missing:
        errors.append(
            f"{username}'s home directory does not exist inside {basedir}, {user_home} not found"
        )

if errors:
    print(
        "The following errors were found when trying to validate that all user home directories exist",
        file=sys.stderr,
    )
    print("\n".join(errors), file=sys.stderr)
    sys.exit(1)

@contextmanager
def archive_dir(dir_path: Path, archive_name: str):
"""
Archive given directory reproducibly to out_path
"""
start_time = time.perf_counter()
with tempfile.TemporaryDirectory() as d:
    target_file = Path(d) / (archive_name + ".tar.gz")
    cmd = [
        get_tar_command(),
        f"--directory={dir_path}",
        "--sort=name",
        "--numeric-owner",
        "--create",
        "--use-compress-program=pigz",
        f"--file={target_file}",
    ] + ["."]
    env = os.environ.copy()
    # Set GZip / pigz option to not write timestamp so we get consistent hashes
    env["GZIP"] = "-n"
    try:
        # Capture output and fail explicitly on non-0 error code
        # Primarily to get rid of tar: Removing leading `/' from member names
        subprocess.check_output(cmd, stderr=subprocess.STDOUT, env=env)
    except subprocess.CalledProcessError as e:
        print(f"Executing {e.cmd} failed with code {e.returncode}", file=sys.stderr)
        print(f"stdout: {e.stdout}", file=sys.stderr)
        print(f"stderr: {e.stderr}", file=sys.stderr)
        sys.exit(1)
    duration = time.perf_counter() - start_time

    file_size_gb = target_file.stat().st_size / 1024 / 1024 / 1024
    print(
        f"Tarballing {dir_path.name}  to {archive_name}.tar.gz ({file_size_gb:0.3f} GB) took {duration:0.2f}s"
    )

    yield target_file

def sha256_file(filepath: Path) -> str:
with open(filepath, "rb") as f:
return hashlib.file_digest(f, "sha256").hexdigest()
def archive_user(
s3_client,
basedir: Path,
username: str,
archive_name: str,
bucket_name: str,
prefix: str,
ignore_missing: bool,
delete: bool,
):
escaped_username = escape(
username, safe=set(string.ascii_lowercase + string.digits), escape_char="-"
).lower()
homedir = basedir / escaped_username
if ignore_missing and not homedir.exists():
    print(f"User {username} does not exist, skipping archival")
    return

print(f"Archiving {username}")
with archive_dir(homedir, archive_name) as archived_file:
    # Make sure the object key has the same extension as the compressed file we have
    object_name = os.path.join(prefix, username, archive_name) + "".join(
        archived_file.suffixes
    )
    sha256sum = sha256_file(archived_file)
    try:
        head_response = s3_client.head_object(Bucket=bucket_name, Key=object_name)
        # If we are here, it means that the file *does* exist
        if head_response["Metadata"].get("sha256sum") == sha256sum:
            # We have already uploaded this, and the hashes match!
            needs_upload = False
        else:
            # This file exists, *but hashes do not match!*
            # This is an error condition, and we abort so we don't overwrite user files
            print(head_response)
            print("AAAAAAAAAAA", file=sys.stderr)
            sys.exit(1)
    except ClientError as e:
        if e.response.get("Error", {}).get("Code") == "404":
            # Does not exist, needs to be uploaded
            needs_upload = True
        else:
            # Some other issue, let's just fail
            raise
    if needs_upload:
        start_time = time.perf_counter()
        print(f"Uploading {username}...")
        s3_client.upload_file(
            archived_file,
            bucket_name,
            object_name,
            ExtraArgs={"Metadata": {"sha256sum": sha256sum}},
        )
        duration = time.perf_counter() - start_time
        print(f"Upload for {username} complete in {duration:0.2f}s")
    else:
        if delete:
            start_time = time.perf_counter()
            print(f"Already uploaded, going to delete {username}")
            shutil.rmtree(homedir)
            duration = time.perf_counter() - start_time
            print(f"Already uploaded, deleted {username} in {duration:0.2f}s")
        else:
            print(f"Username already uploaded, skipping.")

def main():
argparser = argparse.ArgumentParser()
argparser.add_argument(
"--archive-name",
help="Name for user home directory in",
required=True,
)
argparser.add_argument(
"--basedir",
help="Base directory containing user home directories",
required=True,
)
argparser.add_argument(
"--object-store",
choices=("s3",),
default="s3",
help="Type of object store to upload files to",
)
argparser.add_argument(
"--bucket-name",
help="Name of object storage bucket to upload archived files to",
required=True,
)
argparser.add_argument(
"--object-prefix",
help="Prefix to use before username when uploading archives",
default="a/",
)
argparser.add_argument(
"--usernames-file",
help="File with list of usernames to archive, one per line",
required=True,
)
argparser.add_argument(
"--ignore-missing",
help="Ignore missing user home directories",
action="store_true",
)
argparser.add_argument(
"--delete", help="Delete homedirectories after uploading", action="store_true"
)
args = argparser.parse_args()

basedir = Path(args.basedir).absolute()
usernames = []
with open(args.usernames_file) as f:
    for line in f:
        if line.startswith("#"):
            continue
        usernames.append(line.strip())

validate_homes_exist(basedir, usernames, args.ignore_missing)

s3_client = boto3.client("s3")
for username in usernames:
    archive_user(
        s3_client,
        basedir,
        username,
        args.archive_name,
        args.bucket_name,
        args.object_prefix,
        args.ignore_missing,
        args.delete,
    )

if name == "main":
main()

It has some additional features to make sure we don't accidentally delete user home directories before they are properly archived.
There's currently no automatic way for folks to retrieve their home directories post archival, nor guidance about home directory policies that can be given to users before they sign up. For openscapes, their admins have access to s3 buckets via the web console and have taken on the role of sending users their home directories if so desired. 2i2c engineers can not be the ones doing this manual retrieval, however.
An s3 bucket that would move objects to archival status 3 days after was created, and this bucket was used to hold the archive. This was preferred as it would allow us to experiment with creating / deleting archives without incurring the additional cost that comes from doing create / delete operations on archival class objects (see pricing per GET / POST requests for standard vs Glacier Instant retrieval in https://aws.amazon.com/s3/pricing/).

Yet another request for this, now from Smithsonian: https://2i2c.freshdesk.com/a/tickets/1359

Tracked in https://2i2c.productboard.com/feature-board/7803674-product-ideas/features/25937693/insights


Create Armstrong Number

I have created a program to check armstrong number.
Description of Change

Checklist

[ ] Added description of change
[ ] Added file name matches File name guidelines
[ ] Added tests and example, test must pass
[ ] Added documentation so that the program is self-explanatory and educational - Doxygen guidelines
[ ] Relevant documentation/comments is changed or added
[ ] PR title follows semantic commit guidelines
[ ] Search previous suggestions before making a new one, as yours may be a duplicate.
[ ] I acknowledge that all my contributions will be made under the project's license.

Notes:

Seems to be a duplicate of math/armstrong_number.cpp.


Bug 1921458: run bundle-upgrade should handle error gracefully when a previous operator version doesn't exist

Description of the change:
run bundle-upgrade command handles error instead of throwing a panic, when a previous version of the operator bundle doesn't exist in the cluster
Motivation for the change:
Without installing etcd operator,
# operator-sdk run bundle-upgrade quay.io/olmqe/etcd-bundle:0.9.2-share

Result:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1978d29]

Expected behavior: The command should check that the etcd operator doesn't exist in the cluster and handle the error gracefully instead of throwing a panic.
Checklist
If the pull request includes user-facing changes, extra documentation is required:

[x] Add a new changelog fragment in changelog/fragments (see changelog/fragments/00-template.yaml)
[ ] Add or update relevant sections of the docs website in website/content/en/docs

@rashmigottipati please add changelog fragment.

@rashmigottipati please add changelog fragment.

Reproduced this bug locally using kind and also on OCP.
Verified this fix locally and against and it works as expected and doesn't throw panic anymore.
Actual result:
▶ ./build/operator-sdk run bundle-upgrade quay.io/olmqe/etcd-bundle:0.9.2-share
FATA[0001] Failed to run bundle upgrade: no existing operator found in the cluster to upgrade

Reproduced this bug locally using kind and also on OCP.
Verified this fix locally and against and it works as expected and doesn't throw panic anymore.
Actual result:
▶ ./build/operator-sdk run bundle-upgrade quay.io/olmqe/etcd-bundle:0.9.2-share
FATA[0001] Failed to run bundle upgrade: no existing operator found in the cluster to upgrade

/cherry-pick v1.3.x

/cherry-pick v1.3.x


build(vm): remove unused implicit features

Description
As stated in the rust doc
In some cases, you may not want to expose a feature that has the same name as the optional dependency. For example, perhaps the optional dependency is an internal detail, or you want to group multiple optional dependencies together, or you just want to use a better name. If you specify the optional dependency with the dep: prefix anywhere in the [features] table, that disables the implicit feature.
This makes sure no implicit features are created for the optional crates that are imported only to be used in another feature.

@juanbono @fmoletta I solved the conflicts. Ready to be merged


Added university name to project.

In Brazil, many websites use the .com.br extension instead of just .br
Could it be that uniasselvi.com.br.
(https://portal.uniasselvi.com.br/)


Install command fails

Describe the bug
Steamcmd is hanging. Any ideas?

docker@satisfactory:/steamcmd/satisfactory$ /steamcmd/linux32/steamcmd +runscript /app/install.txt
Redirecting stderr to '/app/Steam/logs/stderr.txt'
src/tier0/threadtools.cpp (4071) : Probably deadlock or failure waiting for thread to initialize.
[  0%] Checking for available updates...
Thread failed to initialize
src/tier0/threadtools.cpp (4071) : Probably deadlock or failure waiting for thread to initialize.
Thread failed to initialize
CWorkThreadPool::StartWorkThread: Thread creation failed.
Exiting on SPEW_ABORT

@lowfuel While researching this I did come across others with a Synology NAS and similar issues, right down to the hard locking that happens when it crashes.
If you're up for one more attempt, there's a new image tag that uses Ubuntu 18.04 (instead of 20.04), which just might be enough to fix it, as there's some known issues with steamcmd and even barely modern linux distros, but these issues have been around for years and there's still no official fix, or even a statement acknowledging them.
Anyway, the new tag is didstopia/satisfactory-server:ubuntu-18.04 and as I was able to replicate the thread locking issues on a different machine/server, I also managed to fix it with this new image, so I'm very hopeful about it fixing your issue as well!

Ah! Thanks for looking into it more! I will definitely give it a shot.
Cheers!

Just wanted to say thanks, the 18.04 version works perfectly. First try!


Move contraband text to a separate examine tab

i forgor i had this pr open and deleted it whoops
About the PR
Contraband text is now in a separate examination tab, like armor resistances and item damage.
Why / Balance
That info is related to a single department (security), I don't need to see it every time I examine something.
You can see that tab in the strip menu, meaning it's only a visual difference.
Technical details
ContrabandSystem was refactored a bit for that
Media

Requirements

[X] I have read and am following the Pull Request and Changelog Guidelines.
[X] I have added media to this PR or it does not require an ingame showcase.

Breaking changes
Changelog
no

this is a brilliant addition

is the knife a placeholder? might be better to make it something like handcuffs

@Errant-4 any news on maintainer discussion? AFAIK it's 3 days

@Errant-4 any news on maintainer discussion? AFAIK it's 3 days

Discussion ran into a design vacuum, vote has not started yet

g

Maybe it's just me but I don't like the inspect options, I find them annoying by design.
When it comes to checking people's IDs it's necessary to slow someone who doesn't have a sec hud from immediately identifying IDs.  If you wanna see someone's id you do need to get close and look closely.
But for this I feel like you're just slowing down the player getting info for very little value.  It's not super big of a deal but it's worthy of bringing up at least.

But for this I feel like you're just slowing down the player getting info for very little value.

It's a matter of milliseconds to click a button, plus you're most likely viewing it from the strip menu/from the item on the ground, making speed a non-issue


export/import interface met "reflection_capabilities.js:84Uncaught TypeError: Cannot read property 'annotations' of undefined"

Please provide us with the following information:

OS?   os
Versions.  beta.17
Repro steps.

Was this an app that wasn't created using the CLI? What change did you do on your code? etc.

export en interface in " app/a/a.interface.ts"
export interface PersonalSetInfo{
  userId:string;
  name:string;
}

and import in another component   "app/b/b.component"
import {PersonalSetInfo} from "relative path";
then met err:
reflection_capabilities.js:84Uncaught TypeError: Cannot read property 'annotations' of undefined
The log given by the failure.

Normally this include a stack trace and some more information.

reflection_capabilities.js:84Uncaught TypeError: Cannot read property 'annotations' of undefinedReflectionCapabilities.annotations @ reflection_capabilities.js:84Reflector.annotations @ reflector.js:102NgModuleResolver.resolve @ ng_module_resolver.js:24CompileMetadataResolver.getNgModuleMetadata @ metadata_resolver.js:197RuntimeCompiler._compileComponents @ runtime_compiler.js:130RuntimeCompiler._compileModuleAndComponents @ runtime_compiler.js:68RuntimeCompiler.compileModuleAsync @ runtime_compiler.js:59PlatformRef_._bootstrapModuleWithZone @ application_ref.js:302PlatformRef_.bootstrapModule @ application_ref.js:284(anonymous function) @ main.ts:12__webpack_require__ @ bootstrap 32b1926…:52(anonymous function) @ zone.js:1327__webpack_require__ @ bootstrap 32b1926…:52webpackJsonpCallback @ bootstrap 32b1926…:23(anonymous function) @ main.bundle.js:1
Mention any other details that might be useful.

Thanks! We'll be in touch soon.

What did you do with the interface inside the class?

below the folder where the interface is located：
when i add:
export * from './a.interface.ts';
the err "reflection_capabilities.js:84Uncaught TypeError: Cannot read property 'annotations' of undefined" appears:---

That doesn't answer my question though. For that error to appear, you must be using it somewhere in a component. I need the exact code that you're using to reproduce this error.

3 places  used:  (a bit messy, i just copy)
component A:
@Input() personalSetInfo: PersonalSetInfo;
component B who's child component is component A :
private personalSetInfo: PersonalSetInfo;

ngOnInit(): void {
    this.personalSetInfo = this.route.snapshot.data['personalSet'];
  }

and the  route resolve service:
resolve(route: ActivatedRouteSnapshot): Observable<PersonalSetInfo>|any{
// a service to get the PersonalSetInfo type Object
}

This doesn't seem like a CLI problem really, it seems like a Angular 2 problem. If you open the issue on https://github.com/angular/angular or http://stackoverflow.com/ it's more likely you'll get an answer.


Add command for minimizing/maximizing transparency

-Bind minimize to ctrl+alt+x

LGTM
Thanks

LGTM
Thanks


training problem

I run this command

show

help me pls.

What's your cuda version and your the cuda version of pytorch?

What's your cuda version and your cuda version of pytorch?

cuda9.0 and torch0.4.1 is not available for GPU 2080, so I have changed DCNv2 for pytorch1.x, cuda10.0 and torch1.0.1. but it didn't work well.
So what should I do if I want to train in 2080 ?

Maybe you can change the pytorch to 1.6. Your cuda should be upgraded to 10.1, 10.2 or 9.2, which matches the cuda version of your pytorch. After you change the pytorch version, remember to build the DCNv2 again according to the https://github.com/MCG-NJU/MOC-Detector/tree/pytorch1.6.0/src/network/DCNv2

I have solved it, thks a lot.


waitForJS getting timeout

Hi,
I am not sure if I am doing something wrong, but I have already spent 2 days on it, and with waitForJS and waitForJSVarName I am unable to create Pdf, every time I tried to create pdf, it throws "timeout error" irrespective of whatever timeout value I put in settings. And I don't see any error or a message to indicate what went wrong.
const converter = require("phantom-html-to-pdf")({
    numberOfWorkers: 2,
    timeout: 18000, //18 sec
    portLeftBoundary: 1000,
    portRightBoundary: 4000,
    host: '<IP_ADDRESS>',
    phantomPath: "..\\myapp\\node_modules\\phantomjs\\lib\\phantom\\phantomjs.exe",
    maxLogEntrySize: 1000
});
var createPdf = function (htmlUrl, createHere, token) {
            converter({
                url: htmlUrl,
                footer: '
{#pageNum}/{#numPages}
',
                injectJs:[Path.join(__dirname, '../public/bower_components/jquery/dist/jquery.min.js'),
                     Path.join(__dirname, '../public/dynamic-documents/thePageJS.js')],
                waitForJS: true,
                waitForJSVarName: "htmlReadyToBePrinted",
                settings: {e
                    javascriptEnabled: true
                },
                customHeaders: [{ Authorization: 'Bearer ' + token }],
                format: {
                    quality: 100
                }
            }, function (err, pdf) {
                if (err) {
                    console.error('Error while printing pdf.', err);
                    return;
                }
                console.log(pdf.logs);
                console.log(pdf.numberOfPages);
                var filePath = fs.createWriteStream(createHere);
                pdf.stream.pipe(filePath);
            });

I cross verified the paths of JS, all looks correct.
In my thePageJS.js, all I am doing making a API call and upon getting response (error or success), setting the window.htmlReadyToBePrinted to 'true'.
This is what the error I get everytime:
Error while printing pdf. { Error: Timeout
    at c:\CDN\MyApps\node_modules\phantom-workers\lib\phantomManager.js:117:29
    at c:\CDN\MyApps\node_modules\phantom-workers\lib\phantomWorker.js:120:13
    at c:\CDN\MyApps\node_modules\phantom-workers\lib\phantomWorker.js:136:20
    at Socket.<anonymous> (c:\CDN\MyApps\node_modules\phantom-workers\lib\checkPortStatus.js:45:9)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at TCP._handle.close [as _onclose] (net.js:501:12) weak: true, message: 'Timeout', phantomTimeout: true }

@rathorer hi! can you please prepare a repository with all the code needed to test your problem? we have limited time and the best (and faster) will be to provide a way to reproduce your problem.
Another reason to ask you for a repository is that there are other files referenced in your example which we don't know its contents, so we won't be able to fully check what is happening). Thanks.

@bjrmatos I have created a example repository (hope its helpful for others).
phantom-html-to-pdf-example
As of now in the above example waitForJS is disabled and pdf gets generated without any issues, however if you change it to make it enable, its always ending up in timeout.
So as of now to simulate my webapi i just kept the setTimeout, for which also its failing. So I am not able to populate my pdf with dynamic data.

@rathorer if you don't activate waitForJS and check your console you will see some errors:

the most important is the SyntaxError: Parse error error.. which means that phantomjs is having trouble to understand the javascript in your page, more specifically you are using ES6 (const declarations) in your page which phantomjs does not understand, as the first step to fix your problem just make sure the javascript of your page is ES5 compatible, then check for the other errors.

Hey @bjrmatos , this is helpful, thanks a lot, I was able to fix the example by removing all ES6 related code, I think I should be able to fix my original code as well.

@bjrmatos I am passing URL as parameter rather then the html which has lot of angular code written to it and seems like each time I am getting the timeout error. Note : I have very large amount of data its taking upto 1 minutes to load the view if we are using chrome. But not sure what to set for the timeout.

@ndaiya the default timeout is 180s so if your page load before that, it should be ok. maybe there is some new javascript syntax in your page that phantomjs does not recognize, phantomjs only recognize ES5, maybe you can verify that. or maybe you are not applying the waitForJS signal correctly in your js code. you can verify that by removing some code in your page, and use the waitForJS signal with some simple code, if it works, then maybe you are not calling window.PHANTOM_HTML_TO_PDF_READY = true in the way that you expect.

@bjrmatos Tried with window.PHANTOM_HTML_TO_PDF_READY = true, with the timeout for testing purpose it worked but its generating empty pdf.
What I am seeing  in my logs is :
[ { timestamp: 2017-11-24T06:23:23.592Z,     message: 'Converting in http server based phantomjs 2.1.1',     level: 'debug' },   { timestamp: 2017-11-24T06:23:23.594Z,     message: 'Request http://localhost:9000/shared/e6d0e199-6182-4f89-97b4-ab5527fe4c52',     level: 'debug' },   { timestamp: 2017-11-24T06:23:23.625Z,     message: 'Request http://localhost:9000/app/vendor.css?691d4b1b41983464e407',     level: 'debug' },   { timestamp: 2017-11-24T06:23:23.625Z,     message: 'Request http://localhost:9000/app/appRun.css?691d4b1b41983464e407',     level: 'debug' },   { timestamp: 2017-11-24T06:23:23.626Z,     message: 'Request http://www.gstatic.com/cv/js/sender/v1/cast_sender.js',     level: 'debug' },   { timestamp: 2017-11-24T06:23:23.626Z,     message: 'Request http://localhost:9000/socket.io/socket.io.js',     level: 'debug' },   { timestamp: 2017-11-24T06:23:23.626Z,     message: 'Request http://localhost:9000/app/vendor.bundle.js?691d4b1b41983464e407',     level: 'debug' },   { timestamp: 2017-11-24T06:23:23.627Z,     message: 'Request http://localhost:9000/app/appRun.js?691d4b1b41983464e407',     level: 'debug' },   { timestamp: 2017-11-24T06:23:23.628Z,     message: 'Request http://localhost:35729/livereload.js?snipver=1',     level: 'debug' },   { timestamp: 2017-11-24T06:23:26.970Z,     message: 'Warning: Accessing PropTypes via the main React package is deprecated. Use the prop-types package from npm instead.',     level: 'warn' },   { timestamp: 2017-11-24T06:23:26.970Z,     message: '    : 35',     level: 'warn' },   { timestamp: 2017-11-24T06:23:26.971Z,     message: 'Warning: DebounceInput: React.createClass is deprecated and will be removed in version 16. Use plain JavaScript classes instead. If you\'re not yet ready to migrate, create-react-class is available on npm as a drop-in replacement.',     level: 'warn' },   { timestamp: 2017-11-24T06:23:26.971Z,     message: '    : 35',     level: 'warn' },   { timestamp: 2017-11-24T06:23:27.474Z,     message: 'Request http://localhost:9000/api/poll/currentTime',     level: 'debug' },   { timestamp: 2017-11-24T06:23:27.489Z,     message: 'Request http://localhost:9000/api/la-telemetry/page-visit',     level: 'debug' },   { timestamp: 2017-11-24T06:23:27.498Z,     message: 'Time initialized at - Start: Fri Nov 24 2017 10:50:00 GMT+0530 (IST) End: Fri Nov 24 2017 11:50:00 GMT+0530 (IST)  ',     level: 'debug' },   { timestamp: 2017-11-24T06:23:27.535Z,     message: 'Unable to load resource (#10URL:http://localhost:9000/api/la-telemetry/page-visit)',     level: 'warn' },   { timestamp: 2017-11-24T06:23:27.535Z,     message: 'Error code: 204. Description: Host requires authentication',     level: 'warn' } ]
But I am pretty sure its been working without authentication. As this page doesn't need authorization.

Tried with window.PHANTOM_HTML_TO_PDF_READY = true, with the timeout for testing purpose it worked but its generating empty pdf.

@ndaiya  just to be sure.. have you activated that waitForJS option too? (like the example)

seems like you've not activated the waitForJS or maybe you are calling window.PHANTOM_HTML_TO_PDF_READY = true too early or late in your code.

But I am pretty sure its been working without authentication. As this page is shared.

i don't have more advice here, this seems more like a problem with your other pages, so i can't tell you what is wrong here.

@bjrmatos If I am enabling the waitForJs, each time it ends with the response :
{"weak":true,"message":"Timeout","phantomTimeout":true}
And if not setting then it works but results in empty pdf.

If I am enabling the waitForJs, each time it ends with the response :
{"weak":true,"message":"Timeout","phantomTimeout":true}
And if not setting then it works but results in empty pdf.

yes, it seems like you are not calling window.PHANTOM_HTML_TO_PDF_READY = true at the right moment in your app or maybe you are calling it in an event that it is never emitted or emitted too early. unfortunately i can't give you more insight without running code, if you can post here the content of the external url (html with angular app), i can take a look quickly if it is not too complex.


Improve clarity of beta limit options

In GitLab by @mkovari on Oct 4, 2017, 13:22
The options for how to set the g-value in the beta limit are a little opaque.  (Note that this is distinct from the question of which value of beta to constrain, which is set by iculbl.)  The following might be useful:

Reduce the two switches (iprofile and gtscale) to one switch with three or more options.
Explain the option used more clearly in the OUT.DAT.

Here are the switches and relevant variable:
iprofile /1/ : switch for current profile consistency:
    = 0 use input values for alphaj, rli, dnbeta (but see gtscale option);
    = 1 make these consistent with input q, q0 values (recommendation: use icurr=4 with this option)

dnbeta /3.5/ : (Troyon-like) coefficient for beta scaling; calculated as (4.0*rli) if iprofile=1 (see also gtscale option)

gtscale /0/ : switch for a/R scaling of dnbeta (iprofile=0 only):
    = 0 do not scale dnbeta with eps;
    = 1 scale dnbeta with eps

Note that the term "beta scaling" used above is not very helpful.
The three options listed in the Physics paper might be the way to implement this:

A reference would also be useful, as there isn't one in the physics paper!
(If $l_i$ is used it would help to explain how it is derived.)
EDIT 15/5/24
I have found the reference for $g=4 l_i$: equation 18.57 on page 584 of Fusion Plasma Physics, Stacey, 2nd ed.

I have removed gtscale and added the functionality to iprofile.


Format items as netCDF

This need comes from users of the climatedata.ca website which provides a web interface to make requests to this API. Specifically, I was asked to look into the problem of getting items of the "climate-daily" collection directly in a netCDF format.
AFAIU, a netCDF formatter plugin could be implemented here and activate for items requests? It seems it would be quite straigtforward, adding a xarray (+numpy, pandas) dependency. Would it make sense to this here?
I have time to contribute on this, it would be my first time dealing with pygeoapi.

Thanks @aulemahal.    xarray and numpy are already part of coverage processing in core pygeoapi, so looks tractable at first glance.  Looking forward to how the NetCDF/CF output is put together.  Are you thinking of CF point or time series feature types, or something else?
Another option is that if the plugin is generic enough that it can take a result set of GeoJSON and render as NetCDF, it may be a great candidate for pygeoapi core.  For now, let's start here :)

I never played with this, but it seems that following the "geometries" section of the conventions would make most sense. http://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#geometries
For point geometries the result is almost the same as a "normal" timeseries, so it's a good start.

Hi @tomkralidis. So while starting some work on this, I realized that there was not elegant way fully answer the request I first got through a general formatter. The request is simply too specific. I still think it would be interesting for pygeoapi in general to be able to output netCDFs, but I will not have time to work on this in the near future.
However, as my goal was to return a time series netCDF from the retrieved points, I did write that and if it is of any interest, here's the code:
https://gist.github.com/aulemahal/24608a6bb8ca901448707f06e7e6dd09
A netCDF formatter for pygeoapi could be implemented starting from this, I think. I'll leave the work for later or someone else.

Thanks @aulemahal .  We will keep this here for future reference as we evolve NetCDF output in pygeoapi/msc-pygeoapi.


Managed endpoint doesn't use Azure ML connections (private pip) when building image

Package Name: azureml-mlflow
Package Version: 1.37.0
Operating System: Windows
Python Version: 3.8.11

Describe the bug
When using private pip feeds it is possible to set up a connection (token) which is shared across the Azure ML Studio. This works for creating an environment for training the model. If the model is saved in the MLFLOW format it also works when using an Azure Container Instance for creating a realtime endpoint very easily. But when trying to use the managed endpoint the token is not used.
To Reproduce
When using private pip feeds it is possible to set up a connection (token) which is shared across the Azure ML Studio by running
1:
ws.set_connection(
    name="private-pip",
    category= "PythonFeed",
    authType="PAT",
    target =<EMAIL_ADDRESS>    value = "1234abcd"
)

2: This will then be used if you construct a environment were a private pip is referred and replace the package myprivatepackage with something in you feed:
from azureml.core import Environment
from azureml.core.conda_dependencies import CondaDependencies

env = Environment("test_env")
cd = CondaDependencies.create(
    python_version="3.8.12",
    conda_packages=["pip"],
    pip_packages=["scikit-learn", "pandas","azureml-mlflow","pyyaml","myprivatepackage"]
    )
cd.set_pip_option("--index-url https://pkgs.dev.azure.com/MY-ORG/_packaging/MY-FEED/pypi/simple/")
env.python.conda_dependencies = cd
env.register(ws)
env.build(ws)

3: If you train a model you can deploy it where it connects using the token when building the image too.
First create a training file:
## ml_project/train.py
from sklearn.linear_model import LinearRegression
import mlflow
import mlflow.sklearn
from azureml.core import Run
import yaml

from sklearn.datasets import load_iris
X, y = load_iris(as_frame=True, return_X_y=True)

run = Run.get_context()
ws = run.experiment.workspace
env = run.get_environment()
conda_env = yaml.safe_load(env.python.conda_dependencies.serialize_to_string())

with mlflow.start_run():
    X, y = load_iris(as_frame=True, return_X_y=True)
    model = LinearRegression()
    model.fit(X, y)
    mlflow.sklearn.log_model(model, "model", conda_env=conda_env)

Then then train the model

from azureml.core import Experiment, ScriptRunConfig
from azureml.core.compute import ComputeTarget

cpu_cluster = ComputeTarget(workspace=ws, name="test-cluster")
exp = Experiment(workspace=ws, name="test")

src = ScriptRunConfig(
    source_directory= "mlproject",
    script='train.py',
    compute_target=cpu_cluster,
    environment=env
)
run = exp.submit(src)

Then register the model:
import mlflow
mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri())
model_info = mlflow.register_model(f"runs:/{run.id}/model", "test_model")

4: You can deploy it using Azure Container Instance which works by grabbing the token. It deploys after some time.:
from mlflow.deployments import get_deploy_client
mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri())
client = get_deploy_client(mlflow.get_tracking_uri())
client.create_deployment(model_uri=f'models:/test_model/1', name="mlflow-test-aci")

5: Go to the UI for Azure Machine Learning Studio and go to the wizard for setting up a managed endpoint. As the model is of type MLFLOW it will create enviorment and scoring script for you. When the image builds it will fail as it is expecting the user to put in a password:
Instance status:
SystemSetup: Succeeded
UserContainerImagePull: Succeeded
ModelDownload: Succeeded
UserContainerStart: InProgress

Container logs:
2022-01-11T12:29:04,946980587+00:00 - rsyslog/run
2022-01-11T12:29:04,951881106+00:00 - gunicorn/run

2022-01-11T12:29:04,953826713+00:00 - nginx/run
AzureML image information: mlflow-ubuntu18.04-py37-cpu-inference:20210930.v4

PATH environment variable: /opt/miniconda/envs/amlenv/bin:/opt/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PYTHONPATH environment variable:

Pip Dependencies
---------------
applicationinsights==0.11.10
azureml-inference-server-http==0.3.2
certifi==2021.5.30
charset-normalizer==2.0.6
click==8.0.1
cloudpickle==2.0.0
databricks-cli==0.15.0
entrypoints==0.3
Flask==1.0.3
gitdb==4.0.7
GitPython==3.1.24
gunicorn==20.1.0
idna==3.2
importlib-metadata==4.8.1
inference-schema==1.3.0
itsdangerous==2.0.1
Jinja2==3.0.1
MarkupSafe==2.0.1
mlflow-skinny==1.17.0
numpy==1.21.2
pandas==1.2.4
protobuf==3.18.0
python-dateutil==2.8.2
pytz==2021.1
PyYAML==5.4.1
requests==2.26.0
six==1.16.0
smmap==4.0.0
tabulate==0.8.9
typing-extensions==<IP_ADDRESS>
urllib3==1.26.7
Werkzeug==2.0.1
wrapt==1.12.1
zipp==3.6.0

Entry script directory: /var/mlflow_resources/.

Updating conda environment from /var/azureml-app/azureml-models/test_model/1/model/conda.yaml !
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

Downloading and Extracting Packages

ca-certificates-2020 | 128 KB    |            |   0%
ca-certificates-2020 | 128 KB    | #2         |  13%
ca-certificates-2020 | 128 KB    | ########## | 100%

_openmp_mutex-4.5    | 22 KB     |            |   0%
_openmp_mutex-4.5    | 22 KB     | ########## | 100%

libgcc-ng-11.2.0     | 887 KB    |            |   0%
libgcc-ng-11.2.0     | 887 KB    | ########## | 100%
libgcc-ng-11.2.0     | 887 KB    | ########## | 100%

libzlib-1.2.11       | 59 KB     |            |   0%
libzlib-1.2.11       | 59 KB     | ########## | 100%

certifi-2020.6.20    | 160 KB    |            |   0%
certifi-2020.6.20    | 160 KB    | ########## | 100%

ld_impl_linux-64-2.3 | 667 KB    |            |   0%
ld_impl_linux-64-2.3 | 667 KB    | ########## | 100%
ld_impl_linux-64-2.3 | 667 KB    | ########## | 100%

libnsl-2.0.0         | 31 KB     |            |   0%
libnsl-2.0.0         | 31 KB     | ########## | 100%

sqlite-3.37.0        | 1.5 MB    |            |   0%
sqlite-3.37.0        | 1.5 MB    | ########## | 100%
sqlite-3.37.0        | 1.5 MB    | ########## | 100%

xz-5.2.5             | 438 KB    |            |   0%
xz-5.2.5             | 438 KB    | ########## | 100%
xz-5.2.5             | 438 KB    | ########## | 100%

libstdcxx-ng-11.2.0  | 4.2 MB    |            |   0%
libstdcxx-ng-11.2.0  | 4.2 MB    | ########## | 100%
libstdcxx-ng-11.2.0  | 4.2 MB    | ########## | 100%

pip-20.2.4           | 2.0 MB    |            |   0%
pip-20.2.4           | 2.0 MB    | ########## | 100%
pip-20.2.4           | 2.0 MB    | ########## | 100%

readline-8.1         | 295 KB    |            |   0%
readline-8.1         | 295 KB    | ########## | 100%

libgomp-11.2.0       | 427 KB    |            |   0%
libgomp-11.2.0       | 427 KB    | ########## | 100%
libgomp-11.2.0       | 427 KB    | ########## | 100%

ncurses-6.2          | 1.1 MB    |            |   0%
ncurses-6.2          | 1.1 MB    | ########## | 100%
ncurses-6.2          | 1.1 MB    | ########## | 100%

python-3.8.12        | 26.2 MB   |            |   0%
python-3.8.12        | 26.2 MB   | 5          |   6%
python-3.8.12        | 26.2 MB   | ##9        |  30%
python-3.8.12        | 26.2 MB   | ####6      |  46%
python-3.8.12        | 26.2 MB   | ######9    |  70%
python-3.8.12        | 26.2 MB   | ########## | 100%
python-3.8.12        | 26.2 MB   | ########## | 100%

wheel-0.35.1         | 36 KB     |            |   0%
wheel-0.35.1         | 36 KB     | ########## | 100%

_libgcc_mutex-0.1    | 3 KB      |            |   0%
_libgcc_mutex-0.1    | 3 KB      | ########## | 100%

libffi-3.4.2         | 57 KB     |            |   0%
libffi-3.4.2         | 57 KB     | ########## | 100%

zlib-1.2.11          | 86 KB     |            |   0%
zlib-1.2.11          | 86 KB     | ########## | 100%

openssl-3.0.0        | 2.9 MB    |            |   0%
openssl-3.0.0        | 2.9 MB    | ########## | 100%
openssl-3.0.0        | 2.9 MB    | ########## | 100%

tk-8.6.11            | 3.3 MB    |            |   0%
tk-8.6.11            | 3.3 MB    | ########## | 100%
tk-8.6.11            | 3.3 MB    | ########## | 100%

setuptools-50.3.0    | 902 KB    |            |   0%
setuptools-50.3.0    | 902 KB    | ########## | 100%
setuptools-50.3.0    | 902 KB    | ########## | 100%
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.11.0

Please update conda by running

    $ conda update -n base -c defaults conda

Pip subprocess error:
ERROR: Exception:
Traceback (most recent call last):
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
    return func(self, options, args)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 323, in run
    requirement_set = resolver.resolve(
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 339, in _get_abstract_dist_for
    self._populate_link(req)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 305, in _populate_link
    req.link = self._find_requirement_link(req)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 270, in _find_requirement_link
    best_candidate = self.finder.find_requirement(req, upgrade)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 900, in find_requirement
    best_candidate_result = self.find_best_candidate(
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 883, in find_best_candidate
    candidates = self.find_all_candidates(project_name)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 827, in find_all_candidates
    package_links = self.process_project_url(
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 791, in process_project_url
    html_page = self._link_collector.fetch_page(project_url)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 629, in fetch_page
    return _get_html_page(location, session=self.session)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 441, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 138, in _get_html_response
    resp = session.get(
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/network/session.py", line 421, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 650, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/network/auth.py", line 256, in handle_401
    username, password, save = self._prompt_for_password(parsed.netloc)
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/network/auth.py", line 226, in _prompt_for_password
    username = ask_input("User for {}: ".format(netloc))
  File "/opt/miniconda/envs/userenv/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 259, in ask_input
    return input(message)
EOFError: EOF when reading a line

CondaEnvException: Pip failed

Installing pip dependencies: ...working... Ran pip subprocess with arguments:
['/opt/miniconda/envs/userenv/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/condaenv.i5j0upr4.requirements.txt']
Pip subprocess output:
Looking in indexes: https://pkgs.dev.azure.com/MY-ORG/_packaging/MY-FEED/pypi/simple/
User for pkgs.dev.azure.com:
failed

6: I assume the same errro happens on managed Batch endpoint but I haven't tried it out.
Expected behavior
I expect the managed endpoint to work. I also expect it to work for batch endpoint.
Suggestion
A good solution to this would be to allow for setting a variable in Azure ML studio which was always loaded (at build time) as an env var (PIP_INDEX_URL) when building an environment. That way we don't have to specify the url in the environment/conda file and it can easily be changed if the pip feed changes it address.

Hi @thomasfrederikhoeck, thank you for opening an issue! I'll tag the appropriate folks so we can look into this as soon as possible.

Allowing to set PIP_INDEX_URL globally would also make it easier to follow best practices of using only one private feed described in this Microsoft white paper https://azure.microsoft.com/en-us/resources/3-ways-to-mitigate-risk-using-private-package-feeds/

Hi @mccoyp do you know if this has been picked up by anyone? It is stopping us from using managed endpoints with mlflow as our models uses internal packages :-)

Hi @thomasfrederikhoeck, thanks for the ping! @bandsina, have you been able to look into this, or do you know of someone on the ML team who can?

Hi again @mccoyp  . Sorry for keeping pinging you but this blocks us from using managed endpoints :-) I could seem that @bandsina has disabled notifications. Is it possible that you try another (internal) channel?

@thomasfrederikhoeck this is the place -- I'm sorry you haven't gotten any updates yet! I'll remind the ML team about this issue and re-tag it for their triage system.

@thomasfrederikhoeck Can you please share the conda.yml file from your MLflow model? If it doesn't contain any sensitive information. This will help us understand why AML is not able to identify that a private feed is there, and service connection needs to be used.

@singankit Yes, Here it is:
channels:
- defaults
- conda-forge
dependencies:
- python=3.8.0
- pip
- pip:
  - --index-url https://pkgs.dev.azure.com/MY-ORG/_packaging/MY-FEED/pypi/simple/
  - mlflow
  - scikit-learn==0.24.1
  - mysecretpackage==1.0.0
name: mlflow-env

This then aligns with the connection if have defined:
from azureml.core import Workspace
ws = Workspace(<input>)
ws.list_connections()
>>> [{'id': '/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.MachineLearningServices/workspaces/<workspace>/connections/my-pip', 'name': 'my-pip', 'type': 'Microsoft.MachineLearningServices/workspaces/connections', 'properties': {'category': 'PythonFeed', 'target'<EMAIL_ADDRESS>'authType': 'PAT', 'value': '<PAT>'}}]

env.register(ws)
env.build(ws)
register returns an instance you'd want to build
regenv= env.register(ws)
regenv.build(ws)

Had a very similar situation when trying to build a Docker image that uses a private package.

@MrCordeiro It looks similar and might be related but the error I'm hitting is not the same. Step 1-4 works fine and I think your issues was related to step 1-2. :-) It is when trying to deploying a model of type MLFLOW using Managed endpoint.

Do you have any updates on this issue, @singankit ? :-) We would love to start using managed endpoints but this is holdings us back.

@thomasfrederikhoeck Managed endpoint currently does not support using private feeds. The workaround for now is for customers is to BYOC(bring your own container). Build their own dockerfile with appropriate permission. Did you follow any public documentation for managed endpoint which mentions this scenario? I can take that feedback and provide it to the team to correct. Let me know if there is anything else that I can help you with!

@amolb089  Thank you for the answer. How can I specify my own image when using the MLFlow model format and managed endpoints? I don't think I have seen it described? I just assumed it worked in similar fashion to https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-mlflow-models which does work for private feeds :-)

@thomasfrederikhoeck apologizes for the delay. Since this is an unsupported scenario we are working on a custom sample which can help with this scenario. However it will require using v2 CLI to create Managed Online Deployment. Will share the example in couple of days.

Thank you @singankit . Will you post it here when it is available?

@singankit @amolb089  do you have an update on an example on how to achieve it? :-)

/unresolve

@thomasfrederikhoeck
@singankit
Was there ever a solution to this? I'm having a similar problem when trying to switch to endpoints. I want to be able to update the env in v2 using a private wheel in the same way that you could in the v1 SDK i.e. :
from azureml.core import Environment
from azureml.core.environment import CondaDependencies

conda_dependencies = CondaDependencies(
        conda_dependencies_file_path="../environment.yml"
    )
env = Environment(name="name")
conda_dependencies.set_python_version("version")
env.python.conda_dependencies = conda_dependencies
whl_url = env.add_private_pip_wheel(workspace,
                               file_path="path_to_whl.whl",
                               exist_ok=True
  )
env.python.conda_dependencies.add_pip_package(whl_url)

Any help would be much appreciated

private wheels are not supported in v2. though if you reference it from the storage associated with your ws AzureML will be able to resolve it, as a backward compat support. I'd recommend to use private indexes instead as described above. Other solution if your wheel is relatively small is to use docker context approach and have it there.

@vizhur I'm not sure I follow, how do I actually add the package to azure without using v1 Environment class?, Basically the equivalent of my code above or step 2 in the initial question using Environment from v2?

There is no equivalent in V2 for private packages

TL;DR: You can't use a private index with v2 and MLFlow unless you put your secret in clear text.
@rgreen1995 Not to my knowledge but your problem is different as we use index instead of wheel. We abandoned trying to use managed endpoints (v2 in general).
@vizhur I'm not quite sure that i follow? This whole issue is about private indexes doesn't work for v2 (managed endpoints) unless you put the token for the index as clear text in the requirements file. It can't load the from connections as I have descibed above. I guess as a work around you can build the environment separately and but this doesn't work when using MLFlow as you can't select the enviorment (docker image) in that case.

Hey @thomasfrederikhoeck ,  @rgreen1995 here https://github.com/Azure/azure-sdk-for-python/issues/22441#issuecomment-1263669644 had a question about private wheel feature, that is a bit out of the issue context.

@vizhur I guess :-) But do you know if there is any progress with private pip index for managed endpoints?

@vizhur If you don't know, can you provide who can? :-)

I'd bet on @singankit ;)

Is there any working solution or workaround for this currently to use private repos?

or maybe @azureml-github is better?

+1 any documented working solution would be great, this is the only thing stopping me from transitioning to endpoints

From my understanding connections should still be a thing but it is not very clear if they should still be able to be used for private pip index: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-migrate-from-v1#connection-workspace-connection-in-v1

I saw this but does this mean that it is supported? https://github.com/Azure/azureml-examples/blob/main/sdk/python/resources/connections/connections.ipynb

@thomasfrederikhoeck Yes, workspace connections can be used for private feeds. And the environment you build using private feeds can be used in v2 deployments/endpoints.

@luigiw  I think it would be very useful with an example in https://github.com/Azure/azureml-examples or similar. :-) Does it work by always setting PIP_INDEX_URL when the environment is building, or does it do some matching with url specified with --index-url in requirement files? The first would be cool as this means that the environment is not dependent on the the index keeping the specific url.

I don't think you need to do anything special once a workspace connection is added for your private feed. @vizhur correct me if I was wrong here.

for base image + conda spec case we'll update urls from the spec with the auth from ws connections using the closest match (if there are multiple connections for the same host name)

@vizhur   Hmm. okay. This issue was actually created because this didn't happen with managed endpoints but only for v1 ACIs?
I don't think the logic surrounding private pip is very transparent. What happens if I create a private pip connection but don't specify a --index-url in the pip section of the conda spec? Is it still used?

@luigiw @vizhur What you describe is actually what I did (see top of Issue) but it failed? :-) It only looks like updating of URLs happen in environments created by the user and not the ones created automatically by AzureML when deploying a MLFlow model to managed endpoints (where the env can't be controlled).

@thomasfrederikhoeck this feature is now in private preview in AzureML using a capability called Packages. You can test it out by signing up here: https://aka.ms/AzureML/ModelPackages


Bag buttons not showing

Addon Version
v0.1.27
World of Warcraft Version
Retail
Describe the bug
Normally buttons are shown for the main bag as wel as the indivudual bags/reagent bag. They disappear as soon as I switch from Adibags to Better bags. Replacing a specific bag by a bigger one is impossible now. I don't know if it is a bug or intentional, but the option to show the blizzard bag button only shows the main bag and not the specific bags. I would very much like to be able to show all the bag buttons on screen.
Steps to reproduce
Enable the option to show the blizzard bag button. Only one button shows.

Did you select the "Show Bags" option from the drop down in the bag menu?

Did you select the "Show Bags" option from the drop down in the bag menu?

Thanks for answering.
I see what you mean. But that shows the bags on top of the bag (I believe adibags had that same option). But it still hides the bag buttons next to the main bag.
If it is possible, I would prefer to have an option to show those as well. But I guess this is by design and not a bug.
I just do not know how to change this topic from a bug to a feature request then.


CTC sort_by_thresh does not support nan and multiple comparison operators

Describe the Problem
RRFS needs the CTC linetype to generate ROC diagrams.  The fcst_thresh column contains several nan values and thresholds with multiple comparison operators (e.g. >=4.5 && <=9.1 )
When sorting these fcst threshold column, the sorting logic raises errors as it doesn't explicitly handle these conditions.
Expected Behavior
Provide a clear and concise description of what you expected to happen here.
Environment
Describe your runtime environment:
1. Machine: (e.g. HPC name, Linux Workstation, Mac Laptop)
2. OS: (e.g. RedHat Linux, MacOS)
3. Software version number(s)
To Reproduce
Describe the steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
Post relevant sample data following these instructions:
https://dtcenter.org/community-code/model-evaluation-tools-met/met-help-desk#ftp
Relevant Deadlines
List relevant project deadlines here or state NONE.
Funding Source
Define the source of funding and account keys here or state NONE.
Define the Metadata
Assignee

[ ] Select engineer(s) or no engineer required
[ ] Select scientist(s) or no scientist required

Labels

[ ] Select component(s)
[ ] Select priority
[ ] Select requestor(s)

Projects and Milestone

[ ] Select Organization level Project for support of the current coordinated release
[ ] Select Repository level Project for development toward the next official release or add alert: NEED CYCLE ASSIGNMENT label
[ ] Select Milestone as the next bugfix version

Define Related Issue(s)
Consider the impact to the other METplus components.

[ ] METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Bugfix Checklist
See the METplus Workflow for details.

[ ] Complete the issue definition above, including the Time Estimate and Funding Source.
[ ] Fork this repository or create a branch of main_<Version>.
Branch name: bugfix_<Issue Number>_main_<Version>_<Description>
[ ] Fix the bug and test your changes.
[ ] Add/update log messages for easier debugging.
[ ] Add/update unit tests.
[ ] Add/update documentation.
[ ] Add any new Python packages to the METplus Components Python Requirements table.
[ ] Push local changes to GitHub.
[ ] Submit a pull request to merge into main_<Version>.
Pull request: bugfix <Issue Number> main_<Version> <Description>
[ ] Define the pull request metadata, as permissions allow.
Select: Reviewer(s) and Development issues
Select: Organization level software support Project for the current coordinated release
Select: Milestone as the next bugfix version
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Complete the steps above to fix the bug on the develop branch.
Branch name:  bugfix_<Issue Number>_develop_<Description>
Pull request: bugfix <Issue Number> develop <Description>
Select: Reviewer(s) and Development issues
Select: Repository level development cycle Project for the next official release
Select: Milestone as the next official version
[ ] Close this issue.

This was fixed in #363


Issues with missing 'role="editor"' entries in some bib. items for RFCs

Have only seen this in one document so far, so there are probably more.  Seen so far with RFCs 3473, 4426, 4872, and 6372.
The problem occurs when using https://bib.ietf.org/public/rfc/bibxml/ in the .xml file.
It is resolved by changing (back) to https://xml2rfc.ietf.org/public/rfc/bibxml/, running xml2rfc --clear-cache, and regenerating the output files.
Please see the following:
https://www.rfc-editor.org/v3test/rfc9270-bib-bugs-15July2022.xml
https://www.rfc-editor.org/v3test/rfc9270-bib-bugs-15July2022.txt
https://www.rfc-editor.org/v3test/rfc9270-bib-bugs-15July2022-rfcdiff.html
This file is correct:  https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.4872.xml
This file shows the issues:  https://bib.ietf.org/public/rfc/bibxml/reference.RFC.4872.xml
"Lang, J P." in the listing for RFC 4872 is incorrect as well.

Filed an issue on missing roles (if they are provided in RFC editor’s authoritative data but get lost in conversion, this will get fixed).
I’m not sure how to interpret this note:

"Lang, J P." in the listing for RFC 4872 is incorrect as well.

The missing of role "editor" is a bug. Now filed in https://github.com/relaton/relaton-ietf/issues/92#issuecomment-1192215551

"Lang, J P." in the listing for RFC 4872 is incorrect as well.

The authoritative information comes from the RFC Editor endpoint:
https://www.rfc-editor.org/rfc-index.xml
However, that endpoint does not provide a structured name format:
For example in RFC 4872, the information provided is:
<rfc-entry>
  <doc-id>RFC4872</doc-id>
  <title>
RSVP-TE Extensions in Support of End-to-End Generalized Multi-Protocol Label Switching (GMPLS) Recovery
</title>
  <author>
    <name>J.P. Lang</name>
    <title>Editor</title>
  </author>
  <author>
    <name>Y. Rekhter</name>
    <title>Editor</title>
  </author>
  <author>
    <name>D. Papadimitriou</name>
    <title>Editor</title>
  </author>
  <date>
    <month>May</month>
    <year>2007</year>
  </date>
  <format>
    <file-format>ASCII</file-format>
    <file-format>HTML</file-format>
  </format>
  <page-count>47</page-count>
  <keywords>
    <kw>resource reservation protocol</kw>
    <kw>traffic engineering</kw>
  </keywords>
  <abstract>
    <p>
This document describes protocol-specific procedures and extensions for Generalized Multi-Protocol Label Switching (GMPLS) Resource ReSerVation Protocol - Traffic Engineering (RSVP-TE) signaling to support end-to-end Label Switched Path (LSP) recovery that denotes protection and restoration. A generic functional description of GMPLS recovery can be found in a companion document, RFC 4426. [STANDARDS-TRACK]
</p>
  </abstract>
  <draft>draft-ietf-ccamp-gmpls-recovery-e2e-signaling-04</draft>
  <updates>
    <doc-id>RFC3471</doc-id>
  </updates>
  <updated-by>
    <doc-id>RFC4873</doc-id>
    <doc-id>RFC6780</doc-id>
  </updated-by>
  <current-status>PROPOSED STANDARD</current-status>
  <publication-status>PROPOSED STANDARD</publication-status>
  <stream>IETF</stream>
  <area>rtg</area>
  <wg_acronym>ccamp</wg_acronym>
  <errata-url>
http://www.rfc-editor.org/errata_search.php?rfc=4872
</errata-url>
  <doi>10.17487/RFC4872</doi>
</rfc-entry>

The contributor names are not split into initials vs given name vs surname. We can certainly do a transformation, but we rely on the RPC to tell us what the correct behavior is here.

@ronaldtse I suspect this is not what the sentence meant: we already do name normalization, which includes the extraction of initials (or at least that’s what we have in source data), and we return separate initials in our XML responses (see both xml2rfc and BibXML service links in ticket description). Thus I thought it may have to do with initial formatting.

Thanks for the clarification. Maybe @lbartholomew-rpc just meant that the line with "J.P. Lang" is the line that demonstrates the problem of the missing role="editor"?

The legacy bibxml element is <author initials="J.P." surname="Lang" fullname="J.P. Lang" role="editor">.
In the new data, we have <author fullname="J.P. Lang" surname="Lang" initials="J P"/>.

Hi, Anton and Ronald.
Sorry I wasn't clearer about this!  Ronald, your "We return them without the dots" note demonstrates the issue with "Lang, J P." in the listing for RFC 4872.  Because entries for initials are being returned without the dots, they don't display correctly in output for multiple initials that should have dots instead of a space between them (in other words, "J P." should be "J.P.").
rfc-index.xml can be followed to find out what the outputs should look like.  Another example is A.L.J. Verschuren.  rfc-index.xml is correct, and so is https://xml2rfc.tools.ietf.org/public/rfc/bibxml/reference.RFC.8063.xml, but https://bib.ietf.org/public/rfc/bibxml/reference.RFC.8063.xml is missing the dots because it uses spaces (ditto for two of the three coauthors).
There is some variation, and some authors have explicitly expressed their preferences.  For example, Simon Pietro Romano as of a few years ago wants S P. Romano on the first page (note the space between "S" and "P.".  However, we need to keep S. Romano for RFCs 1020, 1062, 1117, and 6503 (and per rfc-index.xml) but use S P. Romano for RFCs 6504, 7058, 8846, and 8847.  So if the rfc-index.xml entries are pulled for each individual RFC, things should work fine.  I don't know how you extract the data, so I'm guessing and hoping that it's not a hassle.
Thank you!  Please let me know if I need to be clearer about any of this.

@lbartholomew-rpc thank you for the detailed explanation! As you surmised, the bibliographic information for each RFC is converted individually, so we should be able to tweak the parsing to directly accept initials as they are, without doing anything clever.
Let us get back to you on this ticket!

Hi, Ronald.  You're most welcome, and sounds good!

@lbartholomew-rpc we will be fixing the issue here:
https://github.com/relaton/relaton-ietf/issues/95

@ronaldtse -- Hi, Ronald.  Thanks for the note re. https://github.com/relaton/relaton-ietf/issues/95!

While https://github.com/relaton/relaton-ietf/issues/95 is fixed, which means the data source is fixed:
https://github.com/ietf-tools/relaton-data-rfcs/blob/08dd1326979660b46c776d80c3205a069bf26fb4/data/RFC4872.yaml#L26-L35
Yet the output is still unfixed:
https://bib.ietf.org/public/rfc/bibxml/reference.RFC.4872.xml
<author fullname="J.P. Lang" initials="J P" role="editor" surname="Lang"/>

This means either it is a relaton-py issue or bibxml-service issue.
Ping @strogonoff @stefanomunarini .

Is this issue still needed? The original "role" part has a fixing commit in relaton - have we confirmed it fixes the final output here, modulo the initials? The initials are being traced at #264, and the remainder of this (or that) should probably be treated as a duplicate?

@rjsparks this is exactly why I'm keeping this issue open, because this is not yet fixed. Sorry about this. I'm hoping to close this once the necessary fixes are verified.

We have come up with a solution. The work is being tracked at:

https://github.com/relaton/relaton-models/pull/42
https://github.com/relaton/relaton-ietf/issues/99
https://github.com/relaton/relaton-py/issues/39

The fix has been merged in #282 thanks to @strogonoff . Closing.
https://bib.ietf.org/public/rfc/bibxml/reference.RFC.4872.xml
<reference anchor="RFC4872" target="https://www.rfc-editor.org/info/rfc4872">
<front>
<title>
RSVP-TE Extensions in Support of End-to-End Generalized Multi-Protocol Label Switching (GMPLS) Recovery
</title>
<author fullname="J.P. Lang" initials="J.P." role="editor" surname="Lang"/>
<author fullname="Y. Rekhter" initials="Y." role="editor" surname="Rekhter"/>
<author fullname="D. Papadimitriou" initials="D." role="editor" surname="Papadimitriou"/>
<date month="May" year="2007"/>
<abstract>
<t>
This document describes protocol-specific procedures and extensions for Generalized Multi-Protocol Label Switching (GMPLS) Resource ReSerVation Protocol - Traffic Engineering (RSVP-TE) signaling to support end-to-end Label Switched Path (LSP) recovery that denotes protection and restoration. A generic functional description of GMPLS recovery can be found in a companion document, RFC 4426. [STANDARDS-TRACK]
</t>
</abstract>
</front>
<seriesInfo name="RFC" value="4872"/>
<seriesInfo name="DOI" value="10.17487/RFC4872"/>
</reference>


Hide database migration post from table of content list

What this PR does / why we need it:
Hide the database migration post from blog table of content, since that version is staled and we're not going to support that migration script in the future.
Which issue(s) this PR fixes:
Fixes #
Does this PR introduce a user-facing change?:

NONE

/approve

This is a blog post, not documentation. I feel like you basically shouldn't delete it once you publish it as someone is still likely to refer to this page even if it gets outdated.

/approve cancel

I don't have a strong opinion so if you guys are fine, proceed as is.

This is a blog post, not documentation. I feel like you basically shouldn't delete it once you publish it even if it gets outdated as someone is still likely to refer to this page.

I feel you, but the change I made here does not delete the post content. It's possible to access the post by URL directly, just hide it from the table of content list, since migration post may cause users to misunderstand that our PipeCD highly depends on a database, that may not good for the product image.

That makese total sense. In that case fine to me 👍
There you go /approve


Error with Forecaster Attribute 'validation_metric_value'

Hey, I've just started using Scalecast, and was trying to use the RandomForest Model. After initializing Forecaster and adding COVID19 Regressor and Time Trend, I got the error:
AttributeError: 'Forecaster' object has no attribute 'validation_metric_value'

Here is the code that I'm running:

def plot_test_export_summaries(f):
    """ exports the relevant statisitcal information and displays a plot of the test-set results for the last model run
    """
    f.plot_test_set(models=f.estimator,include_train=False)
    plt.title(f'{f.estimator} test-set results',size=16)
    plt.show()
    return f.export('model_summaries',determine_best_by='TestSetMAPE')[
        [
            'ModelNickname',
            'HyperParams',
            'TestSetMAPE',
            'InSampleMAPE',
        ]
    ]

icms = Forecaster(y=df.ICMS, current_dates=df.DATA, test_length=12, freq='MS')
icms.generate_future_dates(12)
icms.plot()

icms.add_covid19_regressor()
icms.add_time_trend()

icms.set_estimator('rf')
rf_grid = {
    'max_depth':[2,3,4,5],
    'n_estimators':[100,200,500],
    'max_features':['auto','sqrt','log2'],
    'max_samples':[.75,.9,1],
}
icms.ingest_grid(rf_grid)
icms.auto_forecast()
plot_test_export_summaries(icms)

Full error:

I'm trying to figure out what is the problem, if anyone have any suggestion, I would aprecciate.
I still want to use other models ('xgboost', 'LSTM' and 'prophet'), but I get the same error. Am I missing something?

Hi,
Before calling icms.auto_forecast() and after running icms.ingest_grid(), allow the object to select the optimal hyperparameters by calling either icms.tune() or icms.cross_validate(). This should avoid the error.
Next scalecast update, I will make sure there is a more descriptive error message or warning when this happens.
Thanks for raising the issue!

Thank you, it solved the problem.


[sil-opaque-values] Don't set reference-counted flag during lowering.

With sil-combine test case.

@swift-ci smoke test and merge.


Why is the DS peripheral configured for rsa_alt rather than rsa? (IDFGH-11891)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report
esp-idf v5.12, mbedtls v3.5.0
I have had issues connecting ESP32-S3 to ThingsBoard using TLS 1.3 mutual authentication and DS. The problem was that the DS peripheral is configured with rsa_alt (see esp_mbedtls_init_pk_ctx_for_ds) and rsa_alt only supports MBEDTLS_PK_RSA, not MBEDTLS_PK_RSASSA_PSS.
This conflicts with mbedtls which does not support MBEDTLS_PK_RSA (legacy?) in mbedtls_ssl_tls13_check_sig_alg_cert_key_match() and mbedtls_ssl_tls13_sig_alg_for_cert_verify_is_supported().
Can the  DS peripheral be configured for rsa (which supports MBEDTLS_PK_RSA and MBEDTLS_PK_RSASSA_PSS)?

Hi @owen63,
Thanks for the issue.
Can you share any error logs that you observed while trying the connection with TLS 1.3?
If I understand correctly, for mbedTLS the ALT layers are to be replaced in future by the PSA Crypto layer but the current layer are supposed to be functioning correctly.
If posible can you share a small reproducible code that I can quickly try at my end?
Thanks,
Aditya

Hi Aditya,
I am putting together a sample project now...
On Mon, 29 Jan 2024 at 14:51, Aditya Patwardhan @.***>
wrote:

Hi @owen63 https://github.com/owen63,
Thanks for the issue.
Can you share any error logs that you observed while trying the connection
with TLS 1.3?
If I understand correctly, for mbedTLS the ALT layers are to be replaced
in future by the PSA Crypto layer but the current layer are supposed to be
functioning correctly.
If posible can you share a small reproducible code that I can quickly try
at my end?
Thanks,
Aditya
—
Reply to this email directly, view it on GitHub
https://github.com/espressif/esp-idf/issues/12977#issuecomment-1913829091,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXRUI77LCGS5KPPLT3VEWCLYQ36A7AVCNFSM6AAAAABB2QH5DOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJTHAZDSMBZGE
.
You are receiving this because you were mentioned.Message ID:
@.***>

Cannot reproduce.


Feature/readme update

SwiftUI is still a bit new. This is why I believe this minor edit would be a useful addition to your documentation.

Hi @piterwilson thanks for your pull request. There's something that sounds wrong to me in the note you added. I'll try to explain my doubts to you, tell me if I'm missing something: in my opinion the sentence

If you want to specify a different transition for push and pop, you can use .asymmetric

is wrong. If you specify an asymmetric transition you're not actually specifying a transition for the push operation and a transition for the pop operation. You're specifying a transition for an insertion operation and a transition for a removal operation, which is different. In SwiftUI an asymmetric transition is a transition that (from https://developer.apple.com/documentation/swiftui/anytransition/3076193-asymmetric):

Returns a composite transition that uses a different transition for insertion versus removal.

This means that asymmetric has nothing to do with the push/pop concept. Both in a push operation and in a pop operation you have an insertion and a removal. Let's make a simple example. Let's say we have this asymmetric transition for the navigation stack:
AnyTransition.asymmetric(insertion: .scale, removal: .opacity)
Now let's say we are in a View1 and we push to a View2. You will have:

.scale applied to View2 (the view that is getting in)
.opacity applied to View1 (the view that is being removed)

Now we are in the View2 and we want to come back to View1 with a pop operation. You will have:

.scale applied to View1 (the view that is getting in)
.opacity applied to View2 (the view that is being removed)

So, insertion != push and removal != pop, insertion and removal are just both applied in push and pop operations.
Does this make sense  to you?

It totally makes sense. Thanks for clarifying.


Add genomeSequences field to schema per spec

The GenomeSequences field is an array of GenomeSequence objs, which have sample collection date, repository url, sequence id, sequence name, sequence length, and notes fields.
AFACIT, none of these fields is required.

 Updates to schema based on standard spec


Added merge/split tests

The main purpose of this PR is to add merge/split tests, but I've also done a couple other things.
First, I've imported merge_split in the main __init__.py, which both builds the documentation and allows users to import tobac and then run tobac.merge_split instead of having to import tobac.merge_split explicitly.
Second, I've changed the default start track number to 0 from -1. I think this caused a great deal of confusion over on the main PR, and threw me for a loop while building the test.
Third, I've added a test that merges two cells and checks to make sure that they are merged (with an appropriate distance set) or not merged (with a distance set too close). I think that this gets us most of the way there on tests, if not all of the way there. I think a figure would help us out with the documentation, and I'm working on that now.
@kelcyno, I'd love to hear your feedback. I'm happy to send along the notebook that I used to create the test. The idea and some of the code for this test comes from @JuliaKukulies and @snilsn.

@kelcyno can you approve the workflows when you get a chance so that I can see what this does for coverage?

@freemansw1 I need to double check, but by having the count begin at -1, it ensures that all features that are not linked to a cell also are not linked to a track. Features that are left un-linked have a cell id of -1, and I think that should be the same case with Tracks.

@freemansw1 I need to double check, but by having the count begin at -1, it ensures that all features that are not linked to a cell also are not linked to a track. Features that are left un-linked have a cell id of -1, and I think that should be the same case with Tracks. BUT I think the correct order here is to begin counting at 0, and if the id == -1, then the track id -1 is given.

Ok, interesting. Before making this change, I was definitely getting a single track with an id of -1, which I assume is not the intended behavior.

I'm happy to (or you can) change it back to starting at -1, but I was getting incorrect behavior before.

Okay, absent codecov.io working, I ran a coverage test on my machine. This test covers most of the lines in merge_split, and I'm reasonably happy with this for the main tobac repo. We'll need to do some user communication to encourage users to add networkx, and ultimately we may decide to add it to our requirements, but I think this we are in a good place for 1.4.0.
@kelcyno if you are happy with my changes, I think we can turn this around to a quick approval and get 1.4.0 released early next week.