Reconsidering what semantic conventions code generation should produce

Context
Today every (or almost every) language SIG generates semantic conventions from yaml files using Jinja templates.
Each SIG separates attributes based on the signal they appear on in one way or another:

resource attributes vs all other attributes
resource, spans, logs/events

While there are some deviations, most of jinja templates and codegen configurations are mostly consistent across SIGs.
Problem
We want attributes to be used across signals. E.g. user_agent.original is a span attribute on server and resource attribute in browser.
Code generation was already broken and is patched to keep the status quo until we find a better solution.
Questions to Otel SIGs:
How language SIGs see the 'perfect' outcome of code generation tooling?

How do we group attributes? Do we group them at all or generate all in one class?
How do we work with experimental vs stable attributes? Should we separate them into stable and preview artifacts?
Do we want semconv libs to have consistent approach/api across languages?
How do we guarantee that YAML refactoring or code generation tweaks would not change stable public API of semconv libs in the future even if we reconsider approach in the p1?
Do we even want to ship stable semantic convention libraries in each language?
anything else?

Language SIGs that generate semconv with Jinja templates:

Java: separates resource vs all other attributes. @open-telemetry/java-maintainers
JS: separates resource vs all other attributes @open-telemetry/javascript-maintainers
Python: separates resource vs all other attributes @open-telemetry/python-maintainers
.NET: separates resource vs trace @open-telemetry/dotnet-maintainers
Go:  separates resource, events, traces @open-telemetry/go-maintainers
Collector: separates resources vs spans  @open-telemetry/collector-maintainers
Rust: separates resource vs all other @open-telemetry/rust-maintainers
C++: separates resource vs all other attributes @open-telemetry/cpp-maintainers
Erlang: separates resource, logs, traces @open-telemetry/erlang-maintainers
Ruby: separates resource,  traces   @open-telemetry/ruby-maintainers
PHP: separates resources vs all other   @open-telemetry/php-maintainers
Swift: separates resources vs all other  @open-telemetry/swift-maintainers

/cc @lquerel in case you have some thoughts or ideas on how to change the existing basic semconv codegen

Proposal
Jinja templates still allow to customize things a lot, I'm proposing these answers as a default behavior, not as hard requirements:

Group attributes by root namespace.

server.address goes to ServerAttributes class and/or file.
http.request.method goes to HttpAttributes - this decision is irreversible
attributes without namespaces MUST get one (alternative is to move them to common Attributes class)

Stable and/or experimental semconv artifacts: up to language SIG.

Jinja templates allow to filter based on stability already
If instrumentation libraries depend on semconv library, we need to have two versions/artifacts for stable and experimental semconv

Strict(ish) consistency across languages: no

it seems they are already mostly consistent and by providing sample templates and generation commands we can keep it this way.
Jinja templates are too hard to write, so having common examples would be great

Write validation tools preventing breaking changes. (API checks, ideally some checks in semconv repo)

Experiment with generating delta between vOld and vNew.
exclude specific attribute namespaces from code generation

Ship dedicated semconv libraries: up to language SIG.

SIGs may decide to provide source generators for user apps, or use shading/shared code in instrumentation libs

@lmolkova, Here are some feedback based on my explorations with the Application Telemetry Schema project and the creation of a Client SDK Generator (similar to protobuf).

To offer the best experience to their users, it seems important to me to allow enough freedom for language SIGs to respect the naming conventions and idioms of each language.
Moving towards a 1 to many relationship between attributes, metrics, and telemetry signals or objects seems crucial and perfectly aligned with what I am trying to do around the Application Telemetry Schema proposal.
Grouping attributes or metrics by root namespace or domain also seems to be a step in the right direction.
For the project I am working on (Client SDK Generator), I also use a template engine compatible with Jinja2. However, I have implemented the following improvements with the aim of making life easier for implementors of Client SDKs, which I think could also be used in the context of this GH issue:

each language can define one or more templates organized in one or multiple sub-directories for generating the client SDKs.
two types of templates exist:

templates that apply to the entire Resolved Application Telemetry Schema (i.e., without reference, and where all override rules have been applied).
templates that apply to a specific instance of a telemetry object. For example, a template named span.jinja will be applied as many times as there are spans in the resolved application telemetry schema. A custom functionis exposed to allow template authors to define the name of the result file from applying this template to the specific instance of the span (or metric, log, event, etc.). This allows to have a different source code file for each root namespace for example.

a series of custom functions and filters are also exposed and accessible at the template level to make life easier for the authors of these templates. This also helps to unify certain functions.
a YAML configuration file can also be optionally added to parameterize the templates or to guide the generation tool. For example, an OTel data type to Language data type mapping table can be defined in this config file. This mapping will be used by a mapping function/filter exposed in the template to facilitate type conversions.

JS splitting the resource attributes out from the other attributes is more of a historical accident than a considered decision. We're currently working on a 2.0 to fix a lot of our historical mistakes right now so I wouldn't worry too much about the current situation.
The primary concern for JS moving forward for something like this is code size. This means no namespaces in the generated code if we can avoid it, and ideally duplicated strings and substrings are kept to a minimum. Our ideal generated code would look something like this:
// declare all common substrings
const http = 'http';
const response = 'response';
... et cetera

export const HTTP_RESPONSE_STATUS_CODE = dotJoin(http, response, 'status_code'); // no need to declare single use strings separately

code written this way is friendly to tree shakers and code minifiers which is very important for JS, particularly in browser contexts. @msnev may be able to expand on this or fill in details i've missed.

Group attributes by root namespace.

👍

Write validation tools preventing breaking changes. (API checks, ideally some checks in semconv repo)

Yeah I think we need checks in both the generation repos and in semantic-conventions. Ideally semconv would have strict checks preventing any breaking changes, but the compatibility rules might not be the same in semconv as in all languages, and so language code generation repos need to verify against their own rules.

@jack-berg curious what you think about above approach and if you'd be concerned about diamond dependency problem from the Java side (outside of the agent).

A little more context with this...

The 1_26 namespace would contain all stable conventions
The 1_26Experimental namespace would contain all stable and experimental conventions
The 1_26Experimental would be updated (add to and changed) until 1.27 is marked stable.

Example Namespaces (.NET focused)

OpenTelemetry.SemanticConventions.1_26_0.Http
OpenTelemetry.SemanticConventions.1_26_0Experimental.Http

For library Releases it gets a little harder

Release version 1.26.0nn will be released incrementally as new conventions are updated in the preview specs
Release version 1.26.xnn will include all previous stable AND experimental classes from a point(1)

Below that, the structure would be the same, and contain namespaces that correspond to semantic conventions sections (.Http and .Messaging etc.)
The general idea being a library can choose to stick with only stable conventions, and be comfortable that regardless of which semconv dependency the client application uses, the conventions will be there (adhering to minimum version constraints).
The concern I had was having the older 1.26 yaml's available for regeneration purposes. However, I think that could be handle through having the tooling work with git tags from the semantic conventions.
(1) what this point is would be debatable, I would say everything starts at 1.26, but it's arguable that this would be "latest stable available at the point that the language started using them. However, with using git tags, I feel we can make sure that all libraries use the same starting point, and therefore libraries that are cross-language and rely on older conventions can still use them.

thanks for the context @martinjt and having a default namespace with stable things sounds great.
A small slarification:

Release version 1.26.0nn will be released incrementally as new conventions are updated in the preview specs

The new minor version of semconv is released  ~every month and older versions are never changed, so essentially once 1.26.0 is generated, it never needs to be updated again. Attribute may change in 1.27.0 or become stable in 1.28.0.

I suppose if there's a regular cadence of the minor versions being released then it might not matter. The idea there is so that when someone is adding new items to the semantic conventions, they don't have to wait for a "version" to happen for months on end, therefore causing them not to use the libraries and therefore negating the point of them being there.
The other thing I think would be really useful is to have a workflow convention so that generation, and ultimately creating maybe a PR, for the new version can be done by the semantic conventions repository to reduce the time that each SIG takes to release the new package. If those are stuck behind in the various languages, we'll get into the same state as we have been around the perception of instability and maturity across languages.

The other thing I think would be really useful is to have a workflow convention so that generatio

Great point!
Created #565 - it should be possible to do it today (as long as it's possible to detect new semconv version), but ultimately jinja templates and generation scripts can't be centralized (e.g. JS doesn't mind having all attributes in one file, but cares about code size).

From this one Go Maintainer's perspective:

As long as the generation tooling doesn't break we are not opposed to any layout changes
It is desirable to have the semantic conventions laid out in files that matched their groups so we can find them easier

@lmolkova joined the 11/30/23 java SIG and we talked about this issue a bit.. On the topic of the diamond dependency problem, we discussed that it would only really be a problem for experimental conventions. Since we have strong backwards compatibility guarantees for the stable generated code, if two libraries have different dependencies on the semconv artifact, they can resolve by using the most recent. This is similar to expectations around resolving dependency conflicts on the API.
However, we currently generate a constant SCHEMA_URL which references the schemaUrl of the semantic convention version the constants were generated from. This will be problematic if different libraries depend on different versions, because one library could intent to indicate that they publish telemetry conforming to 1.22.0, but because of dependency resolution, end up publishing a schema url corresponding to 1.23.0, which has implications on consumes. We'll need to either remove this constant, encode the version in the name of the constant, or something similar.
The experimental conventions ARE problematic since because we don't have compatibility guarantees, we won't be able to instruct users to simply resolve to the most recent version. I think @lmolkova suggestion here is a good solution to that problem. She also mentioned in the java SIG that we might only support generated attributes for some N recent releases of semantic attributes, where N is 5 or something. This would allow us to avoid a bloated artifact, and tell users to upgrade to the latest version, which would work most but not all of the time. This creates an incentive to move conventions from experimental to stable more quickly, which I think is a good thing.

@dyladan thanks for the input!
I believe it's possible (but not easy) to generate code in the way you mentioned even today. It's also possible to add some helpers methods into build-tools to make it much easier.
I just realized you're also suggesting the approach with including the version into import path  import {} from<EMAIL_ADDRESS>as we discussed later in that issue - would the artifact size be a problem here?
Based on the experiment in Java (https://github.com/open-telemetry/semantic-conventions-java/pull/40) we see that every version adds ~70kb to the artifact size.

I believe it's possible (but not easy) to generate code in the way you mentioned even today. It's also possible to add some helpers methods into build-tools to make it much easier.

It is indeed possible (and indeed, not easy). example draft: https://github.com/open-telemetry/opentelemetry-js/pull/4298

I just realized you're also suggesting the approach with including the version into import path import {} from<EMAIL_ADDRESS>as we discussed later in that issue - would the artifact size be a problem here?
Based on the experiment in Java (https://github.com/open-telemetry/semantic-conventions-java/pull/40) we see that every version adds ~70kb to the artifact size.

I don't expect artifact size to be as big of a concern because that method is very friendly to tree-shaking. JS targets that are sensitive to bundle size usually use bundlers so any unused import paths are simply not included in the final bundle. It would increase the size of the package to download, which will have some impact.
Another possibility is import {} from '@otel/semconv' (stable semconv only) and import {} from '@otel/semconv/unstable' (experimental only) which would only be 2 versions (maybe also an /all path). This would potentially result in breaks if a library used an unstable semconv, but a library would typically pin the version they use anyway. The biggest concern for breaks would be if an end-user made custom instrumentation that used the library, but I think an /unstable path is a pretty strong signal of what users should expect.

Before creating the above PR to "attempt" to pull out separate strings, I was envisaging either not using the jinga templates or have to post process the strings out to find the longest common strings to reduce the code size (I opted for the above semi-manual approach or eye-balling the common strings).
Some additional inputs / outputs that I would prefer to see from generation (as that is what this issue is really about, and if I can find time I may prototype) would be that the generation would.

Include ALL of the conventions (strings) for a range of "versions" (say from v1.1 to 1.x), where duplicate strings are only listed once (so you don't have semantic convention version number included in the name or namespace.
As part of this each string could be tagged with the version that it was introduced (or deprecated) in
This would also ideally, have the ability to "drop" (exclude) semantic conventions that have been deprecated for what ever period.

Part of the above could also be achieved by including the additional meta data in the "current" semantic conventions, but that would also require that any of the "dropped" strings are never removed.
To summarize the JavaScript approach as mentioned by @dyladan

JavaScript does NOT want to use namespaces (so that unused / packaged values are tree-shaking out of the packaged bundles)
Because strings cannot be minified, we also would prefer to reduce the overall duplication of strings and have them "created" at runtime (only if required / referenced)

But for languages that don't have the unique challenges of client environments

If they want / prefer to use namespaces, should they be able to be grouped (absolutely), as the overall code size will always be reduced by excluding unused values.
Should there be multiple "namespaces" based on the "version" of the semantic conventions

This will depend on each language (IMHO), so it should be "possible" but not mandatory as stated above my preference would be to include every value in the "group" for the selected range of conventions into a single generated code, with documentation comments highlighting when it was added / removed along with the description etc.

I'm repeating myself so that's it for now.

For swift, we manually generate the semconv when necessary. We essentially have to re-learn how to do it every time, so automatic generation would be greatly appreciated. I don't have great opinion on how they are organized. It may be wise to separate out experimental into their own library, but it would be more for the convenience of downstream dependents. I envision the semconv libraries to be packaged with the sdk, since the instrumentation is dependent on them.

to @lmolkova
cc @open-telemetry/cpp-maintainers
About semantic conventions in C++, from my own perspective:

Grouping of attributes

Group by root namespace sounds good.

Experimental or deprecated semconv

It should be possible to test for experimental semconv in the template, to decide to filter it out or not.
Testing for deprecated semconv is good, but this is not enough in my opinion: what I would also like, is to have a deprecated_since_version property in the yaml file, accessible in the template, so that code generation can decide for example to generate code only if deprecated_since_version >= 1.30 when generating code for semconv 1.35, in effect enforcing that deprecated symbols are maintained only for a given time, not forever. Having this field, each SIG can decide to use it or not.

Workflow for renaming semconv

When "old.thing" is renamed to "new.thing", the yaml file should not just rename the entry.
Instead, please create a new entry "new.thing", so it gets generated, and change the "old.thing" to be deprecated.
This way, both old and new semconv can be generated, without the need for SIG maintainer to manually re introduce an hard coded definition for "old.thing", to avoid breaking changes.

Jinja templates

The complexity of maintaining templates, without knowledge of jinja, without knowledge of the list of primitives available (what is attribute.brief, where does it come from, and how do I know it exists in the first place ?), and without knowing where to find this information, should not be underestimated. Not everyone knows the implementation of the build-tools to find this out.
The most important part is doc with -- working -- examples, and doc that covers the primitives available.

Code generation

One observation from code generation is that it is working reasonably well and is repeatable when using the same tag on semconv.
What is not working well however, is when going from one semconv tag to the next: the code generates, but the output for the new version is unpredictable, generating noisy diffs from the previous release.
I would like to suggest to sort all the semconv by name in the code generation tool first, and then emit code for symbols always in the same order. This way, semconv "foo_a" will always appear before "foo_b" in the generated file. This helps to assess what actually changed, helps with merges, avoid polluting git history with code that just moved, etc.

Semantic convention libraries

Right now, C++ generates semconv header files inside the opentelemetry-cpp repository, keeping up with the latest semconv releases. As a result, users of opentelemetry-cpp are de facto forced to use the latest semconv "library" (header files) that was available when opentelemetry-cpp released, so there is coupling.
We may deliver generated code into a separate repository (opentelemetry-cpp-semconv) later, this is undecided for now, and to be investigated. This would allow an instrumented application to pick opentelemetry-cpp and semconv versions independently, and allow independent release cycles.

What @marcalff wrote up summarizes the situation and opinions for @open-telemetry/erlang-maintainers, as well. We do have an additional complication where Elixir supports deprecated annotations for users but Erlang does not. We'd like to figure that out but that's a codegen issue for us. Either way, it would helpful to to have that and maintaining backwards compatibility for some period of time before it's hard-deprecated/removed is essential imo.

noisy diffs is an annoyance for PHP sig (we have started sorting the attributes by fqn to make diffs easier to comprehend)
being able to exclude classes of attributes that aren't relevant would be nice (I can't imagine we'd ever need jvm., device. etc for our SIG)
we currently generate TraceAttributes and ResourceAttributes, but I'd imagine in the future we'll want to generate some attributes specific to logs and metrics

Thank you all for the information and the feedback. Let me summarize it and break it down into different categories (let me know if I missed something)
Process issues

We should never remove attributes (even experimental ones), but mark them deprecated. Alternatively, codegen tooling should generate attributes that were removed between version X and Y.
Codegen is never validated until semconv/tooling is released

Tooling issues

no clear documentation on otel + Jinja templates (what's available, how to use)

exclude certain attribute groups/namespaces (putting it here because it's possible, probably just need better docsc)
noisy diffs, no sorting - same here, no new features needed, just docs on how to do it + better defaults
generating metric definitions, attributes across different signals - it's supported, need better docs

not fully automated: need to re-learn how to do generation

Tooling feature requests

whatever we do should be backward compatible with current jinja templates and scripts
More flexible grouping (by namespace, or no grouping, but possibly access to namespace fragments and trie-like attribute structure)
ability to separate stable/unstable conventions
better defaults:

sorting attributes to create better diffs

Yaml feature requests

deprecated_since property

I believe we have a prototype  that addresses most of tooling feature-requests - otel/semconvgen:feature-codegen-by-namespace
It allows to

no changes for existing usage
group attributes by root namespace when --file-per-group  root_namespace is added. As a result it generates ServerAttributes, ClientAttributes, HttpAttributes, etc files
it adds a bunch of convenience methods to jinja such as is_stable, is_deprecated, is_template, etc
simplifies jinja by providing a filtered list of attribute definitions (excludes refs)

Usage is demonstrated in

https://github.com/open-telemetry/semantic-conventions-java/pull/40
https://github.com/open-telemetry/opentelemetry-python/pull/3586
you can find jinja examples in the tests

There are some rough edges that we need to polish (e.g. file names pattern depends on the langauge), but it's open to experiment with.
What it does not solve:

JavaScript case when we'd want to represent all attributes in a trie-like structure and format attribute names  at runtime. I can imagine an incremental change that would allow it though.
We still need better processes, automated checks for codegen and docs

hey @lmolkova is this issue still relevant?

What it does not solve:
* JavaScript case when we'd want to represent all attributes in a trie-like structure and format attribute names  at runtime. I can imagine an incremental change that would allow it though.

(@dyladan Please correct me if I'm wrong.) This is no longer a requirement for JavaScript's semconv generation. We are and will be generating straightforward constants, something like export const ATTR_HOST_NAME = 'host.name'; or similar.


4-pane Merge View

It is great to have 3-pane Merge View (https://github.com/microsoft/vscode/issues/69857). However, industry has an even better way to do a merge - 4-pane merge view! The 4-th pane stands for merge base. So the full list of panes is: ours, theirs, base, result.
See how it is done in e.g. kdiff3 (though I prefer to place "base" in the middle):

It really helps a lot in complicated conflicts. Should be quite easy to implement with the current advanced layout mechanism already present in VS Code.

Duplicate of https://github.com/microsoft/vscode/issues/155277

/duplicate - thanks @Igorgro

Thanks for creating this issue! We figured it's covering the same as another one we already have. Thus, we closed this one as a duplicate. You can search for similar existing issues. See also our issue reporting guidelines.
Happy Coding!


GEODE-4816: Add support for authentication to the experimental driver.

Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?

[x] Has your PR been rebased against the latest commit within the target branch (typically develop)?

[x] Is your initial contribution a single, squashed commit?

[x] Does gradlew build run cleanly?

[x] Have you written or updated unit tests to verify your changes?

[n/a] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to<EMAIL_ADDRESS>

Closing to recreate with @galen-pivotal's changes for putIfAbsent.


pip install vs pipenv install for package graphene>=2.0.dev

pip install graphene>=2.0.dev works fine, but pipenv install "graphene>=2.0.dev"  returns:
Locking [packages] dependencies…
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches graphene>=2.0.dev
Tried: 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.1.4, 0.1.5, <IP_ADDRESS>, <IP_ADDRESS>, <IP_ADDRESS>, 0.1.6a1, <IP_ADDRESS>, <IP_ADDRESS>, 0.3.0, 0.4.0, <IP_ADDRESS>, 0.4.1, <IP_ADDRESS>, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9b1, 0.9, 0.9.1, 0.10.0, 0.10.1, 0.10.2, 1.0.dev20160815004752, 1.0.dev20160816073455, 1.0.dev20160822080320, 1.0.dev20160823061102, 1.0.dev20160909040318, 1.0.dev20160909055438, 1.0.dev20160911044410, 1.0.dev20160911051803, 1.0.dev20160917190505, 1.0.dev20160918041239, 1.0.dev20160920015515, 1.0.dev20160920070441, 1.0.dev20160921153356, 1.0.dev20160922020647, 1.0, 1.0.1, 1.0.2, 1.1, 1.1.1, 1.1.2, 1.1.3, 1.2, 1.3, 1.4, 1.4.1, 2.0.dev20170724064308, 2.0.dev20170725043308, 2.0.dev20170725061556, 2.0.dev20170727024417, 2.0.dev20170727061432, 2.0.dev20170801053013, 2.0.dev20170801053013, 2.0.dev20170802065539, 2.0.dev20170802065539

Ok – so I get that graphene>=2.0.dev isn't specific enough to build a hash against. So, I've tried pipenv install "graphene==2.0.dev20170802065539", but get the same error. What should I be specifying here?

It looks like graphene has conflicting sub dependencies.  I don't think you can hash it without resolving that dependency resolution issue in graphene.  @kennethreitz correct me if I'm wrong?

did you try the --pre flag?

You'll also need to use pipenv lock --pre

Since it appears as though you're using a prerelease.

I tried --pre last night and it didn't seem to work, was very very tired though...

this might be a pip-tools bug

weird, i can't get it to find a version at all.

You want pipenv install --pre graphene

that'll grab the latest pre release and add it to your lockfile.

make sure whenever you run pipenv lock you also pass the --pre flag.

This currently resolves to ==2.0.dev20170802065539

If graphene is in my setup.py, how would I specify --pre for it?

What does your pipfile look like?

$ pipenv lock --pre followed by $ pipenv install

your version selector in your setup.py should be blank though, i think.

Ok – so let's take this package I authored over the weekend. It uses graphql-core>=2.0.dev and graphql-server-core>=1.0.dev.
Here is an ignorant approach (still specifying graphene>=2.0.dev in setup.py):
cd aiohttp-graphql
pipenv --three
pipenv install "-e ."  # Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies. [...]

Here is my approach without specifying a version for graphql-core or graphql-server-core in setup.py.
cd aiohttp-graphql
pipenv --rm
rm Pipfile*
# edit setup.py to not include versions for `graphql-core` and `graphql-server-core`.

pipenv --three
pipenv install "-e ."
# Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
#  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
# Could not find a version that matches graphql-server-core
# Tried: 1.0.dev20170322001

I guess I'm stuck.

OK, so I've got something working. It comes with a caveat though.
pipenv --three
pipenv install "-e ." --pre

This approach supports pinning a pre-release version in setup.py (good so that users of my package can actually use the package).
However (the caveat), is that all future installs also need to include --pre. I.e. pipenv install ipython fails to generate a lockfile, while pipenv install ipython --pre will succeed. So it seems that --pre is global, and I need to be somewhat explicit when installing packages to specify their version or version range or else they won't get the latest stable, but a "pre-release" version as well.

yes, you'll always have to pass the --pre flag. I'll see about adding that as an option to the pipfile.

now, in master you can have the following in your pipfile:
[pipenv]
pre = true

and the --pre flag will automatically be honored.

this setting will not automatically get written to your pipfile

this is now allow_prereleases

released

Interesting approach. But is this not doing brain surgery with a bludgeon? Should pre-release flags not be "per-requirement"?
I'm sure that'd require a more technically sophisticated approach, but it seems much safer. What do you think?

it's all or nothing, i'm afraid.

unless you want to re-write all of piptools to support this

which... we're not doing :)

FWIW: I'm hitting the same issue, in that I need to install an "rc1" for one specific package, but can't "allow_prereleases" for all my other dependencies. I'm sad to hear this is hard to fix. I wish I could just pin the version of the one package I want a pre-release for in Pipfile, but when I try that, as discussed above, pip doesn't find it. I think that means I'm stuck and have to revert to using pip/requirements.txt again. If anyone has ideas I'm all ears.

I'm a bit surprised here. AFAIK, pip-tools will resolve pre-release versions if the specifier includes a pre-release tag.
Ex:
echo "graphene>=2.0.dev" | pip-compile -o requirements.txt -
gives
graphene==2.0.dev20170802065539
graphql-core==2.0.dev20170801051721  # via graphene, graphql-relay
graphql-relay==0.4.5      # via graphene
promise==2.1.dev20170724043809  # via graphene, graphql-core, graphql-relay
six==1.11.0               # via graphene, graphql-core, graphql-relay, promise
typing==3.6.2             # via promise

And even if it's the sub dependencies of another package:
Ex:
echo "aiohttp-graphql" | pip-compile -nv -o requirements.txt -
gives
aiohttp-graphql==1.0.0
aiohttp==2.2.5            # via aiohttp-graphql
async-timeout==1.4.0      # via aiohttp
chardet==3.0.4            # via aiohttp
graphql-core==2.0.dev20170801051721  # via aiohttp-graphql, graphql-server-core
graphql-server-core==1.0.dev20170322001  # via aiohttp-graphql
multidict==3.2.0          # via aiohttp, yarl
promise==2.1.dev20170724043809  # via graphql-core, graphql-server-core
pytest-runner==2.12.1     # via aiohttp-graphql
six==1.11.0               # via graphql-core, promise
typing==3.6.2             # via promise
yarl==0.13.0              # via aiohttp

This doesn't seem like  pip-tools bug, but I can't really explain what's going on, as I don't know much of how pipenv integrates with pip-tools.

you can always specify an exact version


ContactSensor not changing state

I cannot get this to work, is the contactsensor function tested?
I try to send mk/status/Ytterdorr/STATUS with { "var" : "door close" } and { "var" : "door open" }
snippet from the homekit2mqtt.json
    "mk//MK Ytterdörr": {
    "service" : "ContactSensor",
    "name" : "Ytterdörr",
    "topic": {
      "statusContactSensorState": "mk/status/Ytterdorr/STATUS"
    },
    "payload": {
        "onContactDetected" : "door closed"
    },
    "manufacturer": "Siemens",
    "model":        "MK 240"
  }

But it responses the same when I update the values on the iphone and get them sent to HAP:
(from log on screen)
< hap Ytterdörr get ContactSensorState
> hap Ytterdörr 1

I made a proposal for change...

Hi, I merged your PR and also improved the behavior of ContactSensor, it now notifies via setCharacteristic when the value changes.
Regards,
Sebastian


How to properly reference the Metalama.Compiler package

I'm trying to write a source transformer and have followed the steps in the readme for this project, but it doesn't seem to be working. In addition, I couldn't find any other projects or samples which implement source transformers, so I haven't easily been able to figure out what I'm doing wrong.
This is the csproj file of the transformer project
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>library</OutputType>
    <TargetFramework>netstandard2.0</TargetFramework>
    <LangVersion>latest</LangVersion>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <WarningsAsErrors>nullable</WarningsAsErrors>
    <EnforceExtendedAnalyzerRules>true</EnforceExtendedAnalyzerRules>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Metalama.Compiler.Sdk" Version="2023.4.6" />
  </ItemGroup>

</Project>

and this is the source transformer (just a test for now)
using Metalama.Compiler;
using Microsoft.CodeAnalysis.CSharp;

namespace Transform;

[Transformer]
public sealed class SourceTransformer : ISourceTransformer
{
    public void Execute(TransformerContext context)
    {
        context.AddSyntaxTrees(CSharpSyntaxTree.ParseText("""
        using System;

        public static class Foo
        {
            public static void Bar() => Console.WriteLine("uwu");
        }
        """));
    }
}

I did try adding an analyzer to the project and referencing the Microsoft.CodeAnalysis.CSharp package which did recognize the analyzer, however when removing this package reference and just referencing Metalama.Compiler.Sdk the analyzer stopped being recognized.
Is there anything I'm missing?

Hello @thinker227, thank you for submitting this issue. We will try to get back to you as soon as possible.
Note to the PostSharp team, this ticket is being tracked in our dashboard under ID TP-34298.

Your test project must reference the Metalama.Compiler package.

Ah, nice. Thanks!


Changes in the Report and the updated resulting data

[x] Relax SpecImportPeerOIFPS to SpecPeerOnlyProviderPolicies.

[x] Relax SpecImportCustomerOIFPS to SpecCustomerOnlyProviderPolicies.

[x] Add SpecImportCustomer, like SpecImportFromNeighbor.

Do we split BadImport/BadExport to the peering/ filter case, or do we just distinguish them after the report generation using whether err_filter is 0? @cunha No.

Do we split the special cases to have whitelisted cases? We could also just distinguish them after the report generation using spec_uphill_tier1 and other counts. @cunha No.

We need to figure these out before regenerating the reports. If no changes are needed, we don't need to regenerate the reports. We need to regenerate the reports for the Only Provider Policies.

If we have the information required for computing the splits in the current output, then we don't need to regenerate the report. We could post-process the existing reports. I usually go for minimizing code complexity, in this case weighting (1) writing code to post-process the existing Reports vs (2) changing the Report-generation code to make the post-processing simpler/trivial.

@cunha, in Import Customer, you are talking about importing Any from customers, but the rules we actually have is for "neighbors". Either we need to change the text, or the rules.

SpecImportCustomer stole slightly over half of the cases for SpecImportFromNeighbor.
And, SpecTier1Pair saw a significant growth, probably because I fixed the pattern matching not falling through while making the other changes.
-         spec_uphill  spec_uphill_tier1  spec_tier1_pair  spec_import_peer_oifps  spec_import_customer_oifps  spec_export_customers  spec_import_from_neighbor  spec_as_is_origin_but_no_route  spec_as_set_contains_origin_but_no_route
- count  2.644760e+07       2.644760e+07     2.644760e+07            2.644760e+07                2.644760e+07           2.644760e+07               2.644760e+07                    2.644760e+07                              2.644760e+07
- mean   4.756757e-01       3.257567e-01     1.092092e-01            1.500892e-02                1.237541e-04           1.716322e-02               7.100750e-02                    1.095841e-01                              5.669660e-02
- std    8.403122e-01       5.378626e-01     3.168373e-01            1.215965e-01                1.112380e-02           1.668009e-01               4.407332e-01                    4.826153e-01                              4.277722e-01
- min    0.000000e+00       0.000000e+00     0.000000e+00            0.000000e+00                0.000000e+00           0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00
- 25%    0.000000e+00       0.000000e+00     0.000000e+00            0.000000e+00                0.000000e+00           0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00
- 50%    0.000000e+00       0.000000e+00     0.000000e+00            0.000000e+00                0.000000e+00           0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00
- 75%    1.000000e+00       1.000000e+00     0.000000e+00            0.000000e+00                0.000000e+00           0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00
- max    9.000000e+00       2.000000e+00     2.000000e+00            2.000000e+00                1.000000e+00           1.700000e+01               2.400000e+01                    3.700000e+01                              7.200000e+01
+         spec_uphill  spec_uphill_tier1  spec_tier1_pair  spec_peer_only_provider_policies  spec_customer_only_provider_policies  spec_export_customers  spec_import_customer  spec_import_from_neighbor  spec_as_is_origin_but_no_route  spec_as_set_contains_origin_but_no_route
+ count  2.644760e+07       2.644760e+07     2.644760e+07                      2.644760e+07                          2.644760e+07           2.644760e+07          2.644760e+07               2.644760e+07                    2.644760e+07                              2.644760e+07
+ mean   4.662592e-01       3.183927e-01     2.539163e-01                      1.641253e-02                          9.577429e-05           1.827671e-02          2.605257e-02               2.110033e-02                    5.095706e-02                              1.488434e-02
+ std    8.275327e-01       5.312293e-01     5.427857e-01                      1.278851e-01                          9.785965e-03           1.342874e-01          1.628525e-01               1.437386e-01                    2.511928e-01                              1.365160e-01
+ min    0.000000e+00       0.000000e+00     0.000000e+00                      0.000000e+00                          0.000000e+00           0.000000e+00          0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00
+ 25%    0.000000e+00       0.000000e+00     0.000000e+00                      0.000000e+00                          0.000000e+00           0.000000e+00          0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00
+ 50%    0.000000e+00       0.000000e+00     0.000000e+00                      0.000000e+00                          0.000000e+00           0.000000e+00          0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00
+ 75%    1.000000e+00       1.000000e+00     0.000000e+00                      0.000000e+00                          0.000000e+00           0.000000e+00          0.000000e+00               0.000000e+00                    0.000000e+00                              0.000000e+00
+ max    9.000000e+00       2.000000e+00     3.000000e+00                      3.000000e+00                          1.000000e+00           2.000000e+00          3.000000e+00               2.000000e+00                    3.000000e+00                              4.000000e+00

Following up. The route special case percentages stacked area plot changes dramatically in the Tier1Pair part, compared to https://github.com/SichangHe/internet_route_verification/issues/91#issue-1986772999. @cunha.

I would have preferred if the special cases on the left increased... but oh well... I think with the 50/50 split between import_customer and import_neighbor we should just stick with import_neighbor to reduce noise. For the graph it might also make sense to join some of the other classes (in the figure and in the text, if we have not already merged them in the text).

New stats data generation after d7c43aafc4aa6d23b3edea75f0dad5555867405a.
$ cargo r --release                     Steven-VM internet_route_verification/route_verification/rib_stats   main
    Finished release [optimized] target(s) in 10.62s
     Running `/home/sichanghe/internet_route_verification/route_verification/target/release/route_verification_rib_stats`
[2024-04-02T13:34:58Z INFO  route_verification_rib_stats] Starting...
[2024-04-02T13:34:58Z DEBUG route_verification_as_rel] Loading AS relationship database `"../../data/20230701.as-rel.bz2"`.
[2024-04-02T13:34:58Z DEBUG route_verification_as_rel] Loaded AS relationship database with 19 cliques & 497271 links.
[2024-04-02T13:35:03Z DEBUG route_verification_rib_stats] Loaded AS Relationship DB and IR for query
[2024-04-02T13:35:03Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.napafrica--rib.20230623.0000.bz2` for collector `route-views.napafr
ica`.
2024-04-02 13:35:03 [info] logging to syslog
Generated stats for 81249 ASes, 173996 AS pairs, 13660056 routes for route-views.napafrica in 2m 42s 937ms.
[2024-04-02T13:37:46Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.napafrica`.
[2024-04-02T13:37:46Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.napafrica` in 521ms.
[2024-04-02T13:37:47Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.napafrica` in 1s 250ms.
[2024-04-02T13:37:47Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc14--bview.20230623.0000.gz` for collector `rrc14`.
2024-04-02 13:37:47 [info] logging to syslog
Generated stats for 81221 ASes, 155176 AS pairs, 9053028 routes for rrc14 in 2m 9s 301ms.
[2024-04-02T13:39:57Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc14`.
[2024-04-02T13:39:57Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc14` in 504ms.
[2024-04-02T13:39:58Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc14` in 1s 100ms.
[2024-04-02T13:39:58Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.telxatl--rib.20230623.0000.bz2` for collector `route-views.telxatl`
.
2024-04-02 13:39:58 [info] logging to syslog
Generated stats for 81094 ASes, 155943 AS pairs, 7680878 routes for route-views.telxatl in 2m 1s 275ms.
[2024-04-02T13:42:00Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.telxatl`.
[2024-04-02T13:42:00Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.telxatl` in 501ms.
[2024-04-02T13:42:01Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.telxatl` in 1s 94ms.
[2024-04-02T13:42:01Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.gorex--rib.20230623.0000.bz2` for collector `route-views.gorex`.
2024-04-02 13:42:01 [info] logging to syslog
Generated stats for 0 ASes, 0 AS pairs, 0 routes for route-views.gorex in 5ms.
[2024-04-02T13:42:01Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.gorex`.
[2024-04-02T13:42:01Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.gorex` in 0ms.
[2024-04-02T13:42:01Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.gorex` in 0ms.
[2024-04-02T13:42:01Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.fortaleza--rib.20230623.0000.bz2` for collector `route-views.fortal
eza`.
2024-04-02 13:42:01 [info] logging to syslog
Generated stats for 81143 ASes, 158677 AS pairs, 5411107 routes for route-views.fortaleza in 1m 8s 521ms.
[2024-04-02T13:43:10Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.fortaleza`.
[2024-04-02T13:43:10Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.fortaleza` in 501ms.
[2024-04-02T13:43:11Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.fortaleza` in 1s 108ms.
[2024-04-02T13:43:11Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc21--bview.20230623.0000.gz` for collector `rrc21`.
2024-04-02 13:43:11 [info] logging to syslog
Generated stats for 81316 ASes, 190029 AS pairs, 23839239 routes for rrc21 in 4m 55s 23ms.
[2024-04-02T13:48:06Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc21`.
[2024-04-02T13:48:07Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc21` in 515ms.
[2024-04-02T13:48:08Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc21` in 1s 339ms.
[2024-04-02T13:48:08Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc20--bview.20230623.0000.gz` for collector `rrc20`.
2024-04-02 13:48:08 [info] logging to syslog
Generated stats for 81306 ASes, 193112 AS pairs, 30046545 routes for rrc20 in 6m 36s 198ms.
[2024-04-02T13:54:45Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc20`.
[2024-04-02T13:54:45Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc20` in 508ms.
[2024-04-02T13:54:46Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc20` in 1s 340ms.
[2024-04-02T13:54:46Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views5--rib.20230623.0000.bz2` for collector `route-views5`.
2024-04-02 13:54:46 [info] logging to syslog
Generated stats for 81344 ASes, 172541 AS pairs, 17765091 routes for route-views5 in 4m 29s 123ms.
[2024-04-02T13:59:15Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views5`.
[2024-04-02T13:59:16Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views5` in 509ms.
[2024-04-02T13:59:17Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views5` in 1s 240ms.
[2024-04-02T13:59:17Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views6--rib.20230623.0000.bz2` for collector `route-views6`.
2024-04-02 13:59:17 [info] logging to syslog
Generated stats for 31850 ASes, 72440 AS pairs, 4836395 routes for route-views6 in 1m 0s 723ms.
[2024-04-02T14:00:18Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views6`.
[2024-04-02T14:00:18Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views6` in 201ms.
[2024-04-02T14:00:19Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views6` in 500ms.
[2024-04-02T14:00:19Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc18--bview.20230623.0000.gz` for collector `rrc18`.
2024-04-02 14:00:19 [info] logging to syslog
Generated stats for 80740 ASes, 120578 AS pairs, 1110311 routes for rrc18 in 9s 326ms.
[2024-04-02T14:00:28Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc18`.
[2024-04-02T14:00:28Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc18` in 489ms.
[2024-04-02T14:00:29Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc18` in 808ms.
[2024-04-02T14:00:29Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.ny---rib.20230623.0000.bz2` for collector `route-views.ny`.
2024-04-02 14:00:29 [info] logging to syslog
Generated stats for 80717 ASes, 119863 AS pairs, 1087343 routes for route-views.ny in 12s 803ms.
[2024-04-02T14:00:42Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.ny`.
[2024-04-02T14:00:43Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.ny` in 491ms.
[2024-04-02T14:00:43Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.ny` in 845ms.
[2024-04-02T14:00:43Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.gixa--rib.20230623.0000.bz2` for collector `route-views.gixa`.
2024-04-02 14:00:43 [info] logging to syslog
Generated stats for 0 ASes, 0 AS pairs, 0 routes for route-views.gixa in 6ms.
[2024-04-02T14:00:43Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.gixa`.
[2024-04-02T14:00:43Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.gixa` in 0ms.
[2024-04-02T14:00:43Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.gixa` in 0ms.
[2024-04-02T14:00:43Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc12--bview.20230623.0000.gz` for collector `rrc12`.
2024-04-02 14:00:43 [info] logging to syslog
Generated stats for 81529 ASes, 236282 AS pairs, 48361161 routes for rrc12 in 11m 22s 688ms.
[2024-04-02T14:12:06Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc12`.
[2024-04-02T14:12:07Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc12` in 511ms.
[2024-04-02T14:12:08Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc12` in 1s 656ms.
[2024-04-02T14:12:08Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc26--bview.20230623.0000.gz` for collector `rrc26`.
2024-04-02 14:12:08 [info] logging to syslog
Generated stats for 81143 ASes, 151199 AS pairs, 4133258 routes for rrc26 in 43s 332ms.
[2024-04-02T14:12:52Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc26`.
[2024-04-02T14:12:52Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc26` in 515ms.
[2024-04-02T14:12:53Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc26` in 1s 23ms.
[2024-04-02T14:12:53Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc16--bview.20230623.0000.gz` for collector `rrc16`.
2024-04-02 14:12:53 [info] logging to syslog
Generated stats for 80973 ASes, 142073 AS pairs, 3636661 routes for rrc16 in 1m 1s 154ms.
[2024-04-02T14:13:54Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc16`.
[2024-04-02T14:13:55Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc16` in 501ms.
[2024-04-02T14:13:56Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc16` in 971ms.
[2024-04-02T14:13:56Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc05--bview.20230623.0000.gz` for collector `rrc05`.
2024-04-02 14:13:56 [info] logging to syslog
Generated stats for 81162 ASes, 164535 AS pairs, 11475978 routes for rrc05 in 2m 58s 341ms.
[2024-04-02T14:16:54Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc05`.
[2024-04-02T14:16:55Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc05` in 506ms.
[2024-04-02T14:16:56Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc05` in 1s 152ms.
[2024-04-02T14:16:56Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.uaeix--rib.20230623.0000.bz2` for collector `route-views.uaeix`.
2024-04-02 14:16:56 [info] logging to syslog
Generated stats for 81223 ASes, 155134 AS pairs, 6117293 routes for route-views.uaeix in 1m 17s 613ms.
[2024-04-02T14:18:13Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.uaeix`.
[2024-04-02T14:18:14Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.uaeix` in 496ms.
[2024-04-02T14:18:15Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.uaeix` in 1s 58ms.
[2024-04-02T14:18:15Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.decix-jhb--rib.20230623.0000.bz2` for collector `route-views.decix-jhb`.
2024-04-02 14:18:15 [info] logging to syslog
Generated stats for 22510 ASes, 26888 AS pairs, 639827 routes for route-views.decix-jhb in 4s 643ms.
[2024-04-02T14:18:20Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.decix-jhb`.
[2024-04-02T14:18:20Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.decix-jhb` in 140ms.
[2024-04-02T14:18:20Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.decix-jhb` in 181ms.
[2024-04-02T14:18:20Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.sfmix--rib.20230623.0000.bz2` for collector `route-views.sfmix`.
2024-04-02 14:18:20 [info] logging to syslog
Generated stats for 81045 ASes, 148630 AS pairs, 4761696 routes for route-views.sfmix in 1m 2s 731ms.
[2024-04-02T14:19:23Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.sfmix`.
[2024-04-02T14:19:23Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.sfmix` in 497ms.
[2024-04-02T14:19:24Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.sfmix` in 1s 12ms.
[2024-04-02T14:19:24Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.isc--rib.20230623.0000.bz2` for collector `route-views.isc`.
2024-04-02 14:19:24 [info] logging to syslog
Generated stats for 81310 ASes, 162170 AS pairs, 6357136 routes for route-views.isc in 1m 9s 995ms.
[2024-04-02T14:20:34Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.isc`.
[2024-04-02T14:20:35Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.isc` in 508ms.
[2024-04-02T14:20:36Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.isc` in 1s 157ms.
[2024-04-02T14:20:36Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc06--bview.20230623.0000.gz` for collector `rrc06`.
2024-04-02 14:20:36 [info] logging to syslog
Generated stats for 81150 ASes, 147791 AS pairs, 4439957 routes for rrc06 in 53s 153ms.
[2024-04-02T14:21:29Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc06`.
[2024-04-02T14:21:30Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc06` in 494ms.
[2024-04-02T14:21:31Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc06` in 1s 13ms.
[2024-04-02T14:21:31Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc04--bview.20230623.0000.gz` for collector `rrc04`.
2024-04-02 14:21:31 [info] logging to syslog
Generated stats for 81154 ASes, 157736 AS pairs, 6942567 routes for rrc04 in 1m 21s 565ms.
[2024-04-02T14:22:52Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc04`.
[2024-04-02T14:22:53Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc04` in 509ms.
[2024-04-02T14:22:54Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc04` in 1s 104ms.
[2024-04-02T14:22:54Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.perth--rib.20230623.0000.bz2` for collector `route-views.perth`.
2024-04-02 14:22:54 [info] logging to syslog
Generated stats for 81285 ASes, 156626 AS pairs, 15578098 routes for route-views.perth in 3m 46s 743ms.
[2024-04-02T14:26:40Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.perth`.
[2024-04-02T14:26:41Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.perth` in 503ms.
[2024-04-02T14:26:42Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.perth` in 1s 100ms.
[2024-04-02T14:26:42Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.wide--rib.20230623.0000.bz2` for collector `route-views.wide`.
2024-04-02 14:26:42 [info] logging to syslog
Generated stats for 80947 ASes, 127469 AS pairs, 2082419 routes for route-views.wide in 19s 403ms.
[2024-04-02T14:27:01Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.wide`.
[2024-04-02T14:27:02Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.wide` in 494ms.
[2024-04-02T14:27:03Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.wide` in 863ms.
[2024-04-02T14:27:03Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc01--bview.20230623.0000.gz` for collector `rrc01`.
2024-04-02 14:27:03 [info] logging to syslog
Generated stats for 81494 ASes, 208992 AS pairs, 41893437 routes for rrc01 in 10m 16s 146ms.
[2024-04-02T14:37:19Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc01`.
[2024-04-02T14:37:20Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc01` in 516ms.
[2024-04-02T14:37:21Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc01` in 1s 473ms.
[2024-04-02T14:37:21Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.soxrs--rib.20230623.0000.bz2` for collector `route-views.soxrs`.
2024-04-02 14:37:21 [info] logging to syslog
Generated stats for 80704 ASes, 121204 AS pairs, 1634938 routes for route-views.soxrs in 35s 213ms.
[2024-04-02T14:37:56Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.soxrs`.
[2024-04-02T14:37:57Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.soxrs` in 491ms.
[2024-04-02T14:37:58Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.soxrs` in 811ms.
[2024-04-02T14:37:58Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views4--rib.20230623.0000.bz2` for collector `route-views4`.
2024-04-02 14:37:58 [info] logging to syslog
Generated stats for 81608 ASes, 186571 AS pairs, 22453694 routes for route-views4 in 5m 45s 956ms.
[2024-04-02T14:43:43Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views4`.
[2024-04-02T14:43:44Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views4` in 507ms.
[2024-04-02T14:43:45Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views4` in 1s 348ms.
[2024-04-02T14:43:45Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.sg--rib.20230623.0000.bz2` for collector `route-views.sg`.
2024-04-02 14:43:45 [info] logging to syslog
Generated stats for 81587 ASes, 180281 AS pairs, 17884085 routes for route-views.sg in 4m 15s 759ms.
[2024-04-02T14:48:01Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.sg`.
[2024-04-02T14:48:02Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.sg` in 508ms.
[2024-04-02T14:48:03Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.sg` in 1s 290ms.
[2024-04-02T14:48:03Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.bknix--rib.20230623.0000.bz2` for collector `route-views.bknix`.
2024-04-02 14:48:03 [info] logging to syslog
Generated stats for 60 ASes, 58 AS pairs, 2241 routes for route-views.bknix in 13ms.
[2024-04-02T14:48:03Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.bknix`.
[2024-04-02T14:48:03Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.bknix` in 0ms.
[2024-04-02T14:48:03Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.bknix` in 0ms.
[2024-04-02T14:48:03Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.nwax--rib.20230623.0000.bz2` for collector `route-views.nwax`.
2024-04-02 14:48:03 [info] logging to syslog
Generated stats for 80939 ASes, 134945 AS pairs, 2017450 routes for route-views.nwax in 38s 691ms.
[2024-04-02T14:48:42Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.nwax`.
[2024-04-02T14:48:42Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.nwax` in 497ms.
[2024-04-02T14:48:43Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.nwax` in 911ms.
[2024-04-02T14:48:43Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.amsix--rib.20230623.0000.bz2` for collector `route-views.amsix`.
2024-04-02 14:48:43 [info] logging to syslog
Generated stats for 81648 ASes, 206652 AS pairs, 26214280 routes for route-views.amsix in 6m 17s 563ms.
[2024-04-02T14:55:01Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.amsix`.
[2024-04-02T14:55:01Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.amsix` in 523ms.
[2024-04-02T14:55:03Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.amsix` in 1s 454ms.
[2024-04-02T14:55:03Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.chicago--rib.20230623.0000.bz2` for collector `route-views.chicago`.
2024-04-02 14:55:03 [info] logging to syslog
Generated stats for 81189 ASes, 165304 AS pairs, 16128146 routes for route-views.chicago in 4m 40s 834ms.
[2024-04-02T14:59:43Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.chicago`.
[2024-04-02T14:59:44Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.chicago` in 498ms.
[2024-04-02T14:59:45Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.chicago` in 1s 145ms.
[2024-04-02T14:59:45Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc22--bview.20230623.0000.gz` for collector `rrc22`.
2024-04-02 14:59:45 [info] logging to syslog
Generated stats for 81114 ASes, 145031 AS pairs, 3930396 routes for rrc22 in 42s 798ms.
[2024-04-02T15:00:28Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc22`.
[2024-04-02T15:00:28Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc22` in 508ms.
[2024-04-02T15:00:29Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc22` in 1s 21ms.
[2024-04-02T15:00:29Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc11--bview.20230623.0000.gz` for collector `rrc11`.
2024-04-02 15:00:29 [info] logging to syslog
Generated stats for 81381 ASes, 163917 AS pairs, 11232909 routes for rrc11 in 2m 12s 810ms.
[2024-04-02T15:02:42Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc11`.
[2024-04-02T15:02:43Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc11` in 502ms.
[2024-04-02T15:02:44Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc11` in 1s 139ms.
[2024-04-02T15:02:44Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.rio--rib.20230623.0000.bz2` for collector `route-views.rio`.
2024-04-02 15:02:44 [info] logging to syslog
Generated stats for 81157 ASes, 163178 AS pairs, 6792491 routes for route-views.rio in 1m 20s 757ms.
[2024-04-02T15:04:05Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.rio`.
[2024-04-02T15:04:05Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.rio` in 500ms.
[2024-04-02T15:04:06Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.rio` in 1s 124ms.
[2024-04-02T15:04:06Z DEBUG route_verification_rib_stats] Starting to process RIB file `pacwave.lax--rib.20230623.0000.bz2` for collector `pacwave.lax`.
2024-04-02 15:04:06 [info] logging to syslog
Generated stats for 5346 ASes, 5449 AS pairs, 25267 routes for pacwave.lax in 155ms.
[2024-04-02T15:04:06Z DEBUG route_verification_rib_stats] Wrote route stats for `pacwave.lax`.
[2024-04-02T15:04:06Z DEBUG route_verification_rib_stats] Wrote AS stats for `pacwave.lax` in 30ms.
[2024-04-02T15:04:06Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `pacwave.lax` in 34ms.
[2024-04-02T15:04:06Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc15--bview.20230623.0000.gz` for collector `rrc15`.
2024-04-02 15:04:06 [info] logging to syslog
Generated stats for 81402 ASes, 228021 AS pairs, 28395509 routes for rrc15 in 5m 59s 254ms.
[2024-04-02T15:10:06Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc15`.
[2024-04-02T15:10:06Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc15` in 503ms.
[2024-04-02T15:10:08Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc15` in 1s 597ms.
[2024-04-02T15:10:08Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.sydney--rib.20230623.0000.bz2` for collector `route-views.sydney`.
2024-04-02 15:10:08 [info] logging to syslog
Generated stats for 81401 ASes, 174485 AS pairs, 14997508 routes for route-views.sydney in 2m 50s 28ms.
[2024-04-02T15:12:58Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.sydney`.
[2024-04-02T15:12:58Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.sydney` in 496ms.
[2024-04-02T15:13:00Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.sydney` in 1s 214ms.
[2024-04-02T15:13:00Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views3--rib.20230623.0000.bz2` for collector `route-views3`.
2024-04-02 15:13:00 [info] logging to syslog
Generated stats for 81585 ASes, 187684 AS pairs, 28526376 routes for route-views3 in 7m 31s 504ms.
[2024-04-02T15:20:31Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views3`.
[2024-04-02T15:20:32Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views3` in 502ms.
[2024-04-02T15:20:33Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views3` in 1s 300ms.
[2024-04-02T15:20:33Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.peru--rib.20230623.0000.bz2` for collector `route-views.peru`.
2024-04-02 15:20:33 [info] logging to syslog
Generated stats for 139 ASes, 138 AS pairs, 3896 routes for route-views.peru in 40ms.
[2024-04-02T15:20:33Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.peru`.
[2024-04-02T15:20:33Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.peru` in 1ms.
[2024-04-02T15:20:33Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.peru` in 1ms.
[2024-04-02T15:20:33Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.bdix--rib.20230623.0000.bz2` for collector `route-views.bdix`.
2024-04-02 15:20:33 [info] logging to syslog
Generated stats for 1144 ASes, 1321 AS pairs, 14130 routes for route-views.bdix in 70ms.
[2024-04-02T15:20:33Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.bdix`.
[2024-04-02T15:20:33Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.bdix` in 7ms.
[2024-04-02T15:20:33Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.bdix` in 9ms.
[2024-04-02T15:20:33Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views2--rib.20230623.0000.bz2` for collector `route-views2`.
2024-04-02 15:20:33 [info] logging to syslog
Generated stats for 75451 ASes, 177308 AS pairs, 26484140 routes for route-views2 in 5m 8s 76ms.
[2024-04-02T15:25:41Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views2`.
[2024-04-02T15:25:42Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views2` in 463ms.
[2024-04-02T15:25:43Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views2` in 1s 204ms.
[2024-04-02T15:25:43Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.linx--rib.20230623.0000.bz2` for collector `route-views.linx`.
2024-04-02 15:25:43 [info] logging to syslog
Generated stats for 81564 ASes, 206678 AS pairs, 38087968 routes for route-views.linx in 8m 55s 122ms.
[2024-04-02T15:34:38Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.linx`.
[2024-04-02T15:34:38Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.linx` in 499ms.
[2024-04-02T15:34:40Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.linx` in 1s 408ms.
[2024-04-02T15:34:40Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.flix--rib.20230623.0000.bz2` for collector `route-views.flix`.
2024-04-02 15:34:40 [info] logging to syslog
Generated stats for 81181 ASes, 161960 AS pairs, 10563393 routes for route-views.flix in 2m 18s 61ms.
[2024-04-02T15:36:58Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.flix`.
[2024-04-02T15:36:58Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.flix` in 497ms.
[2024-04-02T15:36:59Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.flix` in 1s 127ms.
[2024-04-02T15:36:59Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc10--bview.20230623.0000.gz` for collector `rrc10`.
2024-04-02 15:36:59 [info] logging to syslog
Generated stats for 81302 ASes, 179804 AS pairs, 17588584 routes for rrc10 in 3m 17s 595ms.
[2024-04-02T15:40:17Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc10`.
[2024-04-02T15:40:18Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc10` in 503ms.
[2024-04-02T15:40:19Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc10` in 1s 224ms.
[2024-04-02T15:40:19Z DEBUG route_verification_rib_stats] Starting to process RIB file `oix-route-views--oix-full-snapshot-2023-06-23-0000.bz2` for collector `oix-route-views`.
2024-04-02 15:40:19 [info] logging to syslog
Generated stats for 0 ASes, 0 AS pairs, 0 routes for oix-route-views in 21s 775ms.
[2024-04-02T15:40:41Z DEBUG route_verification_rib_stats] Wrote route stats for `oix-route-views`.
[2024-04-02T15:40:41Z DEBUG route_verification_rib_stats] Wrote AS stats for `oix-route-views` in 0ms.
[2024-04-02T15:40:41Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `oix-route-views` in 0ms.
[2024-04-02T15:40:41Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc00--bview.20230623.0000.gz` for collector `rrc00`.
2024-04-02 15:40:41 [info] logging to syslog
Generated stats for 81756 ASes, 233138 AS pairs, 54735676 routes for rrc00 in 11m 49s 658ms.
[2024-04-02T15:52:30Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc00`.
[2024-04-02T15:52:31Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc00` in 497ms.
[2024-04-02T15:52:32Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc00` in 1s 606ms.
[2024-04-02T15:52:32Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.chile--rib.20230623.0000.bz2` for collector `route-views.chile`.
2024-04-02 15:52:32 [info] logging to syslog
Generated stats for 74405 ASes, 111451 AS pairs, 937539 routes for route-views.chile in 8s 755ms.
[2024-04-02T15:52:41Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.chile`.
[2024-04-02T15:52:42Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.chile` in 437ms.
[2024-04-02T15:52:42Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.chile` in 720ms.
[2024-04-02T15:52:42Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc25--bview.20230623.0000.gz` for collector `rrc25`.
2024-04-02 15:52:42 [info] logging to syslog
Generated stats for 81718 ASes, 241339 AS pairs, 51806838 routes for rrc25 in 10m 53s 452ms.
[2024-04-02T16:03:36Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc25`.
[2024-04-02T16:03:36Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc25` in 521ms.
[2024-04-02T16:03:38Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc25` in 1s 704ms.
[2024-04-02T16:03:38Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.mwix--rib.20230623.0000.bz2` for collector `route-views.mwix`.
2024-04-02 16:03:38 [info] logging to syslog
Generated stats for 0 ASes, 0 AS pairs, 0 routes for route-views.mwix in 4ms.
[2024-04-02T16:03:38Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.mwix`.
[2024-04-02T16:03:38Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.mwix` in 0ms.
[2024-04-02T16:03:38Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.mwix` in 0ms.
[2024-04-02T16:03:38Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.eqix--rib.20230623.0000.bz2` for collector `route-views.eqix`.
2024-04-02 16:03:38 [info] logging to syslog
Generated stats for 81286 ASes, 175826 AS pairs, 19738344 routes for route-views.eqix in 4m 0s 555ms.
[2024-04-02T16:07:39Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.eqix`.
[2024-04-02T16:07:39Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.eqix` in 499ms.
[2024-04-02T16:07:40Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.eqix` in 1s 203ms.
[2024-04-02T16:07:40Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.kixp--rib.20230623.0000.bz2` for collector `route-views.kixp`.
2024-04-02 16:07:40 [info] logging to syslog
Generated stats for 80946 ASes, 136999 AS pairs, 3550271 routes for route-views.kixp in 32s 9ms.
[2024-04-02T16:08:12Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.kixp`.
[2024-04-02T16:08:13Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.kixp` in 484ms.
[2024-04-02T16:08:14Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.kixp` in 922ms.
[2024-04-02T16:08:14Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc07--bview.20230623.0000.gz` for collector `rrc07`.
2024-04-02 16:08:14 [info] logging to syslog
Generated stats for 81276 ASes, 162518 AS pairs, 9111652 routes for rrc07 in 2m 19s 177ms.
[2024-04-02T16:10:33Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc07`.
[2024-04-02T16:10:33Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc07` in 492ms.
[2024-04-02T16:10:34Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc07` in 1s 84ms.
[2024-04-02T16:10:34Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc23--bview.20230623.0000.gz` for collector `rrc23`.
2024-04-02 16:10:34 [info] logging to syslog
Generated stats for 81391 ASes, 174172 AS pairs, 11768802 routes for rrc23 in 2m 26s 675ms.
[2024-04-02T16:13:01Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc23`.
[2024-04-02T16:13:02Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc23` in 489ms.
[2024-04-02T16:13:03Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc23` in 1s 183ms.
[2024-04-02T16:13:03Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views2.saopaulo--rib.20230623.0000.bz2` for collector `route-views2.saopaulo`.
2024-04-02 16:13:03 [info] logging to syslog
Generated stats for 81224 ASes, 217626 AS pairs, 24431840 routes for route-views2.saopaulo in 4m 55s 773ms.
[2024-04-02T16:17:59Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views2.saopaulo`.
[2024-04-02T16:17:59Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views2.saopaulo` in 496ms.
[2024-04-02T16:18:01Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views2.saopaulo` in 1s 519ms.
[2024-04-02T16:18:01Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc03--bview.20230623.0000.gz` for collector `rrc03`.
2024-04-02 16:18:01 [info] logging to syslog
Generated stats for 81518 ASes, 234295 AS pairs, 41197877 routes for rrc03 in 9m 23s 681ms.
[2024-04-02T16:27:24Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc03`.
[2024-04-02T16:27:25Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc03` in 504ms.
[2024-04-02T16:27:26Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc03` in 1s 605ms.
[2024-04-02T16:27:26Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc24--bview.20230623.0000.gz` for collector `rrc24`.
2024-04-02 16:27:26 [info] logging to syslog
Generated stats for 81194 ASes, 170238 AS pairs, 11433481 routes for rrc24 in 2m 9s 683ms.
[2024-04-02T16:29:36Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc24`.
[2024-04-02T16:29:37Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc24` in 489ms.
[2024-04-02T16:29:38Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc24` in 1s 162ms.
[2024-04-02T16:29:38Z DEBUG route_verification_rib_stats] Starting to process RIB file `route-views.phoix--rib.20230623.0000.bz2` for collector `route-views.phoix`.
2024-04-02 16:29:38 [info] logging to syslog
Generated stats for 11 ASes, 9 AS pairs, 18 routes for route-views.phoix in 12ms.
[2024-04-02T16:29:38Z DEBUG route_verification_rib_stats] Wrote route stats for `route-views.phoix`.
[2024-04-02T16:29:38Z DEBUG route_verification_rib_stats] Wrote AS stats for `route-views.phoix` in 0ms.
[2024-04-02T16:29:38Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `route-views.phoix` in 0ms.
[2024-04-02T16:29:38Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc19--bview.20230623.0000.gz` for collector `rrc19`.
2024-04-02 16:29:38 [info] logging to syslog
Generated stats for 0 ASes, 0 AS pairs, 0 routes for rrc19 in 3ms.
[2024-04-02T16:29:38Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc19`.
[2024-04-02T16:29:38Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc19` in 0ms.
[2024-04-02T16:29:38Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc19` in 0ms.
[2024-04-02T16:29:38Z DEBUG route_verification_rib_stats] Starting to process RIB file `rrc13--bview.20230623.0000.gz` for collector `rrc13`.
2024-04-02 16:29:38 [info] logging to syslog
Generated stats for 81228 ASes, 160135 AS pairs, 6660218 routes for rrc13 in 1m 3s 356ms.
[2024-04-02T16:30:41Z DEBUG route_verification_rib_stats] Wrote route stats for `rrc13`.
[2024-04-02T16:30:42Z DEBUG route_verification_rib_stats] Wrote AS stats for `rrc13` in 495ms.
[2024-04-02T16:30:43Z DEBUG route_verification_rib_stats] Wrote AS pair stats for `rrc13` in 1s 124ms.
Successfully generated stats for 60 RIB files.
$                                       Steven-VM internet_route_verification/route_verification/rib_stats   main 2h55m55s

Not much different in the plot (note that this plot is only for our reference).

route_all_export_stats3.csv.gz
route_all_import_stats3.csv.gz
route_all_total_stats3.csv.gz
As per https://github.com/SichangHe/internet_route_verification/issues/88#issuecomment-1890291629 update. Little difference.

@cunha, actually, if we look at the 3rd graphs above (counting bottom up), we do see spec_import_customer dominating spec_import_from_neighbor for all RIBs. It must be that route-views2 contains a large portion of spec_import_from_neighbor.
Therefore, I think we can keep the phrasing in the text.

Somewhat good news, although it still seems like it's a small fraction compared to the behemoth that is uphill. =D

I just realized a few plots above are for the single RIB, not all. This might have been misleading.
However, the insights should be correct. So… do we need to plot those ones (route special case, etc.) for all RIBs, @cunha?

Consolidated with the numbers instead:
route special/whitelisted cases stats: https://github.com/SichangHe/internet_route_verification/issues/99#issuecomment-2085521192
route unrecorded cases stats: https://github.com/SichangHe/internet_route_verification/issues/99#issuecomment-2088334134


Add google analytics

 Issue by landonreed
Friday Nov 03, 2017 at 13:36 GMT
Originally opened as https://github.com/catalogueglobal/datatools-ui/issues/45

Google Analytics has great tools for assessing which parts of the website (down to button clicks) are being utilized, which would be helpful for making UI decisions.  It also has features to check for active users which would be helpful when deciding whether to deploy updates.

:tada: This issue has been resolved in version 4.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Clean up memory monitor

In this case when memory stats are not available (e.g. no gpu memory has been allocated), let's have the memory monitor return None and make any caller code handle missing memory stats. This will help prevent empty logging entries.
Originally posted by @ravi-mosaicml in https://github.com/mosaicml/composer/pull/387#discussion_r800957269

Fixed in #1071.


Add --clean for build command

It would be useful to add a clean parameter to the build command if I just need the current latest build target.

Looks good.


Add license

Please add Apache-2.0 license file

Done in https://github.com/PinoutLTD/rrs-odoo-addon/commit/bed870accfc8613387dcaed8bd706b35cb3a5e2c


[BUG] Android interface orientation value changes even with auto rotation off

Describe the bug
If the user disables the auto rotation feature, the interface stays in the current position it was before, that's correct, but the value it sends to js is off because it follows the device orientation.
To Reproduce
Steps to reproduce the behavior:

Checkout the example app
Run the app
Toggle auto rotation off
See the interface orientation value change following the device

Expected behavior
The interface orientation value should not follow the device one with auto rotation off
Smartphone (please complete the following information):

OS: Android
Device: Pixel 8 Emulator
Version: API 30

I've looked into iOS too and it seems that on iOS it behaves differently.
When the user locks the interface, then the orientationDidChange event isn't triggered, therefore we don't have to implement a logic to take care of it, it is done automatically by the system.
There isn't a way to detect if the user has locked the interface too


Backup: add firmware backup for mtdblock devices

This commit add the feature that allow user to backup the image of the flash.

@hnyman @jow-

How is that ipv6 related change related to the backup functionality that is the topic of this PR?

@hnyman  sorry, will drop it.

Hello!
First I’d like to say that this PR is really essential in luci:
It allows user to make a real backup of the system and to restore the whole device with all packages an config's.
A perfect long outstanding feature!
For me it works perfect, but maybe for new users the interface may be confusing.
It’s not clear how to restore the backups and how to deal with mtd-partitions.
If you want to restore u-boot or art partition it is very likely that you already have serial adapter and soldering iron lying around, which is not the everyday case.
I would suggest to remove other partitions than “firmware” or make it visible via “Expert Options”  -checkbox in the Configuration-section (if you lockup the bootloader  it can’t be restored within luci).
Further more it could be handy if the downloaded images have “Model” and “Hostname” in their filenames.
Maybe it could look like this:
(“Keep settings” is a little bit confusing when restoring a backup-firmware but did no harm in my tests.)

@rwalli  Having “Model” and “Hostname” in target file is a good suggestions.
In addition, I think we really need the backup options for u-boot and art partitions, particularly for the developers who work on them. Though these features is not everyday case but it is useful and convenient once we need. Moreover, this can meet the different needs of people who use OpenWrt and luci. So I would prefer to make it visible via “Expert Options” -checkbox in the Configuration-section.
Thank you for your suggestions, @rwalli ! :-)

The help texts should indicate more clearly that this is low-level technical functionality accessing flash partition directly, and is not meant for everyday config backup/restore. I am pretty sure that if the current version is merged, there will be people making config backups this way and then wondering how to restore them.
Should it be in a separate section??? (instead of backup)
Maybe after backup and restore.
Terminology may be a bit confusing, as both this new text and the firmware flash text talk about "image". So, people may try using this mtdblock image for firmware flash...

@hnyman @rwalli  Hi,  how about this ?

@hnyman This is the latest modification. I added a caution sentence saying that "this feature is for professionals!" at the description. Feel free to correct me if you think it is not properly. Thanks.

@hnyman @rwalli  I did not upload the code of the last screenshot, because I was not sure it seems OK.
Now I update my commit, and also fixed the issue (@hnyman) you mentioned. It won't miss the last 1024 bytes any longer. :-)

Thanks. Looks good enough for me.
Seemed to work ok with my wrt3200acm, so let's merge it.


Implement Undo, Redo

This PR features complete Undo, Redo functionality. This PR is compatible with newest commit without merge issues. Please be aware that there still might be some bugs out there that sild in unnoticed.
Closes #1975, #2250

The PR is Ready to be reviewed

Looks like I'll be busy...

crash when (this is difficult XD):

copypaste a file to its same folder
rename new file to something else
ctrlz
ctrlz
ctrly

Are you trying to cast spells with this combination 😆

Here are all the changes requested:

Crash deleting from recycle bin
Crash on drag and drop after hovering on a folder (and so navigating to it)
Cltr-x copies instead of cutting
Cannot copy paste shortcut (.lnk) files
Crash pasting file to windows folder
Does not copy files to usb connected android phone anymore
Witchcraft combination
Remame Status to something more specific
Use FilesystemTasks.Wrap() on ToStorageItem()

I'll report progress on them once they're completed

crash pasting file to windows folder
@gave92 Hmmm... It doesn't seem to crash for me

@d2dyno1 I'll check again 👍

Thanks for your work on this, here some updates on the issues:

crash pasting file to windows folder
-> I can confirm this, pasting to "C:\Program Files (x86)" does the same
"magic combo"
copypaste a file to its same folder
rename new file to something else
ctrlz
ctrlz
ctrly
-> no crash anymore but the "ctrly" does nothing (should redo copy-paste)
quickly pressing ctrlz + ctrly often results in a crash
copypaste file
quickly press ctrlz and ctrly
2020-11-17 19:29:54.0065|ERROR|Files.App|Object reference not set to an instance of an object.|System.NullReferenceException: Object reference not set to an instance of an object.
at Files.Filesystem.FilesystemOperations.DeleteAsync(IStorageItem source, IProgress1 progress, IProgress1 errorCode, Boolean permanently, CancellationToken cancellationToken)
at Files.Filesystem.FilesystemHistory.StorageHistoryOperations.Redo(IStorageHistory history)
at Files.Filesystem.FilesystemHistory.StorageHistoryHelpers.Redo()
at Files.Views.Pages.ModernShellPage.KeyboardAccelerator_Invoked(KeyboardAccelerator sender, KeyboardAcceleratorInvokedEventArgs args)
at System.Threading.WinRTSynchronizationContextBase.Invoker.InvokeCore()

Ok, I got around my issue, but it'd be way cleaner if IStorageItemWithPath implemented IStorageItem

@d2dyno1 glad you solved it. But making IStorageItemWithPath implement IStorageItem is something that we can totally consider.

Fixed Spamming Ctrl+Z, Ctrl+Y with a little help from @gave92 😉 🙌

Hi @d2dyno1, I can confirm that the issue "quickly pressing ctrlz + ctrly often results in a crash" is fixed.
Also I'm not getting a crash anymore when "pasting file to windows folder"
But:

copy pasting a folder to the same location causes issues: folder is not copied (should be duplicated with a "(2)" at the end) and ctrl-z deletes the origin folder
copy pasting a folder "inside itself" and pressing ctrl-z causes a crash

Here are all the changes requested:

[x]  Crash deleting from recycle bin
[x]  Dropping dragged items to a folder causes a crash (and so navigating to it) ******
[x]  CTRL+X copies instead of cutting
[x]  Cannot copy paste shortcut (.lnk) files
[x]  Crash pasting items to /Windows (and /Program Files (x86)) directories
[x]  Cannot delete files from usb connected android phone
[x]  Witchcraft combination
[x]  Remame Status to something more specific
[x]  Use FilesystemTasks.Wrap() on ToStorageItem()
[x]  Quickly pressing ctrlz + ctrly often results in a crash
[x]  Copy pasting a folder to the same location does not duplicate the folder
[x]  Copy pasting a folder "inside itself" and pressing ctrl-z causes a crash

I'll report progress on them once they're completed
***** - TBD

@gave92 looks like everything is done :eyes:

This source.Path == Path.GetDirectoryName(destination) only works if you copy the folder "immediately" inside itself. If you copy the folder at least "two levels down" you'll still have issues (untested)

Good catch!

Now it should be ok

Nope sorry I still get an exception when I press ctrl-z (after pasting a folder inside itself)
2020-11-20 20:55:15.1122|ERROR|Files.App|Object reference not set to an instance of an object.|System.NullReferenceException: Object reference not set to an instance of an object.
at Files.Helpers.RecycleBinHelpers.<>c__DisplayClass5_0.b__0(ShellFileItem shellItem)
at System.Linq.Enumerable.Any[TSource](IEnumerable1 source, Func2 predicate)
at Files.Helpers.RecycleBinHelpers.IsRecycleBinItem(IStorageItem item)
at Files.Filesystem.FilesystemHelpers.DeleteItemsAsync(IEnumerable`1 source, Boolean showDialog, Boolean permanently, Boolean registerHistory)
at Files.Filesystem.FilesystemHistory.StorageHistoryOperations.Undo(IStorageHistory history)
at Files.Filesystem.FilesystemHistory.StorageHistoryHelpers.Undo()
at Files.Views.Pages.ModernShellPage.KeyboardAccelerator_Invoked(KeyboardAccelerator sender, KeyboardAcceleratorInvokedEventArgs args)
at System.Threading.WinRTSynchronizationContextBase.Invoker.InvokeCore()

Oh, probably because it returned invalid value, gonna fix it, gimme a momento

More issues found :)
Pressing ctrl-z after copying a hosrtcut (.lnk) couses this:
2020-11-20 21:33:44.8706|ERROR|Files.App|Object reference not set to an instance of an object.|System.NullReferenceException: Object reference not set to an instance of an object.
at Files.Filesystem.FilesystemHelpers.GetItemSize(IStorageItem item)
at Files.Filesystem.FilesystemOperations.CopyAsync(IStorageItem source, String destination, IProgress1 progress, IProgress1 errorCode, CancellationToken cancellationToken)
at Files.Filesystem.FilesystemOperations.MoveAsync(IStorageItem source, String destination, IProgress1 progress, IProgress1 errorCode, CancellationToken cancellationToken)
at Files.Filesystem.FilesystemHelpers.MoveItemAsync(IStorageItem source, String destination, Boolean registerHistory)
at Files.Filesystem.FilesystemHistory.StorageHistoryOperations.Undo(IStorageHistory history)
at Files.Filesystem.FilesystemHistory.StorageHistoryHelpers.Undo()
at Files.Views.Pages.ModernShellPage.KeyboardAccelerator_Invoked(KeyboardAccelerator sender, KeyboardAcceleratorInvokedEventArgs args)
at System.Threading.WinRTSynchronizationContextBase.Invoker.InvokeCore()

Shortcut items now work with Undo, Redo but... (read description)
Untested, I ran out of time today

Got one more commit for today 😌

@d2dyno1 undo/redo seems to work ok for shortcuts 👍 except for:

crash on sending a shortcut item to recycle bin
2020-11-21 17:09:45.7188|ERROR|Files.App|Unsupported data type. (Exception from HRESULT: 0x8007065E)|System.Exception: Unsupported data type. (Exception from HRESULT: 0x8007065E)
at System.Runtime.InteropServices.WindowsRuntime.IMap2.Insert(K key, V value)  at System.Runtime.InteropServices.WindowsRuntime.MapToDictionaryAdapter.Insert[K,V](IMap2 _this, K key, V value)
at System.Runtime.InteropServices.WindowsRuntime.MapToDictionaryAdapter.Add[K,V](K key, V value)
at Files.Filesystem.FilesystemOperations.DeleteAsync(PathWithType source, IProgress1 progress, IProgress1 errorCode, Boolean permanently, CancellationToken cancellationToken)
at Files.Filesystem.FilesystemHelpers.DeleteItemsAsync(IEnumerable`1 source, Boolean showDialog, Boolean permanently, Boolean registerHistory)
at Files.Views.Pages.ModernShellPage.KeyboardAccelerator_Invoked(KeyboardAccelerator sender, KeyboardAcceleratorInvokedEventArgs args)
at System.Threading.WinRTSynchronizationContextBase.Invoker.InvokeCore()

Hmm... the exception is thrown here:

That's not the case with original code, but the logic used is the same 🤔

:eyes:

I hope that's all, and this commit can be finally fully reviewed @gave92  @yaichenbaum  @tsvietOK

Bug steps:

Delete file to Recycle bin
Clear Recycle bin
Press Ctrl+Z.
Press Ctrl+Y.
Crash

I'm aware it might have not fixed all issues

Should be alright now, @tsvietOK

50th comment!

:+1:

@d2dyno1
Steps:

Create bitmap with any name.
Press F5.
Try to rename this file

For me it shows:

@d2dyno1 By the way, when renaming item several times and trying doing undo several times, it reverts only last change.

@d2dyno1 Steps:

Create folder and place file in this folder.
Remove this folder to recycle bin.
Press Ctrl+Z.
It restore only folder without file inside.

@tsvietOK should be working now

I'd like to thank everyone who helped me with this PR. Without you, this wouldn't be as it is now!
Thanks,
@gave92 @tsvietOK @yaichenbaum @duke7553


Reenable aggregate TestControllerClientSync

@xiaolanz: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/istio-unit-tests.sh
e947bfb4dfc082aa9cb979e25f74b6f1243ceb3c
link
/test istio-unit-tests

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

it was merged already done.


Proposal for a link update and project description added

Hello
Here a proposal: only linking to the 2BitWizard flashable camera mod from Photo! as the licence is more friendly than the other mod.  I've made a fork from it to ease the project making as it is was bit "dry" as it.
I've added a short project description.
Also the current release dates from last year, maybe it could be good to make a new official release.
Up to you to accept or not, it is a friendly proposal as usual
Raphaël

Thank you, that what i wanted to do but did not have time yet.