Update to new jsonschema referencing library

This change-set adds compatibility with recent versions of the jsonschema library.
Changes:

Use new 'Registry' and 'Resource' classes.
Remove file-path patching when converting URIs.
Patch schema URLs to ensure DRAFT201909 and DRAFT202012 are correctly recognized.

CC @clenk @ejratl @rpiazza

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Codecov Report
Attention: Patch coverage is 90.62500% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 94.44%. Comparing base (6d3aeab) to head (ebd4034).

Files
Patch %
Lines

stix2validator/validator.py
90.32%
3 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #228      +/-   ##
==========================================
- Coverage   94.45%   94.44%   -0.02%
==========================================
  Files          66       66
  Lines        6173     6176       +3
==========================================
+ Hits         5831     5833       +2
- Misses        342      343       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

I have added a different PR https://github.com/oasis-open/cti-stix-validator/pull/231 where tests are run also on Windows (just one Python version) so we can be relatively sure that the URI-filepath manipulation works as expected.

Thank you very much! It indeed makes sense.

I like that the tests are passing, thank you @ostefano. However, I'm getting an Unresolvable: http://raw.githubusercontent.com/<...> error when I try to validate using a local custom schema. For example: stix2_validator -s ~/Downloads/test_schemas ~/Downloads/test/tlp-amber.json. Did you test this use case by chance?

Oh snap @clenk  I was under the assumptions that tests were enough to validate the changeset.
I will start testing this use case right away, and open a separate PR.

@clenk can you share the tlp-amber.json file and the content of the test_schemas directory?

There are a few unit tests that use some custom schemas (https://github.com/oasis-open/cti-stix-validator/tree/master/stix2validator/test/v21/test_schemas), for example: https://github.com/oasis-open/cti-stix-validator/blob/2330398fa244228bad748d58b295334eb1d9987f/stix2validator/test/v21/observed_data_tests.py#L774 or https://github.com/oasis-open/cti-stix-validator/blob/2330398fa244228bad748d58b295334eb1d9987f/stix2validator/test/v21/indicator_tests.py#L207. However, I'm realizing now none of the test schemas include $id or $ref.

test_schemas/extension-definition--60a3c5c5-0d10-413e-aab3-9e08dde9e88d.json:
{
  "$id": "tlp2.0.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "tlp2.0-marking-definition-extension",
  "description": "This marking extension was created to apply TLP2.0 data markings",
  "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "description": "The type of this object, which MUST be the literal `marking-definition`.",
          "enum": [
            "marking-definition"
          ]
        },
        "spec_version": {
          "type": "string",
          "enum": [
            "2.1"
          ],
          "description": "The version of the STIX specification used to represent this object."
        },
        "created_by_ref": {
          "$ref": "http://raw.githubusercontent.com/oasis-open/cti-stix2-json-schemas/stix2.1/schemas/common/identifier.json",
          "description": "The created_by_ref property specifies the ID of the identity object that describes the entity that created this Marking Definition."
        },
        "created": {
          "$ref": "http://raw.githubusercontent.com/oasis-open/cti-stix2-json-schemas/stix2.1/schemas/common/timestamp.json",
          "description": "The created property represents the time at which the first version of this Marking Definition object was created."
        },
        "external_references": {
          "type": "array",
          "description": "A list of external references which refers to non-STIX information.",
          "items": {
            "$ref": "http://raw.githubusercontent.com/oasis-open/cti-stix2-json-schemas/stix2.1/schemas/common/external-reference.json"
          },
          "minItems": 1
        },
        "object_marking_refs": {
          "type": "array",
          "description": "The object_marking_refs property specifies a list of IDs of marking-definition objects that apply to this Marking Definition.",
          "items": {
            "allOf": [
              {
                "$ref": "http://raw.githubusercontent.com/oasis-open/cti-stix2-json-schemas/stix2.1/schemas/common/identifier.json"
              },
              {
                "pattern": "^marking-definition--"
              }
            ]
          },
          "minItems": 1
        },
        "granular_markings": {
          "type": "array",
          "description": "The granular_markings property specifies a list of granular markings applied to this object.",
          "items": {
            "$ref": "http://raw.githubusercontent.com/oasis-open/cti-stix2-json-schemas/stix2.1/schemas/common/granular-marking.json"
          },
          "minItems": 1
        }
      },
      "oneOf": [
        {
          "$ref": "#/definitions/tlp_clear"
        },
        {
          "$ref": "#/definitions/tlp_green"
        },
        {
          "$ref": "#/definitions/tlp_amber"
        },
        {
          "$ref": "#/definitions/tlp_amber_strict"
        },
        {
          "$ref": "#/definitions/tlp_red"
        }
      ],
      "required": [
          "id",
          "type",
	  "name",
          "spec_version",
          "created",
	  "extensions"
      ],
  "definitions": {
    "tlp_clear": {
      "description": "The marking-definition object representing Traffic Light Protocol (TLP) Clear.",
      "properties": {
        "id": {
          "type": "string",
          "enum": [
            "marking-definition--94868c89-83c2-464b-929b-a1a8aa3c8487"
          ]
        },
        "name": {
          "type": "string",
          "enum": [
            "TLP:CLEAR"
          ]
        },
        "extensions": {
          "type": "object",
          "properties": {
            "extension-definition--60a3c5c5-0d10-413e-aab3-9e08dde9e88d": {
              "type": "object",
              "properties": {
                "extension_type": {
                  "type": "string",
                  "description": "Defined by STIX 2.1 extension definition rules from 'extension-type-enum'.",
                  "enum": [
                    "property-extension"
                  ]
                },
                "tlp_2_0": {
                  "type": "string",
                  "enum": [
                    "clear"
                  ]
                }
              },
              "required": [
                "extension_type",
                "tlp_2_0"
              ],
              "additionalProperties": false
            }
          },
	  "required": [
	     "extension-definition--60a3c5c5-0d10-413e-aab3-9e08dde9e88d"
	  ],
          "additionalProperties": false
        }
      }
    },
    "tlp_green": {
      "description": "The marking-definition object representing Traffic Light Protocol (TLP) Green.",
      "properties": {
        "id": {
          "type": "string",
          "enum": [
            "marking-definition--bab4a63c-aed9-4cf5-a766-dfca5abac2bb"
          ]
        },
        "name": {
          "type": "string",
          "enum": [
            "TLP:GREEN"
          ]
        },
        "extensions": {
          "type": "object",
          "properties": {
            "extension-definition--60a3c5c5-0d10-413e-aab3-9e08dde9e88d": {
              "type": "object",
              "properties": {
                "extension_type": {
                  "type": "string",
                  "description": "Defined by STIX 2.1 extension definition rules from 'extension-type-enum'.",
                  "enum": [
                    "property-extension"
                  ]
                },
                "tlp_2_0": {
                  "type": "string",
                  "enum": [
                    "green"
                  ]
                }
              },
              "required": [
                "extension_type",
                "tlp_2_0"
              ],
              "additionalProperties": false
            }
          },
          "additionalProperties": false
        }
      }
    },
    "tlp_amber": {
      "description": "The marking-definition object representing Traffic Light Protocol (TLP) Amber.",
      "properties": {
        "id": {
          "type": "string",
          "enum": [
            "marking-definition--55d920b0-5e8b-4f79-9ee9-91f868d9b421"
          ]
        },
        "name": {
          "type": "string",
          "enum": [
            "TLP:AMBER"
          ]
        },
        "extensions": {
          "type": "object",
          "properties": {
            "extension-definition--60a3c5c5-0d10-413e-aab3-9e08dde9e88d": {
              "type": "object",
              "properties": {
                "extension_type": {
                  "type": "string",
                  "description": "Defined by STIX 2.1 extension definition rules from 'extension-type-enum'.",
                  "enum": [
                    "property-extension"
                  ]
                },
                "tlp_2_0": {
                  "type": "string",
                  "enum": [
                    "amber"
                  ]
                }
              },
              "required": [
                "extension_type",
                "tlp_2_0"
              ],
              "additionalProperties": false
            }
          },
          "additionalProperties": false
        }
      }
    },
    "tlp_amber_strict": {
      "description": "The marking-definition object representing Traffic Light Protocol (TLP) Amber+Strict.",
      "properties": {
        "id": {
          "type": "string",
          "enum": [
            "marking-definition--939a9414-2ddd-4d32-a0cd-375ea402b003"
          ]
        },
        "name": {
          "type": "string",
          "enum": [
            "TLP:AMBER+STRICT"
          ]
        },
        "extensions": {
          "type": "object",
          "properties": {
            "extension-definition--60a3c5c5-0d10-413e-aab3-9e08dde9e88d": {
              "type": "object",
              "properties": {
                "extension_type": {
                  "type": "string",
                  "description": "Defined by STIX 2.1 extension definition rules from 'extension-type-enum'.",
                  "enum": [
                    "property-extension"
                  ]
                },
                "tlp_2_0": {
                  "type": "string",
                  "enum": [
                    "amber+strict"
                  ]
                }
              },
              "required": [
                "extension_type",
                "tlp_2_0"
              ],
              "additionalProperties": false
            }
          },
          "additionalProperties": false
        }
      }
    },
    "tlp_red": {
      "description": "The marking-definition object representing Traffic Light Protocol (TLP) Red.",
      "properties": {
        "id": {
          "type": "string",
          "enum": [
            "marking-definition--e828b379-4e03-4974-9ac4-e53a884c97c1"
          ]
        },
        "name": {
          "type": "string",
          "enum": [
            "TLP:RED"
          ]
        },
        "extensions": {
          "type": "object",
          "properties": {
            "extension-definition--60a3c5c5-0d10-413e-aab3-9e08dde9e88d": {
              "type": "object",
              "properties": {
                "extension_type": {
                  "type": "string",
                  "description": "Defined by STIX 2.1 extension definition rules from 'extension-type-enum'.",
                  "enum": [
                    "property-extension"
                  ]
                },
                "tlp_2_0": {
                  "type": "string",
                  "enum": [
                    "red"
                  ]
                }
              },
              "required": [
                "extension_type",
                "tlp_2_0"
              ],
              "additionalProperties": false
            }
          },
          "additionalProperties": false
        }
      }
    }
  }
}

test/tlp-amber.json (typo in tlp_2_0 value is intentional, to verify validator catches it):
{
    "type": "bundle",
    "id": "bundle--63ab8e67-acac-4817-845a-d09f0e86954c",
    "objects": [
        {
            "type": "marking-definition",
            "spec_version": "2.1",
            "id": "marking-definition--55d920b0-5e8b-4f79-9ee9-91f868d9b421",
            "created": "2022-10-01T00:00:00.000Z",
            "name": "TLP:AMBER",
            "extensions": {
                "extension-definition--60a3c5c5-0d10-413e-aab3-9e08dde9e88d": {
                    "extension_type": "property-extension",
                    "tlp_2_0" : "amberx"
                }
            }
        }
    ]
}

@clenk so test/tlp-amber.json with the typo fixed, together with the custom schemas directory, should reproduce the issue?

@clenk fixed it.
I created a new issue https://github.com/oasis-open/cti-stix-validator/issues/234 and will continue the conversation there.
Will open a PR and link it there when ready.


The functions in sha.go are all using the deprecated low level API calls in openssl. These do not go via the fips provider and so are not FIPS compliant. See: https://github.com/golang/go/blob/25aa45afcf40ff6c699b47bad0cb609bba84da12/src/crypto/internal/boring/sha.go#L16 for an example. Instead of doing
SHA1_Init
SHA1_Update
SHA1_Final

We should be doing something like:
    const EVP_MD *md = EVP_get_digestbyname("sha256");
    if (md == NULL) {
        printf("Unknown message digest %s\n", "sha256");
        exit(1);
    }
    EVP_MD_CTX *ctx = EVP_MD_CTX_new();
        if (!EVP_DigestInit_ex(ctx, md, NULL)) {
        printf("Message digest initialization failed.\n");
	ERR_print_errors_fp (stderr);
        EVP_MD_CTX_free(ctx);
        exit(1);
    }
    EVP_DigestUpdate(ctx, data, strlen(data));
    EVP_DigestFinal_ex(ctx, md_value, &md_len);

This uses the high level EVP_ API and will be FIPS compliant. On the page (https://www.openssl.org/docs/manmaster/man7/fips_module.html):
Applications written to use the OpenSSL 3.0 FIPS module should not use any legacy APIs or features that avoid the FIPS module. Specifically this includes:

Low level cryptographic APIs (use the high level APIs, such as EVP, instead)

Also see https://github.com/openssl/openssl/blob/a4cbffcd8998180b98bb9f7ce6065ed37d079d8b/doc/man7/ossl-guide-migration.pod#L1105 where openssl say not to use functions such as SHA1_Init etc and instead use EVP. Apparently there are some convinience functions as well ..or the quick one-shot L<EVP_Q_digest(3)>
Note this bug is a follow-on from https://github.com/golang/go/issues/66513 where I have removed the references about md5 that muddied the waters in the original bug report.
What did you see happen?
All covered in initial section
What did you expect to see?
All covered in initial section

CC @golang/security

These do not go via the fips provider and so are not FIPS compliant.

Applications written to use the OpenSSL 3.0 FIPS module

Go+BoringCrypto doesn't use the OpenSSL 3.0 FIPS module. Is this the case for BoringCrypto as well?

I forgot to mention I am using the go-toolset on Redhat, but the boringcrypto code that calls into openssl is the same, and if openssl is running in FIPS mode (Redhat fips-mode-setup enable so kernel has fips=1 and openssl is setup to use the fips module) I think that means that openssl C API calls in boringcrypto should aim to be using the high level openssl API calls so that they go via the openssl fips provider.

Also when I ran my go code that did:
		result2 := sha1.Sum([]byte("hash me"))
		fmt.Printf("%x\n", result2)

like this: LD_DEBUG=symbols /tmp/test fips-self-test
I got:
...
   3880446:	calling init: /usr/lib64/ossl-modules/fips.so
   3880446:
   3880446:	symbol=OSSL_provider_init;  lookup in file=/usr/lib64/ossl-modules/fips.so [0]
   3880446:	symbol=pthread_mutex_init;  lookup in file=/tmp/test [0]
   3880446:	symbol=pthread_mutex_init;  lookup in file=/lib64/libresolv.so.2 [0]
   3880446:	symbol=pthread_mutex_init;  lookup in file=/lib64/libc.so.6 [0]
   3880446:	symbol=SHA1_Init;  lookup in file=/tmp/test [0]
   3880446:	symbol=SHA1_Init;  lookup in file=/lib64/libresolv.so.2 [0]
   3880446:	symbol=SHA1_Init;  lookup in file=/lib64/libc.so.6 [0]
   3880446:	symbol=SHA1_Init;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
   3880446:	symbol=SHA1_Init;  lookup in file=/lib64/libcrypto.so.3 [0]
   3880446:	symbol=SHA1_Update;  lookup in file=/tmp/s3store [0]
   3880446:	symbol=SHA1_Update;  lookup in file=/lib64/libresolv.so.2 [0]
   3880446:	symbol=SHA1_Update;  lookup in file=/lib64/libc.so.6 [0]
   3880446:	symbol=SHA1_Update;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
   3880446:	symbol=SHA1_Update;  lookup in file=/lib64/libcrypto.so.3 [0]
   3880446:	symbol=SHA1_Final;  lookup in file=/tmp/test [0]
   3880446:	symbol=SHA1_Final;  lookup in file=/lib64/libresolv.so.2 [0]
   3880446:	symbol=SHA1_Final;  lookup in file=/lib64/libc.so.6 [0]
   3880446:	symbol=SHA1_Final;  lookup in file=/lib64/ld-linux-x86-64.so.2 [0]
   3880446:	symbol=SHA1_Final;  lookup in file=/lib64/libcrypto.so.3 [0]

Which shows it is using the FIPS modile in openssl

Ah, you are using Red Hat's fork that uses OpenSSL's module. I'm afraid we don't have the bandwidth upstream to handle issues of the downstream forks.

@FiloSottile OK np, so the boringcrypto code in this repo does not call into openssl the same way Redhat's does?
Looking at the code it seemed like an issue in the base boringcrypto so I was trying to cut out the middle main and get straight to the source as the code in this repo for the openssl C API calls for SHA are the same as in the Redhat fork.

Thanks for the explanation. I will do a bit more digging using standard go and GOEXPERIMENT=boringcrypto at compile time and get back to you, probably tomorrow. Are there any docs you can point me at for how normal boringcrypto mode works please? I'm not sure I fully undersand the difference between standard boringcrypto and Redhat's version.

boringssl only defines SHA1_Init under crypto/fipsmodule/sha
https://github.com/google/boringssl/blob/70b33d39048abaa1c810ad63ace4b05af7b94d15/crypto/fipsmodule/sha/sha1.c#L69 so there's no non fips implementation?

Aaaah. My apologies and thank you both so much for your time. I see now where the standard boringcrypto/ssl and Redhat's version diverge. I was thrown as the function name SHA1_Init is identical to openssl and I hadn't dug into borringssl's code. Closing this as it is invalid and I will raise it on Redhat.


broken links in doc

Solved Broken Links in doc.

I have covered almost all broken links in this PR.


same network discovery

What happens if both peers are on the same network, i.e. behind the same router/gateway/nat?
Would it be possible to figure this out by e.g. adding something unique about the network (gateway mac address? gateway public ip?) to the dns record and resolving to the all LAN IP addresses on that network so that peers could try to discover each other in a LAN?
Another option would be SSDP but then you would have to patch into a systems SSDP server like avahi.

What happens if both peers are on the same network, i.e. behind the same router/gateway/nat?

This depends on the behavior of the gateway device. Some of the gateways I've played with will not forward traffic coming in on the "internal" interface with a destination address of the "external" interface that translates back to the "internal" interface.

Would it be possible to figure this out by e.g. adding something unique about the network (gateway mac address? gateway public ip?) to the dns record and resolving to the all LAN IP addresses on that network so that peers could try to discover each other in a LAN?

I think that the WebRTC approach defined in ICE RFC5245 gives us some ideas. ICE gathers what it calls "candidates" (IP address + port) and attempts to establish connectivity over all candidates. There are different types of candidates, one of which could be learned via STUN (similar to what we do in this project), another could be a local interface. In this case of two peers behind the same layer 3 gateway, ICE will ultimately have them choose the local interface candidate, assuming it is prioritized over the others.
We can't exactly implement ICE since we exist outside of WireGuard, but we can borrow some of its concepts. Candidates are SRV+A/AAAA record pairs. Right now we just have one candidate per peer, which is discovered from the tunnel with the Registry. SRV records can have priorities just as ICE candidates do. We just need a way to "push" host candidates to wgsd. This could be the responsibility of the client. I haven't looked much into the various methods available for pushing DNS updates, but I imagine there is a standard for this. So long as wgsd could verify the push came from peer that the update is intended to modify SRV records for, this could work.

Any updates about "candidates" implementation?


replace deployment reconciler with webhook

Which problem is this PR solving?

this pr prevents containers with valid annotation from being created without a sidecar and getting immediately replaced. This was caused by the reconciler determining that a sidecar was needed after the pod was already launched.
Belongs to https://github.com/jaegertracing/jaeger-operator/pull/1804

Short description of the changes

moved deployment reconciler logic into webhook

cc @rubenvp8510

improved flaky unittest https://github.com/jaegertracing/jaeger-operator/pull/1828/commits/32eeb2afe9fe6af827c96f8f6ed8e4f337fbd0e6 and rebased branch onto main.
cc @rubenvp8510


[Question] Is there a way to dynamically set the plugin configuration ?

@mderevyankoaqa
I am looking for a way to be able to dynamically change the configuration passed to the plugin (ex: hostname, token etc) via Host Environment variable or Jmeter Property notation (${__P(propname, default)}.  Does the plugin already support either of these options?
Thanks!

Hi @leelaprasadv should be supported without any issues.
BR,
Mike

Hi @mderevyankoaqa I was trying to parameterize run id with a jmeter environment value. But could not make it work. Every time the run id is getting set as the defaul01 rather than the actual env value.

I made sure the run_id has a value by appending the env value to the request name and the run id is populating correctly

Is there any limitation to the backend plugin that we cant use the jmeter environment value to parameterise the details?


"Cenei" and "Bobda" are listed in the drop-down menu

Description:
"Cenei" and "Bobda" are listed in the drop-down menu alongside its corresponding county and commune, according to the law
Precondition:
The website is up an running.

Step 1
Type "Cenei" in the search bar.

Expected results
The location is listed in the drop-down menu as "Sat Cenei, TIMIȘ (Cenei)" and "Comuna Cenei, TIMIȘ".

Step 2
Press "x" button.

Expected results
The location was deleted from the search bar.

Step 3
Type "Bobda" in the search bar.

Expected results
The location is listed in the drop-down menu as "Sat Bobda, TIMIȘ (Cenei)".


ci: auto format with prettier

Implementing Prettier to

work on pull request
format the code

Closes https://github.com/stamford-syntax-club/stamfordcenter-backend/issues/77


Pict w/ non origin bounds, masks, various

When a PICT's bounding box is not at the origin, e.g. (18, 15, 100, 263) instead of (0, 0, 82, 248), the DirectCopyBits doesn't draw the right and bottom edges. This is because write_canvas_pixel compares a PICT-local coordinate (the pixel's position) with a GrafPort-local coordinate (the GrafPort's bounds). Instead it should transform the pixel's position into GrafPort's coordinate space.
Attached file Bounds.zip contains a .sit archive (Github doesn't accept .sit, alas) with a file with a resource fork that contains a single PICT resource with non-origin bounds.
Also, when DirectCopyBits uses a mask, a pixel is only drawn when not in the mask, instead of when in the mask (or at least switching the condition makes a DirectCopyBits mask consisting of merely the bounding rectangle work).
Plus a few small fixes and notes, including from my previous pull request (which I closed):

skip_var32_read only skips 2 bytes like skip_var16_read, not 4 bytes as it probably should.
The single-argument constructor of ResourceFile isn't explicit to prevent (probably unintended) implicit conversions.
Pedantic stuff 😉

Thanks for fixing these! I'm a little surprised I never noticed pict_skip_var32 was incorrect for so long.


Upgrade to duckdb 1.1.3

Hello! I am hacking on a project using duckdb rust and noticed that the duckdb version was slightly out of date.
This came to my attention because it seems like a community plugin I am trying to use does not have updated binaries for duckdb 1.1.1 (the bigquery extension, to be specific).
Also, the upgrade script I think was missing crates/duckdb/Cargo.toml, so I added that in.

I'm a little confused about the intent of the failing test.
The panic says "expected error" but I'm wondering if thats a typo and its actually an unexpected error?

I've dug into the error a bit more. It seems like in this version of duckdb, the appender no longer errors when writing an incomplete row.
Instead, the append is ignored, and nothing seems to be written to the table (i.e. no rows are returned when querying the table).
If I can get the crate building against a local checkout of duckdb, I'll see if I can bisect when the regression was introduced.


Fixed interpolateData function

Utils:
-updated interpolateData function to remove x-val equality check which can cause issues with log-spaced x-val data with small magnitudes
Resolves #40

Pull Request Test Coverage Report for Build 61

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.002%) to 99.174%

Totals

Change from base Build 59:
-0.002%

Covered Lines:
840

Relevant Lines:
847

💛  - Coveralls


chore: Refactor logpipeline and logparser validators

Description
Changes proposed in this pull request (what was done and why):

Move logpipeline and logparser validators out of the API packages and into the webhook/validation packages
Refactor interdependent methods

Changes refer to particular issues, PRs or documents:

Prerequisite for: https://github.com/kyma-project/telemetry-manager/issues/872

Traceability

[x] The PR is linked to a GitHub issue.
[ ] The follow-up issues (if any) are linked in the Related Issues section.
[ ] If the change is user-facing, the documentation has been adjusted.
[ ] If a CRD is changed, the corresponding Busola ConfigMap has been adjusted.
[x] The feature is unit-tested.
[ ] The feature is e2e-tested.

TODO: Discuss in daily where to move the Mode type, as it seems we need it in multiple places, and still want to keep the stringer functionality


Asset.get_path frontend changes

Updates the file browser component to make use of the additional data returned from the Asset.get_path endpoint introduced in https://github.com/dandi/dandi-api/pull/312.
These changes eliminate the slowdown that occurs in the file browser mentioned in #600.
Blocked until https://github.com/dandi/dandi-api/pull/312 is merged, so I'll mark this as draft for now.

Given the fix https://github.com/dandi/dandi-api/pull/387/files#diff-2f7f82b6ec9c0331ea6b7f270afb58015baf10237df23dbf41fc5597f5001f75R27 , shouldn't this PR base against staging?


Bad uuidv4 validator

Bad uuidv4 validation for uuid like a d7b5d9fb-57f6-11e7-8328-bc5ff4b8e882.
Tested in http://www.freecodeformat.com/validate-uuid-guid.php and my example uuid is valid.
Looks like bad regex in next line https://github.com/asaskevich/govalidator/blob/73945b6115bfbbcc57d89b7316e28109364124e1/patterns.go#L12.
Tested in https://regex101.com and my example uuid with your regexp is not valid.

I guess bad regexp in next part 4[0-9a-f]{3}

My bad, maybe i used to bad uuid generator.

My bad, maybe i used to bad uuid generator.


No module 'cms.middleware.utils.ApphookReloadMiddleware'

According to the documentation of DjangoCMS 3.2.0 I should add cms.middleware.utils.ApphookReloadMiddleware to MIDDLEWARE_CLASSES in my settings.py file in order to make the server auto reload whenever there's a changed to my AppHooks.
However, when I do this
MIDDLEWARE_CLASSES = (
    ...
    'cms.middleware.utils.ApphookReloadMiddleware',
    ...

I get an error ImportError: No module named utils in terminal and A server error occurred.  Please contact the administrator. on the web page.
When I look inside my venv/lib/python2.7/site-packages/cms (venv is the virtual environment of my project) I indeed don't see a utils.py file.
How do I install it and why is it missing? Are the docs pointing to a wrong location?

@dmk12 what's the output of pip freeze|grep cms?
Middleware is https://github.com/divio/django-cms/blob/develop/cms/middleware/utils.py#L7 and both tar.gz and wheel file in pypi contains the relevant code

Output of pip freeze|grep cms:
cmsplugin-filer==1.0.0
django-cms==3.1.4
djangocms-admin-style==1.0.7
djangocms-column==1.6.0
djangocms-file==0.2.0
djangocms-flash==0.3.0
djangocms-googlemap==0.4.0
djangocms-inherit==0.1.1
djangocms-installer==0.8.1
djangocms-link==1.7.1
djangocms-picture==0.2.0
djangocms-style==1.6.0
djangocms-teaser==0.2.0
djangocms-text-ckeditor==2.8.0
djangocms-video==0.2.0

And /cms/middleware/utils.py is missing in my installation.

@dmk12 apparently you have django-cms 3.1.4, which does not include the middleware

Doh, my bad :) Thanks, @yakky


DeepSpeed FP16 causes bert pretrain ACC decline or NAN?

We use DeepSpeed (v0.4.0) FP16 to train Bert Large, we found some Acc decline cases or NAN cases.
Like this picture:

And this picture:

When we set FP16 config to false or use AMP. The Acc convergence is good.
DeepSpeed FP16 implementation is like apex O2, there is some convergence issue about amp O2.
Is there some solution to the convergence problem of DeepSpeed FP16?

@mpjlu, thanks for reporting this issue. Can you please share your deepspeed config?

My deepspeed config is:
{
2   "gradient_accumulation_steps": 8,
3   "train_micro_batch_size_per_gpu": 32,
4   "steps_per_print": 1000,
5   "prescale_gradients": false,
6   "optimizer": {
7     "type": "Lamb",
8     "params": {
9       "lr": 1e-3,
10       "weight_decay": 0.01,
11       "bias_correction": false,
12       "max_coeff": 0.3,
13       "min_coeff": 0.01
14     }
15   },
16   "gradient_clipping": 1.0,
17
18   "wall_clock_breakdown": false,
19
20   "fp16": {
21     "enabled": true,
22     "loss_scale": 0
23   }
24 }
~
Thanks

@mpjlu, thanks for sharing your deepspeed config. It looks reasonable. I have a few other questions to help understand the problem.

What are those two plots showing? What are the axes?
Can you share the corresponding amp convergence plot as well?
How many GPUs are you using?
Is the dataset publicly available for us to try to repro?

Thanks!

@mpjlu, just wanted to check if this was still an issue? Thanks.

Yes, this is still an issue.
The plots show the steps (X) and the ACC,
The FP32 plot is like this:

We are using 64 GPUs.
The dataset is not publicly available.
DeepSpeed FP16 is like AMP O2, there are many discussions about the convergence issue of O2,  like this:

https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html
So I want to know whether DeepSpeed FP16 has the same problem as amp O2.

@mpjlu, thanks for the clarifications. Yes, DeepSpeed FP16 is similar to amp O2 and should have same issues.

Do you have a plan to fix this problem?

@mpjlu, yes we would like to fix this problem. The challenge is that we don't currently have an easy repro. If you could provide such even on a small case, that would be very helpful.


Added table to show Kubernetes options

When I read this doc it was unclear that the cert could be provided as a secret. The only table on the original version showed spnCertificateFile as a required option. I added a second table to make it clear that in Kubernetes there is a spnCertificate option that does not require a file on disk.
Thank you for helping make the Dapr documentation better!
Please follow this checklist before submitting:

[x] Read the contribution guide
[ ] Commands include options for Linux, MacOS, and Windows within codetabs
[ ] New file and folder names are globally unique
[ ] Page references use shortcodes instead of markdown or URL links
[ ] Images use HTML style and have alternative text
[ ] Places where multiple code/command options are given have codetabs

In addition, please fill out the following to help reviewers understand this pull request:
Description
After talking to MarkF we decided to update the docs to make it more clear there are additional options besides just providing a file path.

There's also a merge conflict due to a recent structure change. I'd rebase on the latest 1.1

There's also a merge conflict due to a recent structure change. I'd rebase on the latest 1.1

Done

@DarqueWarrior - See @AaronCrawfis provided a few suggested edits, can you respond to those?

@orizohar all the comments are outdated... So do you need me to do anything? I am a bit confused.

@orizohar all the comments are outdated... So do you need me to do anything? I am a bit confused.

I'm still seeing suggested edits in the PR . @AaronCrawfis can you confirm?

@orizohar all the comments are outdated... So do you need me to do anything? I am a bit confused.

I'm still seeing suggested edits in the PR . @AaronCrawfis can you confirm?

Yeah my suggestions are still there, although marked as Outdated, hence the confusion. I'll see if I can upodate them


Adds the word issues after the counter

Hello everyone!
The possibility of filtering alphabetically and putting ... (x issues) in the dropdown was being discussed in #38.
I couldn't put it in alphabetical order, but I was able to include the word issues after the counter.
This PR also adds to Filter by Issue Label and Filter by Repository.
This partially resolves #38.
If it's interesting, consider merging. 😉

Hi, just a small nitpick, but I would want the text to show "1 issue" instead of "1 issues" :)

Hi, just a small nitpick, but I would want the text to show "1 issue" instead of "1 issues" :)

Really! I created a conditional to check the number of issues.

Cool, thanks for your contribution!


Error to run tdlib in springboot project in windows 10

i've a springboot and try to run
./mvnw spring-boot:run '-Dspring-boot.run.jvmArguments="-Djava.library.path=src/main/resources/native/windows/x64"'
the tdjni.dll path is in here

$Projectroot/src/main/resources/native/windows/x64/dll

When i run the first script, i got

A fatal error has been detected by the Java Runtime Environment:
EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffd34892f58, pid=17124, tid=5200
JRE version: Java(TM) SE Runtime Environment (21.0.3+7) (build 21.0.3+7-LTS-152)
Java VM: Java HotSpot(TM) 64-Bit Server VM (21.0.3+7-LTS-152, mixed mode, emulated-client, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64)
Problematic frame:
C  [msvcp140.dll+0x12f58]
No core dump will be written. Minidumps are not enabled by default on client versions of Windows
An error report file with more information is saved as:
D:\College\Industrial Practices\socmed-service\hs_err_pid17124.log
[4.569s][warning][os] Loading hsdis library failed
If you would like to submit a bug report, please visit:
https://bugreport.java.com/bugreport/crash.jsp
The crash happened outside the Java Virtual Machine in native code.
See problematic frame for where to report the bug.
and then I tried to run this code in the tdjni.dll directory
java -Djava.library.path=. org.drinkless.MainClass
but, i got
Error: Could not find or load main class .library.path=.
Caused by: java.lang.ClassNotFoundException: /library/path=/
can anyone can help me in this?

just to make sure, so what I need to do is basically recreate https://tdlib.github.io/td/build.html?language=Java but with having a modifiedd CMakeLists.txt, is that correct?

Yes, modifying CMakeLists.txt or downgrading MSVC is supposed to help.

Thank you, i want to make sure again. so , what i need to do is basically convert it into a .jar, add it to maven, and then then include the java.library.path in the command again ( referencing to where the .dll file are) into the arguments when running the springboot using ./mvnw spring-boot:run ?

Yes, after rebuilding you can pack the library in a .JAR file and use it in your project.

hello sir, could you help me again? just to really make sure again. so, i've to follow this instructions https://tdlib.github.io/td/build.html?language=Java, but changed the cmakelist just right after the first line, and then after that continue the installation.
after all the instruction is done, then i've to create jar using this command in the td root directory
jar cf tdlib.jar td
and after that, i've to include the jar into the pom. i also have to include the .dll using argument java.library.path when running the springboot using ./mvnw spring-boot:run, is that correct?

The first paragraph is correct.
I can't confirm exact steps for including the library in a springboot project.

i want to make sure again. everytime that i want to run the project with tdlib.jar being used inside of the java project, i must supply the java.library.path for the tdjni.dll in the script argument?

Yes. You can include only Java classes in .jar, so you need to ship tdjni.dll and its dependent libraries and provide java.library.path to be able to load them.

i'm kinda a little confused. after following the instructions in here https://tdlib.github.io/td/build.html?language=Java , do i've to follow the other instructions in here https://github.com/tdlib/td/tree/master/example/java ? like both of this
cd <path to TDLib sources> mkdir jnibuild cd jnibuild cmake -DCMAKE_BUILD_TYPE=Release -DTD_ENABLE_JNI=ON -DCMAKE_INSTALL_PREFIX:PATH=../example/java/td .. cmake --build . --target install
cd <path to TDLib sources> mkdir jnibuild cd jnibuild cmake -DCMAKE_BUILD_TYPE=Release -DTD_ENABLE_JNI=ON -DCMAKE_INSTALL_PREFIX:PATH=../example/java/td .. cmake --build . --target install

https://github.com/tdlib/td/tree/master/example/java contains generic description of the build process and https://tdlib.github.io/td/build.html?language=Java shows specific instructions for the chosen operating system. It is simpler to use https://tdlib.github.io/td/build.html?language=Java in most cases.

okay, thanks for answering. I think there are several unknown problem to me that I didn't know before ( like somehow the dlls is being imported twice or more in the runtime ).

Hi @jonnjonnjo , did you import tdlib to a spring boot project successful ? can you give me some tips

Yes, I do!
I think i got several errors.

make sure you follow the instructions and change the CMake just like what levlam asked. I literally just copied the whole instructions from  https://tdlib.github.io/td/build.html?language=Java and from my experience, because I executed the whole sequence , i got error because sometimes my connection appeared to failed to clone the git and then my powershell proceed into executing the next task.

Make sure that there's no error in powershell when executing the job ( I got error because i don't have some libraries such as ssh, zlib, or something that i've to download independently )

from that on, you could go to td\tdlib\bin and then create the jar ( I kinda forgot whether to create .jar from directory td\tdlib\bin or td\tdlib\bin\org or td\tdlib\bin\org\drinkless ), just make sure that you also check the jar using jar tf

Add the tdlib dependency into your pom.xml, in my case, i created it just like this one
		<dependency>  		<groupId>org.drinkless</groupId>  		<artifactId>tdlib</artifactId>  		<version>1.0.0</version>  		<scope>system</scope>  		<systemPath>${project.basedir}/src/main/resources/lib/tdlib.jar</systemPath>  	</dependency>
and add the tdlib.jar into src/main/resources/lib/ folder

inside of your td\tdlib\bin, there's several .dll classes that's libcrypto-3-x64, libssl-3-x64, tdjni , and zlib1. copy that into
src\main\resources\native\windows

2 out of 4 classes of that actually exist in C:\Windows\System32, i accidentally copied all of them so i "kinda" replaced the old .dll files to the C:\Windows\System32 ( i think you shouldn't follow this one )

run ./mvnw spring-boot:run '-Dspring-boot.run.jvmArguments="-Djava.library.path=src/main/resources/native/windows"'. i run it from powershell

Hi @jonnjonnjo ,
I followed the tdlib instructions and run it successful, but could not import to other spring boot project.
in the tdlib example, i added maven to it then i have installed tdlib.jar to maven local repository (mvn clean install), then import it to my spring boot project in pom:
 org.drinkless tdlib 1.0.SNAPSHOT
but when compile my spring project, it said "could not find package org.drinkless.tdlib"

Yes, I do!
I think i got several errors.

make sure you follow the instructions and change the CMake just like what levlam asked. I literally just copied the whole instructions from  https://tdlib.github.io/td/build.html?language=Java and from my experience, because I executed the whole sequence , i got error because sometimes my connection appeared to failed to clone the git and then my powershell proceed into executing the next task.
Make sure that there's no error in powershell when executing the job ( I got error because i don't have some libraries such as ssh, zlib, or something that i've to download independently )
from that on, you could go to td\tdlib\bin and then create the jar ( I kinda forgot whether to create .jar from directory td\tdlib\bin or td\tdlib\bin\org or td\tdlib\bin\org\drinkless ), just make sure that you also check the jar using jar tf
Add the tdlib dependency into your pom.xml, in my case, i created it just like this one
		<dependency> <groupId>org.drinkless</groupId> <artifactId>tdlib</artifactId> <version>1.0.0</version> <scope>system</scope> <systemPath>${project.basedir}/src/main/resources/lib/tdlib.jar</systemPath> </dependency>
and add the tdlib.jar into src/main/resources/lib/ folder
inside of your td\tdlib\bin, there's several .dll classes that's libcrypto-3-x64, libssl-3-x64, tdjni , and zlib1. copy that into
src\main\resources\native\windows
2 out of 4 classes of that actually exist in C:\Windows\System32, i accidentally copied all of them so i "kinda" replaced the old .dll files to the C:\Windows\System32 ( i think you shouldn't follow this one )
run ./mvnw spring-boot:run '-Dspring-boot.run.jvmArguments="-Djava.library.path=src/main/resources/native/windows"'. i run it from powershell

what is the expectation should show when use jar tf tdlib.jar bro @jonnjonnjo

@jonnjonnjo , the issue was resolved when i comment out this plugins in file pom of tdlib project :):


Skyblock

Added redstone powered chunk loading

Need to add that if flower pot is next to chunk border, the chunk it touches is also loaded. As per requested by ilmango

Flower pots now load chunks there touching!

If this gets added we need to add a custom crafting recipe for flower pots, & it needs to be hard to obtain!

Fixed issue where Flower Pots still load chunks if broken while powered


[CLOSED] find a way to link to the fb conversation

 Issue by WestleyArgentum
Tuesday Jun 07, 2016 at 05:00 GMT
Originally opened as https://github.com/buzzfeed-openlab/convention-bot/issues/12

in the response view

 Comment by WestleyArgentum
Saturday Jun 25, 2016 at 23:46 GMT

I posted an issue about this but it doesn't seem possible :(


Issue with Persisting User Progress in Flashcard Array

Problem Summary
I am encountering a problem with maintaining the user's progress within a flashcard array component. The key functionality required is for the application to save the user's current position within the flashcard sequence locally on the client side. This way, if a user switches tabs or navigates away and then returns, they can continue from the last flashcard they viewed instead of starting over from the first flashcard. Attempts to use a setCurrentCard function to achieve this have been unsuccessful, leading me to believe that this function might not directly influence how the flashcard is rendered or does not interact with the component's state as expected.
Expected Behavior
When a user navigates away from the flashcard component (switching tabs, navigating to a different page, etc.) and then returns, their progress should be preserved, allowing them to resume from the last flashcard they were viewing.
Actual Behavior
Currently, despite attempts to implement functionality to save and restore the user's progress (using a setCurrentCard approach), the flashcard array resets to the first card whenever the user leaves and returns to the component, indicating a failure to persist user progress across sessions.
Steps to Reproduce

Navigate to the flashcard component and interact with it to change the currently displayed flashcard from the default (first) position.
Switch tabs or navigate away from the page.
Return to the flashcard component and observe that the display has reset to the first flashcard, instead of displaying the last flashcard viewed.

Attempted Solutions and Observations

I attempted to save the user's current flashcard index in localStorage upon navigating away and then restore this index using setCurrentCard upon returning.
The restoration process seems to correctly identify and attempt to set the saved index, but the displayed flashcard does not update accordingly.
This leads to the hypothesis that setCurrentCard may not be affecting the flashcard rendering process or is not properly integrated with the component's state management.

Request for Assistance

Could you provide insight or confirm whether setCurrentCard is intended to influence the rendering of flashcards directly, or if an alternative method is recommended for this purpose?
Is there a recommended approach or best practice for persisting and restoring user progress within the flashcard array component that I might have overlooked?

Hey!
I forgot about this case and I didn’t notice it during development. Give me a few days, I’ll try to add that to the library.


Trouble getting started with target rate sensors

Describe the bug
I cannot seem to get target rate sensors to work - i don't seem to have the binary sensors as shown in the example. Maybe this is user error on my part - can we have some full examples of exactly what we need to configure to get this

Hello. What part of the process are you having issues with? If you "add the integration" following the link a second time after you've added your account, you will be presented with the option to setup a target rate sensor.

I was adding the integration the second time manually - user error. It works fine when following the link
I have the sensor created now


chore: deps linter

Добавлен линтер для проверки зависимостей. Выявляет неиспользуемые и отсутствующие в package.json зависимости

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 80.279%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
7894

Relevant Lines:
8921

💛  - Coveralls


Adding info on customer defined values

This doc should be inside "datacenter network integration" between "border connectivity" and "DNS Integration".

@wamota : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@wamota - Microsoft authors and contributors need to make content contributions through the private repository so the content can be staged and validated by the current validation rules.
After you review staging and fix all validation issues, you can add the sign-off comment to let PR reviewers know that the updates are ready to be merged.
Read about authors working in the public repo for more information.
If you need help getting set up to work in the private repo, see Set up Git repository locally for documentation.
We'll close this PR. Thank you.
Cc @femila as service area content owner

@wamota I'll open a new pr, review the topic, and send you the topic when it is ready to review. Thanks, Matt


Explicitly selecting new publish should clear all publish fields

If you start an edit and abandon it, state is still there when selecting new

I've got this one figured out. I'll open a review for it tomorrow evening.


Fix activate.sh to find CHIP_ROOT

Problem
if CHIP_ROOT is unset when sourcing activate.sh from bash, activate.sh erroneously uses $0 (which is "bash" when in bash) to locate the tree
source <(echo echo $0)

Summary of Changes
fix for bash (tested with zsh, too)

Should fix scripts/bootstrap.sh as well.

Should fix scripts/bootstrap.sh as well.

updated


[contracts] USDT-like asset transfer

Describe the bug
The transfer of USDT-like assets currently results in a double-spend, and we should be more restrictive about what we consider a failed transfer.
We should always bias towards locked funds over double spends.

Is that a new bug you discovered since, or is this supposed to be a summary of Friday's discussion?
It is my understanding that standard-conforming as well as missing-return-value-bug tokens like USDT are safe to use. Only for tokens that ignore the standard even more blatantly -- which means they return something that is not a boolean -- could such a double-spend occur. And we don't support such tokens to begin with. However, in order to introduce an additional safety net, just in case such a token makes it into the system anyway, the suggestion was made to interpret a non-boolean return value as success -- rather than failure, as is currently the case --, thus favoring frozen funds over double-spends, in case our interpretation was wrong (taking also into account that the "frozen" funds can still be withdrawn cooperatively).

This is supposed to be the a summary of the Friday discussion, not a new issue. Your summary is definitely better. Is this a duplicate of #121 ?

Ok, no worries! I just wanted to make sure I don't accidentally discard what would be a very serious issue in case I misunderstood.
It's not a duplicate. While this issue is about transfers out of the channel, #121 deals with transfers into the channel (for Alice). The latter still only work with a strictly standard-compliant token.


Warning: Link: context type router is invalid; it must be a function, usually from React.PropTypes.

##version
react 15.3.0
react-router 2.6.1
use rollup to pack
I found a strange problem. When I use , some msg appears in the consol as follow.
Warning: Link: context type `router` is invalid; it must be a function, usually from React.PropTypes.

And I try to see the call stack. When 's craeatClass has property 'contextTypes', the stack trace is
mixSpecIntoComponent => RESERVED_SPEC_KEYS[contextTypes] => validateTypeDef =>
warning.

So it log in the console.
When I remove the contextType property in node_modules/react-router/lib/Link.js, the warning disappered. I don't know why add the contextType will cause the warning, and why the contextType.router is undefined here.
Any suggestion to solve?
PS: I have a same project use webpack. It doesn't have any warning.
the code is in this repo react-rollup-starter

If you're only seeing this in Rollup and not webpack, I'd follow up with the Rollup folks. We're just doing the normal transpiled ES modules thing.


Fix Nightly cucumber test

https://ci.opensmartgridplatform.org/view/Nightly/job/OSGP_open-smart-grid-platform_nightly/1185/


[Integration] Integration of test suite to run on Browserstack

Hi
We just integrated Material-UI's tests to run on BrowserStack. The test suite will run across latest browsers of Chrome, Firefox, IE and Safari. This uses karma-browserstack-launcher. You can use npm run test:karma. This will run the tests on PhantomJS and BrowserStack browsers.
To enable it on your Travis build, you just have to expose BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY. You can set it at repository level. More information - here.

[x] PR has tests / docs demo, and is linted.
[x] Commit and PR titles begin with [ComponentName], and are in imperative form: "[Component] Fix leaky abstraction".
[ ] Description explains the issue / use-case resolved, and auto-closes the related issue(s) (http://tr.im/vFqem).

Ankur Goel
- BrowserStack Automate

@AnkurGel Thanks for opening this PR, however, the master branch is in a maintenance mode. Would you mind submitting the same PR on the next branch in active development? We use a similar setting with karma.


KeyError: '1' when running methgo snp

I tried running methgo snp with the command below, and got an error:
# /data/home/user/paoyangchen-laboratory-methgo-911acbc/methgo snp -g /data/home/user/genomes/Human/GRCh38/Homo_sapiens.GRCh38.dna.all.fa /data/group/test/TEST_Run001/TEST_Run001_bismark_hg38_samplebysample-12345678/WBGS_oxBS_vB_1000ng_PCR06_R02-12345678/WBGS-oxBS-vB-1000ng-PCR06-R02_S1_L001_R1_001.cutB_pe.sorted.mrgd.bam
Traceback (most recent call last):
  File "/data/home/user/paoyangchen-laboratory-methgo-911acbc/methgo", line 13, in <module>
    module.main()
  File "/data/home/user/paoyangchen-laboratory-methgo-911acbc/scripts/snp/snp.py", line 105, in main
    snpOutput(args.bamfile,args.genomeFile,args.coverage,args.majorAlleleFreq,args.buffer)
  File "/data/home/user/paoyangchen-laboratory-methgo-911acbc/scripts/snp/snp.py", line 75, in snpOutput
    refNuc = openFile[chr][pos]
  File "/data/home/user/.local/lib/python2.7/site-packages/pyfasta/fasta.py", line 128, in __getitem__
    c = self.index[i]
KeyError: '1'

Any ideas?

Can you please check the FASTA file is the same file you used to generate the BAM file?
I guess it's a mismatch in chromosome names, e.g. one maybe use "chr1", but the other use "1".

Dear Wen-Wei Liao,
I ran a small example with these two input files:
https://cloud.curoverse.com/collections/download/qr1hi-4zz18-na0zghh5hrqpgcv/19b2plvcyblkufyh0c30y7q9ct0fmd63nln4zhcqqp6amas3yi/
$HOME/paoyangchen-laboratory-methgo-911acbc/methgo snp -g
$GROUP/Genomes/Human/hs38DH/hs38DH.karyotype.fa
WBGS-oxBS-vB-1000ng-PCR06-R02_S1_L001_R1_001.cutB_pe.sorted.mrgd.karyotype.bam
Traceback (most recent call last):
  File "/bi/home/rocks-av/paoyangchen-laboratory-methgo-911acbc/methgo",
line 13, in <module>
    module.main()
  File
"/bi/home/rocks-av/paoyangchen-laboratory-methgo-911acbc/scripts/snp/snp.py",
line 105, in main

snpOutput(args.bamfile,args.genomeFile,args.coverage,args.majorAlleleFreq,args.buffer)
  File
"/bi/home/rocks-av/paoyangchen-laboratory-methgo-911acbc/scripts/snp/snp.py",
line 71, in snpOutput
    for col in bam.pileup():
  File "pysam/calignmentfile.pyx", line 1066, in
pysam.calignmentfile.AlignmentFile.pileup (pysam/calignmentfile.c:12408)
ValueError: no index available for pileup

On Fri, Dec 11, 2015 at 4:13 PM, Wen-Wei Liao<EMAIL_ADDRESS>wrote:

Can you please check the FASTA file is the same file you used to generate
the BAM file?
I guess it's a mismatch in chromosome names, e.g. one maybe use "chr1",
but the other use "1".
—
Reply to this email directly or view it on GitHub
https://github.com/paoyangchen-laboratory/methgo/issues/1#issuecomment-163977733
.

Before you run SNP module of MethGo, the BAM file need to be indexed first (I'll update this info on the docs later):
$ samtools index your.bam
It'll generate a *.bam.bai in the same place of your BAM file.
After getting the index file, please try to run SNP module again.


Fix brand image url

Link was broken.

Thanks for opening this PR @bbonch. The link needed to be updated at some other places as well and with a different URL so I'm closing this PR superseded by https://github.com/twbs/examples/pull/36.


Video: AI Forum 2023 | The Small Models Revolution

AI Forum 2023 | The Small Models Revolution
Link: AI Forum 2023 | The Small Models Revolution

phi-1

1.3B params
Coding

phi-1.5

1.3B params
Natural Language model with NL

phi-1.5 vs Falcon-7B vs Llama 2-7B

Getting high quality = high educational value by filtering. Filtering by labeling small size of data and developing small classifier. And generated 1B of pure education content.

Compares with major OSS LLMs. Shows near performance of 13B params LLM.


Add auto merge

@conda-forge-admin, please add bot automerge

Hi! This is the friendly automated conda-forge-webservice.
I just wanted to let you know that I added bot automerge in conda-forge/pyiron_gpl-feedstock#3.


Update announcements.yml

Add 3/15 Quarterly Webinar info & registration link

@balhar-jakub please review (double check that link from OMP - we've had issues with this) and merge. TY

I checked the link and it works.


chore(CI): Workflow that will take care of Release Version updates

Workflow that will take care of Release Version updates

Type of change:

[ ] Bug fix
[x] Improvement
[ ] New feature
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] Documentation

Checklist:

[x] I have read the CONTRIBUTING document.
[ ] My PR title is Fixes <issue-number>: <short explanation>
[ ] I have commented on my code, particularly in hard-to-understand areas.
[ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

@akash-jain-10 Should we add the ticket number to follow up?
Other than that, LGTM


fix make target build-images

Signed-off-by: Mike Ng<EMAIL_ADDRESS>tested locally.

same changes as the other 5 repos .


meet deploy problem when using Group Fisher in rtmdet with custom data

Checklist

I have searched related issues but cannot get the expected help.
I have read related documents and don't know what to do.

Describe the question you meet
I use  Group Fisher to prune the rtmdet, and when deploying the pruned model with this scripts, it meets the ERROR:Exception: Forward failed, there may be an error in demo input.here I just want to export the onnx or trt model.
Post related information

The output of pip list | grep "mmcv\|mmrazor\|^torch"
mmcv                    2.0.0rc4
mmrazor                 1.0.0rc2     /home/yaojiahao/zbc/mmrazor
torch                   1.13.0+cu116
torchaudio              0.13.0+cu116
torchvision             0.14.0+cu116
Your config file if you modified it or created a new one.

_base_ = 'mmyolo::rtmdet/rtmdet_s_syncbn_fast_1xb64-300e_fs_4k.py'
fix_subnet = {
    "backbone.stem.0.conv_(0, 16)_16": 13,
    "backbone.stem.1.conv_(0, 16)_16": 16,
    "backbone.stem.2.conv_(0, 32)_32": 32,
    "backbone.stage1.0.conv_(0, 64)_64": 63,
    "backbone.stage1.1.short_conv.conv_(0, 32)_32": 32,
    "backbone.stage1.1.main_conv.conv_(0, 32)_32": 32,
    "backbone.stage1.1.blocks.0.conv1.conv_(0, 32)_32": 31,
    "backbone.stage1.1.final_conv.conv_(0, 64)_64": 64,
    "backbone.stage2.0.conv_(0, 128)_128": 128,
    "backbone.stage2.1.short_conv.conv_(0, 64)_64": 64,
    "backbone.stage2.1.main_conv.conv_(0, 64)_64": 64,
    "backbone.stage2.1.blocks.0.conv1.conv_(0, 64)_64": 64,
    "backbone.stage2.1.blocks.1.conv1.conv_(0, 64)_64": 63,
    "backbone.stage2.1.final_conv.conv_(0, 128)_128": 128,
    "backbone.stage3.0.conv_(0, 256)_256": 256,
    "backbone.stage3.1.short_conv.conv_(0, 128)_128": 127,
    "backbone.stage3.1.main_conv.conv_(0, 128)_128": 128,
    "backbone.stage3.1.blocks.0.conv1.conv_(0, 128)_128": 128,
    "backbone.stage3.1.blocks.1.conv1.conv_(0, 128)_128": 128,
    "backbone.stage3.1.final_conv.conv_(0, 256)_256": 256,
    "backbone.stage4.0.conv_(0, 512)_512": 512,
    "backbone.stage4.1.conv1.conv_(0, 256)_256": 256,
    "backbone.stage4.1.conv2.conv_(0, 512)_512": 512,
    "backbone.stage4.2.short_conv.conv_(0, 256)_256": 256,
    "backbone.stage4.2.main_conv.conv_(0, 256)_256": 256,
    "backbone.stage4.2.blocks.0.conv1.conv_(0, 256)_256": 256,
    "backbone.stage4.2.blocks.0.conv2.pointwise_conv.conv_(0, 256)_256": 256,
    "backbone.stage4.2.final_conv.conv_(0, 512)_512": 511,
    "neck.reduce_layers.2.conv_(0, 256)_256": 256,
    "neck.top_down_layers.0.0.short_conv.conv_(0, 128)_128": 128,
    "neck.top_down_layers.0.0.main_conv.conv_(0, 128)_128": 128,
    "neck.top_down_layers.0.0.blocks.0.conv1.conv_(0, 128)_128": 128,
    "neck.top_down_layers.0.0.blocks.0.conv2.pointwise_conv.conv_(0, 128)_128": 128,
    "neck.top_down_layers.0.0.final_conv.conv_(0, 256)_256": 256,
    "neck.top_down_layers.0.1.conv_(0, 128)_128": 128,
    "neck.top_down_layers.1.short_conv.conv_(0, 64)_64": 64,
    "neck.top_down_layers.1.main_conv.conv_(0, 64)_64": 62,
    "neck.top_down_layers.1.blocks.0.conv1.conv_(0, 64)_64": 61,
    "neck.top_down_layers.1.blocks.0.conv2.pointwise_conv.conv_(0, 64)_64": 64,
    "neck.top_down_layers.1.final_conv.conv_(0, 128)_128": 127,
    "neck.downsample_layers.0.conv_(0, 128)_128": 128,
    "neck.bottom_up_layers.0.short_conv.conv_(0, 128)_128": 128,
    "neck.bottom_up_layers.0.main_conv.conv_(0, 128)_128": 128,
    "neck.bottom_up_layers.0.blocks.0.conv1.conv_(0, 128)_128": 128,
    "neck.bottom_up_layers.0.blocks.0.conv2.pointwise_conv.conv_(0, 128)_128": 128,
    "neck.bottom_up_layers.0.final_conv.conv_(0, 256)_256": 254,
    "neck.downsample_layers.1.conv_(0, 256)_256": 255,
    "neck.bottom_up_layers.1.short_conv.conv_(0, 256)_256": 255,
    "neck.bottom_up_layers.1.main_conv.conv_(0, 256)_256": 256,
    "neck.bottom_up_layers.1.blocks.0.conv1.conv_(0, 256)_256": 255,
    "neck.bottom_up_layers.1.blocks.0.conv2.pointwise_conv.conv_(0, 256)_256": 256,
    "neck.bottom_up_layers.1.final_conv.conv_(0, 512)_512": 406
}
divisor = 16

##############################################################################

architecture = _base_.model

model = dict(
    _delete_=True,
    _scope_='mmrazor',
    type='GroupFisherDeploySubModel',
    architecture=architecture,
    fix_subnet=fix_subnet,
    divisor=divisor,
)

Your train log file if you meet the problem during training.

/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmdet/utils/setup_env.py:83: UserWarning: The current default scope "mmyolo" is not "mmdet", register_all_modules will force the currentdefault scope to be "mmdet". If this is not expected, please set init_default_scope=False.
warnings.warn('The current default scope '
/home/yaojiahao/zbc/mmrazor/mmrazor/utils/setup_env.py:77: UserWarning: The current default scope "mmdet" is not "mmrazor", register_all_modules will force the currentdefault scope to be "mmrazor". If this is not expected, please set init_default_scope=False.
warnings.warn(
/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmdet/utils/setup_env.py:83: UserWarning: The current default scope "mmrazor" is not "mmdet", register_all_modules will force the currentdefault scope to be "mmdet". If this is not expected, please set init_default_scope=False.
warnings.warn('The current default scope '
/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmdet/utils/setup_env.py:83: UserWarning: The current default scope "mmyolo" is not "mmdet", register_all_modules will force the currentdefault scope to be "mmdet". If this is not expected, please set init_default_scope=False.
warnings.warn('The current default scope '
/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/torch/fx/_symbolic_trace.py:564: UserWarning: Was not able to add assertion to guarantee correct input data_samples to specialized function. It is up to the user to make sure that your inputs match the inputs you specialized the function with.
warnings.warn(
/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.)
return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Process Process-2:
Traceback (most recent call last):
File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in call
ret = func(*args, **kwargs)
File "/home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/pytorch2onnx.py", line 63, in torch2onnx
torch_model = task_processor.build_pytorch_model(model_checkpoint)
File "/home/yaojiahao/zbc/mmdeploy/mmdeploy/codebase/mmrazor/deploy/mmrazor.py", line 133, in build_pytorch_model
model.post_process_for_mmdeploy()
File "/home/yaojiahao/zbc/mmrazor/mmrazor/implementations/pruning/group_fisher/prune_deploy_sub_model.py", line 19, in post_process_for_mmdeploy
s = make_channel_divisible(model, divisor=divisor)
File "/home/yaojiahao/zbc/mmrazor/mmrazor/models/utils/expandable_utils/tools.py", line 76, in make_channel_divisible
mutator = to_expandable_model(model)
File "/home/yaojiahao/zbc/mmrazor/mmrazor/models/utils/expandable_utils/tools.py", line 23, in to_expandable_model
mutator.prepare_from_supernet(model)
File "/home/yaojiahao/zbc/mmrazor/mmrazor/models/mutators/channel_mutator/channel_mutator.py", line 110, in prepare_from_supernet
units = self._prepare_from_tracer(supernet, self.parse_cfg)
File "/home/yaojiahao/zbc/mmrazor/mmrazor/models/mutators/channel_mutator/channel_mutator.py", line 307, in _prepare_from_tracer
unit_configs = tracer.analyze(model)
File "/home/yaojiahao/zbc/mmrazor/mmrazor/models/task_modules/tracer/channel_analyzer.py", line 127, in analyze
return self._find_mutable_units(model, unit_configs)
File "/home/yaojiahao/zbc/mmrazor/mmrazor/models/task_modules/tracer/channel_analyzer.py", line 167, in _find_mutable_units
raise Exception(
Exception: Forward failed, there may be an error in demo input.
5. Other code you modified in the mmrazor folder.
change the group_fisher_act_{action}_{model}_coco.py

Fine， change the divisor to 1 can solve this problem but leads to another problem,
RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. Dictionaries and strings are also accepted, but their usage is not recommended. Here, received an input of unsupported type: InstanceData
04/05 12:39:01 - mmengine - ERROR - /home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 - `mmdeploy.apis.pytorch2onnx.torch2onnx` with Call id: 0 failed. exit.

I'm sorry, I cannot reproduce this problem. I can deploy the pruned rtmdet on onnxcpu sucessfullly. Here is the version of my codes.
mmcls           1.0.0rc5
mmcv            2.0.0rc4
mmdet           3.0.0rc6
mmdet3d         1.1.0rc3
mmengine        0.7.0
mmpose          1.0.0rc0
mmrazor         1.0.0rc2
mmrotate        1.0.0rc0
mmsegmentation  1.0.0rc1
mmyolo          0.2.0
.
Please try the following commands and provide the error info if it failed
razor_config={your config}.py
deploy_config=mmdeploy/configs/mmdet/detection/detection_onnxruntime_static.py

python mmdeploy/tools/deploy.py $deploy_config \
    $razor_config \
   {your checkpoint} \
    mmdeploy/tests/data/tiger.jpeg \
    --work-dir ./work_dirs/mmdeploy

this is my env
mmcv       2.0.0rc4   https://github.com/open-mmlab/mmcv

mmdet      3.0.0rc6   https://github.com/open-mmlab/mmdetection

mmengine   0.6.0      https://github.com/open-mmlab/mmengine

mmrazor    1.0.0rc2   /home/yaojiahao/zbc/mmrazor

mmyolo     0.5.0      /home/yaojiahao/zbc/mmyolo

mmdeploy 1.0.0rc3     /home/yaojiahao/zbc/mmdeploy

here is my command
python tools/deploy.py configs/mmdet/detection/detection_onnxruntime_static.py \
../mmrazor/configs/pruning/mmdet/group_fisher/custom/group_fisher_act_deploy_rtmdet_1x_coco.py \
../mmrazor/work_dirs/group_fisher_act_finetune_rtmdet_1x_coco_0.8/best_coco/bbox_mAP_epoch_286.pth \
tests/auto_08856.jpg \
--device cuda:0 \
--work-dir work_dirs/deploy_rtmdet_prune_onnx/\
 --dump-info

here is the error info
04/06 19:06:54 - mmengine - INFO - Export PyTorch model to ONNX: work_dirs/deploy_rtmdet_prune_onnx/end2end.onnx. 04/06 19:06:54 - mmengine - WARNING - Can not find mmdet.models.dense_heads.DETRHead.forward_single, function rewrite will not be applied /home/yaojiahao/zbc/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/single_stage.py:84: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).   img_shape = [int(val) for val in img_shape] /home/yaojiahao/zbc/mmdeploy/mmdeploy/codebase/mmdet/models/detectors/single_stage.py:84: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   img_shape = [int(val) for val in img_shape] /home/yaojiahao/zbc/mmdeploy/mmdeploy/core/optimizers/function_marker.py:160: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   ys_shape = tuple(int(s) for s in ys.shape) /home/yaojiahao/zbc/mmyolo/mmyolo/models/task_modules/coders/distance_point_bbox_coder.py:45: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   assert points.size(-2) == pred_bboxes.size(-2) /home/yaojiahao/zbc/mmyolo/mmyolo/models/task_modules/coders/distance_point_bbox_coder.py:46: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   assert points.size(-1) == 2 /home/yaojiahao/zbc/mmyolo/mmyolo/models/task_modules/coders/distance_point_bbox_coder.py:47: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   assert pred_bboxes.size(-1) == 4 /home/yaojiahao/zbc/mmyolo/mmyolo/models/dense_heads/yolov5_head.py:372: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).   img_meta) in zip(flatten_decoded_bboxes, flatten_cls_scores, /home/yaojiahao/zbc/mmyolo/mmyolo/models/dense_heads/yolov5_head.py:394: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   if scores.shape[0] == 0: /home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmdet/models/utils/misc.py:336: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   num_topk = min(topk, valid_idxs.size(0)) /home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmengine/structures/instance_data.py:296: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.   return len(self.values()[0]) /home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmengine/structures/instance_data.py:139: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.   assert len(value) == len(self), 'The length of ' \ /home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmdet/models/dense_heads/base_dense_head.py:477: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   if with_nms and results.bboxes.numel() > 0: /home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmcv/ops/nms.py:276: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   if boxes.size(-1) == 5: /home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmcv/ops/nms.py:293: TracerWarning: torch.tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.   max_coordinate + torch.tensor(1).to(boxes)) /home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmcv/ops/nms.py:301: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   if boxes_for_nms.shape[0] < split_thr: /home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmcv/ops/nms.py:316: TracerWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).   for id in torch.unique(idxs): /home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmcv/ops/nms.py:123: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   assert boxes.size(1) == 4 /home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/mmcv/ops/nms.py:124: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!   assert boxes.size(0) == scores.size(0) Process Process-2: Traceback (most recent call last):   File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap     self.run()   File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/multiprocessing/process.py", line 108, in run     self._target(*self._args, **self._kwargs)   File "/home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__     ret = func(*args, **kwargs)   File "/home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/pytorch2onnx.py", line 110, in torch2onnx     export(   File "/home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 356, in _wrap     return self.call_function(func_name_, *args, **kwargs)   File "/home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 326, in call_function     return self.call_function_local(func_name, *args, **kwargs)   File "/home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 275, in call_function_local     return pipe_caller(*args, **kwargs)   File "/home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/core/pipeline_manager.py", line 107, in __call__     ret = func(*args, **kwargs)   File "/home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/onnx/export.py", line 131, in export     torch.onnx.export(   File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/torch/onnx/utils.py", line 504, in export     _export(   File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/torch/onnx/utils.py", line 1529, in _export     graph, params_dict, torch_out = _model_to_graph(   File "/home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/onnx/optimizer.py", line 11, in model_to_graph__custom_optimizer     graph, params_dict, torch_out = ctx.origin_func(*args, **kwargs)   File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/torch/onnx/utils.py", line 1111, in _model_to_graph     graph, params, torch_out, module = _create_jit_graph(model, args)   File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/torch/onnx/utils.py", line 987, in _create_jit_graph     graph, torch_out = _trace_and_get_graph_from_model(model, args)   File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/torch/onnx/utils.py", line 891, in _trace_and_get_graph_from_model     trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(   File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/torch/jit/_trace.py", line 1184, in _get_trace_graph     outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs)   File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1190, in _call_impl     return forward_call(*input, **kwargs)   File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/torch/jit/_trace.py", line 127, in forward     graph, out = torch._C._create_graph_by_tracing(   File "/home/yaojiahao/miniconda3/envs/mmyolo/lib/python3.9/site-packages/torch/jit/_trace.py", line 121, in wrapper     out_vars, _ = _flatten(outs) RuntimeError: Only tuples, lists and Variables are supported as JIT inputs/outputs. Dictionaries and strings are also accepted, but their usage is not recommended. Here, received an input of unsupported type: InstanceData 04/06 19:06:58 - mmengine - ERROR - /home/yaojiahao/zbc/mmdeploy/mmdeploy/apis/core/pipeline_manager.py - pop_mp_output - 80 -mmdeploy.apis.pytorch2onnx.torch2onnxwith Call id: 0 failed. exit.
but if I try to use
# print("test raw onnx start")
# test_model    = torch_model.eval()
#  dummy_input = torch.randn(1, 3, 640, 640, device='cuda:0')
# torch.onnx._export(test_model, dummy_input, "output_cuda.onnx", verbose=True, opset_version=11)

before the export function in pytorch2onnx.py to export
onnx manually, it can export correctlly. So this confused me

We find there is a bug when deploy a pruned model using cuda and fix it in this pr. Please try it.

We closed the issue as there has been no response in a long time. You can reopen it when you need.

sry for my no reply, but this pr still can not work in my env.The error info is still the same as before


App freezes

When trying to start a new measurement if a measurement is already running.

Thank you!
#21 should have fixed this by removing the option to start a new one. While not exactly user-friendly, waiting 10 seconds is perhaps not a big issue – on the other hand, stopping the current measurement would have been quite messy.


CS0433 Conflict between OpenTelemetry and Telemetry packages

Description
I am using the OpenTelemetry exporter to export traces to NewRelic. However I also want to export Metrics. Currently there is no OpenTelemetry integration for exporting metrics so I am trying to use the Telemetry package to send metrics directly.
However including both nuget packages in the project results in the following:
CS0433: The type 'TelemetryConfiguration' exists in both
'NewRelic.Telemetry, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=82b1ac20087f0ac7'
'OpenTelemetry.Exporter.NewRelic, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=82b1ac20087f0ac7'

It seems there might be a workaround but it does not appear to me working for me unfortunately (Possibly due to my inexperience with C# and getting it wrong) https://stackoverflow.com/questions/33460667/how-to-use-extern-alias-with-nuget
Steps to Reproduce
Create a C# project and add the two nuget packages from this Repo via nuget.
Expected Behaviour
I am able to use the NewRelic.Telemetry package without issues until OpenTelemetry Metrics exporting is enabled.

@rurounijones Please try the latest version of our exporter package. It has been renamed to NewRelic.OpenTelemetry https://www.nuget.org/packages/NewRelic.OpenTelemetry. The package was renamed in preparation for the upcoming GA of OpenTelemetry, so that all of our packages are prefixed with NewRelic.*.
I believe this issue should be remedied in when using the latest packages.

Will give it a try, thank you for the quick response.
For completeness sake I also got the alias method working using the following
csproj file
  <Target Name="ChangeAliasesOfNugetRefs" BeforeTargets="FindReferenceAssembliesForReferences;ResolveReferences">
    <ItemGroup>
      <ReferencePath Condition="'%(FileName)' == 'NewRelic.Telemetry'">
        <Aliases>workaround</Aliases>
      </ReferencePath>
    </ItemGroup>
  </Target>

code
extern alias workaround;
using workaround::NewRelic.Telemetry;

// <snip>

TelemetryConfiguration telemetryConfig = new TelemetryConfiguration();

Confirmed latest betas play nicely with each other. Apologies for not noticing later versions were available.

Great, glad that resolved things for you! Also thanks for the tip regarding the alias - didn't know this was a thing!


Python: Prevent bad magic in Expr::pointsTo/1.

For some reason, this problem is not apparent when running the query locally, but popped up on a dist-compare.  A more recent master shows no improvement, so I am closing this PR.


⚠ Processors has degraded performance

In 6418db0, Processors ($STATUS_URL) experienced degraded performance:

HTTP code: 200
Response time: 70 ms

Resolved: Processors performance has improved in c337277.


Second try is not displayed in the app

Hi, today I found out that second tries are not displayed in the app.
Expected behavior: The app shows the latest grade for the subject in the overview and lists all tries in the subject view.
Actual behavior: The app only shows the first/missed try and does not show that you passed the exam at the second try.
Background: The Technische Hochschule Köln HIS system lists all exam tries of one course and has the exam date at the far right. From what I can see, the newest exam is listed first; so I assume that the app overwrites the second try when reading the first.
I will look deeper into the issue tomorrow but look forward for your input on the topic.

Hey,
I think the behaviour should be as follows: All tries (also unsuccessful) are listed in the app as individual grades and are displayed in the semester where the exam was taken, respectively.
Possible problems:

examId is not unique anymore in the data model within the app. That's why we decided to use a hash of a combination of fields as primary key. So if there is a distinct combination of examId, semester, name and attempt the grades should also be distinct.
The line in which the latest result is given, might be different and does not match the TRANSFORMER_MAPPING_ITERATOR XPath expression anymore. So the app filters this entry out.

If you could provide (anonymised) source code of the webpage with the table of grades I'm sure I can be of more help than just  speculating :)

To the second problem: Would you please tell me the meaning of TRANSFORMER_MAPPING_ITERATOR? Then, I can see whether it will change.
I will publish the Table of Grades, but only after the weekend (I don't have my notebook with me right now).

Hi @jeyemwey,
the XPath expression TRANSFORMER_MAPPING_ITERATOR selects all table rows which contain grade entries. We basically iterate over that result set and convert each row into a grade entry.
Instead of publishing your (anonymised) table of grades, you can also send it to us via mail to<EMAIL_ADDRESS>Thanks for the help. :)

I just looked into the app again and the grade does appear now, sorry for the inconvenience.


Delete Cluster

dialog upon delete cluster request

Coverage remained the same at 60.027% when pulling 967373a1ccfbbc7d1d5d2a0c72be60b836c96fec on josephkane:delete-cluster into 839ef9e66ebe8fcaa3b2272408459c9f6869b384 on supergiant:master.


2.x ASG status overrides + DI refactorings

NetflixOSS » eureka » eureka-pull-requests #329 SUCCESS
This pull request looks good


Include multiarch buld into Docker

Would be nice to have build process included into the Dockerfile, for example:
ARG ARCH=amd64

# build stage
FROM golang:1.13 AS builder
RUN mkdir -p /go/src/app
WORKDIR /go/src/app
COPY . ./
RUN go mod download
RUN go mod verify
RUN CGO_ENABLED=0 GOOS=linux GOARCH=$ARCH go build -a -o /app .

# final stage
FROM alpine:latest
RUN apk --no-cache add ca-certificates
COPY --from=builder /app ./
RUN chmod +x ./app
ENTRYPOINT ["./app"]
EXPOSE 80

Our build tool actually already supported Docker multi-arch builds. I enabled the flag and now there's a Docker Hub image with multiple architectures


Interim CML refactor for fill_value and dtype

For sanity sake, this is an interim refactor of CML on the dask feature branch, this tee's up the fill_value and dtype changes for following PRs ...

Ping @lbdreyer @dkillick @pp-mo

Thanks @bjlittle You can now rebase onto the dask branch.