Collecting multiple attribute error

Attempt to clos e #2892 (Edit by @davidhewitt - I think there's more we can do with #[pymethods] probably, though might be a fair bit more work.)
I am not sure how to add the test in CI but I have tested manually with the following:
use pyo3::prelude::*;

#[pyclass]
struct Example {
    #[pyo3(foo)]
    #[pyo3(blah)]
    x: i32,
    #[pyo3(pop)]
    y: i32,
}

Originally it gave:
error: expected one of: `get`, `set`, `name`
 --> pytests/src/pyclasses.rs:7:12
  |
7 |     #[pyo3(foo)]
  |            ^^^

error: cannot find attribute `pyo3` in this scope
  --> pytests/src/pyclasses.rs:10:7
   |
10 |     #[pyo3(pop)]
   |       ^^^^
   |
   = note: `pyo3` is in scope, but it is a crate, not an attribute

error: could not compile `pyo3-pytests` (lib) due to 2 previous errors

Now it gives:
error: expected one of: `get`, `set`, `name`
 --> pytests/src/pyclasses.rs:7:12
  |
7 |     #[pyo3(foo)]
  |            ^^^

error: expected one of: `get`, `set`, `name`
 --> pytests/src/pyclasses.rs:8:12
  |
8 |     #[pyo3(blah)]
  |            ^^^^

error: expected one of: `get`, `set`, `name`
  --> pytests/src/pyclasses.rs:10:12
   |
10 |     #[pyo3(pop)]
   |            ^^^

error: could not compile `pyo3-pytests` (lib) due to 3 previous errors

Thanks for the suggestions @davidhewitt i have added the test and have refactored as suggested.

Thank you for your patience. I was busy with EuroPython. I plan to finish this PR this weekend or next week.

(I took the liberty to fix the new merge conflicts.)

Thank you @davidhewitt


Fix a build issue for PRESCOTT on x86_64

Define everything the same way for PRESCOTT as for PENTIUM4.
This is an RFC pull request. I have not actually tested running on a Prescott machine, only that 0.3.27 builds for x86_64.

Can you please expand on what build problem this PR is fixing for you ? 0.3.27 should build as-is for TARGET=PRESCOTT

Can you please expand on what build problem this PR is fixing for you ? 0.3.27 should build as-is for TARGET=PRESCOTT

Cross-compiling for Yocto 5.0 Scarthgap, using this recipe.
With the custom build flags used, I have effectively cmake -DTARGET=NEHALEM -DBUILD_WITHOUT_LAPACK=OFF -DBUILD_WITHOUT_CBLAS=OFF -DNO_AFFINITY=ON -DUSE_OPENMP=OFF -DDYNAMIC_ARCH=ON plus the extra flags for cross-compiling.
The build error was:
| FAILED: kernel/CMakeFiles/kernel_PRESCOTT.dir/CMakeFiles/sgemm_oncopy_PRESCOTT.S.o
| /home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/recipe-sysroot-native/usr/bin/x86_64-oe-linux/x86_64-oe-linux-gcc --sysroot=/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/recipe-sysroot -DUSE_GEMM3M -I/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git/lapack-netlib/LAPACKE/include -I/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git -I/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/build/kernel_config/PRESCOTT -m64 -march=nehalem -mtune=generic -mfpmath=sse -msse4.2 -fstack-protector-strong  -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security  --sysroot=/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/recipe-sysroot  -O2 -pipe -g -feliminate-unused-debug-types -fcanon-prefix-map  -fmacro-prefix-map=/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git=/usr/src/debug/openblas/0.3.27  -fdebug-prefix-map=/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git=/usr/src/debug/openblas/0.3.27  -fmacro-prefix-map=/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/build=/usr/src/debug/openblas/0.3.27  -fdebug-prefix-map=/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/build=/usr/src/debug/openblas/0.3.27  -fdebug-prefix-map=/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/recipe-sysroot=  -fmacro-prefix-map=/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/recipe-sysroot=  -fdebug-prefix-map=/home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/recipe-sysroot-native=   -DNO_AVX512 -DHAVE_C11 -Wall -m64 -fPIC -DSMALL_MATRIX_OPT -DDYNAMIC_ARCH -DNO_AVX512 -DNO_WARMUP -DMAX_CPU_NUMBER=0 -DMAX_PARALLEL_NUMBER=1 -DMAX_STACK_ALLOC=2048 -DVERSION="\"0.3.27\"" -DBUILD_KERNEL -DTABLE_NAME=gotoblas_PRESCOTT  -DTS=_PRESCOTT -msse -msse2 -msse3 -mssse3 -msse4.1 -MD -MT kernel/CMakeFiles/kernel_PRESCOTT.dir/CMakeFiles/sgemm_oncopy_PRESCOTT.S.o -MF kernel/CMakeFiles/kernel_PRESCOTT.dir/CMakeFiles/sgemm_oncopy_PRESCOTT.S.o.d -o kernel/CMakeFiles/kernel_PRESCOTT.dir/CMakeFiles/sgemm_oncopy_PRESCOTT.S.o -c /home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/build/kernel/CMakeFiles/sgemm_oncopy_PRESCOTT.S
| /home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git/kernel/x86_64/gemm_ncopy_4.S: Assembler messages:
| /home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git/kernel/x86_64/gemm_ncopy_4.S:192: Error: invalid operands (*UND* and *ABS* sections) for `*'
| /home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git/kernel/x86_64/gemm_ncopy_4.S:193: Error: invalid operands (*UND* and *ABS* sections) for `*'
| /home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git/kernel/x86_64/gemm_ncopy_4.S:194: Error: invalid operands (*UND* and *ABS* sections) for `*'
| /home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git/kernel/x86_64/gemm_ncopy_4.S:195: Error: invalid operands (*UND* and *ABS* sections) for `*'
| /home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git/kernel/x86_64/gemm_ncopy_4.S:197: Error: invalid operands (*UND* and *ABS* sections) for `*'
| /home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git/kernel/x86_64/gemm_ncopy_4.S:345: Error: invalid operands (*UND* and *ABS* sections) for `*'
| /home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git/kernel/x86_64/gemm_ncopy_4.S:346: Error: invalid operands (*UND* and *ABS* sections) for `*'
| /home/zozo/test-yocto-5.0/tmp-sicom-glibc/work/corei7-64-oe-linux/openblas/0.3.27/git/kernel/x86_64/gemm_ncopy_4.S:348: Error: invalid operands (*UND* and *ABS* sections) for `*'

Interesting, I do not see this in my builds, and I do not think this has ever come up before. PREFETCHSIZE should only be relevant when PREFETCH is defined, so maybe there is something going wrong with the preprocessing... (Not that I think it would necessarily be wrong to apply P4 settings to the equally venerable Prescott, I'd just like to understand how this crops up after so many years - the Prescott kernels were already in the original GotoBLAS of 15+ years ago)

(Not that I think it would necessarily be wrong to apply P4 settings to the equally venerable Prescott, ...

According to wikipedia:

The Prescott was a major architectural revision. Later revisions were the first to feature Intel's x86-64 architecture, enhanced branch prediction and trace cache, and eventually support was added for the NX (No eXecute) bit to implement executable-space protection.

So it's possible that Prescott should be omitted from the supported set of x86 CPUs for x86-64 as not all of them have the AMD64 instruction set. That would probably fix my build error, too.

I'm not convinced that two different products named Prescott would have been released with the same CPUID. And as there isn't even any actual hardware involved in your case, it is probably just your cross-compiler that is making wrong assumptions (and/or speedrunning through all the preprocessor directives that guard instances of PREFETCHSIZE in the assembly code)

Interestingly, the original 0.3.25 recipe from meta-scikit-learn did cross-compile successfully, albeit using the stock Makefile. Updating the recipe to 0.3.27 failed the same way for both make and cmake.

closing as superseded by #4650


Support enter/exit transitions

Is your feature request related to a problem?
Qwik doesn't have any first class support for transitioning elements when they are created or destroyed.
Animating a component into view, or between states, can be done using an eager useClientEffect$.
The difficulty is when an element is destroyed; it's not possible to keep the element in the DOM until the animation is complete.
Describe the solution you'd like
SolidJS allow CSS animations with a Transition element:
<Transition name="slide-fade">
  {show() && (
    <div>
      Lorem ipsum dolor sit amet, consectetur adipiscing elit. Mauris
      facilisis enim libero, at lacinia diam fermentum id. Pellentesque
      habitant morbi tristique senectus et netus.
    </div>
  )}
</Transition>

Svelte has a nice API using html attributes:
{#if visible}
  <p transition:fly="{{ y: 200, duration: 2000 }}">
    Flies in and out
  </p>
{/if}

Describe alternatives you've considered
To animate out an element:

I've tried returning an async function from a useWatch and useClientEffect but the element will still be destroyed immediately.
I was also unable to duplicate the element to inject it back into the dom and manually destory it after the animation completes. But even if I could get this working, an issue would occur when you could have multiple copies of the element if it were to toggle fast enough.

Additional context
No response

This is not specific to qwik but in Chromium based browser you can use the view transition API to achieve that :
https://developer.mozilla.org/en-US/docs/Web/API/View_Transitions_API

Related https://github.com/BuilderIO/qwik/issues/3664 and https://github.com/BuilderIO/qwik/issues/5032

I'd like to merge this into #5032 since we're leaning towards only using the view transitions API, and that's what Qwik UI is using too.


receiver-mock: Add print_metrics parameter

Really a nitpick: previous commits regarding receiver mock in this repo have [receiver-mock] prefix, not receiver-mock:.
Do we change this in this PR or moving forward? :)

We need to stick one. Need to add CONTRIBUTORS.md 😅


Upgrade Kafka dependencies to 2.2.0

Fixes #893

Waiting for https://github.com/charithe/kafka-junit/pull/42 to be merged to update snapshot dependency.


[CreateFullPage]: Header height issues on some older browsers

Package
Carbon for IBM Products
Description
The CreateFullPage header occupies half of the vertical space because minmax(auto) isn’t supported in some browsers

Tested on Firefox ESR 115
Component(s) impacted
CreateFullPage
Browser
Firefox
@carbon/ibm-products (previously @carbon/ibm-cloud-cognitive) version
v2.43.1
Suggested Severity
None
Product/offering
DPM
CodeSandbox or Stackblitz example
https://ibm-products.carbondesignsystem.com/?path=/story/ibm-products-patterns-create-flows-createfullpage--create-full-page-with-header&globals=viewport:basic
Steps to reproduce the issue (if applicable)
Download Firefox ESR 115 and visit this Storybook example
Release date (if applicable)
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] I checked the current issues for duplicate issues

I have a fix ready for this issue and will be creating a pull request for it shortly.


deny_nonsecure_transport grants read-write access to all principals

Description
The policy generated by deny_nonsecure_transport grants access to all AWS principals. This makes the use of IAM to control access to the filesystem impossible when this boolean is set, and is an extremely significant side-effect of the boolean (in contrast to having it set to false and using policy_statements) that is not clear in documentation.
The problematic policy was added in https://github.com/terraform-aws-modules/terraform-aws-efs/pull/21, in an attempt to fix #20 and #11. In particular, I think the policy given in #20 is the incorrect policy to fix the ECS issue because it only works because it grants access to all principals -- which is far too broad of a policy, and probably not the intention.
#20 mentions that the web console generated the policy. However, I believe the policies generated by the web console are intended to be used where IAM is not used to control access to the filesystem: all of them generate a similar policy granting access to all principals, with specific denies; this is because the 'default' EFS policy is to allow access to all principals, and use firewall rules to control access.
I think this module should support using IAM to selectively control access to the EFS filesystem, instead of firewall rules alone. At the very least, it should be made more explicit that deny_nonsecure_transport precludes the use of IAM. I would suggest creating a new boolean that makes it very explicit as to whether a 'allow all principals' policy will be attached; then, this could be set to false to facilitate the use of IAM to control access.

[x] ✋ I have searched the open/closed issues and my issue is not listed.

Versions

Module version [Required]: v1.6.2

Terraform version: v1.6.3

Provider version(s): provider registry.terraform.io/hashicorp/aws v5.40.0

Reproduction Code [Required]
module "efs" {
  source = "terraform-aws-modules/efs/aws"

  # File system
  name           = "example"
  creation_token = "example-token"
  encrypted      = true
  kms_key_arn    = "arn:aws:kms:eu-west-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"

  lifecycle_policy = {
    transition_to_ia = "AFTER_30_DAYS"
  }

  # File system policy
  attach_policy                      = true
  bypass_policy_lockout_safety_check = false
  deny_nonsecure_transport = true
  policy_statements = [
  # XXX: This policy has no effect, because of the 'grant all' policy added by deny_nonsecure_transport!
    {
      sid     = "Example"
      actions = ["elasticfilesystem:ClientMount"]
      principals = [
        {
          type        = "AWS"
          identifiers = ["arn:aws:iam::111122223333:role/EfsReadOnly"]
        }
      ]
    }
  ]

  # Mount targets / security group
  mount_targets = {
    "eu-west-1a" = {
      subnet_id = "subnet-abcde012"
    }
    "eu-west-1b" = {
      subnet_id = "subnet-bcde012a"
    }
    "eu-west-1c" = {
      subnet_id = "subnet-fghi345a"
    }
  }
  security_group_description = "Example EFS security group"
  security_group_vpc_id      = "vpc-1234556abcdef"
  security_group_rules = {
    vpc = {
      # relying on the defaults provdied for EFS/NFS (2049/TCP + ingress)
      description = "NFS ingress from VPC private subnets"
      cidr_blocks = ["<IP_ADDRESS>/24", "<IP_ADDRESS>/24", "<IP_ADDRESS>/24"]
    }
  }
}

Steps to reproduce the behavior:

Create an EFS filesystem using the module that:

Has deny_nonsecure_transport set to true
Uses policy_statements to attempt to grant some form of access to a specific IAM principal

Attempt to mount the EFS filesystem with an IAM principal that you did not grant explicit access to
Notice that the EFS filesystem is mounted successfully

Expected behavior
The EFS filesystem should not be able to be mounted by any IAM principal when deny_nonsecure_transport is true
Actual behavior
The EFS filesystem is able to be mounted by any IAM principal when deny_nonsecure_transport is true, regardless of any allows in policy_statements.

This could be related to #11 / #20 which was fixed in #21 -- the fix in #21 contains the following line:
        variable = "elasticfilesystem:AccessedViaMountTarget"

https://github.com/terraform-aws-modules/terraform-aws-efs/pull/21/files#diff-dc46acf24afd63ef8c556b77c126ccc6e578bc87e3aa09a931f33d9bf2532fbbR124
but when you look at the article cited in #21's description: https://docs.aws.amazon.com/whitepapers/latest/efs-encrypted-file-systems/encryption-of-data-in-transit.html
It suggests checking aws:SecureTransport. Perhaps just user error from @antonbabenko ?

Not stale

Still not stale

I hit this issue when setting the least privileged access to a new EFS file system. I was following this EFS guide and using deny_nonsecure_transport to leave that secure transport policy to the module and trying to add the rest of secure policies, but, as mentioned in this issue, deny_nonsecure_transport adds too broad permissions. I will try without deny_nonsecure_transport, but it's less ideal to repeat the permissions logic that this module already has.

Not stale: issue still exists as far as I can tell


Trivial UI Fixes

The current image upload descriptive text is not accurate and was probably duplicated from another area. Images can be of various sizes as they will be resized accordingly provided they are in either .png or .jpg format.
https://bugzilla.redhat.com/show_bug.cgi?id=1354026
In addition the current text to remove Catalog Items, refers to the VMDB which is unnecessary and confusing.

[RFE] Label for removing catalog item is confusing
https://bugzilla.redhat.com/show_bug.cgi?id=1354044


Request: bracketSpan as a child of section

Brackets are used in modern transcriptions to indicate where a ligature was placed in the original mensural piece. These brackets can be indicated in CMN using the <bracketSpan> element as a child of <measure>.
When scoring up a mensural piece—this is, aligning the voices in score layout, but preserving the original mensural notation values—it is desirable to divide ligatures into the individual notes to better visualize the alignment of the notes in the different voices. So, the use of the <bracketSpan> would also be desired here, to show which individual notes were part of a ligature in the original source. But, because we are still in mensural notation, there are no <measure> elements to contain these <bracketSpan> elements. It would be useful for <bracketSpan> to be allowed as a child of <section> (at least for mensural notation), so that we can visualize these brackets in pieces encoded in mensural notation (with no measures) where ligatures have been separated into their individual component notes (for facilitating the visualization of the alignment of the voices and the vertical sonorities).

Would it work for you to use MEI-all? In general its used should be discouraged, but for transitional statuses that clearly mix notation types, I think it makes sense to use it.

I am editing the customization file for mensural notation to solve this problem. So that <bracketSpan> is available as a child of <section> just for mensural notation. It should work as usual (child of <measure>) if we don't use the mensural customization.

Can be closed because solved differently. See #771


Update restAddProfileByYaml response code

Fix #1670
Update the restAddProfileByYaml REST handler to return a '409(Conflict)'
when unable to unmarshal the request body. The reason for this change is
backwards compatibility with EdgeX 1.x. Disregarding backwards
compatibility, the correct response code is '400(Bad Request)'.
Updated unit tests to adhere to the new expectations of the API.
Signed-off-by: Anthony M. Bonafide<EMAIL_ADDRESS>

Hmmm, I re-ran them locally and they seem to have passed. Am I looking at the wrong thing?

@cherrycl could you execute the blackbox tests against these changes to ensure the bug is resolved?

Codecov Report

Merging #1678 into master will increase coverage by 0.04%.
The diff coverage is 84.21%.

@@            Coverage Diff             @@
##           master    #1678      +/-   ##
==========================================
+ Coverage   34.67%   34.71%   +0.04%
==========================================
  Files         139      139
  Lines       11770    11787      +17
==========================================
+ Hits         4081     4092      +11
- Misses       7488     7494       +6
  Partials      201      201

Impacted Files
Coverage Δ

internal/core/metadata/rest_deviceprofile.go
87.88% <84.21%> (-1.17%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 82597d1...eaf8a81. Read the comment docs.


Counter-example for typing DataField#initial as (source: DataModel.Any) => InitializedType

So far I've only identified one place that a DataField#initial is called with an argument and that is in EmbeddedCollectionField#_cleanType:
return value.map(v => this.schema.clean(v, {...options, source: v}));

this.schema is a SchemaField so this calls SchemaField#clean (no specific override here) so it goes to DataField#clean which has this line:
if ( value === undefined ) return this.getInitialValue(options.source);

and getInitialValue is:
  getInitialValue(data) {
    return this.initial instanceof Function ? this.initial(data) : this.initial;
  }

Therefore DataField#initial is called with source: v which will be some Document. I've typed DataField#initial based off of this but there may be counterexamples out there.

Did regex for clean.*source

EmbeddedDataField passes value, which would be the source data for the embedded model I believe?
EmbeddedCollectionDeltaField is configured similarly to EmbeddedCollectionField but sometimes deferring to TombstoneData rather than the model
TypeDataField similarly calls cls.cleanData(value, {...options, source: value});

I think though in each of these cases we're dealing with the json-serializable _source data rather than the model instance itself.

Update from discord - looks like it's just AnyObject getting passed through. Only example of actual function-based usage was sight.enabled in Token


Fix Injective Swap / update endpoints

Grpc and rest endpoints throw 503 errors.

Steps to Reproduce

Navigate to injective/swap page and insert value
Stuck on loading

Additional context
Add any other context about the problem here, code snippets, json responses and so on.

Screen shots

Logs

Hello @nick134-bit, could you check again, and share the last result with us.

Hey @KaliaNetwork, looks like this has been resolved.


[App]: Cannot open files in the UI for the runs created with a group specified

Current Behavior
If I create the run with the group specified (run = wandb.init(project='my-awesome-project', config=model_params, group="my-run-group")), then I cannot open any file in the "Files" section in the UI.
If my run doesn't belong to any group, the files open just fine.
I see the same incorrect behaviour on both W&B cloud and when running W&B locally using wandb/local:latest docker image.
Expected Behavior
I expect to be able to explore the run files regardless of whether a group was specified or not.
Steps To Reproduce

Create a run with the group run = wandb.init(project='my-awesome-project', config=model_params, group="my-run-group")
Go to wandb.ai
Navigate to Projects / my-awesome-project / Runs /
Open "Files" tab
Click on any file

Screenshots
Trying to open conda-environment.yaml for the run with the group:

Trying to open conda-environment.yaml for the run without a group (note the URL difference with the first screenshot):

Environment
OS: Ubuntu 20.04.5 LTS
Browsers: Chrome Version 104.0.5112.101 (Official Build) (64-bit)
Version:
Additional Context
No response

Hi @dmder, thanks for reporting this and for providing us with all the details! I've checked this and have been able to reproduce this. It seems the URL is changing and the way to get the file is a bit different so this is maybe what's causing the issue.  I've escalated this internally but, in the meantime, you can download files from the API like:
import wandb

api = wandb.Api()
run = api.run('luis_bergua/files_group/jps7x6et')
for f in run.files():
  f.download()

Please let me know if this is helpful!

Hi @dmder, we've been investigating this and it seems if you refresh the page the file is properly shown. Could you please check if you get the same behavior?

WandB Internal User commented:
luisbergua commented:
Hi @dmder, thanks for reporting this and for providing us with all the details! I've checked this and have been able to reproduce this. It seems the URL is changing and the way to get the file is a bit different so this is maybe what's causing the issue.  I've escalated this internally but, in the meantime, you can download files from the API like:
import wandb

api = wandb.Api()
run = api.run('luis_bergua/files_group/jps7x6et')
for f in run.files():
  f.download()

Please let me know if this is helpful!


Language field in the metadata exported incorrectly

Support log ID: NQSVQTXA-refs-euc
Betterbibtex for Zotero 5.4.29
Zotero <IP_ADDRESS>
Linux Mint 20.2 Cinnamon
In Zotero, I keep language fields for works in German as follows:
de for German
de-AT for Austrian
de-CH for Swissgerman
de-At is for German spoken and written in Austria, and de-CH is for German spoken and written in Switzerland.
In the exported biblatex file, I get in the language fields German and Austrian accordingly. They are correct. For de-CH, I get in langid field German instead of Swissgerman. Shouldn't I get Swissgerman here?
Here is the output with Betterbiblatex.

@book{kollerSchweizerischesObligationenrechtBesonderer2012,
title = {Schweizerisches Obligationenrecht Besonderer Teil Band I},
shorttitle = {OR BT I},
author = {Koller, Alfred},
date = {2012},
publisher = {{Stämpfli Verlag}},
location = {{Bern}},
langid = {german}
}

Here is the CSL json format from Zotero:
[
	{
		"id": "kollerSchweizerischesObligationenrechtBesonderer2012",
		"type": "book",
		"event-place": "Bern",
		"language": "de-CH",
		"publisher": "Stämpfli Verlag",
		"publisher-place": "Bern",
		"title": "Schweizerisches Obligationenrecht Besonderer Teil Band I",
		"title-short": "OR BT I",
		"author": [
			{
				"family": "Koller",
				"given": "Alfred"
			}
		],
		"issued": {
			"date-parts": [
				[
					"2012"
				]
			]
		}
	}
]

I currently have a fixed (and rather limited) mapping; I'm looking at this or this to re-use existing effort to get a complete mapping, but both of those have German (Switzerland) rather than Swissgerman for de-CH. Do you know of a maintained mapping that gets swissgerman?

this has swissgerman but does not understand de-CH.

I couldn't find one.
I also checked some other converters, e.g. pandoc. I thought, their mapping might be of help if there is one. But it did not work out.

Check out the biblatex manual as well as IETF BCP47.
The babel and polyglossia manuals might be useful, too. Note that both babel and polyglossia have begun to support BCP47 tags in addition to the traditional ones, but biblatex has not yet (it’s being discussed, though).
Hence my suggestion for a sensible mapping from BCP47 to currently used biblatex tags:

BCP47
biblatex

de
ngerman

de-DE
ngerman

de-AT
naustrian

de-CH
nswissgerman

de-1901
german

de-DE-1901
german

de-AT-1901
austrian

de-CH-1901
swissgerman

babel equates de-CD to German though, and polyglossia to german variant=swiss, spelling=new.

(1) If you're referring to the babel manual, p.21 (extract below), I've seen that, too, but though I’m not sure what this table really is about, I’m sure it can't be the mapping we're talking about. In particular, it also indiscriminately maps (?) all variants of English to "English" – which definitely is not what should be happening when mapping from BCP47 to babel or polyglossia.

I still maintain that both nswissgerman and swissgerman are mentioned in the babel manual as well as the biblatex manual, and both work when using \usepackage[nswissgerman]{babel} or \usepackage[nswissgerman]{babel} in a latex document, so it is these identifiers that should be used when mapping from de-CH or de-CH-1901.
(2) From the polyglossia manual: "For convenience reasons, polyglossia also supports the use of babel names"

Interesting -- yep, I missed that, but I'm now looking at the babel source, and there I find for babel-de-CH.ini:
name.babel.A = german-ch
name.babel.B = nswissgerman
; The following conflicts with gsw:
name.babel.C = swissgerman

so on the upside, this looks to be a machine-readable mapping for language codes I could use, but also that there are multiple names, and assuming that .A means preferred, or default, de-CH  would map to german-ch rather than nswissgerman.

Booh, no nl-BE or nl-rBE.

Ok, but first and foremost, this is about biblatex. Its manual, p. 29, lists, under "Supported Languages":

So these, I take it, are the values that should be used in the langid field.
The biblatex manual also says (p. 28), also on the langid field: "The identifier must be a language name known to the babel/polyglossia packages."
Now all of these (german, austrian, swissgerman, ngerman, naustrian, nswissgerman) are known to babel and polyglossia.
german-ch OTOH, whatever it is supposed to be used for, does not seem to be a user-facing language identifier: Using \usepackage[german-ch]{babel} in a latex document bombs with ! Package babel Error: Unknown option 'german-ch'.

Cool; I'll use the biblatex list to establish preferred alternatives, and will otherwise just pick the name.babel from the babel ini files.

@ah-dagdelen can you verify build 1556 does what you expect?

@retorquere thanks a alot.
I can verify that I get nswissgerman for de-CH tag now. For de and de-At, I get German and Austrian respectively.
I agree with @njbart. Using n versions of the languages seems like a better idea here. n versions refer to the new orthography (Neue Rechtschreibung) while the others refer to the old orthography (Alte Rechtschreibung). The difference may cause some issues regarding word hyphenation.

So in other words, satisfactory? If you have samples of all of those, a right-click-support-log would give me samples to add to my test suite.

Yes satisfactory, the problem is solved for de-CH. Thanks for solving this issue.
Support log: CFWE9TLS-refs-euc

@njbart what does bibtex expect/support in the language field?

Unlike biblatex, bibtex does not seem to officially support either language or langid fields. I don’t expect these fields to cause any problems if included in a bibtex file (and I’ve actually seen bibtex files with a language field in the wild), but I’m not sure whether they’d serve any useful purpose in that format either.

So I might as well leave it out for bibtex then? What are the casing rules for bibtex? Everything title-cased?

So I might as well leave it out for bibtex then?

I guess so. (Unless users complain, of course.)

What are the casing rules for bibtex? Everything title-cased?

The rules for bibtex and biblatex are essentially the same, so, yes, title-cased.
(Still, there are some “small differences between bibtex the program and the btparse library used by biber” we’ve encountered before.)

What are the casing rules for bibtex? Everything title-cased?

The rules for bibtex and biblatex are essentially the same, so, yes, title-cased.

The reason I ask is that in biblatex, I look at the language information to determine whether or not to apply title-casing (to be specific, title-casing for Engl-ish, and no title casing for all others). So for bibtex I should also look at the language info I have? Or just treat everything as Engl-ish and title-case titles?

As I said some time ago: “For bibtex, which does not have a langid field and thus cannot distinguish languages, I would guess that the complete title fields of non-English titles should be wrapped in braces to prevent bibtex from messing with capitalisation.“ I have no reason to assume anything has changed WRT bibtex since then.

Ah right -- non-English is treated special here, rather than biblatex, where English is treated special.

Ah right -- non-English is treated special here, rather than biblatex, where English is treated special.

I don’t think that’s the case. The default, IIRC, is in both cases to sentence-case titles if requested to do so by a style; it’s just that bibtex does not have anything similar to biblatex’s langid field to prevent this from happening, so the only solution is to wrap the whole title in braces.
In addition:
Maybe including langid fields in bibtex files might still clarify why titles were enclosed in braces upon export from BBT, as might some kind of comment in a quality report.
A langid field in a bibtex file would also be picked up by pandoc’s citeproc (that’s about the only case where I’m positive a tool designed for bibtex would know about langid).

... which is probably why I have code that does just all of this ...
Sorry man, it's been a long month.


[ES] New tests

HI:
@bhavin-qryptal @SchulzeStTSI @daniel-eder  could you help me to understand what test are wrong and why? I have seen some errors, and I have remove that test from pull request, but is still block, and I don't know why.

HI:
@bhavin-qryptal @SchulzeStTSI @daniel-eder could you help me to understand what test are wrong and why? I have seen some errors, and I have remove that test from pull request, but is still block, and I don't know why.

Reference
Primarily two tests have failed.

test_cose_schema        : 227 Failures
test_expected_key_usage  : 185 Failures

Reasons:
Schema Failures:
2021-05-25T18:42:52.6364643Z _______________ test_cose_schema[NL/2DCode/raw/010-NL-test.json] _______________
2021-05-25T18:42:52.6365399Z tests/test_ehealth_certs.py:273: in test_cose_schema
2021-05-25T18:42:52.6366104Z     schema_validate(hcert, _get_hcert_schema())
2021-05-25T18:42:52.6367372Z /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/jsonschema/validators.py:934: in validate
2021-05-25T18:42:52.6368252Z     raise error
2021-05-25T18:42:52.6369412Z E   jsonschema.exceptions.ValidationError: '1963' does not match '(19|20)\\d{2}-\\d{2}-\\d{2}'
2021-05-25T18:42:52.6370303Z E
2021-05-25T18:42:52.6371048Z E   Failed validating 'pattern' in schema['properties']['dob']:
2021-05-25T18:42:52.6372095Z E       {'description': 'Date of Birth of the person addressed in the DGC. ISO '
2021-05-25T18:42:52.6373072Z E                       '8601 date format restricted to range 1900-2099',
2021-05-25T18:42:52.6373781Z E        'examples': ['1979-04-14'],
2021-05-25T18:42:52.6374385Z E        'format': 'date',
2021-05-25T18:42:52.6375015Z E        'pattern': '(19|20)\\d{2}-\\d{2}-\\d{2}',
2021-05-25T18:42:52.6375658Z E        'title': 'Date of birth',
2021-05-25T18:42:52.6376252Z E        'type': 'string'}
2021-05-25T18:42:52.6376648Z E
2021-05-25T18:42:52.6377484Z E   On instance['dob']:
2021-05-25T18:42:52.6378043Z E       '1963'
2021-05-25T18:42:52.6378824Z _____________ test_cose_schema[NL/2DCode/raw/108-NL-recovery.json] _____________
2021-05-25T18:42:52.6379604Z tests/test_ehealth_certs.py:273: in test_cose_schema
2021-05-25T18:42:52.6380406Z     schema_validate(hcert, _get_hcert_schema())
2021-05-25T18:42:52.6381528Z /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/jsonschema/validators.py:934: in validate
2021-05-25T18:42:52.6382336Z     raise error
2021-05-25T18:42:52.6383937Z E   jsonschema.exceptions.ValidationError: '1963-00' does not match '(19|20)\\d{2}-\\d{2}-\\d{2}'
2021-05-25T18:42:52.6384837Z E
2021-05-25T18:42:52.6385585Z E   Failed validating 'pattern' in schema['properties']['dob']:
2021-05-25T18:42:52.6386631Z E       {'description': 'Date of Birth of the person addressed in the DGC. ISO '
2021-05-25T18:42:52.6387594Z E                       '8601 date format restricted to range 1900-2099',
2021-05-25T18:42:52.6388347Z E        'examples': ['1979-04-14'],
2021-05-25T18:42:52.6389051Z E        'format': 'date',
2021-05-25T18:42:52.6389701Z E        'pattern': '(19|20)\\d{2}-\\d{2}-\\d{2}',
2021-05-25T18:42:52.6390346Z E        'title': 'Date of birth',
2021-05-25T18:42:52.6390938Z E        'type': 'string'}
2021-05-25T18:42:52.6391336Z E
2021-05-25T18:42:52.6391849Z E   On instance['dob']:
2021-05-25T18:42:52.6392393Z E       '1963-00'
2021-05-25T18:42:52.6393200Z ___________ test_cose_schema[NL/2DCode/raw/049-NL-vaccination.json] ____________
2021-05-25T18:42:52.6393997Z tests/test_ehealth_certs.py:273: in test_cose_schema
2021-05-25T18:42:52.6394685Z     schema_validate(hcert, _get_hcert_schema())
2021-05-25T18:42:52.6395889Z /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/jsonschema/validators.py:934: in validate
2021-05-25T18:42:52.6396754Z     raise error
2021-05-25T18:42:52.6397889Z E   jsonschema.exceptions.ValidationError: '1963' does not match '(19|20)\\d{2}-\\d{2}-\\d{2}'
2021-05-25T18:42:52.6398769Z E
2021-05-25T18:42:52.6399512Z E   Failed validating 'pattern' in schema['properties']['dob']:
2021-05-25T18:42:52.6400551Z E       {'description': 'Date of Birth of the person addressed in the DGC. ISO '
2021-05-25T18:42:52.6401566Z E                       '8601 date format restricted to range 1900-2099',
2021-05-25T18:42:52.6402240Z E        'examples': ['1979-04-14'],
2021-05-25T18:42:52.6402797Z E        'format': 'date',
2021-05-25T18:42:52.6403377Z E        'pattern': '(19|20)\\d{2}-\\d{2}-\\d{2}',
2021-05-25T18:42:52.6403972Z E        'title': 'Date of birth',
2021-05-25T18:42:52.6404520Z E        'type': 'string'}
2021-05-25T18:42:52.6405072Z E
2021-05-25T18:42:52.6405583Z E   On instance['dob']:
2021-05-25T18:42:52.6406119Z E       '1963'
2021-05-25T18:42:52.6406888Z _____________ test_cose_schema[NL/2DCode/raw/116-NL-recovery.json] _____________
2021-05-25T18:42:52.6407658Z tests/test_ehealth_certs.py:273: in test_cose_schema
2021-05-25T18:42:52.6408335Z     schema_validate(hcert, _get_hcert_schema())
2021-05-25T18:42:52.6409556Z /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/jsonschema/validators.py:934: in validate
2021-05-25T18:42:52.6410426Z     raise error
2021-05-25T18:42:52.6411654Z E   jsonschema.exceptions.ValidationError: '1963' does not match '(19|20)\\d{2}-\\d{2}-\\d{2}'
2021-05-25T18:42:52.6412478Z E
2021-05-25T18:42:52.6413164Z E   Failed validating 'pattern' in schema['properties']['dob']:
2021-05-25T18:42:52.6414343Z E       {'description': 'Date of Birth of the person addressed in the DGC. ISO '
2021-05-25T18:42:52.6415403Z E                       '8601 date format restricted to range 1900-2099',
2021-05-25T18:42:52.6416077Z E        'examples': ['1979-04-14'],
2021-05-25T18:42:52.6416631Z E        'format': 'date',
2021-05-25T18:42:52.6417329Z E        'pattern': '(19|20)\\d{2}-\\d{2}-\\d{2}',
2021-05-25T18:42:52.6417933Z E        'title': 'Date of birth',
2021-05-25T18:42:52.6418484Z E        'type': 'string'}
2021-05-25T18:42:52.6418856Z E
2021-05-25T18:42:52.6419334Z E   On instance['dob']:
2021-05-25T18:42:52.6420045Z E       '1963'
2021-05-25T18:42:52.6420784Z _______________ test_cose_schema[NL/2DCode/raw/018-NL-test.json] _______________
2021-05-25T18:42:52.6421522Z tests/test_ehealth_certs.py:273: in test_cose_schema
2021-05-25T18:42:52.6422199Z     schema_validate(hcert, _get_hcert_schema())
2021-05-25T18:42:52.6423421Z /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/jsonschema/validators.py:934: in validate
2021-05-25T18:42:52.6424400Z     raise error
2021-05-25T18:42:52.6425541Z E   jsonschema.exceptions.ValidationError: '1963' does not match '(19|20)\\d{2}-\\d{2}-\\d{2}'
2021-05-25T18:42:52.6426432Z E
2021-05-25T18:42:52.6427172Z E   Failed validating 'pattern' in schema['properties']['dob']:
2021-05-25T18:42:52.6428218Z E       {'description': 'Date of Birth of the person addressed in the DGC. ISO '
2021-05-25T18:42:52.6429173Z E                       '8601 date format restricted to range 1900-2099',
2021-05-25T18:42:52.6429900Z E        'examples': ['1979-04-14'],
2021-05-25T18:42:52.6430560Z E        'format': 'date',
2021-05-25T18:42:52.6431215Z E        'pattern': '(19|20)\\d{2}-\\d{2}-\\d{2}',
2021-05-25T18:42:52.6431854Z E        'title': 'Date of birth',
2021-05-25T18:42:52.6432443Z E        'type': 'string'}
2021-05-25T18:42:52.6432839Z E
2021-05-25T18:42:52.6433348Z E   On instance['dob']:
2021-05-25T18:42:52.6433881Z E       '1963'
2021-05-25T18:42:52.6434797Z _______ test_cose_schema[NL/2DCode/raw/258-NL-recovery+vaccination.json] _______
2021-05-25T18:42:52.6435715Z tests/test_ehealth_certs.py:273: in test_cose_schema
2021-05-25T18:42:52.6436386Z     schema_validate(hcert, _get_hcert_schema())
2021-05-25T18:42:52.6437598Z /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/jsonschema/validators.py:934: in validate
2021-05-25T18:42:52.6438451Z     raise error
2021-05-25T18:42:52.6439556Z E   jsonschema.exceptions.ValidationError: '' does not match '[A-Z]{1,10}'
2021-05-25T18:42:52.6440516Z E
2021-05-25T18:42:52.6441376Z E   Failed validating 'pattern' in schema['properties']['r']['items']['properties']['co']:
2021-05-25T18:42:52.6442267Z E       {'description': 'Country of Vaccination / Test, ISO 3166 where '
2021-05-25T18:42:52.6442895Z E                       'possible',
2021-05-25T18:42:52.6443398Z E        'pattern': '[A-Z]{1,10}',
2021-05-25T18:42:52.6443867Z E        'type': 'string'}
2021-05-25T18:42:52.6444186Z E
2021-05-25T18:42:52.6444774Z E   On instance['r'][0]['co']:
2021-05-25T18:42:52.6445272Z E       ''
2021-05-25T18:42:52.6446031Z __________ test_cose_schema[NL/2DCode/raw/201-NL-test+recovery.json] ___________
2021-05-25T18:42:52.6446793Z tests/test_ehealth_certs.py:273: in test_cose_schema
2021-05-25T18:42:52.6447433Z     schema_validate(hcert, _get_hcert_schema())
2021-05-25T18:42:52.6448544Z /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/jsonschema/validators.py:934: in validate
2021-05-25T18:42:52.6449345Z     raise error
2021-05-25T18:42:52.6450359Z E   jsonschema.exceptions.ValidationError: '' does not match '[A-Z]{1,10}'
2021-05-25T18:42:52.6451162Z E
2021-05-25T18:42:52.6452157Z E   Failed validating 'pattern' in schema['properties']['t']['items']['properties']['co']:
2021-05-25T18:42:52.6453256Z E       {'description': 'Country of Vaccination / Test, ISO 3166 where '
2021-05-25T18:42:52.6454048Z E                       'possible',
2021-05-25T18:42:52.6454658Z E        'pattern': '[A-Z]{1,10}',
2021-05-25T18:42:52.6455251Z E        'type': 'string'}
2021-05-25T18:42:52.6455633Z E
2021-05-25T18:42:52.6456175Z E   On instance['t'][0]['co']:
2021-05-25T18:42:52.6456689Z E       ''
2021-05-25T18:42:52.6457612Z ___________ test_cose_schema[NL/2DCode/raw/078-NL-vaccination.json] ____________
2021-05-25T18:42:52.6458396Z tests/test_ehealth_certs.py:273: in test_cose_schema
2021-05-25T18:42:52.6459089Z     schema_validate(hcert, _get_hcert_schema())
2021-05-25T18:42:52.6460288Z /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/jsonschema/validators.py:934: in validate
2021-05-25T18:42:52.6461156Z     raise error
2021-05-25T18:42:52.6462097Z E   jsonschema.exceptions.ValidationError: 0 is less than the minimum of 1
2021-05-25T18:42:52.6463076Z E
2021-05-25T18:42:52.6463884Z E   Failed validating 'minimum' in schema['properties']['v']['items']['properties']['sd']:
2021-05-25T18:42:52.6465124Z E       {'description': 'Dose Number / Total doses in Series: positive '
2021-05-25T18:42:52.6465966Z E                       'integer, range: [1,9]',
2021-05-25T18:42:52.6466661Z E        'maximum': 9,
2021-05-25T18:42:52.6467214Z E        'minimum': 1,
2021-05-25T18:42:52.6467779Z E        'type': 'integer'}
2021-05-25T18:42:52.6468186Z E
2021-05-25T18:42:52.6468717Z E   On instance['v'][1]['sd']:
2021-05-25T18:42:52.6469130Z E       0
2021-05-25T18:42:52.6470132Z ____ test_cose_schema[NL/2DCode/raw/288-NL-test+recovery+vaccination.json] _____
2021-05-25T18:42:52.6471104Z tests/test_ehealth_certs.py:273: in test_cose_schema
2021-05-25T18:42:52.6471799Z     schema_validate(hcert, _get_hcert_schema())
2021-05-25T18:42:52.6473068Z /opt/hostedtoolcache/Python/3.9.5/x64/lib/python3.9/site-packages/jsonschema/validators.py:934: in validate
2021-05-25T18:42:52.6473939Z     raise error
2021-05-25T18:42:52.6475032Z E   jsonschema.exceptions.ValidationError: '' does not match '[A-Z]{1,10}'
2021-05-25T18:42:52.6475898Z E
2021-05-25T18:42:52.6476752Z E   Failed validating 'pattern' in schema['properties']['t']['items']['properties']['co']:
2021-05-25T18:42:52.6477851Z E       {'description': 'Country of Vaccination / Test, ISO 3166 where '
2021-05-25T18:42:52.6478751Z E                       'possible',
2021-05-25T18:42:52.6479317Z E        'pattern': '[A-Z]{1,10}',
2021-05-25T18:42:52.6479870Z E        'type': 'string'}
2021-05-25T18:42:52.6480225Z E
2021-05-25T18:42:52.6480730Z E   On instance['t'][1]['co']:
2021-05-25T18:42:52.6481209Z E       ''
2021-05-25T18:42:52.6481903Z _______________ test_cose_schema[NL/2DCode/raw/025-NL-test.json] _______________

Key Usage Failures:
Reason: AssertionError: DGC adheres to schema but contains multiple certificates of different types. @SchulzeStTSI and @daniel-eder , Please confirm if this is a valid error OR not a valid error. As I have tried to confirm this here , but have not recd confirm response. Also, please let me know if test logic needs to be updated.

HI:
@bhavin-qryptal @SchulzeStTSI @daniel-eder could you help me to understand what test are wrong and why? I have seen some errors, and I have remove that test from pull request, but is still block, and I don't know why.

@Santivicen @SchulzeStTSI @daniel-eder
All these failures are related to NL files. We need to re-open ONE or BOTH of
https://github.com/eu-digital-green-certificates/dgc-testdata/issues/146
https://github.com/eu-digital-green-certificates/dgc-testdata/issues/107
And also re-introduce NL entry in known issue list.


Correct identifier lexing

Rust uses _ and XID_Start, XID_Continue to define identifiers. We need to add lexer test for some edge cases with these classes.

Is this still required now that this crate is using the rustc lexer?
(Sorry if all the 'is this done'/'this is done' comments on old issues are annoying - I'm looking through open issues to try and find something easy to work on.)

@theotherphil maybe you want to try https://github.com/rust-analyzer/rust-analyzer/issues/1592?

Thanks for the suggestion - I’ve started work on another couple of issues, but I’ll have a look at that one if it’s still open by the time I’m done with these.

yeah, this can be just closed now!


outlook.com calendar issue

I've tried all the variations of URLs for a public outlook.com calendar (webcal:// webcals:// http:// and https:// to the calendar.ics), but the mirror calendar only displays the status (Free, busy, tentative), not the summary text.
My sample event is as follows in the .ics file:
BEGIN:VEVENT
UID:5f4834ab-472a-4a40-a07c-6a94a091fd6a
CLASS:PUBLIC
X-MICROSOFT-CDO-BUSYSTATUS:BUSY
TRANSP:OPAQUE
SEQUENCE:0
CREATED:20160414T203323Z
LAST-MODIFIED:20160414T203323Z
DTSTART;TZID=Eastern Standard Time:20160415T160000
DTEND;TZID=Eastern Standard Time:20160415T170000
SUMMARY:Test Event
PRIORITY:0
BEGIN:VALARM
ACTION:DISPLAY
TRIGGER:-PT15M
END:VALARM
END:VEVENT
But the calendar ist on the mirror only shows events with a status other than "Free", and only shows the status, not the summary text. The above appears as "Busy      in 13 hours" rather than "Test Event    in 13 hours".

Could you supply a test url of a feed with this issue?

test.txt

I just tried it with the test.txt file (your .ics code) above, and I get the following.


"Error: Found an unshardable source dataset" when using Mirrored Strategy with train on batch

System information

Have I written custom code: Custom
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
TensorFlow installed from (source or binary): Tested from compiled source and binary
TensorFlow version (use command below): 2.4.0rc2
Python version: 3.9/3.8/3.7
Bazel version (if compiling from source): 3.7.0
GCC/Compiler version (if compiling from source): 9.3.0/7
CUDA/cuDNN version: 11.1/10.1
GPU model and memory: RTX 3090/ RTX 2080

Describe the current behavior
When using MirroredStrategy with Keras, it produces a lot of outputs (see below) as described in this issue #42146. For each batch, it keeps printing the same logs with the error Error: Found an unshardable source dataset
I tried to disable the tf.data.experimental.AutoShardPolicy.OFF on train_dataset or to set it to DATA, but it seems to not have any effect (probable because the for loop unpack the train dataset)
It works without any problem on TF 2.3.1
Describe the expected behavior
I expect to have the same behavior as on TF 2.3.1: no extensive printed log, no error. Have it works with the default shard policy.
Or at least, to find a way to apply the shard policy on the two subset train_images, train_labels
Standalone code to reproduce the issue
from tensorflow.keras import datasets, layers, models
import tensorflow as tf

options = tf.data.Options()
options.experimental_distribute.auto_shard_policy = tf.data.experimental.AutoShardPolicy.DATA
(train_images, train_labels), (test_images, test_labels) = datasets.cifar10.load_data()
train_images, test_images = train_images / 255.0, test_images / 255.0
train_dataset = tf.data.Dataset.from_tensor_slices((train_images, train_labels))
train_dataset = train_dataset.with_options(options)
train_dataset = train_dataset.batch(32)

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    model = models.Sequential()
    model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    model.add(layers.MaxPooling2D((2, 2)))
    model.add(layers.Conv2D(64, (3, 3), activation='relu'))
    model.add(layers.Flatten())
    model.add(layers.Dense(64, activation='relu'))
    model.add(layers.Dense(10))

    model.compile(optimizer='adam',
                  loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
                  metrics=['accuracy'])

for train_images, train_labels in train_dataset:
    history = model.train_on_batch(train_images, train_labels)

Other info / logs
2020-11-24 17:17:49.116426: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-11-24 17:17:49.885169: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-11-24 17:17:49.885621: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2020-11-24 17:17:49.917150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1727] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 23.69GiB deviceMemoryBandwidth: 871.81GiB/s
2020-11-24 17:17:49.917174: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-11-24 17:17:49.918707: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2020-11-24 17:17:49.918727: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2020-11-24 17:17:49.919263: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2020-11-24 17:17:49.919382: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2020-11-24 17:17:49.920901: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2020-11-24 17:17:49.921285: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2020-11-24 17:17:49.921350: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2020-11-24 17:17:49.922684: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1869] Adding visible gpu devices: 0
2020-11-24 17:17:49.923076: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-11-24 17:17:49.924740: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-11-24 17:17:49.925430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1727] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 3090 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 82 deviceMemorySize: 23.69GiB deviceMemoryBandwidth: 871.81GiB/s
2020-11-24 17:17:49.925441: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-11-24 17:17:49.925448: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2020-11-24 17:17:49.925455: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2020-11-24 17:17:49.925462: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2020-11-24 17:17:49.925468: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2020-11-24 17:17:49.925474: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11
2020-11-24 17:17:49.925480: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2020-11-24 17:17:49.925486: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2020-11-24 17:17:49.926781: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1869] Adding visible gpu devices: 0
2020-11-24 17:17:49.926800: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-11-24 17:17:50.236484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-11-24 17:17:50.236514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1273]      0
2020-11-24 17:17:50.236519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1286] 0:   N
2020-11-24 17:17:50.238508: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1413] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22185 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3090, pci bus id: 0000:01:00.0, compute capability: 8.6)
2020-11-24 17:17:51.610137: W tensorflow/core/grappler/optimizers/data/auto_shard.cc:656] In AUTO-mode, and switching to DATA-based sharding, instead of FILE-based sharding as we cannot find appropriate reader dataset op(s) to shard. Error: Found an unshardable source dataset: name: "TensorDataset/_2"
op: "TensorDataset"
input: "Placeholder/_0"
input: "Placeholder/_1"
attr {
  key: "Toutput_types"
  value {
    list {
      type: DT_DOUBLE
      type: DT_UINT8
    }
  }
}
attr {
  key: "output_shapes"
  value {
    list {
      shape {
        dim {
          size: 32
        }
        dim {
          size: 32
        }
        dim {
          size: 32
        }
        dim {
          size: 3
        }
      }
      shape {
        dim {
          size: 32
        }
        dim {
          size: 1
        }
      }
    }
  }
}

Thank you!

Hi @houseofai, I was able to repro this and I see that the logs look different in 2.3 vs in nightly. I also was able to repro using model.fit instead of model.train_on_batch.
Note that #42146 was in reference to MultiWorkerMirroredStrategy, and the use of an autoshard policy is not necessary when using MirroredStrategy since you only have one machine. This can be a bit confusing (especially given the logs you are seeing), so in case you haven't already taken a look at the Distributed Input guide I am linking it here.

This is a warning, not an error. The warning is expected, as we cannot shard tensor based dataset by FILE.
You can pass the the dataset with options to model.fit() and the option should take effect.

Thanks @nikitamaia
Should I understand that now the default behavior with MirroredStrategy and train_on_batch is to print those verbose warnings all the way long the training?

Yes, the reason the options aren't working is that model.train_on_batch wraps the tensors in a dataset again, so the specified options are not taking effect. If you pass the dataset to model.fit with the options then you won't see the warning. That being said, since there is no autosharding when you're doing single worker training the autosharding options register as no-op, but the code is shared with MWMS so you see the warning regardless. I think ideally the verbose warnings shouldn't be there and in the future we can improve this, but for now the training is working as expected.

Thanks a lot @nikitamaia for the detailed answers

@crccw Why was this closed? As @nikitamaia explained using options to steer the sharding of a dataset passed to model.train_on_batch is not working due to some internals of that function. The proposed "solution" is to not use this function which is hardly a solution.
So may I rephrase the issue/bug report as "Dataset options ignored for dataset passed to model.train_on_batch" with OPs code  as a reproducer. And although sharding is not required for MirroredStrategy this is still a valid bug in TF 2.4: a) The option is clearly ignored and b) the same happens for MultiWorkerMirroredStrategy

Can't believe they close this problem, this is atrocious

Hi all, it seems that my explanation resulted in more confusion.
The solution was not to avoid train_on_batch, but instead to explain that train_on_batch will result in some warnings that you can ignore. Ideally, we would silence these warnings, but for now they are just warnings that don't actually affect the training process.
The sharding options set on the original dataset don't apply to train_on_batch because train_on_batch takes tensors as inputs and creates a new dataset internally. This is intended Keras behavior.
Because the options are not taking effect, you are unable to explicitly set the autoshardpolicy. Thus, the autoshardpolicy is set to AUTO as indicated in the docs:
AUTO: This is the default option which means an attempt will be made to shard by FILE.
The attempt to shard by FILE fails if a file-based dataset is not detected. tf.distribute will then fall back to sharding by DATA.

So first the policy tries to shard by FILE, but it cannot since train_on_batch takes tensors as input. So then the DATA policy is used.Error: Found an unshardable source dataset is indicating that the input could not be sharded by FILE, and will instead be sharded by DATA.
To summarize, it seems the main point of confusion here is the warning Error: Found an unshardable source dataset which in this case is telling us that with model.train_on_batch you cannot shard by FILE. However, the phrasing of the warning is confusing/misleading.
A fix in this case either means writing a better warning message, or from the Keras side setting the autoshardpolicy to DATA when train_on_batch is used. In the end the same effect occurs, but just without the warning message.
Hopefully this clears things up!

Hi all, please be aware that you also have this warring message not only at training time but also when you are just predicting and you want to leverage with MirroredStrategy to increase the batch size.


progresshook on fragment.py: Inconsistent behavior on total_bytes

In the documentation of progress_hooks, it's said that that if total_bytes is unknown, the value would be None:
    progress_hooks:    A list of functions that get called on download
                       progress, with a dictionary with the entries
                       * status: One of "downloading", "error", or "finished".
                                 Check this first and ignore unknown values.
                       If status is one of "downloading", or "finished", the
                       following properties may also be present:
...
                       * total_bytes: Size of the whole file, None if unknown

However this is not the case for fragment.py, where the key simply doesn't exist:
        # This dict stores the download progress, it's updated by the progress
        # hook
        state = {
            'status': 'downloading',
            'downloaded_bytes': 0,
            'frag_index': 0,
            'frag_count': total_frags,
            'filename': ctx['filename'],
            'tmpfilename': ctx['tmpfilename'],
        }

This has caused downstream issues such as KeyError: 'total_bytes'

the following properties may also be present:

None of these fields are guaranteed to be present at all. total_bytes is unknown beforehand for fragment based content. total_bytes_estimate may be available for non live streams.

It may be None for unknown values, as explained so in docs, or the documentation itself is not consistent with the behavior.

It may be present or may not. If present it may contain a value or None. Unless mentioned otherwise, None is equivalent to absence of information.


Toolbar icon's menu is completely broken

Clicking the toolbar button in v1.1.3 on Firefox 54, 55 or 56 results in the following:

And the following error in the Browser Console:

v1.1.2 works fine.

I am having the same problem. Firefox 55 @ Ubuntu.

Thanks you
I fixed it by 358850b24c39c1b0692ee138a63f691e6f128943

@mems, could you push the update to addons.mozilla.org? The latest version there has been broken for more than 2 months now...

@FatOrangutan A new version with additional commit is waiting a review, and should be available soon

The version <IP_ADDRESS> has been approved


Fix test environment

What does this PR do?

Fix Windows support
Finish log setup from https://github.com/DataDog/integrations-core/pull/7616

/azp run PR Changes


Edited accomplishments (qbronze).

Edited accomplishments (qbronze).

👷 Deploy Preview for yunrui-zhang processing.

Name
Link

🔨 Latest commit
6df0aff63c23990329f5938f0dffefaab825885d

🔍 Latest deploy log
https://app.netlify.com/sites/yunrui-zhang/deploys/650aeb97b1d0f0000872e4a4


Equalize histogram HSV

Hey, equalize histogram HSV seems to have an option that can be adjusted. however it seems that it's only fixed to 128. Maybe this is a bug?
Version 3.0.0

Seems to be a bug, maybe it doesn't require parameters at all, so i will look how to fix that


tf_prefix is not resolved

Looks like tf::TransformListener::resolve is missing on frame_ids. Is that correct ? If yes I can make a fix for that.

which code please ?

Oh, sorry I forgot to tell you the code. It's in naoqi_joint_states.
// get base_frame_id (and fix prefix if necessary)
m_privateNh.param("base_frame_id", m_baseFrameId, m_baseFrameId);
m_privateNh.param("odom_frame_id", m_odomFrameId, m_odomFrameId);
m_privateNh.param("use_imu_angles", m_useIMUAngles, m_useIMUAngles);

I made a fix for that, to enable muliple robots:
https://github.com/ros-naoqi/naoqi_bridge/commit/193cca15788e94784d61a6f7c8a89f2e016684a6

actually, that bug would not happen with tf2 right ? Wouldn't it be simpler to switch to tf2 ? That's what the alrosbridge that got announced yesterday does.
https://github.com/ros-naoqi/alrosbridge/blob/master/src/converters/nao_joint_state.cpp


include and category fixup

Fixup full include paths and single "category" issue with Blueprint function.
This fixes compiling issues for 4.22.3

Closing as this is for an old version of the RMC at this point, as v4 is now complete.
Thank you for your support! Feel free to PR in the future!


refactor(merge-tree sequence): remove combining op and related codepaths

Removes ICombiningOp and code gated on different values of it. This code has no code coverage and is generally unused by partners.
Changeset+deprecation PR coming after CI passes.
I'm considering modifying some of the APIs to accept undefined in the place of the combining op to reduce code churn, but based on a quick glance of partner code, this likely won't be necessary or useful.

Deprecation PR: https://github.com/microsoft/FluidFramework/pull/18318


MicrosoftCommonPropsHasBeenImported is set after importing Directory.Build.props

Steps to reproduce
Importing microsoft.build.centralpackageversions SDK (2.0.52) from the Directory.Build.props.
Directory.Build.props
<Project>
  <Import Project="Sdk.props" Sdk="Microsoft.Build.CentralPackageVersions" />
</Project>

Expected behavior
No warning is issued.
Actual behavior
C:\Users\Andrew\.nuget\packages\microsoft.build.centralpackageversions\2.0.52\Sdk\Sdk.props(14,3): warning MSB4011: "C:\Program Files\dotnet\sdk\5.0.100-preview.5.20253.6\Current\Microsoft.Common.props" cannot be imported again. It was already imported at "C:\Program Files\dotnet\sdk\5.0.100-preview.5.20253.6\Sdks\Microsoft.NET.Sdk\Sdk\Sdk.props (40,3)". This is most likely a build authoring error. This subsequent import will be ignored. [D:\dev\SharpGenTools\SdkTests\Functions\Functions.csproj]

Reason
microsoft.build.centralpackageversions correctly checks for MicrosoftCommonPropsHasBeenImported, but it's not set yet. Microsoft.Common.props sets it after importing Directory.Build.props file.
Environment data

dotnet --info
.NET SDK (reflecting any global.json):
 Version:   5.0.100-preview.5.20253.6
 Commit:    7e7a355253

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19613
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.100-preview.5.20253.6\

Host (useful for support):
  Version: 5.0.0-preview.5.20253.6
  Commit:  6d40a8ff51

.NET SDKs installed:
  2.1.800 [C:\Program Files\dotnet\sdk]
  2.1.805 [C:\Program Files\dotnet\sdk]
  3.1.201 [C:\Program Files\dotnet\sdk]
  3.1.300-preview-015135 [C:\Program Files\dotnet\sdk]
  3.1.300-preview-015139 [C:\Program Files\dotnet\sdk]
  5.0.100-preview.3.20170.4 [C:\Program Files\dotnet\sdk]
  5.0.100-preview.5.20253.6 [C:\Program Files\dotnet\sdk]

MSBuild Binary Log
Process = "C:\Program Files\dotnet\dotnet.exe"
MSBuild executable path = "C:\Program Files\dotnet\sdk\5.0.100-preview.5.20253.6\MSBuild.dll"
Command line arguments = ""C:\Program Files\dotnet\sdk\5.0.100-preview.5.20253.6\MSBuild.dll" -maxcpucount -verbosity:m /t:GenerateSharpGenBindings /bl "-distributedlogger:Microsoft.DotNet.Tools.MSBuild.MSBuildLogger,C:\Program Files\dotnet\sdk\5.0.100-preview.5.20253.6\dotnet.dll*Microsoft.DotNet.Tools.MSBuild.MSBuildForwardingLogger,C:\Program Files\dotnet\sdk\5.0.100-preview.5.20253.6\dotnet.dll""
Current directory = "D:\dev\SharpGenTools\SdkTests\Functions"
MSBuild version = "16.7.0-preview-20229-03+2cee6d020"

I think the correct solution here is to only use it in Directory.Build.targets via the <Sdk /> element.
<Sdk Name="Microsoft.Build.CentralPackageVersions" Version="1.0.0" />

I can update the docs to make this more clear.  I'm not sure how to detect that Sdk.props is being imported in Directory.Build.props so it might not be possible to tell the user something is wrong.

@jeffkl wouldn't it be better to set MicrosoftCommonPropsHasBeenImported before importing Directory.Build.props? This way, it enables detection of whether... Microsoft.Common.props has been imported. Fixing the docs doesn't solve the problem of MicrosoftCommonPropsHasBeenImported not being a reliable indicator that Microsoft.Common.props has been imported.

Also, because of the reason above, I believe the move from MSBuild repo is erroneous, since it's not specific to the specific SDK, but instead, it applies to any scenario trying to check from Directory.Build.props (or any descendant import) if Microsoft.Common.props has already been imported.

Yes the root of the problem is that there's no property set that tells Microsoft.Build.CentralPackageVersions that its been imported in Directory.Build.props.  The MicrosoftCommonPropsHasBeenImported should probably be set earlier but that might be a breaking change.  Perhaps we need to add a new property that is set at the very top of Microsoft.Common.props to prevent that?

@jeffkl Ironically - I am looking for just such a property to indicate that Microsoft.Build.CentralPackageVersions has been imported in my SDK! https://github.com/CZEMacLeod/MSBuild.SDK.SystemWeb/issues/26

I'll add a property, tracking in #320

Microsoft.Build.CentralPackageVersions must be imported in Directory.Build.targets so I'm going to make a change to log an error if its imported in Directory.Build.props.


Friends Pairing

💥 Proposal
Given N, friends who want to go to a party on bikes.
Each guy can go as single, or as a couple.
Find the number of ways in which N friends can go to the party.
Have you read the Contributing Guidelines on Pull Requests?
Yes
I want to work on this in C++

/assign


fix status code when container found on unhealthy node

Signed-off-by: Victor Vieux<EMAIL_ADDRESS>

also fix a test for engin-master

LGTM.

LGTM


Enhance Garment Cover Letter Pdf

Codecov Report

Merging #14 (4d9bcb4) into dev (74015e4) will decrease coverage by 0.01%.
The diff coverage is 89.83%.

@@            Coverage Diff             @@
##              dev      #14      +/-   ##
==========================================
- Coverage   92.75%   92.73%   -0.02%
==========================================
  Files         809      809
  Lines       78666    78726      +60
  Branches     4438     4440       +2
==========================================
+ Hits        72965    73006      +41
- Misses       5043     5061      +18
- Partials      658      659       +1

Impacted Files
Coverage Δ

...hipping/CoverLetter/GarmentCoverLetterViewModel.cs
96.93% <ø> (+0.86%)
:arrow_up:

...ructure/GarmentShippingCostStructurePdfTemplate.cs
0.00% <0.00%> (ø)

...calCoverLetter/GarmentLocalCoverLetterViewModel.cs
97.61% <ø> (+1.61%)
:arrow_up:

...ion/ToBeRefactored/Utilities/Base64ImageStrings.cs
100.00% <ø> (ø)

...tshipping/ShippingNote/GarmentShippingNoteModel.cs
92.92% <30.00%> (-7.08%)
:arrow_down:

...ping/GarmentPackingList/GarmentPackingListModel.cs
96.55% <47.36%> (-2.25%)
:arrow_down:

...ingList/GarmentPackingListWithHeaderPdfTemplate.cs
80.56% <50.00%> (+0.10%)
:arrow_up:

...mentShippingInvoice/GarmentShippingInvoiceModel.cs
98.56% <58.33%> (-1.44%)
:arrow_down:

...pping/CoverLetter/GarmentCoverLetterPdfTemplate.cs
99.05% <100.00%> (-0.04%)
:arrow_down:

...SalesDO/GarmentShippingExportSalesDOPdfTemplate.cs
97.40% <100.00%> (-0.06%)
:arrow_down:

... and 30 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Get rid of the workaround to find export annotations on accessed.

A long time ago, @JSExport used not to have @field @getter @setter, and that meant we needed a workaround to go look in sym.accessed for annotations if sym was an accessor. Now that we break binary compatibility, this is not necessary anymore.
Removing the workaround revealed that we had similarly forgot to add the meta-annotations to @JSExportStatic, which we also fix here.
The two test that are removed were specifically designed to simulate a scenario in which we were loading things compiled by an old version. They are not relevant anymore.

Refer to this link for build results (access rights to CI server needed):
https://scala-webapps.epfl.ch/jenkins/job/scalajs-pr/4104/
https://scala-webapps.epfl.ch/jenkins/job/scalajs-matrix-build/5457/
Test PASSed.


Support clear cache when using uv

Discussed in https://github.com/astral-sh/rye/discussions/990

Originally posted by nrbnlulu April  3, 2024
I figure that rye uses some global package cache. If I accidentally modify some package sources I would like to distinct the cache is it possible?

Clear it?
Yes, sorry for the misleading header.

Generally it is useful if for some reason you edited the package and you want to reinstall it.

Willing to help on this one if you'd have time to write some minor guidelines...

FWIW I moved to uv which has quite the same if not better dev-ex.

Also in terms of project management? For anyone interested, here is a comment from the team about rye vs uv: https://github.com/astral-sh/rye/discussions/1164

Also in terms of project management? For anyone interested, here is a comment from the team about rye vs uv: #1164

yep. I only missed the version bump feature though there is a PR for this


[fix](multi catalog)Collect decimal and date type min max statistic value.

Proposed changes
Issue Number: close #xxx
The min and max value of decimal and date columns in hive external table are incorrect, this pr is to parse the min max value in HMS correctly.
Describe your changes.
Checklist(Required)

Does it affect the original behavior:

[ ] Yes
[ ] No
[ ] I don't know

Has unit tests been added:

[ ] Yes
[ ] No
[ ] No Need

Has document been added or modified:

[ ] Yes
[ ] No
[ ] No Need

Does it need to update dependencies:

[ ] Yes
[ ] No

Are there any changes that cannot be rolled back:

[ ] Yes (If Yes, please explain WHY)
[ ] No

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.83 seconds
load time: 507 seconds
storage size:<PHONE_NUMBER>4 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230130154722_clickbench_pr_87294.html


⚠ Secret Website 3 has degraded performance

In ae17004, Secret Website 3 ($SECRET_WEBSITE_3) experienced degraded performance:

HTTP code: 200
Response time: 2844 ms

Resolved: Secret Website 3 performance has improved in 9b26cd6.


RuntimeError: Unable to load global settings of Nest. load() missing 1 required positional argument: 'Loader'

Hi, I just downloaded the Nest and S3N and try to get it run. When importing the nest module, an error shown as title occurs. Any idea can solve this problem? Many thanks!
Input:
from nest import register
Output:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/nest/settings.py", line 126, in
settings = SettingManager()
File "/usr/local/lib/python3.6/dist-packages/nest/settings.py", line 107, in init
self.load()
File "/usr/local/lib/python3.6/dist-packages/nest/settings.py", line 119, in load
self.settings, self.user_settings = SettingManager.load_settings()
File "/usr/local/lib/python3.6/dist-packages/nest/settings.py", line 93, in load_settings
settings = yaml.load(DEFAULT_SETTINGS)
TypeError: load() missing 1 required positional argument: 'Loader'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python3.6/dist-packages/nest/init.py", line 1, in
from nest.parser import run_tasks
File "/usr/local/lib/python3.6/dist-packages/nest/parser.py", line 7, in
import nest.utils as U
File "/usr/local/lib/python3.6/dist-packages/nest/utils.py", line 12, in
from nest.logger import exception
File "/usr/local/lib/python3.6/dist-packages/nest/logger.py", line 7, in
from nest.settings import settings
File "/usr/local/lib/python3.6/dist-packages/nest/settings.py", line 128, in
raise RuntimeError('Unable to load global settings of Nest. %s' % exc_info)
RuntimeError: Unable to load global settings of Nest. load() missing 1 required positional argument: 'Loader'
Platform: Ubuntu 20.04
Python 3.6

It may be caused by the version of YAML. It works after adding "Loader=yaml.Loader" in yaml.load (i.e., yaml.load(xxx, Loader=yaml.Loader)) . Thanks!


Is there a way to validate password and passwordConfirmation onChange, before other validations have passed?

Is there a way to validate password and passwordConfirmation onChange, before other validations have passed? refine validates the passwordConfirmation after all validations.

Is this what you are looking for?
const schemaBase = z.object( {
    password: z.string(),
    passwordConfirmation: z.string(),
} )

const schema = schemaBase
    .passthrough()
    .refine( ( { password, passwordConfirmation } ) => password === passwordConfirmation, {
        message: 'Passwords must match',
        path: [ 'passwordConfirmation' ],
    } )
    .pipe(
        schemaBase.extend( {
            someOtherProp: z.string(),
        } )
    )

const result = schema.safeParse( {
    password: '123',
    passwordConfirmation: '12',
    someOtherProp: 'foo',
} )
!result.success && console.log( result.error.issues )
// [
//     {
//       code: "custom",
//       message: "Passwords must match",
//       path: [ "passwordConfirmation" ],
//     }
// ]

If you found my answer satisfactory, please consider supporting me. Even a small amount is greatly appreciated. Thanks friend! 🙏
https://github.com/sponsors/JacobWeisenburger

@JacobWeisenburger  Thanks. It looks like a paint in my .. 😄 . Yup has these features out of the box 😞 .


Fix links in Ords README

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

:exclamation: No coverage uploaded for pull request base (main@42e9ed6). Click here to learn what that means.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #14   +/-   ##
=======================================
  Coverage        ?   48.04%
=======================================
  Files           ?       48
  Lines           ?     2895
  Branches        ?      421
=======================================
  Hits            ?     1391
  Misses          ?     1468
  Partials        ?       36

Flag
Coverage Δ

unittests
48.04% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Don't have permissions to merge this without a review unfortunately


[Extension] Need to ask user to enter password before signing the deploy

Is your feature request related to a problem? Please describe.
To temporarily mitigate the user info issues with service worker, the extension needs to ask users to enter the password before signing deploy action.
Describe the solution you'd like
Affected features:

Send CSPR
Staking

Suggestion: handle logic in useSigner , @djhahe please advise more about this.
Describe alternatives you've considered
In term of UI, we can reuse this screen and replacing by Password form.

Additional context
Related ticket: #111

Sw is working well and stable not, this one could be close without changes


Need Local File Access Option

From<EMAIL_ADDRESS>on March 24, 2012 11:11:18
What steps will reproduce the problem? 1. Go to local html file on desktop
2. Observe the lack of outliner icon What is the expected output? What do you see instead? I was hoping to see the outliner icon, instead I did not. What version of the product are you using? On what operating system? <IP_ADDRESS> on OSX 10.6.8 (Snow Leopard) Please provide any additional information below. I installed this extension so I could test my HTML projects for outline quality before going live with them, but since the extension does not offer the "Allow access to file URLs" option, I had to modify the extension to get this functionality.
The fix is to change the following line in the manifest file from:
"matches": [ "http:/// "https:/// ]
to
"matches": [ "http:/// "https:/// "file:///" ]
I think most people interested in this very useful extension are developers with similar goals as myself, so having the file protocol option is pretty crucial (in my opinion) to this extension's success.
Very nice extension overall.
Original issue: http://code.google.com/p/h5o/issues/detail?id=15

Moved to https://github.com/h5o/h5o-chrome/issues/4


[BUG] how control Desktop Camera (Wont move Arrow keys or Num pad)

i tried move camera, and far i seen im able pop out Camera, but unable move it round. last Desktop Camera i have v1.2.1
since then i didnt seen it got fixed part able activate by button but right now unable move camera round main issue.
if have changed keybinds, it hasnt been updated on readme
UPDATE
I just checked Desktop Camera Code, kinda disabled keybinds of it. so Desktop unable move camera anymore but able show it.

Mhm, that's right.
Since VRC UI update, both the camera and the UI got completely remade, and thus this mod is almost entirely broken, in such a way that it needs a complete remake.
And so, because of what I said in the readme, I do not plan on updating this mod for now.
I asked a friend from VRCMG to do it if he wanted to, but since he didn't do in the end, DesktopCamera is just archived here and waiting for someone to update it, pretty much.
For anyone reading this:

If you know how to code and want to update this mod, please head to the readme for my contact info;
If you are just a user, then know that, even though most functions (such as custom menu, movements, etc.) are disabled, as the OP mentioned, with this mod as it is, you can still use the camera in-game, since it allows you to activate/deactivate it. Though it's a bit harder without the custom menu, since you'd have to manually click the buttons on the camera itself; To move it around, you can simply use the default VRChat keybinds to move/rotate objects.


Event is not dispatched on the cold start

If you launch the application first from the app icon, close it (by pressing home button on the phone) and then click on the UL link - JS event is captured.
But if you kill the app from task manager, and then click on the link - event is missed.

Actually
it does work, you can close this @nikDemyankov
I was registering to the js event at the wrong time (inside device ready)

With pleasure :)

I still have this problem. I have a long index.html (old app) and have tried to add the event listener direct in a script part in the page and also in the onLoad event. In both cases the UL link event is fired when the app is already loaded in the background. When i kill it and try again, the app opens, but without the event.
I am trying with Android.
Any idea?

Can you show some code, how you do the subscription and where?

I have made some tests. As i wrote, it is an old and big app. It has one big index.html. I include the cordova.ja and 33 other scripts at the beginning. After that i had this code:
<script>
function onLoad() {
	document.addEventListener("deviceready", onDeviceReady, false);
}

function onDeviceReady() {
		console.log('deviceready');
		document.addEventListener("resume", doResumeApp, false);
		document.addEventListener("pause", doPauseApp, false);
		document.addEventListener("online", onOnline, false);
		document.addEventListener("offline", onOffline, false);
		console.log('eventhandler installed')
		Init();
};
function onULOpen(event) {
	alert('onULOpen');
}
document.addEventListener('ULOpen', onULOpen, false);

</script>

It has not worked, when the app was not running in the background. Now i have moved the UL block between the include of cordova.js and the other scripts.
<script>
document.addEventListener('ULOpen', onULOpen, false);
function onULOpen(event) {
	alert('onULOpen');
}
</script>

And this works.
I have tested with a Samsung Galaxy Note 4.

So now it's fine?

Yes. I have now the event in front before loading other scripts.
Is it possible that the event is fired before the page is loaded?
Maybe you should add some information, when the event is called. In my tests i see it before resume and before deviceReady.

When app is launched from the link - event is saved until plugins JS module is initialised. And this can happen before the deviceready  in the index page. So, you need to subscribe to the event as soon as possible.
As a workaround I can add to the plugin JS module (i.e., universalLinks), and people can use it like so:
function onDeviceReady() {
  console.log('deviceready');

  // set handler for the event
  universalLinks.on('event_name', onULOpen);
};

function onULOpen(event) {
  // handle the event
}

Internally, it will store the events, received from the native side; and then universalLinks.on(...) is called -  send them to the callback. Basically, you are subscribing to the event through the plugins module.
What do you think about that?

This would be a good solution, because in the event all cordova stuff is available, when it is called after deviceReady.
This also prevents for timing issues.
PS: Thank you for your work.

Thanks. Closing this issue. Created a separate one with the required feature.


agateau/refactor file read

refactor: delay document reading until we need to scan them
refactor: remove File.from_path()
refactor: remove Files.filenames()
fix: make File.is_longer_than() use string size, not byte size

Codecov Report

Merging #503 (895b84b) into main (25588af) will increase coverage by 0.06%.
The diff coverage is 96.26%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main     #503      +/-   ##
==========================================
+ Coverage   93.84%   93.91%   +0.06%
==========================================
  Files          78       78
  Lines        3623     3647      +24
==========================================
+ Hits         3400     3425      +25
+ Misses        223      222       -1

Flag
Coverage Δ

unittests
93.91% <96.26%> (+0.06%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

ggshield/scan/scanner.py
91.75% <78.57%> (-0.94%)
:arrow_down:

ggshield/scan/scannable.py
95.48% <98.55%> (+1.32%)
:arrow_up:

ggshield/cmd/iac/scan.py
98.61% <100.00%> (ø)

ggshield/cmd/secret/scan/docset.py
67.74% <100.00%> (ø)

ggshield/core/file_utils.py
91.83% <100.00%> (+2.18%)
:arrow_up:

ggshield/iac/filter.py
100.00% <100.00%> (ø)

ggshield/output/text/iac_text_output_handler.py
89.18% <100.00%> (ø)

ggshield/scan/__init__.py
100.00% <100.00%> (ø)

ggshield/scan/docker.py
96.22% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Text gets duplicated and magled when resizing window to minimum width and out again

Environment
Windows build number: Microsoft Windows NT 10.0.19619.0
Windows Terminal version (if applicable): 0.11.1251.0

Any other software? Ubuntu on WSL as default Terminal Profile

Steps to reproduce

Start Terminal
Open new Ubuntu session from drop-down menu
Resize terminal window to minimum width
Resize out to reasonable width again
Repeat previous two steps as many times as you desire thereby increasing the text mashing

Expected behavior

The BASH prompt and other text on the screen should reflow. It should remain intact with no repetitions or extra text inserted where there was no text before.
Actual behavior

The prompt is getting duplicated and mangled after multiple resizes.
Before resizing multiple times:

After resizing multiple times:

You can exacerbate the issue by ensuring your prompt is very long, possibly it requires the prompt to be longer than the minimum window width to trigger the problem, such as by navigating deep into a very deep filesystem hierarchy.

We have #5450 for this.

Thanks for the help triaging @skyline75489!
/dup #5450 #5666


也许需要精简一下结构？

Expected behavior (预期行为)
感觉现在有点太大了…而且好多功能以前有的现在都不流行了，以及一些功能太小众的也在里面，三个scheme也许可以拆成三个theme这样代码阅读起来更好理解结构？不过是我个人的想法，不知道大家怎么想…
还有很不明白swig的主页面不是显示not maintained了吗…有没有必要迁移成其他模板语言呢…
Actual behavior (实际行为)
Steps to reproduce the behavior (重现步骤)
NexT Information
Add the  ✔  sign before an item which is affected by this behavior.
NexT Version:

Master
Latest Release
Old version -

NexT Scheme:

All schemes
Muse
Mist
Pisces

Other Information (Like Browser, System, Screenshots)

@AlynxZhou in English, please.

@ivan-nginx Sorry...I just mean that I think now NexT has too many functions and some of them are not used by the majority, maybe we need to remove some functions, and split three schemes into three themes? Because I used to hack some part of my blog, I find it's hard to remind which part I have changed and how I found it, because there are three schemes and a lot of files...It maybe easier to understand if split.
Plus I find the github repo of swig template language shows that swig is not maintained? Is that means we need to move to another kind of template language? Any future plan here?
However those are just my thoughts and I think final decision needs to be made by @iissnan , I just give some advice. And seems he hasn't showed up for a long time...
I am wondering that once I saw your reply under a Chinese pullrequest/issue, so I guess you use google translate to read Chinese sometimes? :-) Sorry for my forgeting using English for you to read, I'll remember though my English is not so good.

@AlynxZhou ok, it's alright. I use google translate sometimes, but sometimes it's translate that i don't understand what there meaning. Don't forget, my basic language is Russian and my English is not good to.

About migrate form swig to ejs or jade or else language - about this must think @iissnan. Of course, we all must to do decision about how it will be better, but for now, as i thing, swig is already fast and anyone may do what he's want: all variables is worked anywhere - in .swig files, in .styl files and in other basic and custom style variables too. But, if, for example, ejs something will gooder then swig, need to think about migrate. But don't remember, at the begining of this theme Next Vi ported this theme from jade (or ejs, i don't remember, but you may see this in first commits of repository) to swig, so, i think this was do not simply and was true causes for migrate to swig.
About split themes - don't understand what are u real meaning. U want to remove other themes and stay just 1 theme or what? In fact, themes - it's just misc CSS styles what used by scheme option in config. If 1 scheme is used, another scheme styles don't include in main compilation, but not always. We thing about it in our chat and if your interesting of maintenance Next theme, a can ask for Vi to invite you in working team. And about unused/unworked functions - need to talk too. But, my main idea for all of this: make all parts of styles and/or templates to load at compilation by conditional import by Next _config.yml.

@ivan-nginx I understand what you say, I am not meaning that stay just 1 theme, I mean make 3 schemes into 3 different themes may be better to maintain. I understand that final page will only contain 1 scheme codes and won't waste space. Why I think about that is sometimes I want to try to change some code and find it's hard to find the proper file. And in China sometimes it's hard to connect to github and clone a repo, smaller project is easier to clone. ;-p But I think maybe I need to read more and understand the structure clearly. So maybe there is no need to change.
And about maintainance, in fact I love this theme very much, and I'm happy to contribute to this theme, but I am lack of experience of front-end like this, and may not work well. I would like to learn more.
Plus I think this issue can be closed because I got answer to my problem, thanks to your work that make NexT better.

Why I think about that is sometimes I want to try to change some code and find it's hard to find the proper file.
At the beginning of maintenance, i use local search in files content by needed strings. For example, u can see source code in browser and find .any-class-name, and afterthat u may find needed files by .any-class-name key from local search. Anyway, whole structure is basically build on variables and she's really difficult to understand it at beginning.


Expand logging tests

Expand the logging tests to include all console.log and DSTest-style logs

Apologies for any missteps as a newbie but I'd like to try to grok this better so would appreciate a little guidance.
I'd love to be able to flesh out more robust tests.
Are you looking for something like this?
my branch:
https://github.com/shawnharmsen/foundry/commit/bc8ae4d89ba5695b03a6155a43b330ec66b86c31

add more to [testdata/logs/DebugLogs.t.sol]
add to [forge/src/multi_runner.rs]

Check the types from ds-test/test.sol
    event log                    (string);
    event logs                   (bytes);

    event log_address            (address);
    event log_bytes32            (bytes32);
    event log_int                (int);
    event log_uint               (uint);
    event log_bytes              (bytes);
    event log_string             (string);

    event log_named_address      (string key, address val);
    event log_named_bytes32      (string key, bytes32 val);
    event log_named_decimal_int  (string key, int val, uint decimals);
    event log_named_decimal_uint (string key, uint val, uint decimals);
    event log_named_int          (string key, int val);
    event log_named_uint         (string key, uint val);
    event log_named_bytes        (string key, bytes val);
    event log_named_string       (string key, string val);

https://github.com/dapphub/ds-test/blob/2c7dbcc8586b33f358e3307a443e524490c17666/src/test.sol#L19-L36

Yeah, essentially we should make sure we can decode all events from DSTest as well as all calls to console.log. These should be added to DebugLogs.t.sol as well as HardhatLogs.t.sol. We should then add the relevant assertions to multi_runner.rs :)


Haskell shell completion

Motivation for this change
Make shell completion work for more haskell commands.
niv and ormolu require the "support separate bin" commit. Otherwise, it doesn't work when niv.bin is used, which is how pkgs.niv is defined (as opposed to haskellPackages.niv).
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Could you rebase this on the haskell-updates branch?

Done.

Also, is this related at all to NixOS/cabal2nix#474 by @zimbatm?

It has the same goal, but these packages don't use optparse-applicative, so the implementation is different.
I'm surprised to see these overrides implemented in cabal2nix. Doesn't that make it harder to contribute to it?

Thanks!

I'm surprised to see these overrides implemented in cabal2nix. Doesn't that make it harder to contribute to it?

I'm not sure of the full backstory, but I think the idea is that if you put overrides directly into cabal2nix, they will work for anyone running cabal2nix (even if they are not using the package set from nixpkgs).
This used to be important when there were tools like stack2nix that directly made use of cabal2nix, but maybe not as important anymore?

@cdepillabout I had a discussion about that with peti last friday. It's exactly for the reason you say. We are brainstorming right now about redesigning the process (something which we really want to here your opinion about, btw. we‘d like to schedule a meeting as soon as we get to it), one longterm goal might be to migrate all those overrides into nixpkgs and having tooling that can generate derivations will taking overrides defined in nixpkgs into account.

one longterm goal might be to migrate all those overrides into nixpkgs and having tooling that can generate derivations will taking overrides defined in nixpkgs into account.

That's probably the best option. We have been using that approached in the ruby packages and it has been working quite well.

to migrate all those overrides into nixpkgs and having tooling that can generate derivations will taking overrides defined in nixpkgs into account.

This sounds exactly like how I expected it to work. :+1:

@maralorn

We are brainstorming right now about redesigning the process (something which we really want to hear your opinion about

To be honest, I don't really have strong opinions either way.
It seems like a lot of people just end up using callCabal2nix (or the related callHackage), so as long as those work nicely with cabal2nix, I'm happy.

The biggest advantage is that the haskell packages can be fixed, without having to re-release cabal2nix.

I agree that this does sound nice, but maybe another solution would be to add more people as maintainers for cabal2nix and push out releases faster.


Add no-std flag to byteorder

@homu r+

:pushpin: Commit 8dd0143 has been approved by hauleth

:zap: Test exempted - status


Pac4JSecurityFeature should look for annotations in superclass/interface

I am having trouble with @Pac4JSecurity annotations not being located on resource class/methods when the resource class had been dynamically extended (e.g. via a javassist ProxyFactory).
The annotation getters in Pac4JSecurityFeature only check the class itself for annotations and does not continue with superclass or interface. In comparison, similar DynamicFeature implementations in Jersey use org.glassfish.jersey.server.model.AnnotatedMethod utility class (or other similar reflection utils), which has a more elaborate logic of locating annotations. Could something like that be ported/reused in jax-rs-pac4j?
The actual issue cropped up when using dropwizard-pac4j in a Dropwizard application with a subresource instantiated via a UnitOfWorkAwareProxyFactory (suggested as a workaround in dropwizard/dropwizard#1806). This creates a proxy subclass of a sub-resource and the pac4j annotations are lost.
While that use case would go away with some future version of Dropwizard, the general problem with Pac4JSecurityFeature as described should still be valid in general.

Yes, that makes sense!
I will look at it tomorrow. The only thing is that I don't want to rely on AnnotatedMethod from Jersey because we have to support any JAX-RS runtime, but I don't think this is a too complex functionality to implement.
Thanks :)

@andriusvelykis could you check that everything is ok with the latest SNAPSHOT? (you can retrieve it from sonatype's OSS Snapshot repository, documented in the README)

Thanks - just checked, seems to be working in my case!

Great, I will let it steep for a few days and also see if you don't find other bugs or limitation and then we will release a new version then :)

@andriusvelykis do you need it released with version 1.0.1 or can you wait for the next 1.1.0 which will include some new features and be released in a few weeks?

No rush for me - feel free to release as 1.1.0. Thanks!