TS-4571: Fix "pointer to local outside scope" (CID 1356978)

Dereferencing the returned or out-of-scope stack pointer will access an invalid location on the stack after its scope or after the function returns.
In ParentRecord::Init(matcher_line *): Pointer to a local stack variable returned or used outside scope (CWE-562)

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/311/ for details.

trying a manual trigger [approve ci]

👍

👍

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/315/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/209/ for details.

This should probably get back ported to 6.2.0 right ?

@PSUdaemon do you want to change the sprintf to snprintf while you are changing this function?

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/325/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/219/ for details.


Fortran-layout arrays are handled wrong

I cannot write and read back numpy arrays that use the order='F' memory layout. The array shape is recovered correctly, but the array content is scrambled, as if the strides were handled wrong. A brief glance at the created file shows the expected stride attribute.
This code reproduces the problem:
import numpy as np
import asdf

array = np.array([[11,12,13], [21,22,23]], order='F')
file = asdf.AsdfFile({"array": array})
file.write_to("/tmp/strides.asdf")

file2 = asdf.open("/tmp/strides.asdf")
array2 = file2.tree["array"]

print("Original array:\n", array[:,:])
print("Recovered array:\n", array2[:,:])

This outputs:
Original array:
 [[11 12 13]
 [21 22 23]]
Recovered array:
 [[11 13 22]
 [12 21 23]]

Both outputs should be identical.
If I change the order='F' to order='C' (which is also the default), then everything works fine.

Thanks @eschnett for the bug report. Please see #539 for the fix.

@eschnett the fix for this was released today in v2.0.3.


Save files directory when "always ask" is selected?

Where does this save the files when the when Firefox is set to always ask where to save downloads?

Good question! I ran a few tests and figured out that Firefox apparently remembers the last configured download directory and uses that for background downloads, even though you've set the Browser to always ask.

CLosing this issue, since there's nothing more to do about it. Feel free to reopen if necessary.


tag ver 0.6

Can you make a 0.6 tag, or are you still working on 0.6?  Readme has info bout .6 but no tag..

I can make a tag, I just hadn't had the time to do it.

The v0.6 has been added.


Cursor should change when hovering over nav, buttons

The cursor should change to a pointer when the user is hovering over buttons, nav items, and other interactive elements to let the user know that they can click on them.

Great. GO AHEAD.


Fix117: check about key schedule should be consistent

REF: #117
Double check if algorithm has been provisioned, because ALGORITHM might be skipped.

I also notice that there is a key difference between PSK and KEY_EXCHANGE. The PSK does not require NEGOTIATE_ALGORITHMS. But KEY_EXCHANGE requires NEGOTIATE_ALGORITHMS.
See "SPDM commands GET_CAPABILITIES and NEGOTIATE_ALGORITHMS are not required during session key establishment with the PSK option"
I change my mind now. I think we just need add comment to address the reason of inconsistency.
Adding code == Adding size.


Fix the execution order of a given struct.

Describe the bug
When a struct is passed as an argument to the Execute function, the defined methods are executed in alphabetical order. This behavior gets weird when it comes to defining a sequence of methods to insert data that have dependencies on a previous set of inserted data. The user has to manage to name the methods of the struct in an alphabetical and logical order at the same time without getting conflicts at the time to run the tool.
To Reproduce
Steps to reproduce the behavior:

Define a struct just like the example provided by the package
Define a method with a name that starts with X, Y or Z. Make sure the table you're inserting data, has constraints with another.
Define a method with a name with A, B, C, or whatever as a prefix. The table you are inserting data into should be the referenced table by the previous one.
Run the tool with the struct provided and see the error.

Expected behavior
At this point, I'm not sure what should be the expected behavior, for most of the cases. From my point of view, the order used to define the methods in the source code should the source of truth when it comes to the execution, but not sure if that's possible.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: Fedora Linux
Version 34

Additional context
Add any other context about the problem here.

It's not possible fix this, but added some instructions on how to bypass this limitation to the README file #27


#48in24: diamond 2024-12-03

[x] Run bin/configlet sync -u -e diamond --yes --docs --filepaths --metadata --tests include (updates the Markdown files and maybe tests.toml)

docs: instructions unsynced: diamond
metadata: unsynced: diamond

[x] Drop strict types comments from test file and example code (these are useless)
[x] Add and sync test meta data to tests (uuid / @testdox in DocBlocks)
[x] Decide on adding / adjusting / ordering test cases to match current problem specs

Do not redesign the student's interface or add test cases that would invalidate existing community solutions. These are extra tasks, which should be discussed in advance.

I don't think renaming the function is worth breaking all existing solutions.

I don't think renaming the function is worth breaking all existing solutions.

I agree.


Update test_tuples.py

Hi @TusharShandilya, thank you for PR!
It is failing though. Could you please review code style issues?

Thank you for the fixes @TusharShandilya


Allow prop_remove to operate on a list of property types

Removing text properties can be slow as it requires scanning the buffer lines internally. The best way is to use a line range when calling prop_remove. However, due to the way text properties move around when text is modified, placing a text prop with prop_add on line 10 does not mean that it's on line 10 when you might want to remove it. When working with virtual text and semantic tokens, there can be a lot of text properties that need to be refreshed periodically. The simplest way is to remove existing properties matching some ID (or type) and re-add the new ones.
With "classic" text properties, it was possible to assign the same Id to each property and thus remove the props in one scan of the affected range (e.g. whole buffer). With virtual text properties, it's not possible as the ID is generated and returned, so we're left with deleting "all by type" over the range, or recording the IDs of each property type and querying if it's still present (as deleting or undoing can remove it).
We have prop_find that can take a list of types to find, so to avoid having to do multiple scans of the lines and to avoid having to remember and query every ID, provide prop_remove with the ability to remove all properties with one of a list of property types.
Problem: can't remove text properties for multiple types at once
Solution: Allow types to be a list

concrete example:

create text property types for Param, Type and Enum inlay hints, including a type for "padding"
(periodically) clear all Param, Type,  Enum and padding properties and re-create them based on the result of some query to the semantic engine.

currently requires at least 4 calls to prop_remove; can now be done with prop_remove( { 'types': [ 'Param', 'Type', 'Enum', 'padding' ], 'all': 1 } )

Happy to change it so that type can be a number or a list if you prefer that.

I think providing both "type" and "types" should be considered an error, otherwise a mistake might go unnoticed.
It might also be useful to pass a ist of id's.  We could add that later.

I think providing both "type" and "types" should be considered an error, otherwise a mistake might go unnoticed.

Sure thing, will add this.

It might also be useful to pass a ist of id's. We could add that later.

I thought the exact same.

I think providing both "type" and "types" should be considered an error, otherwise a mistake might go unnoticed.

Sure thing, will add this.

Added

Maybe it should be documented:
diff --git a/runtime/doc/textprop.txt b/runtime/doc/textprop.txt
index 9a148455e..470af2743 100644
--- a/runtime/doc/textprop.txt
+++ b/runtime/doc/textprop.txt
@@ -352,6 +352,7 @@ prop_remove({props} [, {lnum} [, {lnum-end}]])
 		{props} is a dictionary with these fields:
 		   id		remove text properties with this ID
 		   type		remove text properties with this type name
+		   types	remove text properties with these type names
 		   both		"id" and "type" must both match
 		   bufnr	use this buffer instead of the current one
 		   all		when TRUE remove all matching text properties,

Actually, it might not be obvious that the value should be a list:
diff --git a/runtime/doc/textprop.txt b/runtime/doc/textprop.txt
index 9a148455e..0c4a12bfc 100644
--- a/runtime/doc/textprop.txt
+++ b/runtime/doc/textprop.txt
@@ -352,6 +352,8 @@ prop_remove({props} [, {lnum} [, {lnum-end}]])
 		{props} is a dictionary with these fields:
 		   id		remove text properties with this ID
 		   type		remove text properties with this type name
+		   types	remove text properties with these type names;
+				the value should be a list
 		   both		"id" and "type" must both match
 		   bufnr	use this buffer instead of the current one
 		   all		when TRUE remove all matching text properties,

Wait I swear I updated the doc in my patch... let me check what I messed up

Wait I swear I updated the doc in my patch... let me check what I messed up

I did. I guess Bram will just update the runtime files at a later date. https://github.com/vim/vim/pull/10945/files#diff-9cdfd291efd41da3f72937703364d769bdb73b9b6145446717674dd1ce6f122a

Ah, my bad.  I didn't check the PR, only the pushed commit.  Yes, the help will probably be updated later.

Wait I swear I updated the doc in my patch... let me check what I messed up

Somehow the merge failed and this part went missing.
It will be in the next runtime files update.
--
There can't be a crisis today, my schedule is already full.
/// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net   \
///                                                                      \
\\        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///


how to inference ./dlrm_s_criteo_kaggle.sh

Hello
I recently read your DLRM paper and decided to try running the code myself. However, I'm a silly teenager who finds this field quite challenging.
I've tried training the model using this command ./bench/dlrm_s_criteo_kaggle.sh --test-freq=1024  and tweaked the values of the Criteo Kaggle dataset to train in various ways. What I'm really curious about is inference.

Does using ./bench/dlrm_s_criteo_kaggle.sh --inference-only execute the inference process simply? (I don't think so, based on the git issues you've responded to in the past. It seems like the model needs to be saved first. Is that correct?)

In the case where --data-generation = dataset what does the input data set structure look like during inference? (Does the original Criteo Kaggle dataset and a dataset where I've manually modified some values have the same input data?)

Thank you.

If you wanted to see the values predicted by the trained model, you would need to train and save it first. Then load it back during inference. If you are only interested in system performance and therefore using random values is fine, you can call the --inference-only option directly.
The code will simply pick part of the dataset indicated in option --raw-data-file or --processed-data-file, if you pass your custom dataset there, it will use it.


Poor performance of getkey compared to docs benchmarks

I have rerun the first set of benchmarks from /docs/speed.jl comparing getindex to getkey. In many instances getkey performs worse (both relative to getindex and relative to the values in speed.jl comments). For getkey in many instances I saw substantially worse performance (> 1 order of magnitude) compared to the getindex calls on my machine, including some allocations where there seemed to be none for the results in the comments.
I have tested this on both 1.4.2 and 1.6.1. 1.6.1 gave me ~2x speedup over the 1.4.2 results, but the numbers were still far off of what's present in the comments for getkey. The environment I ran this in was just the AxisKeys project environment with the latest version of all dependencies + BenchmarkTools. Machine information is included at the end.
Here are my results of running the benchmarks. I included the original values, and then appended my results from @btime afterwards in parentheses.
using AxisKeys, BenchmarkTools # Julia 1.4.2

#==============================#
#===== getkey vs getindex =====#
#==============================#

mat = wrapdims(rand(3,4), 11:13, 21:24)
bothmat = wrapdims(mat.data, x=11:13, y=21:24)
bothmat2 = wrapdims(mat.data, x=collect(11:13), y=collect(21:24))

@btime $mat[3, 4]    # 1.699 ns (1.699 ns)
@btime $mat(13, 24)  # 5.312 ns (124.232 ns (1 allocation: 32 bytes))

@btime $bothmat[3,4]        # 1.700 ns (1.649 ns)
@btime $bothmat[x=3, y=4]   # 1.701 ns (1.965 ns)
@btime $bothmat(13, 24)     # 5.874 ns  (124.299 ns (1 allocation: 32 bytes))
@btime $bothmat(x=13, y=24) # 14.063 ns (132.624 ns (1 allocation: 32 bytes))
@btime $bothmat2(13, 24)    # 16.719 ns (13.606 ns)

ind_collect(A) = [@inbounds(A[ijk...]) for ijk in Iterators.ProductIterator(axes(A))]
key_collect(A) = [@inbounds(A(vals...)) for vals in Iterators.ProductIterator(axiskeys(A))]

bigmat = wrapdims(rand(100,100), 1:100, 1:100);
bigmat2 = wrapdims(rand(100,100), collect(1:100), collect(1:100));

@btime ind_collect($(bigmat.data)); #  9.117 μs (4 allocations: 78.25 KiB) (12.646 μs (4 allocations: 78.25 KiB))
@btime ind_collect($bigmat);        # 11.530 μs (4 allocations: 78.25 KiB) (9.996 μs (4 allocations: 78.25 KiB))
@btime key_collect($bigmat);        # 64.064 μs (4 allocations: 78.27 KiB) (1.248 ms (10004 allocations: 390.77 KiB))
@btime key_collect($bigmat2);      # 718.804 μs (5 allocations: 78.27 KiB) (625.003 μs (5 allocations: 78.27 KiB))

twomat = wrapdims(mat.data, x=[:a, :b, :c], y=21:24)
@btime $twomat(x=:a, y=24)  # 36.734 ns (2 allocations: 64 bytes) (137.505 ns (2 allocations: 64 bytes))

@btime $twomat(24.0)        # 26.686 ns (4 allocations: 112 bytes) (44.374 ns (5 allocations: 144 bytes))
@btime $twomat(y=24.0)      # 33.860 ns (4 allocations: 112 bytes) (49.411 ns (6 allocations: 160 bytes))
@btime view($twomat, :,3)   # 24.951 ns (4 allocations: 112 bytes) (22.302 ns (4 allocations: 112 bytes))

Here's the version of everything from my instantiation:
(AxisKeys) pkg> st
Project AxisKeys v0.1.16
Status `~/Documents/AxisKeys.jl/Project.toml`
  [621f4979] AbstractFFTs v1.0.1
  [587fd27a] CovarianceEstimation v0.2.6
  [8197267c] IntervalSets v0.5.3
  [41ab1584] InvertedIndices v1.0.0
  [1fad7336] LazyStack v0.0.7
  [356022a1] NamedDims v0.2.29
  [6fe1bfb0] OffsetArrays v1.10.0
  [2913bbd2] StatsBase v0.33.8
  [bd369af6] Tables v1.4.3
  [37e2e46d] LinearAlgebra
  [10745b16] Statistics

I'm on a Thinkpad X1 Carbon 7th gen running Ubuntu 20.04. Here's the CPU/RAM info from lshw:
H/W path             Device          Class          Description
===============================================================
                                     system         20QDS3DQ00 (LENOVO_MT_20QD_BU_Think_FM_ThinkPad X1 Carbon
/0                                   bus            20QDS3DQ00
/0/2                                 memory         16GiB System Memory
/0/2/0                               memory         8GiB Row of chips LPDDR3 Synchronous 2133 MHz (0.5 ns)
/0/2/1                               memory         8GiB Row of chips LPDDR3 Synchronous 2133 MHz (0.5 ns)
/0/c                                 memory         256KiB L1 cache
/0/d                                 memory         1MiB L2 cache
/0/e                                 memory         8MiB L3 cache
/0/f                                 processor      Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz
/0/11                                memory         128KiB BIOS
/0/100                               bridge         Coffee Lake HOST and DRAM Controller
/0/100/2                             display        UHD Graphics 620 (Whiskey Lake)
/0/100/4                             generic        Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal
/0/100/8                             generic        Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Pr

That's not good! Thanks for the careful investigation.
It looks like this is the key difference -- the lookup from a Vector is similar, but the lookup from a range 1:100 (without collect) has become very slow:
@btime key_collect($bigmat);        # 64.064 μs (4 allocations: 78.27 KiB) (1.248 ms (10004 allocations: 390.77 KiB))
@btime key_collect($bigmat2);      # 718.804 μs (5 allocations: 78.27 KiB) (625.003 μs (5 allocations: 78.27 KiB))

I rolled back to 0.1.0 and tested that as well
@btime key_collect($bigmat);        # 64.064 μs (4 allocations: 78.27 KiB) (1.264 ms (10004 allocations: 390.77 KiB))

So it seems this is an issue with some dependency rather than an AxisKeys issue :)
I'll make sure we avoid UnitRange indices for the time being, I don't think that'll cause any problems for us. Thanks!


Latest Ubuntu Cloud Image AMI is packaged with AWS CLI version 1.x which causes /etc/eks/bootstrap.sh to silently misconfigure the cluster DNS when the EKS cluster has a custom Service IP CIDR address

What happened:
When an EKS cluster with a custom Kubernetes Service IP CIDR is created with Ubuntu cloud-image worker nodes, the /etc/eks/bootstrap.sh script silently misconfigures the --cluster-dns argument to kubelet in /var/snap/kubelet-eks/70/args file.
The Ubuntu cloud-image AMI (us-east-1, EKS 1.21) is ami-04c4f2c4799614025.  We found this AMI from the official AWS EKS Ubuntu cloud images catalog https://cloud-images.ubuntu.com/docs/aws/eks/.
# Distro details
root@ip-10-109-4-64:~# lsb_release -a 2> /dev/null
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

# Kubernetes version
root@ip-10-109-4-64:~# kubectl version --short
Client Version: v1.21.9
Server Version: v1.21.12-eks-a64ea69

What you expected to happen:
The /etc/eks/bootstrap.sh script to properly configure the --cluster-dns argument to kubelet and not default to <IP_ADDRESS>.
How to reproduce it (as minimally and precisely as possible):
Here is a link to the /etc/eks/bootstrap.sh script:  https://github.com/awslabs/amazon-eks-ami/blob/master/files/bootstrap.sh#L373.
In my test case, I've extracted the command of interest executed by the bootstrap.sh script:
#
# Below is the extracted command from the /etc/eks/bootstrap.sh script, see line 373
#
AWS_DEFAULT_REGION=us-east-1
CLUSTER_NAME=iat-dev-us-east-1-eks
aws eks describe-cluster \
            --region=${AWS_DEFAULT_REGION} \
            --name=${CLUSTER_NAME} \
            --output=text \
            --query 'cluster.{endpoint: endpoint, serviceIpv4Cidr: kubernetesNetworkConfig.serviceIpv4Cidr, serviceIpv6Cidr: kubernetesNetworkConfig.serviceIpv6Cidr, clusterIpFamily: kubernetesNetworkConfig.ipFamily}'

Failure when using AWS CLI version 1.x.x
#
# The Ubuntu cloud-image ami-04c4f2c4799614025 is installed with version 1.18 of the AWS CLI
# AMI image was retrieved for us-east-1 from: https://cloud-images.ubuntu.com/docs/aws/eks/
#
root@ip-10-109-4-64:~# aws --version/
aws-cli/1.18.69 Python/3.8.10 Linux/5.13.0-1031-aws botocore/1.16.19

#
# The aws_describe_cluster script produces None for the Service CIDR address.
#
root@ip-10-109-4-64:~# bash aws_describe_cluster.sh
None    https://6CCB47A35A560104CFDE3CAF89B1A0D6.gr7.us-east-1.eks.amazonaws.com        None    None

Anything else we need to know?:

Working when used with AWS CLI version 2.x.x
See that the 3rd field contains the custom EKS service IP CIDR address.

#
# This host is installed with version 2.7.4 of the AWS CLI
#
root@test-awscli-1:~# aws --version
aws-cli/2.7.4 Python/3.9.11 Linux/5.10.0-11-amd64 exe/x86_64.debian.11 prompt/off

#
# The same script produces the correct Service CIDR address.
#
root@test-awscli-1:~# bash aws_describe_cluster.sh
ipv4    https://6CCB47A35A560104CFDE3CAF89B1A0D6.gr7.us-east-1.eks.amazonaws.com        <IP_ADDRESS>/20  None

Environment:

AWS Region: us-east-1
Instance Type(s): r6i.4xlarge
EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion):

root@ip-10-109-4-64:~# aws eks describe-cluster --name iat-dev-us-east-1-eks --region us-east-1 --query cluster.platformVersion
"eks.7"

Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version):

root@ip-10-109-4-64:~# aws eks describe-cluster --name iat-dev-us-east-1-eks --region us-east-1 --query cluster.version
"1.21"

AMI Version:

ubuntu-eks/k8s_1.21/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20220623
ami-04c4f2c4799614025

Kernel (e.g. uname -a):

Linux ip-10-109-4-64 5.13.0-1031-aws #35~20.04.1-Ubuntu SMP Mon Jun 13 22:30:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Release information (run cat /etc/eks/release on a node):

root@ip-10-109-4-64:~# cat /etc/eks/release
cat: /etc/eks/release: No such file or directory

Workaround:

Update the /var/snap/kubelet-eks/70/args file by hand.
OR update the AWS launch template and provide kube-dns service IP to the --dns-cluster-ip to the /etc/eks/bootstrap.sh script.

Best fixes:

Update the AWS CLI to version 2 in the Ubuntu cloud-image AMI.
Update the bootstrap.sh script to require AWS CLI version 2.

@mw-tlhakhan I think it would be good to report this issue and request the AWS CLI upgrade in the Ubuntu cloud-images bug tracker https://bugs.launchpad.net/cloud-images that is used to track issues for the Ubuntu EKS AMI

Another workaround is to cpecify 172.20.0. 0/16 as cluster service ipv4 cidr.
Probably, the easiest one. If you know it beforehand

Terraform AWS/eks module has it 16 in every example. A lot of people just
copy some parts from the examples..For instance, i found this discussion
after I found the reason, so I decided to leave a comment for luckier
persons.
On Tue, Oct 4, 2022, 17:12 Tenzin Lhakhang @.***> wrote:

@lure https://github.com/lure , by default AWS EKS uses the
<IP_ADDRESS>/16 address block. See documentation here:
https://docs.aws.amazon.com/eks/latest/APIReference/API_KubernetesNetworkConfigRequest.html
.
AWS EKS starting ~ Oct 2021 supported configurable cluster service CIDR.
See post here:
https://aws.amazon.com/about-aws/whats-new/2020/10/amazon-eks-supports-configurable-kubernetes-service-ip-address-range/
.
The primary reason for using a different Kubernetes service CIDR is for
the following use case:
...
Previously, Amazon EKS automatically chose a value for this range based on
the primary CIDR block of the Amazon VPC used by the cluster. While this
worked for most cases, customers with VPCs peered to on-premise networks or
other Amazon VPCs found that the EKS chosen Kubernetes service IP address
range may conflict with other IP ranges in use across their network.
...
This enables customers with clusters running in a peered or direct
connected network environment to ensure that their pods can communicate
with external applications on networks outside the cluster.
—
Reply to this email directly, view it on GitHub
https://github.com/awslabs/amazon-eks-ami/issues/963#issuecomment-1267583481,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACGYK2ZWEBNI52WASEGEXTWBSMSLANCNFSM523BTTBQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

We now install 2.x CLI instead of relying on the version available in the package manager. Unfortunately I don't have an update on Ubuntu's AMI, we don't track those issues here.

We now install 2.x CLI instead of relying on the version available in the package manager. Unfortunately I don't have an update on Ubuntu's AMI, we don't track those issues here.

@cartermckinnon you install 2.x in the AWS EKS AMI, not the Ubuntu AMI. Just to clarify...

Correct; my point was the Ubuntu AMI is maintained by Canonical, not EKS. I've passed along this feedback.


[e2e]: generic push main default slsa3

Repo: https://github.com/slsa-framework/example-package/tree/main
Run: https://github.com/slsa-framework/example-package/actions/runs/2587404599
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.generic.push.main.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.generic.push.main.default.slsa3.yml
Trigger: push
Branch: main
Date: Thu Jun 30 03:57:58 UTC 2022

Repo: https://github.com/slsa-framework/example-package/tree/main
Run: https://github.com/slsa-framework/example-package/actions/runs/2594260225
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.generic.push.main.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.generic.push.main.default.slsa3.yml
Trigger: push
Branch: main
Date: Fri Jul  1 04:09:20 UTC 2022

Repo: https://github.com/slsa-framework/example-package/tree/main
Run: https://github.com/slsa-framework/example-package/actions/runs/2600253152
Workflow file: https://github.com/slsa-framework/example-package/tree/main/.github/workflows/e2e.generic.push.main.default.slsa3.yml
Workflow runs: https://github.com/slsa-framework/example-package/actions/workflows/e2e.generic.push.main.default.slsa3.yml
Trigger: push
Branch: main
Date: Sat Jul  2 04:05:41 UTC 2022
Tests are passing now. Closing this issue.


chore: export css minimizer webpack-plugin for user custom configuration

Summary
There are some constants to be exposed which is necessary for user custom-configuration in CssMinimizerWebpackPlugin.
import { defineConfig } from '@rsbuild/core';
import {
  pluginCssMinimizer,
  CssMinimizerWebpackPlugin,
} from '@rsbuild/plugin-css-minimizer';

export default defineConfig({
  plugins: [
    pluginCssMinimizer({
      pluginOptions: {
        minify: CssMinimizerWebpackPlugin.lightningCssMinify,
        minimizerOptions: {
          // ...
        }
      },
    }),
  ],
});

Related Links
no
Checklist

[ ] Tests updated.
[ ] Documentation updated.

If this usage is recommended, I can update the document later

If this usage is recommended, I can update the document later

Yeah we can document this.


SW-3509 Handle duplicate accession photo filenames

Initially, all the metadata about accession photos was stored in a single table
that had a unique constraint to enforce that a given accession could only have
one photo with a given filename. When the table was split up to add support for
nursery withdrawal photos, we lost that unique constraint and thus the server
started accepting uploads of accession photos with duplicate filenames.
When a client attempted to read one of those photos, the request would fail
because the server still assumed filenames were unique and a database query
that was expected to return a single row instead returned multiple rows.
Change the upload API so that uploading a photo overwrites any existing photo
with the same filename on the same accession. To handle the duplicates that
already exist in the database, update the database queries to only return the
newest file with a given filename.

Current dependencies on/for this PR:

main

PR #1126   👈

This comment was auto-generated by Graphite.

Updated this to have "overwrite existing file" semantics as we discussed in Slack. It's different enough to warrant a fresh review.

Existing duplicates shouldn't hurt anything. The code needs to be able to deal with duplicates at read time no matter what, since there will be a short period after the second copy of a file is uploaded and before the first copy is deleted where both of them will exist.
For context, there are 4 duplicate filenames on prod and 9 on staging.


Add '=' to URL param 'filter'

The API call was missing "=" in in the "filter" parameter.
After the fix, it no longer requests all the posts from the server but only those that have been updated from the last reindex time.

Working perfectly, thanks!


"Failed to parse quantity for ___" and some aren't showing the correct numbers either.

Current Behavior
A few items failed to parse the quantity. Also, the numbers of some items aren't recording properly.
Expected Behavior
The scanner to correctly exports the data
Reproducing a Behavior
Monitor - 3440x1440
Version - Latest
Resolutions used in-game - 1600x900 (used normally works) / 1920x1080 (troubleshoot only)
I always restart the game after changing the resolution so the columns/rows show properly which usually makes the scan go smoothly.
-Open the game, press escape, open InvKamera program, and then scan with only "Char Development Items" and "Materials" checked.
Device OS
Windows 10
Genshin Impact Version
3.5
Inventory Kamera Version
v1.3.8
Screen resolution, screen mode, and UI scale
My monitor's native and selected resolution is 3440x1440, using 1600x900 in-game,
Additional notes and remarks
No response
Evidence
Here is the debug report:
2023-04-08 23:07:37.128|INFO|InventoryKamera|Kamera initialized
2023-04-08 23:07:38.076|INFO|MainForm|Inventory Kamera version 1.3.8
2023-04-08 23:07:38.191|INFO|DatabaseManager|Checking for newer game data...
2023-04-08 23:07:38.621|DEBUG|DatabaseManager|Parsing wiki version history.
2023-04-08 23:07:38.622|DEBUG|DatabaseManager|Parsed release date 4/12/2023 12:00:00 AM for version 3.6
2023-04-08 23:07:38.622|DEBUG|DatabaseManager|Version 3.6 is announced but not released!
2023-04-08 23:07:38.622|INFO|MainForm|Current game data is up to date with data for 3.5
2023-04-08 23:07:56.873|INFO|MainForm|Starting scan
2023-04-08 23:07:56.873|INFO|MainForm|Hotkey registered
2023-04-08 23:07:56.873|INFO|MainForm|Scan settings:
Game Version Data:			 3.5
Weapons:				 False
Artifacts:				 False
Characters:				 False
Dev Items:				 True
Materials:				 True
Min Weapon Rarity:		 3
Min Weapon Level:		 1
Equip Weapons:			 True
Min Artifact Rarity:	 2
Min Artifact Level:		 0
Equip Artifacts:		 True
Delay:					 0
2023-04-08 23:07:56.873|DEBUG|Navigation|Checking for "GenshinImpact".exe
2023-04-08 23:07:56.892|DEBUG|Navigation|Found GenshinImpact.exe
2023-04-08 23:07:56.922|INFO|InventoryKamera|Kamera initialized
2023-04-08 23:07:56.922|INFO|MainForm|Resolution: 1600x900
2023-04-08 23:07:56.922|INFO|InventoryKamera|Logging directory reset
2023-04-08 23:07:57.566|INFO|Scraper|Scraper initialized
2023-04-08 23:07:57.566|DEBUG|InventoryKamera|Added 3 workers
2023-04-08 23:07:57.581|DEBUG|InventoryKamera|Thread #5 priority: Normal
2023-04-08 23:07:57.581|DEBUG|InventoryKamera|Thread #12 priority: Normal
2023-04-08 23:07:57.581|DEBUG|InventoryKamera|Thread #3 priority: Normal
2023-04-08 23:07:57.865|DEBUG|Scraper|8 Engines restarted
2023-04-08 23:07:57.865|INFO|Scraper|Internally set traveler custom name to thaddaios
2023-04-08 23:07:57.865|INFO|Scraper|Internally set wanderer custom name to gakkuri
2023-04-08 23:07:58.107|DEBUG|InventoryKamera|Thread 5 exit
2023-04-08 23:07:58.370|DEBUG|InventoryKamera|Thread 3 exit
2023-04-08 23:07:58.370|DEBUG|InventoryKamera|Thread 12 exit
2023-04-08 23:07:58.370|INFO|InventoryKamera|Scanning character development materials...
2023-04-08 23:08:01.951|DEBUG|MaterialScraper|Scanned mora input: System.Collections.Generic.List`1[System.String]
2023-04-08 23:08:01.951|DEBUG|MaterialScraper|Parsed mora input:
2023-04-08 23:08:01.951|ERROR|UserInterface|Unable to parse mora count
2023-04-08 23:08:02.554|DEBUG|MaterialScraper|Scanning material page 0
2023-04-08 23:08:02.554|DEBUG|MaterialScraper|Located 40 possible item locations on page.
2023-04-08 23:08:03.680|DEBUG|MaterialScraper|1010) -> 1010 -> 1010
2023-04-08 23:08:13.005|DEBUG|MaterialScraper|Finished page of materials. Scrolling...
2023-04-08 23:08:14.419|DEBUG|MaterialScraper|Scanning material page 1
2023-04-08 23:08:14.419|DEBUG|MaterialScraper|Located 40 possible item locations on page.
2023-04-08 23:08:24.880|DEBUG|MaterialScraper|Finished page of materials. Scrolling...
2023-04-08 23:08:26.286|DEBUG|MaterialScraper|Scanning material page 2
2023-04-08 23:08:26.286|DEBUG|MaterialScraper|Located 40 possible item locations on page.
2023-04-08 23:08:34.058|DEBUG|MaterialScraper|» ->  -> 0
2023-04-08 23:08:34.062|DEBUG|MaterialScraper|5] -> 5 -> 5
2023-04-08 23:08:34.687|DEBUG|MaterialScraper|© ->  -> 0
2023-04-08 23:08:34.687|ERROR|UserInterface|Failed to parse quantity for MirrorOfMushin
2023-04-08 23:08:36.249|DEBUG|MaterialScraper|zz ->  -> 0
2023-04-08 23:08:36.249|DEBUG|MaterialScraper|p74 -> 74 -> 74
2023-04-08 23:08:36.249|DEBUG|MaterialScraper|<< ->  -> 0
2023-04-08 23:08:36.249|DEBUG|MaterialScraper|<< ->  -> 0
2023-04-08 23:08:36.270|DEBUG|MaterialScraper|<Z ->  -> 0
2023-04-08 23:08:36.600|DEBUG|MaterialScraper|Finished page of materials. Scrolling...
2023-04-08 23:08:38.002|DEBUG|MaterialScraper|Scanning material page 3
2023-04-08 23:08:38.002|DEBUG|MaterialScraper|Located 40 possible item locations on page.
2023-04-08 23:08:48.568|DEBUG|MaterialScraper|Finished page of materials. Scrolling...
2023-04-08 23:08:49.994|DEBUG|MaterialScraper|Scanning material page 4
2023-04-08 23:08:49.994|DEBUG|MaterialScraper|Located 40 possible item locations on page.
2023-04-08 23:09:00.738|DEBUG|MaterialScraper|Finished page of materials. Scrolling...
2023-04-08 23:09:02.170|DEBUG|MaterialScraper|Scanning material page 5
2023-04-08 23:09:02.170|DEBUG|MaterialScraper|Located 40 possible item locations on page.
2023-04-08 23:09:11.219|ERROR|UserInterface|Failed to parse quantity for NarukamisAffection
2023-04-08 23:09:12.845|DEBUG|MaterialScraper|Finished page of materials. Scrolling...
2023-04-08 23:09:14.244|DEBUG|MaterialScraper|Scanning material page 6
2023-04-08 23:09:14.244|DEBUG|MaterialScraper|Located 32 possible item locations on page.
2023-04-08 23:09:14.409|DEBUG|MaterialScraper|Repeat material found. Scrolling until end
2023-04-08 23:09:17.691|DEBUG|MaterialScraper|Last material scanned, IronTalismanOfTheForestDew.
2023-04-08 23:09:19.713|INFO|InventoryKamera|Done scanning character development materials
2023-04-08 23:09:19.713|INFO|InventoryKamera|Scanning materials...
2023-04-08 23:09:23.712|DEBUG|MaterialScraper|Scanning material page 0
2023-04-08 23:09:23.712|DEBUG|MaterialScraper|Located 40 possible item locations on page.
2023-04-08 23:09:34.038|DEBUG|MaterialScraper|Finished page of materials. Scrolling...
2023-04-08 23:09:35.452|DEBUG|MaterialScraper|Scanning material page 1
2023-04-08 23:09:35.452|DEBUG|MaterialScraper|Located 40 possible item locations on page.
2023-04-08 23:09:45.648|DEBUG|MaterialScraper|Finished page of materials. Scrolling...
2023-04-08 23:09:47.061|DEBUG|MaterialScraper|Scanning material page 2
2023-04-08 23:09:47.061|DEBUG|MaterialScraper|Located 40 possible item locations on page.
2023-04-08 23:09:53.408|DEBUG|MaterialScraper|100) -> 100 -> 100
2023-04-08 23:09:57.167|DEBUG|MaterialScraper|Finished page of materials. Scrolling...
2023-04-08 23:09:58.576|DEBUG|MaterialScraper|Scanning material page 3
2023-04-08 23:09:58.576|DEBUG|MaterialScraper|Located 32 possible item locations on page.
2023-04-08 23:10:03.730|DEBUG|MaterialScraper|bl) ->  -> 0
2023-04-08 23:10:03.734|DEBUG|MaterialScraper|HU ->  -> 0
2023-04-08 23:10:04.370|DEBUG|MaterialScraper|HY ->  -> 0
2023-04-08 23:10:04.373|DEBUG|MaterialScraper|H3 -> 3 -> 3
2023-04-08 23:10:06.267|DEBUG|MaterialScraper|Finished page of materials. Scrolling...
2023-04-08 23:10:07.523|DEBUG|MaterialScraper|Scanning material page 4
2023-04-08 23:10:07.523|DEBUG|MaterialScraper|Located 32 possible item locations on page.
2023-04-08 23:10:07.657|DEBUG|MaterialScraper|Repeat material found. Scrolling until end
2023-04-08 23:10:09.917|DEBUG|MaterialScraper|» ->  -> 0
2023-04-08 23:10:09.921|DEBUG|MaterialScraper|5] -> 5 -> 5
2023-04-08 23:10:09.921|DEBUG|MaterialScraper|D ->  -> 0
2023-04-08 23:10:09.921|DEBUG|MaterialScraper|O ->  -> 0
2023-04-08 23:10:09.936|DEBUG|MaterialScraper|N/ ->  -> 0
2023-04-08 23:10:10.258|DEBUG|MaterialScraper|EF] ->  -> 0
2023-04-08 23:10:10.258|DEBUG|MaterialScraper|F ->  -> 0
2023-04-08 23:10:10.265|DEBUG|MaterialScraper|J ->  -> 0
2023-04-08 23:10:10.265|DEBUG|MaterialScraper|/ ->  -> 0
2023-04-08 23:10:10.265|DEBUG|MaterialScraper|Y/ ->  -> 0
2023-04-08 23:10:10.279|ERROR|UserInterface|Failed to parse quantity for FakeFlyBait
2023-04-08 23:10:11.202|DEBUG|MaterialScraper|Last material scanned, RedrotBait.
2023-04-08 23:10:13.212|INFO|InventoryKamera|Done scanning materials
2023-04-08 23:10:13.212|INFO|MainForm|Data converted to GOOD
2023-04-08 23:10:13.265|INFO|MainForm|Exported data
2023-04-08 23:10:13.265|INFO|MainForm|Hotkey removed
Here are the logging scans that failed:
Mora-

Mirror of Mushin -

Narukamis Affection -

FakeFlyBait -

As for the ones that got scanned:
materials": {"GlowingGem": 4, "HerosWit": 238, "AdventurersExperience": 938, "WanderersAdvice": 100, "SlimeConcentrate": 33, "SlimeSecretions": 107, "SlimeCondensate": 162, "OminousMask": 30, "StainedMask": 102, "DamagedMask": 71, "ForbiddenCurseScroll": 36, "SealedScroll": 110, "DiviningScroll": 542, "WeatheredArrowhead": 18, "SharpArrowhead": 30, "FirmArrowhead": 2, "BlackCrystalHorn": 28, "BlackBronzeHorn": 158, "HeavyHorn": 689, "LeyLineSprout": 23, "DeadLeyLineLeaves": 49, "DeadLeyLineBranch": 62, "ChaosCore": 1, "ChaosCircuit": 15, "MistGrassWick": 9, "MistGrass": 30, "MistGrassPollen": 125, "InspectorsSacrificialKnife": 9, "AgentsSacrificialKnife": 18, "HuntersSacrificialKnife": 23, "LieutenantsInsignia": 36, "SergeantsInsignia": 109, "RecruitsInsignia": 193, "GoldenRavenInsignia": 67, "SilverRavenInsignia": 214, "TreasureHoarderInsignia": 103, "EnergyNectar": 10, "ShimmeringNectar": 53, "WhopperflowerNectar": 36, "FossilizedBoneShard": 1, "SturdyBoneShard": 11, "FragileBoneShard": 105, "FamedHandguard": 11, "KageuchiHandguard": 8, "OldHandguard": 91, "ChaosOculus": 14, "ChaosAxis": 40, "ChaosGear": 275, "PolarizingPrism": 8, "CrystalPrism": 36, "DismalPrism": 115, "SpectralHeart": 11, "SpectralHusk": 12, "ConcealedTalon": 1, "ConcealedUnguis": 11, "ConcealedClaw": 68, "DarkStatuette": 1, "CrystallineCystDust": 6, "LuminescentPollen": 41, "FungalSpores": 95, "RobustFungalNucleus": 32, "DormantFungalNucleus": 130, "InactivatedFungalNucleus": 459, "TrimmedRedSilk": 26, "FadedRedSatin": 34, "ChaosBolt": 4, "ChaosModule": 29, "ChaosStorage": 107, "RadiantPrism": 16, "TurbidPrism": 45, "DamagedPrism": 184, "SturdyShell": 9, "DesiccatedShell": 31, "DvalinsSigh": 3, "TailOfBoreas": 8, "RingOfBoreas": 11, "SpiritLocketOfBoreas": 3, "TuskOfMonocerosCaeli": 8, "ShardOfAFoulLegacy": 13, "ShadowOfTheWarrior": 11, "DragonLordsCrown": 12, "BloodjadeBranch": 8, "GildedScale": 4, "MoltenMoment": 5, "HellfireButterfly": 6, "AshenHeart": 8, "MudraOfTheMaleficGeneral": 8, "TearsOfTheCalamitousGod": 10, "TheMeaningOfAeons": 5, "PuppetStrings": 4, "MirrorOfMushin": 6, "DakasBell": 2, "LightningPrism": 21, "BasaltPillar": 14, "HoarfrostCore": 2, "CleansingHeart": 22, "JuvenileJade": 2, "CrystallineBloom": 2, "MarionetteCore": 27, "PerpetualHeart": 1, "SmolderingPearl": 8, "StormBeads": 20, "DragonheirsFalseFin": 1, "RunicFang": 18, "MajesticHookedBeak": 2, "ThunderclapFruitcore": 2, "QuelledCreeper": 22, "PseudoStamens": 26, "BrilliantDiamondGemstone": 6, "AgnidusAgateChunk": 9, "AgnidusAgateFragment": 9, "AgnidusAgateSliver": 85, "VarunadaLazuriteChunk": 9, "VarunadaLazuriteSliver": 6, "NagadusEmeraldChunk": 9, "NagadusEmeraldFragment": 10, "NagadusEmeraldSliver": 1, "VajradaAmethystGemstone": 2, "VajradaAmethystChunk": 45, "VajradaAmethystFragment": 36, "VajradaAmethystSliver": 16, "VayudaTurquoiseChunk": 9, "VayudaTurquoiseFragment": 57, "VayudaTurquoiseSliver": 153, "ShivadaJadeChunk": 24, "ShivadaJadeFragment": 18, "ShivadaJadeSliver": 44, "PrithivaTopazGemstone": 1, "PrithivaTopazChunk": 15, "PrithivaTopazFragment": 41, "PrithivaTopazSliver": 181, "PhilosophiesOfFreedom": 12, "GuideToFreedom": 73, "TeachingsOfFreedom": 73, "PhilosophiesOfResistance": 18, "GuideToResistance": 55, "TeachingsOfResistance": 21, "PhilosophiesOfBallad": 5, "GuideToBallad": 28, "TeachingsOfBallad": 18, "PhilosophiesOfProsperity": 14, "GuideToProsperity": 310, "TeachingsOfProsperity": 16, "PhilosophiesOfDiligence": 11, "GuideToDiligence": 62, "TeachingsOfDiligence": 101, "PhilosophiesOfGold": 9, "GuideToGold": 25, "TeachingsOfGold": 47, "PhilosophiesOfTransience": 19, "GuideToTransience": 52, "TeachingsOfTransience": 27, "PhilosophiesOfElegance": 16, "GuideToElegance": 46, "TeachingsOfElegance": 24, "PhilosophiesOfLight": 10, "GuideToLight": 26, "TeachingsOfLight": 14, "PhilosophiesOfAdmonition": 9, "GuideToAdmonition": 32, "TeachingsOfAdmonition": 20, "PhilosophiesOfIngenuity": 7, "GuideToIngenuity": 50, "TeachingsOfIngenuity": 17, "PhilosophiesOfPraxis": 0, "GuideToPraxis": 9, "CrownOfInsight": 10, "FragmentOfDecarabiansEpic": 4, "DebrisOfDecarabiansCity": 1, "TileOfDecarabiansTower": 44, "BorealWolfsNostalgia": 1, "BorealWolfsCrackedTooth": 1, "BorealWolfsMilkTooth": 7, "ShacklesOfTheDandelionGladiator": 9, "ChainsOfTheDandelionGladiator": 0, "FettersOfTheDandelionGladiator": 21, "LustrousStoneFromGuyun": 0, "LuminousSandsFromGuyun": 5, "MistVeiledGoldElixir": 0, "MistVeiledMercuryElixir": 10, "MistVeiledLeadElixir": 52, "BitOfAerosiderite": 44, "PieceOfAerosiderite": 11, "GrainOfAerosiderite": 44, "JadeBranchOfADistantSea": 1, "JeweledBranchOfADistantSea": 20, "CoralBranchOfADistantSea": 1, "NarukamisAffection": 0, "NarukamisJoy": 41, "MaskOfTheTigersBite": 19, "MaskOfTheWickedLieutenant": 1, "IronTalismanOfTheForestDew": 19, "OasisGardensKindness": 19, "RemnantGlowOfScorchingMight": 19, "NorthlanderSwordBillet": 2, "NorthlanderClaymoreBillet": 1, "NorthlanderCatalystBillet": 2, "NorthlanderPolearmBillet": 2, "MidlanderSwordBillet": 1, "MidlanderClaymoreBillet": 1, "IronChunk": 48, "WhiteIronChunk": 719, "CrystalChunk": 1043, "MagicalCrystalChunk": 1, "Starsilver": 4, "AmethystLump": 13, "Wolfhook": 117, "Valberry": 299, "Cecilia": 64, "WindwheelAster": 108, "PhilanemoMushroom": 78, "SmallLampGrass": 137, "CallaLily": 113, "DandelionSeed": 36, "JueyunChili": 110, "NoctilucousJade": 82, "SilkFlower": 142, "GlazeLily": 176, "Qingxin": 115, "Starconch": 58, "Violetgrass": 26, "CorLapis": 9, "Onikabuto": 55, "SakuraBloom": 52, "CrystalMarrow": 137, "Dendrobium": 123, "NakuWeed": 33, "SeaGanoderma": 82, "SangoPearl": 15, "AmakumoFruit": 40, "FluorescentFungus": 20, "RukkhashavaMushrooms": 14, "Padisarah": 60, "NilotpalaLotus": 10, "KalpalataLotus": 13, "HennaBerry": 14, "SandGreasePupa": 107, "Scarab": 25, "Horsetail": 16, "MistFlowerCorolla": 12, "FlamingFlowerStamen": 13, "ElectroCrystal": 34, "ButterflyWings": 36, "Frog": 12, "LuminescentSpine": 32, "LizardTail": 62, "CrystalCore": 65, "LoachPearl": 5, "Viparyas": 39, "Mushroom": 147, "SweetFlower": 333, "Carrot": 5, "Radish": 201, "Snapdragon": 1, "Mint": 23, "Wheat": 190, "Cabbage": 95, "Pinecone": 113, "LotusHead": 1023, "Berry": 103, "RawMeat": 387, "BirdEgg": 501, "Matsutake": 1094, "Fowl": 102, "Crab": 9, "Salt": 79, "Onion": 85, "Pepper": 88, "Milk": 5, "Tomato": 114, "Potato": 41, "Fish": 5, "Tofu": 58, "BambooShoot": 12, "Rice": 2, "ShrimpMeat": 1, "ChilledMeat": 12, "EelMeat": 14, "GlabrousBeans": 100, "MysteriousMeat": 6, "Seagrass": 176, "LavenderMelon": 142, "HarraFruit": 85, "ZaytunPeach": 99, "SumeruRose": 87, "AjilenakhNut": 28, "CrabRoe": 10, "Cream": 20, "SmokedFowl": 20, "Butter": 12, "Cheese": 5, "Bacon": 10, "Sausage": 69, "Spice": 12, "BirchWood": 9, "CuihuaWood": 4, "PineWood": 16, "SandbearerWood": 43, "BambooSegment": 46, "FragrantCedarWood": 42, "FirWood": 36, "YumemiruWood": 32, "MapleWood": 15, "AraliaWood": 69, "OtogiWood": 46, "Brightwood": 2, "KarmaphalaWood": 50, "AdhigamaWood": 20, "AthelWood": 53, "Fabric": 43, "RedDye": 1, "YellowDye": 61, "BlueDye": 38, "FruitPasteBait": 4, "RedrotBait": 10, "FalseWormBait": 10, "FakeFlyBait": 9, "Medaka": 5, "GoldenKoi": 1
Here's a sample of some:

Preflight Checklist

[X] I have checked for similar issues in the issue page, and I have not found any similar to mine.
[X] I am using the latest version of the scanner.
[X] I tried setting the scanning delay to maximum, and it still doesn't work.
[ ] I have attached the video recording to the bug report.
[X] I have attached the error log to the bug report.
[X] I am available to later explains the issue.
[X] I have disabled all screen filtering software or plugins (i.e. HDR, Image Sharpening) while scanning.
[X] I use default in-game keybindings or have configured keybindings for the scanner.

This is a known and duplicate issue.


Fix the 10 minutes timeout issue

I have forked DashCast to implement a fix for the timeout problem:
https://github.com/swiergot/dashcast
It is registered at Google under ID 5C517DAC.
This pull request a) switches to this new DashCast version and b) adds a new option to cast_site to enable the fix (for backward compatibility the old behaviour remains the default).
Please note that due to security constraints the new method requires the casted website to be served over HTTPS.

Thank you! I'm a bit wary of relying on a third party for part of CATT's functionality, would it be possible to add that to CATT?
AFAIK we already have a custom receiver that's registered, it's on this branch:
https://github.com/skorokithakis/catt/blob/feat/custom-receiver/catt_receiver/receiver.html

I've deployed your fork to our custom receiver, would it be possible to deploy your code upstream for inclusion?

@swiergot how is your approach different from the iframe approach already present in dashcast?
As I recall, using an iframe means a lot of sites won't work.

@theychx I'm glad you showed up, I was out of my depth here :P

I've deployed your fork to our custom receiver, would it be possible to deploy your code upstream for inclusion?

Where exactly did you deploy it? The link you gave returns 404.
What upstream do you have in mind? https://github.com/stestagg/dashcast?

@swiergot how is your approach different from the iframe approach already present in dashcast? As I recall, using an iframe means a lot of sites won't work.

So now the options are:

DashCast in window.location mode (old or new)

works with all websites,
times out after 10 minutes (unless the target website is a receiver itself - that's what ha-catt-fix does).

Old DashCast in iframe mode

requires the target URL to be HTTPS,
doesn't work with websites that block iframe,
times out after 10 minutes.

New DashCast in iframe mode

requires the target URL to be HTTPS,
doesn't work with websites that block iframe,
does NOT time out after 10 minutes.

The third, new option will be a huge improvement for many people.

Where exactly did you deploy it? The link you gave returns 404.

It's at https://catt.stavros.io/, the link 404s because I moved the file when I added your fork to it.

What upstream do you have in mind? https://github.com/stestagg/dashcast?

Yes, though I now see it hasn't been updated in 6 years, so it'd probably be pointless.

The third, new option will be a huge improvement for many people.

I agree that having the option is worthwhile, even if it doesn't work for everyone. Since it's an option, they can choose "works with every site for 10 minutes" or "works with some sites for however long".

@swiergot OK, thanks for clarifying.

@theychx does this look good to you?

It's at https://catt.stavros.io/, the link 404s because I moved the file when I added your fork to it.

Ok, that looks fine.

Yes, though I now see it hasn't been updated in 6 years, so it'd probably be pointless.

Well, I've done it anyway, won't hurt (https://github.com/stestagg/dashcast/pull/21).

I agree that having the option is worthwhile, even if it doesn't work for everyone. Since it's an option, they can choose "works with every site for 10 minutes" or "works with some sites for however long".

It will work for people casting Home Assistant and similar.
Although after playing with it today I can see some weird Home Assistant UI issues (eg. mini-graph-card not showing the graph). I did not observe that with the ha-catt-fix method.

@skorokithakis I'm not much of a javascript expert, so cant give any immediate critique.
@swiergot Using cast_site repeatedly doesn't appear to be working
(in either mode using https://ard.de and https://bbc.co.uk)

@swiergot Using cast_site repeatedly doesn't appear to be working

Thanks, I will be on vacation for the next two weeks, will look into it afterwards.

Thanks, enjoy your vacation!


Process for Adding New System Collectives

This RFC proposes a means for Polkadot stakeholders to ratify a new collective. The Fellowship should agree to add new collectives to the Collectives parachain runtime that pass the given process.

ChaosDAO would also like to explore this space.

Looks Ok to me :) in general I think the approach of creating new instances of the different pallets that make up a collective like the fellowship is not very approachable nor scalable but I think this works as a start, at least the community will know there's a processes. To make the process easier/automatable for the fellowship I would be more strict about the format used for the submission and include some form of machine readable format that tools can use to auto generate the instantiation code and auto-submit a PR for the new collective. Markdown for the manifesto annotated with a "front matter" section has a good balance of being simple enough for the proposers(even if they don't use a tool) and still machine parsable.
---
collective: Great Collective
type: ranked
member_types: individuals
has_treasury: yes
ranks:
- name: simpletons  # used to generate origin and track
  track:  # overrides for the associated track default config
    decision_period: 5 days
- name: smarty-pants
  salary: 1 DOT
seeding:
- rank: 1
  account: 1universe1ruLer11111111111111111111111111111G91
---

# The Great Collective Manifesto
...

Later on as demand for collectives increases the fellowship might find worth it investing some dev hours creating a permission-less collectives system where new collectives are incubated continuously and every now and then one such collective might apply to become a "system collective" like the technical fellowship.

Later on as demand for collectives increases the fellowship might find worth it investing some dev hours creating a permission-less collectives system where new collectives are incubated continuously

As a new feature/chain this could be valuable, but this is not at all the purpose of system collectives. System collectives are meant to be permissioned; namely they need buy-in ("permission") from the network to be considered part of the Polkadot system, and as such need to justify their reason for existing and the powers they wish to acquire and get explicit approval from stakeholders.
The process is not meant to be scalable (although adding/configuring a few pallets is hardly a huge lift), it is meant to ensure that people have an accepted process to demonstrate that buy-in and a reasonable expectation that this group (the Fellowship) will honor that signal.

/rfc propose

/rfc process 0x4b54385ffd3ef633aa021d7c56d06de833071380f4ced0e747705dce1838f622


popup.html file edited as per design

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes #38
Type of change
Please delete options that are not relevant and insert x in the box to check the box.

popup.html file edited as per design

How Has This Been Tested?

Tested the popup file as per design on chrome.

Checklist:

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas

@garimasingh128
Please review my PR. Thank you.

@garimasingh128
Please review my PR. Thank you.

Conflicting files
HTML_files/popup.html
HTML_files/splash_screen.html
Please resolve these. And don't forget to mention me.

Conflicting files
HTML_files/popup.html
HTML_files/splash_screen.html
Please resolve these. And don't forget to mention me.

@garimasingh128
Check it now. Thank you.

@garimasingh128
Check it now. Thank you.

Good work!

Good work!


JavaScript workers are not properly working on Windows

Describe the bug
After adding a new end-to-end test suite on #134, the CI failed for JavaScript workers on Windows. I could reproduce the issue on a Windows machine and I'm researching the root cause now.
Reproduction steps

Download the latest version of wws
Download a basic JavaScript worker from the examples folder
Run it
Get a 503 error

Expected behavior
The workers returns the expected 200 response.
Additional context
Rust workers are working properly.

After an initial research, I found a "5, Access Denied" error when trying to open the folder that includes the worker source code in Windows. Note that for interpreted language workers like JavaScript, wws relies on saving the source code into a temporary folder that will mount later.
In the case of JavaScript, this is the specific method that opens it:
https://github.com/vmware-labs/wasm-workers-server/blob/5631fb400c03d3239daed03b523914b87717ad83/crates/runtimes/src/modules/javascript.rs#L50-L53
I need now to understand why wws cannot open the folder that was created by the same process before.

Finally, the issue relied on the differences when opening a directory using fs::File. This is a valid operation in Unix, while failing with AccessDenied error in Windows. A lesson learned here is that we cannot rely on fs::File for working with directories.


Implement ZoneMasterView

@dtinth 2 puzzles #43, #44 are still not solved.

@dtinth 2 puzzles #43, #51 are still not solved; solved: #44.

@dtinth the puzzle #43 is still not solved; solved: #44, #51.

@dtinth all 3 puzzles are solved here: #43, #44, #51.

Done!


[TIR] Expose TVM Backend API-related Builtins and Misc

op list:
tvm_thread_allreduce
tvm_access_ptr
tvm_throw_last_error
TVMBackendAllocWorkspace
TVMBackendFreeWorkspace
Used and tested for future TVMScript parser
Co-Authored-By: yongwww<EMAIL_ADDRESS>cc @junrushao1994

See our discussion thread here. Let's do one PR at a time :-)


Unnecessary escape character: \A  no-useless-escape

In this case I want to remove all symbols in the first line. But I get no-useless-escape. why this error is showing up ?
example outlet ++77 Outlet--

outletName.replace(/[^a-zA-Z0-9]+/g, '').replace(/(\A|_$)/, '')

You don't need to escape the A in the regex, are you trying to match a literal A?

This issue appears stale since it has been open for a while with no activity. I'm going to close it for now. If this is still issue, please feel free to leave a comment or open a new issue.


Update ecctl-getting-started.asciidoc

We should encourage users to use https endpoints, and ports accordingly.

I think we should also add a disclaimer that using https with a self-signed certificate (if it's the case) will require the --insecure flag.

Sure @marclop, I agree that statement should in the docs generally,
but for this specific init example I believe it should be implicit


Redirection failed!

outlook phishlet redirect urls are not working.
phishlets get-url outlook https://www.google.com

Not enough info. Closing.

i have tried the outlook phishlet and after successful login and after all authorization tokens intercepted!
it just not redirected to the desire website.
[17:59:45] [+++] [8] all authorization tokens intercepted!
[17:59:47] [imp] [8] redirecting to URL: https://example.com
[17:59:48] [imp] [8] redirecting to URL: https://example.com
[17:59:49] [imp] [8] redirecting to URL: https://example.com
[17:59:50] [imp] [8] redirecting to URL: https://example.com
[17:59:51] [imp] [8] redirecting to URL: https://example.com
[17:59:53] [imp] [8] redirecting to URL: https://example.com
[17:59:54] [imp] [8] redirecting to URL: https://example.com
[17:59:56] [imp] [8] redirecting to URL: https://example.com
[17:59:57] [imp] [8] redirecting to URL: https://example.com
[18:00:06] [imp] [8] redirecting to URL: https://example.com
[18:00:08] [imp] [8] redirecting to URL: https://example.com
[18:00:09] [imp] [8] redirecting to URL: https://example.com
[18:00:11] [imp] [8] redirecting to URL: https://example.com
[18:00:12] [imp] [8] redirecting to URL: https://example.com
[18:00:13] [imp] [8] redirecting to URL: https://example.com
[18:00:15] [imp] [8] redirecting to URL: https://example.com
[18:00:16] [imp] [8] redirecting to URL: https://example.com
[18:00:18] [imp] [8] redirecting to URL: https://example.com
[18:00:19] [imp] [8] redirecting to URL: https://example.com
[18:00:21] [imp] [8] redirecting to URL: https://example.com
[18:00:22] [imp] [8] redirecting to URL: https://example.com
[18:00:23] [imp] [8] redirecting to URL: https://example.com
[18:00:24] [imp] [8] redirecting to URL: https://example.com
[18:00:26] [imp] [8] redirecting to URL: https://example.com
[18:00:27] [imp] [8] redirecting to URL: https://example.com
[18:00:28] [imp] [8] redirecting to URL: https://example.com
[18:00:30] [imp] [8] redirecting to URL: https://example.com
[18:00:31] [imp] [8] redirecting to URL: https://example.com
[18:00:33] [imp] [8] redirecting to URL: https://example.com
[18:00:34] [imp] [8] redirecting to URL: https://example.com
[18:00:35] [imp] [8] redirecting to URL: https://example.com
[18:00:37] [imp] [8] redirecting to URL: https://example.com
[18:00:38] [imp] [8] redirecting to URL: https://example.com
[18:00:40] [imp] [8] redirecting to URL: https://example.com
[18:00:41] [imp] [8] redirecting to URL: https://example.com
[18:00:42] [imp] [8] redirecting to URL: https://example.com
[18:00:43] [imp] [8] redirecting to URL: https://example.com
[18:00:45] [imp] [8] redirecting to URL: https://example.com

i have tested this in chrome, firefox latest version.
on chrome it says
This page isn’t working
outlook.live.phishdomain.com redirected you too many times.
Try clearing your cookies.
ERR_TOO_MANY_REDIRECTS

Are you sure you are not redirecting to your phishing domain or original outlook domain that will be replaced with phishing domain on the fly?
I need to know exactly what is your redirect URL instead of a https://example.com placeholder, otherwise I can't help.

the exact url i tested with https and with http.
i tested https://google.com and https://live.com https://outlook.live.com
all these stuck, no redirect, and browser says too many redirects.
This was the End url in which that browser error message appears.
https://outlook.live.PhishingDomain.com/#nlp=1&gg=092b&pg=aHR0cHM6Ly9vdXRsb29rLmxpdmUuY29t&RpsCsrfState=96e78fd7-3f98-cffe-89c1-51fe173772d1&wa=wsignin1.0


Handling of bad mqtt host is unhelpful.

When the mqtt hostname is malformed, or the actual host is missing, the scada response and error reporting is unhelpful. In both cases the connect call just crashes. This might happen in the mqtt thread which might silently die. For a malformed host, e.g. "foo", we get socket.gaierror: [Errno 8] nodename nor servname provided, or not known. For a missing host, e.g. "www.foo.com", we get "TimeoutError: timed out", which is correct, but there is no appropriate propagation of the error. Both cases show "Task exception was never retrieved" on stderr.

This is a probably a duplicate of #155, in which a comm issue causing a paho crash was discovered in the field. Closing this, but re-open if #155 turns out to be only partially related.


Add controller folder and created callback to handle todo_posts with …

…validation & sanitization using express-validator. Updated front-end form with new label

I should double check the validation is working again for we merge this

I should double check the validation is working again before we merge this

Seems to be working. Also added another commit to this PR which includes related changes that were made to client side.


Update privacy-and-security.md

It's Protected Mode, not Protect Mode

Docs Build status updates of commit 5c2ee6d:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docs/concepts/privacy-and-security.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Remove duplicate extendedBackgroundTaskTime entry

There are two entries for the restricted capability "extendedBackgroundTaskTime." My proposal is to keep the one under the extended execution header and to merge the content of the two entries into that section to ensure none of it is lost.

@cc-cortes,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


DrawerToggle does not work as expected -It doesn't closes the DrawerNavigation

Current Behavior
The behavior of navigation.navigate('DrawerToggle')  is not as expected. On this example the menu button should open and close the drawer navigation. But it's only possible to open it. When the drawer is visible, the button does nothing.
const DrawerNavigation = StackNavigator({
  DrawerStack: {screen: DrawerStack},
}, {
  headerMode: 'float',
  navigationOptions: ({navigation}) => ({
    headerStyle: {backgroundColor: 'green'},
    title: 'Welcome!',
    headerTintColor: 'white',
    headerLeft: <Text onPress={() => {
      navigation.navigate('DrawerToggle')
    }}>Menu</Text>
  }),
});

With DrawerOpen and DrawerClose instead it's working
if (navigation.state.index === 0) {
  navigation.navigate('DrawerOpen')
} else {
  navigation.navigate('DrawerClose')
}

I thought DrawerToggle should be a replacement for this if block, so the drawer opens and closes through the button.
Your Environment

software
version

react-navigation
1.0.0-beta.13

react-native
0.47.2

node
8.6.0

npm or yarn
yarn

can you post full code? ideally an expo snack?

Sorry, it's not possible to post the full code, because it's a project for a customer. When creating the navigation, I followed this tutorial.
There in line 35 you can see the corresponding part.

@DFelten I meant a repository which reproduces the bug. Some thing that people can clone and run to see the error.

I've created an Expo Snack (my first one).
There you can see that it's only possible to open the Drawer with DrawerToggle

Having the same problem, any luck ?

@sunnixx
I'm using the mentioned workaround
if (navigation.state.index === 0) {
  navigation.navigate('DrawerOpen')
} else {
  navigation.navigate('DrawerClose')
}

I updated the Snack to the latest version and it works as expected: https://snack.expo.io/SJvS0p3Lz


.gitignore maintenance

OSX Finder leaves .DS_Store files around; make them ignored.
The mbedk64f target has been renamed to mbed a long time ago;
follow that change in .gitignore, too.

JerryScript-DCO-1.0-Signed-off-by: Akos Kiss<EMAIL_ADDRESS>

LGTM


Bankcoin Cash (BKC) Add Wallet

Token : Bankcoin Cash
Symbol : BKC
Contract :0xc88Be04c809856B75E3DfE19eB4dCf0a3B15317a
Logo Link : https://bankcoin-cash.com/bankcoincash.png
Logo Update

hi, you are suggested to follow this tutorial: https://github.com/consenlabs/token-profile#token-profile-guideline
to provide your token profile.


Add Linux SCHED_DEADLINE support

This adds rudimentary support for SCHED_DEADLINE on Linux systems. It is tested and working on Linux 5.4.0-84-generic #94-Ubuntu SMP Thu Aug 26 20:27:37 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux, but I see no reason it wouldn't work as far back as 3.14 when SCHED_DEADLINE was introduced.
 I also noticed some libc:: dependencies that aren't present on macOS, so I don't know that unix.rs is fully POSIX anyway.
The ThreadId passed in is interpreted as a pid_t instead of being translated properly (solutions to which are also hacky), since the syscall for deadline scheduling expects a kernel tid (userspace pid).
It interprets the priority as nanoseconds for all three deadline parameters, which is not very flexible. It more or less says "this task must run every X nanoseconds, with no flexibility."

Thanks much for the contribution! Is there a way we can test this policy by any chance?

Thanks much for the contribution! Is there a way we can test this policy by any chance?

Sure! I added a test for the deadline policy as-implemented (and cleaned up some formatting stuff - CLion doesn't default to rustfmt).

It looks like the 4.4 kernel on xenial (which is the GitHub Actions runner you're using) will refuse to admit any threads with deadline scheduling when runtime == deadline == period (even though that's supposed to be supported). I'm changing the interface to add a ThreadPriority::Deadline(u64, u64, u64) so the tests can specify values that the scheduler should consistently admit. Will probably push it up tomorrow.

Sorry, I got pulled into other work. Anyway, the tests should be passing now. I added the explicit deadline interface as described and tested (working) on Xenial.

Okay, now it should be working. I removed the clippy warning in that one test (I think it's more clear with the identity multiplication in this specific case) and fixed the CI issues (hopefully).

Enjoy https://crates.io/crates/thread-priority/0.3.0 and thanks much for the contribution!


Spurious analyzer error in some non-null safe subclasses of a null safe class

The smallest reproduction case I've been able to find so far is:
// A library migrated to null safety, foo.dart
class C {
  T? something<T>(covariant Object o, [T? Function()? callback = _default]) =>
      null;
}

Null _default() => null;

// @dart=2.11

import 'foo.dart';

class Bad extends C {}
//    ^^^ Error is here

The error is:

'C.something' ('T Function<T>(Object, [T Function()])') isn't a valid concrete implementation of 'C.something' ('T Function<T>(Object, [T Function()])'). • invalid_implementation_override

The CFE accepts this code.

https://dart-review.googlesource.com/c/sdk/+/212866


Adding instructions for Document Level Access control

Please provide us with the following information:

This issue is for a: (mark with an x)
- [ ] bug report -> please search issues before submitting
- [ x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Mention any other details that might be useful
Any plans to add instructions for enabling document level access control similar to what has been added to the python sample recently?

Thanks! We'll be in touch soon.

This would be a nice addition. Adding to Dec / January work item. We'd have to look into whether the process is the same with Azure Container Apps.

Additional resources -
https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/access-control-in-generative-ai-applications-with-azure/ba-p/3956408


Joey sprint update

This pull request separates the wrf spec, so it can be build with "--only dependencies"

Merging. Will move the env_apps/wrf.yaml to env_wrf/spack.yaml after merging.


fixed splash screen slightly jumps before hiding

Please review @Beamanator.
Details
Previously, We manage to fix this issue by making the status bar translucent from the android theme https://github.com/Expensify/Expensify.cash/pull/2567. but it caused a regression https://github.com/Expensify/Expensify.cash/issues/2577#issuecomment-827244046.
New Solution

Another way to fix this is by setting the status bar translucent, only after the Splash screen is hidden.

Fixed Issues

Fixes #2542
Tests / QA Steps

Kill the chat app if open
Relaunch the app
Notice that the Splash screen should fade away without jumping up or down.

Tested On

[ ] Web
[ ] Mobile Web
[ ] Desktop
[ ] iOS
[x] Android (affected platform)

Screenshots

Web

Mobile Web

Desktop

iOS

Android

https://user-images.githubusercontent.com/24370807/116290968-aee18f80-a7b1-11eb-88c7-c6fdcc07d205.mp4

@marcaaron would you mind also checking out these changes related to BootSplash since you added that code recently?

@Beamanator Updated. Thanks.

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to staging in version: 1.0.33-1🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


proposal: runtime: fall back to fair locks after repeated sleep-acquire failures

Go's locks make no guarantee of fairness. However, there exists at least one common pattern that exhibits severely unfair behavior, to the extent where it might be considered a serious bug. I propose we look into whether we can do anything in the sync.Mutex implementation that would ameliorate this severe unfairness without hurting the common case. I have one concrete suggestion at the bottom, but I hope to spur thought and discussion about this. I don't claim that this specific solution is the right one.
Problem
I've seen two different reports now of severe lock unfairness under roughly the same conditions, in two independent programs. The general problem occurs when there are two goroutines contending for a lock and one of them holds it for a long time while only briefly releasing it, and the other releases it for a long time and only briefly acquires it.
The demo program available by go get rsc.io/tmp/lockskew boils down to this:
// shared state
done := make(chan bool, 1)
var mu sync.Mutex

// goroutine 1
go func() {
	for {
		select {
		case <-done:
			return
		default:
			mu.Lock()
			time.Sleep(100 * time.Microsecond)
			mu.Unlock()
		}
	}
}()

// goroutine 2
for i := 0; i < n; i++ {
	time.Sleep(100 * time.Microsecond)
	mu.Lock()
	mu.Unlock()
}
done <- 1

Here, goroutine 1 basically wants to hold the lock all the time, releasing it briefly, while goroutine 2 only wants to hold the lock for brief amounts of time. Developers naively (but I don't think completely unreasonably) expect that although goroutine 2 may not get the lock quite as much as goroutine 1, it will get the lock once in a while, maybe after a second or two at worst. In fact, on my Linux workstation it's common for goroutine 2 to take 100+ seconds to acquire the lock just once.
When goroutine 1 executes mu.Unlock(), it marks the lock unlocked and wakes up goroutine 2, intending that goroutine 2 will run and acquire the lock itself. But goroutine 2 does not run immediately. Goroutine 1 keeps running, goes around the loop again, calls mu.Lock(), and reacquires the lock. By the time goroutine 2 actually runs and tries to acquire the lock, goroutine 1 has taken it back. The basic pattern is:
g1 holds lock, sleeping
	g2 calls mu.Lock
	- finds lock locked
	- adds itself to waiting queue
	- sleeps
g1 wakes up from time.Sleep
g1 calls mu.Unlock
- marks lock unlocked
- tells runtime g2 is allowed to run
- returns
g1 checks channel (never ready)
g1 calls mu.Lock
- finds lock unlocked
- acquires lock
g1 calls time.Sleep, sleeps
	g2 scheduled to run
	- finds lock locked
	- adds itself to waiting queue
	- sleeps

Given that g2 is woken by g1 only once per 100µs, the fact that g2 takes 100+ seconds to acquire the lock means that this cycle repeats millions of times before g1 is interrupted enough to let g2 run.
If you run lockskew it will acquire the lock as much as it can and print about acquisitions, at most once per second. The performance is sensitive to the specific length of the sleep, but they're all bad. On my system, here is lockskew with 100µs sleeps, as in the program above:
$ ./lockskew
2015/10/28 12:30:24 lock#0 took 180.773982s; average 180.773982s
2015/10/28 12:38:28 lock#1 took 484.603155s; average 332.688568s
2015/10/28 12:40:16 lock#2 took 107.852508s; average 257.743215s
2015/10/28 12:44:00 lock#3 took 223.592777s; average 249.205605s
2015/10/28 12:54:00 lock#4 took 600.067715s; average 319.378027s
2015/10/28 13:03:40 lock#5 took 579.896472s; average 362.797768s
2015/10/28 13:05:11 lock#6 took 91.223369s; average 324.001425s
2015/10/28 13:06:10 lock#8 took 59.019365s; average 258.622936s
2015/10/28 13:11:14 lock#9 took 303.023746s; average 263.063017s
2015/10/28 13:12:22 lock#10 took 68.975316s; average 245.418681s
2015/10/28 13:15:08 lock#11 took 165.555347s; average 238.763403s
2015/10/28 13:15:32 lock#12 took 23.713157s; average 222.221076s
2015/10/28 13:18:58 lock#13 took 205.748355s; average 221.044453s
2015/10/28 13:20:39 lock#14 took 101.183532s; average 213.053725s
2015/10/28 13:21:36 lock#15 took 56.859756s; average 203.291602s
2015/10/28 13:22:59 lock#16 took 83.056960s; average 196.218976s
2015/10/28 13:24:05 lock#17 took 66.815677s; average 189.029904s

And here is lockskew with 100ms sleeps:
$ ./lockskew -t 100ms
2015/10/28 13:26:01 lock#0 took 22.828276s; average 22.828276s
2015/10/28 13:28:04 lock#1 took 123.553891s; average 73.191083s
2015/10/28 13:31:17 lock#2 took 192.438909s; average 112.940359s
2015/10/28 13:31:24 lock#3 took 6.708422s; average 86.382375s
2015/10/28 13:33:04 lock#4 took 100.719121s; average 89.249724s
2015/10/28 13:34:06 lock#5 took 61.773499s; average 84.670353s
2015/10/28 13:34:11 lock#6 took 5.004274s; average 73.289485s
2015/10/28 13:34:14 lock#7 took 2.902557s; average 64.491119s
2015/10/28 13:34:17 lock#8 took 2.702412s; average 57.625707s
2015/10/28 13:34:34 lock#9 took 16.418097s; average 53.504946s
2015/10/28 13:34:39 lock#10 took 4.904286s; average 49.086704s
2015/10/28 13:35:31 lock#11 took 52.456904s; average 49.367554s
2015/10/28 13:37:03 lock#12 took 91.809099s; average 52.632288s
2015/10/28 13:37:17 lock#13 took 13.213271s; average 49.816644s
2015/10/28 13:37:23 lock#15 took 5.204634s; average 43.958641s

Barging vs Handoff
The fundamental problem here is when g1 decides to wake up g2, it leaves the sync.Mutex unlocked and lets g2 deal with reacquiring it. In the time between those two events, any goroutine other than g2 (here g1, but in general any other goroutine) can arrive at mu.Lock and grab the mutex instead of g2. This "drive-by acquisition" is sometimes called barging.
The alternative to barging is for g1 to leave the lock locked when waking g2, explicitly handing off the lock to g2. This coordinated handoff leaves the lock in the "locked" state the entire time, eliminating the window in which barging can occur.
In his work on java.util.concurrent, Doug Lea found that barging helped throughput [1], and that fact has entered the received wisdom surrounding locking. Following standard Java at the time, the evaluation used operating system threads and an operating system scheduler (specifically, Linux 2.4 NPTL on Pentium3/4 and Solaris 9 on UltraSparc2/3). As I understand it, barging is a win because it makes some bad operating system scheduling decisions impossible. Specifically, if the operating system delays the scheduling of g2, making the handoff delay long, not holding the lock during the handoff lets another thread do useful work in the interim.
As the test program and the real programs that inspired it show, however, at least in the Go implementation, barging enables very bad states. Also, the delays that barging helps avoid may not apply to Go, which is primarily using goroutines and a goroutine scheduler.
Alternatives
It's worth noting that there is a known workaround to this problem. If you use a pair of locks, with mu2 protecting mu.Lock, then that makes barging impossible in the two-goroutine case and at least less of an issue in the case of more goroutines. Specifically, the above code changes to:
// shared state
done := make(chan bool, 1)
var mu sync.Mutex
var mu2 sync.Mutex

// goroutine 1
go func() {
	for {
		select {
		case <-done:
			return
		default:
			mu2.Lock()
			mu.Lock()
			mu2.Unlock()
			time.Sleep(100 * time.Microsecond)
			mu.Unlock()
		}
	}
}()

// goroutine 2
for i := 0; i < n; i++ {
	time.Sleep(100 * time.Microsecond)
	mu2.Lock()
	mu.Lock()
	mu2.Unlock()
	mu.Unlock()
}
done <- 1

Now, because g1 sleeps without holding mu2, goroutine 2 can get mu2 and block in mu.Lock, and then when goroutine 1 goes back around its loop, it blocks on mu2.Lock and cannot barge in on goroutine 2's acquisition of mu.
If we take this approach in the lockskew test, it works much better:
$ ./lockskew -2
2015/10/28 12:27:17 lock#0 took 0.000163s; average 0.000163s
2015/10/28 12:27:18 lock#2910 took 0.000164s; average 0.000164s
2015/10/28 12:27:19 lock#5823 took 0.000163s; average 0.000164s
2015/10/28 12:27:20 lock#8727 took 0.000174s; average 0.000164s
2015/10/28 12:27:21 lock#11651 took 0.000166s; average 0.000164s

Note that there is only one print per second but the program is averaging over 2500 loop iterations per second, about 500,000X faster than above.
Ideally developers would not need to know about, much less use, this kind of hack. Another possibility is to wrap this idiom or some other implementation into a (say) sync.FairMutex with guaranteed fairness properties. But again ideally developers would not need to know or think about these awful details. I do not want to see blog posts explaining when to use sync.Mutex and when to use sync.FairMutex.
If Go's sync.Mutex can do something better by default without sacrificing performance, it seems it should.
Proposal
The proposal is that we detect severe unfairness and revert to handoffs in that case. How to do that is an open question, which is why this is not a design doc.
Performance
As an approximate lower bound on the cost of this in the general case, I added one line to the implementation of sync.Mutex's Unlock:
	// Grab the right to wake someone.
	new = (old - 1<<mutexWaiterShift) | mutexWoken
	if atomic.CompareAndSwapInt32(&m.state, old, new) {
		runtime_Semrelease(&m.sema)
		runtime.Gosched()                  <<< ADDED
		return
	}

Today at least, the implementation of Gosched essentially guarantees to run the goroutine just woken up by the Semrelease. This is not strictly speaking a handoff, since barging can still happen between the unlock and the acquisition in g2, but the handoff period is much smaller than it used to be, and in particular g1 is kept from reacquiring the lock. Anyway, the point is to evaluate how costly this might be, not to do a perfect job. A perfect job would require more lines changed in both sync.Mutex and the runtime (although perhaps not many).
With this line, lockskew with only a single lock runs at about the same speed as with two locks:
$ ./lockskew.fast
2015/10/28 13:49:02 lock#0 took 0.000155s; average 0.000155s
2015/10/28 13:49:03 lock#2934 took 0.000165s; average 0.000162s
2015/10/28 13:49:04 lock#5861 took 0.000163s; average 0.000162s
2015/10/28 13:49:05 lock#8783 took 0.000166s; average 0.000162s
2015/10/28 13:49:06 lock#11712 took 0.000162s; average 0.000162s

The pathological case is gone, but what about the common case? Barging is supposed to help throughput, so one expects that average throughput must have gone down. Doug Lea's paper uses as its benchmark multiple threads calling a random number generator protected by a lock. Each call acquires a lock, does a tiny amount of work, and releases the lock. The paper found that with 256 threads running on even single-processor systems, implementations allowing barging ran 10-100X faster than handoffs. Again this was with operating system threads, not goroutines, so the result may not carry over directly. A goroutine switch is much cheaper than a thread switch.
In fact, the result does not seem to carry over to goroutines, at least not in the random number case. Running go test -bench . rsc.io/tmp/lockskew will benchmark Go's rand.Int (with the same mutex around simple math) called from multiple goroutines simultaneously, from 1 to 256. As a worst case, I ran it with GOMAXPROCS=256 even though my system has only 12 cores. Here is the effect of adding the Gosched:
name         old time/op  new time/op  delta
Rand1-256    33.2ns ± 2%  32.8ns ± 1%   -1.24%   (p=0.008 n=10+9)
Rand2-256    40.4ns ±60%  33.4ns ±16%     ~     (p=0.203 n=10+11)
Rand4-256     143ns ± 9%   140ns ± 3%     ~      (p=0.098 n=9+11)
Rand8-256     230ns ± 4%   208ns ± 3%   -9.46%   (p=0.000 n=9+12)
Rand12-256    267ns ± 2%   236ns ± 1%  -11.81%  (p=0.000 n=10+11)
Rand16-256    272ns ± 1%   250ns ± 2%   -8.12%   (p=0.000 n=9+10)
Rand32-256    283ns ± 1%   269ns ± 2%   -4.92%  (p=0.000 n=10+10)
Rand64-256    284ns ± 2%   279ns ± 2%   -1.90%  (p=0.000 n=10+11)
Rand128-256   279ns ± 3%   280ns ± 2%     ~     (p=0.591 n=10+11)
Rand256-256   276ns ± 1%   288ns ± 2%   +4.23%  (p=0.000 n=10+10)

For the most part, the Gosched makes things better, not worse.
Although this microbenchmark doesn't show it, there may be a small degradation in real programs. I ran the golang.org/x/benchmarks/bench program with -bench=http with and without the change. The change seemed to cause a roughly 5% increase in the reported "RunTime" result.
Perhaps with work to detect and fall back instead of having the "handoff" on always, we could keep real programs running as they do today but fix these pathological (but real) behaviors at the same time. For example, one idea is to have the blocked mu.Lock call do the detection. In the pathological behavior, it goes to sleep, is awoken, and then can't acquire the lock and goes back to sleep. If it is woken up to find the lock unavailable (say) three times in a row, it could set a "now we're in handoff mode" bit in the lock word. The handoff mode bit could be sticky for the life of the lock, or it could be cleared on a call to mu.Unlock that finds no goroutines waiting.
[1] http://gee.cs.oswego.edu/dl/papers/aqs.pdf but ignore section 5.2. @RLH says the theoretical analysis there is known to be flawed.
/cc @randall77 @dvyukov @aclements @RLH @dr2chase

Some notes on the measuring strategy.
The same unfair algorithm is used on two levels: sync.Mutex and runtime semaphores. If you change only one of them you can get all kinds of weird non-linear effects that can both improve and worsen fairness and performance.
In the test you statically partition iterations among goroutines. And this strategy strongly prefers fair execution -- all goroutines need to finish at the same time. That can explain better numbers in benchmarks.
I've did a quick prototype of fully fair (FIFO) sync.Mutex and runtime sema:
https://go-review.googlesource.com/#/c/16469/
and changed the test to (RunParallel knows how to do it properly):
func benchmarkRand(b *testing.B, n int) {
	b.SetParallelism(n)
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			rand.Int()
		}
	})
}

Here are results on a silly 2 HT core notebook (and number of goroutines is not as high as in typical Go servers):
benchmark             old ns/op     new ns/op     delta
BenchmarkRand1        50.8          48.1          -5.31%
BenchmarkRand1-2      63.1          576           +812.84%
BenchmarkRand1-4      206           584           +183.50%
BenchmarkRand1-8      209           505           +141.63%
BenchmarkRand2        59.8          112           +87.29%
BenchmarkRand2-2      82.1          500           +509.01%
BenchmarkRand2-4      219           496           +126.48%
BenchmarkRand2-8      238           472           +98.32%
BenchmarkRand4        65.5          230           +251.15%
BenchmarkRand4-2      290           472           +62.76%
BenchmarkRand4-4      261           477           +82.76%
BenchmarkRand4-8      275           478           +73.82%
BenchmarkRand8        64.2          385           +499.69%
BenchmarkRand8-2      85.8          471           +448.95%
BenchmarkRand8-4      285           476           +67.02%
BenchmarkRand8-8      283           485           +71.38%
BenchmarkRand16       65.7          295           +349.01%
BenchmarkRand16-2     305           470           +54.10%
BenchmarkRand16-4     290           498           +71.72%
BenchmarkRand16-8     287           495           +72.47%

You can see that the things Doug is talking about are not fake. Even with GOMAXPROCS=1 slowdown is significant (that's our goroutine switching overhead).

I think we can do something along the lines of your proposal. However, we should not rely on absolute number of failed acquire tries. What matters for both user perceived latency and performance is real time (for performance it is probably frequency of acquisitions, but we can approximate it with real time). If we take a frequently acquired mutex (like the rand test), 3 is effectively the same as 1 (see the benchmark results). User does not care about number of mutex acquisitions, he cares about milliseconds.
10 ms looks like a good number to trigger hand off more (that's the GC latency guarantee and that's the goroutine preemption period).

@dvuykov, thanks for building that prototype and showing what I did wrong. :-)
You reported new benchmark results with two different things changed: both the implementation and the benchmark. What about your correct benchmark with my pseudo-fair implementation? Perhaps your results show only that we don't want complete fairness on by default (or perhaps even at all). My pseudo-fairness change fixed the real-world pathology with only a 5% slowdown in the macrobenchmarks (bench -bench=http), so maybe that's a better starting place.
If you wanted to switch the detection to time instead of number of spurious wakeups, how would you incorporate real time efficiently? My concern is that I didn't want to put new time calls into every Lock acquisition.

If 10ms is the Go preemption period, might we have a crude+cheap counter available based on that?  If the thread at the head of the queue is denied access 3 times in a row, then it can check that counter each denial after that, and if it changes twice, that's between 10 and 19.9 ms.

@dr2chase Sysmon thread can provide such counter, sysmon runs periodically and queries time anyway. However, my plan always was to get rid of sysmon entirely in favor of Windows UMS and Linux fibers.
@rsc Humm... context switch after unlock is exactly what we want to avoid. The micro benchmark does not test the real thing here. 2 points: (1) the context switch will destroy cache locality (in the benchmark there is no significant data), (2) the context switch will increase latency and memory consumption; consider that a server has 10 requests to process, each takes 1 ms to process, if you process them one-by-one to completion first request will be serviced in 1 ms, second in 2 ms, etc;  if you constantly round-robin between them, all requests will be serviced in 10 ms and working set the sum of 10 requests; servers need to prioritize the oldest work and get rid of it entirely as soon as possible to free all associated resources before switching to new work.
Also I suspect that in the benchmark most of the goroutines are blocked on the mutex, so there is no significant number of runnable goroutines and you basically switch directly to the just unblocked goroutine. This is faster and also gives false sense of fairness.

In multiple goroutines, the effect is still there, just not as severe as 2 goroutines illustrated in the test. I think the fundamental cause is probably that the scheduler is always favor to keep the runnable goroutine keep running, instead of even looking at the goroutines in the waiting queue that some might also just ready to run...
I think in Linux there is third queue (if my memory still correct) that used to describe threads that can actually be run (don't block on anything) but just not yet to be scheduled to run. On a scheduling point, all of them are just relatively fair picked. (don't know exactly algorithm but I'd imagine just a randomly pick would provide some basic fairness). I am wondering if this could be somewhat useful in Go, as it seems a very common case where a bunch of goroutines would block on waiting some lock and there is one that's doing something and holding the lock. Upon the time when the holder releases the lock, all of these parked goroutines that are waiting on the same lock should be just in the runnable queue and be fair pick along with the runnable. I think the Gosched() is pretty much just doing this artificially.
Although dvyukov@ also has the point of cache destroy... but hmm I am wondering if generally seconds of block of preventing from continuing doing work would actually be much worse than ctx swtiches in microseconds (and we aren't even talking about constant context switches, in practice, you don't lock and unlock right after that...).

I think in Linux there is third queue (if my memory still correct) that used to describe threads that can actually be run

Go scheduler has just one queue, and it holds exactly such goroutines. The queue is services in a fair manner (FIFO).

in practice, you don't lock and unlock right after that...

It happens quite frequently in practice, e.g. in some helper function that is called frequently and takes a mutex internally.

Inspiration might be found in an older version of the Solaris Nevada sources:
"LIFO queue ordering is unfair and can lead to starvation, but it gives better performance for heavily contended locks..."
https://github.com/alhazred/onarm/blob/2b3c7a498208e08afc7734be7f08cdad7ac51fbe/usr/src/lib/libc/port/threads/synch.c#L569
Obviously, this applies to OS-level thread scheduling, but the principles should remain applicable.

@rsc ping!

NB for your consideration: we have seen this in production in a similar variation:
We have many incoming requests contending on a lock protecting a long-running (order of milliseconds) journal transaction. The lock unfairness stretched the long tail into hours(!). The solution was to track the waiters explicitly in a queue and guard the queue with a mutex, this sorted out nearly all of the unfairness.

I wrote initially "The proposal is that we detect severe unfairness and revert to handoffs in that case. How to do that is an open question, which is why this is not a design doc." That's still true, unfortunately. We should probably plan to investigate this for Go 1.9 though. Thanks for the ping.

Right, would be great if locks didn't observe perfect fairness, but nevertheless followed the principle of least surprise. Thanks for keeping the issue open.

We don't know what to do here. It seems like maybe I should retract this proposal, but this is a problem for some people, and it would be nice to solve it before they realized they had it (that is, not require people to type things like "FairMutex").
Does anyone have any suggestions for what we might do?
/cc @dvyukov @aclements @RLH @dr2chase

Do something crudely random, say allow barging (N-1)/N of the time, hand off 1/N?  Pick N a power of two, use a super-crude (i.e., fast) RNG and just check for the lower log2(N) bits zero.

It seems like the unfairness only becomes a noticeable problem when the lock is held for a "very long time" (or, in other words, when the lock holder reschedules while holding the lock). You could detect that condition by looking at how long the lock is held on average, or by looking at the lock queue depth (or the wait time of the goroutine at the head of the queue).

In other words, if the semaphore was fair (I know it's not now), then you could choose lock handoff if the next goroutine on the semaphore had been waiting for longer than 10ms.

Since Dmitriy figured out a way to do this, proposal accepted.


Add linting to ensure line-breaks between "block-like" statements

This ensures "block-like" statements are padded with a blank line unless they are variable declarations, using the padding-line-between-statements-rule introduced in eslint 4.0.0.
"block-like" means "statements that the last token is the closing brace of blocks; e.g. { }, if (a) { }, and while (a) { }."
This will pass for example:
var foo = "bar";
var baz = function() {
  ...
};

And also:
if (expression) {
  ...
}
else if (expression2) {
  ...
}

But not:
if (expression) {
  ...
}
if (expression2) {
  ...
}

or:
var foo = function() {
  ...
};
foo();

Limitations:
Does not apply to object methods. There is a separate issue/suggestion for that called "padded-object", but it's not implemented (and another one for es6 class methods that is implemented, but we can't use that).
I added the rule as warning (since all the existing rules were). It's a single line in ".eslintrc.json", which could be seen as a bit of a contradiction.

Thank you!


Get started button on the documentation page leads to Page not found :(

Steps to replicate:

Navigate to the documentation link from the main website
Click on Get Started

Expected: We get brought to a relevant webpage (perhaps this one? https://docs.open-metadata.org/v1.1.x/quick-start)
Actual: Page not found :(

hi @sylthiel, do you mean this button?

Just reviewed it and it redirects fine for me. Marking this as closed. Feel free to reopen if I missed something. Thanks