PowerShell 7.1.3 on Raspberry Pi cannot write to writable network file

Steps to reproduce
Context is: PowerShell 7.1.3 running on Raspberry Pi 2B+ with a network mount off an Ubuntu 20.04.2 LTS box running Samba Version 4.11.6-Ubuntu
PS /home/pi> lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 10 (buster)
Release:        10
Codename:       buster
PS /home/pi> cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
PS /home/pi> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.1.3
PSEdition                      Core
GitCommitId                    7.1.3
OS                             Linux 5.10.17-v7+ #1403 SMP Mon Feb 22 11:29:51 GMT 2021
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

pi@jeedom:~ $ resolveip pnjnas
IP address of pnjnas is <IP_ADDRESS>
pi@jeedom:~ $ mount | fgrep <IP_ADDRESS>
//<IP_ADDRESS>/pi on /pnjnas/pi type cifs (rw,nosuid,nodev,noexec,relatime,vers=2.0,cache=strict,username=pi,uid=0,noforceuid,gid=0,noforcegid,addr=<IP_ADDRESS>,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,_netdev)
//<IP_ADDRESS>/Qdownload on /pnjnas/Qdownload type cifs (rw,nosuid,nodev,noexec,relatime,vers=2.0,cache=strict,username=pi,uid=0,noforceuid,gid=0,noforcegid,addr=<IP_ADDRESS>,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,_netdev)
//<IP_ADDRESS>/public on /pnjnas/public type cifs (rw,nosuid,nodev,noexec,relatime,vers=2.0,cache=strict,username=pi,uid=0,noforceuid,gid=0,noforcegid,addr=<IP_ADDRESS>,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,_netdev)
//<IP_ADDRESS>/sto on /pnjnas/sto type cifs (rw,nosuid,nodev,noexec,relatime,vers=2.0,cache=strict,username=sto,uid=0,noforceuid,gid=0,noforcegid,addr=<IP_ADDRESS>,file_mode=0777,dir_mode=0777,soft,nounix,serverino,mapposix,rsize=65536,wsize=65536,bsize=1048576,echo_interval=60,actimeo=1,_netdev)
pi@jeedom:~ $ ls -l /pnjnas/sto/disks/Health/foo.txt
-rwxrwxrwx 1 root root 4 May  2 21:09 /pnjnas/sto/disks/Health/foo.txt
pi@jeedom:~ $ ls -ld /pnjnas/sto/disks/Health
drwxrwxrwx 2 root root 0 May  2 21:06 /pnjnas/sto/disks/Health
pi@jeedom:~ $ echo "foo" >/pnjnas/sto/disks/Health/foo.txt
# no error since file is writable
pi@jeedom:~ $ ls -l /pnjnas/sto/disks/Health/foo.txt
-rwxrwxrwx 1 root root 4 May  3 14:16 /pnjnas/sto/disks/Health/foo.txt
pi@jeedom:~ $ cat /pnjnas/sto/disks/Health/foo.txt
foo

Writing to that same file from PowerShell doesn't work:

"foo" | out-file /pnjnas/sto/disks/Health/foo.txt

Expected behavior
The file /pnjnas/sto/disks/Health/foo.txt should be written to.
Actual behavior

Out-File: Access to the path '/pnjnas/sto/disks/Health/foo.txt' is denied.

$ ls -l /pnjnas/sto/disks/Health/foo.txt
-rwxrwxrwx 1 root root 0 May  3 14:17 /pnjnas/sto/disks/Health/foo.txt

Environment data

@anmenaga and @rjmholt Any idea on what might cause this issue?

I can replicate this myself and thought I would share more information. One thing to note that was briefly mentioned at the end is the file is created it just has no content.
What I've found is a few problems that cause other problems so I'll try and break it down.
Issue 1 - Out-File fails (this issue)
Out-File fails, the file is created but is 0 length (no content).
PS /home/jborean/dev> "foo" | Out-File /mnt/test_smb/folder/foo.txt
Out-File: Access to the path '/mnt/test_smb/folder/foo.txt' is denied.

PS /home/jborean/dev> $Error[0] | Select *

PSMessageDetails      :
Exception             : System.UnauthorizedAccessException: Access to the path '/mnt/test_smb/folder/foo.txt' is denied.
                         ---> System.IO.IOException: Permission denied
                           --- End of inner exception stack trace ---
                           at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source)
                           at System.IO.FileStream.FlushWriteBuffer()
                           at System.IO.FileStream.FlushInternalBuffer()
                           at System.IO.FileStream.Flush(Boolean flushToDisk)
                           at System.IO.FileStream.Flush()
                           at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
                           at System.IO.StreamWriter.Flush()
                           at Microsoft.PowerShell.Commands.OutFileCommand.ProcessRecord() in
                        /PowerShell/src/Microsoft.PowerShell.Commands.Utility/commands/utility/FormatAndOutput/out-file/Out-File.cs:line 252
                           at System.Management.Automation.Cmdlet.DoProcessRecord() in /PowerShell/src/System.Management.Automation/engine/cmdlet.cs:line 173
                           at System.Management.Automation.CommandProcessor.ProcessRecord() in /PowerShell/src/System.Management.Automation/engine/CommandProcessor.cs:line 388
TargetObject          :
CategoryInfo          : NotSpecified: (:) [Out-File], UnauthorizedAccessException
FullyQualifiedErrorId : System.UnauthorizedAccessException,Microsoft.PowerShell.Commands.OutFileCommand
ErrorDetails          :
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

PS /home/jborean/dev> ls -al /mnt/test_smb/folder/
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 06:29 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
-rwxr-xr-x. 1 jborean jborean    0 May  4 06:29 foo.txt

I'm looking at the commands mentioned in the stack trace and it seems like Out-File essentially does the following in .NET which also fails
$fs = [IO.File]::Open("/mnt/test_smb/folder/foo.txt", "Create",  "Write", "Read")
$sw = [IO.StreamWriter]::new($fs)
$sw.WriteLine("foo")
$sw.Dispose()

# Fails with
MethodInvocationException: Exception calling "Dispose" with "0" argument(s): "Access to the path '/mnt/test_smb/folder/foo.txt' is denied."
PS /home/jborean/dev> $Error[0] | Select *

PSMessageDetails      :
Exception             : System.Management.Automation.MethodInvocationException: Exception calling "Dispose" with "0" argument(s): "Access to the path '/mnt/test_smb/folder/foo.txt'
                        is denied."
                         ---> System.UnauthorizedAccessException: Access to the path '/mnt/test_smb/folder/foo.txt' is denied.
                         ---> System.IO.IOException: Permission denied
                           --- End of inner exception stack trace ---
                           at System.IO.FileStream.WriteNative(ReadOnlySpan`1 source)
                           at System.IO.FileStream.FlushWriteBuffer()
                           at System.IO.FileStream.Dispose(Boolean disposing)
                           at System.IO.Stream.Close()
                           at System.IO.StreamWriter.CloseStreamFromDispose(Boolean disposing)
                           at System.IO.StreamWriter.Dispose(Boolean disposing)
                           at CallSite.Target(Closure , CallSite , Object )
                           --- End of inner exception stack trace ---
                           at System.Management.Automation.ExceptionHandlingOps.ConvertToMethodInvocationException(Exception exception, Type typeToThrow, String methodName, Int32
                        numArgs, MemberInfo memberInfo) in /PowerShell/src/System.Management.Automation/engine/runtime/Operations/MiscOps.cs:line 2068
                           at CallSite.Target(Closure , CallSite , Object )
                           at System.Dynamic.UpdateDelegates.UpdateAndExecute1[T0,TRet](CallSite site, T0 arg0)
                           at System.Management.Automation.Interpreter.DynamicInstruction`2.Run(InterpretedFrame frame) in
                        /PowerShell/src/System.Management.Automation/engine/interpreter/DynamicInstructions.Generated.cs:line 141
                           at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) in
                        /PowerShell/src/System.Management.Automation/engine/interpreter/ControlFlowInstructions.cs:line 358
TargetObject          :
CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
FullyQualifiedErrorId : UnauthorizedAccessException
ErrorDetails          :
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : at <ScriptBlock>, <No file>: line 1
PipelineIterationInfo : {}

This is very weird because there isn't any failure in the SMB traffic returned from the server so Samba is doing something to return access denied for some reason. There's no attempt to write to the file, it just opens the handle, sends some set commands to change attributes but no write request. This may be an issue for the .NET team because it's not just PowerShell.
What is even more curious is this issue seems to be related to the FileShare value. I can run the same .NET functions from above successfully if I omit the FileShare parameter or set it to None. Adding even more confusion to the pot is that Samba seems to be setting all of the FileShare values on the actual SMB open packet so something might be up with the client side share logic it has. This will require further investigation to figure out how FileShare actually operates on Linux and how .NET passes along that information to the native syscalls, i.e. .NET -> Samba.
Issue 2 - Add/Set-Content silently fails
Using Set-Content instead also has some confusing behaviour, it will not output an error but these cmdlets will create the file if it doesn't exist but will not set the content.
PS /home/jborean/dev> ls -al /mnt/test_smb/folder
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 06:56 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..

PS /home/jborean/dev> Set-Content /mnt/test_smb/folder/foo.txt "foo"

PS /home/jborean/dev> ls -al /mnt/test_smb/folder
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 06:57 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
-rwxr-xr-x. 1 jborean jborean    0 May  4 06:57 foo.txt

PS /home/jborean/dev> ls -al /mnt/test_smb/folder
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 06:58 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
PS /home/jborean/dev> Add-Content /mnt/test_smb/folder/foo.txt "foo"
PS /home/jborean/dev> ls -al /mnt/test_smb/folder
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 06:58 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
-rwxr-xr-x. 1 jborean jborean    0 May  4 06:58 foo.txt

So things seem to be ok but in reality the commands didn't work.
Issue 3 - Add/Set-Content does not close the file handle
At least with the example above, the SMB open is not closed when the commands are finished. This leaves a handle still opened on the SMB server causing issues when subsequent requests go to open the file with exclusive access or the file is deleted. Using the delete example we can see that the file isn't actually deleted until all the handles are closed (pwsh is closed/GC is collected)
PS /home/jborean/dev> Set-Content /mnt/test_smb/folder/foo.txt "value"

PS /home/jborean/dev> ls -al /mnt/test_smb/folder/
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 07:04 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
-rwxr-xr-x. 1 jborean jborean    0 May  4 07:04 foo.txt

PS /home/jborean/dev> Remove-Item /mnt/test_smb/folder/foo.txt

PS /home/jborean/dev> ls -al /mnt/test_smb/folder/
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 07:04 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..
-rwxr-xr-x. 0 jborean jborean    0 May  4 07:04 foo.txt

PS /home/jborean/dev> [GC]::Collect(); [GC]::WaitForPendingFinalizers()

PS /home/jborean/dev> ls -al /mnt/test_smb/folder/
total 8
drwxr-xr-x. 2 jborean jborean    0 May  4 07:05 .
drwxr-xr-x. 2 jborean jborean 8192 May  4 06:26 ..

You can see that the file is still present after the Remove-Item because the handle opened by Set-Content wasn't closed and it's only until the garbage collector has collected those handles that the file is actually deleted. Because this is dependent on when the GC runs you might get lucky and have it dispose of the handles relatively quickly but you cannot guarantee it will always run without explicitly calling it.
This can cause confusing errors if you try to then write to the file again that's still pending a delete
PS /home/jborean/dev> Set-Content /mnt/test_smb/folder/foo.txt "value"

PS /home/jborean/dev> Remove-Item /mnt/test_smb/folder/foo.txt

PS /home/jborean/dev> Set-Content /mnt/test_smb/folder/foo.txt "value"
Set-Content: Could not find file '/mnt/test_smb/folder/foo.txt'.

It states it couldn't find the file even though you want to create it with Set-Content. This is because the SMB response to opening that file returns STATUS_DELETE_PENDING because the original file still has a handle open to it (from the first Set-Content command).
It seems like there might be a stray handle in the code that Add/Set-Content calls that should be explicitly scoped with a using (...) where it is called.

Good findings @jborean93, thanks! It's really strange that Add/Set-Content fails silently.

@sba923 Since you have the repro environment, can you please file an issue in dotnet/runtime?
@jborean93 shared the C# repro code (written in ps script) in his post:

$fs = [IO.File]::Open("/mnt/test_smb/folder/foo.txt", "Create",  "Write", "Read")
$sw = [IO.StreamWriter]::new($fs)
$sw.WriteLine("foo")
$sw.Dispose()

Opened https://github.com/dotnet/runtime/issues/53182

https://github.com/dotnet/runtime/issues/53182 is now resolved. How is this going to rippple into PowerShell?

I guess we get this in .Net Runtime 6.0 Preview7.

So could make it in 7.2.0-preview.9?

MSFT team works on moving to  .Net Runtime 6.0 Preview7.

Issue also seen on Debian 11. Issue #16056
Cutouts when the dotnet fix is planning to be consumed?

Issue is still present in PowerShell 7.3.0-preview.1

Doesn't the example work too?
$fs = [IO.File]::Open("/mnt/test_smb/folder/foo.txt", "Create",  "Write", "Read")
$sw = [IO.StreamWriter]::new($fs)
$sw.WriteLine("foo")
$sw.Dispose()

I see you continue investigations with .Net team.

Doesn't the example work too?
$fs = [IO.File]::Open("/mnt/test_smb/folder/foo.txt", "Create",  "Write", "Read")
$sw = [IO.StreamWriter]::new($fs)
$sw.WriteLine("foo")
$sw.Dispose()

This fails too:

Having the same issue. PS version info below.
Name                           Value
----                           -----
PSVersion                      7.2.1
PSEdition                      Core
GitCommitId                    7.2.1
OS                             Linux 5.10.63-v7+ #1459 SMP Wed Oct 6 16:41:10 …
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

I have tried all the above examples and all have failed. Any word on if the MS team has plans to address this issue soon?

The core issue has been fixed by https://github.com/dotnet/runtime/issues/61175, now we have to wait until this ripples up to PowerShell 7.2 (via a backport of the fix to .NET 6) and PowerShell 7.3.

FWIW, problem is (expectedly) still present in 7.3.0-preview.5

Based on @adamsitnik's comment in https://github.com/dotnet/runtime/issues/61175, the next steps are:

check if issue is fixed in 7.3.0-preview.6 when available
backport of the fix for https://github.com/dotnet/runtime/issues/61175 in .NET 7 to .NET 6
release of some 7.2.x LTS version incorporating said backport

@adamsitnik I confirm the problem's no longer present in 7.3.0-preview.6:

An issue will be closed when it's confirmed fixed. The preview PowerShell will eventually move to GA release.

An issue will be closed when it's confirmed fixed. The preview PowerShell will eventually move to GA release.

Thanks for the clarification.
This does make sense but just doesn't align with what I used to live with during my 25 R&D years where, considering the possibility of a regression between a test version and GA, customer-reported issues were closed only after having been tested as fixed on the next GA release.

@adamsitnik given that this issue doesn't exist anymore in PowerShell 7.3.1, it's time to backport the underlying .NET fix to 6.0, see https://github.com/dotnet/runtime/issues/61175#issuecomment-1187606602


Error encountered during build of chefPrepEnv

AWS ParallelCluster version: 3.1.2
Cluster Configuration:
Region: us-east-1
Image:
  Os: alinux2
HeadNode:
  InstanceType: t2.micro
  Networking:
    SubnetId: subnet-0aa9d3bd709f86d50
    SecurityGroups:
    - sg-045290f659c3be158
  Ssh:
    KeyName: pcluster
  Iam:
    InstanceRole: arn:aws:iam::716453263077:role/AmazonSSMRoleForAutomationAssumeQuickSetup

Scheduling:
  Scheduler: slurm
  SlurmSettings:
    Dns:
      DisableManagedDns: true
      UseEc2Hostnames: true
  SlurmQueues:
  - Name: oar
    ComputeResources:
    - Name: t2micro
      InstanceType: t2.micro
      MinCount: 1
      MaxCount: 10
    Networking:
      SubnetIds:
      - subnet-0aa9d3bd709f86d50
      SecurityGroups:
      - sg-045290f659c3be158
    Iam:
      InstanceRole: arn:aws:iam::716453263077:role/AmazonSSMRoleForAutomationAssumeQuickSetup

Cluster name: test-cluster
Describe Cluster Output:
{
  "creationTime": "2022-03-18T16:13:42.670Z",
  "headNode": {
    "launchTime": "2022-03-18T16:16:21.000Z",
    "instanceId": "i-0d46daa16277ef29b",
    "instanceType": "t2.micro",
    "state": "running",
    "privateIpAddress": "<IP_ADDRESS>"
  },
  "version": "3.1.2",
  "clusterConfiguration": {
    "url": "https://parallelcluster-e2ca1557272da85b-v1-do-not-delete.s3.amazonaws.com/parallelcluster/3.1.2/clusters/test-cluster-2gk3wqecfyzsjmvm/configs/cluster-config.yaml?versionId=yLZzRZXzmMhqwpQc2YyqZWL6ypW5MnvJ&AWSAccessKeyId=XXXXX&Signature=XXXX&Expires=1647624263"
  },
  "tags": [
    {
      "value": "3.1.2",
      "key": "parallelcluster:version"
    }
  ],
  "cloudFormationStackStatus": "CREATE_FAILED",
  "clusterName": "test-cluster",
  "computeFleetStatus": "UNKNOWN",
  "cloudformationStackArn": "arn:aws:cloudformation:us-east-1:716453263077:stack/test-cluster/6120e280-a6d6-11ec-b9af-0e4b061813b3",
  "lastUpdatedTime": "2022-03-18T16:13:42.670Z",
  "region": "us-east-1",
  "clusterStatus": "CREATE_FAILED"
}

Arn of the cluster CloudFormation main stack: See Above
Bug description and how to reproduce:
Used the above configuration and the chef script failed.
chef-client.log
cfn-init.log
cloud-init.log
Additional context:
Creating cluster on pre-existing private VPC/subnet with a role we use for all of our instances.
The Role includes policies
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateInstanceProfile",
                "iam:ListInstanceProfilesForRole",
                "iam:PassRole",
                "ec2:DescribeIamInstanceProfileAssociations",
                "iam:GetInstanceProfile",
                "ec2:DisassociateIamInstanceProfile",
                "ec2:AssociateIamInstanceProfile",
                "iam:AddRoleToInstanceProfile"
            ],
            "Resource": "*"
        }
    ]
}

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:ListRoles",
                "config:DescribeConfigurationRecorders",
                "compute-optimizer:GetEnrollmentStatus",
                "support:DescribeTrustedAdvisorChecks"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:UpdateServiceSetting",
                "ssm:GetServiceSetting"
            ],
            "Resource": [
                "arn:aws:ssm:*:*:servicesetting/ssm/opsitem/ssm-patchmanager",
                "arn:aws:ssm:*:*:servicesetting/ssm/opsitem/EC2",
                "arn:aws:ssm:*:*:servicesetting/ssm/opsdata/ExplorerOnboarded",
                "arn:aws:ssm:*:*:servicesetting/ssm/opsdata/Association",
                "arn:aws:ssm:*:*:servicesetting/ssm/opsdata/OpsData-TrustedAdvisor",
                "arn:aws:ssm:*:*:servicesetting/ssm/opsdata/ComputeOptimizer",
                "arn:aws:ssm:*:*:servicesetting/ssm/opsdata/SupportCenterCase"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateServiceLinkedRole"
            ],
            "Resource": "arn:aws:iam::*:role/aws-service-role/ssm.amazonaws.com/AWSServiceRoleForAmazonSSM"
        }
    ]
}

Hello @joehellmersNOAA,
from chef-client.log I see
Error executing action `run` on resource 'execute[copy_cluster_config_from_s3]'
Unable to locate credentials. You can configure credentials by running "aws configure".

meaning that head node doesn't have appropriate permission to be able to retrieve the cluster configuration from the PCluster S3 bucket.
Doc here lists the minimal set of policies for the head, while here are listed the minimal set for the compute.
Thanks,
L

Thanks for the info.  I'll make the change and try again soon.

Hi @joehellmersNOAA, were you able to make any progress?

I haven't had a chance to work on it yet.  I will definitely be trying again this week.

Hello.  In the Policy JSON there is a reference to a SECRET_ID.
{
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "arn:aws:secretsmanager:<REGION>:<AWS ACCOUNT ID>:secret:<SECRET_ID>",
            "Effect": "Allow"
 }

Is this secret something that ParallelCluster creates?  Or is it something I'm supposed to create ahead of time?
Thanks.

I was able to get this working by using the "AdditionalIamPolicies" capability of the yaml config file.

Thanks for letting us know!


Does it work with ScrollView?

I have a Scroll View and if the ListView is open, I only can scroll the background and the list is freezing.
Are there any possibilities to fix this?

Did you find any solution ?

NO Solution So Far ;(

Hello, @melixfueller, @shrikanth7698, @Qenawi. To make direct-select-android work in layout with ScrollView or NestedScrollView, you should use this approach:
@Override
public boolean dispatchTouchEvent(MotionEvent ev) {
if (countryListView.getVisibility() == View.VISIBLE) {
((ScrollView) (findViewById(R.id.rootScrollView))).requestDisallowInterceptTouchEvent(true);
}
if (sportListView.getVisibility() == View.VISIBLE) {
((ScrollView) (findViewById(R.id.rootScrollView))).requestDisallowInterceptTouchEvent(true);
}
if (countryListView.getVisibility() != View.VISIBLE && sportListView.getVisibility() != View.VISIBLE) {
((ScrollView) (findViewById(R.id.rootScrollView))).requestDisallowInterceptTouchEvent(false);
}
    return super.dispatchTouchEvent(ev);
}

This method should be overridden in your Activity. Basically what you're doing here is restricting your ScrollView to intercept touch events in case any of your DSListView is visible.


Add GetParameters to RTPReceivers and RTPSenders

Description
Add GetParameters to RTPReceivers and RTPSenders, allowing setting and getting ext map at transceiver level.

@OrlandoCo this LGTM! It is just extending existing W3C structures so I see no problem with it at all :)
I need to read more to understand the distinction, but would be nice if updateFromRemoteDescription and the Plan-B stuff could be merged.

@Sean-Der This PR is ready for review, with this PR we attempt to handle all the media info in transceivers, these way we can handle in single PC peers with different codec capabilities, diferent payloads and ext headers.

Moved to #1554


Rootless Docker issues

Trying to run docker run -it -p 3000:3000 huginn/huginn causing the following error:
foreman stdout | 15:36:47 jobs.1 | fatal: detected dubious ownership in repository at '/app'
15:36:47 jobs.1 | To add an exception for this directory, call:
15:36:47 jobs.1 |

I have configurated my Docker to work in rootless mode as explained in https://docs.docker.com/engine/security/rootless/
This article might be helpful to find the solution for this problem: https://www.kenmuse.com/blog/avoiding-dubious-ownership-in-dev-containers/
I'm running Docker on Debian 11.

I'm also seeing this error when trying to start up a new container.
Generating random APP_SECRET_TOKEN. You are replacing the current local value of without, which is currently "test:development" Using rake 13.0.6 Using ace-rails-ap 2.0.1 Using concurrent-ruby 1.1.10 Using i18n 1.12.0 Using minitest 5.16.3 Using thread_safe 0.3.6 Using tzinfo 1.2.10 Using zeitwerk 2.6.1 Using activesupport 6.0.6 Using builder 3.2.4 Using erubi 1.11.0 Using racc 1.6.0 Using nokogiri 1.13.9 (x86_64-linux) Using rails-dom-testing 2.0.3 Using crass 1.0.6 Using loofah 2.19.0 Using rails-html-sanitizer 1.4.3 Using actionview 6.0.6 Using rack 2.2.4 Using rack-test 2.0.2 Using actionpack 6.0.6 Using nio4r 2.5.8 Using websocket-extensions 0.1.5 Using websocket-driver 0.7.5 Using actioncable 6.0.6 Using globalid 1.0.0 Using activejob 6.0.6 Using activemodel 6.0.6 Using activerecord 6.0.6 Using marcel 1.0.2 Using activestorage 6.0.6 Using mini_mime 1.1.2 Using mail 2.7.1 Using actionmailbox 6.0.6 Using actionmailer 6.0.6 Using actiontext 6.0.6 Using public_suffix 5.0.0 Using addressable 2.8.1 Using aws-eventstream 1.2.0 Using aws-partitions 1.547.0 Using aws-sigv4 1.4.0 Using jmespath 1.6.1 Using aws-sdk-core 3.125.2 Using aws-sdk-kms 1.53.0 Using aws-sdk-s3 1.111.1 Using bcrypt 3.1.16 Using msgpack 1.4.2 Using bootsnap 1.9.3 Using kaminari-core 1.2.2 Using kaminari-actionview 1.2.2 Using kaminari-activerecord 1.2.2 Using kaminari 1.2.2 Using bundler 2.3.10 Using method_source 1.0.0 Using thor 0.20.3 Using railties 6.0.6 Using sprockets 3.7.2 Using sprockets-rails 3.4.2 Using rails 6.0.6 Using bootstrap-kaminari-views 0.0.5 Using buftok 0.2.0 Using coffee-script-source 1.12.2 Using execjs 2.8.1 Using coffee-script 2.4.1 Using coffee-rails 5.0.0 Using cookiejar 0.3.3 Using daemons 1.1.9 Using declarative 0.0.20 Using delayed_job 4.1.9 Using delayed_job_active_record 4.1.4 from https://github.com/dsander/delayed_job_active_record.git (at rails6-zeitwerk@ac2dfef) Using orm_adapter 0.5.0 Using responders 3.0.1 Using warden 1.2.9 Using devise 4.8.1 Using unf_ext <IP_ADDRESS> Using unf 0.1.4 Using domain_name 0.5.20170404 Using dotenv 2.0.1 from source at vendor/gems/dotenv-2.0.1Using dotenv-rails 2.0.1 from source atvendor/gems/dotenv-2.0.1Using hashie 3.5.7 Using multi_json 1.15.0 Using multipart-post 2.1.1 Using faraday 0.17.4 Using jwt 2.3.0 Using multi_xml 0.6.0 Using oauth2 1.4.7 Using dropbox-api 0.5.0 from https://github.com/dsander/dropbox-api.git (at 86cb7b5@86cb7b5) Using eventmachine 1.2.7 Using em-socksify 0.3.2 Using http_parser.rb 0.6.0 Using em-http-request 1.1.7 Using equalizer 0.0.11 Using polyglot 0.3.5 Using treetop 1.6.10 Using erector 0.10.0 from https://github.com/dsander/erector.git (at rails6@821c2fa) Using et-orbi 1.2.6 Using ffi 1.15.5 Using ethon 0.12.0 Using evernote-thrift 1.25.1 Using oauth 0.5.8 Using evernote_oauth 0.2.3 Using faraday_middleware 0.12.2 Using sax-machine 1.3.2 Using feedjira 3.2.0 Using rb-fsevent 0.11.0 Using rb-inotify 0.10.1 Using sass-listen 4.0.0 Using sass 3.7.4 Using font-awesome-sass 4.7.0 Using forecast_io 2.0.1 Using foreman 0.63.0 Using raabro 1.4.0 Using fugit 1.5.2 Using gems 1.2.0 Using geokit 1.13.1 Using geokit-rails 2.3.2 Using gmail_xoauth 0.4.2 Using memoist 0.16.2 Using os 1.1.4 Using signet 0.16.0 Using googleauth 0.17.1 Using httpclient 2.8.3 Using trailblazer-option 0.1.2 Using uber 0.1.0 Using representable 3.1.1 Using retriable 3.1.2 Using rexml 3.2.5 Using webrick 1.7.0 Using google-apis-core 0.4.1 Using google-apis-discovery_v1 0.7.0 Using google-apis-generator 0.4.0 Using google-api-client 0.53.0 Using google-cloud-env 1.5.0 Using google-cloud-errors 1.2.0 Using google-cloud-core 1.6.0 Using google-protobuf 3.21.9 (x86_64-linux) Using googleapis-common-protos-types 1.4.0 Using grpc 1.50.0 (x86_64-linux) Using googleapis-common-protos 1.3.12 Using rly 0.2.3 Using google-gax 1.8.2 Using google-cloud-translate 2.3.0 Using haversine 0.3.0 Using httparty 0.14.0 Using hipchat 1.2.0 Using mimemagic 0.4.3 Using httmultiparty 0.3.16 Using http-cookie 1.0.3 Using http-form_data 1.0.1 Using http 2.1.0 Using huginn_agent 0.6.1 Using hypdf 1.0.10 Using jquery-rails 4.2.2 Using json 2.6.1 Using jsonpath 1.1.0 Using kgio 2.11.4 Using kramdown 2.3.1 Using libv8-node <IP_ADDRESS> (x86_64-linux) Using liquid 5.3.0 Using listen 3.0.8 Using systemu 2.6.4 Using macaddr 1.7.1 Using memoizable 0.4.2 Using mime-types-data 3.2021.1115 Using mime-types 3.4.1 Using mini_magick 4.11.0 Using mini_racer 0.6.3 Using mqtt 0.3.1 Using mysql2 0.5.4 Using naught 1.1.0 Using net-ftp-list 3.2.8 Using netrc 0.11.0 Using omniauth 1.9.1 Using omniauth-oauth2 1.7.2 Using omniauth-dropbox-oauth2 0.3.0 from https://github.com/huginn/omniauth-dropbox-oauth2.git Using omniauth-oauth 1.1.0 Using omniauth-evernote 1.2.1 Using omniauth-google-oauth2 0.8.2 Using omniauth-tumblr 1.2 Using omniauth-twitter 1.3.0 Using raindrops 0.20.0 Using rb-kqueue 0.2.4 Using rest-client 2.0.2 Using rturk 2.12.1 Using uuid 2.3.7 Using ruby-growl 4.1 Using rufus-scheduler 3.8.1 Using sassc 2.3.0 Using tilt 2.0.10 Using sassc-rails 2.1.2 Using sass-rails 6.0.0 Using select2-rails <IP_ADDRESS> Using simple_oauth 0.3.1 Using slack-notifier 1.0.0 Using spectrum-rails 1.3.4 Using tumblr_client 0.8.5 from https://github.com/albertsun/tumblr_client.git (at e046fe6@e046fe6) Using twilio-ruby 5.62.0 Using twitter 5.15.0 from https://github.com/sferik/twitter.git Using twitter-stream 0.1.15 from https://github.com/cantino/twitter-stream.git (at huginn@a80822d) Using typhoeus 1.3.1 Using uglifier 2.7.2 Using unicorn 6.1.0 Using weibo_2 0.1.7 from https://github.com/albertsun/weibo_2.git (at master@ac38d04) Using xmpp4r 0.5.6 Bundle complete! 110 Gemfile dependencies, 204 gems now installed. Gems in the groups 'test' and 'development' were not installed. Bundled gems are installed into./vendor/bundle`
2022-11-19 19:34:39,232 INFO Included extra file "/etc/supervisor/conf.d/bootstrap.conf" during parsing
2022-11-19 19:34:39,233 INFO Included extra file "/etc/supervisor/conf.d/foreman.conf" during parsing
2022-11-19 19:34:39,233 INFO Included extra file "/etc/supervisor/conf.d/mysqld.conf" during parsing
2022-11-19 19:34:39,241 INFO RPC interface 'supervisor' initialized
2022-11-19 19:34:39,241 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-11-19 19:34:39,242 INFO supervisord started with pid 1
2022-11-19 19:34:40,245 INFO spawned: 'stdout' with pid 377
2022-11-19 19:34:40,248 INFO spawned: 'bootstrap' with pid 378
2022-11-19 19:34:40,257 INFO success: bootstrap entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2022-11-19 19:34:40,401 INFO spawned: 'foreman' with pid 380
2022-11-19 19:34:40,402 INFO success: foreman entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2022-11-19 19:34:40,420 INFO exited: bootstrap (exit status 0; expected)
2022-11-19 19:34:41,968 INFO success: stdout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
bootstrap stdout | DATABASE_HOST=mariadb-infra-db.service.dc1.consul
foreman stderr | fatal: detected dubious ownership in repository at '/app'
To add an exception for this directory, call:
git config --global --add safe.directory /app

foreman stderr | fatal: detected dubious ownership in repository at '/app'
To add an exception for this directory, call:
git config --global --add safe.directory /app

foreman stderr | fatal: detected dubious ownership in repository at '/app'
To add an exception for this directory, call:
git config --global --add safe.directory /app

foreman stderr | fatal: detected dubious ownership in repository at '/app'
To add an exception for this directory, call:
git config --global --add safe.directory /app

foreman stderr | rake aborted!
Mysql2::Error::ConnectionError: Access denied for user<EMAIL_ADDRESS>(using password: NO)
foreman stderr | /app/vendor/bundle/ruby/2.7.0/gems/mysql2-0.5.4/lib/mysql2/client.rb:95:in connect' /app/vendor/bundle/ruby/2.7.0/gems/mysql2-0.5.4/lib/mysql2/client.rb:95:in initialize'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/connection_adapters/mysql2_adapter.rb:24:in new' /app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/connection_adapters/mysql2_adapter.rb:24:in mysql2_connection'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:887:in new_connection' /app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:931:in checkout_new_connection'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:910:in try_to_checkout_new_connection' /app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:871:in acquire_connection'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:593:in checkout' /app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:437:in connection'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:1125:in retrieve_connection' /app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/connection_handling.rb:221:in retrieve_connection'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/connection_handling.rb:189:in connection' /app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/tasks/database_tasks.rb:238:in migrate'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/railties/databases.rake:86:in block (3 levels) in <main>' /app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/railties/databases.rake:84:in each'
/app/vendor/bundle/ruby/2.7.0/gems/activerecord-6.0.6/lib/active_record/railties/databases.rake:84:in block (2 levels) in <main>' /app/vendor/bundle/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in <top (required)>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
2022-11-19 19:34:50,234 INFO exited: foreman (exit status 1; not expected)`

I was receiving this error too. I made a new Dockerfile that overlays the Huginn one with the following:
FROM huginn/huginn:latest
WORKDIR /app

USER root
RUN git config --system --replace-all safe.directory '*'
# RUN git config --system --add safe.directory /app
USER 1001

I was going to integrate this into the main Dockerfile and send a PR, but I'm having trouble building the docker image on my Mac. /cc @knu

@iikkart I don't think this is related to your Docker setup being rootless.
This is surely caused by the fact that all processes are run by user default and /app is still mostly owned by root, /app/.git in particular. The solution suggested by @cantino is one way to go about it.
Alternatively, when building the image, ownership of /app could just be transferred to default/1001 recursively. Not sure if there are any safety concerns doing this...maybe one of the devs could chime in.
In any case, this seems to be just a cosmetic issue in my setup...everything seems to be working.


Skip explanation_answer page if UD case

I made it so that including this page doesn't cause an error, and shows explanations for relevant defenses, although none of the unlawful detainer defenses are included.  Also using collapse_template() to make the answer easier.  I think there's a lot of value to seeing what defenses are in the answer before looking at the answer and the evidence list.


kube-prometheus/jsonnet: Use jsonnet-bundler

This PR has some large changes on the kube-prometheus sub-project. It now uses a packaging approach of jsonnet code using jsonnet-bundler. This allows each component to be individually packaged up and then brought together in the now meta-package kube-prometheus. As such, Grafana has already been extracted into it's own package: kubernetes-grafana. All other components are currently still living in this repository but the goal is to eventually make them standalone packages. Additionally the majority of alerting rules and dashboards have been extracted to the Kubernetes monitoring mixins. Not all alerting rules and dashboards have been migrated there, but I'd prefer to get this merged and continue with the migration piece by piece (which is why dashboards and alerting rules have not been entirely removed from this repository).
@mxinden could you just play through the instructions in the readme against a local minikube as described in the readme? If that works I think we can merge this.

Tested latest head and addressed all comments. Could you give this a re-review @mxinden ? :slightly_smiling_face: (I put all fixes in a separate commit so it's better reviewable, ultimately I would like to squash the two though when we merge)


Web console e2e test failing locally

I am trying to run e2e tests locally on the latest master branch with druid processes running at standard ports without any security enabled. I can access the console at localhost:8888. However, when I run jest --config jest.e2e.config.js --detectOpenHandles e2e-tests, I get the following error -
node_modules/.bin/jest --config jest.e2e.config.js --detectOpenHandles e2e-tests
 FAIL  e2e-tests/tutorial-batch.spec.ts (32.427s)
  Tutorial: Loading a file
    ✕ Loads data from local disk (31378ms)

  ● Tutorial: Loading a file › Loads data from local disk

    TimeoutError: waiting for selector ".raw-lines" failed: timeout 30000ms exceeded

      66 |   private async validateConnect(validator: (preview: string) => void) {
      67 |     const previewSelector = '.raw-lines';
    > 68 |     await this.page.waitFor(previewSelector);
         |                     ^
      69 |     const preview = await this.page.$eval(previewSelector, el => (el as HTMLTextAreaElement).value);
      70 |     validator(preview!);
      71 |   }

      at new RerunnableTask (node_modules/playwright-core/lib/frames.js:822:34)
      at Frame._scheduleRerunnableTask (node_modules/playwright-core/lib/frames.js:770:32)
      at Frame._waitForSelectorInUtilityContext (node_modules/playwright-core/lib/frames.js:741:35)
      at Frame.waitForSelector (node_modules/playwright-core/lib/frames.js:485:35)
      at Frame.<anonymous> (node_modules/playwright-core/lib/helper.js:55:31)
      at Frame.waitFor (node_modules/playwright-core/lib/frames.js:710:25)
      at Frame.<anonymous> (node_modules/playwright-core/lib/helper.js:55:31)
      at Page.waitFor (node_modules/playwright-core/lib/page.js:383:33)
      at Page.<anonymous> (node_modules/playwright-core/lib/helper.js:55:31)
      at DataLoader.<anonymous> (e2e-tests/component/load-data/data-loader.ts:68:21)
      at step (node_modules/tslib/tslib.js:136:27)
      at Object.next (node_modules/tslib/tslib.js:117:57)
      at node_modules/tslib/tslib.js:110:75
      at Object.__awaiter (node_modules/tslib/tslib.js:106:16)
      at DataLoader.Object.<anonymous>.DataLoader.validateConnect (e2e-tests/component/load-data/data-loader.ts:111:24)
      at DataLoader.<anonymous> (e2e-tests/component/load-data/data-loader.ts:62:16)
      at step (node_modules/tslib/tslib.js:136:27)
      at Object.next (node_modules/tslib/tslib.js:117:57)
      at fulfilled (node_modules/tslib/tslib.js:107:62)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        32.98s, estimated 33s
Ran all test suites matching /e2e-tests/i.

What might be the issue ? @ccaominh

ok figured out the issue, I was not running druid from tar ball but directly from IDE so the location to file wikiticker-2015-09-12-sampled.json.gz was different and that's why it was failing.

Sorry for the late response, but I'm adding some comments here in case anyone finds this issue when trying to debug the end-to-end tests.
From the stack trace, the data loader failed during the "connect" step. Specifically, the preview of the raw data did not appear in the web console.
When the e2e test fails, it'll dump a screen shot of the of the web console at the time of test failure, which can be helpful for debugging. For this particular test, the screen shot is named load-data-from-local-disk-screenshot.png.
Disabling headless mode while running the tests can also be helpful for debugging. One way of doing this is by modifying tutorial-batch.spec.ts to change:
import { createBrowserNormal as createBrowser } from './util/playwright';

to
import { createBrowserDebug as createBrowser } from './util/playwright';

Once that's done, running the tests will launch a chromium instance and you can watch all the interactions with the web console.

thanks makes sense, I was able to find the problem after looking at the screenshot but took me a long time to find it. Disabling headless mode sound nice, it would be great to check in these notes probably in web console readme.

I've opened https://github.com/apache/druid/pull/9864 to add some docs to the web console readme.


[COOK-3631] scm_provider Chef::Provider::RemoteFile::Deploy broken

I was unable to get the RemoteFile cookbook working with my web hosted war files.  I decided to try the cookbook's example which also failed the same way.
application 'jenkins' do
  path         '/usr/local/jenkins'
  owner        node['tomcat']['user']
  group        node['tomcat']['group']
  repository   'http://mirrors.jenkins-ci.org/war/latest/jenkins.war'
  revision     '6facd94e958ecf68ffd28be371b5efcb5584c885b5f32a906e477f5f62bdb518-1'
	scm_provider Chef::Provider::RemoteFile::Deploy

  java_webapp do
    context_template 'jenkins-context.xml.erb'
  end

  tomcat
end

NoMethodError
-------------
undefined method `create' for nil:NilClass
Cookbook Trace:
---------------
/var/chef/cache/cookbooks/application_java/libraries/provider_remote_file_deploy.rb:53:in `action_sync'
Compiled Resource:
------------------
# Declared in /var/chef/cache/cookbooks/application/providers/default.rb:122:in `run_deploy'
deploy_revision("jenkins") do
provider Chef::Provider::Deploy::Revision
action [:deploy]
retries 0
retry_delay 2
deploy_to "/usr/local/jenkins"
repository_cache "cached-copy"
revision "6facd94e958ecf68ffd28be371b5efcb5584c885b5f32a906e477f5f62bdb518-1"
rollback_on_error true
remote "origin"
shallow_clone true
scm_provider Chef::Provider::RemoteFile::Deploy
keep_releases 5
cookbook_name "app_billingagent"
repo "http://mirrors.jenkins-ci.org/war/latest/jenkins.war"
user "tomcat7"
group "tomcat7"
restart_command #Proc:0x000000031c7b10@/var/chef/cache/cookbooks/application/providers/default.rb:139
before_migrate #Proc:0x000000028eda30@/var/chef/cache/cookbooks/application/providers/default.rb:166
before_symlink #Proc:0x0000000290fab8@/var/chef/cache/cookbooks/application/providers/default.rb:169
before_restart #Proc:0x00000002a57240@/var/chef/cache/cookbooks/application/providers/default.rb:172
after_restart #Proc:0x00000002bc5d98@/var/chef/cache/cookbooks/application/providers/default.rb:175
shared_path "/usr/local/jenkins/shared"
destination "/usr/local/jenkins/shared/cached-copy"
current_path "/usr/local/jenkins/current"
end

+1 Just noticed this while upgrading chef-client from 11.4.4 to latest 12.4. :(

+1 Just ran into this today with version 3.0.0


[markWriter] Add MarkAttachmentType flags to mkmk

Should resolve https://github.com/googlei18n/ufo2ft/issues/52, for now.
r? @khaledhosny

This does not seem to work, the mkmk feature is not applied at all now. I think this is because the mark class does not include the base marks.

Hmm :frowning: it seems to be working fine with Noto. I tried building Mada but got an assertion error from feaLib. Not sure how to debug this.

try updating the fonttools requirement to the latest HEAD. Behdad fixed something with Extension subtables overflowing.

Here is a UFO that should be buildable with fontmake: Mada-Medium.ufo.zip
Testing the string بَ بَّ نَّ بِّ بِ نَ with font generated by my code, I get:

But with ufo2ft I get:

That's a helpful test string, thanks. I'm still not convinced this is a ufo2ft bug though, because I'm getting correct results (AFAICT) from NotoSansArabic-Regular.ufo.zip. I guess this will take more debugging.

What surprises me actually is that this example string doesn't render correctly from the output without this PR. I only see an issue with بِّ, because the bottom-most mark is attaching to the base glyph rather than the dot below. But why would that be, if a mark attachment is defined on the dot below and that's the closest preceding character? I don't understand this aspect of GPOS.
Anyways, I guess this PR is unsuitable after all for resolving your case. So let's revisit the idea using group prefixes, maybe defining something as a class attribute similar to what we do with the kern feature writer.

I’m checking Noto Sans Arabic to see what is the difference and if I can get my font to work the same way.

Oh, it seems that Noto Sans Arabic is not using the mkmk anchors at all, but instead ligates the marks:
$ hb-shape NotoSansArabic-Regular.ttf "بِّ" --no-positions --cluster-level=1
[shadda_kasra.ar=1|beh.ar=0]

For comparison, in Mada this gives:
$ hb-shape Mada-Regular.ttf "بِّ" --no-positions --cluster-level=1
[uni0650=2|uni0651=1|ar1Dot.below=0|arBeh.isol=0]

Yea, I noticed that as well. It was strange to me that uni0650 was attaching to arBeh.isol rather than ar1Dot.below.

That is because the MarkAttachmentType class does not include ar1Dot.below (since it is a base mark) but only uni0650 and uni0651 etc., so the mkmk lookup ends up not being applied at all and the regular mark lookup is applied instead.
If we want to continue with the current approach, then we will need to add the base marks to the class used in MarkAttachmentType.

I'm talking about the output without this PR, without using the MarkAttachmentType flag. ufo2ft on master generates this feature source:
markClass uni0650 <anchor 0 0> @MC_markBelow;
...
markClass uni0650 <anchor 0 0> @MC_markBelowDot;
...
feature mark {
  ...
  lookup mark9 {
    ...
    pos base arBeh.isol <anchor 402 -139> mark @MC_markBelow;
    ...
  } mark9;
} mark;

feature mkmk {
  ...
  lookup mkmk10 {
    pos mark ar1Dot.below <anchor 0 -140> mark @MC_markBelowDot;
    ...
  } mkmk10;
} mkmk;

Why would the mark rule take precedence over the mkmk rule for this string? I'd expect the opposite since ar1Dot.below is closer to uni0650.

Because the mkmk lookup will stop at the first mark glyph, so it will stop at uni0651, check if the lookup has an anchor to apply between it and uni0650 (there is none) and stops there. That is where the MarkAttachmentType flag is needed, it should tell the lookups to ignore any glyphs other than uni0650 and ar1Dot.below, so the lookup will skip uni0651 and stop at ar1Dot.below, finds the anchor and applies it.

OK that makes sense. But I still feel like this should be automate-able somehow. I guess I still don't get what should go into the class for the MarkAttachmentType flag -- would it work to include the glyphs used as bases in the lookup, in addition to the glyphs used as marks?

Yes, I think this should work.

It does work, but only with UseMarkFilteringSet since there are commonly base glyphs shared between lookups. But actually, I just tried to compile the output feature syntax with makeotf, and it worked! So maybe that's a good solution?

PTAL

LGTM

Thanks so much for your patience! Let me know if this is still insufficient in any way.


Add 'series' and 'transferring body' fields

Fields required to allow downstream process to correctly identify bags and process them accordingly

@kurtismash

Is this repo still used for message schemas?
TDR still uses it to construct the bag available message. Can't speak for other services

And what happens if we start sending messages with the new fields without redeploying TRE first?

TRE devs suggested it will be OK, but it is a breaking change for the TRE code which will need updating. The unit tests for a couple of TRE repos are affected

I hope adding fields is a minor change that consumers don't need to be pre-warned about.
Me too ...

DDT will need to be informed as the new fields will not get processed unless we have managed them.
From: Kurtis Mash @.>
Sent: Wednesday, September 11, 2024 4:36 PM
To: nationalarchives/da-transform-schemas @.>
Cc: Ashworth, Phil @.>; Review requested @.>
Subject: Re: [nationalarchives/da-transform-schemas] Add 'series' and 'transferring body' fields (PR #65)
EXTERNAL: This email originates from outside of The National Archives.
@kurtismash approved this pull request.
Looks sensible.
Is this repo still used for message schemas? And what happens if we start sending messages with the new fields without redeploying TRE first? I hope adding fields is a minor change that consumers don't need to be pre-warned about.

Reply to this email directly, view it on GitHubhttps://github.com/nationalarchives/da-transform-schemas/pull/65#pullrequestreview-2297626887, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARXLT77J7GJOW5OGJBYETN3ZWBPM7AVCNFSM6AAAAABOA6BCH6VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDEOJXGYZDMOBYG4.
You are receiving this because your review was requested.Message ID<EMAIL_ADDRESS>Please don't print this e-mail unless you really need to. ----------------------------------------------------------------------------------- National Archives Disclaimer This email and any files transmitted with it are intended solely for the use of the individual(s) to whom they are addressed. If you are not the intended recipient and have received this email in error, please notify the sender and delete the email. Opinions, conclusions and other information in this message and attachments that do not relate to the official business of The National Archives are neither given nor endorsed by it. ------------------------------------------------------------------------------------


🔄 Issue links are broken

Reproduction url
n/a
Good version
n/a
Bad version
n/a
Description of the regression
I tried to file a feature request, but you can't anymore.
I blame dbfefa316c6710c1179a0f0c2ca762525ea32140

Yup: https://github.com/webpro-nl/knip/blob/dbfefa316c6710c1179a0f0c2ca762525ea32140/.github/ISSUE_TEMPLATE/03-feature_request.yaml
Big red error there. You want type: input, not type:markdown.

For reference, my feat request was going to be adding in support for TS 5.5 @import comments. :)
Both becuase I leave them lying around everywhere, and they should be considered dependencies.

Thanks @lishaduck! Should be fixed now.
Import comments should work (eg this fixture has some: https://github.com/webpro-nl/knip/blob/main/packages/knip/fixtures/jsdoc/index.ts) - feel free to open an issue with a reproduction.


Run package syncs using self hosted runners

Not run out of time on 6h+ syncs, and maybe spend less money on GH runners but I'm not sure if leafcloud is any cheaper

GH runners are free for public repos. If it works then it's fine by me, but I expect the time is spent waiting for tasks in Ark either way.


vim9script: Segmentation fault when running :import Foo from "foo.vim" in command-line mode

Describe the bug
Vim crashes when an exported vim9script function is imported in command-line mode with :import.
To Reproduce

Add the following to ~/.vim/import/foo.vim:

vim9script
export def Foo(): number
    return 0
enddef

Run `vim -c "import Foo from 'foo.vim'"

Vim will crash with the following message:
Vim: Caught deadly signal SEGV
Segmentation fault

Expected behavior
Importing vim9script functions in command-line mode using :import should probably be disabled.
Environment
Vim version 8.2.983

backtrace
valgrind log
asan log


Corrected Latin_1_General_BIN to Latin1_General_BIN

The collation "Latin1_General_BIN" was misspelled.

@WilliamDAssafMSFT
Can you review the proposed changes?
When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

Thanks @dhmacher!


material ui bug

when i select different image for map and ao,both textures are always the same

This issue has been fixed.


Failed to batch ingest runs: LangSmithError - 403 Client Error: Forbidden

Issue you'd like to raise.
I am getting this error while trying to use langschain and langsmith on my application:
Failed to batch ingest runs: LangSmithError('Failed to POST https://api.smith.langchain.com/runs/batch in LangSmith API. HTTPError('403 Client Error: Forbidden for url: https://api.smith.langchain.com/runs/batch', '{"detail":"Forbidden"}')')
I wrote a Jupiter Notebook to test the core components of my application and I don't have the same error in the notebook.
The application and the notebook use the same Python environment and environment variables including the same LANGCHAIN_API_KEY.
These are the langchain packages I am using:
langchain==0.2.6
langchain-community==0.2.6
langchain-core==0.2.11
langchain-experimental==0.0.62
langchain-openai==0.1.13
langchain-pinecone==0.1.1
langchain-text-splitters==0.2.1
langchainhub==0.1.20
langdetect==1.0.9
langsmith==0.1.83
I also attached the full requirements.txt:
requirements.txt
Interestingly,  I keep getting the error message even when I set the environment variable LANGCHAIN_TRACING_V2 to false. Given this strange behavior, I added the following debugging print statements to client.py:
def _post_batch_ingest_runs(self, body: bytes):
    try:
        for api_url, api_key in self._write_api_urls.items():

            print(f"******** Posting batch ingest runs to: {api_url}")
            print(f"******** api_key          : {api_key}")
            print(f"******** LANGCHAIN_API_KEY: {os.environ["LANGCHAIN_API_KEY"]}")

            self.request_with_retries(
                "POST",
                f"{api_url}/runs/batch",
                request_kwargs={
                    "data": body,
                    "headers": {
                        **self._headers,
                        X_API_KEY: api_key,
                    },
                },
                to_ignore=(ls_utils.LangSmithConflictError,),
                stop_after_attempt=3,
            )
    except Exception as e:
        logger.warning(f"Failed to batch ingest runs: {repr(e)}")

This is the output:
********Posting batch ingest runs to https://api.smith.langchain.com
********API Key:
********LANGCHAIN_API_KEY:
Failed to batch ingest runs: LangSmithError('Failed to POST https://api.smith.langchain.com/runs/batch in LangSmith API. HTTPError('403 Client Error: Forbidden for url: https://api.smith.langchain.com/runs/batch', '{"detail":"Forbidden"}')')
The variable api_key matches the value of os.environ["LANGCHAIN_API_KEY"] but is not the one I set. I also have a print statement on my app, and the os.environ["LANGCHAIN_API_KEY"] that I get matches the one I set using dot_env.
I don't have any setting of LANGCHAIN_API_KEY within my app besides the loading of the .env file.
To validate that they issue is the wrong api_key I manually set it up in client.py:
def _post_batch_ingest_runs(self, body: bytes):
    try:
        for api_url, api_key in self._write_api_urls.items():

            api_key = "my real api_key"

            self.request_with_retries(
                "POST",
                f"{api_url}/runs/batch",
                request_kwargs={
                    "data": body,
                    "headers": {
                        **self._headers,
                        X_API_KEY: api_key,
                    },
                },
                to_ignore=(ls_utils.LangSmithConflictError,),
                stop_after_attempt=3,
            )
    except Exception as e:
        logger.warning(f"Failed to batch ingest runs: {repr(e)}")

The issue disappears and I finally can see my traces on the hosted langsmith page.
Somehow the environment variable LANGCHAIN_API_KEY is being changed. The changed value is shorter than the original value.
Suggestion:
No response

In general a 403 error typically results from having the wrong API key, which aligns with what you've seen.
If i understand correctly:

Prior to executing code, you have an API key set in the environment via LANGCHAIN_API_KEY. You also have LANGCHAIN_TRACING_V2 set to true.
In your code, you load from a dotenv to overwrite that API key.
Within the client, you see that the client's API key matches the original environment's key, not the one you tried to overwrite it with

Are you by any chance loading your dotenv file after executing other code?
In general we don't recommend changing env vars mid-execution since the results may be undefined (we use caching, trace in a background thread, etc. to minimize tracing overhead)

I am loading my dotenv file at the start of my application and not changing the environment variables after that. Tracing only begins after someone chats with the chatbot.
The dotenv file has the correct LANGCHAIN_API_KEY and LANGCHAIN_TRACING_V2 set to true.   After loading the dotenv file, I have print statements that confirm that the environment variables are set correctly. I still get a 403 error.
For that reason, I included a print statement in the client.py file of the langsmith library and I validated that the api_key being used by _post_batch_ingest_runs is incorrect and shorter than expected.

My app is divided into several servers and I found a call to langchain in one of them occurring before loading the dotenv file.
The issue was solved by loading the dotenv file in the first module to be initialized.


⚡Add sensitive data Cookie types

💭 Introduction
Akto allows users to find sensitive params in request and response of APIs. These sensitive params include PII data such as SSN, Credit card number, phone number etc. You can check how it works in the docs.
Your task is to add cookie types as sensitive params.
👀 Impact
Your task will give users the ability to detect cookie types sensitive data types.
🎯 Requirements

Understanding of regex
Understanding of unit test

✅ Task summary:

[ ] Ask to be assigned to the issue.
[ ] Wait to be assigned. We will try to assign in less than 2 hours.
[ ] Make regular expressions for below cookie types. Feel free to add your own too.
[ ] Please provide the reference URL that mentions the nature of the cookie.
[ ] Add an entry in fintech.json file
[ ] Add a test case entry in TestFintechTypes.java
[ ] Submit a pull request here

facebook,  intercom,  google ads,  hotjar,  fullstory,  mixpanel,  appsflyer)
🙋🏼♂ Questions:
If you have questions, need any help, or just want to hang out, make sure to join us on our Discord server.

Hi Ankita, I would like to try on this issue and contribute. Can you please assign me to this issue?

HI @bhavya-error404, Assigned to you! Happy hackfesting 🎉

Hi @Ankita28g , I need to add regex for facebook, intercom etc cookie types. For example a regex that would be able to detect a facebook cookie, right?

Hi @bhavya-error404 - yes. In this case, you might want to add the regexes based on keys rather than values. They can cookie names for products or infrastructures -
For example,

_ga _gid is for Google Analytics
intercom.* is for Intercom

examples for infra based are -

.*srf.* is for CSRF token
JSESSIONID is for Jetty session
AWS.* for AWS Loadbalancer cookies

Hi @bhavya-error404  thanks for your submission in Hackfest. 🔥 We are reviewing your work. Do these two below:
Join this group on discord for discussions around prizes? 🚀 🏆
Please fill this form your PR to be considered for prizes!

hi @bhavya-error404 did you submit a PR here?

yes ankita

here is the link https://github.com/akto-api-security/akto/pull/251

@ankush-jain-akto , @Ankita28g I can pick this up. Can you please assign this to me.

facebook-
intercom-
Cookies: intercom*
Ref- https://www.intercom.com/help/en/articles/2361922-intercom-messenger-cookies
google ads-
Cookies:
csrf.*
.cookie.test.
hjCampaignId
optanon.
Ref-
hotjar-
fullstory-
mixpanel-
appsflyer-

@Ankita28g @ankush-jain-akto - I have analysed list of cookies and below are details - Is it good to proceed? Also do I need to have separate entry in fintech.json for each product/infrastructure?
Facebook Cookies:
c_user.*
datr
csrf
_fbp, _fbc

Ref: https://www.facebook.com/privacy/policies/cookies/?annotations[0]=explanation%2F1_common_cookies_and_uses
intercom- Cookies: intercom.* Ref- https://www.intercom.com/help/en/articles/2361922-intercom-messenger-cookies
google ads- Cookies:
_ga
_gid

Ref-
hotjar- Cookies:
csrf.*
optanon.*
_dc_gtm_UA_.*
.*consent
cf.*
__cf.*
user.*
ts.*
nsid

Ref-https://help.hotjar.com/hc/en-us/articles/115011789248-Cookies-on-hotjar-com#cookies_set_by_hotjar_website
fullstory- Cookies:
fs_.*
_fs.*

Ref: https://help.fullstory.com/hc/en-us/articles/360020829513-Why-FullStory-uses-First-Party-Cookies
mixpanel- Cookies:
.*mixpanel-.*
mp.*

Ref: https://cookiedatabase.org/service/mixpanel/
appsflyer- Cookies:
_hj.*
_ym.*

Ref- https://www.appsflyer.com/legal/cookie-list/

@Ankita28g / @ankush-jain-akto - is it good for me to proceed? If yes could you assign this to me?

Hi @harshalkh
Your analysis is correct, but we should be cautious about over-tagging, especially with common cookies like csrf, as it could lead to false positives. To minimize this, please refine the list, focusing on unique cookies specific to Facebook, like _fbp and _fbc. I've assigned the issue to you in the meantime.

@avneesh-akto - I want to understand how can I test the new regex after I add in Test class.
I have tried using mvn clean install - but getting errors about dependency -
https://github.com/harshalkh/akto/actions/runs/6609614157/job/17950029280
Could you please suggest any other steps to test this?

@avneesh-akto - I want to understand how can I test the new regex after I add in Test class.
I have tried using mvn clean install - but getting errors about dependency -
harshalkh#1
https://github.com/harshalkh/akto/actions/runs/6609614157/job/17950029280
Could you please suggest any other steps to test this?

Hi @harshalkh
Your analysis is correct, but we should be cautious about over-tagging, especially with common cookies like csrf, as it could lead to false positives. To minimize this, please refine the list, focusing on unique cookies specific to Facebook, like _fbp and _fbc. I've assigned the issue to you in the meantime.

@avneesh-akto - I have made the changes and tested by modifying json file link inside test case.
Please review - https://github.com/akto-api-security/akto/pull/682


Made links more readable

I hovered over one of the links (guidelines and templates) so you'd see how it looks when its active and when its not

hello, @1Joda did you change the CSS style, how did you make the link's contrast readable?

hello, @1Joda did you change the CSS style, how did you make the link's contrast readable?
I did
I left the color they used, removed the underline and made it the tiniest bit transparent so that when you click on it, it becomes opaque

I initially made it opaque and transparent on hover but it was blending in with the titles in the pages ('June 20, 2022' and 'guidelines and templates' for instance look similar but i was hovering over 'guidelines and templates) so i changed it

There is already an issue #417 and a PR to review here #430
Could you check the open PR and make suggestion there, instead of having a similar PR? Thanks a lot

I will close this issue but issue #417 and PR #430 are still open.


Fatal Exception 'Empty "blog" value'

When submitting to Akismet v2 returns Empty "blog" value from the Akismet API. I opened #45 with more details.
I'm not sure how long this was an issue as the x-debug exception was added with v2.
Possibly related to #6 but that was so long ago who knows ;)

Thanks BrookeDot, just released v2.0.1


Failed to execute "adb push"

* ERROR: Failed to execute: [D:\Scrcpy\scrcpy-win64-v1.10], [push], ["D:\Scrcpy\scrcpy-win64-v1.10\scrcpy-server.jar"], ["/data/local/tmp/scrcpy-server.jar"]
* ERROR: Could not execute "adb push"

You probably set the environment variable ADB to D:\Scrcpy\scrcpy-win64-v1.10.
You probably wanted to set it to D:\Scrcpy\scrcpy-win64-v1.10\adb.exe.

You probably set the environment variable ADB to D:\Scrcpy\scrcpy-win64-v1.10.
You probably wanted to set it to D:\Scrcpy\scrcpy-win64-v1.10\adb.exe.

Thank you!
^_^

You probably set the environment variable ADB to D:\Scrcpy\scrcpy-win64-v1.10.
You probably wanted to set it to D:\Scrcpy\scrcpy-win64-v1.10\adb.exe.

ths,useful~


A dependency(grpc) of Google-gax is blocking our ruby upgrade (2.6.5 => 2.7.0)

Environment details

OS: macOs Catalina
Gem name and version:   google-cloud-text_to_speech (0.6.0)

Error
You get this error when you attempt to upgrade
bundle install
The dependency tzinfo-data (>= 0) will be unused by any of the platforms Bundler is installing for. Bundler is installing for ruby but the dependency is only for x86-mingw32, x86-mswin32, x64-mingw32, java. To add those platforms to the bundle, run `bundle lock --add-platform x86-mingw32 x86-mswin32 x64-mingw32 java`.
Fetching gem metadata from https://rubygems.org/...........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
grpc-1.27.0-universal-darwin requires ruby version < 2.7.dev, >= 2.3,
which is incompatible with the current version, ruby 2.7.1p83

Please is there is plan on resolving this?
You can find a related issue here https://github.com/grpc/grpc/issues/21514

You should be able to bundle update grpc to get the latest version (1.32, which is compatible with Ruby 2.7.)
You should also consider updating google-cloud-text_to_speech. The latest version is 1.1.1. (The 0.x versions were built using an old code generator that we no longer support, and that may have some issues/warnings on Ruby 2.7. Note that newer versions do not even use google-gax anymore.)


Using the googlemaps api key?

📚 What are you trying to do?
Hi All,
I am really stuck with this. So i use the demo code
<ScriptGoogleMaps
    ref="googleMapsRef"
    :center="query"
    :markers="markers"
    :api-key="googleMapsApiKey"
    class="group w-full h-64 rounded-lg"
    above-the-fold
    @ready="handleReady"
  />

const googleMapsApiKey = process.env.NUXT_GOOGLE_MAPS_API_KEY;
but this doesnt work? When i use my api key directly in the file it works. But putting it in the .env i wont load the map? I thought placing your api key is not secure? Already asked chatgpt but no solution.
🔍 What have you tried?
I looked at the docs. Nothing about it. I tried chatgpt and different solutions nothing works.
ℹ Additional context
No response

You should use runtimeConfig

The setup instructions need to be improved for this page. Using .env is supported but you need to follow the format of NUXT_PUBLIC_SCRIPTS_GOOGLE_MAPS_API_KEY
// nuxt.config.ts
export default defineNuxtConfig({
  // you need to provide a runtime config to access the environment variables
  runtimeConfig: {
    public: {
      scripts: {
        googleMaps: {
          apiKey: '', // NUXT_PUBLIC_SCRIPTS_GOOGLE_MAPS_API_KEY
        },
      },
    },
  },
})

# .env
NUXT_PUBLIC_SCRIPTS_GOOGLE_MAPS_API_KEY=<your-token>

I'll leave open to track doc update

Thank you i got it working. Is using public not a security issue? I read that I can enable on google cloud what site can use the api key. Or is the key hidden in the script?

Thank you i got it working. Is using public not a security issue? I read that I can enable on google cloud what site can use the api key. Or is the key hidden in the script?

Google Maps needs a public API key. For other APIs you definitely don't want it public (e.g. the stripe private key). Talked a bit about that in this video

Yes just to add on to this, make sure you lock down your google maps API key to specific hosts!
Very important step https://developers.google.com/maps/api-security-best-practices#restricting-api-keys


Add support for specifying separate addresses for non-inlinable libraries on different networks

🌟 Feature Request for hardhat-zksync plugin
Related discussion from vmaark here: https://github.com/zkSync-Community-Hub/zksync-developers/discussions/763
💥 Plugin name
hardhat-zksync
📝 Description
Add support for specifying separate addresses for non-inlinable libraries on different networks in the compiler settings. Currently, non-inlinable libraries need to be deployed separately, with the address specified in the hardhat config. However, this single address setup poses issues when deploying to multiple networks, such as a testnet and mainnet, or different chains built with ZK Stack.
This new feature would allow users to provide a mapping with unique library addresses for each network, enabling seamless deployment to multiple targets without needing to manually change configuration files.
🤔 Rationale
This feature is essential for teams working with multiple deployment targets, as it ensures flexibility and efficiency when deploying to various networks. By supporting network-specific addresses, developers can automate deployments across different environments more effectively, saving time and reducing potential for error.
🖼 Mockups/Examples
If implemented, the configuration could look like this:
module.exports = {
  solidity: "0.8.7",
  networks: {
    mainnet: {
      libraryAddresses: {
        SomeLibrary: "0xMainnetAddress",
      },
    },
    testnet: {
      libraryAddresses: {
        SomeLibrary: "0xTestnetAddress",
      },
    },
  },
};

📋 Additional Context
This enhancement would be particularly useful for projects utilizing ZK Stack or deploying across both mainnet and testnet environments. It provides a streamlined approach for managing library dependencies across various network configurations.

The ML team has worked on compiler improvements to support linking libraries during deployment. As a result, this feature is no longer necessary. We will be releasing a new version of the plugins that allows users to link libraries when deploying a contract, similar to the functionality we have on L1. Here a link to the PR


cmake build failed when compile dlib.so for python

err msg:
/usr/bin/ld: /usr/local/lib/libpython2.7.a(abstract.o): relocation R_X86_64_32S against `_Py_NotImplementedStruct' can not be used when making a shared object; recompile with -fPIC
error: cmake build failed!
I have run "python setup.py install" on Ubuntu 16.04 server in python virtualenvwrapper, it begin compilation, but in the end print err_msg.
What should I do?

Something is wrong with your python install.  I would install python and libpython with apt-get.


Added description and labels fields to VoiceDesign

The server-side requirements for v1/voice-generation/create-voice include a description and labels field. The mismatch between the previous setup of VoiceDesign and the server-side requirements was causing HTTPError422, Unprocessable Entity.
Updated API.md to match the new design of VoiceDesign
closes #54
closes #94

Hi, thanks! Voice design has been fixed in v0.2.25.


Allow content_type of response to be None, e.g. for HTTP-401.

Hi Martin,
test is added. Can you merge the bugfix please?
Thanks,
Sebastian

Merged, thank you!


Please document: What happens if there is a hash collision?

Please document clearly what happens if there is a hash collision.
I read the existing docs, but I didn't see any mention of this scenario.  I looked at the source code, and I think this code answers the question:
// internal/archiver/archiver.go:166

func (arch *Archiver) saveChunk(ctx context.Context, chunk chunker.Chunk, p *restic.Progress, token struct{}, file fs.File, resultChannel chan<- saveResult) {
    defer freeBuf(chunk.Data)

    id := restic.Hash(chunk.Data)
    err := arch.Save(ctx, restic.DataBlob, chunk.Data, id)
    // TODO handle error
    if err != nil {
        debug.Log("Save(%v) failed: %v", id, err)
        fmt.Printf("\nerror while saving data to the repo: %+v\n", err)
        panic(err)
    }

    p.Report(restic.Stat{Bytes: uint64(chunk.Length)})
    arch.blobToken <- token
    resultChannel <- saveResult{id: id, bytes: uint64(chunk.Length)}
}

...and...
// internal/archiver/archiver.go:96

// Save stores a blob read from rd in the repository.
func (arch *Archiver) Save(ctx context.Context, t restic.BlobType, data []byte, id restic.ID) error {
    debug.Log("Save(%v, %v)\n", t, id)

    if arch.isKnownBlob(id, restic.DataBlob) {
        debug.Log("blob %v is known\n", id)
        return nil
    }

    _, err := arch.repo.SaveBlob(ctx, t, data, id)
    if err != nil {
        debug.Log("Save(%v, %v): error %v\n", t, id, err)
        return err
    }

    debug.Log("Save(%v, %v): new blob\n", t, id)
    return nil
}

So, it seems that if there were a hash collision, the newer data would just be assumed to be the same as the older data, and would therefore not be stored in the repository (silent data loss).  During a restore, the older data would be used instead of the new data, resulting in a corrupted restored file... but at least the corruption would be limited to that one item, and the rest of the Restic system will continue to function.
...Is this correct?
Thanks!

I found this page to be a good explanation for why we don't really need to worry about a collision:  https://www.reddit.com/r/crypto/comments/6rxgs1/what_is_the_significance_of_a_sha2256_hash/
In particular, I liked this comment by 'atoponce':
> Physics also places a lower bound on the required energy of computation.
> I don't have a number. But it's probably ridiculously high, too.

Energy is calculated as

    Energy = kT

Where 'k' is Boltzmann's constant of 1.38064852×10−16 ergs per Kelvin, and "T" is the
temperature of the system. The background temperature of the Universe is about
2.72548 Kelvin. Running a computer at a lower temperature would require more energy,
thus it would be less efficient.

As a result, this means that to flip a single bit with our ideal computer, it requires:

    Energy = (1.38064852×10^−16 ergs per Kelvin) × (2.72548 Kelvin) = 3.762929928×10^-16ergs

The total energy output from our star is about 1.2×1034 Joules per year. Because one
Joule is the same as 1×107 ergs, then the total annual energy output of the Sun is about
1.2×1041 ergs. So, doing some basic math:

    Bits flipped = (1.2×10^41 ergs) / (3.762929928×10^-16 ergs per bit) = 3.189004374×10^56 bits

3.189004374×1056 bits means I can flip every combination of bits in a 187-bit number, if I
could harness 100% of the solar energy output from the sun each year.

Our star is a weak one. Rather, if we harnessed 100% of the energy released in a hypernova,
we could do a lot more work. A hypernova is calculated to release something around
1046 Joules or 1053 ergs of energy. Doing that math:

    Bits flipped = (10^53 ergs) / (3.762929928×10^-16 ergs per bit) = 2.657503608×10^68 bits

2.657503608×1068 bits is approximately 227-bits. This is a computation orgy turned up to 11.

Of course, I would have to harness 100% of that energy into my ideal computer, to flip every
combination of these bits. Never mind finding transportation to get me to that hypernova, the
time taken in that travel (how many millions of light years away is it?), and the cost of the
equipment to harness the released energy.

This strongly implies that is it just not physically possible to flip every combination of bits in a
256-bit number.

After reading the source code, these are my guesses for what happens for various kinds of collisions:
    If Hash(Chunk) collides:  One chunk of data would be lost, resulting in corruption
                              of dependent file(s).

    If a Pack ID collides:  The extent of damage depends on the backend.
        * The built-in backends (local, sftp, rest-server) save packs like this:

                  fs.OpenFile(filename, os.O_CREATE|os.O_EXCL|os.O_WRONLY, backend.Modes.File)

          The O_CREATE combined with O_EXCL will prevent overwriting of
          old files.  Therefore, the save operation will produce an error, and
          Restic will probably crash.  The user would then need to try to save
          the data again, and new IDs would be used, and everything would be OK.

        * The built-in cloud backends (azure, ...) also try to use a "Create()" API
          call, to indicate that the backend should not overwrite existing data.

        * 'rclone' backends (for example 'ftp', 'webdav', 'dropbox') use "Update()"
          instead of a "Create" instruction.  Therefore, I think a pack collision
          would cause the old pack to be replaced by the new pack, which would
          cause major, unrecoverable Restic Repository corruption.

    If an Index ID collides:  The 'restic' program would probably crash, but the
                              data in the repository would be fine.  The user could
                              work around the problem by disabling the index, or
                              maybe by re-building the index.

    If a Snapshot ID collides:  Similar to Packs -- the result depends on the backend:
        * If the backend is well-behaved, 'restic' will crash but won't corrupt data,
          and the user will be able to try again and everything will be OK.

        * If the backend uses Update() instead of Create, the previous snapshot
          will be overwritten, which will cause some minor repository corruption.
          The user can work around the issue by deleting corrupted snapshot files.

Thanks for the in-depth analysis, I think it is correct :)
The only thing that's really bad (because the user wouldn't be notified) is if Hash(chunk) collides. The probability for that is extremely low, after all we're using a cryptographic hash function (SHA-256), one requirements for such a function is that it must be really really hard to produce a collision on purpose. Therefore, it's very unlikely that this happens by accident.

@likebike You seem to understand these things and have done a great job looking into it. Would you mind writing up a PR for the documentation? I imagine we'd put this information in some of the technical sections, as it's on a technical level that most users won't care about. And it's not a FAQ so it probably doesn't go in there.

Maybe an info in a log would be good if a collision was found or an info by the next check. For example, if the file size is different, but the hash is the same you can save a file "collision" like the "lockfile", then it would be clear. Or is that a performance problem? .. or add a cmd-flag like --paranoid for checking at backup whitch would be a bit slower :) .
I have read that CERN is planing to use restic.. what if there is by some PByte only one collision??

About the comment of @fd0. It really is very, very, very, very unlikely to get a hash collision.
Just have a look at the maths and discussions like this.
It is in fact so unlikely, that things like the earth getting completely destroyed by an asteroid in the next second is much, much more likely. The probability that any part of your computer (CPU, RAM, whatever) just produces a wrong result out of the blue is much more likely than this.
If we document this, please add a disclaimer that this is only a theoretical discussion and add some references to some of the discussions about hash collision probability.
@cd1188: I do not think that an extra check is needed if we also do not (can not) check for much more likely things where we just assume that things are working.

The probability for a hash collision needs to take the birthday paradox into account. Using the formula p<=n*(n-1)/2*1/2^m (m=length of a hash in bits) [1], this gives a probability of 1/2^197 = approx. 5e-60 for 1MB chunks in a 1 PB dataset.  The probability that some disks or your whole computer fails are much much higher than the risk of a hash collision.
Regarding the physical limits on bit flips: There seem to be some theoretical results that could allow for a more energy efficient computation.
[1] https://csyhua.github.io/csyhua/hua-PIEEE.pdf

My experience is that when things can go wrong, they will. So if there is a chance that the collision happens, then you simply do not have a reliable backup solution. You do not want to figure out during recovery that the file is corrupted. According to the probability this might happen just for one person on the planet, but this is wrong assumption for a reliable backup solution. Zero probability should be the target. Just my two cents.

@oldium Please see https://crypto.stackexchange.com/questions/8765/is-there-a-hash-function-which-has-no-collisions . While it would be great, what you want is not practical, so there has to be tradeoffs like with everything else. Also note that Git has the same "problem", yet people use it.
If you are truly concerned about this problem, instead of trying to fix one single software (which again isn't doable), you should instead make sure that you back up your data with two differently working softwares (of which one is restic, of course ;) - then if there were to be a collision in one of them, the other should be fine (although theoretically you could have them collide in the very same place, but it's undescribably unlikely assuming the softwares does hash things differently).

@oldium Sorry to be direct, but you are wrong here.

Zero probability should be the target. Just my two cents.

Please tell me how you would achieve this given you have a backup tool with doesn't use hashing? For example, your storage - hard disc, SSD, tape; whatever you take - has a probability greater than zero of flipping random bits. How do you get zero probability of loosing data here?
The point is, all things which are implemented to "secure" data are just measurements to decrease the probability of "things going wrong" down to a ridiculous low value - there is no way for zero probability in real life, sorry to tell you!

Please tell me how you would achieve this given you have a backup tool which doesn't use hashing? For example, your storage - hard disc, SSD, tape; whatever you take - has a probability greater than zero of flipping random bits. How do you get zero probability of loosing data here?

That is a completely valid point here. To overcome storage failure, you can duplicate the storage data - copy it to two different locations. You can work-around this. But what if your backup software, which is used to make your copies, is source of the error? You cannot workaround it by duplicating the storage, because you duplicate the error actually.
The problem is not the hashing, the problem is that the hashes are used as the only measure to match identical blocks – without checking if the blocks are really equal. A simple check would give you 0% probability to have an error, the hashes should be there just to speed-up searching for the identical block, not to be the only measure.
And one funny fact: A simple copy to a date-based folder is more reliable than a backup using hashes, when other measures to ensure block equality are not used.

To overcome storage failure, you can duplicate the storage data - copy it to two different locations.

No, you don't overcome it. Each of your two storages can has a probability greater than zero, that there is a bitflip. The probability that the bitflip occurs on both storages at the exact same position exists and is not zero!
(And, as you hopefully realize, you don't get zero probability, if you add as many copies as you want)

No, you don't overcome it. Each of your two storages has a probability greater than zero, that there is a bitflip. The probability that the bitflip occurs on both storages at the exact same position is not zero!

Correct. But this probability should not be increased by the backup software, shouldn't it?

@oldium Excuse me, but I have to ask; How do you suggest that the verification that blocks etc are identical is done, in practice? Hashes are used in order to have a fixed (or at least smaller) length of signature than the original data itself has. Since the only way to identify a piece of data with 0% probability of collisions is to use all of it, are you suggesting that every time a block of data has to be identified in restic, all of it is read? That would make things immensely slower. So what are you proposing as an alternative to how it works now?

@rawtaz You do not need to verify the colliding block immediately during backup, you can upload it as duplicated block, mark it for verification and deduplicate it as soon as it is verified.

How will you know which blocks are colliding blocks and which ones are not colliding but just deduplicated?

How will you know which blocks are colliding blocks and which ones are not colliding but just deduplicated?

@oldium:

Zero probability should be the target.

Quoting from Stackoverflow:

All harddisks ever produced on Earth can't hold enough 1MB files to get a likelihood of a collision of even 0.01% for SHA-256.

That makes it pretty much zero probability IMHO.

@rawtaz During upload of „unseen“ block with existing hash, you assume it could be a collision and create a duplicate for verification. The „unseen“ logic needs to use heuristics for decision if the block was already uploaded, for example if the block is from the same file, same date/time, same position, same size, same hash, than it does not need to go through the verification again.
I know that the probability is pretty much zero. I just think that the software should be able to verify that the collision happened and then it should handle it in a way that does not lead to a lost file. When you verify the backup consistency and it tells you that the file hash does not match, it is already too late. Low probability does not mean that it cannot happen, this is why it is non-zero. For example a 1% probability of failure means that the error happens (in average) once in 100 tries, but it could be the first try out of 100, but also the last one, but also first 100 out of 10000 tries. Be it 1 in 10^100 tries or more, who decided that it cannot be the first one?

I think you are trying to solve a problem that isn't a problem, in a way that I hardly even understand but that sounds completely messy. Using two different software for backing up your data, if you are this concerned about it's integrity, is a much much better solution :)

@oldium While your goal (zero probability of errors in software) is laudable, it can be never reached. What about software bugs? Bugs in the compiler? Bugs in the OS? Hardware bugs on the system that compiles your software? Hardware bugs on the system that runs your software?
In fact it is always about mitigating risks and lowering error probabilities. This is a question about priorities. When dealing wit hardware issues, I feel your priorites are right. If you fear that the probability of any bug in restic is too high for you, the best mitigation strategy would be IMHO to use another backup tool in addition, as @rawtaz also suggested.

@rawtaz During upload of „unseen“ block with existing hash, you assume it could be a collision and create a duplicate for verification. The „unseen“ logic needs to use heuristics for decision if the block was already uploaded, for example if the block is from the same file, same date/time, same position, same size, same hash, than it does not need to go through the verification again.

A "heuristic" is, by definition, a potentially incorrect way of solving a problem. Your heuristics for determining "unseen" blocks can fail with non-zero probability, just like hashes: a file can be modified in such a way that a block inside has different contents but the same hash, and then the file's modification date can be updated to be the same as before. Then the "unseen blocks heuristics" won't be able to detect the change.
You can't have 100% reliable backup algorithm, unless it copies all data each time (or directly compares the new data with the already recorded data).

Thanks for all the comments on this issue, I think we're clear how to proceed: add a short section to the manual to document it.

A "heuristic" is, by definition, a potentially incorrect way of solving a problem. Your heuristics for determining "unseen" blocks can fail with non-zero probability, just like hashes: a file can be modified in such a way that a block inside has different contents but the same hash, and then the file's modification date can be updated to be the same as before. Then the "unseen blocks heuristics" won't be able to detect the change.

Not really. With some assumptions, like verifying the whole file hash, this could be doable. When you detect file hash issue, you would need to upload the whole file for deduplication. Moreover, with verifying the file hash together with all block hashes you even lower the risk of not detecting file difference. And solving two hashes failure is like solving problem of two consecutive errors. I just want to solve one error, which is storing the wrong file content in the backup, not that the change was not detected (by the way, even simpler heuristics is used nowadays - just using block hashes for the modified file - isn't it?).

Not really. With some assumptions, like verifying the whole file hash, this could be doable. When you detect file hash issue, you would need to upload the whole file for deduplication. Moreover, with verifying the file hash together with all block hashes you even lower the risk of not detecting file difference. And solving two hashes failure is like solving problem of two consecutive errors. I just want to solve one error, which is storing the wrong file content in the backup, not that the change was not detected (by the way, even simpler heuristics is used nowadays - just using block hashes for the modified file - isn't it?).

All these measures can only reduce the probability of having wrong data in the backup, not bring it to zero. You can further classify these failures by dividing them into "failures to detect change", "failures to detect difference due to hash collision" and probably something else, and try to eliminate only specific classes of failure; but at the end of the day it's irrelevant: you'll never bring the final probability to zero without copying and comparing the entire data set. (Note that I'm talking here with certain assumptions in mind, namely: perfectly implemented algorithm that runs on perfectly reliable hardware; backup source is a modern general-purpose filesystem, which can't provide guarantees that its files/directories weren't changed, or report specific changed places.)
All you can do is to reduce the risk of having wrong data in the backup. And I think it's more important to prevent things going wrong in varying and unexpected user configurations/environments (weird file systems, eventually consistent backup storage, unreliable network), rather then trying to combat potential failures that provably have extremely low probability of occurring.
Also, you need to take into account the performance cost of all these measures. So it's important to compare the cost and the benefits, and employ sane trade-offs. Very few people will want to copy the entire dataset each time to eliminate, say, a 10^-20 chance of backing up wrong data due to hash failure; in practice, it will prevent people from making backups often enough (or even at all), and thus will actually greatly increase their chance of losing data, instead of reducing it.

The would be happy if the whole file hash could be used as an additional measure to verify the file content as mentioned in #1620.


Fix(Webview): Webview not loading fix.

Move internal server to express instead of unreliable/over-
complex HAPI.
Fixes #359

I gave it a quick try and it seems to solve the issue. Let me use this for a bit while before we effectively merge this, just in case there is some hidden issue.


wip: Use factory and DI for action creators

... don't hate me.

I think we've solved this with the ActionCreatorService pattern. Feel free to reopen if you disagree.


Don't highlight arguments including fn in their name

e.g. fn part of fn_foo in fn foobar<F: Fn()>(fn_foo: F) { was highlighted.

Hello @sinkuu,
Thank you for the PR.
I need to check it before I will merge it.

Everything is fine, but I want you to add another change.
There is the "test.rs" file in the repository's root.
Add there the following source code:

text
fn foo(fn_foo: Foo) {}
text

I saw the following line on top of test.rs, and added an automated test to spec/rust-spec.coffee.
Do I still have to add it to test.rs?
// TODO: Move all of these to automated grammar specs in spec/rust-spec.coffee

@sinkuu,
I read what zargany had told me.
You shouldn't do what I asked.

Nice fix. Thanks for submitting, @sinkuu.

@zargony,
As you are here, should I sometimes ping you to publish a new version?


Tests fixed and passed

Checklist

[x] I checked the Contributing Guidelines before creating this request.
[x] New extensions support iOS 8 or later.
[x] New extensions are written in Swift 3.
[x] New extensions were created in development branch.
[x] Pull request was created to development head branch.
[x] All extensions have a clear comments explaining their functionality, all parameters and return type in English.
[x] All comments headears have the word SwifterSwift: before description.

@SiarheiFedartsou Great work, Thank you!
will merge soon.

@SiarheiFedartsou Please add a description to your new extension, I was not able to understand what does it do


Package's Control

For Idea:
Need to development Plugin's Control.

[ ] Preferences > Package Settings > Auto Refresh

#SublimeText3 #Python #Plugin

Fixed with f5e43c2


Authentication tenant needs to be configured

I run yarn startand go to http://localhost:3000/. I click the Login button.  I am asked for my username and password and the 2 factor (authenticatior app). After this it returns to the login page an nothing happens.
If I click the second time only the 2nd factor is asked but still nothing happens. Interestingliy the checkbox Remember for 14 days is unchecked again.
My system:

Mac OS 10.15.6
node 14.12.0
yarn 1.22.5
Microsoft Edge 85.0.564.68

I found it. The tenant needs to be configured same as the client id in msalApp.ts. this should be documented.

... a verbose error message in the gui would be nice too. Now it is hidden in the JS console.


[TF 2.0 API Docs] tf.RegisterGradient

Related with https://github.com/tensorflow/tensorflow/issues/29309
tf.RegisterGradient's __init__ can raise TypeError.
So Added it.

I signed it!


shared/net: extend JoinHostPort to support names

fixed typo in a comment

safer this time I think @karasz

shared/v0.4.6 tagged


added workflow for closing old PRs

Description
This feature aims to automate the management of open PRs in a repository, ensuring that outdated or abandoned PRs are closed in a timely manner. By doing so, it helps maintain repository hygiene, improves workflow efficiency, and provides a better experience for both maintainers and contributors.
Related Issues
Fixes #168
Changes
List the detailed changes made in this PR.
Added a new feature to close old PRs
Checklist
Make sure to check off all the items before submitting. Mark with [x] if done.

[X] I have performed a self-review of my code
[X] I have commented my code, particularly in hard-to-understand areas
[X] My changes generate no new warnings
[X] I am working on this issue under GSSOC

@Satyam1923 please review and merge this PR
Also, please add labels to this PR