Installing VisualStudio appears to work, but actually doesn't

Not sure if this is the right place to put this...
I have a chef recipe that runs the following code block:
include_recipe 'chocolatey'

chocolatey 'visualstudio2013ultimate'

chocolatey 'visualstudio2013ultimate' do
  args '--force'
end

On the first run I get the output:
       Recipe: windows-build-agent::chocostudio

           * execute[install package visualstudio2013ultimate] action run
             - execute "C:\ProgramData\chocolatey/bin/choco" install -y  visualstudio2013ultimate

        (up to date)

On the second run I get the output:
       Recipe: windows-build-agent::chocostudio
        (up to date)
        (up to date)

neither of them actually install VS.  If I log into the instance and run choco install, I get this:
PS C:\programdata\chocolatey> .\choco.exe install visualstudio2013ultimate
Chocolatey v<IP_ADDRESS>
Installing the following packages:
visualstudio2013ultimate
By installing you accept licenses for the packages.
VisualStudio2013Ultimate v12.0.21005.20141031 already installed.
 Use --force to reinstall, specify a version to install, or try upgrade.

Chocolatey installed 0/1 package(s). 0 package(s) failed.
 1 package(s) had warnings.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).
Warnings:
 - visualstudio2013ultimate

The programs do not show up on disk, nor start menu.  The log file is as follows:
2015-09-23 18:22:13,475 [DEBUG] - Command line: "C:\ProgramData\chocolatey\choco.exe" install -y visualstudio2013ultimate
2015-09-23 18:22:13,475 [DEBUG] - Received arguments: install -y visualstudio2013ultimate
2015-09-23 18:22:13,553 [DEBUG] -
NOTE: Hiding sensitive configuration data! Please double and triple
 check to be sure no sensitive data is shown, especially if copying
 output to a gist for review.
2015-09-23 18:22:13,569 [DEBUG] - Configuration: CommandName='install'|
CacheLocation='C:\Users\ADMINI~1\AppData\Local\Temp'|
ContainsLegacyPackageInstalls='True'|
CommandExecutionTimeoutSeconds='2700'|
Sources='https://chocolatey.org/api/v2/'|Debug='False'|Verbose='False'|
Force='False'|Noop='False'|HelpRequested='False'|RegularOutput='True'|
PromptForConfirmation='False'|AcceptLicense='True'|
AllowUnofficialBuild='False'|Input='visualstudio2013ultimate'|
AllVersions='False'|SkipPackageInstallProvider='False'|
PackageNames='visualstudio2013ultimate'|Prerelease='False'|
ForceX86='False'|OverrideArguments='False'|NotSilent='False'|
IgnoreDependencies='False'|AllowMultipleVersions='False'|
ForceDependencies='False'|Information.PlatformType='Windows'|
Information.PlatformVersion='6.3.9600.0'|
Information.PlatformName='Windows Server 2012 R2'|
Information.ChocolateyVersion='<IP_ADDRESS>'|
Information.ChocolateyProductVersion='<IP_ADDRESS>'|
Information.FullName='choco, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=79d02ea9cad655eb'|

Information.Is64Bit='True'|Information.IsInteractive='False'|
Information.IsUserAdministrator='True'|
Information.IsProcessElevated='True'|Features.AutoUninstaller='False'|
Features.CheckSumFiles='True'|Features.FailOnAutoUninstaller='False'|
ListCommand.LocalOnly='False'|
ListCommand.IncludeRegistryPrograms='False'|
UpgradeCommand.FailOnUnfound='False'|
UpgradeCommand.FailOnNotInstalled='False'|
UpgradeCommand.NotifyOnlyAvailableUpgrades='False'|
NewCommand.AutomaticPackage='False'|SourceCommand.Command='unknown'|
FeatureCommand.Command='unknown'|PushCommand.TimeoutInSeconds='0'|
PinCommand.Command='unknown'|
2015-09-23 18:22:13,585 [DEBUG] - _ Chocolatey:ChocolateyInstallCommand - Normal Run Mode _
2015-09-23 18:22:13,585 [INFO ] - Installing the following packages:
2015-09-23 18:22:13,585 [INFO ] - visualstudio2013ultimate
2015-09-23 18:22:13,585 [INFO ] - By installing you accept licenses for the packages.
2015-09-23 18:22:17,538 [INFO ] -
DotNet4.5.1 v<IP_ADDRESS>40606
2015-09-23 18:22:17,585 [DEBUG] - Contents of 'C:\ProgramData\chocolatey\lib\DotNet4.5.1\Tools\ChocolateyInstall.ps1':
2015-09-23 18:22:17,600 [DEBUG] - if(-not (test-path "hklm:\SOFTWARE\Microsoft\.NETFramework\v4.0.30319\SKUs\.NETFramework,Version=v4.5.1")) {
Install-ChocolateyPackage 'dotnet451' 'exe' "/Passive /NoRestart /Log $env:temp\net451.log" 'http://download.microsoft.com/download/7/4/0/74078A56-A3A1-492D-BBA9-865684B83C1B/NDP451-KB2859818-Web.exe' -validExitCodes @(0,3010)
}
else {
     Write-Host "Microsoft .Net 4.5.1 Framework is already installed on your machine."
 }

2015-09-23 18:22:17,616 [DEBUG] - Calling command ['"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -NoLogo -ExecutionPolicy Bypass -Command "[System.Threading.Thread]::CurrentThread.CurrentCulture = '';[System.Threading.Thread]::CurrentThread.CurrentUICulture = ''; & import-module -name 'C:\ProgramData\chocolatey\helpers\chocolateyInstaller.psm1'; & 'C:\ProgramData\chocolatey\helpers\chocolateyScriptRunner.ps1' -packageScript 'C:\ProgramData\chocolatey\lib\DotNet4.5.1\Tools\ChocolateyInstall.ps1' -installArguments '' -packageParameters ''"']
2015-09-23 18:22:18,491 [INFO ] -  Microsoft .Net 4.5.1 Framework is already installed on your machine.
2015-09-23 18:22:18,522 [DEBUG] - Command ['"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -NoLogo -ExecutionPolicy Bypass -Command "[System.Threading.Thread]::CurrentThread.CurrentCulture = '';[System.Threading.Thread]::CurrentThread.CurrentUICulture = ''; & import-module -name 'C:\ProgramData\chocolatey\helpers\chocolateyInstaller.psm1'; & 'C:\ProgramData\chocolatey\helpers\chocolateyScriptRunner.ps1' -packageScript 'C:\ProgramData\chocolatey\lib\DotNet4.5.1\Tools\ChocolateyInstall.ps1' -installArguments '' -packageParameters ''"'] exited with '0'
2015-09-23 18:22:18,522 [DEBUG] - Calling command ['"shutdown" /a']
2015-09-23 18:22:18,569 [DEBUG] - Command ['"shutdown" /a'] exited with '1116'
2015-09-23 18:22:18,585 [DEBUG] - Capturing package files in 'C:\ProgramData\chocolatey\lib\DotNet4.5.1'
2015-09-23 18:22:18,585 [DEBUG] -  Found 'C:\ProgramData\chocolatey\lib\DotNet4.5.1\dotnet.png'
  with checksum '0D5A15B93F47DF7F91B24A14C7CE91FE'
2015-09-23 18:22:18,585 [DEBUG] -  Found 'C:\ProgramData\chocolatey\lib\DotNet4.5.1\DotNet4.5.1.nupkg'
  with checksum '644E236231ADA16E07DA6C9B9612F508'
2015-09-23 18:22:18,585 [DEBUG] -  Found 'C:\ProgramData\chocolatey\lib\DotNet4.5.1\Tools\ChocolateyInstall.ps1'
  with checksum '805DDE099F331DF8A4F0D47BE8D47E7C'
2015-09-23 18:22:18,600 [DEBUG] - Attempting to create directory "C:\ProgramData\chocolatey\extensions".
2015-09-23 18:22:18,600 [DEBUG] - Attempting to create directory "C:\ProgramData\chocolatey\.chocolatey".
2015-09-23 18:22:18,616 [DEBUG] - Attempting to create directory "C:\ProgramData\chocolatey\.chocolatey\DotNet<IP_ADDRESS>.5.1.20140606".
2015-09-23 18:22:18,631 [DEBUG] - Attempting to delete file "C:\ProgramData\chocolatey\.chocolatey\DotNet<IP_ADDRESS>.5.1.20140606\.sxs".
2015-09-23 18:22:18,631 [DEBUG] - Attempting to delete file "C:\ProgramData\chocolatey\.chocolatey\DotNet<IP_ADDRESS>.5.1.20140606\.pin".
2015-09-23 18:22:18,631 [INFO ] -  The install of dotnet4.5.1 was successful.
2015-09-23 18:22:18,631 [INFO ] -
VisualStudio2013Ultimate v12.0.21005.20141031
2015-09-23 18:22:18,631 [DEBUG] - Contents of 'C:\ProgramData\chocolatey\lib\VisualStudio2013Ultimate\Tools\ChocolateyInstall.ps1':
2015-09-23 18:22:18,631 [DEBUG] - . (Join-Path $(Split-Path -parent $MyInvocation.MyCommand.Definition) 'common.ps1')

$adminFile = (Join-Path $(Split-Path -parent $MyInvocation.MyCommand.Definition) 'AdminDeployment.xml')
$customArgs = $env:chocolateyInstallArguments
$env:chocolateyInstallArguments=""

$settings = Initialize-VS-Settings $customArgs $adminFile
$installerArgs = Get-VS-Installer-Args $settings.ProductKey

Install-ChocolateyPackage 'VisualStudio2013Ultimate' 'exe' $installerArgs 'http://download.microsoft.com/download/C/F/B/CFBB5FF1-0B27-42E0-8141-E4D6DA0B8B13/vs_ultimate_download.exe'

2015-09-23 18:22:18,631 [DEBUG] - Calling command ['"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -NoLogo -ExecutionPolicy Bypass -Command "[System.Threading.Thread]::CurrentThread.CurrentCulture = '';[System.Threading.Thread]::CurrentThread.CurrentUICulture = ''; & import-module -name 'C:\ProgramData\chocolatey\helpers\chocolateyInstaller.psm1'; & 'C:\ProgramData\chocolatey\helpers\chocolateyScriptRunner.ps1' -packageScript 'C:\ProgramData\chocolatey\lib\VisualStudio2013Ultimate\Tools\ChocolateyInstall.ps1' -installArguments '' -packageParameters ''"']
2015-09-23 18:22:19,928 [INFO ] -  Downloading VisualStudio2013Ultimate 32 bit
2015-09-23 18:22:19,944 [INFO ] -    from 'http://download.microsoft.com/download/C/F/B/CFBB5FF1-0B27-42E0-8141-E4D6DA0B8B13/vs_ultimate_download.exe'
2015-09-23 18:22:22,428 [INFO ] -  Installing VisualStudio2013Ultimate...
2015-09-23 18:22:29,038 [INFO ] -  VisualStudio2013Ultimate has been installed.
2015-09-23 18:22:29,053 [DEBUG] - Command ['"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -NoLogo -ExecutionPolicy Bypass -Command "[System.Threading.Thread]::CurrentThread.CurrentCulture = '';[System.Threading.Thread]::CurrentThread.CurrentUICulture = ''; & import-module -name 'C:\ProgramData\chocolatey\helpers\chocolateyInstaller.psm1'; & 'C:\ProgramData\chocolatey\helpers\chocolateyScriptRunner.ps1' -packageScript 'C:\ProgramData\chocolatey\lib\VisualStudio2013Ultimate\Tools\ChocolateyInstall.ps1' -installArguments '' -packageParameters ''"'] exited with '0'
2015-09-23 18:22:29,053 [DEBUG] - Calling command ['"shutdown" /a']
2015-09-23 18:22:29,085 [DEBUG] - Command ['"shutdown" /a'] exited with '1116'
2015-09-23 18:22:29,100 [DEBUG] - Capturing package files in 'C:\ProgramData\chocolatey\lib\VisualStudio2013Ultimate'
2015-09-23 18:22:29,100 [DEBUG] -  Found 'C:\ProgramData\chocolatey\lib\VisualStudio2013Ultimate\VisualStudio2013Ultimate.nupkg'
  with checksum '307AA81594B864085363287C08054360'
2015-09-23 18:22:29,100 [DEBUG] -  Found 'C:\ProgramData\chocolatey\lib\VisualStudio2013Ultimate\Tools\AdminDeployment.xml'
  with checksum '01F61D4825533688114F2225072F92E4'
2015-09-23 18:22:29,100 [DEBUG] -  Found 'C:\ProgramData\chocolatey\lib\VisualStudio2013Ultimate\Tools\ChocolateyInstall.ps1'
  with checksum '557ED94554A30A376F8C4580ADA76041'
2015-09-23 18:22:29,100 [DEBUG] -  Found 'C:\ProgramData\chocolatey\lib\VisualStudio2013Ultimate\Tools\ChocolateyUninstall.ps1'
  with checksum 'F1A1396EC8B29639871B06C1360ACC3D'
2015-09-23 18:22:29,100 [DEBUG] -  Found 'C:\ProgramData\chocolatey\lib\VisualStudio2013Ultimate\Tools\common.ps1'
  with checksum 'CAD2FE3515E9D169FA58CD3ECC49A900'
2015-09-23 18:22:29,100 [DEBUG] - Attempting to create directory "C:\ProgramData\chocolatey\.chocolatey\VisualStudio2013Ultimate.12.0.21005.20141031".
2015-09-23 18:22:29,100 [DEBUG] - Attempting to delete file "C:\ProgramData\chocolatey\.chocolatey\VisualStudio2013Ultimate.12.0.21005.20141031\.sxs".
2015-09-23 18:22:29,100 [DEBUG] - Attempting to delete file "C:\ProgramData\chocolatey\.chocolatey\VisualStudio2013Ultimate.12.0.21005.20141031\.pin".
2015-09-23 18:22:29,100 [INFO ] -  The install of visualstudio2013ultimate was successful.

If I manually run choco install --force visualstudio2013ultimate after all of this, it will install just fine.
Any suggestions?
Windows server 2012R2 with .NET 4.5.2 installed (manually) build from an AWS AMI

TL;DR: I think the installation exe needs to be run in interactive mode, which you can replicate in Chef by importing a scheduled task XML file that explicitly runs chocolatey as the local administrators group.
I had this same problem with VS 2015 Enterprise and had to come up with a somewhat unorthodox solution. My thesis is that the VS installation program needs to be run in interactive mode in order to work. When chocolatey is executed during a chef converge it is not running in an interactive session in our environment.
However, if you schedule a task that fires off the exact same chocolatey command it will work since you can set the task to run as an interactive user. The key for me was having the task run as a the administrators group, but you can't specify a runas group with schtasks on the command line. You can, however, dump the XML definition of said task and lay it down with a chef template!
windows_task 'import_task_xml' do
  command "schtasks /create /xml '#{your_hacky_xml_path}' /tn vs_install"
  frequency :monthly
  run_level :highest
  action [:create, :run]
end

template your_hacky_xml_path do
  source 'install_visualstudio_task.xml.erb'
  action :create
end

powershell_script 'install visualstudio' do
  not_if { is_visualstudio_installed }
  code <<-EOF
schtasks /run /tn vs_install
do { Sleep 15 } while (ps -name choco)
do { Sleep 15 } while (ps -name vs_install_exe_thingy)
EOF
  action :run
end

I made the task in the task scheduler GUI and played with it until I got it into the state where it showed the chocolatey package downloading in a separate cmd window and the VS installation progress bar showed up, then did a right-click and export to the cookbook's file directory.
That last powershell_script resource I would say is optional if you are in charge of building the chocolatey package since you can bake the looping logic into the install script. In my case I was not, so I had to hack around it.
Might be an idea to get you unstuck ^_^

Are you running the chef client remotely via winrm? The issue with dotnet and visual studio is that they pull down some bits via windows update server which will fail via winrm. Using a scheduled task will allow it to run in a local context. It's not that it needs an interactive user so much as a local one.

@mwrock Spot on! In my particular case I am triggering the installation via packer, which is using remote winrm.

Same is true for SQL Server over WinRM, something I discovered recently with TestKitchen. There the root of the issue is a Crypto permissions problem, however it's defeated the same way (scheduled task  route, using a special provider the name of which is obvious but eludes me anyway!).

@jedugas Interesting, if you remember what that provider is let me know. I'd love to clean up my hackiness :)

This is is, specific to TestKitchen...
https://github.com/smurawski/chef-zero-scheduled-task
On Thu, Dec 24, 2015 at 11:05 AM -0800, "Dan Sedlacek"<EMAIL_ADDRESS>wrote:
@jedugas Interesting, if you remember what that provider is let me know. I'd love to clean up my hackiness :)
—
Reply to this email directly or view it on GitHub.

Thanks! :+1:

closing now since it seems lie this is not an issue with the cookbook and workarounds have been found.


Add xDai Support

More info about using thegraph in xdai in their dev documentation

https://thegraph.com/docs/deploy-a-subgraph#deploying-the-subgraph-to-multiple-ethereum-networks

The SC of KlerosLiquid in xDai can be found here:  https://github.com/kleros/xdai-kleros-liquid

Closed in 15a4d07daf620431aa07c2bcf4b13e8c59558ebc


Amazon Web Services Kql validation Failure Fix

Required items, please complete
Change(s):

Fixed failing kql validation for hunting queries in Amazon Web Services Solution

Reason for Change(s):

Failing KQL Validations

Version Updated:

Required only for Detections/Analytic Rule templates
See guidance below

Testing Completed:

See guidance below

Checked that the validations are passing and have addressed any issues that are present:

See guidance below

Guidance <- remove section before submitting

Before submitting this PR please ensure that you have read the following sections and filled out the changes, reason for change and testing complete sections:
Thank you for your contribution to the Microsoft Sentinel Github repo.

Details of the code changes in your submitted PR.  Providing descriptions for pull requests ensures there is context to changes being made and greatly enhances the code review process.  Providing associated Issues that this resolves also easily connects the reason.

Change(s):

Updated syntax for XYZ.yaml

Reason for Change(s):

New schema used for XYZ.yaml
Resolves ISSUE #1234

Version updated:

Yes
Detections/Analytic Rule templates are required to have the version updated

The code should have been tested in a Microsoft Sentinel environment that does not have any custom parsers, functions or tables, so that you validate no incorrect syntax and execution functions properly.  If your submission requires a custom parser or function, it must be submitted with the PR.

Testing Completed:

Yes/No/Need Help

Note: If updating a detection, you must update the version field.

Before the submission has been made, please look at running the KQL and Yaml Validation Checks locally.
https://github.com/Azure/Azure-Sentinel#run-kql-validation-locally

Checked that the validations are passing and have addressed any issues that are present:

Yes/No/Need Help

Note: Let us know if you have tried fixing the validation error and need help.

References:

Guidance for Detection checks
General contribution guidance
PR validation troubleshooting

I am following up with owner.

Hi, we are waiting for update in schema of Azure Monitor Logs in the ASI-Common repository, so we have this PR's updates on hold, thanks!

Hi, we are still waiting for schema update, thanks!

Hi, we are still waiting for schema update, thanks!

Hi, waiting for schema update, thanks!

Hi, waiting for schema update, thanks!

Hi, we are still waiting for schema update, thanks!

Hi, waiting for schema update, thanks!

Hi, we are still waiting for schema update, thanks!

Hi, waiting for schema update, thanks!

Following up changes in PR #8074 as the schema has been updated


Rsizing of output area (with too much data?) generates error.

Description:
Resizing of output area to maximum generates error
Steps to Reproduce:
Steps:
Download Data:
https://www.statistik-nord.de/fileadmin/Dokumente/Statistische_Berichte/bevoelkerung/A_I_S_1_j_H/A_I_S1_j16.xlsx
run script:
# %%
import pandas as pd
path = 'A_I_S1_j16.xlsx'
data = pd.read_excel(path, sheet_name='T3_1')
data.head(8)

then resize output area to maximum width
Versions:
Atom    : 1.23.3
Electron: 1.7.11
Chrome  : 58.0.3029.110
Node    : 7.9.0
error-log.txt

I believe this to be fix. Please test this on a current version of hydrogen. If you still get an error, feel free to open a new issue.


Add package eventmux

#46  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @jespert and the rest of your teammates on  Graphite


Const opSlice, upperBound, lowerBound and equalRange for rbtree

Const versions of opSlice, upperBound, lowerBound and equalRange functions are added. They return ConstRange that is a range of const type elements.

I think this is the wrong approach. First, you should not be casting away const to run an algorithm which is actually const. You should do ti the other way around (implement the const version, then cast within the mutable one).
Second, this is missing the immutable equivalent (easy to add when you have the boilerplate for const). This also will not work with inout unfortunately.
Third, the best way to fix this is to fix the language. Although, we can likely include a reasonable compromise here and then remove the compromise when the language is fixed.

Thank you, I agree, more work is needed here. I'll try to fix it.

First, you should not be casting away const to run an algorithm which is actually const. You should do it the other way around (implement the const version, then cast within the mutable one).

That would mean call the const opSlice on a mutable this, cast the result back to mutable, and return it, right? I think that's no good. Casting away const and then mutating is undefined behaviour.
The other way around (cast this from const to mutable, call opSlice on it, return as const) doesn't have that problem, as there's no mutating going on when this is briefly painted mutable.
-> I think it's correct the way it is, and it wouldn't be correct the way you suggest.
However, simply duplicating the function bodies is an option, too. opSlice, upperBound, and lowerBound are one-liners. Just repeating them with different range types would be shorter and possibly less bug-prone than casting cleverness. equalRange is more complex, I'd avoid duplicating that one.

Third, the best way to fix this is to fix the language. Although, we can likely include a reasonable compromise here and then remove the compromise when the language is fixed.

We did the very same ConstRange/ImmutableRange thing with Array (#2631). No one likes it, but no one came up with something better either.

Casting away const and then mutating is undefined behaviour.

Casting away immutable and then mutating is undefined behavior. Const is OK as long as the actual data is mutable. Obviously this is true from a mutable version of the function.

However, simply duplicating the function bodies is an option, too. opSlice, upperBound, and lowerBound are one-liners. Just repeating them with different range types would be shorter and possibly less bug-prone than casting cleverness.

This is an excellent point, can we just do this? I'd rather avoid casting wherever possible, even if it means not all the implementations use the same "tricks"

Casting away immutable and then mutating is undefined behavior. Const is OK as long as the actual data is mutable.

I have my doubts about this. I'm pretty sure it's wrong in the general case, and I think it may be wrong in this specific case, too.
Surely the following is not ok, is it?
void f(const int* c) pure {* cast(int*) c = 2;}
void main()
{
    int x = 1;
    const int* c = &x;
    f(c);
}

x is mutable, but the compiler should be allowed to assume that f doesn't change it.
When we inline f, we get this:
void main()
{
    int x = 1;
    const int* c = &x;
    * cast(int*) c = 2;
}

Which you're saying is ok, right? It looks less problematic, but unless I'm mistaken, which I well may be, the two snippets should be equivalent, i.e. either both ok or both bad.
Anyway, what's wrong with casting the other way around?

I'm pretty sure it's wrong in the general case

Yes, definitely if you aren't sure if the const data is immutable or mutable, you cannot mutate.

the compiler should be allowed to assume that f doesn't change it.

I think in this case, yes, it should be able to assume that.
But that is not the same here. The casting occurs locally, not wrapped inside a function.
This is more of the equivalent:
const int *f(const int *c) pure { return c;}

void main()
{
   int x;
   auto y = f(&x);
   *(cast(int *)y) = 2;
}

Now, can the compiler assume that y doesn't point at x? I don't see how it can assume that.

what's wrong with casting the other way around?

Because this turns off the compiler checks for const. One is free to mutate inside the function without realizing you are compromising the const/immutable versions, and the compiler won't complain. If you do it the other way, the dangerous code is not the implementation, it's simply plumbing.

This is an excellent point, can we just do this?
Sure, I will try to manage without castings.

Yes, definitely if you aren't sure if the const data is immutable or mutable, you cannot mutate.

I meant that the data being mutable is not enough to allow casting away const and then mutating. You're agreeing on the example then, so I think we're on the same page here.

But that is not the same here. The casting occurs locally, not wrapped inside a function.

I'm not completely sure if it's correct, but I'm under the impression that when a line of code itself is ok, then turning it into a function should be ok, too. Just like the other way around, inlining any function call should be ok (ignoring recursion for a moment).

This is more of the equivalent:
const int *f(const int *c) pure { return c;}

Fixed: const(int*) f(const int *c) pure { return c;}

void main()
{
int x;
auto y = f(&x);
*(cast(int *)y) = 2;
}
Now, can the compiler assume that y doesn't point at x? I don't see how it can assume that.

I agree that it can't assume that *y is immutable, but I think it should be allowed to assume that x won't be mutated through y, as that's the const promise.
And when the compiler can rely on the const promise, then I think it could put x into immutable storage here:
There are two references to the data: x is a mutable reference, y is a const reference. x is only ever used to initialize y, i.e. x is never used to change the data. y can't change the data, says the const promise. => The data won't change, and can be put into immutable storage.

what's wrong with casting the other way around?

Because this turns off the compiler checks for const. One is free to mutate inside the function without realizing you are compromising the const/immutable versions, and the compiler won't complain. If you do it the other way, the dangerous code is not the implementation, it's simply plumbing.

I see. Yes, ensuring correctness and maintainability are problems. I still think that the alternative is outright invalid, though.

I agree that it can't assume that *y is immutable, but I think it should be allowed to assume that x won't be mutated through y, as that's the const promise.

const promises go away when you cast. If the compiler sees the cast, it has to assume the worst.
And in any case, I need to refine the analogous example. Because the function that does the cast isn't actually mutating anything. So really it looks more like this:
const(int) *f1(const(int)* x) { return x;}
int *f2(int *x) { return cast(int *)f1(x);}

void main()
{
   int x;
   auto y = f2(&x);
   *y = 2;
}

It seems to me there can't be any invalid assumptions made by the compiler due to const in this case when it examines each function.

I still think that the alternative is outright invalid

I don't see in the spec where that is so. Only modifying immutable data is invalid that I can find.

const promises go away when you cast. If the compiler sees the cast, it has to assume the worst.

Once you cast away const, it's up to you to guarantee that those promises aren't violated, and as I understand it, casting away const does not stop the compiler from making assumptions about any const references to the same data. So, if you cast away const and mutate the object, even if the object is actually mutable underneath the hood, you risk running afoul of assumptions made by the compiler. As long as the compiler can't do anything based on the assumption that a const variable isn't going to change, then it won't matter, but if it can do something based on that, then as I understand it, it's going to. So, if it's doing something based on the assumption that the variable's value didn't change, and you changed it by casting away const and mutating it anyway, then you're going to have problems. What makes it so that you can usually get away with it is the fact that there are very few assumptions that can be made based on const (since to be able to assume that the value wasn't changed, it has to be sure that there are no mutable references to the same data which are getting accessed at the same time, which can sometimes be known thanks to pure, but usually, it can't be known). But if you cast away const and mutate the value, you do run the risk of the compiler getting smarter later and making more assumptions based on const than it does now. So, even if it works now, that doesn't guarantee that it will later.
So, personally, my take on it is that you just don't cast away const and mutate. It's too risky. Yes, you can theoretically get away with it if you're careful (particularly since it is very hard to do much based on const as opposed to immutable), but I wouldn't risk it.
Casting away const without mutating is fine, if you're careful, though it's ugly, so I really wouldn't do that either unless I had no choice, but it doesn't run the same risks. You just have to make sure that you're not actually mutating anything.

casting away const does not stop the compiler from making assumptions about any const references to the same data

Const provides no guarantees of immutability to the compiler. A const variable can easily be mutated via a mutable reference. The spec rightly says it's UB to cast away const/immutable and mutate immutable data.

Casting away const without mutating is fine, if you're careful, though it's ugly

What if you KNOW that the underlying data is mutable (as in this case)? And in this case actually, the code that knows the data is mutable is not actually mutating, but just restoring the mutable attribute via a cast.
Really, what you and @aG0aep6G are asking is that the compiler be intelligent enough to prove that one variable can only be accessed through const references, but to willfully ignore the casting of such a variable. I think that's quite a silly proposition. IMO, the cast should disable any such optimizations.
The one case where I think this is a problem is if a pure function can accept a const reference, with no other mutable references that could possibly lead back to that const reference, casts away that const and modifies. Other than that, I think it's difficult to imagine a compiler that is so highly intelligent in flow analysis, but ignores one very important aspect of it.
I ask again, is there any part of the spec that says you cannot cast away const and modify if the underlying data is mutable? If not, I can't see how this is even worth discussing. The overwhelmingly logical choice here is to let the compiler prove the implementation is valid for const, and then do the casting outside where you know the cast is valid.

Really, what you and @aG0aep6G are asking is that the compiler be intelligent enough to prove that one variable can only be accessed through const references, but to willfully ignore the casting of such a variable. I think that's quite a silly proposition. IMO, the cast should disable any such optimizations.

Even if it is smart enough to not make assumptions around the point where the cast is made (which I would hope that it would be that smart), as soon as other function calls get involved, you're asking it to be that smart across function boundaries, and optimizations don't generally work that way.

I ask again, is there any part of the spec that says you cannot cast away const and modify if the underlying data is mutable? If not, I can't see how this is even worth discussing. The overwhelmingly logical choice here is to let the compiler prove the implementation is valid for const, and then do the casting outside where you know the cast is valid.

I don't know what the spec says, but it has come up before - including in discussions that included Walter - that the compiler can make assumptions based on const not mutating data and that it is undefined behavior to mutate an object that was const - precisely because the compiler is free to assume that data is not mutated via a const reference. e.g.
const myFoo = getFoo();
pureFunc(myFoo);
// It should be impossible for the value of myFoo to have changed at this point

If pureFunc were not pure, then there the compiler can't know that myFoo hasn't changed, because pureFunc would then have been able to access data that wasn't passed to it and thus might have been able to access the same data that myFoo refers to via another reference, but since it is pure, it does know that no other references to what myFoo refers to can have been accessed and thus knows that its state has not changed.
Now, whether the compiler currently does much based on that knowledge, I don't know. But I'm 100% certain that casting away const to mutate an object is undefined behavior, even if you know that the underlying object can't be immutable, because this has been discussed quite a bit in the past - particularly with regards to logical const. And if the spec isn't clear on that, then I would consider it a bug in the spec.

const(int) f1(const(int) x) { return x;}
int *f2(int *x) { return cast(int *)f1(x);}
void main()
{
int x;
auto y = f2(&x);
*y = 2;
}
It seems to me there can't be any invalid assumptions made by the compiler due to const in this case when it examines each function.

I don't think this changes anything. If the compiler is allowed to look at all the code, it's the same situation as before, isn't it? My take on it is that once you've gone const you can't go back to mutable (from the const reference) and mutate.

Really, what you and @aG0aep6G are asking is that the compiler be intelligent enough to prove that one variable can only be accessed through const references, but to willfully ignore the casting of such a variable. I think that's quite a silly proposition. IMO, the cast should disable any such optimizations.

It's a silly, unrealistic scenario, yes. But I think a hypothetical, silly compiler like that should be allowed.

I ask again, is there any part of the spec that says you cannot cast away const and modify if the underlying data is mutable? If not, I can't see how this is even worth discussing.

You agree that there are cases where casting away const and then mutating is not ok, even when the data is mutable (see the first example in https://github.com/D-Programming-Language/phobos/pull/3501#issuecomment-123503312 and your answer to that). If that's not in the spec, I hope you agree that we should add it.
Now, your case is different from that one, but the spec also doesn't say that it's ok to cast away const and then mutate when the data is mutable. And adding that would be bad, given the example above.
So we can either go the simple route and disallow casting away const and then mutating. Or we can go a more complex route that allows your use case while forbidding my example. I'd vote for the simple one as it's easier to remember, easier to verify to be correct, and easier to apply correctly.

The overwhelmingly logical choice here is to let the compiler prove the implementation is valid for const, and then do the casting outside where you know the cast is valid.

I can't see how your take on this is the "overwhelmingly logical choice". I think the key point where we're apart is that you think that the compiler should recognize un-const-ing casts and not rely on the const promise accordingly, whereas Jonathan and I think that the const promise still applies and the programmer is then responsible for maintaining it.

Actually, what I'm saying is exactly how inout should work if it were allowed to be applied here. I think this is simply a forced implementation of inout, and should be allowed.
What you have for the conditions is:

Inside a function, the data is treated as const.
Once the function is finished, the data is returned to the same state it was before const was applied.

If you have this, I think it should be, and is, defined and valid. What is not valid is casting away const (and mutating) while inside the function (which is not aware of the mutability of the data inside the function).

I don't think this changes anything. If the compiler is allowed to look at all the code, it's the same situation as before, isn't it?

No. If the compiler can look at (and inline) all the code, it becomes:
auto y = &x;

And it works fine.

If that's not in the spec, I hope you agree that we should add it.

I agree the spec should identify that if the the compiler can prove that a variable cannot be modified inside a pure function (and it is allowed to assume no casts occur inside the function), then it can assume the variable has not been modified by the function.

But I'm 100% certain that casting away const to mutate an object is undefined behavior, even if you know that the underlying object can't be immutable

Can you back this up besides anecdotal evidence? I can't see at all why this is UB:
int x;
const int *y = &x;
*(cast(int *)y) = 5;

const only exists in the compiler, it doesn't survive to generated code, the CPU has no concept of it. Any compiler that doesn't just compile the above and set x to 5 is wrong.

No. If the compiler can look at (and inline) all the code, it becomes:
auto y = &x;
And it works fine.

You've omitted the type changes. With them:
auto y = cast(int*) cast(const int*) &x;

Your position (as I understand it) is that the compiler should either be dumb and not see a const promise in that cast to const, or if it's smart, it must recognize that the promise is taken back by the cast to mutable. You're disallowing silly/ridiculous compilers that would recognize the const promise but not the cast to mutable.

I agree the spec should identify that if the the compiler can prove that a variable cannot be modified inside a pure function (and it is allowed to assume no casts occur inside the function), then it can assume the variable has not been modified by the function.

That is, you're restricting the const promise to function parameters.

Actually, what I'm saying is exactly how inout should work if it were allowed to be applied here. I think this is simply a forced implementation of inout, and should be allowed.
What you have for the conditions is:
Inside a function, the data is treated as const.
Once the function is finished, the data is returned to the same state it was before const was applied.

If you have this, I think it should be, and is, defined and valid. What is not valid is casting away const (and mutating) while inside the function (which is not aware of the mutability of the data inside the function).

I'm having trouble understanding these conditions, but I guess it comes down to restricting the const promise to function parameters as above.
I don't think const is defined that way. It may actionable to do so. That is, allow casting away const and then mutating as long you don't cross function boundaries. Or something more precise along these lines.
I can't say that I understand all the implications that may have, and I think it's significantly more complex (and therefore prone to be subtly wrong) than simply saying "don't cast away const and then mutate". So I'm not a fan.
I understand that the spec doesn't actually say "don't cast away const and then mutate" either. All it says (which I could find) is "you can't mutate through a const reference".
I think we may have reached a point in the discussion where everyone understands the other side's points, yet neither of us has been convinced by them.
Shall we take it to a broader audience? I made a thread on the forum:
<EMAIL_ADDRESS>

I would point out that regardless of the issues of whether casting away const and mutating is ever defined behavior, I think that doing anything with const ranges at this point is a bad idea. We need to come up with a general solution for this before we start trying to add it to the containers. const and ranges simply do not mix as it stands. You have to go to great lengths to make it work. It's simpler to get a range over const elements from a container than it is to get a tail-const range, but that's really what needs to be solved, and how that is solved could affect how we get ranges over const elements from a container.

It's simpler to get a range over const elements from a container than it is to get a tail-const range

That's what this PR does.

That's what this PR does.

Yes. But I don't think that we should be doing even that without solving the more general problem, because how that is solved could affect how this should be done.

Not really, although I'd love to solve the general problem. I think whatever we came up with would allow a drop-in replacement. All that happens is magically mutable ranges can now be implicitly cast to const ranges. We can leave the ConstRange alias to whatever a tail-const version of the range is.

Are you guys sure the current language is unable to express a const member function that returns a tail-const mutable range? I didn't look too closely at the details, but I remember writing similar code before, where it's just a matter of explicitly specifying the types of your iteration pointers so that they are tail-const, which the compiler allows assigning const pointers to without casting:
struct MyContainer
{
    Data* root;
    auto getRange() inout
    {
        struct Range
        {
            inout(Data)* currentElem; // N.B. this pointer is tail-const
            ...
            void popFront()
            {
                // AFAIK this is allowed, it's assigned const(Data*) to const(Data)*
                currentElem = currentElem.next;
            }
        }
        return Range(root);
    }
}

Are you guys sure the current language is unable to express a const member function that returns a tail-const mutable range?

Yeah, you can. The issue is, we need inout to run the algorithm (so it goes back to the constancy of this when the algorithm is done). Otherwise, you have to copy/paste the whole thing. And you can't make a tail-inout type, since inout can't be used on a struct member (and the compiler wouldn't know what to do with it if you did).

@quickfur I didn't read your code sample closely enough. what you wrote will not work, because you cannot make a struct with an inout member. You can make const or immutable versions of the range, but they won't implicitly cast right, which is what this PR is about.

@jmdavis @aG0aep6G I concede the UB argument. It's clear that we could make certain cases of behavior defined, but the more I think about it, the more this seems like a horrible hack fix to cover over the lack of tail-const.
We can fix this anyway with @aG0aep6G's suggestion of copying the implementation to const/mutable/immutable for all 3 functions (upperBound lowerBound and equalRange). However, one thing is needed. We need to make _firstGreater and _firstGreaterEqual inout, since those actually deal in Node (a raw pointer which we can tail-ify).
So essentially (and I think you can just do this without changing any code, it's already done for _find), signatures become:
inout(RBNode)* _firstGreater(Elem e) inout
inout(RBNode)* _firstGreaterEqual(Elem e) inout

And then your implementations for all three constancies of each higher-level function can be (mostly) identical. It's repeated code, but you can't template based on the constancy of this. Hopefully these plumbing functions will be replaceable with one function once we can do tail-inout properly.
Sorry for the friction.

Unnecessary castings are eliminated. Current implementation is similar to this one that @schveiguy suggested above.

We could get rid of the duplication with template this parameters:
    private alias QualifiedRange(This : immutable RedBlackTree) = ImmutableRange;
    private alias QualifiedRange(This : const RedBlackTree) = ConstRange;
    private alias QualifiedRange(This : RedBlackTree) = Range;

    auto opSlice(this This)()
    {
        return QualifiedRange!This(_begin, _end);
    }

    /* ... analogous with upperBound, lowerBound, equalRange ... */

This would turn those methods into templates, meaning they're not virtual. But the class is final, so all methods aren't virtual anyway.
There may be other downsides I fail to see right now.
I don't think the duplication is too bad, though. So this looks good to me either way. (Aside from the PointerTarget ugliness mentioned above.)

Thank you, I fixed PointerTarget ugliness.
I like this approach with template this parameter.
Actually I missed the fact that template this parameterized method gets the same qualifier as this. It should be described in documentation explicitly (like in changelog). I also changed _equalRange helper to template this parameterized method.

Nice. LGTM.

One more thing: When we did this for Array (#2631) we moved RangeT out of the container to reduce template bloat. We should probably do the same here.
I.e., before:
class RedBlackTree(T)
{
    struct RangeT(V, N) {}
}

after:
class RedBlackTree(T) {}
struct RangeT(V, N) {}

to reduce template bloat

This will do more than just reduce the name bloat, because RedBlackTree has some additional parameters. For example the following 2 ranges could theoretically use the same external range type:
RedBlackTree!int rbt1;
RedBlackTree!(int, "a > b") rbt2;

Since the comparison is only used for insertion and searching, not iteration.
I noticed this given the new possible improvement -- why does RangeT need 2 parameters? One can be constructed from the other I think.
Also, don't call it RangeT if it's outside the class, call RBRange or something like that, to avoid conflict.

I moved RangeT out of the container and renamed it as RBRange. I also made the range depending on 1 parameter.

Otherwise, you have to copy/paste the whole thing. And you can't make a tail-inout type, since inout can't be used on a struct member (and the compiler wouldn't know what to do with it if you did).

Actually I believe this is strange limitation of the curent inout that I never understood completely. To me the fact that with templates on T you invariably end up with scope-level inout variables is obvious yet it is not implemented.
It should be fairly trivial to treat inout like const in every aspect on the scope-level isn't it? I mean scope is not returned anywhere so we can even ignore the exact modifier inout there implies.

Oh and on topic - LGTM

@Groterik ping reviewers after pushing new stuff there is no "source changed" notification from GitHub.

LGTM - @schveiguy ?

Auto-merge toggled on


Port two security fixes from Envoy 1.16.1

For an explanation of how to fill out the fields, please see the relevant section
in PULL_REQUESTS.md
Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
[Optional Fixes #Issue]
[Optional Deprecated:]

I did a merge with Envoy 1.16.1: https://github.com/istio/envoy/pull/287


New admin menu

This one follows the design guidelines we defined for the admin theme in O1.
It also collapses on small devices and show a fly-out menu.
I don't want to do more things on it before we can have more actual usage and feedback

We could have the container items be clickable if the chevron was driving the expansion
It could be converted to a client side rendering and integrate dynamic elements (content tree)

In Bootstrap 4, it seems the use of FlexBox will be by default : https://github.com/twbs/bootstrap/pull/21389

For the dropdown icon, please use chevron down when collapsed and chevron up when expanded?

Add a different background for sub level and a padding to align vertically a sub link on the parent link.

Use https://github.com/jamesflorentino/nanoScrollerJS to have a thin scrollbar for the menu when it is too long.

Suggestion:
Always show a burger icon (different of the one used to display the fly out menu on small screens).
Place it on the left, before the title.
If you click on it, only the icons will be displayed, not the labels.
In that mode, display the labels over the content: on mouse hover for desktop screens and onclick for tablets.

I will create an issue with the suggestions, I want to have a discussion first about the changes.


Add MSUnmerged initStandalone && Read AllUnmerged from file

Fixes #11904
Status

Description
With the Current PR I provide an initialization script for running the MSUnmergred service standalone. It requires the service config files in order to be able to run it. Additional feature added is the ability to read all unmerged files from disk. This way we avoid uploading huge lists in the order of GBs in memory.
Is it backward compatible (if not, which system it affects?)
YES
Related PRs
None
External dependencies / deployment changes
service_configs
WMCore virtual environment:

@amaltaro  Take look at these two functions  from the current PR:

getUnmergedfromFile
and
filterUnmergedFromFile

They are basically exactly the same as the respective methods at the MSUnmerged class: https://github.com/dmwm/WMCore/blob/9a703d621b4a80b25a3989d8e53d684e2fc6b082/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py#L599
and https://github.com/dmwm/WMCore/blob/9a703d621b4a80b25a3989d8e53d684e2fc6b082/src/python/WMCore/MicroService/MSUnmerged/MSUnmerged.py#L665
The only difference is that they do all the job through io operations to disk instead of memory. When it comes to huge lists in the order of GBs this implroves the service performance and  memory consumption tremendously. I was actually surprised to see the running speed did not degrade as much as I expected. I am considering adding this functionality as optional parameters to the already existing two methods. What do you think?

With my latest commit, I finally get the direct deletions  with the os based library right. Together with the lfn deletions I now  also update the rse counters, such that  upon the RSE cleanup we will have everything tracked just as it was with gfal and we can upload to MongoDB the document for the finally cleaned RSE. This way the manual cleanup will not get into the way of properly tracking the cleanup history of the RSE itself and will not mess up with the cleanup sequence later when we the RSE is to be cleaned automatically again.
I have tested everything with 50 directories. So far so good. Tomorrow morning I'll shoot for cleaning the whole area of T2_CH_CERN
FYI @amaltaro

@amaltaro  No need for  a detailed review. Just try to grasp the idea of the two big improvements this code provides:

The ability to reduce memory foot print on the cost of disk i/o operations: https://github.com/dmwm/WMCore/pull/11916#issuecomment-1970933014
The ability to drive deletions with os libraries - for the cases like T2_CH_CERN where we have a direct mount  point  to the data: cleanRSEOs

We may consider adding these functionalities  to the main code as well. Refactoring them to become proper MSUnmerged methods is not too much work.

The initStandalnoe.py script just finished running.  It took almost 3,5 h: [1]. The RSE is clean and the status  document properly uploaded to MongoDB:

The detailed document (with the status of all directories deleted{failed|sucess} && protected): https://cmsweb.cern.ch/ms-unmerged/data/info?rse=T2_CH_CERN&detail=True

The shortened document (including only timestamps and counters): https://cmsweb.cern.ch/ms-unmerged/data/info?rse=T2_CH_CERN&detail=False

{"result": [
 {
  "wmcore_version": "2.3.1",
  "microservice_version": "2.3.1",
  "microservice": "MSManager",
  "query": "rse=T2_CH_CERN&detail=False",
  "rseData": [
    {
      "name": "T2_CH_CERN",
      "pfnPrefix": "/eos/cms",
      "rucioConMonStatus": null,
      "isClean": false,
      "timestamps": {
        "rseConsStatTime": "2023-11-02T16:06:26.189066",
        "prevStartTime": "2024-03-01T09:54:28.719355",
        "startTime": "2024-03-01T10:05:24.840614",
        "prevEndTime": "2024-03-01T09:56:10.504660",
        "endTime": "2024-03-01T13:23:18.124024"
      },
      "counters": {
        "totalNumFiles": 0,
        "totalNumDirs": 24501,
        "dirsToDelete": 23678,
        "filesToDelete": 0,
        "filesDeletedSuccess": 0,
        "filesDeletedFail": 0,
        "dirsDeletedSuccess": 23678,
        "dirsDeletedFail": 0,
        "gfalErrors": {}
      }
    }
  ]
}]}

NOTE: Please notice the mistaken field isClean: False. This  is because I just forgot to update it before exiting the cleanup cycle and uploading the document at MongoDB. I am fixing it with my latest commit. The RSE is for real clear - All successfully deleted directories  rseData->counters->dirsDeletedSuccess match the numbers in  rseData->counters->dirsToDelete
Here I also upload the full log from this cleanup process.
initStandalone-T2_CH_CERN-2024-03-01T11:05+01:00-RealRun.log
[1]
(WMCore.MSUnmergedStandalone) [user@unit01 srv]$ time ipython WMCore/src/python/WMCore/MicroService/MSUnmerged/initStandalone.py > initStandalone-`date -Im`-RealRun.log &
[1] 1397433

(WMCore.MSUnmergedStandalone) [user@unit01 srv]$
real	201m11.981s
user	2m3.947s
sys	5m6.828s

[1]+  Done                    time ipython WMCore/src/python/WMCore/MicroService/MSUnmerged/initStandalone.py > initStandalone-`date -Im`-RealRun.log

hi @amaltaro

I understand that the bulk of that implementation is in the getUnmergedfromFile and filterUnmergedFromFile methods, right?

Correct

for each file from ConMon dump, short it to a given path length and add it to a set of LFN paths (dirs.allUnmerged field)

Yes but this is not new this is how things  work even right now.

remove LFN paths from dirs.allUnmerged If they are also present in the protected list

No this is not correct. The lists in dirs.allUnmerged are never touched, but those are not too long at the first place. Once the huge file lists are  reduced  to few level up in the tree and are merged in a set (from the previous step) we usually end up with few hundreds (at most thousand) records. we do not have sites  which hold more than few  hundreds workflows and in this process  we end up with less than 10 records per workflow on average. So this is really not a big record. And not changed with this code.

then iterate over each LFN path in dirs.allUnmerged and
a) open the ConMon dump, and scan each file/line for the LFN path, yielding those lines

Again no. If you refer to this: https://github.com/dmwm/WMCore/pull/11916/files#diff-d829adc30e84e637ed37c8770f8a0f84eb357772a5f210b79c8ba0f385091dfdR353-R360
    # Now create the filters for rse['files']['toDelete'] - those should be pure generators
    # A simple generator:
    def genFunc(pattern, filePath):
        with open(filePath, 'r') as fd:
            for line in fd:
                if line.startswith(pattern):
                    yield line.rstrip()

These are simple closures which are used  as file list generators (indeed by opening and parsing the main file holding the full list of lfns at the site), but none of those are actually executed. Those are only function definitions recorded in the dictionary with toplevel directory paths. The key is the toplevel directory itself, the value is a pointer referring to the proper address in memory where those function definitions are. No file is actually opened during this process of filling the records in the dictionary. No file descriptors are created at this stage. We refer to those for executing the actual function and opening  the main file for iterating through it (in order to filter the lists of lfns), only for those deletions  for which we fail to delete the topmost path due to errors of the sort : Directory not empty. Then (and only then) we need to enter the recursive operations for purging the whole tree starting from the leafs, and only then we need to open and read the file with lfns. During the whole process of cleaning CERN it did not happen even once. But of course there are going to be sites  where it will happen. And again this is not new. This was exactly the same mechanism in the past, the only difference is that we were holding the huge list of lfns in the memory and we were iterating through those through the pointer to this list in memory. Now we do the same, but we just keep a file descriptor instead. And we sacrifice the fast access to memory for the i/o operations to disk (only for extreme cases and definitely not for all)

Which means, if we have 10k unique directories, we would open that file 10k times and scan it as a whole 10k times.

No we are not doing that. I explained the process above.

either make a feature request to the Rucio ConMon to provide a sorted list of files, or open the ConMon dump and sort files by name (well, by LFN path)

That may help. And we can use many search optimization algorithms.  But indeed starting with a sorted list is much faster.

slice the ConMon dump by X files/lines (e.g. 50k entries)
execute the MSUnmerged logic on each slice separately

This was my exact approach at the beginning. And I ran into complications of properly doing the bookeeping in the database. those were not that difficult to sort, but would need require additional code. At the end I went boldly for working with the whole (the largest seen so far) list of lfns for CERN and it did not take more than few hundreds megabytes in memory, even though the file itself was  in the order  of Gb.
https://github.com/dmwm/WMCore/pull/11916/files#diff-d829adc30e84e637ed37c8770f8a0f84eb357772a5f210b79c8ba0f385091dfdR353-R360


Evaluate embedded composer for extensions

We should evaluate embedded composer for managing extensions.
It's also used in sculpin.io and flow3 to manage extension/plugins.

Embedded composer is not developed any further. Also Sculpin stopped using embedded composer.


HDR images appear dark and dull

Disclaimer:

I don't know whether this is something pnd-jpegxl should handle.
It seems Paint.NET only supports 8 bit and have no plans for supporting HDR.

070208 "Popular Feature Requests": https://forums.getpaint.net/topic/2940-popular-feature-requests/
150501 "open exr and .hdr file support": https://forums.getpaint.net/topic/31763-open-exr-and-hdr-file-support

Opening JPEG XL HDR images in Paint.NET with pdn-jpegxl makes the images appear dark and dull compared to opening the same image with a program that seems to handle HDR better, like the Thorium web browser in my example.
Sample image

Web page containing multiple images: https://people.csail.mit.edu/ericchan/hdr/hdr-jxl.php
Image in question: https://people.csail.mit.edu/ericchan/hdr/jxl_images/20140606_102418_IMGP0297.jxl

Screenshot showing the difference

My environment

Windows 11 22621
Paint.NET v5.0.11 from Microsoft Store
pdn-jpegxl v1.0.3
Laptop with monitor that doesn't support HDR (Dell XPS 7590 with 1080p non-HDR panel)
External monitor that supports HDR10 according to manufacturer (Samsung LS27A600NAUXEN), but Windows 11 say they don't support HDR.

Vs. other programs
For comparison, so far I've found that following seems to display this image correctly:

GIMP v2.10.34
PhotoQt v3.4
Thorium Browser (thorium-win-avx2) v117.0.5938.157

While following doesn't:

ImageGlass v<IP_ADDRESS> and <IP_ADDRESS>8_RC_x64_net6
IrfanView v4.62 with plugins v4.62
JPEGView v<IP_ADDRESS>
Paint.NET v5.0.11 with pdn-jpegxl v1.0.3
XnViewMP v1.6.1

That HDR image is 16-bits-per-channel. GIMP natively supports 16 and 32 bits-per-channel images.
I am not sure what the other applications do when loading it.
I do not think that I have any control over this. The plugin tells libjxl that it wants 8-bits-per-channel output, and libjxl provides it.
How libjxl converts HDR data to 8-bits-per-channel is a question for the libjxl repository.


Remove splash/ripple effect on header when tapHeaderToExpand = true

In the ExpandableButton class, could you please replace InkWell with GestureDetector, because I don't want the ripple effect, or at least give me the option to disable the ripple effect.
Thank you in advance.

+1, i'm going to create a fork for this one.

@topex-psy  https://github.com/bantunes22/flutter-expandable

This is fixed in version 4.0.0, where useInkWell: false can be specified as a theme setting.


[cluster-test] increase chunk size of state sync in load test

Motivation
(Write your motivation for proposed changes here.)
Have you read the Contributing Guidelines on pull requests?
(Write your answer here.)
Test Plan
(Share your test plan here. If you changed code, please provide us with clear instructions for verifying that your changes work.)
Related PRs
(If this PR adds or changes functionality, please take some time to update the docs at https://github.com/diem/diem/developers.libra.org, and link to your PR here.)

/land


KAFKA-9533: Fix JavaDocs of KStream.transformValues

We might want to cherry-pick this back to 2.3.
Call for review @bbejeck

Java 8 passed.
Java 11: kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
As this is just a JavaDoc change, I'll just merge directly.

Merged to trunk and cherry-picked to 2.5, 2.4, and 2.3.


Custom events by item click

Hello, you have a nice plugin for Vue. But I want to ask about not links in your plugin. For example, in first item I want typical link, like a or router-link, but...
For example I have some component (may be from Material), and this component displays modal window by click it. May be add some opportunity for passing components to your plugin?

If you don't mind, I can make PR with example how I see this

No, sorry, I think that I will be rewrite many code for my fork because I don't want use Font Awesome because it's very overweight (~150KB for 3-4 icons, no, thanks), and for my fork I will remove iconClass and will use svg files through custom icon component.
Sorry to waste your time.

ok, if somebody need this plugin with pass components you can get it from fork https://github.com/HawkeyePierce89/vue-quick-menu


Making the number of CPU cores used for sorting postings lists configurable

This PR adds MemPostingsEnsureOrderConcurrency as an option on tsdb Options and HeadOptions. If it is not set or set to a <=0 value, then GOMAXPROCS is used, which matches the previous behaviour.
Moreover, this PR changes the signature of MemPostings.EnsureOrder() by introducing parameter numberOfConcurrentProcesses, representing the maximal number of CPU cores used for this operation. If it is less or equal to 0, GOMAXPROCS is used. GOMAXPROCS was the default before introducing this parameter.

As discussed with @codesome last week, I have opened this PR.

I think my main question is why having a new setting in tsdb.Options? To my understanding, MemPostings.EnsureOrder() is basically called when we init TSDB after the WAL replay, so couldn't we just WALReplayConcurrency for EnsureOrder() too? /cc @codesome

It is OK for me not to introduce the new flag, and to pass WALReplayConcurrency to EnsureOrder() if it conceptually makes sense. Let's hear @codesome's point of view too and I will modify my PR.

pass WALReplayConcurrency to EnsureOrder() if it conceptually makes sense.

It makes sense, because it is kind of part of WAL replay (since it is in Init()). Also, not having a new parameter is a plus.


Fix integration tests failing - pgspecial version

Describe your changes
Downgrade pgspecial version to 2.0.1
Issue number
Closes https://github.com/ploomber/jupysql/issues/347
Checklist before requesting a review

[ ] Performed a self-review of my code
[ ] Formatted my code with pkgmt format
[ ] Added tests (when necessary).
[ ] Added documentation in the docstring and changelog (when needed)

:books: Documentation preview :books:: https://jupysql--349.org.readthedocs.build/en/349/

Yeah local LGTM

Meanwhile I am checking how to update our code to let it use latest pgspecial version


6+ Vega tdxminer mining

Hello. Is there any way to run more than 6 Vegas with rocm? I spent last 48 hours trying to run more by using different kernels etc. and still no result.

We run 12 GPU on servers today.  The issue is your ussing consumer motherboard with out PLX PCIe Gen3  Swtich on x16 or x8 PCIe lane to support Polaris based GPU.   Only. Vega10 today supports abilty to use PCIe Gen2 in x1 lane if you shut off SDMA in 1.8.1 ( this is tempory for this release, future release it just work with SDMA on)  Also we working on getting Firmware addressed to support GFX 8 parts


[Warning][Django1.10]RemovedInDjango110Warning

Hi man, i test today this package and i get the next warnings:
RemovedInDjango110Warning: Support for string view arguments to url() is deprecated and will be removed in Django 1.10 (got rules_list). Pass the callable instead.
  url(r'^$', 'rules_list', name='robots_rule_list'),

/run/media/salahaddin/Data/Proyectos/Trabajo/ccct/lib/python3.5/site-packages/robots/urls.py:8: RemovedInDjango110Warning: django.conf.urls.patterns() is deprecated and will be removed in Django 1.10. Update your urlpatterns to be a list of django.conf.urls.url() instances instead.
  url(r'^$', 'rules_list', name='robots_rule_list'),

Thnks for your work, :D

Fixed in #44

We are waiting for the new version, thanks!

@SalahAdDin right :)
Care to test a prerelease? First time I release django-robots, needs some tests


Handling of bison locations appears to break common uses of this functionality

Hello,
Unless I am missing something, RE-flex's automatic handling of bison locations does not appear to work as expected and the automatic features appears to prevent common uses and a users ability to fix it.
Unexpected behavior 1: In my experience, calculated locations reference the beginning of the match -1 and the end of a match. Looking at the generated re-flex code, I see:
virtual int yylex (semantic_type* yylval, location_type* yylloc,...)
  {
    yylloc->step();
    int ret = yylex(*yylval);
    yylloc->end.line = matcher().lineno();
    yylloc->end.column = matcher().columno();
    return ret;
  }

My interpretation is that the above code assigns the begin to previous matched end which is fine, after the call to yylex, it sets the end to the current values of end to the beginning of the match not the end. That is to say, I would have expected it to be:
yylloc->end.line = matcher().lineno() + determine_if_match_has_newline(); // or similar
yylloc->end.column = matcher().columno() + size();

Unexpected behavior 2: In addition to simply calculating location data, any non-trivial use of a Lexer needs to be able to access location data within rules for decent error reporting. bison includes examples of use that it supports and recommends. For example in section <IP_ADDRESS> Calc++ Scanner of the bison documentation where numeric overflows detected in the scanner are handled and thrown back to the parser via: yy::parser::syntax_error. It does not appear to be possible to accomplish this in re-flex without inheriting from the generated Lexer and redefining yylex (semantic_type* yylval, location_type* yylloc,...) and stashing the location value for use in a rule. If the actual generated rule handling was handled in  yylex (semantic_type* yylval, location_type* yylloc,...) instead of yylex (semantic_type* yylval,...) I don't believe this would be an issue.
Unexpected behavior 3: There doesn't appear to be a straightforward way of setting other values of the location without derivation of the generated lexers and overriding the lex function. For example, setting filenames if you are using the bison-generated locations.
Most if not all of the above idiosyncrasies can be dealt with by redefining YY_USER_ACTION but my read of your documentation suggests that your would like to have automatic location tracking as a feature.
My suggestion for your consideration would be:

Have the generated rules be put into the appropriate yylex function for the options. That is, if locations are enabled, place them intoyylex (semantic_type* yylval, location_type* yylloc,...). This will allow location data to be used in rules.

Move the automatic location update functionality out of the yylex... functions and into the generated Lexer. To me, the issue with the location it is in now is that the information needed to appropriately set the end needs to come after the content is matched but before the rule action code is handled. This is why YY_USER_ACTION was a popular place to handle location updates.  The messy part of this was that the code defined in YY_USER_ACTION was duplicated for each rule. It would seem that re-flex could put this in a more appropriate place in the generated Lexer after the content was matched but before the rule actions were executed. To me, it also makes sense for the user to be able to override any automatic behavior which a virtual Lexer class method would suffice. For example from the documentation:

/**
  match is known at the call site so begin and end can be properly set. The
  user can override for any custom behavior.
*/
virtual bool LEXER::update_locations(location_type* yylloc) {...}

virtual int LEXER::yylex (semantic_type* yylval, location_type* yylloc,...)
{
    static const reflex::Pattern PATTERN_INITIAL("(?m)(REGEX)");
    if (!has_matcher())
    {
      matcher(new Matcher(PATTERN_INITIAL, stdinit(), this));
      YY_USER_INIT
    }
    CODE
    while (true)
    {
      switch (matcher().scan() && update_locations(yyloc)) // or semantically equivalent
      {
        case 0:
          if (matcher().at_end())
          {
            return 0;
          }
          else
          {
            output(matcher().input());
          }
          YY_BREAK
        case 1:
          YY_USER_ACTION
          ACTION
          YY_BREAK
      }
    }
}

Thanks for the details. Will take a look at this for the next release.

Hypothetically your approach won't work well when more() or less(n) are used in one of the switch arms (i.e. the code specified in the lexer spec), because these affect the length of the match. In that case the reported end position stored inllocp->last_column will not line up with the actual match size, because update_locations(yyloc) is called too soon before more() or less(n) are called. Perhaps a minor detail. Also, I like the idea having a choice of user-defined positions update function or a built-in update function generated by reflex (but one that is corrected as you point out). Perhaps using an #define to allow the generated update code to be replaced. I'll explore that.

I've implemented an updated bison-locations approach as suggested in v1.1.1.

Hm. I think a small change needs to be made to v1.1.1, now that I am trying more examples. The ending column reported by the new yylloc_update Lexer method is the column after the match, not the last column of the match. So I will change the assignment to yylloc.end.column = yylloc.begin.column + matcher().columns() - 1, thus with one less. Everything else checks out fine. Please take a look if you agree.

I'll check but it sounds correct.
Thanks again for doing this.


Marshes into Cascade

Thread outlining progress and issues towards bringing marshes from BarrierBMFT into Cascade

Basic Current Structure of Cascade:

Initialize a specific instance of the Cascade Class from your run file script
While making an instance of Cascade Initialize the BRIE coupler
BRIE Coupler initializes multiple instances of BRIE
Initalize_equal function makes multiple instances of Barrier3D and equalizes values between BRIE and Cascade
Use cascade.update() to update time steps of Cascade
Update multiple instances of Barrier3D
If alongshore transport is active, update AST using Cascade otherwise update cross-shore dynamics in Barrier3D
If human dynamics are active, run human modules

New revised Cascade structure (Assume no human modules are active)

Initialize a specific instance of the Cascade Class from your run file script

While making an instance of Cascade Initialize the BRIE coupler

BRIE Coupler initializes multiple instances of BRIE

Initalize_equal function makes multiple instances of Barrier3D and equalizes values between BRIE and Cascade

If marsh dynamics are turned on, initialize a back-barrier transect from PyBMFT for each B3D section

Use cascade.update() to update time steps of Cascade

Update multiple instances of Barrier3D

If alongshore transport is active, update AST using Cascade otherwise update cross-shore dynamics in Barrier3D

Update each PyBMFT transect based on changes in Barrier3D

Update back barrier elevation in Barrier3D based on changes in PyBMFT

(Unknown if needed) Update back barrier elevation in Brie based on changes in Barrier3D

If human dynamics are active, run human modules

@anardek Here's my basic outline

This looks really good to me! Just make sure to follow the same structure
as the other couplers/human dynamics modules and create a class for pyBMFT
that you then call within the main CASCADE update loop. Also, make sure to
pull any recent changes from CASCADE before you start.
On Friday, November 4, 2022, Benton Franklin @.***>
wrote:

@anardek https://github.com/anardek Here's my basic outline
—
Reply to this email directly, view it on GitHub
https://github.com/UNC-CECL/CASCADE/issues/32#issuecomment-1304066176,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ANXYL56TYL6SF4YBNLMOZJ3WGVP65ANCNFSM6AAAAAARXMZM5Q
.
You are receiving this because you were mentioned.Message ID:
@.***>

@anardek I have run a coupled version of Cascade and PyBMFT using 1 B3D grid. I'm next going to try running multiple B3D grids in Cascade. My updated branch is on Git. You'll need to have PyBMFT installed from the CECL Github to run it


Parsing of (previously saved) multi-line strings

Hi,
thanks for your very helpful piece of software.
I run into an issue with saving and parsing of multi-line strings or text.
SharpConfig is part of an Unity3D project.
The code checks if a configuration file exists. If not it is created based on the default settings in the code.
A text field from an UI element is saved correctly and looks like the following in the config file. The line-breaks are correctly placed and everything seams fine.

IntroText="Line 1
Line 2
Line 4
Line 5"

But parsing of this config file entry fails on the first line-break within the multi-line string. I get the following error:

ParserException: Line 2: setting assignment expected.

ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index

If I remove the line-breaks form the previously saved configuration file, the parsing does work again.
Apparently the parser looks for the next config file entry on the next line and ignores the quotations marks.
Is there a trick how to work with "text" or strings with line breaks?
I already tried "RawValue" but it is the same issue.
Or maybe I do something wrong?
Here is a short code snippet of the saving and loading of the text.
cfg["UI"]["IntroText"].StringValue = introText.GetComponent<TMPro.TextMeshProUGUI>().text;
introText.GetComponent<TMPro.TextMeshProUGUI>().text = cfg["UI"]["IntroText"].StringValue;
For the meantime I can use Richtext tags to have my line breaks. Still it seams odd to me, that SharpConfig creates a config file that it cannot parse afterwards.
Many thanks!
Peter

Hi @CSLVRlab-PW / Peter,
this is unfortunately a feature that's missing in SharpConfig. It can be fixed by improving the parser to correctly read quoted strings across multiple lines. The parser will soon be reimplemented entirely anyway, so that this too is possible.
My question: Do you have to have the config file in text form? If not, you may load/save it in binary form by using the appropriate overloads.

Thanks. That was an fast answer!
I was just wondering if I am doing something wrong :)
Yes. I want to use the config file to tweak the applications setup.
I will have a look on the Parser but probably I will simply store the various intro texts in separate files and then use the config file to reference them.
Thanks again!

You're welcome.
Referencing the text sounds like a good idea. By the looks of it, your config files might become too cluttered anyway once you include more text in them. Regardless, I'll reopen this issue just to have it on a TODO list for the next patch.
Cheers!

Alright. Sorry I closed it already.
Yes it might be cluttered. But this is an app for scientific purposes only. Thus it does not matter that much.


Only non-empty passphrase when creating key

We should only pass non-empty passphrases when creating a key. If we
don't avoid empty passphrases, certificates without passphrases will
throw an error when attempting to create the key.

I came across this annoyance with a development certificate. Basically, I didn't put a passphrase on the certificate since I was just using it for testing locally. Since I didn't have a passphrase, I didn't include it in my push call from CLI (and I didn't have the environment variable set). This gave me an error. I could literally give the command any passphrase and it would accept it. So we should just avoid sending a passphrase if it's empty.


[Bug] accept tcp host:port use of closed network connection on server shutdown

Describe the bug
I'm using Hertz with the standard transport like this:

https://github.com/EinStack/glide/blob/develop/pkg/api/http/config.go#L43-L69

On shutdown though, Hertz constantly returns me this error:
2024-02-11T12:41:25.093+0200    ERROR   HERTZ: Error=accept tcp <IP_ADDRESS>:9099: use of closed network connection
2024-02-11T12:41:25.093+0200    ERROR   api/servers.go:46       error on running HTTP server    {"error": "accept tcp <IP_ADDRESS>:9099: use of closed network connection"}
glide/pkg/api.(*ServerManager).Start.func1
        /Users/roman/Projects/ideas/Glide/pkg/api/servers.go:46
^C^C2024-02-11T12:41:30.093+0200        INFO    HERTZ: Execute OnShutdownHooks finish
make: *** [run] Error 1

Does anyone know what I'm doing wrong here? 🤔
Expected behavior
I expect Hertz server to shutdown without errors.
Screenshots
If applicable, add screenshots to help explain your problem.
Hertz version:
v0.7.3
Environment:
GO111MODULE='on'
GOARCH='amd64'
GOBIN=''
GOCACHE='/Users/usr/Library/Caches/go-build'
GOENV='/Users/usr/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/usr/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/usr/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/Cellar/go/1.21.5/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/Cellar/go/1.21.5/libexec/pkg/tool/darwin_amd64'
GOVCS=''
GOVERSION='go1.21.5'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/Users/usr/Project/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/s1/ts3nxvl965lfts126qzdcw300000gn/T/go-build662925335=/tmp/go-build -gno-record-gcc-switches -fno-common'

I have a feeling it's similar to this issue.

@li-jin-gou hey, you seem to reference this issue probably by mistake 😌  Could please review that link?

https://github.com/cloudwego/hertz/issues/878

The fastest solution is to use netpoll transport.

@li-jin-gou  thank you for the reference! I will take a look at the chat history.
I was using netpoll, but the thing is that Glide needs to support:

windows
uploading of voice stream which looks like a file uploading (still not sure what the final implementation will look like)

These two cases don't look like the ideal for netpoll usage right?

@li-jin-gou thank you for the reference! I will take a look at the chat history.
I was using netpoll, but the thing is that Glide needs to support:

windows
uploading of voice stream which looks like a file uploading (still not sure what the final implementation will look like)

These two cases don't look like the ideal for netpoll usage right?

Yes

@li-jin-gou if we batched the voice streams and send them as websocket messages, would this be still not the ideal case for netpoll?

If the QPS is not high, netpoll is fine. refer to https://www.cloudwego.io/docs/hertz/tutorials/basic-feature/network-lib/#choosing-appropriate-network-library

@li-jin-gou I actually have seen that one. But the doc is more focused on the request size rather than throughput. Is the assumption here that with somewhat low throughput it's unlikely that the request size would cause any memory issues?

I actually have seen that one. But the doc is more focused on the request size rather than throughput. Is the assumption here that with somewhat low throughput it's unlikely that the request size would cause any memory issues?

Yes

Refer to https://github.com/cloudwego/hertz/issues/878 and duplicate issue, I'll close it for now.

This error is returned if a accepting listener is closed. It is a normal returning process, and it is safe to ignore.
Try this code below to check:
func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/ping", func(writer http.ResponseWriter, request *http.Request) {
		writer.WriteHeader(200)
		writer.Write([]byte("pong!"))
	})
	server := &http.Server{Addr: "<IP_ADDRESS>:8888", Handler: mux}

	go func() {
		time.Sleep(5 * time.Second)
		server.Close()
	}()

	err := server.ListenAndServe()
	if err != nil {
		fmt.Println(err.Error())
	}
}

I think it is an optimized point that we ignore this Server closed error when closing the server.
I will put it into the Good first issues pool. Please feel free to open a pr to do so if you want to.

@welkeyever
I would like to try to optimize this problem,
plz assign to me


Add configurable include_filename? setting to allow s to include the test file path.

Hello! I'm submitting this PR because I ran into an issue on CircleCI where their split-by=timings option required the ingested JUnit XML to include a filename. I have tested the generated XML on my forked branch, and it does fix the CircleCI timings problem for Elixir.
However, I didn't want to force it to be true by default here because I found a few issues where including the file attribute caused issues with the XML parser.
Let me know if you find any issues with the code.

Coverage increased (+0.2%) to 98.039% when pulling 0cf2f4e19a74d57fbc1dfdfab40981d9070ef27d on danadaldos:master into dd24a87e2d795ccb60250ec7d1b7d8116c271e5b on victorolinasc:master.

This is the old case pictured perfectly by xkcd... https://xkcd.com/927/
I am okay with the approach of this not being default. What worries me a bit (but not much) is the extensive configuration we are having in this project =/
But I am fine with the implementation. Would like to test it a bit more on umbrella projects though. I'll try to find some time this week to do so.
Thanks a lot for your contribution! I'll merge/comment as soon as I can test it locally.

This is the old case pictured perfectly by xkcd... https://xkcd.com/927/

😀

I'll try to find some time this week to do so.

👍

Thanks a lot for your contribution! I'll merge/comment as soon as I can test it locally.

Thank you! I look forward to any feedback you have.

Everything seems fine :) I will merge and release now!

@victorolinasc
Thanks for merging it, this will save some future CircleCI users some frustration.
🎉 Woo! My first contribution to an open source project! 🎉

Published as 3.1.0.
Thanks for your contribution.


Selection handlers should take precedence over page navigation arrows

Bug Description
Currently, when selection handlers are at the same location as the page navigation and the handler is clicked, the navigation action is triggered.

Expected Behaviour
Selection handlers should take precedence over page navigation arrows

Steps to Reproduce

Add an element
Resize it so that its resize handlers are in the same place as the page navigation
Try resizing now
See the page navigating instead

Screenshots

Additional Context
Follow-up to https://github.com/google/web-stories-wp/issues/2463, see also https://github.com/google/web-stories-wp/issues/2463#issuecomment-651275296

Plugin Version:
Operating System:
Browser:

Do not alter or remove anything below. The following sections will be managed by moderators only.
Acceptance Criteria

Implementation Brief

@littlemilkstudio Saw that you moved this issue to In Progress and then unassigned -- should this go back to Ready for Sprint (as it's for 1.2) or is it still In Progress?

@barklund This is currently set for 1.2.0, should it be 1.1.0 instead (since it's in the Sprint backlog)?


Support Kafka-clients 3.9.0

KAFKA-17060 broke the retrieval of autocommit enabled in
AbstractParallelEoSStreamProcessor#getAutoCommitEnabled.
Relates to #840
Checklist

[ ] Documentation (if applicable)
[x] Changelog

It might be worth upgrading the Docker image used for integration tests to the latest one (or run tests against multiple Kafka versions), especially given now an image is provided by Apache Kafka. Not sure if it should be done in this change though.

kafka-clients version < 3.8.0 have a known security problem CVE-2024-31141  need fix, so this pr is really useful.

/sem-approve


Feature add:  Display Transaction statistics

While running Movr it would be good to show the transaction times
instead of
[INFO] (MainThread) connected to movr database @ postgres://root@drew-geo-partitioning-0012:26257/movr?sslmode=disable
[INFO] (MainThread) simulating movr load for cities ['chicago']
[INFO] (MainThread) warming up....

show something like the following output put from workload or YCSB
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
      1s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
      1s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
      1s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
      1s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
      1s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
      2s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
      2s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
      2s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
      2s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
      2s        0            0.0            0.0      0.0      0.0      0.0      0.0 stockLevel
      3s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
      3s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
      3s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
      3s        0            0.0            0.0      0.0      0.0      0.0      0.0 payment
      3s        0            1.0            0.3     92.3     92.3     92.3     92.3 stockLevel
      4s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
      4s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
      4s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
      4s        0            3.0            0.7     35.7     58.7     58.7     58.7 payment
      4s        0            0.0            0.2      0.0      0.0      0.0      0.0 stockLevel
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
      5s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
      5s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
      5s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
      5s        0            0.0            0.6      0.0      0.0      0.0      0.0 payment
      5s        0            0.0            0.2      0.0      0.0      0.0      0.0 stockLevel
      6s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
      6s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
      6s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
      6s        0            0.0            0.5      0.0      0.0      0.0      0.0 payment
      6s        0            0.0            0.2      0.0      0.0      0.0      0.0 stockLevel
      7s        0            0.0            0.0      0.0      0.0      0.0      0.0 delivery
      7s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
      7s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus
      7s        0            0.0            0.4      0.0      0.0      0.0      0.0 payment
      7s        0            0.0            0.1      0.0      0.0      0.0      0.0 stockLevel
      8s        0            1.0            0.1     46.1     46.1     46.1     46.1 delivery
      8s        0            0.0            0.0      0.0      0.0      0.0      0.0 newOrder
      8s        0            0.0            0.0      0.0      0.0      0.0      0.0 orderStatus

@nstewart, for context, this suggesiton came out of a discussion about my geo-partitioning instructions: https://cockroachlabs.atlassian.net/wiki/spaces/ENG/pages/109314241/Geo-Partitioning+a+Multi-Region+Cluster. We discussed why we're not using the Statements page (transaction pipelining). Drew agreed that showing full replicaton latency in the UI would be helpful (e.g., via a transactions page), but he mentioned that customers really want this to include the app latency as well.

Awesome. Will try to do this ahead of our 19.1 demo.


Draft - Arbitration

Background

User Story
As a panel chair,  I would like to see applications when I am on the score sheet page so that I can see where original panel does not agree with a moderating panel.
As a panel chair on the score sheet page, I would like to allow SETs to enter the outcome: final grade and note so that I can view all necessary data.
Benefit(s) to user (if not already clear from User story)
On the sift score page, the panel chair can view live data.
Feature(s) Description
Application, overview, score sheet - see screen shot above.
Acceptance Criteria
It is done when a panel chair is on the score sheet page and able to see the applications related to the original panel and moderating panel.
It is done when the panel chair is able to enter the final grade and note for each SET on the score sheet page.
Feature(s) Description
see screen shot
##User testing steps
Enter grade and note for each SET.
Feature Champion
Devs

Hey team! Please add your planning poker estimate with Zenhub @drieJAC @HalcyonJAC @tomlovesgithub @warrensearle

The image isnt displaying @NormaJAC @nickaddy
Is it possible to re-upload it please?

@drieJAC  Does this help?


Adjust how slang and slangc uses a profile to manage the stage of an entry-point

Fixes #4656
Changes:

If the stage specified by a profile for a target is not equal to the stage specified by a entry-point, we throw a capability error.
Resolving the stage of an entry point was changed to function (mostly) equally for when 0 entry-points are specified versus to when there are 1 or more. This makes erroring locations consistent across the 2 methods to find entry-points.
Changed CapabilitySet Iterator so it is invalid if backing data is nullptr to avoid a potential crash. This should never happen unless validation fails, [[unlikely]] would be a nice-to-have for this change.

It could be better if you were able to split your changes into smaller PRs.

It could be better if you were able to split your changes into smaller PRs.

This PR solves the core problem of https://github.com/shader-slang/slang/issues/4656.
I believe I could have have 2 PRs, but then PR 1 would depend on PR 2.
PR1 would have been the changes to source/slang/slang-capability.h (1 line of code)

Errors would have been wrong, #4656 would still not be solved, but we would no longer crash.

PR2 would add diagnostics and fix how profiles are assigned to entry-points (all are required together, else the diagnostics added would have still been wrong)

I can still break the code into the 2 PRs if preferred for some reason in particular.

I am not suggesting to break down the current PR because it will be a non trivial extra work. But please try to make your changes smaller next time which I have been telling multiple times already.
Incomplete implementation is not a big problem as long as it does cause a regression. The project is always incomplete anyway; from a bigger picture. The concept of complete and incomplete is how you define the problem domain.
Although my suggestion to make the PR smaller comes from the inconvenience for me as a reviewer, it has other benefits as well when the PR is limited to smaller size.
From a long term maintenance cost perspective, the readability is very important and when a commit is big, the readability goes down. Consider also that writing a code happens only once and reading and maintaining the code happens by the number of people who read the change throughout multiple years. Spending some time rewriting the code for better readability and maintainability in longer term will pay off eventually.


Improve intrinsic sizing , dryLayout and performLayout in the RenderFlex if crossAxisAlignment is baseline.

This PR fixes #95400 and #95398.
Pre-launch Checklist

[x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
[x] I read the [Tree Hygiene] wiki page, which explains my responsibilities.
[x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement].
[x] I signed the [CLA].
[x] I listed at least one issue that this PR fixes in the description above.
[ ] I updated/added relevant documentation (doc comments with ///).
[ ] I added new tests to check the change I am making, or this PR is [test-exempt].
[ ] All existing and new tests are passing.

@mohammadtaherri Can you please add a test to this to ensure that we never break this again in the future? Thanks!

Closing this since there hasn't been any follow-up. We do have issues on file for the problems.


cleanup: change #!/bin/sh to #!/bin/bash

Several scripts already required /bin/bash, so we shouldn't lose
compatibility.
This allows future improvements to our scripts, e.g.:

"[[" instead of "["
"source" instead of "."
"set -o nounset" etc.

Is there a reasonable guarantee that bash is installed?  It is common in many desktop distributions, but there is certainly no certainty that it is expected even in linux distributions.
It is not in the LSB.
https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Common/LSB-Common/book1.html

Is there a reasonable guarantee that bash is installed? It is common in many desktop distributions, but there is certainly no certainty that it is expected even in linux distributions.
It is not in the LSB.
https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Common/LSB-Common/book1.html

Various scripts already require /bin/bash to build and test Charm++, so this doesn't add an additional dependency. E.g.,

src/util/pup_f.f90.sh
src/scripts/testrun
src/scripts/commitid.sh
ssh script generated by src/util/charmrun-src/charmrun.C
src/arch/win/gather*.local

Bumping this again. Imho, we can rely on the presence of /bin/bash on any system. Even though a user might have a different shell, /bin/bash only needs to be present, not the user's shell. Debian, somewhat famously, tried and failed to make bash it removable (https://wiki.debian.org/Proposals/RemoveBashFromEssential). macOS has it. Windows WSL and cygwin have it. As mentioned above, any build of Charm++ requires /bin/bash currently, even without this patch.
Putting a requirement on /bin/bash avoids hard-to-find bugs such as #2506,  f4d526293c11950081120b18d6f419c04b502885 or d4bed4db01124bc879235139a86e34d9a7531fd7, apart from the other potential improvements mentioned above.

If we switch to bash and replace all [ and test with [[, I would expect build times to improve on Windows, where process launch is expensive.

I think we can merge this after 6.10 and see if anyone complains about the dependence

I think we can merge this after 6.10 and see if anyone complains about the dependence

Agreed. The idea is to merge this to the master branch, which won't go into 6.10 anymore.

Needs another review.


"Error fetching event data" when clicking "RSVP" button in gmail

Describe the bug
For the last few days anytime I try to RSVP an event invitation from a Gmail service, clicking on the "RSVP" button in the invitation will result in a popup showing "Error fetching event data".
Similarly, if I click into an invitation, the top bar that usually shows information about the event is missing.
This happens for two different gmail accounts I have and works fine when opening gmail via browser.
I'm on OSX 11.4 and have latest Franz version (5.7.0) and latest Google Chrome.
I tried clearing cache, restarting Franz, restarting Google Chrome.


How to post an Image to server in ESP32S2? (IDFGH-6685)

Hello,
I'm using ESP32S2 Dev module for my project, ESP32S2 is connected to OV5640 camera sensor, so what I want to do is capture an Image  from camera sensor and POST it to webserver. I'm able to capture the image, but I need to know how to POST image to web server?

https://github.com/nopnop2002/esp-idf-http-camera

@subhash-coder
Please check out https://github.com/espressif/esp-idf/tree/master/examples/protocols/http_server/file_serving#simple-http-file-server-example for this. This example has POST method usage for uploading files to SPIFFS partition on device.
Please post how-to questions on https://esp32.com forum.


Potential dereference of null pointer

Select your issue type: (check at least one)

[x] Bug
[ ] Question
[ ] Suggestion
[ ] Other (please describe):

Describe your issue:
Potential dereference of nullptr.
Please see clang report in my gitlab-ci (link may expire)
That's why I don't like macros, because it makes things hard to read
Ways to Reproduce the issue (optional):

Do you need that trylock all the time? if it failed to lock, the corresponding value would not be set. This is not what I expect as a user of the library.

could you confirm this now ?

Ok, I pulled your update to my git submodule and I don't see any complains from clang now.


Reconnect spawns new connection in infinite loop

Hi,
this issue is the follow-up to a chat on your website today (I am ipool_dev@....de).
The problem was that a reconnect in the client (e.g. because of maintenance of your services) increases the number of connections to the server until the connection limit is reached.
The problem is even more serious than that because the client spawn new websocket connect threads in an infinite loop. This results in an OutOfMemoryError after some time and the application becomes unusable.
The problem is easy to reproduce: create a client, connect (I guess it's important to activate the heartbeat) and let the application sleep. Remove the network connectivity and after some time the client starts to reconnect. Activate the network again -> now we have the infinite loop (and the callbacks are called periodically).
Best regards
Lars

Thanks for the heads-up @lbeuster, we'll be looking over this asap.

Thanks for the fix. I tested my use case too and it worked now ;)


No syntax highlighting for Unity

This may have to be implemented in the future, once you guys have further unity support, but basically no matter which theme I use, Unity specific variables / types / classes do not have syntax highlighting.

VSCode Version: 1.2.1
OS Version: OSX 10.11.5

Steps to Reproduce:

Open unity project with VS Code
Use some unity specific variables / types / classes (e.g Camera, Vector3, etc..)
Observe no syntax highlighting

This isn't limited to Unity, there is no special syntax highlighting for e.g. the System namespace either, unless I'm suffering from a misconfiguration of course.
Pretty sure that it is intended this way though, since intellisense and such work fine for Unity or otherwise here, just the highlighting is limited to primitive types, keywords, and depending on the theme method return types and such.

Assuming this is about C#, please report this to https://github.com/OmniSharp/omnisharp-vscode


feat: Adding DAAC archive config

Closes #376

Missing "dataVersion": "123"  # TODO ask user to provide it in config,

Missing: User can optionally add "type" + "file extension" to choose selective files to be archived to DAAC.

Added missing terms