New to PowerShell - Syntax Question on PowerShell Command

Since I am new to writing code and using Windows PowerShell scripts, I apologize up front if I offend anyone with the newbie "dumb" PowerShell question (please be patient with me!).
I am having a lot of issues with understanding the correct syntax for PowerShell commands, cmdlets, and parameters.
As one of my first test cases, I am trying to use the Microsoft tool to build a SBOM for Serilog (https://github.com/serilog/serilog).  I cloned both this repo and the serilog repo.
In the SBOM tool readme (https://github.com/microsoft/sbom-tool/blob/main/README.md), I going down the path of using PowerShell in order to generate the SBOM (obviously in SPDX format).
I first ran:
Invoke-WebRequest -Uri "https://github.com/microsoft/sbom-tool/releases/latest/download/sbom-tool-win-x64.exe" -OutFile "sbom-tool.exe"
I got as far as:
Run the tool
SBOM Generation
Once you have installed the command line tool for your OS, run the tool using this command:
sbom-tool generate -b  -bc  -pn  -pv  -ps  -nsb
As far as I can tell, I think that this command should generate the SBOM.  Instead PowerShell gives me this error:
##[error]Encountered an error while generating the manifest.
##[error]Error details: Could not find file "<*****>_manifest\spdx_2.2"
where <*****> represents the full directory path, which I am obscuring for privacy reasons.
Why am I seeing this error message?  What step and/or syntax am I missing?  Am I overlooking the applicable information at https://learn.microsoft.com/en-us/powershell/?view=powershell-5.1?  I am so new to PowerShell that I am sure that I am overlooking a simple answer.

@Charles1992Sail can you try running the tool from the command line to see if you get the same error? Can you provide the parameters that you are using to call the tool?

@Charles1992Sail looks like you are missing some required parameters: pn and pv. Also, I'm curious, do you get the same error if you don't specify the -m parameter?

Yes, I am getting the same error message regardless if I use or if I leave out the -m, -pn, and -pv parameters.
For -b and -m, I even tried the full folder paths with:
-- forward slashes
-- back slashes
-- enclosed in double quotes
-- not enclosed in any quotes
.\sbom-tool generate -b "C:------\Documents\GitHub\serilog" -m "C:------\Documents\SBOM\Reports" -pn "Serilog SBOM" -pv "03-17-2023v1" -ps "Serilog Contributors" -nsb "https://github.com/serilog/serilog"

@Charles1992Sail  please execute the command with -V Verbose and share the complete log so we can investigate further

With the dashes representing parts of the full directory path that I redacted, I ran the command:
<----------------------------------------->
.\sbom-tool generate -b "C:/---/Documents/GitHub/serilog" -m "C:/---/Documents/SBOM/Reports" -pn "Serilog SBOM" -pv "03-17-2023v1" -ps "Serilog Contributors" -nsb "https://github.com/serilog/serilog" -V Verbose
<----------------------------------------->
The resulting log was:
<----------------------------------------->
.\sbom-tool generate -b "C:/---/Documents/GitHub/serilog" -m "C:/---/Documents/SBOM/Reports" -pn "Serilog SBOM" -pv "03-17-2023v1" -ps "Serilog Contributors" -nsb "https://github.com/serilog/serilog" -V verbose
##[debug]Starting SBOM generation workflow.
##[error]Encountered an error while generating the manifest.
##[error]Error details: Could not find file 'C:---\Documents\SBOM\Reports_manifest\spdx_2.2'.
##[information]Finished execution of the Generate workflow SBOMTelemetry {Result=Failure, Errors=ErrorContainer1 {Count=0, Errors=[]}, Parameters=Configuration {BuildDropPath=ConfigurationSetting1 {Value="C:/-/-/Documents/GitHub/serilog", Source=CommandLine, IsDefaultSource=False}, BuildComponentPath=null, BuildListFile=null, ManifestPath=null, ManifestDirPath=ConfigurationSetting1 {Value="C:/U-/-/Documents/SBOM/Reports\\_manifest", Source=CommandLine, IsDefaultSource=False}, OutputPath=null, Parallelism=ConfigurationSetting1 {Value=8, Source=Default, IsDefaultSource=True}, Verbosity=ConfigurationSetting1 {Value=Verbose, Source=CommandLine, IsDefaultSource=False}, ConfigFilePath=null, ManifestInfo=ConfigurationSetting1 {Value=null, Source=Default, IsDefaultSource=True}, HashAlgorithm=ConfigurationSetting1 {Value=AlgorithmName {Name="SHA256"}, Source=Default, IsDefaultSource=True}, RootPathFilter=null, CatalogFilePath=null, ValidateSignature=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, IgnoreMissing=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, ManifestToolAction=Generate, PackageName=ConfigurationSetting1 {Value="Serilog SBOM", Source=CommandLine, IsDefaultSource=False}, PackageVersion=ConfigurationSetting1 {Value="03-17-2023v1", Source=CommandLine, IsDefaultSource=False}, PackageSupplier=ConfigurationSetting1 {Value="Serilog Contributors", Source=CommandLine, IsDefaultSource=False}, FilesList=null, PackagesList=null, TelemetryFilePath=null, DockerImagesToScan=null, ExternalDocumentReferenceListFile=null, AdditionalComponentDetectorArgs=null, NamespaceUriUniquePart=null, NamespaceUriBase=ConfigurationSetting1 {Value="https://github.com/serilog/serilog", Source=CommandLine, IsDefaultSource=False}, GenerationTimestamp=null, FollowSymlinks=ConfigurationSetting1 {Value=True, Source=Default, IsDefaultSource=True}, DeleteManifestDirIfPresent=ConfigurationSetting`1 {Value=False, Source=Default, IsDefaultSource=True}}, SBOMFormatsUsed=[], Timings=[Timing {EventName="Total generation time", TimeSpan="00:00:00.0333174"}], Switches={}, Exceptions={["System.IO.FileNotFoundException"]="Could not find file 'C:\-\-\-\SBOM\Reports\_manifest\spdx_2.2'."}}
<----------------------------------------->

Thank you @Charles1992Sail , can you please help me confirm which version of the tool you are running? You can call;
sbom-tool Version

sbom-tool Version
0.3.3

Can you try running the following? Adding a forward slash to the folder path in the -m parameter value:
sbom-tool generate -b "C:/---/Documents/GitHub/serilog" -m "C:/---/Documents/SBOM/Reports**/**" -pn "Serilog SBOM" -pv "03-17-2023v1" -ps "Serilog Contributors" -nsb "https://github.com/serilog/serilog" -V verbose

@edgarrs thank you for getting back to me and for your patience!  My coding skills are very rough - I have a lot of learning yet to do for sure.
I updated the tool to version 1.0.2 and added the forward slash as you mentioned.  That being said, I am still encountering the same error message, i.e., the "_manifest\spdx_2.2" file is missing.
I apologize up front for my coding skills being so rough.  Hopefully, the solution to my issue is just a matter of me overlooking something very elemental on my end.  This effort of using the tool is forcing me to go into uncharted territory of slowly expanding my coding skills - I feel bit like a vision-challenged person trying to "see" an elephant!
Let me know if I need to provide you anything in addition to the log file below.
.\sbom-tool --version
1.0.2
sbom-tool generate -b "C:/Users//Documents/GitHub/serilog" -m "C:/Users//Documents/SBOM/Reports/" -pn "Serilog SBOM" -pv "03-17-2023v1" -ps "Serilog Contributors" -nsb "https://github.com/serilog/serilog" -V verbose
##[debug]Starting SBOM generation workflow.
##[error]Encountered an error while generating the manifest.
##[error]Error details: Could not find file 'C:\Users*\Documents\SBOM\Reports_manifest\spdx_2.2'.
##[information]Finished execution of the Generate workflow SBOMTelemetry {Result=Failure, Errors=ErrorContainer1 {Count=0, Errors=[]}, Parameters=Configuration {BuildDropPath=ConfigurationSetting1 {Value="C:/Users//Documents/GitHub/serilog", Source=CommandLine, IsDefaultSource=False}, BuildComponentPath=null, BuildListFile=null, ManifestPath=null, ManifestDirPath=ConfigurationSetting1 {Value="C:/Users/***/Documents/SBOM/Reports/_manifest", Source=CommandLine, IsDefaultSource=False}, OutputPath=null, Parallelism=ConfigurationSetting1 {Value=8, Source=Default, IsDefaultSource=True}, Verbosity=ConfigurationSetting1 {Value=Verbose, Source=CommandLine, IsDefaultSource=False}, ConfigFilePath=null, ManifestInfo=ConfigurationSetting1 {Value=null, Source=Default, IsDefaultSource=True}, HashAlgorithm=ConfigurationSetting1 {Value=AlgorithmName {Name="SHA256"}, Source=Default, IsDefaultSource=True}, RootPathFilter=null, CatalogFilePath=null, ValidateSignature=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, IgnoreMissing=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, ManifestToolAction=Generate, PackageName=ConfigurationSetting1 {Value="Serilog SBOM", Source=CommandLine, IsDefaultSource=False}, PackageVersion=ConfigurationSetting1 {Value="03-17-2023v1", Source=CommandLine, IsDefaultSource=False}, PackageSupplier=ConfigurationSetting1 {Value="Serilog Contributors", Source=CommandLine, IsDefaultSource=False}, FilesList=null, PackagesList=null, TelemetryFilePath=null, DockerImagesToScan=null, ExternalDocumentReferenceListFile=null, AdditionalComponentDetectorArgs=null, NamespaceUriUniquePart=null, NamespaceUriBase=ConfigurationSetting1 {Value="https://github.com/serilog/serilog", Source=CommandLine, IsDefaultSource=False}, GenerationTimestamp=null, FollowSymlinks=ConfigurationSetting1 {Value=True, Source=Default, IsDefaultSource=True}, DeleteManifestDirIfPresent=ConfigurationSetting`1 {Value=False, Source=Default, IsDefaultSource=True}}, SBOMFormatsUsed=[], Timings=[Timing {EventName="Total generation time", TimeSpan="00:00:00.0215677"}], Switches={}, Exceptions={["System.IO.FileNotFoundException"]="Could not find file 'C:\Users\**\Documents\SBOM\Reports\_manifest\spdx_2.2'."}}

I downloaded sbom-tool v1.1.1 (released at 17:58 UTC on 15 May 2023), however I am still having issues.
Thank you all in advance for your patience.  My PowerShell skills are very rudimentary.  I am sure I am overlooking some very elementary and basic steps which are second-nature and automatic for the more experienced programmers here on GitHub.
BACKGROUND:  By default, I saved the "sbom-tool-win-x64.exe" file to my local user download folder.  I am on a Windows machine with a fresh installation of Windows 11 running PowerShell v5.1.22621.963 and PowerShell v7.3.4.
ISSUE:  When I double-click the ."sbom-tool-win-x64.exe" file, the CMD prompt open very briefly (less than one second) then automatically closes.  (I can't figure out how to get CMD to stop closing automatically.)
I was able to run my smartphone camera in slow mode to capture the error message: "No action was specified".
I also encountered this issue when I downloaded "sbom-tool-win-x64.exe" file for version 1.1.0 (released at 22:05 UTC on 08 May 2023).

How do I get the newly downloaded .exe file to run so that I can migrate to sbom-tool v1.1.1?  I feel like I am so close yet so far from my goal of getting sbom-tool to work.
I somehow previously overlooked the "sbom-tool-cli-reference.md" file, which addresses my issue with sbom-tool v0.3.3:
<------------------------------------>
Here, you configured the tool to generte an SBOM for all the files in the c:\outputDrop folder, the c:\Users\test\TestProject path will be searched for *.csproj or packages.config files to get a list of dependency packages that will be added to the SBOM. The package name and version were configured using the -pn and -pv parameter, and the -nsb parameter specifices the namespace base uri which will be used to generate the document namespace in the final SPDX 2.2 SBOM. If no -nsb is provided then a default namespace base uri that complies with SPDX2.2 specifications will be provided.
The generated SBOM will be placed in the path specified by the -b argument inside a _manifest\spdx_2.2\ folder. So, in our case the SBOM will be located here: c:\outputDrop_manifest\spdx_2.2\manifest.spdx.json
Place the generated SBOM in a seperate folder
By default the generated SBOM will be placed in a newly created folder called _manifest inside the BuildDropPath (-b), but in case you want to place the SBOM in a different path, you can specify the ManifestDirPath -m parameter like so.
Validating an SBOM
Now that you have generated an SBOM you can use the tool to validate the SBOM. To do so you can run
./sbom-tool-win-x64.exe validate -b c:\outputDrop -o c:\validationOutputPath\output.json -mi SPDX:2.2
This is the minimal setup required to validate an SBOM where -b should be the path same path you provided when generating the SBOM. In this scenario the tool will default to searching for an SBOM at the c:\outputDrop_manifest\spdx_2.2\manifest.spdx.json path. -o is the output path where the tool will write the validation results to.
This can be any file path on the system, in this case the tool will look for the validationOutputPath directory and create a file named output.json and write the validation output. -mi is the ManifestInfo, A list of the name and version of the manifest format that we are using.
Currently only SPDX2.2 is supported.
Common scenarios where you can provide additional parameters
SBOM was placed in a different folder
If at generation the SBOM was created using the following parameters:
./sbom-tool-win-x64.exe generate -b c:\outputDrop -bc c:\Users\test\TestProject -pn TestProject -pv 1.0.0 -ps MyCompany -nsb http://mycompany.com -m c:\sboms
Then the sbom will not be found at the default location. In order to allow the tool to validate the SBOM at the different location you must provide the path to the _manifest that was created in that directory:
./sbom-tool-win-x64.exe validate -b c:\outputDrop -o c:\validationOutputPath\output.json -mi SPDX:2.2 -m c:\sboms_manifest
https://github.com/microsoft/sbom-tool/blob/main/docs/sbom-tool-cli-reference.md
<------------------------------------>

@Charles1992Sail, would it be possible for you to generate exact step by step instructions? We know you're editing your strings and responses for privacy but it's making it a bit hard to determine what's going on, if you can find a scenario which doesn't include those strings anyway that would be super helpful.

Thank you for engaging my issue!
I can't seem to get past the issue with the tool having issues with generating the manifest.

I am working in the PowerShell CLI with $PSVersionTable:
<---------------------------------------------------->
<---------------------------------------------------->
Name                           Value

PSVersion                      7.3.6
PSEdition                      Core
GitCommitId                    7.3.6
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0
<---------------------------------------------------->
<---------------------------------------------------->

I installed the current ./sbom-tool-win-x64.exe file (version 1.2.0 which was released on Thursday, 27 July 2023 at 21:30 GMT):
<---------------------------------------------------->
<---------------------------------------------------->
./sbom-tool-win-x64.exe --version
1.2.0
<---------------------------------------------------->
<---------------------------------------------------->

I executed this command in order to generate the manifest:
<---------------------------------------------------->
<---------------------------------------------------->
./sbom-tool-win-x64.exe generate -b "C:\Users-----------\Documents\GitHub\serilog" -m "C:\Users-----------\Documents\SBOM\Reports\Serilog" -ps "Serilog" -nsb "https://github.com/microsoft/sbom-tool"
<---------------------------------------------------->
<---------------------------------------------------->

I am consistently encountering this error when I run this and other previous versions of the sbom-tool-win-x64.exe:
<---------------------------------------------------->
<---------------------------------------------------->
##[error]Encountered an error while generating the manifest.
##[error]Error details: Could not find file 'C:\Users-----------\Documents\SBOM\Reports\Serilog_manifest\spdx_2.2'.
<---------------------------------------------------->
<---------------------------------------------------->

I even tried executing the command with just the -m parameter to no avail:
<---------------------------------------------------->
<---------------------------------------------------->
./sbom-tool-win-x64.exe generate -b "C:\Users-----------\Documents\GitHub\serilog" -m -ps -nsb
ManifestDirPath directory not found for '-ps'
<---------------------------------------------------->
<---------------------------------------------------->

I am all ears if you need more information from me or if there are steps that I overlooked executing.
Thanks!!!!

Just in case anyone is wondering, I accidently closed the issue when I meant to delete some text.

I used the steps mentioned previously in order to verify I am on the latest version (1.3.2 released 04 August 2023).  I am running the sbom tool in the CLI for PS Version 7.3.6.
I also tried the sbom tool in Visual Studio Code with the PowerShell extension enabled:
Version: 1.81.0 (user setup)
Commit: 6445d93c81ebe42c4cbd7a60712e0b17d9463e97
Date: 2023-08-02T12:37:13.485Z
Electron: 22.3.18
ElectronBuildId: 22689846
Chromium: 108.0.5359.215
Node.js: 16.17.1
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621
I am still unsuccessful with generating the SBOM.  I am still getting the same errors listed above.  I even went as far as building a "_manifest" subfolder in the destination path populated with blank "spdx_2.2.txt", "spdx.2.2.json", " and "spdx_2.2.spdx" files.
I ran this command:
./sbom-tool-win-x64.exe generate -b "C:\Users_\Documents\GitHub\serilog" -m "C:\Users_\Documents\SBOM\Reports\Serilog_manifest" -D true -ps "Serilog" -nsb "https://github.com/microsoft/sbom-tool"
The sbom tool won't progress past the manifest issues.
Error messages include:

##[error]Encountered an error while generating the manifest.
##[error]Error details: Unable to create manifest directory at path C:\Users_\Documents\GitHub\serilog_manifest. Error: Access to the path 'spdx_2.2.spdx' is denied.
##[warning]Manifest generation failed, however we were unable to delete the partially generated manifest.json file and the C:\Users__\Documents\GitHub\serilog_manifest directory.
##[error]Encountered an error while generating the manifest.
##[error]Error details: Could not find file 'C:\Users___\Documents\SBOM\Reports\Serilog_manifest_manifest\spdx_2.2'.

I tried generating a SBOM with version 1.5.0, which was released on Thursday, 17 August 2023.
I am running PS Version 7.3.6 as an administrator both in the CLI and in VS Code:
<------------------------------------->
<------------------------------------->
$PSVersionTable

Name                                          Value
  ----                                             -----
PSVersion                                    7.3.6
PSEdition                                     Core
GitCommitId                                7.3.6
OS                                               Microsoft Windows 10.0.22621
Platform                                      Win32NT
PSCompatibleVersions               1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion                    <IP_ADDRESS>
WSManStackVersion                   3.0
<------------------------------------->
<------------------------------------->

When I execute:
<------------------------------------->
<------------------------------------->
./sbom-tool-win-x64.exe Generate -b "C:\Users\***\Documents\GitHub\serilog" -bc "C:\Users\***\Documents\GitHub\serilog" -D "true" -pn "serilog" -pv "1" -ps "serilog" -m "C:\Users\***\Documents\MyManifests"
<------------------------------------->
<------------------------------------->

I am receiving this error message:
<------------------------------------->
<------------------------------------->
##[error]Encountered an error while generating the manifest.
##[error]Error details: Could not find file 'C:\Users\***\Documents\MyManifests\_manifest\spdx_2.2'.
<------------------------------------->
<------------------------------------->

I encountered the same error message even when I switched from backslash to a forward slash:
<------------------------------------->
<------------------------------------->
./sbom-tool-win-x64.exe Generate -b "C:/Users/***/Documents/GitHub/serilog" -bc "C:/Users/***/Documents/GitHub/serilog" -D "true" -pn "serilog" -pv "1" -ps "serilog" -m "C:/Users/***/Documents/MyManifests"
<------------------------------------->
<------------------------------------->

Given that ChatGPT is of limited use as a generative AI / LLM tool, even ChatGPT is stumped:
<------------------------------------->
<------------------------------------->
The error message "Could not find file 'C:\Users\***\Documents_manifest\spdx_2.2'" indicates that the tool is expecting a specific file in the _manifest directory, but it's not finding it. The program seems to be appending spdx_2.2 to the _manifest directory path, which suggests that it's looking for a file named spdx_2.2 within that directory.

It's possible that the tool is not handling the specified output path correctly. To troubleshoot this further:

Check Documentation: Double-check the documentation for the sbom-tool to ensure that you're using the correct syntax for specifying the output manifest directory using the -m flag.

Try a Different Manifest Directory: Instead of using the root Documents directory, try specifying a subdirectory within it for the manifest output, for example:
<------------------------------------->
<------------------------------------->

 Other ChatGPT suggestions include:
<------------------------------------->
<------------------------------------->
From the interactions you've provided, it seems like you've diligently followed the instructions and tried different variations of the command to generate the SBOM using the sbom-tool. The maintainers of the tool have been providing guidance, but the issue still persists.

Based on the discussions and attempts you've made, here are a few suggestions:

Check Folder Paths: Ensure that the paths you're providing for the -b and -m parameters are correct and accessible. Verify that the folders exist and that you have the necessary permissions to read, write, and create files in those directories.

Escape Special Characters: If your folder paths contain spaces or special characters, make sure to properly escape them or enclose the paths in double quotes. For example, "C:\Users\username\My Folder".

Run as Administrator: It's good to note that you've been running PowerShell and the tool as an administrator. This is important for ensuring you have the necessary permissions to perform file operations.

File Permissions: The error message "Access to the path 'spdx_2.2.spdx' is denied" suggests a file permission issue. Ensure that you have the necessary permissions to create files in the target directory.

Provide Full Manifest Path: Instead of relying on the tool's default behavior for generating the manifest directory and file, consider providing the full path, including the file name, for the -m parameter. For example, -m "C:\Users\...\manifest.spdx.json".

Update PowerShell: While your PowerShell version is up-to-date, you might also consider checking for any updates to PowerShell or your operating system, as sometimes issues can be related to the environment.

Use Environment Variables: If possible, try using environment variables for the folder paths. This can sometimes help avoid issues related to specific user profiles.

Debugging: The sbom-tool seems to provide a verbose mode (-V verbose) which might provide more detailed information about what's happening during the process. This could help you identify the exact point where things go wrong.

Report to Maintainers: Given that you've followed the guidance from the maintainers of the sbom-tool and the issue persists, you should continue to update your GitHub issue thread with the new information, including the results of your latest attempts. The maintainers are likely to have a deeper understanding of the tool's behavior and might be able to provide more specific assistance.

Remember, the maintainers of the sbom-tool are the best resource to help you troubleshoot and resolve this issue, as they have intimate knowledge of how the tool works and its potential pitfalls. Continue collaborating with them through the GitHub issue thread, providing detailed information about your environment, the exact commands you're using, and the error messages you're encountering. This will give them the context they need to provide accurate guidance.
<------------------------------------->
<------------------------------------->

Based on those suggestions, I even expanded my command to:
<------------------------------------->
<------------------------------------->

./sbom-tool-win-x64.exe Generate -b "C:\Users*\Documents\GitHub\serilog" -bc "C:\Users*\Documents\GitHub\serilog" -D "true" -pn "serilog" -pv "1" -ps "serilog" -m "C:\Users***\Documents\MyManifests\manifest.spdx.json" -V "debug"
<------------------------------------->
<------------------------------------->
I ran -V through the six options (Verbose, Debug, Information, Warning, Error, and Fatal"):
<------------------------------------->
<------------------------------------->
"Verbose"
##[information]No namespace URI base provided, using unique generated default value https://spdx.org/spdxdocs/sbom-tool-1.5.0-67fb15ad-dc59-4d91-8129-8875716f2aab
##[debug]Starting SBOM generation workflow.
##[error]Encountered an error while generating the manifest.
##[error]Error details: Could not find file 'C:\Users*\Documents\MyManifests\manifest.spdx.json_manifest\spdx_2.2'.
##[debug]Wrote telemetry object to path
##[warning]0 Packages were detected during the Generate workflow.
##[information]Finished execution of the Generate workflow SBOMTelemetry {Result=Failure, Errors=ErrorContainer1 {Count=0, Errors=[]}, Parameters=Configuration {BuildDropPath=ConfigurationSetting1 {Value="C:\Users\\Documents\GitHub\serilog", Source=CommandLine, IsDefaultSource=False}, BuildComponentPath=ConfigurationSetting1 {Value="C:\\Users\\***\\Documents\\GitHub\\serilog", Source=CommandLine, IsDefaultSource=False}, BuildListFile=null, ManifestPath=null, ManifestDirPath=ConfigurationSetting1 {Value="C:\Users\\Documents\MyManifests\manifest.spdx.json\_manifest", Source=CommandLine, IsDefaultSource=False}, OutputPath=null, Parallelism=ConfigurationSetting1 {Value=8, Source=Default, IsDefaultSource=True}, Verbosity=ConfigurationSetting1 {Value=Verbose, Source=CommandLine, IsDefaultSource=False}, ConfigFilePath=null, ManifestInfo=ConfigurationSetting1 {Value=null, Source=Default, IsDefaultSource=True}, HashAlgorithm=ConfigurationSetting1 {Value=AlgorithmName {Name="SHA256"}, Source=Default, IsDefaultSource=True}, RootPathFilter=null, CatalogFilePath=null, ValidateSignature=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, IgnoreMissing=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, ManifestToolAction=Generate, PackageName=ConfigurationSetting1 {Value="serilog", Source=CommandLine, IsDefaultSource=False}, PackageVersion=ConfigurationSetting1 {Value="1", Source=CommandLine, IsDefaultSource=False}, PackageSupplier=ConfigurationSetting1 {Value="serilog", Source=CommandLine, IsDefaultSource=False}, FilesList=null, PackagesList=null, TelemetryFilePath=null, DockerImagesToScan=null, ExternalDocumentReferenceListFile=null, AdditionalComponentDetectorArgs=null, NamespaceUriUniquePart=null, NamespaceUriBase=ConfigurationSetting1 {Value="https://spdx.org/spdxdocs/sbom-tool-1.5.0-67fb15ad-dc59-4d91-8129-8875716f2aab", Source=Default, IsDefaultSource=True}, GenerationTimestamp=null, FollowSymlinks=ConfigurationSetting1 {Value=True, Source=Default, IsDefaultSource=True}, DeleteManifestDirIfPresent=ConfigurationSetting1 {Value=True, Source=CommandLine, IsDefaultSource=False}, FailIfNoPackages=ConfigurationSetting`1 {Value=False, Source=Default, IsDefaultSource=True}}, SBOMFormatsUsed=[], Timings=[Timing {EventName="Total generation time", TimeSpan="00:00:00.0053489"}], Switches={}, Exceptions={["System.IO.FileNotFoundException"]="Could not find file 'C:\Users\*\Documents\MyManifests\manifest.spdx.json\_manifest\spdx_2.2'."}}
<------------------------------------->
<------------------------------------->
"Debug"
PS C:\Users*\Documents\GitHub\serilog> ./sbom-tool-win-x64.exe Generate -b "C:\Users*\Documents\GitHub\serilog" -bc "C:\Users*\Documents\GitHub\serilog" -D "true" -pn "serilog" -pv "1" -ps "serilog" -m "C:\Users*\Documents\MyManifests\manifest.spdx.json" -V "Debug"
##[information]No namespace URI base provided, using unique generated default value https://spdx.org/spdxdocs/sbom-tool-1.5.0-d7f7a000-c0cc-4c82-9802-83ce83183cf5
##[debug]Starting SBOM generation workflow.
##[error]Encountered an error while generating the manifest.
##[error]Error details: Could not find file 'C:\Users*\Documents\MyManifests\manifest.spdx.json_manifest\spdx_2.2'.
##[debug]Wrote telemetry object to path
##[warning]0 Packages were detected during the Generate workflow.
##[information]Finished execution of the Generate workflow SBOMTelemetry {Result=Failure, Errors=ErrorContainer1 {Count=0, Errors=[]}, Parameters=Configuration {BuildDropPath=ConfigurationSetting1 {Value="C:\Users\\Documents\GitHub\serilog", Source=CommandLine, IsDefaultSource=False}, BuildComponentPath=ConfigurationSetting1 {Value="C:\\Users\\***\\Documents\\GitHub\\serilog", Source=CommandLine, IsDefaultSource=False}, BuildListFile=null, ManifestPath=null, ManifestDirPath=ConfigurationSetting1 {Value="C:\Users\\Documents\MyManifests\manifest.spdx.json\_manifest", Source=CommandLine, IsDefaultSource=False}, OutputPath=null, Parallelism=ConfigurationSetting1 {Value=8, Source=Default, IsDefaultSource=True}, Verbosity=ConfigurationSetting1 {Value=Debug, Source=CommandLine, IsDefaultSource=False}, ConfigFilePath=null, ManifestInfo=ConfigurationSetting1 {Value=null, Source=Default, IsDefaultSource=True}, HashAlgorithm=ConfigurationSetting1 {Value=AlgorithmName {Name="SHA256"}, Source=Default, IsDefaultSource=True}, RootPathFilter=null, CatalogFilePath=null, ValidateSignature=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, IgnoreMissing=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, ManifestToolAction=Generate, PackageName=ConfigurationSetting1 {Value="serilog", Source=CommandLine, IsDefaultSource=False}, PackageVersion=ConfigurationSetting1 {Value="1", Source=CommandLine, IsDefaultSource=False}, PackageSupplier=ConfigurationSetting1 {Value="serilog", Source=CommandLine, IsDefaultSource=False}, FilesList=null, PackagesList=null, TelemetryFilePath=null, DockerImagesToScan=null, ExternalDocumentReferenceListFile=null, AdditionalComponentDetectorArgs=null, NamespaceUriUniquePart=null, NamespaceUriBase=ConfigurationSetting1 {Value="https://spdx.org/spdxdocs/sbom-tool-1.5.0-d7f7a000-c0cc-4c82-9802-83ce83183cf5", Source=Default, IsDefaultSource=True}, GenerationTimestamp=null, FollowSymlinks=ConfigurationSetting1 {Value=True, Source=Default, IsDefaultSource=True}, DeleteManifestDirIfPresent=ConfigurationSetting1 {Value=True, Source=CommandLine, IsDefaultSource=False}, FailIfNoPackages=ConfigurationSetting`1 {Value=False, Source=Default, IsDefaultSource=True}}, SBOMFormatsUsed=[], Timings=[Timing {EventName="Total generation time", TimeSpan="00:00:00.0065438"}], Switches={}, Exceptions={["System.IO.FileNotFoundException"]="Could not find file 'C:\Users\\Documents\MyManifests\manifest.spdx.json\_manifest\spdx_2.2'."}}
PS C:\Users**\Documents\GitHub\serilog>
<------------------------------------->
<------------------------------------->
"Information"
##[information]No namespace URI base provided, using unique generated default value https://spdx.org/spdxdocs/sbom-tool-1.5.0-cd46eafb-6e02-4386-a20d-8eb848825d0c
##[error]Encountered an error while generating the manifest.
##[error]Error details: Could not find file 'C:\Users*\Documents\MyManifests\manifest.spdx.json_manifest\spdx_2.2'.
##[warning]0 Packages were detected during the Generate workflow.
##[information]Finished execution of the Generate workflow SBOMTelemetry {Result=Failure, Errors=ErrorContainer1 {Count=0, Errors=[]}, Parameters=Configuration {BuildDropPath=ConfigurationSetting1 {Value="C:\Users\\Documents\GitHub\serilog", Source=CommandLine, IsDefaultSource=False}, BuildComponentPath=ConfigurationSetting1 {Value="C:\\Users\\***\\Documents\\GitHub\\serilog", Source=CommandLine, IsDefaultSource=False}, BuildListFile=null, ManifestPath=null, ManifestDirPath=ConfigurationSetting1 {Value="C:\Users\\Documents\MyManifests\manifest.spdx.json\_manifest", Source=CommandLine, IsDefaultSource=False}, OutputPath=null, Parallelism=ConfigurationSetting1 {Value=8, Source=Default, IsDefaultSource=True}, Verbosity=ConfigurationSetting1 {Value=Information, Source=CommandLine, IsDefaultSource=False}, ConfigFilePath=null, ManifestInfo=ConfigurationSetting1 {Value=null, Source=Default, IsDefaultSource=True}, HashAlgorithm=ConfigurationSetting1 {Value=AlgorithmName {Name="SHA256"}, Source=Default, IsDefaultSource=True}, RootPathFilter=null, CatalogFilePath=null, ValidateSignature=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, IgnoreMissing=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, ManifestToolAction=Generate, PackageName=ConfigurationSetting1 {Value="serilog", Source=CommandLine, IsDefaultSource=False}, PackageVersion=ConfigurationSetting1 {Value="1", Source=CommandLine, IsDefaultSource=False}, PackageSupplier=ConfigurationSetting1 {Value="serilog", Source=CommandLine, IsDefaultSource=False}, FilesList=null, PackagesList=null, TelemetryFilePath=null, DockerImagesToScan=null, ExternalDocumentReferenceListFile=null, AdditionalComponentDetectorArgs=null, NamespaceUriUniquePart=null, NamespaceUriBase=ConfigurationSetting1 {Value="https://spdx.org/spdxdocs/sbom-tool-1.5.0-cd46eafb-6e02-4386-a20d-8eb848825d0c", Source=Default, IsDefaultSource=True}, GenerationTimestamp=null, FollowSymlinks=ConfigurationSetting1 {Value=True, Source=Default, IsDefaultSource=True}, DeleteManifestDirIfPresent=ConfigurationSetting1 {Value=True, Source=CommandLine, IsDefaultSource=False}, FailIfNoPackages=ConfigurationSetting`1 {Value=False, Source=Default, IsDefaultSource=True}}, SBOMFormatsUsed=[], Timings=[Timing {EventName="Total generation time", TimeSpan="00:00:00.0062780"}], Switches={}, Exceptions={["System.IO.FileNotFoundException"]="Could not find file 'C:\Users\*\Documents\MyManifests\manifest.spdx.json\_manifest\spdx_2.2'."}}
<------------------------------------->
<------------------------------------->
"Warning"
##[error]Encountered an error while generating the manifest.
##[error]Error details: Could not find file 'C:\Users*\Documents\MyManifests\manifest.spdx.json_manifest\spdx_2.2'.
##[warning]0 Packages were detected during the Generate workflow.
PS C:\Users*\Documents\GitHub\serilog>
What step or steps am I missing?
Thank you in advance for any assistance!
<------------------------------------->
<------------------------------------->
"Error"
##[error]Encountered an error while generating the manifest.
##[error]Error details: Could not find file 'C:\Users***\Documents\MyManifests\manifest.spdx.json_manifest\spdx_2.2'.
<------------------------------------->
<------------------------------------->
"Fatal"
\Documents\MyManifests\manifest.spdx.json" -V "Fatal"
<------------------------------------->
<------------------------------------->
These error messages support ChatGPT's "suggestion" (or "hallucination", based on your point of view) that the tool is having issues with appending spdx_2.2 to the _manifest directory path.
Is there any impact from this line in the test file "sbom-tool/test/Microsoft.Sbom.Api.Tests/Executors/ExternalDocumentReferenceWriterTest.cs" (https://github.com/search?q=repo%3Amicrosoft%2Fsbom-tool %2Fmanifest&type=code)?
   var jsonFilePath = "/root/_manifest/manifest.json";

What step or steps am I missing?
Thank you in advance for any assistance you can provide!
I have to admit that ChatGPT is using in some respects.   I was not aware that the wrong kind of quotation marks will cause issues in PowerShell.
<------------------------------------->
<------------------------------------->

The issue you're facing is likely due to the fact that you're using the wrong type of quotation marks (“”) around the argument values. The program is expecting regular double quotation marks ("). This can cause PowerShell to treat the entire value as a single argument, which is likely why you're seeing the "Unexpected argument: serilog" error.
<------------------------------------->
<------------------------------------->

Could my error be tied to the file SPDX22ManifestConfigHandler.cs (https://github.com/microsoft/sbom-tool/blob/05494772c801e2b29fb7db9d882a80bbd9ee1789/src/Microsoft.Sbom.Api/Manifest/ManifestConfigHandlers/SPDX22ManifestConfigHandler.cs#L26)?
private string ManifestDirPath => configuration.ManifestDirPath?.Value;
// directory path for SPDX 2.2 is
// root/_manifest/spdx_2.2/
private string SbomDirPath => fileSystemUtils.JoinPaths(ManifestDirPath, $"{Constants.SPDX22ManifestInfo.Name.ToLower()}_{Constants.SPDX22ManifestInfo.Version.ToLower()}");

// sbom file path is manifest.spdx.json in the sbom directory.
private string SbomFilePath => fileSystemUtils.JoinPaths(SbomDirPath, $"manifest.{Constants.SPDX22ManifestInfo.Name.ToLower()}.json");

Today (Sunday, 27 August 2023), I updated to version 1.5.1 (released Friday, 24 August 2023 at 16:04 GMT).
I downloaded the sample manifest file posted to https://github.com/microsoft/sbom-tool/tree/71d1184db22f27159d82859f82e1b2dc3c3ace49/samples.
I even played with variations of the theme of changing both the file name and the location for the sample manifest .json file.  For some reason, the sbom-tool does not like my path name for manifest .json file.
For example, in the Terminal in Visual Studio Code, I ran:
./sbom-tool-win-x64.exe Generate -b "C:\Users*\Documents\GitHub\serilog_manifest" -bc "C:\Users*\Documents\GitHub\serilog" -D "true" -pn "serilog" -pv "1" -ps "serilog" -m "C:\Users***\Documents\MyManifests_manifest\spdx_2.2" -V "Verbose"
I received this error code:
##[information]Could not start execution of workflow. Logging telemetry SBOMTelemetry {Result=Failure, Errors=null, Parameters=InputConfiguration {BuildDropPath=ConfigurationSetting1 {Value="C:\\Users\\***\\Documents\\GitHub\\serilog\\_manifest", Source=CommandLine, IsDefaultSource=False}, BuildComponentPath=ConfigurationSetting1 {Value="C:\Users\\Documents\GitHub\serilog", Source=CommandLine, IsDefaultSource=False}, BuildListFile=null, ManifestPath=null, ManifestDirPath=ConfigurationSetting1 {Value="C:\\Users\\***\\Documents\\MyManifests\\_manifest\\spdx_2.2", Source=CommandLine, IsDefaultSource=False}, OutputPath=null, Parallelism=null, Verbosity=ConfigurationSetting1 {Value=Verbose, Source=CommandLine, IsDefaultSource=False}, ConfigFilePath=null, ManifestInfo=ConfigurationSetting1 {Value=null, Source=Default, IsDefaultSource=True}, HashAlgorithm=ConfigurationSetting1 {Value=AlgorithmName {Name="SHA256"}, Source=Default, IsDefaultSource=True}, RootPathFilter=null, CatalogFilePath=null, ValidateSignature=null, IgnoreMissing=null, ManifestToolAction=Generate, PackageName=ConfigurationSetting1 {Value="serilog", Source=CommandLine, IsDefaultSource=False}, PackageVersion=ConfigurationSetting1 {Value="1", Source=CommandLine, IsDefaultSource=False}, PackageSupplier=ConfigurationSetting1 {Value="serilog", Source=CommandLine, IsDefaultSource=False}, FilesList=null, PackagesList=null, TelemetryFilePath=null, DockerImagesToScan=null, ExternalDocumentReferenceListFile=null, AdditionalComponentDetectorArgs=null, NamespaceUriUniquePart=null, NamespaceUriBase=null, GenerationTimestamp=null, FollowSymlinks=null, DeleteManifestDirIfPresent=ConfigurationSetting1 {Value=True, Source=CommandLine, IsDefaultSource=False}, FailIfNoPackages=null}, SBOMFormatsUsed=null, Timings=[], Switches={}, Exceptions={["PowerArgs.ValidationArgException"]="ManifestDirPath directory not found for 'C:\Users\\Documents\MyManifests\_manifest\spdx_2.2'"}}
ManifestDirPath directory not found for 'C:\Users*\Documents\MyManifests_manifest\spdx_2.2'
##[information]No namespace URI base provided, using unique generated default value https://spdx.org/spdxdocs/sbom-tool-1.5.1-04741ad6-9b01-417d-976f-6cbedfdca823
##[debug]Starting SBOM generation workflow.
##[error]Encountered an error while generating the manifest.
##[error]Error details: The directory name is invalid. : 'C:\Users*\Documents\MyManifests_manifest\spdx_2.2'
##[warning]Manifest generation failed, however we were unable to delete the partially generated manifest.json file and the C:\Users*\Documents\MyManifests_manifest directory because the directory was not empty.
##[debug]Wrote telemetry object to path
##[warning]0 Packages were detected during the Generate workflow.
##[information]Finished execution of the Generate workflow SBOMTelemetry {Result=Failure, Errors=ErrorContainer1 {Count=0, Errors=[]}, Parameters=Configuration {BuildDropPath=ConfigurationSetting1 {Value="C:\Users\\Documents\GitHub\serilog\", Source=CommandLine, IsDefaultSource=False}, BuildComponentPath=ConfigurationSetting1 {Value="C:\\Users\\***\\Documents\\GitHub\\serilog", Source=CommandLine, IsDefaultSource=False}, BuildListFile=null, ManifestPath=null, ManifestDirPath=ConfigurationSetting1 {Value="C:\Users\\Documents\MyManifests\_manifest", Source=CommandLine, IsDefaultSource=False}, OutputPath=null, Parallelism=ConfigurationSetting1 {Value=8, Source=Default, IsDefaultSource=True}, Verbosity=ConfigurationSetting1 {Value=Verbose, Source=CommandLine, IsDefaultSource=False}, ConfigFilePath=null, ManifestInfo=ConfigurationSetting1 {Value=null, Source=Default, IsDefaultSource=True}, HashAlgorithm=ConfigurationSetting1 {Value=AlgorithmName {Name="SHA256"}, Source=Default, IsDefaultSource=True}, RootPathFilter=null, CatalogFilePath=null, ValidateSignature=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, IgnoreMissing=ConfigurationSetting1 {Value=False, Source=Default, IsDefaultSource=True}, ManifestToolAction=Generate, PackageName=ConfigurationSetting1 {Value="serilog", Source=CommandLine, IsDefaultSource=False}, PackageVersion=ConfigurationSetting1 {Value="1", Source=CommandLine, IsDefaultSource=False}, PackageSupplier=ConfigurationSetting1 {Value="serilog", Source=CommandLine, IsDefaultSource=False}, FilesList=null, PackagesList=null, TelemetryFilePath=null, DockerImagesToScan=null, ExternalDocumentReferenceListFile=null, AdditionalComponentDetectorArgs=null, NamespaceUriUniquePart=null, NamespaceUriBase=ConfigurationSetting1 {Value="https://spdx.org/spdxdocs/sbom-tool-1.5.1-04741ad6-9b01-417d-976f-6cbedfdca823", Source=Default, IsDefaultSource=True}, GenerationTimestamp=null, FollowSymlinks=ConfigurationSetting1 {Value=True, Source=Default, IsDefaultSource=True}, DeleteManifestDirIfPresent=ConfigurationSetting1 {Value=True, Source=CommandLine, IsDefaultSource=False}, FailIfNoPackages=ConfigurationSetting`1 {Value=False, Source=Default, IsDefaultSource=True}}, SBOMFormatsUsed=[], Timings=[Timing {EventName="Total generation time", TimeSpan="00:00:00.0120215"}], Switches={}, Exceptions={["System.IO.IOException"]="The directory name is invalid. : 'C:\Users\*\Documents\MyManifests\_manifest\spdx_2.2'"}}
Does anyone know why the sbom-tool doesn't like the directory name?
Exceptions={["System.IO.IOException"]="The directory name is invalid. : 'C:\Users\***\Documents\MyManifests\_manifest\spdx_2.2'"}}

@Charles1992Sail The -m parameter you provided is a valid name as long as it already exists on your device and it is a writeable location. For example, if you have an existing directory named 'C:\Users***\Documents\MyManifests' and used it as your ManifestDirPath then the tool will write the _manifest and spdx_2.2 directories on that path.
It is worth noting that there can only be one _manifest directory on a given path so if you're planning on storing several SBOMs on this path you should consider nesting the directory one layer more. Ex: 'C:\Users***\Documents\MyManifests\SBOM1' and making sure it exists before passing it as a parameter.
Let us know if this helps. Thanks.

@sebasgomez238
 Thank you for reviewing my issue.  You have confirmed that I was at least heading down the right path, so my own sense of sanity feels better.

 Your point on the ManifestDirPath was exactly where I was getting stuck.... until just now while I was typing up a response to you.

 Bottom line up front (BLUF): If all else fails, re-check the obvious!!

 For background, I am running Windows 11 on a Surface Pro 9.

 For some reason, there is a read access for my folders on the hard drive.  Even though I have myself as an admin, the "Read-only" attribute remains checked for any folder on the hard drive. I can't get the attribute to stay unchecked.  While I can save any other file just fine otherwise to my Surface Pro's hard drive, this error was enough of a road-block to stop the sbom-tool from writing to the hard drive.

 However, on a whim, I switched the path for my -m argument to my OneDrive folder, and magically the tool produced a SBOM!  Yay!!!!!

Detailed background:
I am running PS 7.3.6 from the CLI as an Administrator along with the most recent sbom-tool version (1.5.2 released on Monday, 28 August 2023 at 22:29 GMT).
PS C:\Users___\Documents\GitHub\sbom-tool> ./sbom-tool-win-x64.exe --version
1.5.2
PS C:\Users___\Documents\GitHub\sbom-tool> ./sbom-tool --version
1.5.2
<----------------------------------------------->
<----------------------------------------------->
I added two arguments based on:

-nsb per the arguments at https://github.com/microsoft/sbom-tool/blob/main/docs/sbom-tool-arguments.md.  I also got an error message on that one.
"No namespace URI base provided, using unique generated default value https://spdx.org/spdxdocs/sbom-tool-1.5.2-cc35c395-3897-4251-9c1e-c76278a5cf57".

I added -D "true" after finding it when I ran ./sbom-tool -h.  -D is not listed on https://github.com/microsoft/sbom-tool/blob/main/docs/sbom-tool-arguments.md

<----------------------------------------------->
<----------------------------------------------->
DeleteManifestDirIfPresent (-D)           If set to true, we will delete any previous manifest
directories that are already present in the
ManifestDirPath without asking the user for
confirmation. The new manifest directory will then
be created at this location and the generated SBOM
will be stored there.
<----------------------------------------------->
<----------------------------------------------->
Here is the command that finally worked:
<----------------------------------------------->
<----------------------------------------------->
./sbom-tool Generate -b "C:\Users_\Documents\GitHub\serilog" -bc "C:\Users_\Documents\GitHub\serilog" -m "C:\Users__________\OneDrive\Documents" -nsb "https://spdx.org/spdxdocs/sbom-tool-1.5.2-cc35c395-3897-4251-9c1e-c76278a5cf57" -D "true" -pn "serilog" -pv "1" -ps "serilog" -V "Verbose"
<----------------------------------------------->
<----------------------------------------------->
I finally realized that a simple folder permission issue fixed the problem.  That's OK!  I learned a lot of good stuff during this journey, so the effort is not wasted.


UK data sources

Goal: explore OA-level 2020 data. A possible use case could be adding as a layer in the LTN and 15m tools.
Geometry
Fernando made this for SPC:
https://ramp0storage.blob.core.windows.net/nationaldata-v2/GIS/OA_2011_Pop20.geojson
But let's track down the originals... https://geoportal.statistics.gov.uk/search?collection=Dataset&sort=name&tags=all(BDY_OA%2CDEC_2021)
Full extent, full clipped, or generalised clipped.

https://github.com/virgesmith/UKCensusAPI is an option. I'm trying to work out if simple bulk downloads can just be built in the browser. https://www.nomisweb.co.uk/api/v01/help

Number of cars: https://www.ons.gov.uk/filters/432658ed-9740-46f4-9565-e0321181ea75/dimensions, or https://www.ons.gov.uk/datasets/TS045/editions/2021/versions/3/filter-outputs/a20437fb-ae7f-439b-bc91-de261335038b#get-data. I tried generating the API link through https://www.nomisweb.co.uk, but got an internal error of some sort.
For now, I'd be fine doing the data downloads manually.
But now it's time to figure out how to more compactly represent all the datasets for the A/B Street map model.

How do you actually get the number of people in an OA? https://www.ons.gov.uk/census/maps/choropleth/population/population-density/population-density/persons-per-square-kilometre?oa=E00091042 and then multiply by the area?

E00104221 (and many other areas) have a point crossing itself in the final clipped GeoJSON:

Where in the pipeline is this being introduced?

The TopoJSON is over-simplified. Need to look into the mapshaper options!

I've made a start on this, and here are some notes:
Geometries
There are three different versions of the Output Areas (OAs) available from the ONS. By ad-hoc visual inspection, the Generalised Clipped EW (BGC) is at least as good as the current data - but will review.
Each of these versions has 188880 records. This is consistent with the census data (see below). I presume from the filename that the OA are from 2011, and that's the reason for the discrepancy between the number of records in the census data and the geometry data.

Output Areas (Dec 2021) Boundaries Generalised Clipped EW (BGC)

WebUI: https://geoportal.statistics.gov.uk/datasets/ons::output-areas-dec-2021-boundaries-generalised-clipped-ew-bgc
WFS https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Output_Areas_Dec_2021_Boundaries_Generalised_Clipped_EW_BGC_2022/FeatureServer/0

Output Areas (Dec 2021) Boundaries Full Clipped EW (BFC)

WebUI: https://geoportal.statistics.gov.uk/datasets/ons::output-areas-dec-2021-boundaries-full-clipped-ew-bfc
WFS: https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Output_Areas_Dec_2021_Boundaries_Full_Clipped_EW_BFC_2022/FeatureServer

Output Areas (2021) EW BFE (?Boundary Full Extent?)

webui:  https://geoportal.statistics.gov.uk/datasets/ons::output-areas-2021-ew-bfe/

There does not seem to be a straightforward way to programmatically download the data. The best (or least worst) way to use the WFS. However, there is a limit of 4000 records per request. It seems that OGR has a way of handling this:
https://gis.stackexchange.com/questions/422609/downloading-lots-of-data-from-wfs
Other options are:

Manually download the data and save it somewhere.
The file is downloaded from this URL, but there are authentication/session-cookie issues. Whilst it might be possible to hack this, it seems like a very fragile solution as no doubt the service has been designed to prevent this:
https://stg-arcgisazurecdataprod1.az.arcgis.com/exportfiles-1559-17258/Output_Areas_Dec_2021_Boundaries_Generalised_Clipped_EW_BGC_2022_-5131276949491721609.geojson?sv=2018-03-28&sr=b&sig=pH%2Fg7tbnYF5L%2BXB9lyHIs8kViwICm3YEIta7u9JT9Ug%3D&se=2023-04-21T14%3A34%3A44Z&sp=r

Census data from browser download
The WebUI to access the car ownership census data is here:
https://www.ons.gov.uk/datasets/TS045/editions/2021/versions/3/filter-outputs/a20437fb-ae7f-439b-bc91-de261335038b#get-data
This need to be downloaded manually.
The csv can be downloaded using wget from this URL (though it is unclear if this URL will remain valid)):
CENSUS_URL = "https://static.ons.gov.uk/datasets/a20437fb-ae7f-439b-bc91-de261335038b/TS045-2021-3-filtered-2023-03-13T16:49:47Z.csv"
This has 944401 lines (including the header)
944400 / 188880 = 5 categories per OA, which is what we expect.
Census data using the NOMIS API and UKCensusAPI

* keep a permalink to the extract? (let's see if https://www.nomisweb.co.uk/Query/GetFile?filename=2422351643080675.csv lasts over time)

I tried this on 2023/04/21 and got this error message:
Session timed out or not logged in: This page is only available when you query data on Nomis, you cannot bookmark it or directly link to it.

So unfortunately, that's not an option...

I tried generating the API link through https://www.nomisweb.co.uk, but got an internal error of some sort.

I also tried this and got the same error message. However, in the help, it states that there is a 25000 cell limit per request, which might be the issue we're hitting. It is possible to obtain an API key which increases the limit to 1 million cells. See https://github.com/virgesmith/UKCensusAPI#api-key
1M limit would be enough for the whole of England, for five categories per OA, but would not be sufficient for more complex census queries. In these cases, we could loop through batches of OA codes and then merge the results. Might be useful if we want to be able to get lots of different data from NOMIS. It's not clear if the UKCensusAPI package can do this. If not, then it might be appropriate to make a PR to that package rather than adding it here.

A question: Where does this pipeline need to be run? Are there any restrictions on the environment? (I note there is a comment above about processing bulk downloads in the browser).

Thanks for the investigation!

Whilst it might be possible to hack this, it seems like a very fragile solution as no doubt the service has been designed to prevent this

Hmm, I can get to a download link with a few clicks manually. I wonder if we can write a simple spider to mimic what a human does; https://github.com/yt-dlp/yt-dlp is a modern example that works great for many sources. Or we probably have some ONS contacts and could ask them what they want the right approach to be for reproducible pipelines.

If not, then it might be appropriate to make a PR to that package rather than adding it here.

+1 to improving something that exists if other projects are already using it and it's well-established

Where does this pipeline need to be run? Are there any restrictions on the environment?

At the moment, manually in a Linux or Mac environment. Conceivably we might want to run it on a cloud VM someday. As long as the dependencies are easy to install (one or two commands) and things won't quietly break over time (so, something with lockfiles pinning to specific versions), I'd be happy. On that note, I'd love to ask what the current best practices for Python dependencies/environments are from REG's perspective -- I've had most luck with Poetry, but still not perfect. :\

I’ve been looking again at the UK - but mostly England - census data. I've been trying to find a solution that doesn’t require manually finding the URL of every table we require. There seem to be three main (slightly overlapping options):
1. The UKcensusAPI package.
As far as I can tell:

It only supports 2011 data, not 2021. We could write a PR for this, but that might be non-trivial, and basically incorporates much of the work for option 2.
Even with an API key, it still limits queries to a fixed number of rows per query. Downloading tables of the whole country exceeds this. We could write a PR to add pagination.

How does this tally with the experience of using it within SPC?
2. The main Nomisweb REST API

There is a machine-readable list of available data here:

"https://www.nomisweb.co.uk/api/v01/dataset/def.sdmx.json"

It seems reasonably easy to then download the data - eg try this link (gets the first 25000 rows); https://www.nomisweb.co.uk/api/v01/dataset/NM_1_1.data.csv?geography=TYPE304
It is not obvious (to me) how to tell which geography applies to each table. It might be possible to workout using some rule-based automated trial and error (or it could be that it exists in the metadata, just I’ve not found it yet)
It is necessary to add code to handle pagination, but there is a record count with every row.

3. Use the list of “Bulk Downloads”
See https://www.nomisweb.co.uk/sources/census_2021_bulk
It would be possible to use some web-scraping process (i.e. Beautiful soup) to get a list of zip files to download. Then download all of them and process the results. This feels like it would be quicker to implement but would be a lot more fragile.
My impression is that option 2 is the best option, though creating a PR for UKcensusAPI might be a good way to achieve the same thing. Any opinions? I would be particularly keen to hear from those who have used UKcensusAPI.


Fix CI in bevy-0.13 branch

Fixes the doc test in the bevy-0.13 branch to make CI green again.

Thanks!


Refinement: remove unnecessary GC & export methods for 3rd party

This PR resolves the performance issue of the new geodata decoder and export methods and errors to 3rd parties.

I see almost no functional change...

The reason why the CPU gets crazy is that too many GCs explicitly.

may not necessary to public the emitBytes. call Decode() for thrid_party is fine.


Column names imod-wq well IPF's: x, y, top, and bot

In GitLab by @Huite on May 22, 2019, 14:50
For reference, it turns out the IPF reader in imod-wq does rely on labels:
      xc = 0; yc = 0; vc = 0; tc = 0; bc = 0
      read(lun,*) nlist
      read(lun,*) nc
      do i = 1, nc
         read(lun,'(a)') line
         line = adjustl(line)
         call cfn_s_upcase(line)
         if (line(1:1).eq.'X') then
            xc = i
         else if (line(1:1).eq.'Y') then
            yc = i
         else if (line(1:3).eq.'TOP') then
            tc = i
         else if (line(1:3).eq.'BOT') then
            bc = i
         else if (trim(line).eq.trim(ipf_label)) then
            vc = i
         end if
      end do

So these should be the only possible output for the wel.save method.
Not sure if we should support top and bot: wells could still end up in aquitards.

In GitLab by @Huite on May 22, 2019, 22:44
closed


Arbitrum Uniswap V3 TWAP bot bug: Front-end shows TWAP bot active after all TWAP cleared

I had a TWAP with two trades: https://arbiscan.io/tx/0x802a3399ee0862943b79e36e1ddd2c570b0a3687eb1141ad8e42342bd89350fc
The last one went through at Jan-17-2024 08:38:14 PM +UTC: https://arbiscan.io/tx/0xf01bcc181111bb9a2d0cae7dd2fa4b78c11cf0f777115dea3201ae71958b66f7
But at Jan-17-2024 08:47:14 PM +UTC mybot page is still showing the bot as active:

What's needed:

understand why it's happening
All TWAP bots should become inactive and removed from My Bots to Old Bots on the FE, shortly after all TWAP clear out

@webelf101 for some reason withdraw block wasn't set on a transaction where the remaining_count is 0.
select deposit_id,bot,dex_name,number_trades,remaining_counts,withdraw_block,contract from deposits;

0|twap|uniswapv3|2|0|19028014|0x7cb5d8DD2EC6aCa4F006b44De8b734Cc1298f40D
0|twap|uniswapv3|2|2||0x37D3F3B7a5F091aBF36459430Da9BB883A710E69
1|twap|uniswapv3|2|0||0x37D3F3B7a5F091aBF36459430Da9BB883A710E69
0|twap|uniswapv3|1|1||0x14dBCf69DadA3e5A9aFC54a24fdd1467c2bbd346
1|twap|uniswapv3|2|1||0x14dBCf69DadA3e5A9aFC54a24fdd1467c2bbd346

@rootedbox If remaining_counts is zero, we can consider it is done.


CoC: remove redundant verbage and simplify

This simplifies the Code of Conduct by removing language that is redundant, too verbose, or otherwise unnecessary for the reader to understand the point of the Code of Conduct.

no


[Minor] Fix some indent issues.

What changes were proposed in this pull request?
Fix some indent issues.
How was this patch tested?
existing tests.

Test build #77477 has started for PR 18133 at commit c90b252.

Test build #77477 has finished for PR 18133 at commit c90b252.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77477/
Test PASSed.

Indent issues look ununified description:

OK. I don't think this is worth bothering with.

Merged to master anyway


attu-setup.exe installation failure

Describe the bug:
Could not install attu-setup.exe on Windows
Attu version:
attu-Setup-2.4.1.exe
attu-Setup-2.3.10.exe
Windows version:
Windows 10 Professional x64

That's weird .


Add reference implementations

These projects are fully featured and tested. In the next two weeks, they will be battle tested.
They were developed as part of rocketry project for my student association for EuRoC competition.
They might not be the best reference implementations, but I wasn't able to find anything else working here on github. They might not be up to the standards, but they work.
Thank you for enabling this project with your library.

Good Morning @not7cd :wave: :coffee:
I'm glad to read that you've been able to use this library for your EuRoc project. Maybe you want to present it in the UAVCAN forum?
Unfortunately your project do not contain sufficient functionality to be called a reference implementation, it misses the register functionality for dynamic configuration. I know the same is true for the GNSS and ToF node, but those codes are under my sphere of influence so I can always update them while the same can't be said for your code. My compromise would be to list them under a "See also" header, very similar it's been done here (https://github.com/UAVCAN/platform_specific_components/pull/16). Would that be acceptable to you?


Resource page with Data Explorer or preview

A page for viewing an individual resource exists and it previews the data (if a preview for that data type exists).
In particular, for CSV etc and data in CKAN DataStore it displays the Data Explorer.
NOTE: ⚠ there is quite a bit of thinking here about how we refactor the data previewing and data explorer. Read the notes in analysis section below. Note, in particular, that the acceptance and tasks are not yet updated to reflect this (we will probably need to refactor this issue into multiple issues).
Acceptance

[ ] Resource page works from the mocks, e.g., localhost:3000/@world-bank/gdp/r/gdp loads and displays resource metadata WIP<EMAIL_ADDRESS>[ ] For CSV we get a data preview (as per data explorer)
[ ] Ditto for data in CKAN DataStore

Tasks

[ ] Implement /[org]/[dataset]/r/[resource] page that renders necessary components for the resource page:

[x] Navbar
[x] About resource component, e.g., metadata table with info such as resource name, title, description, format, created/modified date and link to downloading

[ ] Data Explorer or Data Preview component based on given resource type:

[ ] If in datastore, we can show the data explorer that has preview + query builder + map/chart builder + pagination
[ ] If not in datastore, we only render a preview based on format:

[ ] tabular data => table
[ ] image/website/pdf => web view
[ ] geo data => map
[ ] anything else => nothing

Analysis
Re-using existing Data Explorer library might not be the best way to go here because:

Data Explorer we've built using React + Redux wasn't the easiest library to maintain, for example, if you need to fix something in one of the components (eg, Map Builder), you have to go there and do:

fix it
if unit tests there, run them
test with fixtures (cosmos)
test with yarn start to load complete app
build the dist version by yarn build:package
push to github
publish to NPM
install it in Data Explorer and build the bundle
import the bundles (normally huge ~>2-3MB due to dependencies such as Plotly) into the HTML page

As we can see, it is quite a long process and over time we've seen little feature development but only bug fixes. I believe the process was discouraging for developers in general.
Other points:

Currently, using Data Explorer as a React component is not clear as it was always been used as bundles that doesn't get called explicitly with arguments, but inits on load and traverses the DOM to find the HTML tag with data-pacakge attribute. It parses it and uses it as props for child components.
If we want to use Data Explorer and its components in NextJS, we have to refactor it as it isn't compatible right away, for example, importing CSS, see https://github.com/vercel/next.js/blob/master/errors/css-npm.md

How would I build Data Explorer today with NextJS?

Make the Data Explorer a component rather than a library
Control components such as Query/Map/Chart builder from NextJS so basically our frontend app becomes the Data Explorer
Use something like SWR for client-side data fetching - https://swr.now.sh/

http://tech.datopian.com/data-explorer
graph TD

DataExplorer
DataPackageViews
DataStoreQueryBuilder
ChartBuilder
MapBuilder
Redux

subgraph old
  Redux -.- DataExplorer
  DataExplorer --> DataPackageViews
  DataExplorer --> DataStoreQueryBuilder
  DataExplorer --> ChartBuilder
  DataExplorer --> MapBuilder
end

graph TD

NextJSPage
DataPackageViews
DataStoreQueryBuilder
ChartBuilder
MapBuilder
GraphQL

subgraph new
  NextJSPage --init--> DataPackageViews
  NextJSPage --init--> DataStoreQueryBuilder
  NextJSPage --init--> ChartBuilder
  NextJSPage --init--> MapBuilder
  GraphQL -.- NextJSPage
  GraphQL -.- DataPackageViews
  GraphQL -.- DataStoreQueryBuilder
  GraphQL -.- ChartBuilder
  GraphQL -.- MapBuilder
end

User:  GET a resource page - /@org/{dataset-name}/r/{resource-name}
NextJS (server side) fetches DMS API to get resource metadata which is then cached by GraphQL
NextJS responses with HTML + JSON for hydrating the tempalte
Browser renders the page with components
DataPackageViews component fetches data from the source url (data can be in filestore [or datastore])

OR: could load the data server side and pass down ...

Renders the data in a table (for tabular data)

TODO: other renderings ...``

First steps:

Review current Data Explorer lib [0.5h]
Review datapackage views component [0.5h]
Refactore NextJS page for resource - https://github.com/datopian/portal.js/blob/master/pages/[org]/[dataset]/r/[resource]/index.tsx

we already implemented server side data fetching (resource object)
we next need to create DataPackageView component which should:

Remove DataExplorer component and create DataPackageView compoent instead
Copy over implementation of DataExplorer
fetch data in browser (eg, use https://swr.vercel.app/) [4h]
render table based on that data [4h]

we can re-use existing datapackage-views-js here

Notes

Resource pages do simple (pre)view by default
Explorer page for rich exploration (requires)

# url structure
/dataset/myname/ => [implicitly] /dataset/myname/show/default/ # default-branch-latest-revision/
/dataset/myname/resource/xxx => [implicitly] renders  /dataset/myname/show/default/{resource-id}

# explicit revisions
/dataset/myname/show/{revref}/              # dataset showcase
/dataset/myname/show/{revref}/{resource-id} # resource showcase

Re-using existing Data Explorer library might not be the best way to go here because:

Data Explorer we've built using React + Redux wasn't the easiest library to maintain, for example, if you need to fix something in one of the components (eg, Map Builder), you have to go there and do:

fix it
if unit tests there, run them
test with fixtures (cosmos)
test with yarn start to load complete app
build the dist version by yarn build:package
push to github
publish to NPM
install it in Data Explorer and build the bundle
import the bundles (normally huge ~>2-3MB due to dependencies such as Plotly) into the HTML page

As we can see, it is quite a long process and over time we've seen little feature development but only bug fixes. I believe the process was discouraging for developers in general.
Other points:

Currently, using Data Explorer as a React component is not clear as it was always been used as bundles that doesn't get called explicitly with arguments, but inits on load and traverses the DOM to find the HTML tag with data-pacakge attribute. It parses it and uses it as props for child components.
If we want to use Data Explorer and its components in NextJS, we have to refactor it as it isn't compatible right away, for example, importing CSS, see https://github.com/vercel/next.js/blob/master/errors/css-npm.md

How would I build Data Explorer today with NextJS?

Make the Data Explorer a component rather than a library
Control components such as Query/Map/Chart builder from NextJS so basically our frontend app becomes the Data Explorer
Use something like SWR for client-side data fetching - https://swr.now.sh/

@rufuspollock I've added couple of comments above, please, take a look.

@anuveyatsu ok - for now we build the page without preview and move one whilst we decide what we want to do here 😄

This is how resource page looks like at the moment:
<EMAIL_ADDRESS>

WONTFIX. Closing for now as Catalog related and we aren't going to do much morer right now. Note we have implemented views support in the basic portal app in #72.


Python interpreter not working in aiohttp/aiohttp directory

Long story short
When executing the python interpreter or calling python under the aiohttp/aiohttp directory, it crashes with the following output:
19:17 $ python
Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
  File "/Users/manuelmiranda/.virtualenvs/aiohttp/bin/../lib/python3.5/io.py", line 52, in <module>
  File "/Users/manuelmiranda/Projects/aiohttp/aiohttp/abc.py", line 1, in <module>
ImportError: No module named 'asyncio'
Abort trap: 6

This is because having a module named abc.py in that directory.
Steps to reproduce
cd aiohttp/aiohttp;python
Your environment
MAC, using virtualenv and without using it.
After checking this module it seems classes there are not exposed to public functionality. If it's fine, I could try to do a pull request with the changes (i.e renaming the file to something like abstracts.py or abstract_classes.py).
Cheers!

Do not name your directories asyncio it conflicts with asyncio module.

It's not asyncio/aiohttp issue but the rule how Python is working.

@popravich @asvetlov d'oh, sry for the mess, when I was writing asyncio I meant aiohttp. I've just updated the description =/. Guess it doesn't affect because the issue is already closed but wanted to correct it.

anyway it's a problem with path names -- try renaming your paths so that they dont match python lib names

@popravich they are not my paths, I just cloned the aiohttp repo, then cd aiohttp/aiohttp and then python. Then it's giving the crash.


ABI breakage in 0.7 due to new kflags member in io_uring_cq

Hi!
Steinar H. Gunderson has reported in Debian (https://bugs.debian.org/972758) that the 0.7 release broke the ABI in commit 0f0517331307605e576b3492c93dc4988e06fdc0 when adding the kflags member to struct io_uring_cq. I think at this point the best way forward might be to just quickly release 2.0 and request any downstream to avoid 0.7 and package that instead?

Oops yes, that is unfortunate. As 0.7 is 3 months old, it's definitely out there. I guess people just recompiled, I hadn't heard any complaints.
In any case, I don't think we need to be quick on the next release, we're already more than 3 months into this. For 2.0, at least the padding is there for future cases.

I hadn't heard any complaints.

Should clarify, before this one :-)


Artifactory NuGet task - set nuget protocol v2 as default and allow p…

…rotocol selection
This PR was created following the discussion in https://github.com/jfrog/artifactory-azure-devops-extension/issues/248

[x] All tests passed. If this feature is not already covered by the tests, I added new tests.
[x] This pull request is on the dev branch.
[x] I used npm run format for formatting the code before submitting the pull request.

Should we do the same for .NET tasks?

True!

Should we do the same for .NET tasks?

True!

I suggest we release this change gradually. Let's wait for some feedback, and see if the usage of the v2 protocol is also required fo .NET core CLI.
Maybe it is required, and it makes more sense to have v3 as the default...


Add text in FacetGrid subplots

Adding subplots numbers in the FacetGrid subplots. This will help in the easy generation of scientific publication quality figures.
MCVE Code Sample

import xarray as xr

airtemps = xr.tutorial.open_dataset('air_temperature')
air = airtemps['air']
t = air.isel(time=slice(0, 365 * 4, 250))
g_simple = t.plot(x='lon', y='lat', col='time', col_wrap=3)

#adding sub plot number
string = ['a','b','c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l']
for i, ax in enumerate(g_simple.axes.flatten()):
    ax.text(0.85, 0.85,'(' + string[i] +')',transform=ax.transAxes,
            bbox=dict(boxstyle="square", fc="w", ec="k"))

Expected Output

Problem Description

Currently, there is no argument that can enable the subplot numbers in the FacetGrid subplot. In the case of scientific publications, subplots need to be numbered. The addition of this feature will enable the easy generation of good quality figures.
Versions
Output of xr.show_versions()

INSTALLED VERSIONS
commit: None
python: 3.7.7 (default, May  6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 166 Stepping 0, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None
libhdf5: 1.10.4
libnetcdf: 4.7.3
xarray: 0.15.1
pandas: 1.0.3
numpy: 1.18.1
scipy: 1.4.1
netCDF4: 1.5.3
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.1.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.16.0
distributed: 2.16.0
matplotlib: 3.1.3
cartopy: 0.17.0
seaborn: 0.10.1
numbagg: None
setuptools: 46.2.0.post20200511
pip: 20.0.2
conda: 4.8.3
pytest: None
IPython: 7.13.0
sphinx: 3.0.3

Thanks @pratiman-91 . Our goal with the plotting functions to enable easy visualization of datasets, not fully complete scientific plots. For this reason, labelling subplots seems out of scope to me.
I recommend looking at proplot: https://proplot.readthedocs.io/en/latest/subplots.html#A-b-c-subplot-labels though I have not used it myself. I personally use a utility function that looks a lot like what you came up with.

Thanks a lot for the information.


Change Mehow's BIO

New bio for Mehow
"When I was younger I worked as a consultant hacking corporate machines, if you think your electronics and financial systems are secure, do I have news for you. Now I’m building technology that eliminates theft and fraud so that these issues can never happen again."
Tim feels as though the old logo:
"When I was younger I worked as a consultant hacking voting machines, if you think your elections are secure, do I have news for you. Now I’m building technology that eliminates theft and fraud so that these issues can never happen again."
Is too political

Updated


fix(wit): remove extra spaces between keywords and generic

From the format "spec" there are no spaces between keywords and generics.  E.g. list<u8>.
If this was a personal preference then perhaps it's worth adding another CLI option.

@bnjjj I couldn't figure out why this seemingly small change changed the order in the example wit file.  While this might not be an issue for non-human uses, it would affect the generated documentation order which would be very odd for the end user.

Yes no worries for the order. It looks good to me ! Thank you


(WIP) first stab at a modal component

Coverage decreased (-0.4%) to 98.077% when pulling a099b0e29b2bb0bb7ed26cb17c5d12a38bdcc1d9 on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage decreased (-10.0%) to 88.525% when pulling d5dd2fed08dc437c9d7e63a62db3fba9b5a6cecd on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage decreased (-9.08%) to 89.431% when pulling 3dad7f74bdbe5557f37d0c7f805d5d97c3d257e0 on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage decreased (-9.08%) to 89.431% when pulling 172003c6b45078f315fdacc616c2626236f15e36 on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage decreased (-13.07%) to 85.441% when pulling 7471c2aaee808f3ed7cba56da2c98714903e39e2 on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage decreased (-13.07%) to 85.441% when pulling 0d59bec30074da3b8befa69b84e5fa2fce730f4d on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage decreased (-12.3%) to 86.166% when pulling 9c8a6a0dfd97a978c15e1af57f323ba56c37beb1 on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

@arizzitano I think this is good for a look when you have some time.
TODO: Figure out the Warning: Failed prop type: Invalid prop `buttons[0]` supplied to `Modal`. in Modal warning in Storybook.
TODO: Write tests.
TODO: Accessibility review.
TODO: More?

Coverage decreased (-13.0%) to 85.547% when pulling c57146dc3b64390ed4b3c3b6e6c1cf4ff5c7dc6f on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage decreased (-0.07%) to 98.438% when pulling 440471cc9ff95375d4f57939b123096ea69b93ec on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage decreased (-0.07%) to 98.438% when pulling 8dce90f181d2ad86cc784a72b6b9293d23a32a12 on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

I'm not sure what's going on with the Button coverage?

Coverage decreased (-0.07%) to 98.438% when pulling 2b57fe2ef685d34ce85f92b87fd89c6cda164149 on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage increased (+0.7%) to 99.219% when pulling 7930f8a5d64530815dd1709152eb696a6c87681a on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage increased (+0.7%) to 99.24% when pulling 2dc96e9f40861e5755d2d0fbe181b4f8142581ed on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage increased (+0.7%) to 99.24% when pulling 068865de4c8b9ed31c76021cc42e6901a7644527 on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

@arizzitano This should be good for a re-review when you have a chance. I believe I've addressed all of your comments.

Coverage increased (+0.7%) to 99.24% when pulling 2aa51307f015b57d9849081cbab18ac8e0c7a16a on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.

Coverage increased (+0.7%) to 99.24% when pulling cb7d0048d242d62e6b44d255c815a0ff41c65b8f on mroytman/modal-component into 35d2bf83e66a1fee0c3a3956fb164acda0466b08 on master.


Question about the hyperparamter used to train the model

Hi, first of all, congratulations on your work accepted to ICML!
I have a few questions about the code details to reproduce your reported results.
First, I have found some mismatches between the code and the paper.

It is written that \mathcal{L}_{lowlevel} is used with \alpha_2 of 0.016 in your paper. However, I have found that --no_blurry_recon argument is used in the accel.slurm file and the value of blur_scale is 0.5 in the file.
It is written that \alpha_1 value is 0.033 but the value of clip_scale is 1 in the accel.slurm file.
There is no scalar for \mathcal{L}_{prior} but the value of prior_scale is 30 in the accel.slurm file.
There is no scalar for \mathcal{L}_{softclip}(VIC, \hat VIC) but I have found that the loss is multiplied by 0.1

            if blurry_recon:
                image_enc_pred, transformer_feats = blurry_image_enc_

                image_enc = autoenc.encode(2*image-1).latent_dist.mode() * 0.18215
                loss_blurry = l1(image_enc_pred, image_enc)
                loss_blurry_total += loss_blurry.item()

                if epoch < int(mixup_pct * num_epochs):
                    image_enc_shuf = image_enc[perm]
                    betas_shape = [-1] + [1]*(len(image_enc.shape)-1)
                    image_enc[select] = image_enc[select] * betas[select].reshape(*betas_shape) + \
                        image_enc_shuf[select] * (1 - betas[select]).reshape(*betas_shape)

                image_norm = (image - mean)/std
                image_aug = (blur_augs(image) - mean)/std
                _, cnx_embeds = cnx(image_norm)
                _, cnx_aug_embeds = cnx(image_aug)

                cont_loss = utils.soft_cont_loss(
                    nn.functional.normalize(transformer_feats.reshape(-1, transformer_feats.shape[-1]), dim=-1),
                    nn.functional.normalize(cnx_embeds.reshape(-1, cnx_embeds.shape[-1]), dim=-1),
                    nn.functional.normalize(cnx_aug_embeds.reshape(-1, cnx_embeds.shape[-1]), dim=-1),
                    temp=0.2)
                loss_blurry_cont_total += cont_loss.item()

                loss += (loss_blurry + 0.1*cont_loss) * blur_scale #/.18215

So, I wonder which one should I have to use to reproduce your results.
Secondly, I wonder if you adjusted learning rate or epochs to compensate the reduced iterations when using multiple gpus.
For example, it says that the batch_size is equal to 24 for single subject training in the accel.slurm file, but is it the same for the 4gpus? (in other words, is the global batch size equal to 24*4=96 for 4gpus?)
In addition, the value of max_lr is 3e-4, but does it multiplied by 4 for the 4gpus?
Thank you in advance.
Juhyeon Park

Hi Juhyeon thanks for the interest in our work!
The alphas you are referring to were rewritten in the paper to be relative to the largest scale value. That is, prior_scale = 30 blur_scale = 0.5 and clip_scale = 1.0 are correct but for writing the paper we divided all these numbers by the highest value of 30. If you change the scales to prior_scale = 30/30, blur_scale = .5/30, clip_scale = 1/30 you'll get the numbers as reflected in the manuscript.

There is no scalar for \mathcal{L}_{softclip}(VIC, \hat VIC) but I have found that the loss is multiplied by 0.1
This is correct, we neglected to mention in the manuscript that we multiplied this loss by 0.1.

The code in the github reflects what was used to train the model, so if you are looking to replicate I'd advise to follow the code as opposed to the manuscript equations.

Secondly, I wonder if you adjusted learning rate or epochs to compensate the reduced iterations when using multiple gpus.
For example, it says that the batch_size is equal to 24 for single subject training in the accel.slurm file, but is it the same for the 4gpus? (in other words, is the global batch size equal to 24*4=96 for 4gpus?)
In addition, the value of max_lr is 3e-4, but does it multiplied by 4 for the 4gpus?

The code does adjust the batch size and number of epochs depending on the number of gpus, but it does not adjust learning rate.

batch_size = int(os.environ["GLOBAL_BATCH_SIZE"]) // num_devices

Thank you for your response.
If possible, could you share the script file you used to train the final model (trained with 8 A100 gpus) for the reported values in the paper?
I'm struggling to reproduce your results, so it would be very helpful to me.
Thank you.
Juhyeon Park

What part of reproducing are you finding problems with? We have already had collaborators use our code to successfully reproduce our results, so I believe the code on the GitHub should be in working order. It has only been minimally modified from the actual original code we used for the paper.
For the exact original code you can attempt to browse through my public wandb runs:

I'm facing a problem that the performance degrades when I train a model with multiple gpus.
I have setup the configurations of accelerator as you did.
I will check your shared wandb log and get back to you.
I appreciate your help!

OK yes please let me know if you find a problem where multi-gpu is leading to performance problems!

Well, I cloned your repository and tried to train a model for subject, w/o pretrained model, with 1 session of data.
The training works with 1 gpu setting, but the process of training stucks when multiple gpus are used.
Could you check this problem?
Thank you.

@PaulScotti
I trained a model with 1 session of subj1 data without pretrained model.
final_subj01_notpretrained_1sess_24bs is trained with a single gpu, final_subj01_notpretrained_1sees_24bs_2gpu is trained with 2 gpus and batch size of 24 (i.e. global batch size is equal to 48) and
final_subj01_notpretrained_1sees_12bs_2gpu is trained with 2gpus and batch size of 12 (i.e. global batch size is equal to 24)
I used learning rate of 3e-4 for training the models.
The test log is as follows.

It seems that all of test metrics degrades when multiple gpus are used.

I see I was thinking you were referring to slight performance degradation with multi-gpu but these are huge differences!
I think you must have some sort of issue in your multi-gpu setup in that case, you can take a look as some of my public wandb runs and see that running the model with 1 sess and 8 gpus trains fine -- https://stability.wandb.io/paul-scotti/mindeye/runs/subj01_pretrained_1sess_1024hid_nolow_seed8
I'd advise you do some debugging with your setup to ensure every batch is indeed being handled properly by each gpu in the multi-gpu case, also look into ensuring your accelerate config is properly configured as discussed in the readme