CI/CD --> 'm365' is not recognized as an internal or external command, operable program or batch file.

Hi @AkashSatoliya, I created a separate issue for this as the comment trails might overlap otherwise.

Hi @martinlingstuyl,
I am also getting the same error. I am trying to create CD pipeline in Azure DevOps to deploy sppkg to SharePoint. I was getting error "The system cannot find the file specified." when running task to login to Sharepoint.
When I tried to login using m365 cli on one of my VM I was getting the same error:
m365 login --authType password --userName<EMAIL_ADDRESS>--password ***
The system cannot find the file specified.

When I ran "npm view @pnp/cli-microsoft365" to check details on m365 CLI, I got below output:
C:\Users\User>npm view @pnp/cli-microsoft365

<EMAIL_ADDRESS>| MIT | deps: 24 | versions: 488
Manage Microsoft 365 and SharePoint Framework projects on any platform
https://pnp.github.io/cli-microsoft365/

keywords: office 365, microsoft 365, sharepoint framework, o365, m365, spfx, sha
repoint online, sharepoint, microsoft teams, microsoft graph, microsoft flow, az
ure active directory, azure ad, azure, microsoft

bin: m365, m365_comp, microsoft365

But When I run "m365 version" I'm getting below error:
C:\Windows\system32>m365 version
'm365' is not recognized as an internal or external command,
operable program or batch file.

Nodejs path(C:\Program Files\nodejs) was already included in PATH variable.
nodejs version: V14.15.0
Operating system: Windows
Shell: PowerShell
Originally posted by @AkashSatoliya in https://github.com/pnp/cli-microsoft365/issues/3565#issuecomment-1223752643

Hi @akashsatoliya,
Could you show us the relevant parts of your yaml file?
Also: are you using the CLI for Microsoft 365 action?

Hi @martinlingstuyl,
Thank you so much for looking into it.
I am using classic pipeline for CD release. I have added tasks mentioned in this document.
Tasks are as follows:

Node.js tool installer, v12.22.1
npm install Microsoft 365 CLI:
install -g @pnp/cli-microsoft365
Command line to connect to SPO:
m365 login -t password -u $(username) -p $(password)
Command line to Add the solution to SP App Catalog:
m365 spo app add -p $(System.DefaultWorkingDirectory)/D:\a\r1\a\_Application_CI\drop\App_Folder1\App_Folder2\sharepoint\solution\Package_Name.sppkg
Command line to Deploy SP App:
m365 spo app deploy --name Package_Name.sppkg

Variables defined are :

catalogsite: sites/appcatalog
username
password
tenant

I have tried running CD pipeline using MS hosted Windows and Ubuntu agents. The deployment tasks runs successfully till step 2 i.e. npm install Microsoft 365 CLI. I am getting error for step 3, i.e. Command line to connect to SPO.
The error is as below:

========================== Starting Command Output ===========================
##[debug]Entering Invoke-VstsTool.
##[debug] Arguments: '/D /E:ON /V:OFF /S /C "CALL "D:\a_temp\032fe971-b3c4-4662-b7dc-544dd7b568d1.cmd""'
##[debug] FileName: 'C:\Windows\system32\cmd.exe'
##[debug] WorkingDirectory: 'D:\a\r1\a'
##[command]"C:\Windows\system32\cmd.exe" /D /E:ON /V:OFF /S /C "CALL "D:\a_temp\032fe971-b3c4-4662-b7dc-544dd7b568d1.cmd""
The system cannot find the file specified.
##[debug]Exit code: 1
##[debug]Leaving Invoke-VstsTool.
##[error]Cmd.exe exited with code '1'.

I am not using CLI for Microsoft 365 action. I have configured only steps mentioned in the document.

The line
install -g @pnp/cli-microsoft365
should be:
npm install -g @pnp/cli-microsoft365
Could you please try it and see if it makes any difference?

Hi @waldekmastykarz,
Thank you so much for your suggestion. As suggested, I tried npm install -g @pnp/cli-microsoft365 but it failed while installing Microsoft 365 CLI in step 2 with below error:

##[warning]Couldn't find a debug log in the cache or working directory
##[error]Error: Npm failed with return code: 1

Thank you for trying it and such a quick reply. Is there a way to get more detailed output log of this step that could help us understand what's wrong? Could you also share with us the configuration of this in your pipe? Thank you!

Hi @waldekmastykarz,
I am using classic pipeline for CD release.
Below are the task configurations:

Node.js tool installer, v12.22.1
npm task with Custom command to install Microsoft 365 CLI:
npm install -g @pnp/cli-microsoft365
Command line to connect to SPO:
m365 login -t password -u $(username) -p $(password)
Command line to Add the solution to SP App Catalog:
m365 spo app add -p $(System.DefaultWorkingDirectory)/D:\a\r1\a\_Application_CI\drop\Folder1\Folder2\sharepoint\solution\Package_Name.sppkg
Command line to Deploy SP App:
m365 spo app deploy --name Package_Name.sppkg

Variables defined are :

catalogsite: sites/appcatalog
username
password
tenant

Detailed log for step 2 running command npm install -g @pnp/cli-microsoft365 is too long(450+ lines). Is it ok to post entire logs here?

Hi @AkashSatoliya,
Okay, rollback to the step where the install completed successfully,
Could you run the following commands in a cmd task on a windows pipeline and see what the output is?
Check if the CLI is in fact installed:
npm list -g --depth 0

Check the path where global modules are installed and verify if that path is in the PATH variable
npm root -g

echo %PATH%

Hi @martinlingstuyl,
Thanks for reply.
Below are the updates for shared command:

Check if the CLI is in fact installed:

    npm list -g --depth 0
<EMAIL_ADDRESS>

Path where global modules are installed

npm root -g
C:\npm\prefix\node_modules

Verify if that path is in the PATH variable. Global module path "C:\npm\prefix\node_modules" is not available in PATH variable.

echo %PATH%

Okay,
That's the reason. It cannot find the m365 executable, so it cannot execute its commands.
I find that odd though. You are running on GitHub? Azure DevOps? Your own host? Hosted vm?
The workaround is to add the global npm folder to your path. You can do this using the following statement:
set PATH=%PATH%;c:\etc

This adds the directory to the path in the current session and should cause m365 to be reachable.

Hi @martinlingstuyl
I am running it on Azure DevOps Microsoft hosted agent.
I tried above command but it adds npm path to current task only. When next task is run path is not available in path variable.
I tried adding npm path to system path variable using below PowerShell and setx command but they also saved it to current task only, not to the system. When next task is run value is not available.

PowerShell

$PATH = [Environment]::GetEnvironmentVariable("PATH")
$npm_path = "C:\npm\prefix\node_modules"
$NewPath = "$(PATH);$npm_path"
[Environment]::SetEnvironmentVariable("PATH", $NewPath)

-Setx command
setx /M path "%path%;C:\npm\prefix\node_modules

Hi @AkashSatoliya,
What's the content of your $PATH?
Maybe some node.js folder is in it.
In that case you could change the place where the npm install globally installs the packages to something already in the PATH. You can do this using an environmentvariable npm_config_cache, or using the --prefix option
npm install --prefix <path/to/prefix_folder> -g

You could also install the cli package locally (npm install @pnp/cli-microsoft365) and run the executable something like this:
./node_modules/.bin/m365 login

@martinlingstuyl,
Thank you for your suggestion.
I have tried running command with PowerShell instead of command tasks.
$PATH = [Environment]::GetEnvironmentVariable("PATH")
$npm_path = "C:\npm\prefix\node_modules"
[Environment]::SetEnvironmentVariable("PATH", $NewPath)
Write-Output $Env:PATH

m365 login -t password -u ${username} -p ${password}

With above script I'm not getting error for m365 command. It attempts to login but fails with below error
Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token

Hi @AkashSatoliya,
Okay, great that the command is now being executed at least.
Did you try running the login command on your local pc? Do you have the same issues there?

Hi @martinlingstuyl,
I tried to run login command on local but there I am still getting 'm365' is not recognized as an internal or external command, operable program or batch file. error. Even though npm module folder path  C:\npm\prefix\node_modules  is already there in system PATH variable.

Hi @martinlingstuyl,
I tried installing version 5.5.0 and latest version using install -g @pnp/cli-microsoft365@latest  and run the command but still it fails with same network_error 400

I'm not sure, but is it possible that the syntax ${password} is incorrect when you want to reference a pipeline variable in a powershell task? I believe it should be $(password)

Hi @martinlingstuyl,
When trying with $(password) it results in variable reference error. PowerShell works with ${password} and $password.
When tried with ${password} and $password it tries to login and fails with network error.

Ok, so just to be sure: did you try this on your local pc as well? Signin In using username and password? Same problem there?

Hi @martinlingstuyl,
I tried to run login command on local but there I am still getting 'm365' is not recognized as an internal or external command, operable program or batch file. error. Even though npm module folder path C:\npm\prefix\node_modules is already there in system PATH variable.

Hi @AkashSatoliya,
On your local machine, could you run the following script to check if CLI is installed, if so where, and if that directory is on the path?
Same as before... If the directory is not on the path, add it.
npm list -g --depth 0

npm root -g

write-host $Env:PATH

When trying with $(password) it results in variable reference error.

@AkashSatoliya, reading this again. I think this means that the variable $(password) might be correctly written, but is not available.
Did you read the documentation on setting variables in a pipeline?: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=classic%2Cbatch#set-variables-in-pipeline

Hi @martinlingstuyl ,
Thank you so much for reply.
I've already defined the password variable in variables section of the pipeline.
When I tried to run the pipeline with m365 login -t password -u "$(username)" -p "$(password)", it throws below error:
Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token

When tried to manually reach https://login.microsoftonline.com/organizations/oauth2/v2.0/token, I'm getting below error
AADSTS900561: The endpoint only accepts POST, OPTIONS requests. Received a GET request.

Is there any change required in the URL?
Also could you please suggest if we can use an account, using which we can manually deploy to SharePoint site, but it doesn't have the administrative permissions on the tenant.

Hi @martinlingstuyl ,
I tried to run the m365 login command on my system in PowerShell and it failed with same Network error. It looks like it might be a bug with special character in password. A part of password is highlighted like a variable. When debugged the script, the part of password which is like a variable it empty. Hence the value of password variable goes like incorrect password.

Hi @martinlingstuyl
I tried with new password which does not include special character used to define variable in PowerShell but still I'm getting same network error.

Is it possible you have a $ in your password? You could try escaping that:
"mypass`$word"

If that solves it locally, it might not solve it in the pipeline. (Do try though) if it does not you could try updating your password.

By the way: that's the backtick "`" that you need to use to escape dollar chars in posh. It's not a single quote.

Hi @martinlingstuyl
My previous password included $ but new password doesn't have $ in it. However it includes & and anything after & gets excluded when running 'm365 login' command
When I tried to run the login command in command prompt I am getting below error:
Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token
'XXxxXX' is not recognized as an internal or external command, operable program or batch file.

Part "XXxxXX" is after & in password which gets excluded and treated as command.

Ok, try escaping the & with the backtick, like I showed with the dollar sign. It should work locally.

Ah wait, no, escaping the ampersand is not necessary in PowerShell.
I've just tried it myself and I can sign in successfully using a password with ampersand.

I tried on PowerShell 5, 7 and windows cmd. They all work fine on my end.

I tried on PowerShell 5, 7 and windows cmd 6.3.9600, running on windows server 2012.
When I tried with escaping ampersand it results in same network error.
When tried to provide password within double quotes it results in same network error.
When tried to provide password without double quotes it results in  'XXxxXX' is not recognized as an internal or external command, operable program or batch file.

When tried to provide password within double quotes it results in same network error.

Just to be clear: you should indeed use quotes around the password
So just to be sure:
You are now running CLI 5.7 and have a password without any special chars?
By the way: The network error also occurs if you make a mistake in writing your password or upn. We do not have a nice 'credentials invalid' error message here, it appears.

I am using PowerShell version 5 and 7(tried on both).
Password includes special characters.
UPN and Passwords are correct as I'm using the same credentials for login and deployment using browser.

You are now running CLI 5.7 and have a password without any special chars?

What is the output when you run m365 version?

Hi @martinlingstuyl
Thanks for reply.
I tried with another account using method like m365 login -t password -u<EMAIL_ADDRESS>-p "zxc098!@#$%^&*()" and it resulted in same network error.
When tried to login with m365 login, it went to next step and showed message to open the page https://microsoft.com/devicelogin and enter a code for device login. After entering code it needs access resources in organization that only an admin can grant.
As I don't have admin I'll request admin to grant permission to the account.
After granting device access, can we use the same account to login using pipeline?

Ok, we've found the problem:
You need to consent to the PnP application before being able to use the m365 login -t password.
I've just gone through our documentation, and I can find this exactly nowhere. So I'll create an issue for us to update the documentation about in what cases you can expect this error and that consent is necessary.
Thanks for getting to the bottom with me on this and sorry for all the trouble!

As I don't have admin I'll request admin to grant permission to the account.

So if you take this step, you should be able to connect afterwards. Also: MFA should be disabled on your account, but I assume you guessed that.

Hi @martinlingstuyl
Thank you so much for looking into it. I've requested for permissions. I'll update once permission is granted and I'm able to test it.

Hi @martinlingstuyl
Finally able to move one step ahead but still getting same error.
After running 'm365 login' command and feeding the code to https://microsoft.com/devicelogin, I directly got below message instead of asking for permissions as shown in screenshot on https://pnp.github.io/cli-microsoft365/user-guide/connecting-office-365/#log-in-using-the-default-device-code-flow

PnP Management Shell
You have signed in to the PnP Management Shell application on your device. You may now close this window.

I'm not sure if consent happened appropriately as it didn't show any list of permissions and directly gave above message.
As mentioned in message, I closed the window and tried to login with method like m365 login -t password -u<EMAIL_ADDRESS>-p "zxc098!@#$%^&*()[]{}" but I'm still getting same error
Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token

I tried with a simple password without any special characters as well but still getting the same error.

Hi @AkashSatoliya, just to be sure: MFA is disabled on that account?

Hi @martinlingstuyl ,
Yes, MFA has already been disabled for this account.

Hi @martinlingstuyl,
Any luck with this error?
Also I wonder if consent is completed appropriately as it didn't show the list of permissions. Is it expected behavior?
Thanks.

Hi @AkashSatoliya, if consent is completed you do not get another message about it, so yes, that's expected behavior.
Tell me: if you run m365 login (the regular interactive sign in) and use the account you were using on the password type signin, does that work? Can you sign in successfully and execute commands?

Hi @martinlingstuyl
Thanks for reply. Yes, I'm able to run 'm365 login' command interactively using the account I was trying to login using username and password. After device login I'm able to run commands.
For example, I was able to run below commands successfully:
m365 aad user get --id "@meId"
m365 spo site get --url https://contoso.sharepoint.com/sites/contoso

But it fails when I try to login using same account using m365 login -t password -u<EMAIL_ADDRESS>-p "zxc098!@#$%^&*()[]{}" method.

Hi @martinlingstuyl
Is there any specific requirement for running login command with username and password?
I tried to run same command in Azure CD pipeline but it results in same error.

Hi @AkashSatoliya, OK, good to hear it is in fact running using the interactive way.
My recent research has showed that the username/password login flow can return this error in the following scenario's:

Your username or password is incorrectly written.
MFA is enabled
Consent has not been given.

I also got this error just now because I had accidentally written -userName  instead of --userName. (missing one dash) The validation did not throw for a missing userName, which it does when just omitting the userName option entirely.
There might be more scenario's though...

Hi @martinlingstuyl
Thanks for reverting with suggestion.

Username and passwords are correct as I logged in with same one in interactive login. I tried with m365 login --authType password --userName<EMAIL_ADDRESS>--password "zxc098!@#$%^&*()[]{}" method as well. I tried with exact username case as well but still result is same.
MFA is already disabled as the account I'm using is a service account.
As updated  earlier, I believe consent is already provided. After Consent I'm able to login interactively.

As you mentioned there might be more scenario's, do we need to setup something in network end?

Another thought: I'm assuming you are using the UPN, and not the emailaddress in case UPN and e-mailaddress are not exactly the same?

Hi @martinlingstuyl,
The Earlier account was AD account. I'm not sure if pass through auth is enable or not and any changes would require different set of approvals. Hence I have requested new AAD account. Awaiting for new AAD account.

Fantastic @AkashSatoliya, we'll wait for it.

Any updates on this @AkashSatoliya?

Hi @martinlingstuyl ,
Still awaiting for account. Received an updated that I'll get it in couple of days.

Hi @martinlingstuyl,
Finally, I've received new account.
The new account ia an AAD account,  MFA disabled and has admin permissions.
I tried for device login using m365 login command. It gave a code which I entered in https://microsoft.com/devicelogin.
After that I tried to login using command m365 login -t password -u<EMAIL_ADDRESS>-p "&zxc098!@#$%^*()" in both PowerShell and command prompt. This resulted in below error:

Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token
'zxc098!@#$%^*()' is not recognized as an internal or external command, operable program or batch file.

The password includes & hence I tried adding back tick before & in password like `&zxc098!@#$%^*(). But it still resulted in error

Well, that's not the message we were hoping for 😂
There must be some other situation we've not covered. Let me think on this a bit.
I'm really sorry you're having such troubles with this!

I was really optimistic that I would be able to log in with my new account, which satisfies all requirements. The error was the last thing I expected to encounter, yet it was the first thing I encountered.😂
Is there anything new on this?

Yeah, same here.
I've re-read the list of possibilities. We're concerned with the Resource Owner Password Flow here:
https://learn.microsoft.com/en-us/azure/active-directory/develop/v2-oauth-ropc
I think we covered all areas there. The password should not contain a trailing or starting space, but I don't suspect it's that easy?

The password doesn't contain any trailing or starting space.
I again checked with admin for current permission level for new account. It is as follows:

Full Control, Limited Access
Read, Limited Access
Contribute, Limited Access

When I tried the password as '"&zxc098!@#$%^*()"' using single inverted comma before and after " I'm getting below error

Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token

After using single inverted comma, I'm not getting internal/external command related error.

Hi @AkashSatoliya,
It doesn't matter what permissions the account has when signing in. You should be able to sign in with any Azure AD account, regardless of permissions, as long as the consent is taken care of and MFA is disabled.

When I tried the password as '"&zxc098!@#$%^*()"' using single inverted comma before and after " I'm getting below error

You would need to write it as follows: '&zxc098!@#$%^*()'.
It's a verbatim PowerShell string when using single quotes. You don't need the double quotes in there.

After using single inverted comma, I'm not getting internal/external command related error.

I'm not sure what you mean with this?

Hi @martinlingstuyl,
I'm not sure why string interpretation is incorrectly happening with m365 command. I tried same with Write-Host, substring, etc. and it works as usual.

PS C:\Users\User1> m365 login -t password -u<EMAIL_ADDRESS>-p "&1zxc098!@#$%^()"
Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token
'1zxc098!@#$%^()' is not recognized as an internal or external command,operable program or batch file.
PS C:\Users\User1> Write-Host "&1zxc098!@#$%^()"
&1zxc098!@#$%^()
PS C:\Users\User1>$Pass = "&1zxc098!@#$%^()"
PS C:\Users\User1> Write-Host $Pass
&1zxc098!@#$%^()
PS C:\Users\User1> $Pass.Substring(1,5)
1zxc0

I'm not sure what is causing it to treat the string as a command while using m365 cli. I tried in both PowerShell and cmd.exe.

I tried converting password to a secure string and now I'm not getting the below error:

'1zxc098!@#$%^()' is not recognized as an internal or external command,operable program or batch file.

But still getting this error:

PS C:\Users\User1>$Pass =  ConvertTo-SecureString -AsPlainText '&1zxc098!@#$%^()' -Force
PS C:\Users\User1>m365 login -t password -u<EMAIL_ADDRESS>-p $Pass
Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token

Hi @AkashSatoliya, I don't think you can use a SecureString. that is a .NET object, while the CLI for Microsoft 365 is a Node.JS app.
You should just add the password as a regular string (with single quotes or double quotes should both work the same)

Hi @martinlingstuyl,
I'm not sure why string interpretation is incorrectly happening with m365 command. I tried same with Write-Host, substring, etc. and it works as usual.

PS C:\Users\User1> m365 login -t password -u<EMAIL_ADDRESS>-p "&1zxc098!@#$%^()"
Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token
'1zxc098!@#$%^()' is not recognized as an internal or external command,operable program or batch file.
PS C:\Users\User1> Write-Host "&1zxc098!@#$%^()"
&1zxc098!@#$%^()
PS C:\Users\User1>$Pass = "&1zxc098!@#$%^()"
PS C:\Users\User1> Write-Host $Pass
&1zxc098!@#$%^()
PS C:\Users\User1> $Pass.Substring(1,5)
1zxc0

I'm not sure what is causing it to treat the string as a command while using m365 cli. I tried in both PowerShell and cmd.exe.

By the way, is this literally the error you get when running the command with<EMAIL_ADDRESS>and the mentioned password?
I just tried it literally as well, but it just works on my end (except that it won't login because the account is nonexistent)
works on both PowerShell 7.2 and 5:

Hi @martinlingstuyl ,
I'm getting this error when tried with both<EMAIL_ADDRESS>and actual service account credentials.

Can you try running the following?
m365 login -t password -u<EMAIL_ADDRESS>-p '&1zxc098!@#$%^()' --debug

It should give you an extra log line on top, like this:
Executing command login with options {"options":{"authType":"password","userName":"abc@domain.com","password":"&1zxc098!@#$%^()","debug":true,"output":"json"}}

The value in password is what is received by the m365 application. On my end it reads exactly like I'm pasting above here.

Hi @martinlingstuyl ,
I'm still getting the same error even with --debug option

Hmm, I'm not sure why, but I think you're not using the correct quote symbols here. PowerShell seems to not see your value as a string.

What is the code page in use? I think you should be able to query that using one of the following:
[System.Text.Encoding]::Default.EncodingName

chcp

[Console]::InputEncoding

If I may chime in, @AkashSatoliya, could you try running:
m365 login -t password -u<EMAIL_ADDRESS>-p "&1zxc098!@#$%^()" --debug

I saw you tried running it before, but like @martinlingstuyl mentioned, with the --debug flag we should get more information about what's actually being sent over the wire. Last time you tried it with single quotes you got a different error, so perhaps we should get back to double quotes and see if that gets us any closer.

I do think you might need to escape (at least) the dollar sign with a backtick:
`$

Hi @waldekmastykarz and @martinlingstuyl ,
Thank you so much for your reply.
I tried both the command in both ways, like with double quotes with --debug option and single quotes with --debug option and both double and single quotes without --debug option. In all the cases result remains same:

PS C:\Users> m365 login -t password -u<EMAIL_ADDRESS>-p "&1zxc098!@#$%^()" --debug
Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attemptedto reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token '1zxc098!@#$%()' is not recognized as an internal or external command, operable program or batch file.
PS C:\Users> m365 login -t password -u<EMAIL_ADDRESS>-p '&1zxc098!@#$%^()' --debug
Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token '1zxc098!@#$%()' is not recognized as an internal or external command, operable program or batch file.
PS C:\Users> m365 login -t password -u<EMAIL_ADDRESS>-p "&1zxc098!@#`$ %^()"
Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token '1zxc098!@#$%()' is not recognized as an internal or external command, operable program or batch file.
PS C:\Users> m365 login -t password -u<EMAIL_ADDRESS>-p '&1zxc098!@#$ %^()' Error: network_error: Network request failed. Please check network trace to determine root cause. | Fetch client threw: Error: HTTP status code 400 | Attempted to reach: https://login.microsoftonline.com/organizations/oauth2/v2.0/token'1zxc098!@#$%()' is not recognized as an internal or external command, operable program or batch file.
PS C:\Users>

I have below questions:
Do we need any specific character Encoding?
Access to any specific port need to be enabled? Anything need to check from network end?
Not sure why every time it is asking for network error?

Hi @AkashSatoliya,
As far as I understand, this is not an issue with the CLI for Microsoft 365, but with PowerShell and possibly the input encoding you are using.
PowerShell throws '1zxc098!@#$%()' is not recognized as an internal or external command, operable program or batch file.
PowerShell is apparently interpreting your string password as a command. Possibly it sees the first character & as a reserved character. You can see this because the & is excluded in the string when it says '1zxc098!@#$%()' is not recognized... Which is odd though, it does not do so on my end with & characters.
Could you retry and escape the & character with a backtick?
m365 login -t password -u<EMAIL_ADDRESS>-p "`&1zxc098!@#`$%^()" --debug

Please copy paste the return exception if it still errors.
Also: Your original intent was to use this in a ci/cd pipeline, wasn't it? You could try if that works now, as you're now using a cloud account, it's possible the pipeline DOES work as intended.

Hi @martinlingstuyl ,
You are right. There is some issue with PowerShell. When I tried to login using cmd I am able to login.
I tried in ci/cd pipeline as well but there, even though pipeline was showing as successfully completed. But when I checked the log I am getting below error:

Error: invalid_grant: AADSTS50079: Due to a configuration change made by your administrator, or because you moved to a new location, you must enroll in multi-factor authentication to access 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'.

I believe this must be due to MFA. Let me check admin team internally.
I genuinely appreciate your help in getting this problem resolved.
Thank you so much! :)

Error: invalid_grant: AADSTS50079: Due to a configuration change made by your administrator, or because you moved to a new location, you must enroll in multi-factor authentication to access 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'.

Okay, that's clearly MFA indeed. You'll need to ask your IT admins to exclude this account from any conditional access policy that enforces MFA.

I genuinely appreciate your help in getting this problem resolved.

Of course, 🙂 that's what we're here for. I hope we can resolve this issue ASAP.

Could you retry and escape the & character with a backtick?

Did you try escaping the ampersand?

Okay, that's clearly MFA indeed. You'll need to ask your IT admins to exclude this account from any conditional access policy that enforces MFA.

Yes, I've again requested IT team to exclude this from MFA.

Of course, 🙂 that's what we're here for. I hope we can resolve this issue ASAP.

That's so nice of you  🙂

Did you try escaping the ampersand?

Nope. I simply tried without escaping the ampersand. Like &1zxc098!@#$%^().

Ok, I'm still curious whether that solves it on your local Pc though. Just for future reference. 🤗 who knows, someone else might come up with the same issue.

Hi @martinlingstuyl,
Hope you are doing good.
MFA disabling is still in progress. Team is working on it.
While I'm waiting for response from them I was working commands for next steps like adding sppkg package and perform deployment. While running command to Add the solution to SP App Catalog, I got error 403

C:\Windows\system32>m365 spo app add -p "C:\Users\User1\Downloads\drop (2)\drop\Application\App\sharepoint\solution\AppPkg.sppkg"
Error: Request failed with status code 403

This is because of authorization error. When I again checked for access level I got and update that the account has "Full Control, Limited Access". When working on browser using same account, I'm able to deploy the package. But when trying to use CLI it is throwing above error.
Is it due to this "Full Control, Limited Access" access level?
Thanks

Hi @martinlingstuyl,
Hope you are doing good.
MFA disabling is still in progress. Team is working on it.
While I'm waiting for response from them I was working commands for next steps like adding sppkg package and perform deployment.
I'm able to add the package and deploy it. After deployment I want to copy few files from local drive C:\Users\User1\Downloads\app\temp\  to a folder /sites/Site1/SiteAssets/AppFolder in SharePoint. But I'm getting error Specified path of the file to add does not exist.

C:\Users\User1>m365 spo file add --webUrl https://abc.sharepoint.com/sites/Site1 --folder `/sites/Site1/SiteAssets/AppFolder' --path 'C:\Users\User1\Downloads\app\temp\manifests.js'
Error: Specified path of the file to add does not exist

File is available at C:\Users\User1\Downloads\app\temp\manifests.js but still getting this error.

Hi @AkashSatoliya, I've been at a conference this week. No time to respond. I'll get back to you!

Hi @martinlingstuyl,
Thank you for looking into it.
Thank

m365 spo file add --webUrl https://abc.sharepoint.com/sites/Site1 --folder `/sites/Site1/SiteAssets/AppFolder' --path 'C:\Users\User1\Downloads\app\temp\manifests.js'
Error: Specified path of the file to add does not exist

Hi @AkashSatoliya, it might just be because it's dummy data, but it seems like you are using different types of quotes here: ` and '
Can you check again with proper quote usage? It might just block you there.

Could you navigate to the location of the file in your terminal and execute it like this? You can even remove all quotes and see if that works: (replace path-spaces with %20 though)
m365 spo file add --webUrl https://abc.sharepoint.com/sites/Site1 --folder SiteAssets/AppFolder --path manifests.js

Hi @martinlingstuyl ,
Thank you so much for suggestion. It worked after navigating to the location of the file in the terminal and executing the command using method mentioned above.
I was trying the command like below referring to this document:
m365 spo file add --webUrl https://contoso.sharepoint.com/sites/project-x --folder 'Shared Documents/Sub Folder 1' --path 'C:\MS365.jpg'

Yes, and it should work. Incidentelly, it also works on my machine when using the full path. :-) But something odd is going on with your PC and your use of quotes. I'm not sure what or how.

Hey @martinlingstuyl @AkashSatoliya, do we have any next steps that we could investigate or have we exhausted all options and we close this issue with all the information that we were able to gather for future reference?

MFA disabling is still in progress. Team is working on it.

We got sidetracked, but I think we where waiting on the above to be completed to check if that actually fixed the issue. @AkashSatoliya any updates?

Hi @martinlingstuyl ,
Yes, it is correct. The solution provided is to disable MFA. I've raise request to disable MFA for the account. The MFA cannot be disabled and instead they'll whitelist Microsoft DevOps IP addresses which will not ask for MFA. It is still waiting for security approval.
I'm not sure how much time it will take to get the approval. Hence if required, please close this thread. I'll Update as soon as I get approval and try the solution.

Thank you for the recap. If there are additional things that we're checking, we can keep this thread open. I just want to ensure that it's still active :)

Hi @martinlingstuyl,
Disabling MFA worked. After disabling MFA I'm able to deploy the site and upload files to site assets.
Thank you so much @martinlingstuyl and @waldekmastykarz  for getting to the bottom of the problem and providing its solution.

And thank you for sticking with us and not giving us. I'm sure the information we've got in this thread will help others too.

That's great to hear @AkashSatoliya! It took a while, but we finally got there! Thanks a lot for your patience 👍


Add MAC arm build to bioconda

There was a previous issues requesting MAC M1/M2/M3 support.  That has been closed, and it looks like linux-aarch64 builds have been added but there is no build uploaded for MAC, perhaps because Bioconda did not support this.
It looks like bioconda added MAC ARM support in July. Is it possible to upload an osx-arm AGC build to bioconda? Thanks

Thank you for pointing support of MAC ARM by Bioconda. At the moment, I'm working on a new release of AGC with better speed and ratio (at least in some cases). I need a few weeks to complete. I think this will be an excellent moment to add osx-arm build to Bioconda.

Thanks @sebastiandeorowicz  - looking forward to the new release changes!

AGC 3.1 for MAC ARM is in Bioconda. The recently released AGC 3.2 version will be available soon.


Enable Save and continue on all pages (Phase 1)

Close #549
Affected pages: Upload, Errors, Warnings
Changes

Save and continue always enabled but triggers error display when step is incomplete.

Displayed with secondary styling when step is not complete
Displayed with primary styling when step is complete

Warnings - Displays error state + message in Verification well when appropriate

How to test this PR

See screenshots

Screenshots
Upload
https://github.com/cfpb/sbl-frontend/assets/2592907/838046e8-989c-4a17-951a-f4b1ab60f393
Errors & Warnings
https://github.com/cfpb/sbl-frontend/assets/2592907/716c748f-b6e0-4b73-bea2-91a3f17611f9

Heyo @meissadia, can we lift this out of draft?

I haven't forgotten about this PR, but I'm wary about adding it until the content stuff is done. Can I save it for last?


corr MemoryError

Whenever I am calculating corr on a dask data frame with a larger but not enormous number of columns (200-2000) my worker memory is blowing up and getting  a MemoryError at one of two lines:
from /dask/dataframe/core.py

3885     x = np.where(keep, mat[:, None, :], np.nan)
3886     sums = np.nansum(x, 0)

This is reliably reproduced  with the code below, exact breaking point will obviously vary with worker,  I'm currently using 4 thread, 24GB workers.
arr = da.random.randint(100, size=(10_000, 500), chunks=1_000)
df  = dd.from_array(arr, columns=[f'col_{i}' for i in range(arr.shape[1])])
df.to_parquet(path)
df = dd.read_parquet(path)
corr = df.corr().compute()

Workers are reporting " distributed.worker - WARNING - Memory use is high but worker has no data to store to disk.  Perhaps some other process is leaking memory?  Process memory: 22.55 GB -- Worker memory limit: 24.00 GB"
Dask will break with memory errors at sizes small enough to process the entire calculation in memory with Pandas. I will need to scale to 10's of millions of rows and 1-3 thousand columns so staying in memory doesn't look to a be a good option.

it looks like there are NumPy arrays created that are of shape (n_rows, n_cols, n_cols) for each DataFrame partition. Perhaps this could be the memory bottleneck herecd

That makes sense to me.  My guess is that there is a way to implement this function in smaller memory.

I'm going to mark this as good first issue, not necessarily because it's easy (it's actually quite an interesting problem) but rather because it requires little knowledge of dask itself.  The goal is to reimplement the cov_corr_chunk function in dask/dataframe/core.py in a way that produces the same result, but uses less memory.  This probably requires some good skill with numpy or pandas.


Warning when voting power high

Closes https://github.com/penumbra-zone/web/issues/1892
https://github.com/user-attachments/assets/9c5a2a0f-58e6-41ac-bfc2-c5a06417e379

That sounds great but it could be in a future PR


Should faq.html updated to mentions the creation of a .openjfx directory under the user directory?

Can it be that a .openjfx directory was recently created at my user directory because of sparrowwallet?
I used the suggestions at faq.html#im-running-on-a-high-resolution-screen-and-sparrow-icons-and-text-are-tiny, and faq.html#sparrow-is-too-big-for-my-screen. Which boils to suggesting _JAVA_OPTIONS=-Dglass.gtk.uiScale=.... Does such usage, or maybe any usage of sparrowwallet, would lead to the creation of a .openjfx directory? If the answer is yes, can the faq.html point that out?
Reference: /JabRef/jabref/issues/5817.

Yes, this is standard behaviour in JavaFX. The location of the .openjfx folder (into which is extracted the native libraries JavaFX requires) can be changed, for example
JAVA_OPTIONS=-Djavafx.cachedir=/another/folder /opt/sparrow/bin/Sparrow

It might be possible to include these libraries in the application installation instead, avoiding this. I'll rename and keep this issue open to investigate.

Also see https://bugs.openjdk.org/browse/JDK-8316276


update help

Thanks for the PR. Can you fix the shellcheck warning?  I don’t remember why are there tests in this package, I probably copied the skeleton from some other plugin ;) If there is no way around it (I understand that dokku reads this somehow, but it’s not used in the script itself), just leave a comment and I’ll merge anyway.

What is shellcheck warning? You mean Travis CI? I will check the logs.

All checks have passed.
Thanks,


Resized image missing EXIF

Image taken in portrait mode after resized became improper orientation

fixed look at branch EXIF-support

This branch should be merged immediately. Otherwise people will face unexpected behavior like me in production.


Test Problem

I copy RepPoints src and configs into mmdetection,
and use
python tools/test.py configs/reppoints/reppoints_moment_r101_fpn_2x.py weights/reppoints_moment_r101_fpn_2x.pth --eval bbox --json_out a.json
to test the model in the mmdetection dir.
However, I get an error.

loading annotations into memory...
Done (t=0.06s)
creating index...
index created!
[>>>>>>>>>>>>>>>>>>                                ] 7311/20288, 9.2 task/s, elapsed: 796s, ETA:  1412sTraceback (most recent call last):
File "tools/test.py", line 239, in
main()
File "tools/test.py", line 199, in main
outputs = single_gpu_test(model, data_loader, args.show)
File "tools/test.py", line 32, in single_gpu_test
result, det_time, nms_time = model(return_loss=False, rescale=not show, **data)
File "/home/shengtao/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/shengtao/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/shengtao/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/shengtao/models/RepPoints/mmdet/core/fp16/decorators.py", line 49, in new_func
return old_func(*args, **kwargs)
File "/home/shengtao/models/RepPoints/mmdet/models/detectors/base.py", line 88, in forward
return self.forward_test(img, img_meta, **kwargs)
File "/home/shengtao/models/RepPoints/mmdet/models/detectors/base.py", line 79, in forward_test
return self.simple_test(imgs[0], img_metas[0], **kwargs)
File "/home/shengtao/models/RepPoints/mmdet/models/detectors/single_stage.py", line 66, in simple_test
outs = self.bbox_head(x)
File "/home/shengtao/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/shengtao/models/RepPoints/mmdet/models/anchor_heads/reppoints_head.py", line 291, in forward
return multi_apply(self.forward_single, feats)
File "/home/shengtao/models/RepPoints/mmdet/core/utils/misc.py", line 24, in multi_apply
return tuple(map(list, zip(*map_results)))
File "/home/shengtao/models/RepPoints/mmdet/models/anchor_heads/reppoints_head.py", line 280, in forward_single
self.relu(self.reppoints_cls_conv(cls_feat, dcn_offset)))
File "/home/shengtao/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, kwargs)
File "/home/shengtao/models/RepPoints/mmdet/ops/dcn/deform_conv.py", line 236, in forward
self.dilation, self.groups, self.deformable_groups)
File "/home/shengtao/models/RepPoints/mmdet/ops/dcn/deform_conv.py", line 55, in forward
cur_im2col_step)
RuntimeError: input image is smaller than kernel (shape_check at mmdet/ops/dcn/src/deform_conv_cuda.cpp:127)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x45 (0x7f3f5d088dc5 in /home/shengtao/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: shape_check(at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int) + 0x720 (0x7f3f0e5686a0 in /home/shengtao/models/RepPoints/mmdet/ops/dcn/deform_conv_cuda.cpython-37m-x86_64-linux-gnu.so)
frame #2: deform_conv_forward_cuda(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, int) + 0xf5 (0x7f3f0e569325 in /home/shengtao/models/RepPoints/mmdet/ops/dcn/deform_conv_cuda.cpython-37m-x86_64-linux-gnu.so)
frame #3:  + 0x233f0 (0x7f3f0e57d3f0 in /home/shengtao/models/RepPoints/mmdet/ops/dcn/deform_conv_cuda.cpython-37m-x86_64-linux-gnu.so)
frame #4:  + 0x2354e (0x7f3f0e57d54e in /home/shengtao/models/RepPoints/mmdet/ops/dcn/deform_conv_cuda.cpython-37m-x86_64-linux-gnu.so)
frame #5:  + 0x1ff91 (0x7f3f0e579f91 in /home/shengtao/models/RepPoints/mmdet/ops/dcn/deform_conv_cuda.cpython-37m-x86_64-linux-gnu.so)
frame #11: THPFunction_apply(_object, _object) + 0x691 (0x7f3f830d8891 in /home/shengtao/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)

Who can help me ?
What's more, I find the weight reppoints_moment_r101_dcn_fpn_2x.pth has no params for dcn part.
The code
python tools/test.py configs/reppoints/reppoints_moment_r101_dcn_fpn_2x.py weights/reppoints_moment_r101_dcn_fpn_2x.pth --eval bbox --json_out a.json
gets warnings

loading annotations into memory...
Done (t=0.44s)
creating index...
index created!
The model and loaded state dict do not match exactly
missing keys in source state_dict: backbone.layer3.8.conv2_offset.bias, backbone.layer3.11.conv2_offset.weight, backbone.layer3.5.conv2_offset.weight, backbone.layer3.22.conv2_offset.weight, backbone.layer3.4.conv2_offset.weight, backbone.layer3.10.conv2_offset.weight, backbone.layer3.18.conv2_offset.weight, backbone.layer3.20.conv2_offset.weight, backbone.layer3.17.conv2_offset.weight, backbone.layer4.1.conv2_offset.weight, backbone.layer2.0.conv2_offset.weight, backbone.layer3.13.conv2_offset.weight, backbone.layer3.9.conv2_offset.weight, backbone.layer3.14.conv2_offset.bias, backbone.layer3.16.conv2_offset.bias, backbone.layer3.11.conv2_offset.bias, backbone.layer3.13.conv2_offset.bias, backbone.layer4.2.conv2_offset.weight, backbone.layer3.0.conv2_offset.weight, backbone.layer3.16.conv2_offset.weight, backbone.layer3.0.conv2_offset.bias, backbone.layer3.19.conv2_offset.bias, backbone.layer3.21.conv2_offset.bias, backbone.layer4.0.conv2_offset.weight, backbone.layer3.21.conv2_offset.weight, backbone.layer3.1.conv2_offset.bias, backbone.layer3.6.conv2_offset.weight, backbone.layer3.5.conv2_offset.bias, backbone.layer2.0.conv2_offset.bias, backbone.layer2.2.conv2_offset.bias, backbone.layer3.20.conv2_offset.bias, backbone.layer3.7.conv2_offset.weight, backbone.layer3.12.conv2_offset.bias, backbone.layer3.14.conv2_offset.weight, backbone.layer3.4.conv2_offset.bias, backbone.layer4.1.conv2_offset.bias, backbone.layer3.15.conv2_offset.bias, backbone.layer3.12.conv2_offset.weight, backbone.layer2.3.conv2_offset.weight, backbone.layer3.10.conv2_offset.bias, backbone.layer4.0.conv2_offset.bias, backbone.layer3.7.conv2_offset.bias, backbone.layer3.19.conv2_offset.weight, backbone.layer3.3.conv2_offset.bias, backbone.layer3.8.conv2_offset.weight, backbone.layer2.3.conv2_offset.bias, backbone.layer3.22.conv2_offset.bias, backbone.layer3.9.conv2_offset.bias, backbone.layer2.1.conv2_offset.bias, backbone.layer2.2.conv2_offset.weight, backbone.layer3.17.conv2_offset.bias, backbone.layer3.1.conv2_offset.weight, backbone.layer3.3.conv2_offset.weight, backbone.layer2.1.conv2_offset.weight, backbone.layer3.18.conv2_offset.bias, backbone.layer3.6.conv2_offset.bias, backbone.layer3.2.conv2_offset.bias, backbone.layer3.15.conv2_offset.weight, backbone.layer4.2.conv2_offset.bias, backbone.layer3.2.conv2_offset.weight

but the code
python tools/test.py configs/reppoints/reppoints_moment_r101_fpn_2x.py weights/reppoints_moment_r101_dcn_fpn_2x.pth --eval bbox --json_out a.json
gets nothing.

I'm getting a similar error for training.

https://github.com/open-mmlab/mmdetection/issues/1453#issuecomment-538602013
This might help you.

i

open-mmlab/mmdetection#1453 (comment)
This might help you.

It works for me. Thx :)


update Synapse Client dependency in order to update log4j dependency to 2.15.0

update Synapse Client dependency in order to update log4j dependency to 2.15.0
Details: https://sagebionetworks.jira.com/browse/SC-377

please fix build failure.


Add live output option for prediction

This PR adds live-output to prediction if you specify --live_output command line switch.
Let me know if there's anything that should be changed, happy to improve upon it more.

I discussed with Aero the fact that this code can't run on TPUs in the Discord. Closing PR for now while they work on a fix.


Keys as atoms

Rebar and relx templates use atoms as keys. Could mustache convert the data key to an atom if the value isn't found for the string, at least?

Ok.
In mustache, you can use innumerable key, but the number of atom on ErlangVM there is limit.
Therefore, there is no option to support the atom only.
But, it is great that people who use this library can choose their favorite type (string, atom and binary) as keys.
Please wait a little because corresponding to the weekend.

Master has been updated ! (v0.3.0)

Specify the type in key_type (See #4)

# Default
1> mustache:render(<<"{{a}}">>, [{"a", "string"}]).
<<"string">>
2> mustache:render(<<"{{a}}">>, [{a, "atom"}], [{key_type, atom}]).
<<"atom">>
3> mustache:render(<<"{{a}}">>, [{<<"a">>, "binary"}], [{key_type, binary}]).
<<"binary">>

It is not exist conversion for key when key_type is the binary
https://github.com/soranoba/mustache/blob/v0.3.0/src/mustache.erl#L314

Take over the key_type when you use the Lamda expressions.
https://github.com/soranoba/mustache/blob/v0.3.0/src/mustache.erl#L138

All key in assoc list or maps must be same type.
https://github.com/soranoba/mustache/blob/v0.3.0/src/mustache.erl#L71

1> #{"hoge" => "hoge", "fugo" => "fugo"}. % Ok
2> #{"hoge" => "hoge", atom => "atom"} % Bad


Recipe for fine-tuning with lower resolution images

🚀 The feature, motivation and pitch
Is fine-tuning the Vision models at a lower resolution supported? If so, can you please add a recipe for that(or add a note in recipes/quickstart/finetuning/finetune_vision_model.md for how to do that.  I tried setting the size param in processor as
 processor = AutoProcessor.from_pretrained(train_config.model_name if train_config.tokenizer_name is None else train_config.tokenizer_name,
                                          size={"height": 336, "width": 336})

But this results in a tensor size mismatch error
RuntimeError: The size of tensor a (577) must match the size of tensor b (1601) at non-singleton dimension 2

Alternatives
No response
Additional context
No response

@avaradarajanfigma Thanks for your question. I believe the processor will automatically do the resize process as shown here. You do not need to modify any code to get our vision model working with lower resolution images. Of course you can run fine-tuning on you own lower resolution images using finetune_vision_model.md, if you want.  Let me know if you have any questions.

My question is, is there a way to initialize the model/processor in such a way that it doesn't scale up the lower resolution images first(resulting in fewer visual tokens), so that it is possible to fine tune with less memory?

I do not think it is possible to feed lower resolution image into our vision model. (1) Processor will always resize as # do_resize=False is not supported stated here. (2) Even if somehow you bypass the processor, our model still can not take lower resolution image input directly. As for saving memory for fine-tuning, I recommend you use FSDP+peft. Let me know if you have more questions

Thanks. What GPUs were the recipes/quickstart/finetuning/finetune_vision_model.md recipe tested on? I tried on 4 A10 GPUs and it OOMs.

What is your command to run the fine-tune?

I am using the same command as in recipes/quickstart/finetuning/finetune_vision_model.md, except I changed batch size to 1
torchrun --nnodes 1 --nproc_per_node 4  recipes/quickstart/finetuning/finetuning.py --enable_fsdp --lr 1e-5  --num_epochs 3 --batch_size_training 1 --model_name meta-llama/Llama-3.2-11B-Vision-Instruct --dist_checkpoint_root_folder ./finetuned_model --dist_checkpoint_folder fine-tuned  --use_fast_kernels --dataset "custom_dataset" --custom_dataset.test_split "test" --custom_dataset.file "recipes/quickstart/finetuning/datasets/ocrvqa_dataset.py"  --run_validation True --batching_strategy padding  --use_peft --peft_method lora

Can I see the error message? I think Lora should work as estimated from this table.  I was using the H100 to test the fine-tune but I will investigate more about this problem.

Below are the OOM errors. Based on the model sizes in that table - they are not referring to the vision models.
torchrun --nnodes 1 --nproc_per_node 4  recipes/quickstart/finetuning/finetuning.py --enable_fsdp --lr 1e-5  --num_epochs 3 --batch_size_training 1 --model_name meta-llama/Llama-3.2-11B-Vision-Instruct --dist_checkpoint_root_folder ./finetuned_model --dist_checkpoint_folder fine-tuned  --use_fast_kernels --dataset "custom_dataset" --custom_dataset.test_split "test" --custom_dataset.file "recipes/quickstart/finetuning/datasets/ocrvqa_dataset.py"  --run_validation True --batching_strategy padding  --use_peft --peft_method lora
W1008 13:22:01.883000<PHONE_NUMBER>10528 torch/distributed/run.py:757]
W1008 13:22:01.883000<PHONE_NUMBER>10528 torch/distributed/run.py:757] *****************************************
W1008 13:22:01.883000<PHONE_NUMBER>10528 torch/distributed/run.py:757] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W1008 13:22:01.883000<PHONE_NUMBER>10528 torch/distributed/run.py:757] *****************************************
Clearing GPU cache for all ranks
--> Running with torch dist debug set to detail
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00,  6.66it/s]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00,  6.70it/s]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00,  6.47it/s]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00,  6.45it/s]
--> Model meta-llama/Llama-3.2-11B-Vision-Instruct

--> meta-llama/Llama-3.2-11B-Vision-Instruct has 10670.220835 Million params

trainable params: 5,898,240 || all params: 10,676,119,075 || trainable%: 0.0552
bFloat16 enabled for mixed precision - using bfSixteen policy
trainable params: 5,898,240 || all params: 10,676,119,075 || trainable%: 0.0552
trainable params: 5,898,240 || all params: 10,676,119,075 || trainable%: 0.0552
trainable params: 5,898,240 || all params: 10,676,119,075 || trainable%: 0.0552
--> applying fsdp activation checkpointing...
--> applying fsdp activation checkpointing...
--> applying fsdp activation checkpointing...
--> applying fsdp activation checkpointing...
--> Training Set Length = 1800
--> Validation Set Length = 200
length of dataset_train 1800
custom_data_collator is used
--> Num of Training Set Batches loaded = 450
length of dataset_train 1800
custom_data_collator is used
--> Num of Training Set Batches loaded = 450
length of dataset_train 1800
custom_data_collator is used
--> Num of Training Set Batches loaded = 450
--> Num of Validation Set Batches loaded = 50
--> Num of Validation Set Batches loaded = 50
Starting epoch 0/3
train_config.max_train_step: 0
--> Num of Validation Set Batches loaded = 50
--> Num of Validation Set Batches loaded = 50
Starting epoch 0/3
train_config.max_train_step: 0
length of dataset_train 1800
custom_data_collator is used
--> Num of Training Set Batches loaded = 450
--> Num of Validation Set Batches loaded = 50
--> Num of Validation Set Batches loaded = 50
Starting epoch 0/3
train_config.max_train_step: 0
/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/cuda/memory.py:330: FutureWarning: torch.cuda.reset_max_memory_allocated now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.
  warnings.warn(
/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/cuda/memory.py:330: FutureWarning: torch.cuda.reset_max_memory_allocated now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.
  warnings.warn(
Training Epoch: 1:   0%|                                                                                | 0/450 [00:00<?, ?it/s]/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/cuda/memory.py:330: FutureWarning: torch.cuda.reset_max_memory_allocated now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.
  warnings.warn(
Training Epoch: 1:   0%|                                                                                | 0/450 [00:00<?, ?it/s]--> Num of Validation Set Batches loaded = 50
--> Num of Validation Set Batches loaded = 50
Starting epoch 0/3
train_config.max_train_step: 0
/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/cuda/memory.py:330: FutureWarning: torch.cuda.reset_max_memory_allocated now calls torch.cuda.reset_peak_memory_stats, which resets /all/ peak memory stats.
  warnings.warn(
Training Epoch: 1:   0%|                                                                                | 0/450 [00:00<?, ?it/s]/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py:91: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py:91: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py:91: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py:91: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/recipes/quickstart/finetuning/finetuning.py", line 8, in <module>
[rank0]:     fire.Fire(main)
[rank0]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
[rank0]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank0]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
[rank0]:     component, remaining_args = _CallAndUpdateTrace(
[rank0]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
[rank0]:     component = fn(*varargs, **kwargs)
[rank0]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/src/llama_recipes/finetuning.py", line 313, in main
[rank0]:     results = train(
[rank0]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/src/llama_recipes/utils/train_utils.py", line 175, in train
[rank0]:     loss.backward()
[rank0]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/_tensor.py", line 525, in backward
[rank0]:     torch.autograd.backward(
[rank0]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 267, in backward
[rank0]:     _engine_run_backward(
[rank0]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
[rank0]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank0]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 301, in apply
[rank0]:     return user_fn(self, *args)
[rank0]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 320, in backward
[rank0]:     torch.autograd.backward(outputs_with_grad, args_with_grad)
[rank0]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 267, in backward
[rank0]:     _engine_run_backward(
[rank0]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
[rank0]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 19.93 GiB. GPU
[rank2]: Traceback (most recent call last):
[rank2]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/recipes/quickstart/finetuning/finetuning.py", line 8, in <module>
[rank2]:     fire.Fire(main)
[rank2]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
[rank2]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank2]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
[rank2]:     component, remaining_args = _CallAndUpdateTrace(
[rank2]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
[rank2]:     component = fn(*varargs, **kwargs)
[rank2]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/src/llama_recipes/finetuning.py", line 313, in main
[rank2]:     results = train(
[rank2]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/src/llama_recipes/utils/train_utils.py", line 175, in train
[rank2]:     loss.backward()
[rank2]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/_tensor.py", line 525, in backward
[rank2]:     torch.autograd.backward(
[rank2]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 267, in backward
[rank2]:     _engine_run_backward(
[rank2]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
[rank2]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank2]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 301, in apply
[rank2]:     return user_fn(self, *args)
[rank2]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 320, in backward
[rank2]:     torch.autograd.backward(outputs_with_grad, args_with_grad)
[rank2]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 267, in backward
[rank2]:     _engine_run_backward(
[rank2]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
[rank2]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank2]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 19.93 GiB. GPU  has a total capacity of 21.99 GiB of which 5.56 GiB is free. Including non-PyTorch memory, this process has 16.42 GiB memory in use. Of the allocated memory 10.76 GiB is allocated by PyTorch, and 5.21 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank1]: Traceback (most recent call last):
[rank1]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/recipes/quickstart/finetuning/finetuning.py", line 8, in <module>
[rank1]:     fire.Fire(main)
[rank1]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
[rank1]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank1]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
[rank1]:     component, remaining_args = _CallAndUpdateTrace(
[rank1]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
[rank1]:     component = fn(*varargs, **kwargs)
[rank1]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/src/llama_recipes/finetuning.py", line 313, in main
[rank1]:     results = train(
[rank1]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/src/llama_recipes/utils/train_utils.py", line 175, in train
[rank1]:     loss.backward()
[rank1]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/_tensor.py", line 525, in backward
[rank1]:     torch.autograd.backward(
[rank1]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 267, in backward
[rank1]:     _engine_run_backward(
[rank1]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
[rank1]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank1]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 301, in apply
[rank1]:     return user_fn(self, *args)
[rank1]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 320, in backward
[rank1]:     torch.autograd.backward(outputs_with_grad, args_with_grad)
[rank1]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 267, in backward
[rank1]:     _engine_run_backward(
[rank1]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
[rank1]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank1]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 19.93 GiB. GPU  has a total capacity of 21.99 GiB of which 5.56 GiB is free. Including non-PyTorch memory, this process has 16.42 GiB memory in use. Of the allocated memory 10.76 GiB is allocated by PyTorch, and 5.21 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank3]: Traceback (most recent call last):
[rank3]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/recipes/quickstart/finetuning/finetuning.py", line 8, in <module>
[rank3]:     fire.Fire(main)
[rank3]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
[rank3]:     component_trace = _Fire(component, args, parsed_flag_args, context, name)
[rank3]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
[rank3]:     component, remaining_args = _CallAndUpdateTrace(
[rank3]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
[rank3]:     component = fn(*varargs, **kwargs)
[rank3]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/src/llama_recipes/finetuning.py", line 313, in main
[rank3]:     results = train(
[rank3]:   File "/home/ubuntu/llama-recipes-github-repo/llama-recipes/src/llama_recipes/utils/train_utils.py", line 175, in train
[rank3]:     loss.backward()
[rank3]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/_tensor.py", line 525, in backward
[rank3]:     torch.autograd.backward(
[rank3]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 267, in backward
[rank3]:     _engine_run_backward(
[rank3]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
[rank3]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank3]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/function.py", line 301, in apply
[rank3]:     return user_fn(self, *args)
[rank3]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/utils/checkpoint.py", line 320, in backward
[rank3]:     torch.autograd.backward(outputs_with_grad, args_with_grad)
[rank3]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 267, in backward
[rank3]:     _engine_run_backward(
[rank3]:   File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 744, in _engine_run_backward
[rank3]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank3]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 19.93 GiB. GPU  has a total capacity of 21.99 GiB of which 5.56 GiB is free. Including non-PyTorch memory, this process has 16.42 GiB memory in use. Of the allocated memory 10.76 GiB is allocated by PyTorch, and 5.21 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Training Epoch: 1:   0%|                                                                                | 0/450 [00:15<?, ?it/s]
Training Epoch: 1:   0%|                                                                                | 0/450 [00:14<?, ?it/s]
Training Epoch: 1:   0%|                                                                                | 0/450 [00:14<?, ?it/s]
Training Epoch: 1:   0%|                                                                                | 0/450 [00:15<?, ?it/s]
E1008 13:22:46.931000<PHONE_NUMBER>10528 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 11317) of binary: /home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/bin/python
Traceback (most recent call last):
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/distributed/run.py", line 879, in main
    run(args)
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/distributed/run.py", line 870, in run
    elastic_launch(
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/ubuntu/.cache/pypoetry/virtualenvs/test-llama-recipes-K8dcJ3Te-py3.10/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
recipes/quickstart/finetuning/finetuning.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2024-10-08_13:22:46
  host      : ip-10-142-7-161.us-west-2.compute.internal
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 11318)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2024-10-08_13:22:46
  host      : ip-10-142-7-161.us-west-2.compute.internal
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 11319)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2024-10-08_13:22:46
  host      : ip-10-142-7-161.us-west-2.compute.internal
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 11320)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-10-08_13:22:46
  host      : ip-10-142-7-161.us-west-2.compute.internal
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 11317)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html


Can't accept incoming connection: EOF

Background
With Autopilot on, one outgoing channel opened successfully.  Since then, a single external node is repeatedly(Approaching a day every few minutes) been trying and failing to open a channel to my node.  I cannot see why it is failing.  Blocking the IP in UFW appears results in no apparent incoming connection attempts.
Your environment
LND: 0.5 commit=3b2c807288b1b7f40d609533c1e96a510ac5fa6d
OS: Linux RaspPi 4.14.79-v7+ #1159 SMP Sun Nov 4 17:50:20 GMT 2018 armv7l GNU/Linux
BTC Client: Bitcoin Core RPC client version v<IP_ADDRESS>
Steps to reproduce
Start LND and monitor with journalctl
Expected behaviour
New inbound connection stops repeating either by no future attempts or channel successfully opens
Actual behaviour
Same node repeatedly attempts and fails to open an inbound connection roughly every 2.5 minutes.  No other inbound channels appear to be trying to connect.
Incoming_log.txt

I have a script that runs periodically and that uses nc -z ip_addr 9735 to test whether my node is reachable or not, and to notify me if it goes down. I noticed in the lnd logs that they produce exactly the same message every time my script pings it. Is it possible that this is something similar? Maybe an explorer that heard about you and is just making sure you're still alive?

You are using an old version of lnd. You should upgrade to v0.5.1.

Doesn't seem like an issue other than that node potentially being faulty/malicious. In the future, we'll detect peers that do this through a DoS engine and ban them.


Option to precache multiple requests simultaneously

If I always have to wait for the full installation of the service worker before the page is displayed, why not to tell the service worker that assets can be loaded not sequentially, but as quickly as possible-in parallel?
On a slow connection, we have to wait a very long time for a sequential download, even on http2
My case (slow 3g connection):
13 page assets are loaded in parallel in 5 seconds and then 22 assets are loaded with service worker sequentially for about a minute over http2 (13 of them are the same ones that loaded in 5 seconds for the page and the main size of all assets falls on them)

You can read some background on this new behavior introduced in Workbox v6 at https://github.com/GoogleChrome/workbox/issues/2528
Basically, we went from not imposing a limit at all—which would sometimes incur net::ERR_INSUFFICIENT_RESOURCES errors, and more often, would lead to critical resources loaded by the main page loading slowly when the service worker was registered too early—to a model in which precaching requests are made one at a time.
I still believe that's the "safer" thing to do overall, and my general view of precaching requests is that they should be considered low-priority things that happen in the background. But there are use cases in which folks really do want more aggressive precaching to take place, which you and a few other folks have now asked for.
I'm open to making this configurable in a future release, with the default still being one at a time.

Our application is completely dependent on the service worker and therefore we have to wait for its full installation.
If the constructor of the service worker had an option that allows you to configure the loading of the service worker in parallel when it is necessary - this would save our application from a long 1st start as well as starts in iOS (it is famous for periodically completely deleting the application)

By the way, could you explain a bit more why your web application can't do anything until after the service worker has fully installed? Service workers are normally intended to be progressive enhancement, and (if possible) it's usually best to treat them as such instead of as a hard requirement.

Our application protection against CSRF attacks is based on a Double Submit Cookie when a new token is issued to the client for each session in response header and session cookie.
Due to the fact that Google has made the so - called "session" cookies (cookies without expiration date) eternal in the "Automatic session recovery" mode - the whole idea of protecting against CSRF attacks becomes vulnerable-cookies are eternal and the attacker has a chance to pick up a token knowing approximately the algorithm for its creation.
In order to solve this problem with the violation of the RFC by Google, we wrote a plugin for the service worker that looks at all POST requests and puts a token in the CSRF header that the service worker receives at the beginning of the session of the service worker and the server now has the ability to change the token not only for the session, but also for each request without problems with the Back button and without problems for the site open in different tabs
Therefore, our application cannot work correctly without installing and starting the service worker

While I was writing the answer , I came up with an idea - can I not pre-cache the resources before installing the service worker, but start post caching ?

"The Offline Cookbook" offers an overview of various ways a service worker can be used to cache and keep content up to date. Roughly speaking, each of the concepts in that cookbook map into something you can do with a combination of workbox-routing and workbox-strategies, or workbox-precaching.
It's definitely legitimate to create a service worker that doesn't precache during installation. Normally, you'd "cache as you go" via runtime caching, instead of caching things upfront. When taking this approach, you'd need to think about what kind of freshness guarantees you need for your HTML and subresources, and choose an appropriate strategy (NetworkFirst, StaleWhileRevalidate, etc.) to use for each runtime route. Things get more complicated if you have sets of assets that all need to be expired or refreshed together, atomicly—that's the sort of thing that workbox-precaching is optimized for, but requires more manual intervention when you're using dynamic runtime caching.
I would not describe any of this as "post-install precaching" though. It's more cache-as-you-go runtime caching. If you have a set of assets that all need to be added to a cache before you service worker activates, then using precaching (which has to happen during install) is really your best bet.

I'm sorry, I'm not very clear - what are the reasons not to do the same precaching after installing the service worker?
why can't we use the same algorithm that is used during the installation, after activating the service worker?

The upgrade flow for workbox-precaching—where you have an atomic set of previously cached URLs, and you migrate to a new set, with one or more of those cached URLs updated—relies heavily on the service worker lifecycle. Specifically, we need to cache new entries during install, perform cleanup of no longer needed entries during activate, and use the transition from post-install waiting to active to allow open applications to opt-in to updating to the latest set of precached assets.
If we decouple those operations from the service worker lifecycle, it would break a number of common use cases. For instance, single page apps that lazily load their subresources would end up loading the HTML from the "old" precache, and then may end up lazily-loading their subresources from the "new" precache. We would move from a model with well-defined cache consistency to one with inherent race conditions. (This discussion on why you shouldn't unconditionally call skipWaiting() covers the same material.)
I've been helping developers with service workers since they first launched in Chrome, and I can say with confidence that there are not a large number of applications that rely on early control the service worker. We strongly encourage developers  to treat the presence of a service worker as optional when it comes to app functionality, and it's trivial for users to shift-reload a page, which will trigger a navigation without the service worker being in control.
I really don't know enough about your web app's security constraints to speak to what you're doing for CSRF protection, but I can tell you it's not a common use case for service workers.

time passes and if earlier the worker service was more like a kind of feature for decoration, today it is a working tool and is already not optional for many applications, but mandatory.
I know a lot of developments where increased confidentiality and security are required (document management with digital signatures, crypto tokens, etc.), all of them actively use the worker service. But since they are mostly used on desktops , the issue with a long initial installation is not as acute for them as for me.
I understand that it is difficult to detach the logic of the precache module from the lifecycle, but the implementation of the functionality for monitoring and updating assets for new versions in my situation will largely duplicate the logic laid down in the precache module

We don't have any plans to change the workbox-precaching behavior to detach it from the service worker install and activate events. Those service worker events and the service worker lifecycle in general were designed to be used in the way Workbox uses them.
Allowing developers to customize the number of simultaneous precaching requests that are in flight at any given time is still something we will consider for a future release, and I'll leave this issue open to track that.

based on your experience, can you suggest an algorithm for supporting pre-caching assets for service worker versions in the runtime cache?

It wouldn't be called precaching.
You can set up runtime caching routes to match whatever assets you want to be added to the cache, along the lines of what's described at https://developers.google.com/web/tools/workbox/guides/get-started#routing_and_caching_strategies

+1 for making this configurable, these days simultaneous requests can improve loading assets a lot to be honest.

+1 request - is another +1 potential point of failure and +1 another request that can increase the loading time of the site

Any update about this, or can you add some simple explaination how this can be changed in V6?
For me, my situation is that I want to provide a offline docs for users, but I have every small asssets generated with hash name by default, and currently a 1 by 1 fetch offen takes more then 3 minute to install a new service worker. The only request came from user maybe the search api, so it's totally fine for me to parallel requests, e.g.: 10 to 20.
It would be great if anyone points out some workaround for me to achieve this, thanks.

Can the code for Install be updated to below so that there are 10 requests in parallel.
const fetchQueueDepth = 10; can be configured as a parameter.
install(event) {
    function MakeQuerablePromise(promise) {
        // Don't modify any promise that has been already modified.
        if (promise.isFulfilled) return promise;
        // Set initial state
        var isPending = true;
        var isRejected = false;
        var isFulfilled = false;

        // Observe the promise, saving the fulfillment in a closure scope.
        var result = promise.then(
            function(v) {
                isFulfilled = true;
                isPending = false;
                return v;
            },
            function(e) {
                isRejected = true;
                isPending = false;
                throw e;
            }
        );
        result.isFulfilled = function() { return isFulfilled; };
        result.isPending = function() { return isPending; };
        result.isRejected = function() { return isRejected; };
        return result;
    }
    // waitUntil returns Promise<any>
    // eslint-disable-next-line @typescript-eslint/no-unsafe-return
    return waitUntil(event, async () => {
        const installReportPlugin = new PrecacheInstallReportPlugin();
        this.strategy.plugins.push(installReportPlugin);
        var promises = [];
        // Cache entries 10 at a time.
        const fetchQueueDepth = 10;
        // See https://github.com/GoogleChrome/workbox/issues/2528
        for (const [url, cacheKey] of this._urlsToCacheKeys) {
            const integrity = this._cacheKeysToIntegrities.get(cacheKey);
            const cacheMode = this._urlsToCacheModes.get(url);
            const request = new Request(url, {
                integrity,
                cache: cacheMode,
                credentials: 'same-origin',
            });
            const result = Promise.all(this.strategy.handleAll({
                params: { cacheKey },
                request,
                event,
            }));
            promises.push(MakeQuerablePromise(result));
            if (promises.length >= fetchQueueDepth) {
                await Promise.any(promises);
            }
            promises = promises.filter(function(value){
                return value.isPending();
            });
        }
        const { updatedURLs, notUpdatedURLs } = installReportPlugin;
        if (process.env.NODE_ENV !== 'production') {
            printInstallDetails(updatedURLs, notUpdatedURLs);
        }
        return { updatedURLs, notUpdatedURLs };
    });
}

at least for http2, you can safely download multiple assets simultaneously

and if you also use a lower priority for these requests, there are no barriers at all for parallel loading of all assets at once

I strongly second this as opt-in behaviour.
@jeffposnick's argument as to why it works as it does is sound, but there are still many edge cases where allowing parallel downloads is appropriate - it would be great if Workbox simply allowed to toggle this as an option, with the default behaviour still being sequential downloads.

We're precaching a huge amount of very small requests for some dynamically generated pages (few kbs each) and this basically bricks our install experience. Just showing my love for this feature request.

So far, I think it's possible to solve this using injectManifest and a similar file:
// workbox
import { cacheNames, clientsClaim } from "workbox-core";
import { registerRoute } from "workbox-routing";
import { precacheAndRoute } from "workbox-precaching";

precacheAndRoute(/* ... */); //basic and important files
registerRoute(/* ... */);

self.skipWaiting();
clientsClaim();

// files
import async from "async";
let files = ["http://google.com/", "http://google.com/"] //a lot of links to other files (which are not in the precache), can be taken from a string or file
let concurrency = 10;
async function main() {
    let cache = await caches.open(cacheNames.runtime);
    await async.mapLimit(files, concurrency, (x) => cache.add(x));
}
main();

So far, I think it's possible to solve this using injectManifest and a similar file:
// workbox
import { cacheNames, clientsClaim } from "workbox-core";
import { registerRoute } from "workbox-routing";
import { precacheAndRoute } from "workbox-precaching";

precacheAndRoute(/* ... */); //basic and important files
registerRoute(/* ... */);

self.skipWaiting();
clientsClaim();

// files
import async from "async";
let files = ["http://google.com/", "http://google.com/"] //a lot of links to other files (which are not in the precache), can be taken from a string or file
let concurrency = 10;
async function main() {
    let cache = await caches.open(cacheNames.runtime);
    await async.mapLimit(files, concurrency, (x) => cache.add(x));
}
main();

These files will be added to the cache, if they are not there, during installation. If the installation is interrupted, the remaining part will be added at the next startup.

will there be any competition when downloading between your code and the workbox code?
workbox at the start of the service worker will also start downloading the same files!!

will there be any competition when downloading between your code and the workbox code? workbox at the start of the service worker will also start downloading the same files!!

Important and necessary files are added to the precacheAndRoute. All the others are added manually, and the workbox just serves them with route.

Been more than a year now, any updates on this? I'm making a coding ide which is meant to work offline, and since there are like over 1k fileicons, downloading them serially takes more than 2 minutes, thus making the installation button come a lot after 🥲. The icon sizes are ranging from 1-3 kb so it shouldn't be a problem downloading them in parallel. Out users are asked to provide enough bandwidth for the first time load/installation of the app in order to work offline. (Less than 10mb though 🙃) hopefully you guys provide an inbuilt way to change requests to parallel since I'm using next pwa package, which depends on workbox and I would prefer to not have to edit the sw.js file that is generated during build time to inject the above code posted by Alex. Thanks in advance.

Any updates ?
On my side, my cypress tests crash because to load 20 precache I need to wait for 10 secondes.
After those 10 secondes, the SW reload the page in the middle of my cypress tests..
Adding a 10 seconds wait for each test is not a solution.
So it could be very very nice to parallelize requests.
Or maybe have an option to avoid SW reload after precaching files.

@JayPe69 a forced reload is not standard SW behavior and is to my knowledge not induced by Workbox either.

@JayPe69 a forced reload is not standard SW behavior and is to my knowledge not induced by Workbox either.

Oh, I got this reload, when I added precache functionality, and I tried to locate if it comes from my SW code or not.
I'll double check.

I applied this with patch-package, now 500 assets are cached in 3 seconds instead of 4.5 minutes.
diff --git a/node_modules/workbox-precaching/PrecacheController.js b/node_modules/workbox-precaching/PrecacheController.js
index e00975e..7380049 100644
--- a/node_modules/workbox-precaching/PrecacheController.js
+++ b/node_modules/workbox-precaching/PrecacheController.js
@@ -5,6 +5,7 @@
   license that can be found in the LICENSE file or at
   https://opensource.org/licenses/MIT.
 */
+import eachLimit from 'async-es/eachLimit'
 import { assert } from 'workbox-core/_private/assert.js';
 import { cacheNames } from 'workbox-core/_private/cacheNames.js';
 import { logger } from 'workbox-core/_private/logger.js';
@@ -150,9 +151,8 @@ class PrecacheController {
         return waitUntil(event, async () => {
             const installReportPlugin = new PrecacheInstallReportPlugin();
             this.strategy.plugins.push(installReportPlugin);
-            // Cache entries one at a time.
             // See https://github.com/GoogleChrome/workbox/issues/2528
-            for (const [url, cacheKey] of this._urlsToCacheKeys) {
+            await eachLimit(this._urlsToCacheKeys, 10, async ([url, cacheKey]) => {
                 const integrity = this._cacheKeysToIntegrities.get(cacheKey);
                 const cacheMode = this._urlsToCacheModes.get(url);
                 const request = new Request(url, {
@@ -165,7 +165,7 @@ class PrecacheController {
                     request,
                     event,
                 }));
-            }
+            })
             const { updatedURLs, notUpdatedURLs } = installReportPlugin;
             if (process.env.NODE_ENV !== 'production') {
                 printInstallDetails(updatedURLs, notUpdatedURLs);

you'll have to add async-es to your own package.json

Hi there,
Workbox is moving to a new engineering team within Google. As part of this move, we're declaring a partial bug bankruptcy to allow the new team to start fresh. We realize this isn't optimal, but realistically, this is the only way we see it working. For transparency, here're the criteria we applied:

For PRs, we closed everything that doesn't target the current v7 branch.
For Issues, we closed everything that was created before 2024.

Thanks, and we hope for your understanding!
The Workbox team

Reopening this issue, as it has an active PR associated with it. Sorry.