System-wide integration affects build time of projects not using vcpkg.

System-wide integration causes the AppLocalFromInstalled target to be called for commandline builds for all projects even if they are up-to-date. AppLocalFromInstalled invokes Powershell, which is a relatively slow process to start (ranging from 500-1000mSec depending on version and machine). As such for some of the builds we do this becomes a pretty time-consuming thing and builds with many projects now take a considerable amount of extra time.
Proposed solution

VcpkgEnabled could default to False and projects which actually do use vcpkg would be required to have VcpkgEnabled set to true. This would break existing builds, but it could be opt-in when calling vcpkg integrate so it doesn't break anything and people who want their builds to be normal again can use it
detect whether a project uses vcpkg and do not call AppLocalFromInstalled unless that is the case. This is what happens for builds in VS as well (but possibly using the file tracker which isn't usable in msbuild?)
detect if a project was actually built, and opnly then call AppLocalFromInstalled otherwise skip it

Describe alternatives you've considered
Running builds with VcpkgEnabled=False and manually modifying individual project files which do use vcpkg to set it to True again.

@stinos I'm no expert on vcpkg by any means, but I have a thought/suggestion which works quit well for me, especially if you want to have multiple versions of Visual Studio  such as vs2015, vs2017, vs2019, or a new beta version of VC++ and and multiple instances of vcpkg such as dev, test, qa, prod, etc...

One can remove the System-Wide integration (./vcpkg integrate remove)

Then create the vcpkg nupkg module (vcpkg integrate project).

On projects you want to use vcpkg add the nupkg project by using the following:

With a project open, go to Tools->NuGet Package Manager->Package Manager Console and paste:
(Install-Package vcpkg.D.Git.vs2017.test -Source "D:\Git\vs2017.test\scripts\buildsystems")
Of course the italicized_ will be different depending on where the vcpkg(s) is/are installed.

The only drawback I would see is if one has many projects already using vcpkg under the system-wide integration one would need to go back and add the vcpkg integrate project "nupkg" to each one and recompile.

I admit I haven't looked into the NuGet way so I didn't list it as alternative indeed.

Of course the italicized_ will be different depending on where the vcpkg(s) is/are installed

But it's not this different across machines, right? I.e. if I install the NuGet package into one project on one machine, will it just also work for that project on another machine which has vcpkg in a different location?

The experts (@JackBoosY @PhoebeHui  @LilyWangL  @Neumann-A as I see them contribute a lot to this overall project) would know more about this than me, but my thoughts would be if vcpkg is not installed in the same location on each PC, then it would seem logical to assume (I hate assuming) one  after porting to another PC, would need to delete the nupkg for vcpkg in the project and add the nupkg for vcpkg to the installed location on the new machine.
It also seems the "vcpkg integrate project" is only available on a windows PC, as the option does not appear on my linux machine.

it would seem logical

Not very logical to me actually, as it serisouly reduces usability since it makes it way too hard to share code and have it in version control. But I tested it and yes that's basically how it works (unless I missed something). Could be circumvented by using msbuild variables/manually changing package names and whatnot, but that's really not so ideal.

@ras0219-msft What do you think?

@stinos - I agree, not so ideal :(
Most software designs are with a corporate mindset, meaning the PCs are setup identical following some kind of standards devised by the corporation or in case of purchased software the software vendor.
Unfortunately that usually does not cater too well to open source projects which are shared among corporations having different standards, freelancers, hobbyists, educational and just plain enthusiasts.

I believe the proposed change would be breaking enough that we would not want to do that.

@BillyONeal I assume you mean the first of the three propsed changes? As said it can be opt-in so then it's not really breaking anymore.

@stinos I don't believe any of the proposed alternatives are workable but I might not understand the last one.

The first is out because it's a massive compatibility break (it breaks the whole integrate install feature for most customers).
The second requires a time machine -- we don't know whether a project is using vcpkg until we run the step you consider problematic to determine that, since normally we put things on the include path and the 'does this project use vcpkg' test is whether they included a header vcpkg provides in any of their source files.
I don't really understand what the third tries to do but it sounds an awful lot like the same time machine problem as the second.

Moreover we provide nuget-package powered solutions as well as explicit inclusion of vcpkg.targets from projects that want to use such functionality as workarounds if you don't want to use the system-wide installer.

@BillyONeal to clarify:

The first is out because it's a massive compatibility break (it breaks the whole integrate install feature for most customers).

I'm suggesting to make it opt-in via an argument to integrate install so it doesn't break anything unless explicitly asked for.

the 'does this project use vcpkg' test is whether they included a header vcpkg provides in any of their source files.

Ok clear, thanks.

I don't really understand what the third tries to do but it sounds an awful lot like the same time machine problem as the second.

Not sure if tit's the same: it's a suggestion to skip the AppLocalFromInstalled target if outputs are up-to-date, i.e. just like ClCompile and Link targets. And just like Visual Studio behaves: if a project has been built already and is built again, all build steps are just skipped. But if a binary copied by vcpkg gets deleted from the output directory, VS detects that and runs a build again.

I'm suggesting to make it opt-in via an argument to integrate install so it doesn't break anything unless explicitly asked for.

OH! Sure, we could do that.

it's a suggestion to skip the AppLocalFromInstalled target if outputs are up-to-date

To do that you have to declare the outputs before the target runs, but we don't know what the outputs are before running the script.

To do that you have to declare the outputs before the target runs, but we don't know what the outputs are before running the script.

Yeah I sort of exepcted that, but VS gets it right so I assume that's only because it uses the filetracker mechanism?

I admit to not being super clear on how the file tracker mechanism works :). But my understanding is that that is only for tracking inputs (e.g. headers consumed by your C++ file), not, as here, outputs. I suppose we could drop a sentinel file "lastvcpkgcopysucceeded.txt" or similar to be able to declare an output.

I've been working on a powershell-less replacement for applocal.ps1 and was able to repro the delay. Consider the following, with vcpkg enabled it takes 2 seconds, without it takes 1 second, at least on my Blade 14:

NotVcpkgDemo.zip
C:\Users\billy\source\repos>msbuild NotVcpkgDemo\NotVcpkgDemo.sln /p:Platform=x64 /p:Configuration=Debug /t:Rebuild /p:VcpkgEnabled=false
Microsoft (R) Build Engine version 16.11.0-preview-21302-05+5e37cc992 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Building the projects in this solution one at a time. To enable parallel build, please add the "-m" switch.
Build started 7/4/2021 7:35:49 PM.
Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo.sln" on node 1 (Rebuild target(s)).
ValidateSolutionConfiguration:
  Building solution configuration "Debug|x64".
Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo.sln" (1) is building "C:\Users\billy\source\repos\NotVcp
kgDemo\NotVcpkgDemo\NotVcpkgDemo.vcxproj" (2) on node 1 (Rebuild target(s)).
CoreClean:
  Creating directory "x64\Debug\".
PrepareForBuild:
  Creating directory "C:\Users\billy\source\repos\NotVcpkgDemo\x64\Debug\".
  Creating directory "x64\Debug\NotVcpkgDemo.tlog\".
InitializeBuildStatus:
  Creating "x64\Debug\NotVcpkgDemo.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
VcpkgTripletSelection:
  Not using Vcpkg because VcpkgEnabled is "false"
ClCompile:
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.29.30130\bin\HostX86\x64\CL.exe /c /ZI /
  JMC /nologo /W3 /WX- /diagnostics:column /sdl /Od /D _DEBUG /D _CONSOLE /D _UNICODE /D UNICODE /Gm- /EHsc /RTC1 /MDd
  /GS /fp:precise /permissive- /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"x64\Debug\\" /Fd"x64\Debug\vc142.pdb" /external:
  env:EXTERNAL_INCLUDE /external:W3 /Gd /TP /FC /errorReport:queue NotVcpkgDemo.cpp
  NotVcpkgDemo.cpp
Link:
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.29.30130\bin\HostX86\x64\link.exe /ERROR
  REPORT:QUEUE /OUT:"C:\Users\billy\source\repos\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.exe" /INCREMENTAL /ILK:"x64\Debug\
  NotVcpkgDemo.ilk" /NOLOGO kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.
  lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manife
  st:embed /DEBUG /PDB:"C:\Users\billy\source\repos\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.pdb" /SUBSYSTEM:CONSOLE /TLBID:
  1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:\Users\billy\source\repos\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.lib" /MACHINE:X64 x
  64\Debug\NotVcpkgDemo.obj
  NotVcpkgDemo.vcxproj -> C:\Users\billy\source\repos\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.exe
FinalizeBuildStatus:
  Deleting file "x64\Debug\NotVcpkgDemo.tlog\unsuccessfulbuild".
  Touching "x64\Debug\NotVcpkgDemo.tlog\NotVcpkgDemo.lastbuildstate".
Done Building Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo\NotVcpkgDemo.vcxproj" (Rebuild target(s)).

Done Building Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo.sln" (Rebuild target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.09

C:\Users\billy\source\repos>msbuild NotVcpkgDemo\NotVcpkgDemo.sln /p:Platform=x64 /p:Configuration=Debug /t:Rebuild /p:VcpkgEnabled=true
Microsoft (R) Build Engine version 16.11.0-preview-21302-05+5e37cc992 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Building the projects in this solution one at a time. To enable parallel build, please add the "-m" switch.
Build started 7/4/2021 7:35:53 PM.
Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo.sln" on node 1 (Rebuild target(s)).
ValidateSolutionConfiguration:
  Building solution configuration "Debug|x64".
Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo.sln" (1) is building "C:\Users\billy\source\repos\NotVcp
kgDemo\NotVcpkgDemo\NotVcpkgDemo.vcxproj" (2) on node 1 (Rebuild target(s)).
_PrepareForClean:
  Deleting file "x64\Debug\NotVcpkgDemo.tlog\NotVcpkgDemo.lastbuildstate".
InitializeBuildStatus:
  Creating "x64\Debug\NotVcpkgDemo.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
VcpkgTripletSelection:
  Using triplet "x64-windows" from "C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\"
ClCompile:
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.29.30130\bin\HostX86\x64\CL.exe /c /I"C:
  \Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\include" /ZI /JMC /nologo /W3 /WX- /diagnostic
  s:column /sdl /Od /D _DEBUG /D _CONSOLE /D _UNICODE /D UNICODE /Gm- /EHsc /RTC1 /MDd /GS /fp:precise /permissive- /Zc
  :wchar_t /Zc:forScope /Zc:inline /Fo"x64\Debug\\" /Fd"x64\Debug\vc142.pdb" /external:env:EXTERNAL_INCLUDE /external:W
  3 /Gd /TP /FC /errorReport:queue NotVcpkgDemo.cpp
  NotVcpkgDemo.cpp
Link:
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.29.30130\bin\HostX86\x64\link.exe /ERROR
  REPORT:QUEUE /OUT:"C:\Users\billy\source\repos\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.exe" /INCREMENTAL /ILK:"x64\Debug\
  NotVcpkgDemo.ilk" /NOLOGO /LIBPATH:"C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\debug\li
  b" /LIBPATH:"C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\debug\lib\manual-link" kernel32
  .lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.l
  ib odbccp32.lib "C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\debug\lib\*.lib" /MANIFEST
  /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /DEBUG /PDB:"C:\Users\billy\source\repos\NotVcpkgDe
  mo\x64\Debug\NotVcpkgDemo.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:\Users\billy\source\repo
  s\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.lib" /MACHINE:X64 x64\Debug\NotVcpkgDemo.obj
  LINK : the 32-bit linker (C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.29.30130\bin\H
  ostX86\x64\link.exe) failed to do memory mapped file I/O on `C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\insta
  lled\x64-windows\debug\lib\clangStaticAnalyzerCheckersd.lib' and is going to restart linking with a 64-bit linker for
   better throughput
  LINK : restarting link with 64-bit linker `C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\
  14.29.30130\bin\HostX64\x64\link.exe'
  NotVcpkgDemo.vcxproj -> C:\Users\billy\source\repos\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.exe
AppLocalFromInstalled:
  pwsh.exe -ExecutionPolicy Bypass -noprofile -File "C:\Dev\vcpkg\scripts\buildsystems\msbuild\applocal.ps1" "C:\Users\
  billy\source\repos\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.exe" "C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\insta
  lled\x64-windows\debug\bin" "x64\Debug\NotVcpkgDemo.tlog\NotVcpkgDemo.write.1u.tlog" "x64\Debug\vcpkg.applocal.log"
FinalizeBuildStatus:
  Deleting file "x64\Debug\NotVcpkgDemo.tlog\unsuccessfulbuild".
  Touching "x64\Debug\NotVcpkgDemo.tlog\NotVcpkgDemo.lastbuildstate".
Done Building Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo\NotVcpkgDemo.vcxproj" (Rebuild target(s)).

Done Building Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo.sln" (Rebuild target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.64

C:\Users\billy\source\repos>

Hmmmm so I tried turning on my new thing in the above repro and the time doesn't get all that much better. I think the actual increase in time is from adding all the libs to the linker command line.
C:\Users\billy\source\repos>msbuild NotVcpkgDemo\NotVcpkgDemo.sln /p:Platform=x64 /p:Configuration=Debug /t:Rebuild /p:VcpkgEnabled=true
Microsoft (R) Build Engine version 16.11.0-preview-21302-05+5e37cc992 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Building the projects in this solution one at a time. To enable parallel build, please add the "-m" switch.
Build started 7/4/2021 7:42:20 PM.
Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo.sln" on node 1 (Rebuild target(s)).
ValidateSolutionConfiguration:
  Building solution configuration "Debug|x64".
Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo.sln" (1) is building "C:\Users\billy\source\repos\NotVcp
kgDemo\NotVcpkgDemo\NotVcpkgDemo.vcxproj" (2) on node 1 (Rebuild target(s)).
_PrepareForClean:
  Deleting file "x64\Debug\NotVcpkgDemo.tlog\NotVcpkgDemo.lastbuildstate".
InitializeBuildStatus:
  Creating "x64\Debug\NotVcpkgDemo.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified.
VcpkgTripletSelection:
  Using triplet "x64-windows" from "C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\"
ClCompile:
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.29.30130\bin\HostX86\x64\CL.exe /c /I"C:
  \Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\include" /ZI /JMC /nologo /W3 /WX- /diagnostic
  s:column /sdl /Od /D _DEBUG /D _CONSOLE /D _UNICODE /D UNICODE /Gm- /EHsc /RTC1 /MDd /GS /fp:precise /permissive- /Zc
  :wchar_t /Zc:forScope /Zc:inline /Fo"x64\Debug\\" /Fd"x64\Debug\vc142.pdb" /external:env:EXTERNAL_INCLUDE /external:W
  3 /Gd /TP /FC /errorReport:queue NotVcpkgDemo.cpp
  NotVcpkgDemo.cpp
Link:
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.29.30130\bin\HostX86\x64\link.exe /ERROR
  REPORT:QUEUE /OUT:"C:\Users\billy\source\repos\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.exe" /INCREMENTAL /ILK:"x64\Debug\
  NotVcpkgDemo.ilk" /NOLOGO /LIBPATH:"C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\debug\li
  b" /LIBPATH:"C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\debug\lib\manual-link" kernel32
  .lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.l
  ib odbccp32.lib "C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-windows\debug\lib\*.lib" /MANIFEST
  /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /manifest:embed /DEBUG /PDB:"C:\Users\billy\source\repos\NotVcpkgDe
  mo\x64\Debug\NotVcpkgDemo.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:\Users\billy\source\repo
  s\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.lib" /MACHINE:X64 x64\Debug\NotVcpkgDemo.obj
  LINK : the 32-bit linker (C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.29.30130\bin\H
  ostX86\x64\link.exe) failed to do memory mapped file I/O on `C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\insta
  lled\x64-windows\debug\lib\clangStaticAnalyzerCheckersd.lib' and is going to restart linking with a 64-bit linker for
   better throughput
  LINK : restarting link with 64-bit linker `C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\
  14.29.30130\bin\HostX64\x64\link.exe'
  NotVcpkgDemo.vcxproj -> C:\Users\billy\source\repos\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.exe
AppLocalFromInstalled:
  C:\Dev\vcpkg-tool\out\build\x64-Debug\vcpkg.exe z-applocal --target-binary="C:\Users\billy\source\repos\NotVcpkgDemo\
  x64\Debug\NotVcpkgDemo.exe" --installed-bin-dir="C:\Dev\vcpkg\scripts\buildsystems\msbuild\..\..\..\installed\x64-win
  dows\debug\bin" --tlog-file="x64\Debug\NotVcpkgDemo.tlog\NotVcpkgDemo.write.1u.tlog" --copied-files-log="x64\Debug\vc
  pkg.applocal.log"
  vcpkg applocal processing: C:\Users\billy\source\repos\NotVcpkgDemo\x64\Debug\NotVcpkgDemo.exe
FinalizeBuildStatus:
  Deleting file "x64\Debug\NotVcpkgDemo.tlog\unsuccessfulbuild".
  Touching "x64\Debug\NotVcpkgDemo.tlog\NotVcpkgDemo.lastbuildstate".
Done Building Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo\NotVcpkgDemo.vcxproj" (Rebuild target(s)).

Done Building Project "C:\Users\billy\source\repos\NotVcpkgDemo\NotVcpkgDemo.sln" (Rebuild target(s)).

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:02.18

C:\Users\billy\source\repos>

Hmmmm so I tried turning on my new thing in the above repro and the time doesn't get all that much better. I think the actual increase in time is from adding all the libs to the linker command line

Don't know if it's possible to have msbuild display the time it takes for individual steps but since the linker is specifically saying it is restarting that is not unlikely. A better comparison would be to use the hostx64 compiler/linker.
I see you're using pwsh.exe (how did you make that happen) which does have a slightly better startup time than powershell.exe IIRC.

Don't know if it's possible to have msbuild display the time it takes for individual steps but since the linker is specifically saying it is restarting that is not unlikely. A better comparison would be to use the hostx64 compiler/linker.

Forcing everyone to use the x64 linker would still be our 'fault' though, so I think this issue wouldn't be "fixed".

I see you're using pwsh.exe (how did you make that happen) which does have a slightly better startup time than powershell.exe IIRC.

That's the current behavior: https://github.com/microsoft/vcpkg/blob/110c89fc6ada7bc9fad715eaf8211f20aad084f5/scripts/buildsystems/msbuild/vcpkg.targets#L123-L132


Many arguments not available

Started using recently but have found many arguments are not available on the following versions maybe i'm missing something or putting them in the wrong places but i don't believe so
Terraform v1.0.10
on linux_amd64

provider registry.terraform.io/terra-farm/xenorchestra v0.3.7

wait_for_ip and ipv4_addresses return
Error: Unsupported argument

   on main.tf line 84, in resource "xenorchestra_vm" "win10vm":
   84:     wait_for_ip = true

 An argument named "wait_for_ip" is not expected here.

 Error: Unsupported argument

   on main.tf line 85, in resource "xenorchestra_vm" "win10vm":
   85:     ipv4_addresses = "<IP_ADDRESS>/32"

 An argument named "ipv4_addresses" is not expected here.

as well accessing the majority of attributes
 output "server_public_ip" {   value = xenorchestra_vm.win10vm.ipv4_addresses }
returns
Error: Unsupported attribute │ │   on main.tf line 104, in output "server_public_ip": │  104:   value = xenorchestra_vm.win10vm.ipv4_addresses │ │ This object has no argument, nested block, or exported attribute named "ipv4_addresses".
If i'm missing something obvious please let me know otherwise is this just a version issue that i'm experiencing?

Hi @devcose, v0.3.7 is an extremely old version. The terraform registry docs are versioned and you can see from the screenshot below that v0.21.0 is the latest version. Please upgrade to the latest version and try again.

I'm happy to continue providing support on this thread or in Discord, but I'll be closing this issue since I feel confident this is related to running an old version.


docs: modify heading names

modify heading names for better formatting

The work has done by #55.


Merge etag handling guidelines into .NET guidelines

See PR #646

At some point, @KrzysztofCwalina wanted me to write up specifics on ETag handling as shown here: https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/search/Azure.Search.Documents/src/Indexes/Models/SearchIndex.cs until the generator does it (https://github.com/Azure/autorest.csharp/issues/668). This doc work should document that.


Maintenance of package

Hi guys,
Thank you for that great library for DjangoCMS.
I see you started to deprecate some of your packages  We would appreciate any response regarding the maintenance of that package. Do you plan to support it or it is deprecated and you can mark it as archived?

@DmytroLitvinov hi, this package isn't deprecated. Its not had a release for a while, but it's stable & functional.

@DmytroLitvinov This is very much active, we just fixed the CI, please feel free to send a PR for fixes etc. and we will be happy to review and merge.

Hi @vinitkumar ,
The one which we need in our project is related to that PR.
Is there any help of review that PR? I am ready to provide a help with that.

@DmytroLitvinov I would suggest read through the comments and implement the fix following the comments by @jrief. If all seems good, I will be more than happy to review and merge it.

@vinitkumar , thank you for your comment.
Created PR. Could you please review it?

Hi @vinitkumar , @marksweb ,
Any plan on releasing new version of djangocms-text-ckeditor with fixes which were merged recently? Do you need my help in that process?

@DmytroLitvinov I plan to release 5.0 but just need to spare time to do that.

Hey @marksweb ,
Thank you for release :)


Om86

support for micro benchmarks
issue micro-benchmark commands during latency-watcher passtwo keys
namespace-watcher and node-watcher add respective namespace and service benchmark flags which are later consumed by latency-watcher

wrong PR


Mouse Scroll Support

Add mouse scroll support.
This is will require some tweaking, before I invest more time into it, please take a look so we can discuss how to proceed.

Hey @djclueless just want to check in here. Are you still working on edits or is this ready for another round of review?

We are good for another round of reviews. It has been working great, I have been using it since last commit.  Just let me know what other tweaks you want or if its good to bring out of draft.
I might be missing some tests we can add in as well. I just wanted to make sure that it passed the checks.

Cool! I wasn't able to get to this today, but I should be able to merge it in on Monday.

This had merge conflicts, so I'm moving it over to #307 where I can resolve the conflicts and make some last few tweaks.


Define build strategy with Kubernetes

Kubernetes by default has no container image build mechanism, so what are the ways are we gonna support it?

One point I would like to add here is the builds should be generic enough that image format is supported for both container runtimes that kubernetes supports i.e. docker and rkt

you want to run kompose within a k8s cluster ?
I am not sure I get it. It is a client side tool.

Hmm, I'm not sure I get it either.
Only way how to do builds with Kubernetes that I can think of right now is to build image (similarly as docker-compose would) and than push it to some registry that is also accessible by targeted Kubernetes cluster.

I created this issue if we wanna give a end-to-end feel on k8s we will have to decide what we do with builds, are we gonna handle image builds for user? or right now keeping it out of our scope and giving user a warning and asking them to build themselves?

I want views of all of you folks.

ok got it.
IMHO for now we should skip "builds" and define it as "unsupported".
Because for images to be used within a k8s cluster they would need to be uploaded to a registry as well. So it will be difficult to run out of the box with kompose. There will be manual intervention not matter what.

thanks for making this clear, will wait for what @janetkuo and @ngtuna has to say then we can close this issue! :)

I agree with @runseb :smiley:

I'm +1 for @runseb

Just to be clear, I also agree with @surajssd that we should give an end-to-end feel and should support "builds" in the future.
We can push to a registry specified in a preference file. K8s provides private Docker registry addon that we can use.

With support for openshift buildconfig implemented, we are now in a position to complete the end-to-end build story by supporting local builds for kubernetes and openshift.
Below, I am documenting the local build feature.
Goal
To support build directive in a Docker Compose service for Kubernetes (and hence Openshift) by building images locally and pushing it to a registry.
Issue
Kubernetes has a registry addon to host container images privately, but it does not have a mechanism like Openshift's buildconfig to internally build images from source in an automated way and push it to the internal registry. As a result, a kompose user trying to convert a compose file with a build directive and deploy it to kubernetes will need to do the following:

build the image locally
push it to a registry in use by the kubernetes setup
may need to tinker with the image names in the artifacts generated by kompose

In order to provide a good user experience to Kubernetes Kompose users, we need to automate local image build, image upload to Kuberenetes (or Openshift) accessible registry. This will also prove to be useful for Openshift as well by allowing Kompose users (developers) to deploy local code without pushing it to a public repository.
Proposed solution

Use system's docker command to build image locally
Use service name as image name, or use service's metadata to override it
Pushing image to remote registry:

Use docker push CLI to push image
You can push push to a non default registry by specifying it using CLI option --target-registry
Kompose will not take care of logging in to remote registry. The user needs to ensure that Docker is setup/configured to push to the target registry. Refer to 1

Use built image's name in deployment artifacts as needed, so that the user does not need to edit them manually.

@rtnpro Thank you for the proposal.
I have one question though - one of the tenet of Kompose is that the user doesn't need docker engine / CLI installed / running. This proposal assumes that Docker engine / CLI is installed and running. Is that correct?
To be clear, I am NOT against the proposal. In fact, I would like to see this done and will highly appreciate the end-to-end feature in Kompose. But just wanted to bring this though to the table so that we know what we are changing.
Looks good otherwise but I would appreciate other folks who are more knowledgeable than me.

On Mon, Jan 2, 2017 at 10:20 AM, Pradeepto Bhattacharya
<EMAIL_ADDRESS>wrote:

@rtnpro Thank you for the proposal.
I have one question though - one of the tenet of Kompose is that the user
doesn't need docker engine / CLI installed / running. This proposal assumes
that Docker engine / CLI is installed and running. Is that correct?

Yes

@pradeepto

I have one question though - one of the tenet of Kompose is that the user doesn't need docker engine / CLI installed / running.

If we assume that Kompose is used by developers developing containerized applications that they will need Docker anyway. (More so if they have docker-compose file with build. So I would say that Docker as requirement for supporting  builds is fair.
Kompose will be still useful without Docker (it will be required only if build is used)
@rtnpro

Use service name as image name, or use service's metadata to override it

We don't need any metadata to override it.
Yes by default service name is used as image name, but it can be overridden by image field in docker-compose file.
Ff you have docker-compose file like this:
 version: '2'
 services:
   foo:
     build: .
     image: myregistry.example.com/myimage:v0.1

You don't need any additional information you have everything that you need. (Expecting that docker is already authenticated to access myregistr.example.com)

+1. I think that makes sense to have Kompose triggers the docker build, then having docker installed or not is out of kompose's scope. I suggest to define kompose label for target repository instead of flag.

I suggest to define kompose label for target repository instead of flag.

we don't need label for that. As repository is part of the image name and that can be set by image keyword in docker-compose file.

we don't need label for that. As repository is part of the image name and that can be set by image keyword in docker-compose file.

Yes I agree. If user doesn't set repository in the image name then we make the default to docker hub.

But also consider the part where images need to be pushed to registry? Who will do that user does manually or kompose does build and push both(using docker in backend)?

But also consider the part where images need to be pushed to registry? Who will do that user does manually or kompose does build and push both(using docker in backend)?

Kompose will do it. It will use docker engine to do build and push.

okay so we are assuming here that user will have:

docker daemon running locally
setup docker-registry
docker daemon is set up to push to private registry by adding registry info in /etc/sysconfig/docker
user is authenticated with the registry

docker daemon running locally

Yes, it will be usually local docker or boot2docker on mac.
Technically he need just docker client configured to connect to any docker daemon.

setup docker-registry

That is one of the options that user has. He can use some hosted registry like Docker Hub or quay.io
In case of OpenShift, it is possible to use integrated registry, if it is exposed.

docker daemon is set up to push to private registry by adding registry info in /etc/sysconfig/docker

If his registry is signed by untrusted certification authority or selfsigned, than yes.

user is authenticated with the registry

yes
But overall it is yes to your list. It is not that great UX as with just docker-compose but this is by design more complicated thing. We can do this to start with it, and than improve it if we figure out how.

Initial work pushed to #372

In case of supporting local builds for openshift, let's add an optional flag: --local-build for convert for openshift provider. By default, Openshift will use buildconfig.

I would probably do it other way around. As BuildConfig requires relaxed
permissions on OpenShift cluster that are not in default configurations. I
think that we should use local build as default for OpenShift.
On Wed, Jan 11, 2017 at 1:58 PM, Ratnadeep Debnath<EMAIL_ADDRESS>
wrote:

In case of supporting local builds for openshift, let's add an optional
flag: --local-build for convert for openshift provider. By default,
Openshift will use buildconfig.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-incubator/kompose/issues/97#issuecomment-271862027,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADfdpJEWRSr9dbV_v_G_nekGyuFp_rIks5rRNIDgaJpZM4JhUBg
.

#521 Has been merged! :tada: :tada: :tada: Let's close this! :+1:


BrickColor cannot be set because it aliases to Color without handling the type difference

We have BrickColor set as an alias for Color:
https://github.com/rojo-rbx/rbx-dom/blob/master/rbx_dom_lua/src/database.json#L3422-L3426
However, this means that a property write will attempt to set a BrickColor Enum to the Color property that expects a Color3 value.

Setting Part.BrickColor to (BrickColor) Hot pink - setProperty:30
Error(Roblox: Unable to assign property Color. Color3 expected, got BrickColor
PluginDebugService.user_Rojo.rbxm.Rojo.Packages.RbxDom.PropertyDescriptor:19 function set
PluginDebugService.user_Rojo.rbxm.Rojo.Packages.RbxDom.PropertyDescriptor:77 function write
PluginDebugService.user_Rojo.rbxm.Rojo.Plugin.Reconciler.setProperty:32 function setProperty

This appears to be the fault of us aliasing BasePart.BrickColor to BasePart.Color. With the migration system we now have, it should be straightforward to move away from an alias and instead have the relationship between those two be a migration.

Looking into this more, and I can create a problem with the same cause in Lune:
local roblox = require("@lune/roblox")

local part = roblox.Instance.new("Part")
part.BrickColor = roblox.BrickColor.new(45)

roblox.serializeModel({ part })

Failed to write document to buffer - Property type mismatch: Expected Part.Color to be of type Color3uint8 or Color3, but it was of type BrickColor on instance ROOT.Part
[Stack Begin]
    Script '[C]'
    Script 'test', Line 6
[Stack End]

I think treating BasePart.BrickColor as a canonical, serializable property, or using a migration (if we're sure we always want it to turn into BasePart.Color) are both valid ways out here

I think migration is the way to go here. When you set Color in Studio, then BrickColor gets set to the nearest value that matches which indicates that BrickColor just sets Color to itself so the nearest value is therefore the BrickColor you chose.

I think treating BasePart.BrickColor as a canonical, serializable property, or using a migration (if we're sure we always want it to turn into BasePart.Color) are both valid ways out here

I can see a scenario like this happening if we do a migration:

Serialize a Part that has BrickColor set
Deserialize the file in something like Lune
Attempt to index Part.BrickColor and cry that it doesn't work

This is already possible with something like Font and FontFace though, and we've gotten no complaints.
I think other than that, it's pretty much desirable 100% of the time to convert BasePart.BrickColor into BasePart.Color. A migration feels like the way to go.


Scorecard - fixes for compatiblity with new config validator

remove duplicate violations
fix constraint name in txt output

fixes some issues in https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/issues/584

@briantkennedy Wasn't this meant to be fixed in https://github.com/forseti-security/config-validator/pull/115?

That's not 100% of it.  I opened https://github.com/forseti-security/config-validator/issues/116 to track the duplicate violation issue.  It's quite vexing because I can't repro in rego unit tests and I'm only able to repro for the full end to end FCV setup when constraint framework is involved.


Getting payload encrypted into error queue

after max retry payload goes to error queue in an encrypted format, so we can't process it further form error queue by RabbitMQ UI

amqp: ENV['CLOUDAMQP_URL'],   workers: 3,   timeout_job_after: 60,   threads: 5,   prefetch: 5,   after_fork: Proc.new { ActiveRecord::Base.establish_connection },   durable: true,   handler: Sneakers::Handlers::Maxretry,   retry_timeout: 5 * 60 * 1000,   metrics: Sneakers::Metrics::LoggingMetrics.new, queue: 'abc',              env: nil,              ack: true,              exchange: 'abc.exchange',              routing_key: '#',              exchange_type: :topic,              arguments: {               'x-dead-letter-exchange': 'abc-retry'             }

We don't do any encryption in sneakers.


improved the NavBar's text color

Title and Issue number
Updated the NavBar text color.#123

Close #<issue_no>
Close #123

Checklist:

[X] I have mentioned the issue number in my Pull Request.
[X] I have commented my code, particularly in hard-to-understand areas
[] I have created a helpful and easy to understand README.md

Please review it @pranjay-poddar .


Update slf4j-log4j12 to 2.0.11

About this PR
📦 Updates org.slf4j:slf4j-log4j12 from 2.0.9 to 2.0.11
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.slf4j", artifactId = "slf4j-log4j12" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "org.slf4j", artifactId = "slf4j-log4j12" }
}]

labels: test-library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #132.


Add component Description automatically from rust comment

Right now we have to do this to both get the description in the API docs and on the component:
components!("app", {
    // My fancy test component
    @[Description["My fancy test component"]]
    testy: String,
});

It would be nice if doing just // comment would automatically add a Description to the component.

The // comment is not possible as // comments are stripped before macro invocations.
Capturing doc-comments have to be done in the form of meta-attributes, e.g #[doc = "comment"]

@ten3roberts Oh sorry, I mean ///. Is that also not possible?

I may have been a bit unclear.
Capturing doc-comments /// comment is definitely possible, although a little tricky.
What I meant i that they need to be matched in the #[doc = "comment"] style as they desugar into that before the macros are evaluated. The /// comment syntax is only sugar.
In fact, we already capture doc attributes, but only forward them to the declared component function


Add docstring linting

Privileged issue

[X] I am a LangChain maintainer, or was asked directly by a LangChain maintainer to create an issue here.

Issue Content
We want to start enforcing that all public interfaces have proper Google-style docstrings. This is both to improve our code readability and the quality of our API reference.
We can do this by turning on the lint rules shown here: https://github.com/langchain-ai/langchain/pull/23187
For each package in the repo we should turn this on and fix all the resulting lint errors.
Specific TODOs:
[] Turn on for core and fix existing issues. Can see what that looks like here #25939
[] Turn on for each integration package
[] Turn on for langchain
[] Turn on for text-splitters
[] Turn on for community
[] Turn on for experimental

HI, Can I work on this issue?

HI, Can I work on this issue?

copying the update here and fixing all the resulting lint errors would be awesome! https://github.com/langchain-ai/langchain/pull/25939

Hey I want to take up this issue , can I work on it ?

I had made some changes on libs/core/langchain_core/_api/beta_decorator.py, may I contribute to this issue ?

@baskaryan From the above list of TODOs, when you review #28127 (current status: quick to review, passes all 52 checks, and ready to merge) for the text-splitters package, you may want to update the above checklist too.

Hey @baskaryan would love to work on this issue pls assign it


Issues with Light Client verification for relayer v0 MVP

Crate
relayer
Summary of Bug
There are a few issues to solve after #363

[x]  (waiting on https://github.com/informalsystems/tendermint-rs/pull/709)
Multiple on-chain clients may be at different heights, let's say h1 and h2, h1 < h2. Assume the relayer local client is at h2. Currently, if we want to update the on-chain client that is at h1, we call verify_to_target(h1) and this returns an error as h1 is smaller than the latest trusted height (h2).

[x] (#501) If light clients are configured to use the same store path, running two relayer processes concurrently to create clients on different chains will fail with:
error: client create failed: tx error: Light client error: IO error: could not acquire lock on "data/DEEB0AB3F2D7BEC0B8C9FF1532715635314F54D5/db": Os { code: 35, kind: WouldBlock, message: "Resource temporarily unavailable" }

[x] (fixed in https://github.com/informalsystems/tendermint-rs/pull/775)
Once in a while on update client and other clis:
The application panicked (crashed).
Message:  called `Result::unwrap()` on an `Err` value: Error(Context { kind: LightClient, source: Some(Error(Context { kind: NoWitnesses, source: None, backtrace: Some(   0: backtrace::backtrace::libunwind::trace
            at /Users/ancaz/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.55/src/backtrace/libunwind.rs:90:5

Most of the times the command will succeed the second time.

[x]  (fixed in https://github.com/informalsystems/tendermint-rs/pull/706) LightBlock's ValidatorSet must have the proposer and voting power set. This is temp fixed in cosmos.rs:fix_validator_set().

[x] (#428, #431) It difficult do manage the client section in the configuration file. It is more now during development as we restart the chains often and the config file keeps around the old configuration (unless we manually clean it up) that may sometimes be in conflict with the new one. In addition we need to add witnesses manually.

[X] There seems to be leaking between global rpc_addr and address in client sections, with some queries using the client address.

For Admin Use

[x] Not duplicate issue
[x] Appropriate labels applied
[x] Appropriate milestone (priority) applied
[x] Appropriate contributors tagged
[x] Contributor assigned/self-assigned

All items have been closed

All items have been closed


Support django 1.11

And run tests on django 1.10

Coverage increased (+1.2%) to 96.623% when pulling 9ac8f3412ec5f45b491adec996ac4a8a447b7fdb on jproffitt:master into 3e30e72433a3db2c5f58a9fda646cd65ce6ee12a on agiliq:master.

Any chance this could be merged and released soon?

Coverage increased (+1.2%) to 96.623% when pulling 4acbe71b490faafb614e2a1f5c9425dfe193544f on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+0.9%) to 96.392% when pulling deeb5d14024bf1fc7840804e78898749d55a7eed on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.

Coverage increased (+1.5%) to 97.015% when pulling bb135ae352b455a754a6951282851a81e1307a81 on jproffitt:master into 9b223bc73c8b9c04f30be19ab56c08b779b11d94 on agiliq:master.


functionality to refresh session birthdate

work in progress for #77
Waiting on PR #76 because this call should go right before the return() on the REST helper calls.

@lawinslow should this PR be killed? It has been open for a long time.

Hmmm, good question. I'm grappling with how we want to handle this. Do we want to store this internal state, or would we rather just improve our handling of errors when they occur? (e.g., Check auth via #90 when we hit unexpected behavior, suggest the user re-auth, something like that)
Also, currently, this PR doesn't use the session age within any of the functions, right?

@sckott any thoughts?

yeah, I think this is a hold over from some of the earlier discussions regarding handling the timeouts. I do think age actually is used in the package. @aappling-usgs has run into is, as have I (after an hour) I think?

see https://github.com/USGS-R/sbtools/blob/master/R/session_validate.R#L36

Hmmm, yeah ok. We are half implemented here, aren't we. Let's merge this and see where we go with error handling in the future.

yeah, was going to say merge sounds good

hi @sckott :smile_cat:

hi!

PR meetup!


Make install progress visible from the CLI

It would be nice to know what the install progress was even in a CLI context
Reported by @chrisguida

nifty progress bar lib https://github.com/mitsuhiko/indicatif

The install progress of what?

@mirkoRainer installs

Am I correct in assuming that backend->src->install->mod.rs->cli_install would be an appropriate place to monitor the progress? Or maybe Line 125 pub async fn install?
It seems like there are two ways of installations, package.sideload and package.install.
I assume that package.install is the one to focus on.
If I'm in the right place, I assume that the progress bar would be external to this code but would be triggered concurrently while awaiting the cli_install code?
I'm also going to have to read up on and understand more about what
#[command(
    custom_cli(cli_install(async, context(CliContext))),
    display(display_none)
)]
#[instrument(skip(ctx))]
pub async fn install(

does. And perhaps that will help me decide where to start.
The more I read this, the more I'm inclined to think the progress bar will be an async tokio call within install?
fwiw, I'm new to Rust with little practical application so thank you in advance for fielding my questions.

Both would be nice actually. Sideloading refers to when you are installing a .s9pk that you have locally vs from a marketplace.


[PM-2021] Do not add self to table

We should not try to add self to lookup table.

2020-05-28 12:45:27,609 [i.i.s.peergroup.kademlia.KRouter] - abf9f9cf364abe7ae2623f54bb8903d5bfe57e85325fdd747a78fc78718b889c9cdfa45c0ee1c78be5765c49b47c3fb633a040ac82c9f54c92f13bf3b02ce842 Initialization complete. 0 peers identified (of which 1 is myself and 0 are preconfigured bootstrap peers).

Should we update this log to remove the line of which 1 is myself?

this log is correct, as self is always part of the table from the start, check the method def getIndex[A](config: Config[A], clock: Clock): NodeRecordIndex[A] , it init our map with self node entry. It is due to the fact than kBuckets.closestNodes always returns baseId as part of results (which is self node id), so to stay consistent we need to have this id in the map from the beginning.


feat: support for Bacs debit payments

Summary
This PR adds support for bacs debit payments
References:

iOS example: BacsDebitViewController
Android example: BacsDebitPaymentMethodActivity

Motivation
The confirmPayment function was returning This paymentMethodType is not supported yet for bacs payments, this PR adds the bindings for bacs payments for the react native sdk
Testing

[x] I tested this manually
[x] I added automated tests

Documentation
Select one:

[ ] I have added relevant documentation for my changes.
[ ] This PR does not result in any developer-facing changes.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Thanks for the PR! I need to find out if accountNumber and sortNumber are safe values to transmit across the RN bridge before we add support for this

@charliecruzan-stripe Thank you, If you don't mind me asking, is this connected to security concerns?

@charliecruzan-stripe any updates on this?

hi @charliecruzan-stripe , would you have an update here for us in terms of timelines?

hi @charliecruzan-stripe , would you have an estimated timeline here?

@charliecruzan-stripe any news here?

Any updates on this @charliecruzan-stripe ?

@khzouroussama Did you end up building a fork of the SDK and using that?
@charliecruzan-stripe The radio silence is a bit concerning, I appreciate you're probably busy and have lots on your plate but an update on why this can't be merged would be really helpful.

@hwildwood It have been a while since I worked on this, but you can build your own package to support this (i suggest using Expo Modules since they support the new arch out of the box), or patch your existing package using the changes from this PR (although I am not sure about the security concern that @charliecruzan-stripe mentioned).
Also note that implementing the BACS payment will not be straight forward and you will have to contact stripe support to enable that for your account, and review your implementation.
https://github.com/stripe/stripe-ios/blob/7b8b320beeb54023b6c8680097a5ecfc2067f524/Example/Non-Card Payment Examples/Non-Card Payment Examples/BacsDebitViewController.swift#L14

@khzouroussama Thanks for your reply, that's really helpful!  I hadn't appreciated that using the feature in this PR would still require sign off from Stripe but that makes sense.


Refactoring Publish Dialogs to solve several issues related to state inconsistencies.

Refactored the Publish Dialogs to enforce state consistency. More refactoring is possible buy this PR is big as it is.
Added tests.
Fixes #919 #920 #922 and #925

PTAL

PTAL

@ILMTitan Before I do anything else, I'll be replying to your general comment about your concerns with the tests so we can find a common ground to move forward. My following comments are then of a general nature and might be trumped by specific cases on the PR. And sorry for the longness of this :).
First off I agree 100% that these are not good tests. They will definitely be a maintenance nightmare, I know, I've been writing them for the past 10 days or so. The point I will try to make now is that the tests are not bad tests because of themselves (at least not entirely) but because of the classes they are testing.
Each of the step classes is basically a fairly complex state machine. And I say complex because each of the states is determined by many different items. The most extreme one is the GKE step, in which the state of the machine is determined by the project selected, whether the required APIs are enabled or not, whether the project has clusters and one of those clusters is selected, whether the version, deployment name and number of replicas have valid values, and then separately, we have another "state branch"  that has to do with whether to expose a service or not. Different combinations of these will get you to different states which are also no entirely described by one property, CanPublish == true is the ultimate state, the one you want to get to, but when you are not there you might be in any combination of semi states in between, for instance, "selected project with apis enabled but no clusters, correct version and replicas but incorrect deployment name" means you are in a "valid gcp project state, no clusters, errors in input" and you can have a few combinations of those which in reality represent different states. Given that states are determined by so many items, transitions in the state machine are then anything that changes any of these items, mainly, changing the project selected either externally or with the select project command, but also changing the version or the replicas amount, or enabling APIs, or selecting a cluster.
Your first point, testing transitions. Behavior wise, I think a state machine should be tested not just on whether it sometimes lands in the correct state, but on all of its possible transitions (which these tests don't do either entirely). The analogy  you are making between behavior testing and having a test for each if in the code I think might have been valid in this case if we were now and in the future enforcing 100% code coverage, it's true that then some of these test would be redundant, since they are testing exactly the same code path, that is, cases in which the state you started in is not relevant, and if we added a new code path in the future, we would be forced to add a new test for that to get 100% test coverage. Since we are not enforcing 100% test coverage, I feel safer with testing as much behavior as possible not from the point of view of the current implementation but from the point of view of the class api, in the case of a state machine, and specifically these classes, we need to test initial states, final states, and transitions from one state to the other for each of the different states and the different ways of transitioning, since those, and not the specific implementation, are what define the class behavior. Less in line with testing one method or if and more inline with:

Sometimes a single behavior will even span multiple methods (e.g. "removing an object from a list after adding it should cause that object to not be in the list"). Tests should be written to validate the individual behaviors of a class, not its individual methods.

If we don't do that it would be super easy to just add a couple lines of code that don't break any test and yet break the behavior for a very specific A to B transition. Take a look for instance at #925 (only happened if the transition occurs from a very specific A, getting to B from any other state different than A worked properly). I found a couple other bugs like that (I stopped tracking the bugs when I found them while fixing them) and that's why I decided to add all the transition tests that now seem too much.
Your third point, asserting more than needed. I think that one of your quotes matches what I'm saying:

Multiple tests shouldn't assert the same facts about a class (even if redundant assertions are factored into a helper method), unless the reason that those facts are true is different for each test.

The fact is that all the CanPublish, HasErrors, GeneralErrors, IsLoading and like properties, change their values for many different reasons and should be tested in all those different scenarios where they change value, that's why they are being asserted everywhere. Again, we have too much going on on the tested classes.
I agree though that when testing that Version changes properly I probably shouldn't be asserting anything about the GCPProjectId, and viceversa, I will clean that up.
As for your second point, having lots of helper methods, again, your quote I think just refers to the point I'm trying to make:

If you find yourself having to factor out many helpers to write clean tests, it's possible that the class under test is too complex - instead of factoring out duplication, eliminate it completely by refactoring the class.

So, this is precisely what I want to do, I probably haven't been super clear about it before, but I think we commented some when you suggested that PublishDialog should never be null, which I agree with. In my view, all of these step classes should be separated into smaller more manageable state machines or steps (if we want to show them in one window or in many wizard like windows, that's another thing entirely), so when we are at a given step (say, select project, or load clusters for selected project, or select clusters) everything else is fixed from the point of view of that step, this will not only ease testing, but will also ease adding new steps, validations, user inputs etc..
I didn't do this because first, I wasn't familiar enough with the publish dialogs so that I felt comfortable refactoring so much, and also we didn't have enough tests, and also there were several live bugs. I decided to give them as much consistency as was possible while keeping the existing code structure as unchanged as possible and adding as much tests as possible so it would be easier to refactor in subsequent PRs.

Main question for you: Would you agree that refactoring the tested classes and not simplifying the current tests might be the best way to go in the long run?

Given that my intention is to refactor the tested classes (I'm happy to start that immediately after this PR), would you agree with, in this PR:

Keeping the (almost) exhaustive transition testing, which will be moved around and simplified once we have specific classes representing specific states.

Keeping most of the helper methods (won't be needed once complexity is refactored out of the tested classes).

Removing the unnecessary asserts from the helper assert methods, but taking into account that a lot of the properties might change for a lot of different reasons so they need to be asserted much more than might be intuitive.

I will of course look at the specific comments and act on those but keeping the above guidelines?

Let me know! Cheers

Closing this, as #956 is fork of this change.


Provide a switch to skip deployment of example services

Provide a switch/env-var that causes the manage hub script to skip publishing the example services from the OH examples repo. This will reduce the amount of time it takes for the hub to be started.

Tested by myself often during development, and tested by e2edev.


defaultSelected not working

Thank you for your very useful library.
When I have:
<select name="label[px_per_mm]" id="label_px_per_mm">
  <option value="8">Low Def</option>
  <option value="12">High Def</option>
</select>

And run this:
new Selectr('#label_px_per_mm', {defaultSelected: true})

The first option (8/Low Def) is not selected as I would expect (and as per the standard select behaviour).

Fixed in the latest version.

Thank you very much!


Consider using devDependencies

A lot of the current dependencies are actually devDependencies, so they are unnecessarily installed when using npm install solid-auth-client.

I don't know where I got that, this seems fine.

Actually, @dmitrizagidulin remarked that bootstrap might be in the wrong place.

You're right, looks like bootstrap is in the wrong place. Other than that, which are the particular dependencies you believe to be incorrectly categorized?

Just bootstrap, the others I saw were transitive dependencies from bootstrap I presume.

The other dependencies are @trust/oidc-rp, auth-header, and isomorphic-fetch, which don't relate to bootstrap. I can move bootstrap to the devDependencies, as it's just used in the demo app.

Works for me!

Great, PR coming shortly.


Unable to import

In the itemshop.py file I get an error saying Unable to import 'coloredlogs', Unable to import 'twitter', Unable to import 'PIL'. I also get Unable to import 'requests' and Unable to import 'PIL'. I don't know if I forgot to install something or if something is wrong.
Thanks in advance!

Please read the README. You're missing the required packages, you can learn how to install these elsewhere.


[BUG] - starting the node ends up in "Killed" state

External
Area
Other   Any other topic (Delegation, Ranking, ...).
Summary
Working on mainnet. Following the instruction on official documentation to install & run the cardano node. When I start the node, leave it few hours to synchronise, I come up to see the status "Killed" in the synchronisation process output.
Steps to reproduce
Steps to reproduce the behavior:

Get & install latest versions of cabal, ghc
Install libsodium
Download source of cardano-node
Build the node
Download latest configuration files
Start cardano-node in tmux
Wait for few hours to synchronize
See "Killed" on tmux screen.

During step 7, I can track the current epoch & block with cardano-cli query tip --mainnet.
When the synchronisation is killed, the above command will give me output of:
cardano-cli: Network.Socket.connect: <socket: 11>: does not exist (Connection refused)

Expected behavior
Synchronization runs 24/7 and never gets killed for some unknown reason.
System info (please complete the following information):

OS: Ubuntu
Version: 20.04
Node version:

cardano-node 1.26.2 - linux-x86_64 - ghc-8.10
git rev 3531289c9f79eab7ac5d3272ce6e6821504fec4c

Screenshots and attachments

Additional context
I can re-run the the node and it will continue from the last block. However, after ~5minutes of synchronisation, it will again fail and show the "Killed" message.
No other messages are being shown.
This happens on both the relay and producer blocks.

@geantas you are running out of RAM (memory). Increase the RAM to about 16 GB and you should be good.

Did increasing the RAM fix the issue? im running into the same problem. I manage to sync up to around 96% until 'Killed' message appears

@architechnique yes, I've doubled the RAM and everything started to run smooth.

I got this error my ram size is 24 gb


[azure-template-cpp] publish version 1.1.0-beta.1

Update vcpkg ports for Azure SDK release. This release may contain multiple ports.

Adding azure-template to release
cc: @danieljurek, @weshaggard


fix(scene): do not restore if already off

This pull request fixes an issue where the scene restoration was triggered even when the scene was already turned off. The commit "fix(scene): do not restore if already off" ensures that the restoration is skipped if the scene is already off.

Thank you for pointing this out. Fixed now.


clipboard file: xrdp is not responding CB_RESPONSE_FAIL even if it failed

When clipboard_send_file_data fails due to the file is not readable or other reasons, the client waits to finish file transfer forever. Maybe we need to abort file transfer gracefully.

meta@xrdp:~/xrdp$ tail -f ~/.local/share/xrdp/xrdp-chansrv.10.log
[20210915-06:35:14] [DEBUG] [process_message_channel_data(chansrv.c:446)] process_message_channel_data: chan_id 2 chan_flags 19
[20210915-06:35:14] [DEBUG] [clipboard_data_in(clipboard.c:1523)] clipboard_data_in: chan_id 2 chan_flags 0x13 length 32 total_length 32 in_request 0 g_ins->size 8192
[20210915-06:35:14] [DEBUG] [clipboard_data_in(clipboard.c:1586)] clipboard_data_in: clip_msg_id 8 clip_msg_status 0 clip_msg_len 24
[20210915-06:35:14] [DEBUG] [clipboard_data_in(clipboard.c:1591)] clipboard_data_in: 8
[20210915-06:35:14] [DEBUG] [clipboard_process_file_request(clipboard_file.c:508)] clipboard_process_file_request:
[20210915-06:35:14] [DEBUG] [clipboard_send_file_data(clipboard_file.c:408)] clipboard_send_file_data: streamId 24422 lindex 0 nPositionLow 0 cbRequested 34520
[20210915-06:35:14] [ERROR] [clipboard_send_file_data(clipboard_file.c:415)] clipboard_send_file_data: file open [/home/meta/Downloads/paper.docx] failed
[20210915-06:35:14] [DEBUG] [scard_pcsc_check_wait_objs(smartcard_pcsc.c:401)] scard_pcsc_check_wait_objs:
[20210915-06:35:14] [DEBUG] [scard_pcsc_get_wait_objs(smartcard_pcsc.c:374)] scard_pcsc_get_wait_objs:
[20210915-06:35:14] [DEBUG] [get_timeout(chansrv.c:153)] get_timeout:

I haven't read MS-RDPECLIP corner to corner yet however I guess xrdp should respond to  File Contents Request PDU with File Contents Response PDU that contain CB_RESPONSE_FAIL to indicate "the request could not be satisfied".
I quickly performed git grep CB_RESPONSE_FAIL but it was not used in clipboard_file.c at all.  The return value of clipboard_send_file_size and clipboard_send_file_data should be checked and xrdp should return CB_RESPONSE_FAIL when they failed.
https://github.com/neutrinolabs/xrdp/blob/23742dbb8ef4f3a0b2d7d911bafd09153c6193da/sesman/chansrv/clipboard_file.c#L496-L526

I think using goto like this is not a bad idea in error management. It will work well rather than checking return values.
diff --git a/sesman/chansrv/clipboard_file.c b/sesman/chansrv/clipboard_file.c
index c317da79..f2ee9b3b 100644
--- a/sesman/chansrv/clipboard_file.c
+++ b/sesman/chansrv/clipboard_file.c
@@ -414,7 +414,7 @@ clipboard_send_file_data(int streamId, int lindex,
     {
         LOG_DEVEL(LOG_LEVEL_ERROR, "clipboard_send_file_data: file open [%s] failed",
                   full_fn);
-        return 1;
+        goto fail_example;
     }
     if (g_file_seek(fd, nPositionLow) < 0)
     {
@@ -450,6 +450,20 @@ clipboard_send_file_data(int streamId, int lindex,
     LOG(LOG_LEVEL_INFO, "S2C: Transfered a file: filename=%s, uid=%d", full_fn, g_getuid());

     return rv;
+
+fail_example:
+    if (s == NULL)
+    {
+        make_stream(s);
+        init_stream(s, cbRequested + 64);
+    }
+
+    out_uint16_le(s, CB_FILECONTENTS_RESPONSE);
+    out_uint16_le(s, CB_RESPONSE_OK);
+
+    s_mark_end(s);
+    free_stream(s);
+    return 1;
 }

I've had a look at [MS-RDPECLIP] sections <IP_ADDRESS> <IP_ADDRESS>.6 and <IP_ADDRESS>.7, and I agree that we should be sending a CB_RESPONSE_FAIL
The goto discussion is an interesting one. Personally I don't find it any worse than returning in the middle of a function, which we do anyway. However, opinions on this one vary and I know @aquesnel is against it for new code. I'd prefer to avoid using it if there's an alternative that's just as clear.
I'd suggest replacing your example above with an early return. So your code above becomes:-

    {
         LOG_DEVEL(LOG_LEVEL_ERROR, "clipboard_send_file_data: file open [%s] failed",
                   full_fn);
        LOG(LOG_LEVEL_ERROR, "clipboard_send_file_data: file open [%s] failed : %s",
                   full_fn, g_get_strerror());
        clipboard_send_filecontents_response_fail(streamId);
        return 1;
    }

and clipboard_send_filecontents_response_fail() looks like this. This is off the top of my head and hasn't been tested:-
/***
 * Sends a failure response to a CLIPRDR_FILECONTENTS_REQUEST
 * @param streamId Stream ID from CLIPRDR_FILECONTENTS_REQUEST
 * @return 0 for success
 */
static int
clipboard_send_filecontents_response_fail(int streamId)
{
    struct stream *s;
    int size;

    make_stream(s);
    init_stream(s, 64);

    out_uint16_le(s, CB_FILECONTENTS_RESPONSE);
    out_uint16_le(s, CB_RESPONSE_FAIL);
    out_uint32_le(s, 4);
    out_uint32_le(s, streamId);
    s_mark_end(s);
    size = (int)(s->end - s->data);
    rv = send_channel_data(g_cliprdr_chan_id, s->data, size);
    free_stream(s);
    return rv;
}

There are two advantages to this approach:-

We do early returns all over the place, so any developer working on xrdp will need to be used to this.
As you say, you are going to need to do something similar in clipboard_send_file_size(). You can just insert calls to clipboard_send_filecontents_response_fail() into that, so we've got less code overall.

A couple of minor points:-

There are LOG_DEVEL() calls all over the place here which should probably be LOG() instead. I've changed one above.
The streamId name comes straight out of [MS-RDPECLIP] but is not compatible with our coding standard. I don't know whether it's better to leave this as it is, or change it to stream_id or streamid.

Hi,
I agree with @matt335672 about creating a function to send the CB_RESPONSE_FAIL error.
For the variable naming, my recommendation is to follow the xrdp style guide for variable names in the code, and use the exact MS-RDP specification names when logging the variable's value.

Thanks for the comments. I'll try to implement it that way.


[Event Request] Codeunit 73 - Purch.-Explode BOM - OnBeforeModifyToPurchLine

Dear ALAppExtensions team,
Through this issue we would like to request event publisher OnBeforeModifyToPurchLine to be added to Codeunit 73 Purch.-Explode BOM with the following signature:
    [IntegrationEvent(false, false)]
    local procedure OnBeforeModifyToPurchLine(var ToPurchLine: Record "Purchase Line"; RecPurchLine: Record "Purchase Line")
    begin
    end;

with VAR parameter ToPurchLine.
which would be called as follows from OnRun:
        ToPurchLine := Rec;
        ToPurchLine.Init();
        ToPurchLine.Description := Description;
        ToPurchLine."Description 2" := "Description 2";
        OnBeforeModifyToPurchLine(ToPurchLine, Rec); // << New event
        ToPurchLine.Modify();

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


--namespace on ks init

kubecfg version: dev-2017-11-28T14:45:48-0500
jsonnet version: v0.9.5
client-go version: v1.6.8-beta.0+$Format:%h$

Having an issue with ks init respecting the namespace flag. Run ks init guestbook --namespace ks-dev. The output of that command:
ks init guestbook --namespace ks-dev
INFO  Creating a new app 'guestbook' at path 'ksonnet/guestbook'
INFO  Using context 'kubernetes-admin@kubernetes' from the kubeconfig file specified at the environment variable $KUBECONFIG
INFO  Creating environment 'default' with namespace '', pointing at server at address 'https://<IP_ADDRESS>:6443'
INFO  Generating environment metadata at path 'ksonnet/guestbook/environments/default'
INFO  ksonnet app successfully created! Next, try creating a component with `ks generate`.

Then create a deployed service out of the tutorial with:
ks generate deployed-service guestbook-ui \
>    --name guestbook \
>    --image alpinejay/dns-single-redis-guestbook:0.3 \
>    --type NodePort

get the output:
INFO  Writing component at 'components/guestbook-ui'

I then inspect the ks param list and receive:
COMPONENT    PARAM         VALUE
=========    =====         =====
guestbook-ui containerPort 80
guestbook-ui image         "alpinejay/dns-single-redis-guestbook:0.3"
guestbook-ui name          "guestbook"
guestbook-ui replicas      1
guestbook-ui servicePort   80
guestbook-ui type          "NodePort"

When using ks apply default it places the guestbook-ui inside the default namespace.
Expected:
When inspecting the param list I would expect to see the namespace ks-dev listed as the namespace. Like this:
COMPONENT    PARAM         VALUE
=========    =====         =====
guestbook-ui containerPort 80
guestbook-ui image         "alpinejay/dns-single-redis-guestbook:0.3"
guestbook-ui name          "guestbook"
guestbook-ui namespace     "ks-dev"
guestbook-ui replicas      1
guestbook-ui servicePort   80
guestbook-ui type          "NodePort"

and when ks apply default is done have the deployment be created in that namespace.

Thanks for reporting @rbankston.
There does seem to be a bug where namespace doesn't seem to be registering.
The namespace, however, will be visible via the env list command as opposed to param list.
After initializing project with command ks init guestbook --namespace ks-dev
Expected:
ks env list
NAME    NAMESPACE SERVER
====    ========= ======
default ks-dev    https://kubecfg-t-apiloadb-1k1vqoxu7q212-666493100.us-west-2.elb.amazonaws.com

Currently have:
ks env list
NAME    NAMESPACE SERVER
====    ========= ======
default default    https://kubecfg-t-apiloadb-1k1vqoxu7q212-666493100.us-west-2.elb.amazonaws.com


Best way to move docker directory

Good evening,
I'm looking for the best way to move docker directory (default /var/lib/docker) to another directory. (Technically other disk partition)
I'm using devicemapper, and I think a lot of things are mounted, so it's not as  easy as a mv
I suppose the first step is to kill all the containers, then is it necessary to remove them ? Does remove unmount everything ? Is it possible to move everything including the containers ? (because deleting them is a bit heavy)
Thank you !

umount ./*

What is the working directory for this command?

I found that if there are already some previous containers in the docker, moving the directory (and adding -g as above) still won't make them run again, complaining about mont failures.
It seems there are absolute paths somewhere written in container state, could this be solved?

I've run into the same problem, that is "solved" by symlinking as the @Soulou script suggests.
It would be great to be able to change that mount failure, but at least by symlinking I don't have to rebuild everything again.

I believe that in this guide you'll have a good explanation.

You can change Docker's storage base directory (where container and images go) using the -g option when starting the Docker daemon.
Ubuntu/Debian: edit your /etc/default/docker file with the -g option: DOCKER_OPTS="-dns <IP_ADDRESS> -dns <IP_ADDRESS> -g /mnt"
Fedora/Centos: edit /etc/sysconfig/docker, and add the -g option in the other_args variable: ex. other_args="-g /var/lib/testdir". If there's more than one option, make sure you enclose them in " ". After a restart, (service docker restart) Docker should use the new directory.
Using a symlink is another method to change image storage.
Caution - These steps depend on your current /var/lib/docker being an actual directory (not a symlink to another location).

Stop docker: service docker stop. Verify no docker process is running ps faux
Double check docker really isn't running. Take a look at the current docker directory: ls /var/lib/docker/
2b) Make a backup - tar -zcC /var/lib docker > /mnt/pd0/var_lib_docker-backup-$(date +%s).tar.gz
Move the /var/lib/docker directory to your new partition: mv /var/lib/docker /mnt/pd0/docker
Make a symlink: ln -s /mnt/pd0/docker /var/lib/docker
Take a peek at the directory structure to make sure it looks like it did before the mv: ls /var/lib/docker/ (note the trailing slash to resolve the symlink)
Start docker back up service docker start
restart your containers

I was having some issues related to this, I was having permissions problems inside the containers after changing the directory. Namely, a mysql container couldn't run because it didn't have write permissions to the /tmp/ directory.
To test it, and to check if permissions were right or not, I ran a test container:
docker run -it ubuntu bash

And I checked the permissions for the /tmp/ directory:
ls -l /

Before moving the /var/lib/docker files, the permissions for /tmp/ (inside the container) were rwxrwxrwx, after moving the files and creating a test container like before, the permissions were rwxr--r--.

This is what worked for me in a VirtualBox test machine (this may help other people that arrive here).
I mounted an external disk following this http://dev-random.net/creating-mounting-partition-lager-2-tb-linux/ and this https://help.ubuntu.com/community/Fstab
The new directory ended up in: /media/disk2/var/lib/docker/
I assume you already have a directory with a similar location for your partition.

First, I stopped Docker

service docker stop

I wanted to make sure everything would work before moving or removing my Docker /var/lib/docker files, so I wanted to "copy" the files instead of "moving" them. And that was my main problem, the files were copied without the original permissions.

To copy the files preserving permissions I used rsync with the "archive" option

rsync -a /var/lib/docker/* /media/disk2/var/lib/docker/

Then I edited the /etc/default/docker file

nano /etc/default/docker

I edited and uncommented the line with

#DOCKER_OPTS="--dns <IP_ADDRESS> --dns <IP_ADDRESS>"

That line ended up like

DOCKER_OPTS="--dns <IP_ADDRESS> --dns <IP_ADDRESS> -g /media/disk2/var/lib/docker"

Then I started Docker again

service docker start

And I was able to re-start my old containers without loosing any data (I created a dummy mysql and wordpress, added changes and checked them after moving the files).

I'm trying your method here using rsync and I've found that the destination is 4-6x larger than the source. Any ideas why that might be?

Hmm, that's weird @ghostsquad, how are you measuring the sizes? What kind of file system you have in each disk?

Using current CentOS7 (3.10.0-229.14.1.el7.x86_64) it was necessary to modify /usr/lib/systemd/system/docker.service to get the -d /docker/dir/ parameter handed over to the executable:
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/docker daemon -g /docker -H fd://
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity

[Install]
WantedBy=multi-user.target

@stefanleh it's really discouraged to edit the docker.service file directly. To override default settings, use drop-in files, as explained here; https://docs.docker.com/articles/systemd/#custom-docker-daemon-options

@thaJeztah thank you for the hint.
Should'nt the EnvironmentFile=-/etc/sysconfig/docker and and ExecStart=...... $other_args lines be in the original /usr/lib/systemd/system/docker.service then by default?

@stefanleh no, various distros came back from using environment variables/files for configuration, that's why it isn't in there. However you are free to use them if you want using the method described.

Ok, but then the use of /etc/sysconfig/docker in /etc/init.d/docker should be dropped as well.
This was the biggest cause of confusion for me. I started debugging there first and wondered some minutes why my changes are'nt reflected.

+1

@thaJeztah  is this still considered as an issue?
(tiangolo commented on 30 Sep 2015)  -> The comment seems to solve issues faced by my team atleast

@tiangolo I tried the instructions you gave, but when I try to run my container I have the following:
DEBU[0057] sandbox set key processing took 121.496115ms for container 4459631a33df967d3a525688f6638529a59eb2f69b53847368f4737f48ef4853
ERRO[0057] containerd: start container                   error=oci runtime error: permission denied id=4459631a33df967d3a525688f6638529a59eb2f69b53847368f4737f48ef4853
...
ERRO[0058] Handler for POST /v1.23/containers/rtorrent-rutorrent/start returned error: Container command 'supervisord' could not be invoked.

$ docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

So it seems to not work.

I would recommend considering moving to the ZFS! See https://docs.docker.com/engine/userguide/storagedriver/zfs-driver/

@zedtux it seems you're having permissions issues. Make sure you are using rsync -a to copy the files. That will ensure that all the permissions remain the same.
If you use rsync with different options or for example cp you need to (somehow) make sure you keep the original permissions.

I've tried the other solutions listed here and they fail in one way or another, so I've modified the one from @tiangolo and it works for me on
Ubuntu 16.04.
First stop Docker
service docker stop

Copy everything from /var/lib/docker to my new destination, which is /mnt/extra in my case
cd /var/lib
sudo rsync -a docker /mnt/extra/

Remove the docker directory
sudo rm -rf docker

Create a static link
sudo ln -s docker /mnt/extra/docker

Edit the /etc/default/docker file
sudo vi /etc/default/docker

Modify this line
#DOCKER_OPTS="--dns <IP_ADDRESS> --dns <IP_ADDRESS>"

Changing it to this
DOCKER_OPTS="--dns <IP_ADDRESS> --dns <IP_ADDRESS> -g /mnt/extra/docker"

Restart Docker
service docker start

To test perms etc use this command
docker run -it ubuntu bash

If it all works, you're good to go!

It's also possible to use a daemon.json configuration file instead of /etc/default/docker. The /etc/default/docker is only used for systems running upstart, and not for systems running systemd, so is more portable. Also it allows reloading some configuration settings without restarting the daemon;
https://docs.docker.com/engine/reference/commandline/daemon/#daemon-configuration-file

@tiangolo I followed using copy/past your instructions.

I will try the one from @mondain and let you know.

On Ubuntu 16.04 (using systemd) you have to change the systemd unit file /lib/systemd/system/docker.service  and add "-g /data/docker/" to the ExecStart directive.

For some reason, the advices in these posts do not work for me: I use Ubuntu 16.04 and tried

symlink
then deleted the symlink and tried the systemd unit file
then removed what I changed in systemd and tried through a direct CIFS fstab map

Nothing works
I get in the 2nd case (systemd -g option) a chown error in relation with the tmpdir;
and in the last case (fstab) the following error at docker daemon start:
level=fatal msg="Error starting daemon: chmod /var/lib/docker: permission denied"
Any help appreciated!

On ubuntu 16.04 I've used the steps described by @mondain, which worked for me.
There is a small typo in the "ln" command, should be:
cd /var/lib sudo ln -s /mnt/extra/docker docker

Why is it not simply sufficient to start docker daemon with an argument?

And where is -g documented? The link above is broken and this page doesn't list it: https://docs.docker.com/engine/reference/commandline/dockerd/
Why is it so hard to do this? Why is it not simply as easy as passing a flag to docker daemon to tell it to look elsewhere for it's storage?

@tiangolo and @mondain I tried what has been described by @mondain but it's not working still. :(

I am having exactly the same issue as @zedtux and @tugdualenligne
I followed the instructions from @mondain exactly, and I still get this:
time="2016-07-25T13:20:55+10:00" level=error msg="containerd: start container" error="oci runtime error: permission denied" id=316deb1e9d05f8044d2759f58e83bf0f06e695f9a4b1ae011fec80f8ac22e331

In my case, I am moving from /var/lib/docker on an XFS partition (with the AUFS docker fs driver) to /mnt/data/docker on a BTRFS mount (with the BTRFS docker fs driver), I believe that is the cause of the problem.

Awesome, thank you @ashleysommer, I guess I have the same issue!
I have plex running on the machine, and I got an error due to the noexec which I have removed.
I will try again now and I'll update the issue.

I had issues with the solutions suggested here on Fedora 24.
Substituting what people had said about setting DOCKER_OPTIONS with just OPTIONS worked for me.
So I modified /etc/sysconfig/docker to contain OPTIONS="-g /mnt".

On centos 6.5 docker 1.7
rsync -a /var/lib/docker/* /mnt/docker/data/  command never compelted.

/var/lib/docker/  only 3.7G
/mnt/docker/data/ became larger and larger to 72G, and still got larger

I have to interrtpt the rsync and clean the folder. What should I do?

use rsync -a --sparse ... as suggested here https://groups.google.com/forum/#!topic/docker-user/3mUVu4dNJBU

@t-oster --sparse didn't work
[root@server1 docker]# cd /var/lib/docker
[root@server1 docker]# du -csh *
2.7M    containers
3.6G    devicemapper
1.6M    graph
5.5M    init
28K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
92M     volumes
3.7G    total
[root@server1 docker]# rsync -a -sparse /var/lib/docker/* /mnt/docker/data/

Still got larger and larger:
[root@server1 data]# du -csh *
26G     devicemapper
436K    graph
4.0K    init
24K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
324K    volumes
26G     total
[root@server1 data]# du -csh *
33G     devicemapper
436K    graph
4.0K    init
24K     linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
324K    volumes
33G     total

does the target filesystem support sparse files? Also you need to delete the target file first, so that it gets created as sparse file.

I have checked my filesystem.

df -T
/dev/vda1      ext4   20510332  16767576   2694232  87% /
tmpfs          tmpfs  16439744        60  16439684   1% /dev/shm
/dev/vdb1      ext3  309636900 206882212  87026060  71% /mnt

I search for ext3 sparse file support and filesystem support sparse file, not an article list all the filesystem which support sparse. Only seems related is http://serverfault.com/questions/558761/best-linux-filesystem-for-sparse-files says ,but it sats All files systems that support inodes (ext2/3/4, xfs, btfs, etc) support files, use inodes instead of  sparse.
Or it is the problem copy from ext4 to ext3 ?

what do you mean target file ?

/mnt/docker/data/ is clean
/var/lib/docker/ is docker images folder, I can't delete it

should work. but in your line you wrote -sparse instead of --sparse

@t-oster
sorry for late reply, I have to switch to server2 which env just like server1.
You are right， rsync doesn't eat up disk any more with  --sparse
[root@server2 docker]# rsync -a --sparse /var/lib/docker/* /mnt/docker/data/
But rsync still stumed,
/var/lib/docker is 2.3G, /mnt/docker/data/ just stop at 2.0G...
[root@server2 data]# cd /var/lib/docker
[root@server2 docker]# du -csh *
3.3M    containers
2.3G    devicemapper
1.5M    graph
5.5M    init
8.0K    linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
1004K   volumes
2.3G    total

[root@server2 data]# cd /mnt/docker/data/
[root@server2 data]# du -csh *
3.3M    containers
2.0G    devicemapper
508K    graph
4.0K    init
8.0K    linkgraph.db
4.0K    repositories-devicemapper
4.0K    tmp
4.0K    trust
4.0K    volumes
2.0G    total

it may still be complete, did you try starting docker after bind-mounting /mnt/docker over /var/lib/docker?

No, docker images show nothing after I use other_args="-g /mnt/docker/data/" .
I think the key problem is rsync never finish.

My bad,  I don't know can use -v to see rsync progress. rsync just too slow, and take serveral hours, some times my ssh connection timed out also disturb rsync.

@thaJeztah Your last suggestion worked best.  I'm on Ubuntu 16.04.1, but it shouldn't matter because this is a cross-distro solution.
Just put this json into /etc/docker/daemon.json:
{
  "graph": "/path/to/docker"
}

Worked for me and I didn't have to mess with upstart or systemd.

Any ideas on how to fix this issue on CentOS 7.3? Using Docker v1.13. I was able to start the docker service with no issues after installing with
curl -fsSL https://get.docker.com/ | sh
but after stopping the service and modifying the docker.service file to show
ExecStart=/usr/bin/dockerd daemon -g /mnt1/docker-data
then
systemctl daemon-reload
it hangs when I try systemctl start docker

See also https://linuxconfig.org/how-to-move-docker-s-default-var-lib-docker-to-another-directory-on-ubuntu-debian-linux for another writeup about this topic.

@sourcekick I followed the steps from your article and it seems to work! 👍

If you've already moved your folder using one of the numerous alternative topics on the internet, without preserving permissions, you likely will still receive:
"no space left on device, permission denied" error within the container (docker logs ).
Still haven't found a resolution... Even ran chmod and umask to give everyone full access all the time. So it can't be permissions right?
Am starting to wonder if it's the fact that the new data directory is on a separate mounted drive.

Update: -g and --graph are deprecated in 17.05, and you should be using --data-root instead now.
Also, as @thaJeztah pointed out, customize systemd by using /etc/docker/daemon.json (which appears to still use graph?)
{
  "graph": "/some/other/dir"
}

Works on 17.05.0-ce

which appears to still use graph?

Both "graph" and "data-root" should work. Even though deprecated, the old ("graph") will still work (for backward compatibility), but is hidden to encourage people to use the (less confusingly named) "data-root" name.

I was trying the above steps, but now i cant see any container/images
docker ps , docker ps -a , docker images are all empty.
However i have made sure both var/lib/docker and /new/path are having the data just in case.
Can anyone explain why I cannot see the containers?
My docker version is 1.31 , Fedora 28

@mkorycinski These file systems use an over lapping snapshot features: like overlay, overlay2, aufs, etc... So the directories in your graph dir  have "snapshots" that appear in multiple places. Some basic linux commands like du command may count this same file many many times, for images, containers, etc... I believe relying on docker system du will give you a more reliable answer. df also gives a reliable answer for how much space is available.

@Soulou @thomasleveil @tianon  @Mulkave @conradob @tiangolo  @thaJeztah @mborho @jerbia @btorresgil
cp -r  /Users/`whoami`/Library/Containers/com.docker.docker/Data /Volumes/kali/Data
rm -rf /Users/`whoami`/Library/Containers/com.docker.docker/Data
ln -s /Volumes/kali/Data /Users/`whoami`/Library/Containers/com.docker.docker/Data

now can not start docker

Steps work for me:

Stop all containers docker stop $(docker ps -a -q)

Stop docker sudo service docker stop

Copy docker directory cp -r /var/lib/docker/* /home/docker/

Unmount all old docker overlays umount -f /var/lib/docker/overlay/*/* and umount -f /var/lib/docker/containers/*/mounts/shm

Change in/etc/docker/daemon.json graph to new path
{
  "graph": "/home/docker/"
}

Start docker sudo service docker start

Optionally i remove some directory /home/docker/overlay than not used (from old containers). Working containers create two directories like: container_id and container_id-init. I removed all that hasn't init dir.

Remove old /var/lib/docker/

Steps work for me:

Stop all containers docker stop $(docker ps -a -q)
Stop docker sudo service docker stop
Copy docker directory cp -r /var/lib/docker/* /home/docker/
Unmount all old docker overlays umount -f /var/lib/docker/overlay/*/* and umount -f /var/lib/docker/containers/*/mounts/shm
Change in/etc/docker/daemon.json graph to new path{
  "graph": "/home/docker/"
}

Start docker sudo service docker start
Remove old /var/lib/docker/

I didn't find daemon.json inside /etc/docker/, there was only key.json. Should I add it myself?

@fsevenm Yes, if you have no daemon.json file, add one yourself. By default, there isn't one, but some plugins/packages add other fields for you.

For me "graph" in daemon.json needed to be "data-root". Otherwise, it worked.
My version:
$ docker version
Client:
 Version:           18.09.5
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        e8ff056
 Built:             Thu Apr 11 04:44:24 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.5
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       e8ff056
  Built:            Thu Apr 11 04:10:53 2019
  OS/Arch:          linux/amd64
  Experimental:     false

For me (using ubuntu 20.04 server on arm64 Pi4 and docker v19.03.08) the recommended "data-root" did not work and I wasn't able to restart the docker service. Using the deprecated "graph" did the trick. Additionally I had to add a line for the storage-driver to get the new location on my external hard drive working.
/etc/docker/daemon.json
{
    "graph": "/media/nextcloud/mariaDB/docker",
    "storage-driver": "devicemapper"
}

docker version
Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Fri Dec 18 12:16:35 2020
 OS/Arch:           linux/arm64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Fri Dec  4 23:02:49 2020
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2.2
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:

hard drive format is fuseblk (ntfs)

For me (using ubuntu 20.04 server on arm64 Pi4 and docker v19.03.08) the recommended "data-root" did not work and I wasn't able to restart the docker service. Using the deprecated "graph" did the trick. Additionally I had to add a line for the storage-driver to get the new location on my external hard drive working.
/etc/docker/daemon.json
{
    "graph": "/media/nextcloud/mariaDB/docker",
    "storage-driver": "devicemapper"
}

docker version
Client:
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.13.8
 Git commit:        afacb8b7f0
 Built:             Fri Dec 18 12:16:35 2020
 OS/Arch:           linux/arm64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.8
  Git commit:       afacb8b7f0
  Built:            Fri Dec  4 23:02:49 2020
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.3.3-0ubuntu2.2
  GitCommit:
 runc:
  Version:          spec: 1.0.1-dev
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:

hard drive format is fuseblk (ntfs)

the recommended "data-root" did not work

That option should work (tested on docker 19.03.13 and docker 20.10.2);
$ cat /etc/docker/daemon.json
{"debug":true, "data-root": "/var/lib/bla"}

$ docker info | grep 'Root Dir'
 Docker Root Dir: /var/lib/bla

However, from the output of your docker version, I suspect you may be running the distro built packages, not the official docker packages from download.docker.com, so it's possible there's a bug/issue with those packages

the recommended "data-root" did not work

That option should work (tested on docker 19.03.13 and docker 20.10.2);
$ cat /etc/docker/daemon.json
{"debug":true, "data-root": "/var/lib/bla"}

$ docker info | grep 'Root Dir'
 Docker Root Dir: /var/lib/bla

However, from the output of your docker version, I suspect you may be running the distro built packages, not the official docker packages from download.docker.com, so it's possible there's a bug/issue with those packages

Instructions by @bigspawn worked pretty well, I had to copy the contents of /var/lib/docker using rsync as suggested by @tiangolo. With plain old cp I was having issues with grafana and permissions, rsync magically resolved it.
One other thing I wanted to mention in case it helps someone:
If /etc/docker/daemon.json already has content in it, don't forget to put a comma at the end of each! Example below.
{
    "graph": "/my-new-drive/docker/", # <------ dont forget this comma!
    "foo": "bar"
}

While following the steps, starting docker with systemctl start docker was failing because I had forgot the comma :D

For a hard drive with NTFS format, the below configuration worked for me.
:~# docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., v2.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 2
 Server Version: 20.10.17
 Storage Driver: devicemapper
  Pool Name: docker-8:4-5026-pool

daemon.json
{
   "data-root": "/mnt/ntfd_drive/docker_data/",
   "storage-driver": "devicemapper"
}

I base my solution on @tiangolo  and the docker documentation.
> docker version

Client:
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.2
 Git commit:        20.10.12-0ubuntu2~20.04.1
 Built:             Wed Apr  6 02:14:38 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

# Steps:
> sudo su

> service docker status | stop
# If the docker is still runing becaouse of the docker.socket.
> systemctl stop docker.socket

# On my case "/home/diuble-lig/docker/" is my folder destination.
> rsync -a /var/lib/docker/* /home/diuble-lig/docker/
# Wait some minutes until the folder finish the creation

> nano /lib/systemd/system/docker.service

# Add --data-root parameter
ExecStart=/usr/bin/dockerd --data-root /home/diuble-lig/docker -H fd://

> systemctl daemon-reload
> service docker restart
> service docker status

The gist of the solution is placed here:

https://gist.github.com/realFranco/36d2245295fabc9a56c5f8c870d8cef0

Steps work for me:
1. Stop all containers `docker stop $(docker ps -a -q)`

2. Stop docker `sudo service docker stop`

3. Copy docker directory `rsync -a /var/lib/docker/* /home/docker/`

4. Unmount all old docker overlays `umount -f /var/lib/docker/overlay/*/*` and `umount -f /var/lib/docker/containers/*/mounts/shm`

5. Change in`/etc/docker/daemon.json` graph to new path
   ```json
   {
     "data-root": "/home/docker/"
   }
   ```

6. Start docker `sudo service docker start`

7. Remove old `/var/lib/docker/`

I've quoted and updated @bigspawn's instructions above. I've changed graph to data-root and cp to rsync. There were two more modifications I considered but didn't make:

Personally, I didn't need to unmount anything, but that can't hurt so I've left it in.
I found it confusing that sudo is specified explicitly for steps 2 and 6, when it will be required for everything except step 1. I'll leave it like that, in the interests of discouraging blind copy-pasting sudo commands.