Kubernetes > 1.22

Problem
Hi, we are running a recent version of Kubernetes, and for some reason, scrapandre does not pick up any pods. I've looked into it a bit, and it might be related to the kube-sync crate. It is using a fairly old version of k8s-openapi that only supports Kubernetes 1.22. Looking into this a bit more, there may be an issue with the kube-sync in the near future as the k8s-openapi removed some of the used features in favor of the kube library (API clients) see here.
Solution
Thus, it might be better to migrate away from the kube-sync crate to the kube crate to stay up to date with newer Kubernetes releases.
Alternatives
Update the kube-sync to work with newer versions of Kubernetes.

I have a cluster on k8s 1.26.8 with scaphandre docker image with tag dev and it works well.

We are running v1.29. It could also be a different reason why the pods don't show up. However, the deprecation of the kube-openapi API deprecation will be an issue sooner or later.

The error i see in the logs is this btw:
scaphandre::exporters: watching kubernetes...
isahc::client: send; method=GET uri=https://<IP_ADDRESS>:443/api/v1/namespaces//pods?
isahc::handler: handler;
isahc::handler: handler; id=0
isahc::handler:   Trying <IP_ADDRESS>:443...
isahc::handler: Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 443 (#0)
isahc::handler: ALPN: offers h2,http/1.1
isahc::handler: TLSv1.3 (OUT), TLS handshake, Client hello (1):
isahc::handler: TLSv1.3 (IN), TLS handshake, Server hello (2):
isahc::handler: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
isahc::handler: TLSv1.3 (IN), TLS handshake, Request CERT (13):
isahc::handler: TLSv1.3 (IN), TLS handshake, Certificate (11):
isahc::handler: TLSv1.3 (IN), TLS handshake, CERT verify (15):
isahc::handler: TLSv1.3 (IN), TLS handshake, Finished (20):
isahc::handler: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
isahc::handler: TLSv1.3 (OUT), TLS handshake, Certificate (11):
isahc::handler: TLSv1.3 (OUT), TLS handshake, Finished (20):
isahc::handler: SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
isahc::handler: ALPN: server accepted h2
isahc::handler: Server certificate:
isahc::handler:  subject: CN=kube-apiserver
isahc::handler:  start date: Jan  5 14:52:55 2024 GMT
isahc::handler:  expire date: Jan  4 14:57:56 2025 GMT
isahc::handler:  subjectAltName: host "<IP_ADDRESS>" matched cert's IP address!
isahc::handler:  issuer: CN=kubernetes
isahc::handler:  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
isahc::handler: using HTTP/2
isahc::handler: h2h3 [:method: GET]
isahc::handler: h2h3 [:path: /api/v1/namespaces//pods?]
isahc::handler: h2h3 [:scheme: https]
isahc::handler: h2h3 [:authority: <IP_ADDRESS>]
isahc::handler: h2h3 [accept: */*]
isahc::handler: h2h3 [accept-encoding: deflate, gzip]
isahc::handler: h2h3 [authorization: Bearer OMITTED]
isahc::handler: h2h3 [user-agent: curl/8.0.1-DEV isahc/1.7.2]
isahc::handler: h2h3 [content-length: 0]
isahc::handler: Using Stream ID: 1 (easy handle 0x563c2768d6d0)
isahc::handler: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
isahc::handler: Connection #0 to host <IP_ADDRESS> left intact

I have the exact same problem on kubernetes v1.28.8. The Kubernetes API call hangs in my case. I think this is caused by the empty namespace in the URL, because hardcoded specific namespace name works fine. So it is not possible to list any pod.
According to the Kubernetes documentation, specifying the namespace in the URL is required. And from what I found, it looks like the methods used from the k8s-sync and k8s-openapi libraries are designed to specify non-empty namespace name.
isahc::client: send; method=GET uri=https://localhost:6443/api/v1/namespaces//pods?
isahc::handler: handler;
isahc::handler: handler; id=0
isahc::handler:   Trying <IP_ADDRESS>:6443...
isahc::handler: Connected to localhost (<IP_ADDRESS>) port 6443 (#0)
isahc::handler: ALPN: offers h2,http/1.1
isahc::handler: TLSv1.3 (OUT), TLS handshake, Client hello (1):
isahc::handler: TLSv1.3 (IN), TLS handshake, Server hello (2):
isahc::handler: TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
isahc::handler: TLSv1.3 (IN), TLS handshake, Request CERT (13):
isahc::handler: TLSv1.3 (IN), TLS handshake, Certificate (11):
isahc::handler: TLSv1.3 (IN), TLS handshake, CERT verify (15):
isahc::handler: TLSv1.3 (IN), TLS handshake, Finished (20):
isahc::handler: TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
isahc::handler: TLSv1.3 (OUT), TLS handshake, Certificate (11):
isahc::handler: TLSv1.3 (OUT), TLS handshake, CERT verify (15):
isahc::handler: TLSv1.3 (OUT), TLS handshake, Finished (20):
isahc::handler: SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
isahc::handler: ALPN: server accepted h2
isahc::handler: Server certificate:
isahc::handler:  subject: CN=kube-apiserver
isahc::handler:  start date: Nov  1 17:11:01 2021 GMT
isahc::handler:  expire date: Oct 31 00:59:44 2024 GMT
isahc::handler:  subjectAltName: host "localhost" matched cert's "localhost"
isahc::handler:  issuer: CN=rke2-server-ca@1635786661
isahc::handler:  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
isahc::handler: using HTTP/2
isahc::handler: h2h3 [:method: GET]
isahc::handler: h2h3 [:path: /api/v1/namespaces//pods?]
isahc::handler: h2h3 [:scheme: https]
isahc::handler: h2h3 [:authority: localhost:6443]
isahc::handler: h2h3 [accept: */*]
isahc::handler: h2h3 [accept-encoding: deflate, gzip]
isahc::handler: h2h3 [user-agent: curl/8.0.1-DEV isahc/1.7.2]
isahc::handler: h2h3 [content-length: 0]
isahc::handler: Using Stream ID: 1 (easy handle 0x561ea8743f70)
isahc::handler: TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

I was wrong. The empty namespace in the URL is not the problem. My problem was related with freezing of the parsing of the Kubernetes API response by the k8s-sync library. This problem occurred in my case if the response was too large.
I created a patch in the k8s-sync library. Using my patch the parsing works fine.
After finding this, I am not sure if my problem is related with this issue, so apologize for spam. But still, I think that the parsing function in the k8s-sync is not ideal.


Error message when using the "-j" option

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm requesting a feature unrelated to a specific site
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2023.02.17 (update instructions) or later (specify commit)
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood
This is about entering a URL that is not supported by the current version.
For example, suppose you execute the following command
.\yt-dlp.exe -j https://google.com
At this time, the error message displayed is as follows
WARNING: [generic] Falling back on generic information extractor
ERROR: Unsupported URL: https://www.google.com/

The "-j" option is used to request output in Json format, but it does not seem to output in Json format if the URL does not support it.
My request would be to output the following when the "-j" option is used, but what do you think?
{
    error: true
}

I have looked through the README.md and am sorry if there are any options I missed.
Also, I am not a good English speaker so I am using the help of a translation tool.
I apologize if I have written rude English.
Thank you for reading this far.
Provide verbose output that clearly demonstrates the problem

[ ] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[ ] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
No response

https://github.com/yt-dlp/yt-dlp/issues/1659


Fix mass-assignment error when using protected_attributes

The following code does not work as intended because we use protected_attributes gem in Rails 4:
SimpleCaptchaData.new(key: key)

Sorry that it took so long!
I've merged your changes and fixed all test issues & released new Gem version 0.4.0


GitHub sync failed for Pulumi

The GitHub Sync job failed today:
 /home/runner/work/_actions/pulumi/actions/4204b4e8a7e703da96ba5dd4c3a667adeee35812/webpack:/pulumi-github-action/node_modules/@pulumi/pulumi/automation/errors.js:77
                      : new CommandError(result);
  ^
  CommandError: code: -2
   stdout:
   stderr: Command failed with exit code 255: pulumi up --yes --skip-preview --diff --exec-agent pulumi/actions@v3 --color auto --exec-kind auto.local --stack sigstore/github-prod --non-interactive
  [resource plugin github-5.12.0] installing
  error: this resource can only be used in the context of an organization, "sigstore" is a user
  error: this resource can only be used in the context of an organization, "sigstore" is a user
  error: update failed

Originally posted by @evankanderson in https://github.com/sigstore/community/issues/286#issuecomment-1612122451

See https://github.com/integrations/terraform-provider-github/issues/769 for some thoughts. Someone said it could be an expired token?
cc @cpanato

Working now! It was a token, had missed the open PR to update the token. https://github.com/sigstore/community/actions/runs/5406065230/jobs/9822469211


Simple image building with BuildImageFromDockerfileAsync does not build an image

Output of dotnet --info:
.NET SDK:
 Version:   7.0.100
 Commit:    e12b7af219

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19044
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\7.0.100\

Host:
  Version:      7.0.0
  Architecture: x64
  Commit:       d099f075e4

.NET SDKs installed:
  2.1.700 [C:\Program Files\dotnet\sdk]
  2.1.701 [C:\Program Files\dotnet\sdk]
  2.2.300 [C:\Program Files\dotnet\sdk]
  2.2.301 [C:\Program Files\dotnet\sdk]
  6.0.100-preview.1.21103.13 [C:\Program Files\dotnet\sdk]
  7.0.100 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0-preview.1.21103.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0-preview.1.21102.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0-preview.5.21301.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 6.0.0-preview.1.21103.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 7.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
  arm64 [C:\Program Files\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\arm64\InstallLocation]
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

What version of Docker.DotNet?:
3.125.12

Steps to reproduce the issue:
Just run below code as a ConsoleApp (ofc. you need docker deamon running):
using Docker.DotNet;
using Docker.DotNet.Models;
using System.Text;
using ICSharpCode.SharpZipLib.Tar;

var dockerEngineUrl = new Uri("npipe://./pipe/docker_engine");
using var client = new DockerClientConfiguration(dockerEngineUrl).CreateClient();

const string imageName = "myimage";
const string dockerfilePath = "./../../../source";
await BuildImage(client, imageName, dockerfilePath);

async Task BuildImage(IDockerClient client, string imageName, string dockerFilePath, bool rebuild = false)
{
    if (!rebuild)
        rebuild = !await ImageAlreadyExists(client, imageName); //If should not rebuild existing image, check if it exists. If yes then do not build.

    if (rebuild)
    {
        Console.WriteLine($"Image named '{imageName}' not found. Building image from {dockerFilePath}. {DateTime.Now.ToLongTimeString()}");

        var imageBuildParameters = new ImageBuildParameters
        {
            Dockerfile = dockerFilePath
        };

        var progress = new Progress<JSONMessage>();
        progress.ProgressChanged += (sender, e) =>
        {
            Console.WriteLine($"From: {e.From}");
            Console.WriteLine($"Status: {e.Status}");
            Console.WriteLine($"Stream: {e.Stream}");
            Console.WriteLine($"ID: {e.ID}");
            Console.WriteLine($"Progress: {e.ProgressMessage}");
            Console.WriteLine($"Error: {e.ErrorMessage}");
        };

        var stream = CreateTarFileForDockerfileDirectory(dockerFilePath);

        await client.Images.BuildImageFromDockerfileAsync(imageBuildParameters, stream, null, null, progress);

        Console.WriteLine($"Image named '{imageName}' built. {DateTime.Now.ToLongTimeString()}");
    }
}

async Task<bool> ImageAlreadyExists(IDockerClient client, string imageName)
{
    var listImagesParameters = new ImagesListParameters { All = true };
    var images = await client.Images.ListImagesAsync(listImagesParameters);
    return images.Any(image => image.RepoTags.Contains(imageName));
}

Stream CreateTarFileForDockerfileDirectory(string directory)
{
    var stream = new MemoryStream();
    var filePaths = Directory.GetFiles(directory, "*.*", SearchOption.AllDirectories);

    using var archive = new TarOutputStream(stream, Encoding.UTF8);

    //Prevent the TarOutputStream from closing the underlying memory stream when done
    archive.IsStreamOwner = false;

    //Add files to tar archive
    foreach (var file in filePaths)
    {
        var tarName = Path.GetFileNameWithoutExtension(file);

        var entry = TarEntry.CreateTarEntry(tarName);
        using var fileStream = File.OpenRead(file);
        entry.Size = fileStream.Length;
        archive.PutNextEntry(entry);

        var localBuffer = new byte[32 * 1024];

        while (true)
        {
            var numberOfBytesSavedToBuffer = fileStream.Read(localBuffer, 0, localBuffer.Length);
            if (numberOfBytesSavedToBuffer <= 0)
                break;

            archive.Write(localBuffer, 0, numberOfBytesSavedToBuffer);
        }
        archive.CloseEntry();
    }

    archive.Close();

    stream.Position = 0;
    return stream;
}

Source directory content:
Dockerfile:
FROM alpine
ADD test.txt .
ENTRYPOINT ["/bin/cat", "test.txt"]

test.txt:
Hello from test.txt

What actually happened?:
BuildImageFromDockerfileAsync went through without an error and there is no "myimage" visible under docker image list.
What did you expect to happen?:
"myimage" visible under docker image list
Additional information:

Accordingly to your other questions in the past days you might wanna take a look at Testcontainers. It looks like that it covers everything you are trying to implement.

Can you please give me an example on how you build your tarball?

Here you go.

I also struggled a bit to build an image from a Dockerfile, but it works. For me, the problem was that I didn't specify the 'Tags' property in the ImageBuildParameters.
    private static Stream CreateTarballForDockerfileDirectory(string directory)
    {
        var tarball = new MemoryStream();
        var files = Directory.GetFiles(directory, "*.*", SearchOption.AllDirectories);

        using var archive = new TarOutputStream(tarball)
        {
            //Prevent the TarOutputStream from closing the underlying memory stream when done
            IsStreamOwner = false
        };

        foreach (var file in files)
        {
            //Replacing slashes as KyleGobel suggested and removing leading /
            string tarName = file.Substring(directory.Length).Replace('\\', '/').TrimStart('/');

            //Let's create the entry header
            var entry = TarEntry.CreateTarEntry(tarName);
            using var fileStream = File.OpenRead(file);
            entry.Size = fileStream.Length;
            archive.PutNextEntry(entry);

            //Now write the bytes of data
            byte[] localBuffer = new byte[32 * 1024];
            while (true)
            {
                int numRead = fileStream.Read(localBuffer, 0, localBuffer.Length);
                if (numRead <= 0)
                    break;

                archive.Write(localBuffer, 0, numRead);
            }

            //Nothing more to do with this entry
            archive.CloseEntry();
        }
        archive.Close();

        //Reset the stream and return it, so it can be used by the caller
        tarball.Position = 0;
        return tarball;
    }

    public async Task BuildDockerImage()
    {
        try
        {
            var workingDirectory = "YOUR_WORKING_DIRECTORY";

            await using var tarball = CreateTarballForDockerfileDirectory(workingDirectory);

            IList<string> tags = new List<string>()
            {
                "my-image"
            };

            var imageBuildParameters = new ImageBuildParameters
            {
               Tags = tags
            };

            var progress = new Progress<JSONMessage>();

            using var dockerClient = new DockerClientConfiguration(new Uri("npipe://./pipe/docker_engine")).CreateClient();
            await dockerClient.Images.BuildImageFromDockerfileAsync(imageBuildParameters, tarball, null, null, progress);

        }
        catch (Exception ex)
        {
            Debug.WriteLine("[ERROR] Error during docker image build");
        }
    }

ImageBuildParameters.Dockerfile is relative to the Dockerfile inside the tarball. It is also the file path to the Dockerfile

Can you explain what that means? How can a path to a file in a file system be relative to a file in a tarball?

Can you explain what that means? How can a path to a file in a file system be relative to a file in a tarball?

To build a Docker image, the Docker daemon receives a tarball that contains all necessary files. The Dockerfile arg represents the path inside the tarball.

Where can I find these tests? I don't see any calls to any BuildImageFromDockerfileAsync() override in the test project.

It is part of Testcontainers for .NET.

Thanks for the clarification! Also, sorry -- I thought you were referring to the Docker.DotNet tests.

With your setup (of course with fixed Dockerfile path) i still get an error. The BuildImageFromDockerfileAsync Method does not wait for the Build process to end. Even If it is awaited it just returns after step 7/19 without an error in the build process for me.
Wasn't this supposed to be fixed with Issue #445 and Pull Request #507 ?
If not the Obsolete Annotation for the old BuildImageFromDockerfileAsync Method is wrong, as the new one does not wait either.
(IImageOperations.cs)
Task BuildImageFromDockerfileAsync(ImageBuildParameters parameters, Stream contents, IEnumerable<AuthConfig> authConfigs, IDictionary<string, string> headers, IProgress<JSONMessage> progress,  CancellationToken cancellationToken = default(CancellationToken));

[ObsoleteAttribute("This method does not wait for build to complete. Use new BuildImageFromDockerfileAsync(contents, parameters, authConfigs, headers, progress, cancellationToken,  instead.)", false)]
Task<Stream> BuildImageFromDockerfileAsync(Stream contents, ImageBuildParameters parameters, CancellationToken cancellationToken = default(CancellationToken));

Got it to work using #423


We're sorry, there has been an error. To get help, please use the Help > Report Issues menu option and include the following information: Session d5673be7-f770-4a3f-9fc4-37dadb6ba693. Failed request to assessment endpoint with status code 403

Investigative information
Please provide the following:

UTC Timestamp:
Migration Tool App version :
OS Version:

Repro steps
Provide the steps required to reproduce the problem:

Expected behavior
Provide a description of the expected behavior.

Actual behavior
Provide a description of the actual behavior observed.

Known workarounds
Provide a description of any known workarounds.

Hi @shannuvijaya - thank you for letting us know this happened. I haven't yet been able to reproduce the behavior, so this may be something specific to your setup that I'm not matching. I have also confirmed that other requests to the endpoint do appear to be getting through even very close to that time period, so the endpoint does not appear to have been down (or not for long). Does there happen to be any particular network intermediaries that might block or adjust requests between your client machine and the internet? Does browsing https://readinessapi.trafficmanager.net/ yourself from the machine also return a 403? I ask since I don't see the logs that would indicate the request made it all the way to the assessment site.

An alternative option now also would be to try using the PowerShell scripts that were recently released - these do not require outbound connectivity to do the assessment step (and will assess all sites on the server at once) - the scripts are a lightweight option that involves simply dropping a directory of scripts on the machine (or at least a machine-accessible location) in order to try it out.
Main info page: https://github.com/Azure/App-Service-Migration-Assistant/wiki/PowerShell-Scripts
Place to download: https://azure.microsoft.com/en-us/services/app-service/migration-assistant/


Bug/515 notifications from blocked users

Also fixes https://github.com/OkunaOrg/okuna-api/issues/489

Can you cherry pick the commit from the video branch so tests get cleared?

Issue https://github.com/OkunaOrg/okuna-api/issues/515 can be closet, too?


Update documentation

We don't use EIP anymore, so we need an alternative way for SSH'ing into the instances.
As this is nothing critical, and it is just documentation, I tried to keep things
as simple as possible. We can get more detailed later on, if we think the effort involved (and added complexity) is worth. For now, we have bigger fish to fry.
Also, with the upcoming modularisation things may change again, so there is no point on spending time on something we may need to undo soon.
Also, it should be quite unlikely devs need to SSH into the instances - probably only if there are issues - and that should not be that frequent, hopefully.

It might be worth adding instructions on how to get the master EC2 public dns from the AWS console (how to find the correct EC2, say filtering on tags).

After a chat with @poveyd, it seems we currently don't have a tag we can filter by - so we can address that in a new PR, after merging this one.


Install scripts only support deb distro for now

[  ]  ./install.sh should auto detect the distro release version
[  ] upload artifact for rh distros.


[common-utils]: Bug fix: Install zsh on an image previously built with "installZsh:false"

Closes: https://github.com/devcontainers/features/issues/647
Testes locally with an image provided in https://github.com/devcontainers/features/issues/647#issuecomment-1674386348. Works as expected!
Unfortunately, I wasn't able to write a test for such a complex scenario, feedbacks on how to write a test are welcome 🙏

Please also see https://github.com/devcontainers/features/pull/653

The important bit is that installing zsh won't be blocked if PACKAGES_ALREADY_INSTALLED?

@joshspicer Yes, that's correct!


datadog 6: logging agent

Issue description
Datadog6 also has a "logging agent" (see agent status) output, which returns:
Logs Agent is not running.
The expected response is that it is running when it is instructed to run via:
    services.datadog-agent.extraConfig = {
      logs_enabled = true;
    };

Steps to reproduce
agent status | grep 'Logs Agent is not running'

Related people
@rvl @domenkozar
Technical details
master

this works in v7


Svelte-kit SPA with adapter static and globally disabled prerender and ssr stills executes app code on node.js

Describe the bug
According to this (taken from official docs) load functions defined on +page.ts when ssr = false and prerender = false would only run on the client.
This is not true, in the liked repo I used browser-only code carelessly within the load function and during the build I got
ReferenceError [Error]: Worker is not defined, meaning the load function is being executed on the server despite prerendering and ssr being disabled, contrary to what the docs state.

Reproduction
I made a minimal reproducible example based on the svelte-kit demo app.
The key files are src/lib/worker.ts, src/lib/worker_handler.ts and src/routes/+page.ts
https://github.com/kran6a/spa-bug
Logs
><EMAIL_ADDRESS>build
> vite build

vite v5.0.11 building SSR bundle for production...
"confetti" is imported from external module "@neoconfetti/svelte" but never used in "src/routes/sverdle/+page.svelte".
✓ 108 modules transformed.

node:internal/event_target:1096
  process.nextTick(() => { throw err; });
                           ^
ReferenceError [Error]: Worker is not defined
    at new WorkerWrapper (file://spa-bug/.svelte-kit/output/server/chunks/worker_handler.js:2:3)
    at file://spa-bug/.svelte-kit/output/server/chunks/worker_handler.js:9:16
    at ModuleJob.run (node:internal/modules/esm/module_job:218:25)
    at async ModuleLoader.import (node:internal/modules/esm/loader:329:24)
    at async Promise.all (index 0)
    at async analyse (file://spa-bug/node_modules/@sveltejs/kit/src/core/postbuild/analyse.js:58:16)
    at async MessagePort.<anonymous> (file://spa-bug/node_modules/@sveltejs/kit/src/utils/fork.js:22:16)

Node.js v21.5.0

System Info
System:
    OS: Windows 11 10.0.22621
    CPU: (12) x64 AMD Ryzen 5 3600XT 6-Core Processor
    Memory: 22.43 GB / 63.93 GB
  Binaries:
    Node: 21.5.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.21 - ~\AppData\Roaming\npm\yarn.CMD
    npm: 10.2.4 - C:\Program Files\nodejs\npm.CMD
    pnpm: 8.14.0 - ~\AppData\Roaming\npm\pnpm.CMD
  Browsers:
    Edge: Chromium (120.0.2210.133)
    Internet Explorer: 11.0.22621.1
  npmPackages:
    @sveltejs/adapter-auto: ^3.0.0 => 3.1.0
    @sveltejs/adapter-static: ^3.0.1 => 3.0.1
    @sveltejs/kit: ^2.0.0 => 2.3.5
    @sveltejs/vite-plugin-svelte: ^3.0.0 => 3.0.1
    svelte: ^4.2.7 => 4.2.9
    vite: ^5.0.3 => 5.0.11

Severity
blocking all usage of SvelteKit
Additional Information
No response

The load function will only be called on the client, but +page.ts still needs to be evaluated during the build to find out what it exports, and that imports code that refers to browser-only APIs at the top-level, and so fails to run on the server.
You will either need to import worker_handler.js lazily only while load is being called, or you will need to adjust worker_handler.js so importing it doesn't eagerly try to use browser APIs.

The load function will only be called on the client, but +page.ts still needs to be evaluated during the build to find out what it exports, and that imports code that refers to browser-only APIs at the top-level, and so fails to run on the server.
You will either need to import worker_handler.js lazily only while load is being called, or you will need to adjust worker_handler.js so importing it doesn't eagerly try to use browser APIs.

But there is no need to import the whole file as is just to check some exports. Exports could be checked in the corresponding AST and code could be treeshaken so that anything not needed to compute those exports' values is never executed on node.
I get that it is way more complex than just importing the whole file but it is a pain for people developing client-side SPAs that do not need to prerender. Plus people won't be expecting their build tool to execute the code. At least I wasn't expecting that behavior with adapter-static despite having experience with svelte and svelte-kit (this was my first SPA that did not need prerendering).
If it is not planned I would at least state in the docs about adapter-static that code in +page.ts and +layout.ts files will be executed in node even if you are using adapter-static as it is a big point to consider for people developing no prerendering 100% client-side websites (eg: embedded devices, IoT config/monitoring, kiosks, ...).

I have just hit the same problem. Our app is client-side only. It would be nice if adapter-static had an option to avoid running +page.ts during build-time.


unpack_bytes_verified does not advance the underlying reader

Bug description
The unpack_bytes_{un,}verified methods do not advance the underlying slice while unpacking slices of bytes.
To reproduce:
use packable::unpacker::SliceUnpacker;
use packable::{Packable, PackableExt};

fn main() {
    let src: Vec<u8> = (0..10).collect();
    // This advances the underlying reader
    let mut unpacker = SliceUnpacker::new(src.as_slice());
    for byte in &src {
        let unpacked = u8::unpack(&mut unpacker, None).unwrap();
        assert_eq!(*byte, unpacked);
    }
    // This does not advance the underlying reader
    let bytes = src.as_slice();
    for byte in &src {
        let unpacked = u8::unpack_bytes_verified(bytes, &()).unwrap();
        // will panic after first iteration
        assert_eq!(*byte, unpacked);
    }
}

This is because of the use of bytes.as_ref that copies the underlying slice address into a new address. Hence when we pass slice: &mut bytes.as_ref() into a function that at some point calls *slice = new_slice, the original bytes slice is not affected. This could help illustrate the argument:
let src: Vec<u8> = (0..10).collect();
let bytes = src.as_slice();
let new_bytes = bytes.as_ref();
assert!(format!("{:p}", &bytes) != format!("{:p}", &new_bytes));

If this is the intended behavior, I reckon it should be documented.
Otherwise you could consider using &[u8] instead of AsRef<u8> to ensure that the underlying reader is advanced, which seems like the expected behavior.
Rust version
1.76

You can't advance a slice, so I'm not sure what you expect from this. You need the cursor to do so.

You can't advance a slice, so I'm not sure what you expect from this. You need the cursor to do so.

That is certainly true.
The motivation for this report comes from dealing with the 0.4.0 version of the library, that implemented Unpacker for &mut [u8] allowing one to use both Packable and PackableExt directly on a mutable slice with inconsistent side-effects.
I came across this issue while going over Thoralf's (@Thoralf-M)  first take on the stardust-snapshot parser. In particular, I couldn't figure out why this part does not introduce parsing errors, since the vector of Outputs comes immediately after the FullSnapshotHeader.
With mut slice: &[u8] one would very much expect that unpack::<_, _>(&mut slice.as_ref()) wrapped in the PackableExt would have the same side effects as unpack::<_, _>(&mut slice). But this is not the case for the reasons I discussed in the issue's description.
Nevertheless, the introduction of the SliceUnpacker in later versions overcomes indeed this problem and as you point out one shouldn't expect any side effects on the slice itself when using PackableExt. Feel free to close the issue.


merge package from arlac77/npm-package-template

package.json

chore(package<EMAIL_ADDRESS>chore(scripts): cover@#overwrite c8 --temp-directory build/coverage ava && c8 report -r lcov --temp-directory build/coverage
chore(package): add nyc from template
chore(package): set $.ava.require='esm' as in template
chore(package): set $.ava.files='tests/-test.js,tests/-test.mjs' as in template
chore(package): set $.ava.extensions='js,mjs' as in template

Codecov Report

Merging #455 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master   #455   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files           1      1
  Lines          29     29
=====================================
  Hits           29     29

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c3ea82c...d0285bd. Read the comment docs.

Coverage remained the same at 97.619% when pulling d0285bdc0717d0a2c6313a38c971d056b7b9f21c on template-sync-1 into c3ea82c1f57963485fbc5c02a75101e4fb22ca2e on master.


Investigate the creation of an AnyM2 type

This would represent a monad with 2 type parameters (such as Reader, Either, Try, Xor, Ior), and would facilitate type safe implementation of ReaderT, XorT, TryT etc.

AnyM2 added in 2.0.0-MI4


SRC-6 Vault standard

Abstract
The following standard allows for the implementation of a standard API for token vaults such as yield bearing token vaults. This standard is an optional add-on to the SRC-20 standard.
Motivation
Token vaults allow users to own shares of variable amount of assets, such as lending protocols which may have growing assets due to profits from interest. This pattern is highly useful and would greatly benefit from standardisation
Prior Art
Token vaults have been thoroughly explored on Ethereum and with EIP 4626 they have their own standard for it. However as Fuel's native assets are fundamentally different to Ethereum's ERC-20 tokens, the implementation will differ, but the interface may be used as reference.
Specification
Required public functions
The following functions MUST be implemented (on top of the SRC-20 functions) to follow the SRC-4 standard
fn deposit(receiver: Identity)
Method that allows depositing of the underlying asset in exchange for shares of the vault.
MUST revert if any AssetId  other than the underlying is forwarded.
MUST mint preview_deposit(deposited_assets) amount of shares to receiver
MUST increase total_assets by deposited_assets (through any means including std::context::this_balance(ASSET_ID) if applicable)
MUST increase total_supply by newly minted shares
fn withdraw(receiver: Identity)
Method that allows the redeeming of the vault shares in exchange for a pro-rata amount of the underlying asset
MUST revert if any AssetId other than the AssetId of the self contract is forwarded.
MUST send preview_withdraw(redeemed_shares) amount of assets to receiver
MUST burn the received shares
MUST reduce total_assets by preview_withdraw(redeemed_shares)
MUST reduce total_supply by amount of burnt shares
fn total_assets() -> u64
Method that returns the total assets under management by vault. Includes assets controlled by the vault but not directly possessed by vault
MUST return total amount of assets of underlying AssetId under management by vault
fn convert_to_shares(assets: u64) -> u64
Helper method for converting
The parameters of the functions may change depending on the implementation of native multi tokens on Fuel.
Required logs
The following logs MUST be emitted at the specified occasions
pub struct Deposit {
    caller: Identity,
    receiver: Identity,
    assets: u64,
    shares: u64,
}

caller has called the deposit method sending assets assets of the underlying asset_id, in exchange for shares shares sent to the receiver receiver
The Deposit struct MUST be logged whenever new shares are minted via the deposit method
pub struct Withdraw {
    caller: Identity,
    receiver: Identity,
    assets: u64,
    shares: u64,
}

caller has called the withdraw method sending shares shares in exchange for assets assets to the receiver receiver
The Withdraw struct MUST be logged whenever shares are redeemed for assets via the withdraw method
Rationale
The ABI discussed is simple and covers the known use cases of token vaults while allowing safe implementations
Backwards compatibility
This standard is fully compatible with the SRC-20 standard
Security Considerations
Incorrect implementation of token vaults could allow attackers to steal underlying assets. It is recommended to properly audit any code using this standard to ensure exploits are not possible.
Reference implementation
Full reference implementation can be seen here
This is a draft standard

Closed by https://github.com/FuelLabs/sway-standards/pull/24


R package install with GPU support fails

This used to work:
FROM nvidia/cuda:11.0-devel-ubuntu20.04

RUN apt-get update && \
    DEBIAN_FRONTEND="noninteractive" apt-get install -y software-properties-common apt-transport-https

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 && \
    add-apt-repository 'deb [arch=amd64] https://cran.rstudio.com/bin/linux/ubuntu focal-cran40/' && \
    apt-get update && \
    apt-get install -y r-base

RUN apt-get install -y git wget libcurl4-openssl-dev default-jdk-headless libssl-dev libxml2-dev cmake

ENV MAKE="make -j$(nproc)"

RUN R -e 'install.packages(c("R6","data.table","jsonlite"), repos = "https://cran.rstudio.com/")'

RUN apt-get install -y libboost-dev libboost-system-dev libboost-filesystem-dev ocl-icd-opencl-dev opencl-headers clinfo

RUN mkdir -p /etc/OpenCL/vendors && \
    echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd   ## otherwise lightgm segfaults at runtime (compiles fine without it)

RUN git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && \
    Rscript build_r.R --use-gpu

Now, I get this error:
Cloning into 'LightGBM'...
Submodule 'include/boost/compute' (https://github.com/boostorg/compute) registered for path 'compute'
Submodule 'eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'eigen'
Submodule 'external_libs/fast_double_parser' (https://github.com/lemire/fast_double_parser.git) registered for path 'external_libs/fast_double_parser'
Submodule 'external_libs/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'external_libs/fmt'
Cloning into '/LightGBM/compute'...
Cloning into '/LightGBM/eigen'...
Cloning into '/LightGBM/external_libs/fast_double_parser'...
Cloning into '/LightGBM/external_libs/fmt'...
Submodule path 'compute': checked out '36c89134d4013b2e5e45bc55656a18bd6141995a'
Submodule path 'eigen': checked out '8ba1b0f41a7950dc3e1d4ed75859e36c73311235'
Submodule path 'external_libs/fast_double_parser': checked out 'ace60646c02dc54c57f19d644e49a61e7e7758ec'
Submodule 'benchmark/dependencies/abseil-cpp' (https://github.com/abseil/abseil-cpp.git) registered for path 'external_libs/fast_double_parser/benchmarks/dependencies/abseil-cpp'
Submodule 'benchmark/dependencies/double-conversion' (https://github.com/google/double-conversion.git) registered for path 'external_libs/fast_double_parser/benchmarks/dependencies/double-conversion'
Cloning into '/LightGBM/external_libs/fast_double_parser/benchmarks/dependencies/abseil-cpp'...
Cloning into '/LightGBM/external_libs/fast_double_parser/benchmarks/dependencies/double-conversion'...
Submodule path 'external_libs/fast_double_parser/benchmarks/dependencies/abseil-cpp': checked out 'd936052d32a5b7ca08b0199a6724724aea432309'
Submodule path 'external_libs/fast_double_parser/benchmarks/dependencies/double-conversion': checked out 'f4cb2384efa55dee0e6652f8674b05763441ab09'
Submodule path 'external_libs/fmt': checked out 'cc09f1a6798c085c325569ef466bcdcffdc266d4'
* checking for file '/LightGBM/lightgbm_r/DESCRIPTION' ... OK
* preparing 'lightgbm':
* checking DESCRIPTION meta-information ... OK
* cleaning src
Warning in system2(command, args, stdout = NULL, stderr = NULL, ...) :
  error in running command
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
WARNING: directory 'lightgbm/src/compute/test' is empty
* looking to see if a 'data/datalist' file should be added
* building 'lightgbm_<IP_ADDRESS>.tar.gz'

* installing to library '/usr/local/lib/R/site-library'
* installing *source* package 'lightgbm' ...
** using staged installation
** libs
installing via 'install.libs.R' to /usr/local/lib/R/site-library/00LOCK-lightgbm/00new/lightgbm
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- R version passed into FindLibR.cmake: 4.0.3
-- Found LibR: /usr/lib/R
-- LIBR_EXECUTABLE: /usr/bin/R
-- LIBR_INCLUDE_DIRS: /usr/share/R/include
-- LIBR_CORE_LIBRARY: /usr/lib/R/lib/libR.so
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenCL (missing: OpenCL_LIBRARY) (found version "2.2")
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindOpenCL.cmake:150 (find_package_handle_standard_args)
  CMakeLists.txt:138 (find_package)

-- Configuring incomplete, errors occurred!
See also "/tmp/RtmpvcXiAX/R.INSTALL14755eba078/lightgbm/src/build/CMakeFiles/CMakeOutput.log".
Error in .run_shell_command("cmake", c(cmake_args, "..")) :
  Command failed with exit code: 1
* removing '/usr/local/lib/R/site-library/lightgbm'
Error in .run_shell_command(install_cmd, install_args) :
  Command failed with exit code: 1
Execution halted
The command '/bin/sh -c git clone --recursive https://github.com/microsoft/LightGBM &&     cd LightGBM &&     Rscript build_r.R --use-gpu' returned a non-zero code: 1

If I build the docker image with the last RUN entry commented out:
FROM nvidia/cuda:11.0-devel-ubuntu20.04

RUN apt-get update && \
    DEBIAN_FRONTEND="noninteractive" apt-get install -y software-properties-common apt-transport-https

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 && \
    add-apt-repository 'deb [arch=amd64] https://cran.rstudio.com/bin/linux/ubuntu focal-cran40/' && \
    apt-get update && \
    apt-get install -y r-base

RUN apt-get install -y git wget libcurl4-openssl-dev default-jdk-headless libssl-dev libxml2-dev cmake

ENV MAKE="make -j$(nproc)"

RUN R -e 'install.packages(c("R6","data.table","jsonlite"), repos = "https://cran.rstudio.com/")'

RUN apt-get install -y libboost-dev libboost-system-dev libboost-filesystem-dev ocl-icd-opencl-dev opencl-headers clinfo

RUN mkdir -p /etc/OpenCL/vendors && \
    echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd   ## otherwise lightgm segfaults at runtime (compiles fine without it)

#RUN git clone --recursive https://github.com/microsoft/LightGBM && \
#    cd LightGBM && \
#    Rscript build_r.R --use-gpu

with
sudo docker build -t gbmperf_gpu .

and then run it:
sudo nvidia-docker run --rm -ti gbmperf_gpu /bin/bash

then I can run things manually:
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && \
    Rscript build_r.R --use-gpu

gives the same error.
However, just compiling lightgbm (not the R package) seems fine:
git clone --recursive https://github.com/microsoft/LightGBM  && \
cd LightGBM  &&  mkdir build  &&  cd build  &&  cmake -DUSE_GPU=1 ..  &&  make -j4

as here:
...
Submodule path 'external_libs/fast_double_parser/benchmarks/dependencies/abseil-cpp': checked out 'd936052d32a5b7ca08b0199a6724724aea432309'
Submodule path 'external_libs/fast_double_parser/benchmarks/dependencies/double-conversion': checked out 'f4cb2384efa55dee0e6652f8674b05763441ab09'
Submodule path 'external_libs/fmt': checked out 'cc09f1a6798c085c325569ef466bcdcffdc266d4'
-- The C compiler identification is GNU 9.3.0
-- The CXX compiler identification is GNU 9.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.2")
-- OpenCL include directory: /usr/include
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found suitable version "1.71.0", minimum required is "1.56.0") found components: filesystem system
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /LightGBM/LightGBM/build
make[1]: warning: -j0 forced in submake: resetting jobserver mode.
Scanning dependencies of target lightgbm
Scanning dependencies of target _lightgbm
[  1%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/boosting.cpp.o
[  2%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt.cpp.o
[  4%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/gbdt.cpp.o
[  7%] Building CXX object CMakeFiles/lightgbm.dir/src/boosting/boosting.cpp.o
[  7%] Building CXX object CMakeFiles/lightgbm.dir/src/main.cpp.o
[  8%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o
[ 10%] Building CXX object CMakeFiles/lightgbm.dir/src/application/application.cpp.o
...

though I also see
/usr/include/CL/cl_version.h:34:104: note: #pragma message: cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 220 (OpenCL 2.2)
   34 | #pragma message("cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 220 (OpenCL 2.2)")

but it compiles anyway:
[ 98%] Linking CXX shared library ../lib_lightgbm.so
[100%] Linking CXX executable ../lightgbm
[100%] Built target _lightgbm
[100%] Built target lightgbm

So there must be something in the R package(?) cc @jameslamb

Thanks very much for the detailed report and great reproducible example! I'll take a look at this later today.

Thanks very much for the detailed report and great reproducible example! I'll take a look at this later today.

Thank you so much @jameslamb

Thank you so much @jameslamb

Actually, the old way of installing the R package works:
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && sed -i "s/use_gpu <- FALSE/use_gpu <- TRUE/"  R-package/src/install.libs.R && Rscript build_r.R

Actually, the old way of installing the R package works:
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && sed -i "s/use_gpu <- FALSE/use_gpu <- TRUE/"  R-package/src/install.libs.R && Rscript build_r.R

Oh, I was wrong, the old method above compiles indeed, but it actually does not work:
Error in lgb.call(fun_name = "LGBM_BoosterCreate_R", ret = handle, train_set_handle,  :
  [LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1

Oh, I was wrong, the old method above compiles indeed, but it actually does not work:
Error in lgb.call(fun_name = "LGBM_BoosterCreate_R", ret = handle, train_set_handle,  :
  [LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1

@szilard Hey!
Let me put my two cents in it 🙂 . I completely out of ideas how sed-based installation is working while argument-based one is not. Hope @jameslamb will find the root cause.
But I took a look at your Docker file. Seems that you are inheriting from nvidia/cuda:11.0-devel-ubuntu20.04 image. devel part in it makes me sure that that image has already OpenCL libraries installed. So you don't need to install separate packages ocl-icd-opencl-dev opencl-headers via apt. You just need to make env variables to point at appropriate paths. Please take a look at this section of our Docker example.
https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/docker/gpu/dockerfile.gpu#L15-L20
The following action I can see in your Docker already 👍 https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/docker/gpu/dockerfile.gpu#L59-L61
In addition, performing those actions might not be enough. Then you can add similar paths from the example below to CMake commands
https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/docker/gpu/dockerfile.gpu#L87
here
https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/R-package/src/install.libs.R#L166-L168
and
https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/docker/gpu/dockerfile.gpu#L88
here
https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/R-package/src/install.libs.R#L135
I suspect that CMake goes mad somehow due to two OpenCL installations. Look at these lines from your log:
-- Looking for CL_VERSION_2_2 - found
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenCL (missing: OpenCL_LIBRARY) (found version "2.2")

"found" and "Could NOT find" at the same time.

@szilard Hey!
Let me put my two cents in it 🙂 . I completely out of ideas how sed-based installation is working while argument-based one is not. Hope @jameslamb will find the root cause.
But I took a look at your Docker file. Seems that you are inheriting from nvidia/cuda:11.0-devel-ubuntu20.04 image. devel part in it makes me sure that that image has already OpenCL libraries installed. So you don't need to install separate packages ocl-icd-opencl-dev opencl-headers via apt. You just need to make env variables to point at appropriate paths. Please take a look at this section of our Docker example.
https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/docker/gpu/dockerfile.gpu#L15-L20
The following action I can see in your Docker already 👍 https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/docker/gpu/dockerfile.gpu#L59-L61
In addition, performing those actions might not be enough. Then you can add similar paths from the example below to CMake commands
https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/docker/gpu/dockerfile.gpu#L87
here
https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/R-package/src/install.libs.R#L166-L168
and
https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/docker/gpu/dockerfile.gpu#L88
here
https://github.com/microsoft/LightGBM/blob/f997a0692ca0f26740d2bdef2695c3e881d4e918/R-package/src/install.libs.R#L135
I suspect that CMake goes mad somehow due to two OpenCL installations. Look at these lines from your log:
-- Looking for CL_VERSION_2_2 - found
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenCL (missing: OpenCL_LIBRARY) (found version "2.2")

"found" and "Could NOT find" at the same time.

BTW,

However, just compiling lightgbm (not the R package) seems fine:

You just need to make env variables to point at appropriate paths.

Probably this is the reason... I mean, maybe appropriate env variables are already set in Docker command line, but R doesn't see them.

BTW,

However, just compiling lightgbm (not the R package) seems fine:

You just need to make env variables to point at appropriate paths.

Probably this is the reason... I mean, maybe appropriate env variables are already set in Docker command line, but R doesn't see them.

Well, as I corrected myself later, the sed version actually does not work properly anymore either (it compiles, but it does not add GPU support actually).
Yeah, my Dockerfile has a history of additions over the years (and hacks like the echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd thing), I'll see if I can clean it up with your suggestions @StrikerRUS.
However, lightgbm compiles fine outside the R package, so it seems it's only the R package that gets confused about OpenCL.

Well, as I corrected myself later, the sed version actually does not work properly anymore either (it compiles, but it does not add GPU support actually).
Yeah, my Dockerfile has a history of additions over the years (and hacks like the echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd thing), I'll see if I can clean it up with your suggestions @StrikerRUS.
However, lightgbm compiles fine outside the R package, so it seems it's only the R package that gets confused about OpenCL.

Trying to understand what's going on, trying to strip down things as much as possible:
If I remove the ocl-icd-opencl-dev opencl-headers clinfo install from my docker:
FROM nvidia/cuda:11.0-devel-ubuntu20.04

RUN apt-get update && \
    DEBIAN_FRONTEND="noninteractive" apt-get install -y software-properties-common apt-transport-https

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 && \
    add-apt-repository 'deb [arch=amd64] https://cran.rstudio.com/bin/linux/ubuntu focal-cran40/' && \
    apt-get update && \
    apt-get install -y r-base

RUN apt-get install -y git wget libcurl4-openssl-dev default-jdk-headless libssl-dev libxml2-dev cmake

ENV MAKE="make -j$(nproc)"

RUN R -e 'install.packages(c("R6","data.table","jsonlite"), repos = "https://cran.rstudio.com/")'

##RUN apt-get install -y libboost-dev libboost-system-dev libboost-filesystem-dev ocl-icd-opencl-dev opencl-headers clinfo
RUN apt-get install -y libboost-dev libboost-system-dev libboost-filesystem-dev

RUN mkdir -p /etc/OpenCL/vendors && \
    echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd   ## otherwise lightgm segfaults at runtime (compiles fine without it)

Then the R install errors out (obviously) with:
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && \
    Rscript build_r.R --use-gpu

ERROR:
...
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - not found
-- Looking for CL_VERSION_2_1
-- Looking for CL_VERSION_2_1 - not found
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - not found
-- Looking for CL_VERSION_1_1
-- Looking for CL_VERSION_1_1 - not found
-- Looking for CL_VERSION_1_0
-- Looking for CL_VERSION_1_0 - not found
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenCL (missing: OpenCL_LIBRARY OpenCL_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindOpenCL.cmake:150 (find_package_handle_standard_args)
  CMakeLists.txt:138 (find_package)
...

The non-R install works even if striped down to this (no need for any ENV variables or any other compiler flags mentioned above by @StrikerRUS ):
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && mkdir build && cd build && \
    cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda/include/ .. && \
    make

WORKS OK:

...
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - not found
-- Looking for CL_VERSION_2_1
-- Looking for CL_VERSION_2_1 - not found
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - found
-- Found OpenCL: /usr/local/cuda/lib64/libOpenCL.so (found version "1.2")
-- OpenCL include directory: /usr/local/cuda/include
...

Removing any of DOpenCL_LIBRARY or DOpenCL_INCLUDE_DIR flag breaks:
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && mkdir build && cd build && \
    cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda/lib64/libOpenCL.so .. && \
    make

ERROR:

-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - not found
-- Looking for CL_VERSION_2_1
-- Looking for CL_VERSION_2_1 - not found
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - not found
-- Looking for CL_VERSION_1_1
-- Looking for CL_VERSION_1_1 - not found
-- Looking for CL_VERSION_1_0
-- Looking for CL_VERSION_1_0 - not found
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenCL (missing: OpenCL_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindOpenCL.cmake:150 (find_package_handle_standard_args)
  CMakeLists.txt:138 (find_package)

git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && mkdir build && cd build && \
    cmake -DUSE_GPU=1 -DOpenCL_INCLUDE_DIR=/usr/local/cuda/include/ .. && \
    make

ERROR:

-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - not found
-- Looking for CL_VERSION_2_1
-- Looking for CL_VERSION_2_1 - not found
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - found
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenCL (missing: OpenCL_LIBRARY) (found version "1.2")
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindOpenCL.cmake:150 (find_package_handle_standard_args)
  CMakeLists.txt:138 (find_package)

If I add back the ocl-icd-opencl-dev opencl-headers clinfo install to my docker:
then the R package fails (this was yesterday's result, just including here to see the error message in context):
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && \
    Rscript build_r.R --use-gpu

ERROR:

-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenCL (missing: OpenCL_LIBRARY) (found version "2.2")
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindOpenCL.cmake:150 (find_package_handle_standard_args)
  CMakeLists.txt:138 (find_package)

but now the non-R install can be stripped down even more:
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && mkdir build && cd build && \
    cmake -DUSE_GPU=1 .. && \
    make

WORKS:

-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.2")
-- OpenCL include directory: /usr/include

and notice it has found now OpenCL 2.2 instead of previously the old 1.2 included in nvidia/cuda:11.0-devel-ubuntu20.04 or the other NVIDIA docker images like the one you guys are using.
Based on this it seems to me something's up with the R package (likely it can't get the OpenCL_LIBRARY value when it's calling cmake).

Trying to understand what's going on, trying to strip down things as much as possible:
If I remove the ocl-icd-opencl-dev opencl-headers clinfo install from my docker:
FROM nvidia/cuda:11.0-devel-ubuntu20.04

RUN apt-get update && \
    DEBIAN_FRONTEND="noninteractive" apt-get install -y software-properties-common apt-transport-https

RUN apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 && \
    add-apt-repository 'deb [arch=amd64] https://cran.rstudio.com/bin/linux/ubuntu focal-cran40/' && \
    apt-get update && \
    apt-get install -y r-base

RUN apt-get install -y git wget libcurl4-openssl-dev default-jdk-headless libssl-dev libxml2-dev cmake

ENV MAKE="make -j$(nproc)"

RUN R -e 'install.packages(c("R6","data.table","jsonlite"), repos = "https://cran.rstudio.com/")'

##RUN apt-get install -y libboost-dev libboost-system-dev libboost-filesystem-dev ocl-icd-opencl-dev opencl-headers clinfo
RUN apt-get install -y libboost-dev libboost-system-dev libboost-filesystem-dev

RUN mkdir -p /etc/OpenCL/vendors && \
    echo "libnvidia-opencl.so.1" > /etc/OpenCL/vendors/nvidia.icd   ## otherwise lightgm segfaults at runtime (compiles fine without it)

Then the R install errors out (obviously) with:
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && \
    Rscript build_r.R --use-gpu

ERROR:
...
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - not found
-- Looking for CL_VERSION_2_1
-- Looking for CL_VERSION_2_1 - not found
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - not found
-- Looking for CL_VERSION_1_1
-- Looking for CL_VERSION_1_1 - not found
-- Looking for CL_VERSION_1_0
-- Looking for CL_VERSION_1_0 - not found
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenCL (missing: OpenCL_LIBRARY OpenCL_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindOpenCL.cmake:150 (find_package_handle_standard_args)
  CMakeLists.txt:138 (find_package)
...

The non-R install works even if striped down to this (no need for any ENV variables or any other compiler flags mentioned above by @StrikerRUS ):
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && mkdir build && cd build && \
    cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda/lib64/libOpenCL.so -DOpenCL_INCLUDE_DIR=/usr/local/cuda/include/ .. && \
    make

WORKS OK:

...
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - not found
-- Looking for CL_VERSION_2_1
-- Looking for CL_VERSION_2_1 - not found
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - found
-- Found OpenCL: /usr/local/cuda/lib64/libOpenCL.so (found version "1.2")
-- OpenCL include directory: /usr/local/cuda/include
...

Removing any of DOpenCL_LIBRARY or DOpenCL_INCLUDE_DIR flag breaks:
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && mkdir build && cd build && \
    cmake -DUSE_GPU=1 -DOpenCL_LIBRARY=/usr/local/cuda/lib64/libOpenCL.so .. && \
    make

ERROR:

-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - not found
-- Looking for CL_VERSION_2_1
-- Looking for CL_VERSION_2_1 - not found
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - not found
-- Looking for CL_VERSION_1_1
-- Looking for CL_VERSION_1_1 - not found
-- Looking for CL_VERSION_1_0
-- Looking for CL_VERSION_1_0 - not found
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenCL (missing: OpenCL_INCLUDE_DIR)
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindOpenCL.cmake:150 (find_package_handle_standard_args)
  CMakeLists.txt:138 (find_package)

git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && mkdir build && cd build && \
    cmake -DUSE_GPU=1 -DOpenCL_INCLUDE_DIR=/usr/local/cuda/include/ .. && \
    make

ERROR:

-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - not found
-- Looking for CL_VERSION_2_1
-- Looking for CL_VERSION_2_1 - not found
-- Looking for CL_VERSION_2_0
-- Looking for CL_VERSION_2_0 - not found
-- Looking for CL_VERSION_1_2
-- Looking for CL_VERSION_1_2 - found
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenCL (missing: OpenCL_LIBRARY) (found version "1.2")
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindOpenCL.cmake:150 (find_package_handle_standard_args)
  CMakeLists.txt:138 (find_package)

If I add back the ocl-icd-opencl-dev opencl-headers clinfo install to my docker:
then the R package fails (this was yesterday's result, just including here to see the error message in context):
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && \
    Rscript build_r.R --use-gpu

ERROR:

-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find OpenCL (missing: OpenCL_LIBRARY) (found version "2.2")
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindOpenCL.cmake:150 (find_package_handle_standard_args)
  CMakeLists.txt:138 (find_package)

but now the non-R install can be stripped down even more:
git clone --recursive https://github.com/microsoft/LightGBM && \
    cd LightGBM && mkdir build && cd build && \
    cmake -DUSE_GPU=1 .. && \
    make

WORKS:

-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.2")
-- OpenCL include directory: /usr/include

and notice it has found now OpenCL 2.2 instead of previously the old 1.2 included in nvidia/cuda:11.0-devel-ubuntu20.04 or the other NVIDIA docker images like the one you guys are using.
Based on this it seems to me something's up with the R package (likely it can't get the OpenCL_LIBRARY value when it's calling cmake).

If I make this change in R-package/src/install.libs.R:
-  cmake_args <- c(cmake_args, "-DUSE_GPU=ON")
+  cmake_args <- c(cmake_args, "-DUSE_GPU=ON -DOpenCL_LIBRARY=/usr/lib/x86_64-linux-gnu/libOpenCL.so")

then it finds OpenCL:
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.2")
-- OpenCL include directory: /usr/include

However, then it can't find Boost now:
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find Boost (missing: filesystem system) (found suitable version
  "1.71.0", minimum required is "1.56.0")
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindBoost.cmake:2179 (find_package_handle_standard_args)
  CMakeLists.txt:144 (find_package)

It seems like the R build script cannot find the necessary paths anymore somehow, not only OpenCL.

If I make this change in R-package/src/install.libs.R:
-  cmake_args <- c(cmake_args, "-DUSE_GPU=ON")
+  cmake_args <- c(cmake_args, "-DUSE_GPU=ON -DOpenCL_LIBRARY=/usr/lib/x86_64-linux-gnu/libOpenCL.so")

then it finds OpenCL:
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.2")
-- OpenCL include directory: /usr/include

However, then it can't find Boost now:
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
  Could NOT find Boost (missing: filesystem system) (found suitable version
  "1.71.0", minimum required is "1.56.0")
Call Stack (most recent call first):
  /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake-3.16/Modules/FindBoost.cmake:2179 (find_package_handle_standard_args)
  CMakeLists.txt:144 (find_package)

It seems like the R build script cannot find the necessary paths anymore somehow, not only OpenCL.

It seems like the R build script cannot find the necessary paths anymore somehow (for the GPU install), not only OpenCL.

Could you please try adding Boost paths (-DBOOST_INCLUDEDIR=) to cmake_args  here as well?
cmake_args <- c(cmake_args, "-DUSE_GPU=ON -DOpenCL_LIBRARY=/usr/lib/x86_64-linux-gnu/libOpenCL.so")

On our CI it finds Boost in usr/include:
-- Found Boost: /usr/include (found suitable version "1.74.0", minimum required is "1.56.0") found components: filesystem system

Actual path for your Docker you can take from a successful installation from command line, I believe. Though, it's quite strange.
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found suitable version "1.71.0", minimum required is "1.56.0") found components: filesystem system

And I think having ocl-icd-opencl-dev opencl-headers clinfo installed it's preferable because it's a newer OpenCL version and also takes care of the include and lib paths at least in the non-R install

I'm not sure that newer version from Ubuntu ppa is better than preinstalled native version from NVIDIA in case you are really using NVIDIA cards for training.

It seems like the R build script cannot find the necessary paths anymore somehow (for the GPU install), not only OpenCL.

Could you please try adding Boost paths (-DBOOST_INCLUDEDIR=) to cmake_args  here as well?
cmake_args <- c(cmake_args, "-DUSE_GPU=ON -DOpenCL_LIBRARY=/usr/lib/x86_64-linux-gnu/libOpenCL.so")

On our CI it finds Boost in usr/include:
-- Found Boost: /usr/include (found suitable version "1.74.0", minimum required is "1.56.0") found components: filesystem system

Actual path for your Docker you can take from a successful installation from command line, I believe. Though, it's quite strange.
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found suitable version "1.71.0", minimum required is "1.56.0") found components: filesystem system

And I think having ocl-icd-opencl-dev opencl-headers clinfo installed it's preferable because it's a newer OpenCL version and also takes care of the include and lib paths at least in the non-R install

I'm not sure that newer version from Ubuntu ppa is better than preinstalled native version from NVIDIA in case you are really using NVIDIA cards for training.

@jameslamb I believe R-package needs the same additions command line options for GPU-version as our Python-package:
- boost-root
- boost-dir
- boost-include-dir
- boost-librarydir
- opencl-include-dir
- opencl-library

https://github.com/microsoft/LightGBM/tree/master/python-package#build-gpu-version

@jameslamb I believe R-package needs the same additions command line options for GPU-version as our Python-package:
- boost-root
- boost-dir
- boost-include-dir
- boost-librarydir
- opencl-include-dir
- opencl-library

https://github.com/microsoft/LightGBM/tree/master/python-package#build-gpu-version

I can't make it pass Boost by adding -DBOOST_INCLUDEDIR= to cmake_args.

I can't make it pass Boost by adding -DBOOST_INCLUDEDIR= to cmake_args.

All this strange, because last time I ran the benchmarks (September 2020) it was all working.

All this strange, because last time I ran the benchmarks (September 2020) it was all working.

Please try -DBOOST_INCLUDEDIR=/usr/include/boost in cmake_args. And in case of failure again, add -DBOOST_LIBRARYDIR=/usr/lib/x86_64-linux-gnu as well.

Please try -DBOOST_INCLUDEDIR=/usr/include/boost in cmake_args. And in case of failure again, add -DBOOST_LIBRARYDIR=/usr/lib/x86_64-linux-gnu as well.

Indeed, with this Lightgbm commit 7e11d4aeabd4a39ff (Aug 30, 2020) and using the old sed method:
sed -i "s/use_gpu <- FALSE/use_gpu <- TRUE/"  R-package/src/install.libs.R && Rscript build_r.R

it works:
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.2")
-- OpenCL include directory: /usr/include
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found suitable version "1.71.0", minimum required is "1.56.0") found components: filesystem system
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/RtmpwQ9CUI/R.INSTALLa3474e269e7/lightgbm/src/build
Building lib_lightgbm
Scanning dependencies of target _lightgbm
[ 15%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 15%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 18%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o

Indeed, with this Lightgbm commit 7e11d4aeabd4a39ff (Aug 30, 2020) and using the old sed method:
sed -i "s/use_gpu <- FALSE/use_gpu <- TRUE/"  R-package/src/install.libs.R && Rscript build_r.R

it works:
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.2")
-- OpenCL include directory: /usr/include
-- Found Boost: /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake (found suitable version "1.71.0", minimum required is "1.56.0") found components: filesystem system
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/RtmpwQ9CUI/R.INSTALLa3474e269e7/lightgbm/src/build
Building lib_lightgbm
Scanning dependencies of target _lightgbm
[ 15%] Building CXX object CMakeFiles/_lightgbm.dir/src/io/bin.cpp.o
[ 15%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/gbdt_prediction.cpp.o
[ 18%] Building CXX object CMakeFiles/_lightgbm.dir/src/boosting/prediction_early_stop.cpp.o

Well, actually I don't know, it compiles, though now I'm on an instance without GPU, so I'm not sure if it adds GPU support (yesterday it seemed that the old sed thing compiles with the latest commit, but it does not add GPU support)

Well, actually I don't know, it compiles, though now I'm on an instance without GPU, so I'm not sure if it adds GPU support (yesterday it seemed that the old sed thing compiles with the latest commit, but it does not add GPU support)

Sorry for possible confusion, maybe I did not explain it the best way, what I mean is that
sed -i "s/use_gpu <- FALSE/use_gpu <- TRUE/"  R-package/src/install.libs.R && Rscript build_r.R

seems to always compile OK, but in the latest lightgbm version it actually does not add GPU support.
In fact I think I'm able to see if GPU support was added even on this non-GPU instance, because with old commit  7e11d4a (Aug 30, 2020) after compiling with the sed thing,  I get:
Error in lgb.last_error() : api error: No OpenCL device found
Error in initialize(...) : lgb.Booster: cannot create Booster handle

while with the latest Lightgbm commit I get (compiling with the sed thing):
Error in lgb.call(fun_name = "LGBM_BoosterCreate_R", ret = handle, train_set_handle,  :
  [LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1

So it seems old sed method of installing R with GPU support compiles, but at some point since September 2020 it does not actually add GPU support.
I'm not sure if it would help us fix the main issue if we find out which commit broke this (it's weird that the old sed thing still compiles even now).

Sorry for possible confusion, maybe I did not explain it the best way, what I mean is that
sed -i "s/use_gpu <- FALSE/use_gpu <- TRUE/"  R-package/src/install.libs.R && Rscript build_r.R

seems to always compile OK, but in the latest lightgbm version it actually does not add GPU support.
In fact I think I'm able to see if GPU support was added even on this non-GPU instance, because with old commit  7e11d4a (Aug 30, 2020) after compiling with the sed thing,  I get:
Error in lgb.last_error() : api error: No OpenCL device found
Error in initialize(...) : lgb.Booster: cannot create Booster handle

while with the latest Lightgbm commit I get (compiling with the sed thing):
Error in lgb.call(fun_name = "LGBM_BoosterCreate_R", ret = handle, train_set_handle,  :
  [LightGBM] [Fatal] GPU Tree Learner was not enabled in this build.
Please recompile with CMake option -DUSE_GPU=1

So it seems old sed method of installing R with GPU support compiles, but at some point since September 2020 it does not actually add GPU support.
I'm not sure if it would help us fix the main issue if we find out which commit broke this (it's weird that the old sed thing still compiles even now).

With this:
-  cmake_args <- c(cmake_args, "-DUSE_GPU=ON")
+  cmake_args <- c(cmake_args, "-DUSE_GPU=ON -DOpenCL_LIBRARY=/usr/lib/x86_64-linux-gnu/libOpenCL.so -DBOOST_LIBRARYDIR=/usr/lib/x86_64-linux-gnu")

(added boost libdir but not the include)
it compiles:
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.2")
-- OpenCL include directory: /usr/include
-- Found Boost: /usr/include (found suitable version "1.71.0", minimum required is "1.56.0") found components: filesystem system

and it will probably run.
I get
Error in lgb.last_error() : api error: No OpenCL device found
Error in initialize(...) : lgb.Booster: cannot create Booster handle

but on a box without GPU (good sign), I'll have to try it out on an instance with GPU.
The code I'm running btw:
suppressMessages({
library(data.table)
library(ROCR)
library(lightgbm)
library(Matrix)
})

set.seed(123)

d_train <- fread("https://s3.amazonaws.com/benchm-ml--main/train-1m.csv", showProgress=FALSE)
d_test <- fread("https://s3.amazonaws.com/benchm-ml--main/test.csv", showProgress=FALSE)

d_all <- rbind(d_train, d_test)
d_all$dep_delayed_15min <- ifelse(d_all$dep_delayed_15min=="Y",1,0)

d_all_wrules <- lgb.convert_with_rules(d_all)
d_all <- d_all_wrules$data
cols_cats <- names(d_all_wrules$rules)

d_train <- d_all[1:nrow(d_train)]
d_test <- d_all[(nrow(d_train)+1):(nrow(d_train)+nrow(d_test))]

p <- ncol(d_all)-1
dlgb_train <- lgb.Dataset(data = as.matrix(d_train[,1:p]), label = d_train$dep_delayed_15min, free_raw_data = FALSE)

cat(system.time({
  md <- lgb.train(data = dlgb_train,
            objective = "binary",
            nrounds = 100, num_leaves = 512, learning_rate = 0.1,
            categorical_feature = cols_cats,
            device = "gpu",
            verbose = 0)
})[[3]]," ",sep="")

phat <- predict(md, data = as.matrix(d_test[,1:p]))
rocr_pred <- prediction(phat, d_test$dep_delayed_15min)
cat(performance(rocr_pred, "auc")@y.values[[1]],"\n")

With this:
-  cmake_args <- c(cmake_args, "-DUSE_GPU=ON")
+  cmake_args <- c(cmake_args, "-DUSE_GPU=ON -DOpenCL_LIBRARY=/usr/lib/x86_64-linux-gnu/libOpenCL.so -DBOOST_LIBRARYDIR=/usr/lib/x86_64-linux-gnu")

(added boost libdir but not the include)
it compiles:
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.2")
-- OpenCL include directory: /usr/include
-- Found Boost: /usr/include (found suitable version "1.71.0", minimum required is "1.56.0") found components: filesystem system

and it will probably run.
I get
Error in lgb.last_error() : api error: No OpenCL device found
Error in initialize(...) : lgb.Booster: cannot create Booster handle

but on a box without GPU (good sign), I'll have to try it out on an instance with GPU.
The code I'm running btw:
suppressMessages({
library(data.table)
library(ROCR)
library(lightgbm)
library(Matrix)
})

set.seed(123)

d_train <- fread("https://s3.amazonaws.com/benchm-ml--main/train-1m.csv", showProgress=FALSE)
d_test <- fread("https://s3.amazonaws.com/benchm-ml--main/test.csv", showProgress=FALSE)

d_all <- rbind(d_train, d_test)
d_all$dep_delayed_15min <- ifelse(d_all$dep_delayed_15min=="Y",1,0)

d_all_wrules <- lgb.convert_with_rules(d_all)
d_all <- d_all_wrules$data
cols_cats <- names(d_all_wrules$rules)

d_train <- d_all[1:nrow(d_train)]
d_test <- d_all[(nrow(d_train)+1):(nrow(d_train)+nrow(d_test))]

p <- ncol(d_all)-1
dlgb_train <- lgb.Dataset(data = as.matrix(d_train[,1:p]), label = d_train$dep_delayed_15min, free_raw_data = FALSE)

cat(system.time({
  md <- lgb.train(data = dlgb_train,
            objective = "binary",
            nrounds = 100, num_leaves = 512, learning_rate = 0.1,
            categorical_feature = cols_cats,
            device = "gpu",
            verbose = 0)
})[[3]]," ",sep="")

phat <- predict(md, data = as.matrix(d_test[,1:p]))
rocr_pred <- prediction(phat, d_test$dep_delayed_15min)
cat(performance(rocr_pred, "auc")@y.values[[1]],"\n")

Error in lgb.last_error() : api error: No OpenCL device found

Nice, given that the error happens on non-GPU machine! Indeed good sign!
But please note that successfully compiled GPU version and using device_type='gpu' in params may still result in training on CPU. This can occur with CPU that have onboard graphics and some combination of system-wide default platform and device (refer to gpu_platform_id and gpu_device_id). So to be 100% sure LightGBM uses real GPU please take a look at training log and find this line

[LightGBM] [Info] Using GPU Device: GeForce MX150, Vendor: NVIDIA Corporation

Error in lgb.last_error() : api error: No OpenCL device found

Nice, given that the error happens on non-GPU machine! Indeed good sign!
But please note that successfully compiled GPU version and using device_type='gpu' in params may still result in training on CPU. This can occur with CPU that have onboard graphics and some combination of system-wide default platform and device (refer to gpu_platform_id and gpu_device_id). So to be 100% sure LightGBM uses real GPU please take a look at training log and find this line

[LightGBM] [Info] Using GPU Device: GeForce MX150, Vendor: NVIDIA Corporation

or use nvidia-smi ;)

or use nvidia-smi ;)

I ran it on an instance with GPU (p3 with V100):
With this patch:
-  cmake_args <- c(cmake_args, "-DUSE_GPU=ON")
+  cmake_args <- c(cmake_args, "-DUSE_GPU=ON -DOpenCL_LIBRARY=/usr/lib/x86_64-linux-gnu/libOpenCL.so -DBOOST_LIBRARYDIR=/usr/lib/x86_64-linux-gnu")

that is by using this hack in my Dockerfile (and with ocl-icd-opencl-dev opencl-headers clinfo added back):
RUN git clone --recursive https://github.com/microsoft/LightGBM && cd LightGBM && \
    sed -i 's/cmake_args <- c(cmake_args, "-DUSE_GPU=ON")/cmake_args <- c(cmake_args, "-DUSE_GPU=ON -DOpenCL_LIBRARY=\/usr\/lib\/x86_64-linux-gnu\/libOpenCL.so -DBOOST_LIBRARYDIR=\/usr\/lib\/x86_64-linux-gnu")/'  R-package/src/install.libs.R && \
    Rscript build_r.R --use-gpu

it is compiling and running OK.
Full Dockerfile: https://github.com/szilard/GBM-perf/blob/f34c37357e82f7dd3d8f30e5625a7f268a3b98a5/gpu/Dockerfile
Full R code running: https://github.com/szilard/GBM-perf/blob/f34c37357e82f7dd3d8f30e5625a7f268a3b98a5/gpu/run/3-lightgbm.R
I wonder if on other systems it works out of the box or not (without adding the paths with the patch).

I ran it on an instance with GPU (p3 with V100):
With this patch:
-  cmake_args <- c(cmake_args, "-DUSE_GPU=ON")
+  cmake_args <- c(cmake_args, "-DUSE_GPU=ON -DOpenCL_LIBRARY=/usr/lib/x86_64-linux-gnu/libOpenCL.so -DBOOST_LIBRARYDIR=/usr/lib/x86_64-linux-gnu")

that is by using this hack in my Dockerfile (and with ocl-icd-opencl-dev opencl-headers clinfo added back):
RUN git clone --recursive https://github.com/microsoft/LightGBM && cd LightGBM && \
    sed -i 's/cmake_args <- c(cmake_args, "-DUSE_GPU=ON")/cmake_args <- c(cmake_args, "-DUSE_GPU=ON -DOpenCL_LIBRARY=\/usr\/lib\/x86_64-linux-gnu\/libOpenCL.so -DBOOST_LIBRARYDIR=\/usr\/lib\/x86_64-linux-gnu")/'  R-package/src/install.libs.R && \
    Rscript build_r.R --use-gpu

it is compiling and running OK.
Full Dockerfile: https://github.com/szilard/GBM-perf/blob/f34c37357e82f7dd3d8f30e5625a7f268a3b98a5/gpu/Dockerfile
Full R code running: https://github.com/szilard/GBM-perf/blob/f34c37357e82f7dd3d8f30e5625a7f268a3b98a5/gpu/run/3-lightgbm.R
I wonder if on other systems it works out of the box or not (without adding the paths with the patch).

I wonder if on other systems it works out of the box or not (without adding the paths with the patch) as it used to run for me as well.

Those paths are default ones. Very strange that they are not propagated into R...

I wonder if on other systems it works out of the box or not (without adding the paths with the patch) as it used to run for me as well.

Those paths are default ones. Very strange that they are not propagated into R...

Sounds great @jameslamb

Sounds great @jameslamb

Thanks to both of you for all the great information, and a nice reproducible example!
I've proposed what I think could be a fix, in https://github.com/microsoft/LightGBM/pull/3779. It wouldn't "just work", but would at least allow you to pass in these paths as command-line args like you can in the Python package, so no one would need to use sed to re-write install.libs.R.

Thanks to both of you for all the great information, and a nice reproducible example!
I've proposed what I think could be a fix, in https://github.com/microsoft/LightGBM/pull/3779. It wouldn't "just work", but would at least allow you to pass in these paths as command-line args like you can in the Python package, so no one would need to use sed to re-write install.libs.R.

Thanks @jameslamb for fix. I changed the Dockerfile in my repo GBM-perf to take advantage of this fix (replaced the sed hack with flags to the build script): https://github.com/szilard/GBM-perf/commit/3b56bf0b474edd5dcf8039c9ddd86cddb9c1d845 Thanks.

Thanks @jameslamb for fix.

Thanks @StrikerRUS , I fixed it now. Yeah, strange indeed it was compiling with the == as well.

Thanks @StrikerRUS , I fixed it now. Yeah, strange indeed it was compiling with the == as well.

Thanks @StrikerRUS , I fixed it now. Yeah, strange indeed it was compiling with the == as well.

Thanks @StrikerRUS , I fixed it now. Yeah, strange indeed it was compiling with the == as well.


Requester portfolio design update

Updated design of requester portfolio (with more relevant information):

one comment, otherwise LGTM

@nistala  : moved the file to fixtures

Merged. Collaboration reference #255


Discussion: forward compatibility and cluster downgrade

Currently RisingWave will maintain backward compatibility for our monthly released versions. In other words, new codes can work with old data/metadata. However, sometimes user may perform downgrade from new version (both release and nightly version) to old version if there are new bugs or performance regression. For example, in #13717, compactor enters a crashloop after a downgrade.
Generally speaking, maintaining forward compatibility is hard so I don't think we should make forward compatibility a requirement. Some ideas to improve the situation:

Maintain forward compatibility in best effort if possible if the code changes are not significantly more complex. Normally we don't need to maintain forward compatibility all the time. Only maintaining it across the recent minor versions should be sufficient.
Add feature flag to guard experimental features that can potentially break forward compatibility. The feature flag should be turn off until we think it is fully stable and production ready.
Encourage user to take a kernel backup before version upgrade (and enforce it in our cloud offering). This ensures that a rollback is always possible after version upgrade with potential data rewind.

Materialized only promise backwards compatibility

+1 for

Encourage user to take a kernel backup before version upgrade (and enforce it in our cloud offering). This ensures that a rollback is always possible after version upgrade with potential data rewind.


When running pytest-pretty in github action the output is still standard pytest

Hello, i've added in my yml the indication to install pytest-pretty
       pip install flake8 pytest rich pytest-pretty
and I call pytest as
    - name: Test with pytest
      run: |
        pytest .
      env:

but the output is the same as what what i had before. Am I doing something wrong?

You must be coming something wrong, it works fine for me on multiple projects.

You must be coming something wrong, it works fine for me on multiple projects.

Could you please show a screenshot of your output and maybe your yml? So that I can debug it


Developement environment preparation

I would suggest you to add instructions on how to build the same development environment as yours. For example, a short introduction on virtual box, vagrant or at least puPHPet would be nice to have as a chapter. Why do I say so? I say so because I tried to run unit tests on Windows computer (I use one with Windows 8.1 OS) and then I got some major errors... I was not able to run unit tests the way you wrote (for example, at page 103 of the first edition book). Running the same commands on Linux results in success.
This is the place you could read more about the problem I got:
http://stackoverflow.com/questions/27999373/why-phpunit-test-does-not-work-zf2
Many thanks for The Book! It helps a lot.

Hi sugalvojau, thanks for the great suggestion! We can think of adding instructions in the website to create such an environment( with Vagrant and Virtualbox ). If there is interest we can include these instructions also in the future versions of the book.

Created the https://github.com/slaff/learnzf2-box repository. The initial version of the Vagrant script will follow soon.

@sugalvojau The box can be downloaded and tested from https://github.com/slaff/learnzf2-box. It would be great if you can test it and give your feedback.

Hi slaff, you are quite fast with changes. Happy to see the result. However, running unit tests does not work for me. I get exactly the same result as before (see the previous link to stackoverflow). I am not so sure why I am going into the same situation over and over again, but I have some assumptions on reasons.. I will test more in coming days. Will give a feedback.

@slaff The line of composer.json file used in the book: "phpunit/phpunit" : "3.7.*"
I am reading the first edition of the book, not so sure if the second edition contains the same line?
I have changed "phpunit/phpunit" : "3.7.*" to "phpunit/phpunit" : "4.5.*" in my own copy of the code as 4.5 version of PHPUnit is the current stable.
Then, of course: php composer.phar update
Then this one will work nicely:
ZF2_PATH=`pwd`/vendor/zendframework/zendframework/library \php vendor/phpunit/phpunit/phpunit -c module/User/tests/
The command above is not the same as in the book.
Tested on Windows 8.1, Ubuntu Trusty 14.04 LTS x64, the isolated development environment you provide. Works well.
There is also a way to make PHPUnit 3.7.* work on all systems (same systems tested by me), but the command must be changed to:
ZF2_PATH=`pwd`/vendor/zendframework/zendframework/library \php vendor/phpunit/phpunit/composer/bin/phpunit -c module/User/tests/
The command above is not the same as in the book.
So... Here we have two ways mentioned to go with PHPUnit tests to make them work. It is up to you to decide witch of two is better. I would prefer the one with increasing PHPUnit version to 4.5 at the moment.
As far as I have tested the isolated development environment you provide - the environment works well. Of course, except the PHPUnit tests part. By the way, as mentioned in stackowerflow, I have tried running PHPUnit 3.7.* in my shared hosting environment (I believe it has RedHat or CentOS installed) and the examples from the book worked well in that hosting environment.
Environment configuration changes to make the environment fit the latest version of the book (vs.) code changes to make code portable. What is the right way to go?


Issues loading sub-modules

First of, great work, I've been looking for something like this for a while now, powerful yet simple to use.
I am trying to import the keras from outside the module. I added the parent folder to my PYTHONPATH variable, but when I run the scripts, i'm getting errors loading the modules below the root:
e.g.
/Users/simon.hughes/GitHub/keras/activations.py in ()
24     return x
25
---> 26 from utils.generic_utils import get_from_module
27 def get(identifier):
28     return get_from_module(identifier, globals(), 'activation function')
ImportError: No module named generic_utils
I've tried adding some of the subfolders to the python path:
sys.path.insert(0, "/Users/simon.hughes/GitHub")
sys.path.insert(0, "/Users/simon.hughes/GitHub/keras")
sys.path.insert(0, "/Users/simon.hughes/GitHub/keras/utils")
But that prevents loading modules like utils.generic_utils. It sees generic_utils as a module, but not utils.generic_utils.
Would it be possible to create a setup.py script to install an egg file? Or is there something simple I can do to make this work? The only way i can run code successfully is from the examples folder.

I have added a setup.py. Hopefully this solves your issue : )

Yes thanks!


The current changes disappear when the file is reopened.

When I'm editing a file and not save the changes (because I am still working on it) and open the same file from the file explorer, It has the last saved changes not the currenty unsaved ones.
I usually do this because I have many tabs open at the same time, so the file is easily lost and it is more convenient to open it from the file explorer.
Steps:

Open a file a start editing it and do not save it.
Open too many tabs at the same time (I do not know If this is caused by a memory perfomance but only occurs when too many tabs are opened).
Navigate between the tabs until the edited file is lost.
Open it from the file explorer, It should has the last saved changes.

/gifPlease

Thanks for reporting this issue! Unfortunately, it's hard for us to understand what issue you're seeing. Please help us out by providing a screen recording showing exactly what isn't working as expected. While we can work with most standard formats, .gif files are preferred as they are displayed inline on GitHub. You may find https://gifcap.dev helpful as a browser-based gif recording tool.
If the issue depends on keyboard input, you can help us by enabling screencast mode for the recording (Developer: Toggle Screencast Mode in the command palette).
Happy coding!

I think this was fixed in a recent version, I can no longer replicate it.
Thank you.


Fixes broken readme example

The example in the md had some issue.

nice one, let me know if ready to merge

Ready :)

Releasing now in 3.3.1

Great! :)

I (unfortunately) still have ideas to improve ;-) I'm also going to publish a neo4j-jest preset using this project. I started using neo4j because I'm a little tired of SQL and mongo. And it seems there is many tooling missing in the Node world for neo.


Relax constaints on sizeless types

I don't know if it is proper to open this issue here.
Currently, Clang doesn't allow us to use RVV intrinsic types inside struct. The same as SVE intrinsic types. However, I noticed that Rust does allow sizeless types in structs, with a constraint that it must be the last field of the struct. Can we also implement this in C++? I think this feature is very useful for those wrapper libraries, and makes RVV intrinsic easier to use.

This is a duplicate question with #139. Marking this as duplicate and closing.
This is definitely a topic we will dwell on, but the intrinsic API needs a v1.0 release now. So the discussion will be held later.


Update Project Invoicing

Fixes https://github.com/HyphaApp/hypha/issues/2354

@frjo, this branch is rebased on top of Add Vendor https://github.com/HyphaApp/hypha/pull/2401; And adds the invoicing feature mentioned in https://github.com/HyphaApp/hypha/issues/2354
This could also be deployed on test to test out the invoicing feature.
I have not yet removed Payment Requests models etc..it would be easier to remove them when we have completed the flow with Invoicing(i.e. approval etc.)..as then Payment Requests models won't have any references at other places.
I have also added the feature to update invoice status..which I see was not mentioned in https://github.com/HyphaApp/hypha/issues/2354, let me know if we need to disable it for this pr..we can test that out later when we work on the next stage of invoicing.

@sks444 I have committed https://github.com/HyphaApp/hypha/pull/2401 now so this can be rebased of main and then I can deploy it to test.

Done @frjo.
We can deploy this feature to test for testing.
I will fix the failing tests in the meantime.

Deploying to test now.

Deployed the latest to test now.

@sks444 This has been on test for some time and it seems to work as it should. Can we merge this in to main now or do you want to add more to it first?

@frjo, Yes, this can be merged, thanks!