[WIP][R-package] miscellaneous changes to comply with CRAN requirements

This PR attempts to address the most recent request for changes from CRAN, https://github.com/microsoft/LightGBM/issues/3293#issuecomment-682001047, on our way to #629
See changes to cran-comments.md for details.

@guolinke here is a package built from this branch, that can be submitted:
lightgbm_3.0.0.tar.gz

BTW, is it still pre-checks from CRAN? Because I don't see errors for Solaris that we haven't fixed yet.

We made it past the pre-checks (which is just Debian Linux and Windows) and are now into human review.
I think it is possible that we'll be accepted, then fail the Solaris check, and then be told "hey you are failing the Solaris check, you have 30 days to fix it or we're taking you down".
It's also possible that they'll allow us to stay up indefinitely without passing that check. It isn't easy to understand what the exact expectation is :grimacing:
There are some packages currently in ERROR status on Solaris that have been allowed to stay on CRAN: https://cran.r-project.org/web/checks/check_details_r-patched-solaris-x86.html

... or we're taking you down"

Sounds scary!
I hope that we will be able to adopt that patch and it will work.

... or we're taking you down"

Sounds scary!
I hope that we will be able to adopt that patch and it will work.

haha yeah, it can happen.
I'm going to try that patch this weekend (maybe even tonight) based on this advice: https://github.com/microsoft/LightGBM/issues/629#issuecomment-668920988
worst-case scenario, if that doesn't work we can figure out how to disable distributed training on Solaris. I doubt anyone trying to do distributed training in LightGBM is going to provision a cluster of Solaris 10 machines :joy:
I think that should be the only thing the network code is needed for.

@jameslamb
Dear maintainer,
package lightgbm_3.0.0.tar.gz does not pass the incoming checks automatically, please see the following pre-tests:
Windows: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_3.0.0_20200828_032044%2FWindows%2F00check.log&data=02|01|guolin.ke%40microsoft.com|822df97f9a794c5659cc08d84b150b28|72f988bf86f141af91ab2d7cd011db47|1|0|637341901057745862&sdata=Y9kE%2FpOSK7VaxLisXcruFs8vdqH%2B7D75ZcSVRwOi524%3D&reserved=0
Status: 2 NOTEs
Debian: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_3.0.0_20200828_032044%2FDebian%2F00check.log&data=02|01|guolin.ke%40microsoft.com|822df97f9a794c5659cc08d84b150b28|72f988bf86f141af91ab2d7cd011db47|1|0|637341901057745862&sdata=EjkSroKsewumq2f8mHt5BhOUE30KPAVfcKmIh9yKcGo%3D&reserved=0
Status: 2 NOTEs
Please fix all problems and resubmit a fixed version via the webform.
If you are not sure how to fix the problems shown, please ask for help on the R-package-devel mailing list:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-package-devel&data=02|01|guolin.ke%40microsoft.com|822df97f9a794c5659cc08d84b150b28|72f988bf86f141af91ab2d7cd011db47|1|0|637341901057755854&sdata=bOgHMSqtOTVA0ZHVgjvMwdeLFLmBKcS%2F3Bs2hFz0U9I%3D&reserved=0
If you are fairly certain the rejection is a false positive, please reply-all to this message and explain.
More details are given in the directory:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_3.0.0_20200828_032044%2F&data=02|01|guolin.ke%40microsoft.com|822df97f9a794c5659cc08d84b150b28|72f988bf86f141af91ab2d7cd011db47|1|0|637341901057755854&sdata=ltIZ2uMZK38H7%2FLc0X7CYD8gfyuXhQXF3S7v7OdrNgk%3D&reserved=0
The files will be removed after roughly 7 days.
No strong reverse dependencies to be checked.
Best regards,
CRAN teams' auto-check service

more details:
Flavor: r-devel-linux-x86_64-debian-gcc, r-devel-windows-ix86+x86_64
Check: CRAN incoming feasibility, Result: NOTE
  Maintainer: 'Guolin Ke<EMAIL_ADDRESS>
  New submission

  Possibly mis-spelled words in DESCRIPTION:
    Guolin (13:52)
    Ke (13:48)
    LightGBM (14:20)
    al (13:62)
    et (13:59)

Flavor: r-devel-linux-x86_64-debian-gcc, r-devel-windows-ix86+x86_64
Check: top-level files, Result: NOTE
  Non-standard files/directories found at top level:
    'docs' 'lightgbm-hex-logo.png' 'lightgbm-hex-logo.svg'

ahhhh sorry! the image files are my mistake. I expect they'll allow us to keep those other words because they ard not misspellings.
I'll push a change here and attach a new package in a few minutes

Ok I've made some changes. Could you try this one? After you submit,  could you reply-all to that email and tell them that we have submitted a fix for the unnecessary files, but that we think the "mispellings" note is a false  positive?
lightgbm_3.0.0.tar.gz

@jameslamb
still waiting for the human check...
BTW, do we need to wait for several days every time we update the new version to CRAN?

I think we cannot block the release plan by cran.
If we still don't get the reply tomorrow, maybe we should release the 3.0.0 version for github/pypi first.

I think we cannot block the release plan by cran.
If we still don't get the reply tomorrow, maybe we should release the 3.0.0 version for github/pypi first.

I'm ok with that! Let's not hold up #3293 any longer. It's ok for CRAN to come a bit later.

BTW, do we need to wait for several days every time we update the new version to CRAN?

We shouldn't have to. In my experience, projects are scrutinized much more closely the first time they're submitted to CRAN. Once we've been accepted once, we should be able to get accepted automatically for future releases.

@StrikerRUS @guolinke I found this interesting site that checks the status of CRAN submissions!
https://lockedata.github.io/cransays/articles/dashboard.html
It has a dashboard that is the result of polling ftp://cran.r-project.org/incoming/. It shows we're stuck in newbies:

a specific queue for the manual inspection of first time CRAN submissions.

When I checked https://lockedata.github.io/cransays/articles/dashboard.html last night, there were 16 packages ahead of us (in "newbies" for longer than we've been).
Now there are only two. So I hope we'll hear back in the next 24 hours!

I've just uploaded a 3.0.0 source package, in response to #3350 .
https://github.com/microsoft/LightGBM/releases/download/v3.0.0/lightgbm-3.0.0-r-cran.tar.gz
I'll update it if CRAN asks for changes and we have to update this PR, but this was our users can install the 3.0.0  release of the R package easily.
I can prepare binaries in a day or two. And have not forgotten about doing #3283 so that happens automatically.

Please write references in the description of the DESCRIPTION file in
the form
authors (year) doi:...
authors (year) arXiv:...
authors (year, ISBN:...)
or if those are not available: authors (year) https:...
with no space after 'doi:', 'arXiv:', 'https:' and angle brackets for
auto-linking.
(If you want to add a title as well please put it in quotes: "Title")
\dontrun{} should only be used if the example really cannot be executed
(e.g. because of missing additional software, missing API keys, ...) by
the user. That's why wrapping examples in \dontrun{} adds the comment
("# Not run:") as a warning for the user.
Does not seem necessary.
Please unwrap the examples if they are executable in < 5 sec, or replace
\dontrun{} with \donttest{}.
Please do not modify the global environment (e.g. by using <<-) in your
functions. This is not allowed by the CRAN policies.
Please always add all authors, contributors and copyright holders in the
Authors@R field with the appropriate roles.
From CRAN policies you agreed to:
"The ownership of copyright and intellectual property rights of all
components of the package must be clear and unambiguous (including from
the authors specification in the DESCRIPTION file). Where code is copied
(or derived) from the work of others (including from R itself), care
must be taken that any copyright/license statements are preserved and
authorship is not misrepresented."
e.g.: Microsoft Corporation, Dropbox Inc.
Please explain in the submission comments what you did about this issue.
Please fix and resubmit.

gaaaaah ok, let me fix these.

ok @guolinke can you try again with the attached package and this response?
Thank you for your thorough review. We believe this new submission addresses all questions.

The paper citation has been adjusted as requested. We were using 'glmnet' as a  guide on how to include the URL but maybe they are no longer in compliance with CRAN policies: https://github.com/cran/glmnet/blob/b1a4b50de01e0cd24343959d7cf86452bac17b26/DESCRIPTION

All authors from the original LightGBM paper have been added to Authors@R as "aut". We have also added Microsoft and DropBox, Inc. as "cph" (copyright holders). These roles were chosen based on the guidance in https://journal.r-project.org/archive/2012-1/RJournal_2012-1_Hornik~et~al.pdf.

lightgbm's code does use '<<-', but it does not modify the global environment.  The uses of '<<-' in R/lgb.interprete.R and R/callback.R are in functions which are called in an environment created by the lightgbm functions that call them, and this operator is used to reach one level up into the calling function's environment.

We chose to wrap our examples in \dontrun{} because we found, through testing on https://builder.r-hub.io/ and in our own continuous integration environments, that their run time varies a lot between platforms, and we cannot guarantee that all examples will run in under 5 seconds. We intentionally chose \dontrun{} over \donttest{] because this item in the R 4.0.0 changelog (https://cran.r-project.org/doc/manuals/r-devel/NEWS.html) seems to indicate that \donttest will be ignored by CRAN's automated checks: "R CMD check --as-cran now runs \donttest examples (which are run by example()) instead of instructing the tester to do so. This can be temporarily circumvented during development by setting environment variable _R_CHECK_DONTTEST_EXAMPLES_ to a false value."

We run all examples with 'R CMD check --as-cran --run-dontrun' in our continuous integration tests on every commit to the package, so we have high confidence that they are working correctly.

lightgbm_3.0.0.tar.gz

by the way, this is why  DropBox has to be included as a  copyright holder: https://github.com/microsoft/LightGBM/blob/ca066d499d4b58f74fbd7ab875855cb9f3cb07a1/src/io/json11.cpp#L1
it doesn't mean they hold a copyright on the whole  package. Just on that code we've copied in from them.

and here is what I changed for https://github.com/microsoft/LightGBM/pull/3338#issuecomment-687211351:
https://github.com/microsoft/LightGBM/pull/3338/commits/541d2d96146de2ab14883acc37b81f4990bd15da

@jameslamb oh, very sorry, it seems I accidentally merge master to this PR.
Is that okay for the cran release?

@guolinke yep not a problem! I can reset VERSION.txt and R-package/configure on this branch

This is so awesome! Thank you for working on getting this onto CRAN. I'll be the flag waiver to get it incorporated into tidymodels and modeltime for Time series.

Thanks, if the running time is the only reason to wrap the examples in
\dontrun, please replace \dontrun by \donttest (\donttest examples are
not executed in the CRAN checks).
Please replace cat() by message() or warning() in your functions (except
for print() and summary() functions). Messages and warnings can be
suppressed if needed.
Missing Rd-tags:
lightgbm/man/dimnames.lgb.Dataset.Rd: \value
lightgbm/man/lgb.Dataset.construct.Rd: \value
lightgbm/man/lgb.prepare.Rd: \value
....
Please add the tag and explain in detail the returned objects.
Please fix and resubmit.
Best,
Swetlana Herbrandt

You are getting close. The way CRAN works is to 1st focus on your DESCRIPTION. They seem to be ok with it now. This is actually progress.
Next they work on your code. They have a few issues with cat() vs message() which is actually a good call.
Next they evaluate your tests.
The toughest part is honestly the DESCRIPTION and documentation. They always find broken links and their responses can be cryptic.
But I think you are close based on what they are looking at.

Once you get it on CRAN, future submissions will be much faster because the initial barriers have been passed.

ok @guolinke could you please submit the attached, with this comment?
All examples have been wrapped with `\donttest` as requested. We have replied to Swetlana Herbrandt asking for clarification on the donttest news item in the R 4.0.2 changelog (https://cran.r-project.org/doc/manuals/r-devel/NEWS.html).

All uses of `cat()` have been replaced with `print()`. We chose `print()` over `message()` because it's important that they be written to stdout alongside all the other logs coming from the library's C++ code. `message()` and `warning()` write to stderr.

All exported objects now have `\value{}` statements in their documentation files in `man/`.

lightgbm_3.0.0.tar.gz
After that, can you please reply-all directly to that most recent CRAN  email,  with this message?
Swetlana,

Thank you for your comments. I've just created a new submission that I believe addresses them.

Can you help us understand something? In your message you said "\donttest examples are
not executed in the CRAN checks)", but in https://cran.r-project.org/doc/manuals/r-devel/NEWS.html  we see the following:

> "`R CMD check --as-cran` now runs \donttest examples (which are run by example()) instead of instructing the tester to do so. This can be temporarily circumvented during development by setting environment variable `_R_CHECK_DONTTEST_EXAMPLES_` to a false value."

Could you help us understand how both of those statements can be true?

@StrikerRUS I removed your ✅ . No need to re-review yet, but I think that by the time this is done there will be a lot of changes to review.

ugh I just realized  the changes I made (replacing cat() with print()) now prints a literal \t

I'll fix it in the next revision (either the next submission we have to make for 3.0.0 or the next release of  {lightgbm}). Sorry :/

@jameslamb
just get the reply.
The reguklar CRAN checks do not execute the \donttest examples and hence they do not add to the checktime we are looking at. But irregularly, we run the donttest examples to see if they work - knowing we need more resources for that.
Best,
Uwe Ligges

ah I see, ok. I still think their wording was very confusing but I understand now. Thank you!

@jameslamb lt seems we are stuck for 9 days...

@guolinke ugh I know, it's so frustrating! I think that CRAN fell behind a lot when they had 10 days completely shut down, and now it's hard for them to catch up. As long as we're still in the "newbies" folder I think it's ok and we should eventually expect an answer.
I'll start taking some of the changes from this pull request (like \dontrun being changed to \donttest) and putting them in other pull requests, so they aren't all stuck waiting on this PR.

we just have to be careful not to merge master into this branch any more so that the 3.0.0 release is really 3.0.0.
If we get accepted, this will go a lot faster for our next releases. They just make it really difficult the first time.

Thanks,
on its way to CRAN.
Best,
Gregor Seyer

@jameslamb
Dear maintainer,
Please see the problems shown on
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fweb%2Fchecks%2Fcheck_results_lightgbm.html&data=02|01|guolin.ke%40microsoft.com|1a450f60f5344fbdc18c08d85e29f17a|72f988bf86f141af91ab2d7cd011db47|1|0|637362881848351418&sdata=Z7HjK6TKtWqYVdBU7t%2Fk56sVWYqgPQz4OKhWoh4Xxvg%3D&reserved=0.
Please correct before 2020-10-05 to safely retain your package on CRAN.
The CRAN Team

@jameslamb
Dear maintainer,
Please see the problems shown on
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fweb%2Fchecks%2Fcheck_results_lightgbm.html&data=02|01|guolin.ke%40microsoft.com|1a450f60f5344fbdc18c08d85e29f17a|72f988bf86f141af91ab2d7cd011db47|1|0|637362881848351418&sdata=Z7HjK6TKtWqYVdBU7t%2Fk56sVWYqgPQz4OKhWoh4Xxvg%3D&reserved=0.
Please correct before 2020-10-05 to safely retain your package on CRAN.
The CRAN Team

exciting! It's our friend the Solaris error. I think that @StrikerRUS 's solution is a good one, I can test it tonight. https://github.com/microsoft/LightGBM/issues/629#issuecomment-665091451
unfortunately I think we need to keep this pull request open and add it here...otherwise we'll be putting changes that came after 3.0.0 into the 3.0.0 release of the R package.

I think that the Solaris issue is the only thing we need to fix. The "installed size" note can be safely ignored.
{xgboost}, for example, has the "installed library size" NOTE on many operating systems: https://cran.r-project.org/web/checks/check_results_xgboost.html

whenever we eventually are read to merge this, bumping the version will fix CI

@jameslamb I had experience with debugging Solaris checks with XGBoost. If you get stuck, let me know and I can find some time next week.

@jameslamb I had experience with debugging Solaris checks with XGBoost. If you get stuck, let me know and I can find some time next week.

thanks! I did consider your suggestion about the VM from https://github.com/jeroen/solarisvm but the version of R in that image is too old for {lightgbm} (3.3.0, we require >= 3.5.0).
I had ok success with R Hub last night. It was taking about 15 minutes for each run. So a slow feedback cycle, but I think I'm making progress with it. If I don't get it in the next couple days, I will definitely ask for you help!

@jameslamb Got it. XGBoost only requires R 3.3.0+, so it worked for us. If push comes to shove, we would need to try installing latest R in the VM image.

@guolinke I've been trying too add the ifaddrs patch for Solaris. Have tried a lot of small changes, but I'm stuck. Can you or @shiyu1994 help me out?
I kind of expect that I am just doing something simple incorrectly.
this commit: https://github.com/microsoft/LightGBM/pull/3338/commits/8b6a3a711cdaca343f72f7c03598948837dc90f2
#> In file included from network/linkers.h:22:0,
 854#> from network/linkers_socket.cpp:19:
 855#> network/socket_wrapper.hpp: In function ‘void LightGBM::freeifaddrs(ifaddrs*)’:
 856#> network/socket_wrapper.hpp:89:30: error: call of overloaded ‘freeifaddrs(ifaddrs*&)’ is ambiguous
 857#> freeifaddrs(ifp->ifa_next);
 858#> ^
 859#> network/socket_wrapper.hpp:82:6: note: candidate: void LightGBM::freeifaddrs(ifaddrs*)
 860#> void freeifaddrs (struct ifaddrs *ifp)
 861#> ^
 862#> In file included from network/socket_wrapper.hpp:44:0,
 863#> from network/linkers.h:22,
 864#> from network/linkers_socket.cpp:19:
 865#> network/ifaddrs_patch.h:28:6: note: candidate: void freeifaddrs(ifaddrs*)
 866#> void freeifaddrs(struct ifaddrs *ifp);
 867#> ^
 868#> In file included from network/linkers.h:22:0,
 869#> from network/linkers_socket.cpp:19:
 870#> network/socket_wrapper.hpp: In function ‘int LightGBM::getifaddrs(ifaddrs**)’:
 871#> network/socket_wrapper.hpp:171:22: error: call of overloaded ‘freeifaddrs(ifaddrs*&)’ is ambiguous
 872#> freeifaddrs(*ifap);

The relevant code is here: https://github.com/giampaolo/psutil/tree/master/psutil/arch/solaris/v10
Right now I'm not worrying about #ifdef ON_SOLARIS types of guards. I can figure that out later.
I'm testing this on R Hub, using this  code
sh build-cran-package.sh

and in R:
result <- rhub::check(
    path = "lightgbm_3.0.0.tar.gz"
    , email =<EMAIL_ADDRESS>    , check_args = c(
        "--as-cran"
    )
    , platform = c(
        "solaris-x86-patched"
        , "solaris-x86-patched-ods"
    )
    , env_vars = c(
        "R_COMPILE_AND_INSTALL_PACKAGES" = "always"
    )
)

Hi @jameslamb did you include .c file? https://github.com/giampaolo/psutil/blob/master/psutil/arch/solaris/v10/ifaddrs.c

Hi @jameslamb did you include .c file? https://github.com/giampaolo/psutil/blob/master/psutil/arch/solaris/v10/ifaddrs.c

I included that code directly in socket_wrapper.hpp. Does that not work?

@jameslamb I think it is better to use the .c style.
And some headers seems missing: https://github.com/giampaolo/psutil/blob/f9b1b5e431ecbecf76b255737d5738ba2c69c1ba/psutil/arch/solaris/v10/ifaddrs.c#L10-L15
And there is an extern keyword. I remember that the implementation of extern var/function should be in c/cpp file.

or you can try to remove these extern declarations.

@guolinke thanks for your help. Based on your help, I was able to get this working!!!!
I built a package as of https://github.com/microsoft/LightGBM/pull/3338/commits/e7eb0c4ccded1fa189a8440c93ae6b4bf95173ec and submitted it to R Hub. The ERRORs are gone on Solaris, replaced by one warning and one notte
 916#> /opt/csw/share/aclocal/gtk.m4:7: warning: underquoted definition of AM_PATH_GTK
 917#> /opt/csw/share/aclocal/gtk.m4:7: run info Automake 'Extending aclocal'
 918#> /opt/csw/share/aclocal/gtk.m4:7: or see https://www.gnu.org/software/automake/manual/automake.html#Extending-aclocal

 961#> * checking compilation flags used ... NOTE
 962#> Compilation used the following non-portable flag(s):
 963#> ‘-march=pentiumpro’

I don't think either of these are things we control, so I'm hoping they will not show up on CRAN
full R CMD CHECK logs (click me)
        #> * using log directory ‘/export/home/X5aMYXV/lightgbm.Rcheck’
 888#> * using R version 4.0.2 (2020-06-22)
 889#> * using platform: i386-pc-solaris2.10 (32-bit)
 890#> * using session charset: UTF-8
 891#> * using option ‘--as-cran’
 892#> * checking for file ‘lightgbm/DESCRIPTION’ ... OK
 893#> * checking extension type ... Package
 894#> * this is package ‘lightgbm’ version ‘3.0.0’
 895#> * package encoding: UTF-8
 896#> * checking CRAN incoming feasibility ... WARNING
 897#> Maintainer: ‘Guolin Ke ’
 898#> Insufficient package version (submitted: 3.0.0, existing: 3.0.0)
 899#> Days since last update: 4
 900#> * checking package namespace information ... OK
 901#> * checking package dependencies ... OK
 902#> * checking if this is a source package ... OK
 903#> * checking if there is a namespace ... OK
 904#> * checking for executable files ... OK
 905#> * checking for hidden files and directories ... OK
 906#> * checking for portable file names ... OK
 907#> * checking for sufficient/correct file permissions ... OK
 908#> * checking whether package ‘lightgbm’ can be installed ... OK
 909#> * checking installed package size ... OK
 910#> * checking package directory ... OK
 911#> * checking for future file timestamps ... NOTE
 912#> unable to verify current time
 913#> * checking DESCRIPTION meta-information ... OK
 914#> * checking top-level files ... WARNING
 915#> Output from running autoreconf:
 916#> /opt/csw/share/aclocal/gtk.m4:7: warning: underquoted definition of AM_PATH_GTK
 917#> /opt/csw/share/aclocal/gtk.m4:7: run info Automake 'Extending aclocal'
 918#> /opt/csw/share/aclocal/gtk.m4:7: or see https://www.gnu.org/software/automake/manual/automake.html#Extending-aclocal
 919#> A complete check needs the 'checkbashisms' script.
 920#> See section ‘Configure and cleanup’ in the ‘Writing R Extensions’
 921#> manual.
 922#> Files ‘README.md’ or ‘NEWS.md’ cannot be checked without ‘pandoc’ being installed.
 923#> * checking for left-over files ... OK
 924#> * checking index information ... OK
 925#> * checking package subdirectories ... OK
 926#> * checking R files for non-ASCII characters ... OK
 927#> * checking R files for syntax errors ... OK
 928#> * checking whether the package can be loaded ... OK
 929#> * checking whether the package can be loaded with stated dependencies ... OK
 930#> * checking whether the package can be unloaded cleanly ... OK
 931#> * checking whether the namespace can be loaded with stated dependencies ... OK
 932#> * checking whether the namespace can be unloaded cleanly ... OK
 933#> * checking loading without being on the library search path ... OK
 934#> * checking use of S3 registration ... OK
 935#> * checking dependencies in R code ... OK
 936#> * checking S3 generic/method consistency ... OK
 937#> * checking replacement functions ... OK
 938#> * checking foreign function calls ... OK
 939#> * checking R code for possible problems ... OK
 940#> * checking Rd files ... OK
 941#> * checking Rd metadata ... OK
 942#> * checking Rd line widths ... OK
 943#> * checking Rd cross-references ... OK
 944#> * checking for missing documentation entries ... OK
 945#> * checking for code/documentation mismatches ... OK
 946#> * checking Rd \usage sections ... OK
 947#> * checking Rd contents ... OK
 948#> * checking for unstated dependencies in examples ... OK
 949#> * checking contents of ‘data’ directory ... OK
 950#> * checking data for non-ASCII characters ... OK
 951#> * checking data for ASCII and uncompressed saves ... OK
 952#> * checking line endings in shell scripts ... OK
 953#> * checking line endings in C/C++/Fortran sources/headers ... OK
 954#> * checking line endings in Makefiles ... OK
 955#> * checking compilation flags in Makevars ... OK
 956#> * checking for GNU extensions in Makefiles ... OK
 957#> * checking for portable use of $(BLAS_LIBS) and $(LAPACK_LIBS) ... OK
 958#> * checking use of PKG_*FLAGS in Makefiles ... OK
 959#> * checking use of SHLIB_OPENMP_*FLAGS in Makefiles ... OK
 960#> * checking pragmas in C/C++ headers and code ... OK
 961#> * checking compilation flags used ... NOTE
 962#> Compilation used the following non-portable flag(s):
 963#> ‘-march=pentiumpro’
 964#> * checking compiled code ... OK
 965#> * checking examples ... OK
 966#> * checking examples with --run-donttest ... OK
 967#> * checking for unstated dependencies in ‘tests’ ... OK
 968#> * checking tests ...
 969#> Running ‘testthat.R’ [11s/12s]
 970#> OK
 971#> * checking PDF version of manual ... OK
 972#> * checking for non-standard things in the check directory ... OK
 973#> * checking for detritus in the temp directory ... OK
 974#> * DONE
 975#> Status: 2 WARNINGs, 2 NOTEs
 976#> See
 977#> ‘/export/home/X5aMYXV/lightgbm.Rcheck/00check.log’
 978#> for details.

I am going to work now on adding a define -DON_SOLARIS and the code in configure.ac to detect that, so this patch is only used on Solaris. I'll have a package ready to submit in a few hours.

ok @guolinke , could you please submit the attached package, with a note that we think it will fix the ERROR on Solaris at https://cran.r-project.org/web/checks/check_results_lightgbm.html?
I checked the logs of all the failing CI jobs...they are passing all unit tests, and just failing because of this thing we can ignore:
* checking CRAN incoming feasibility ... WARNING
Maintainer: ‘Guolin Ke<EMAIL_ADDRESS>
Insufficient package version (submitted: 3.0.0, existing: 3.0.0)

Days since last update: 4

Note that this version includes adding three more copyright holders to DESCRIPTION (the authors or the ifaddrs patch). That patch is licensed under BSD 3-Clause so it's safe to use.
lightgbm_3.0.0.tar.gz

Thanks @jameslamb
We still use the cran-r submit web form to update?

yes it's the same submit form. But they'll automatically detect that we are already on cran, so it should go faster.
I don't know if we have to increment the version for this change...hopefully not

Dear maintainer,
package lightgbm_3.0.0.tar.gz does not pass the incoming checks automatically, please see the following pre-tests:
Windows: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_3.0.0_20200925_042449%2FWindows%2F00check.log&data=02|01|guolin.ke%40microsoft.com|e80d55ea1ac044b2862d08d8610de4e9|72f988bf86f141af91ab2d7cd011db47|1|0|637366060120388634&sdata=96XA6NtTogozB6%2BbSTIhuns7qYH9AqIt7WvmHMKacDQ%3D&reserved=0
Status: 1 WARNING
Debian: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_3.0.0_20200925_042449%2FDebian%2F00check.log&data=02|01|guolin.ke%40microsoft.com|e80d55ea1ac044b2862d08d8610de4e9|72f988bf86f141af91ab2d7cd011db47|1|0|637366060120388634&sdata=P4fIiBaxjTSsgl9L9iTJ8WpTeIXaYwzVGcGfj9CyXRk%3D&reserved=0
Status: 1 WARNING
Last released version's CRAN status: ERROR: 1, NOTE: 5, OK: 5
See: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fweb%2Fchecks%2Fcheck_results_lightgbm.html&data=02|01|guolin.ke%40microsoft.com|e80d55ea1ac044b2862d08d8610de4e9|72f988bf86f141af91ab2d7cd011db47|1|0|637366060120388634&sdata=nsCiF%2ByOR2zr26B4ABkG0CulCVarHWoLV6ZZR7feWww%3D&reserved=0
Last released version's additional issues:
gcc-UBSAN https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.stats.ox.ac.uk%2Fpub%2Fbdr%2Fmemtests%2Fgcc-UBSAN%2Flightgbm&data=02|01|guolin.ke%40microsoft.com|e80d55ea1ac044b2862d08d8610de4e9|72f988bf86f141af91ab2d7cd011db47|1|0|637366060120388634&sdata=pk%2F%2FmrnKbUBUAsdfxoOHetCEOr0rzIMvwS8DneGCLSA%3D&reserved=0
valgrind https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.stats.ox.ac.uk%2Fpub%2Fbdr%2Fmemtests%2Fvalgrind%2Flightgbm&data=02|01|guolin.ke%40microsoft.com|e80d55ea1ac044b2862d08d8610de4e9|72f988bf86f141af91ab2d7cd011db47|1|0|637366060120388634&sdata=o5sRGC4Vo%2BkKZMdnl07MjRykyHwPy2u1JIE5%2B6%2BS%2BSc%3D&reserved=0
CRAN Web: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fpackage%3Dlightgbm&data=02|01|guolin.ke%40microsoft.com|e80d55ea1ac044b2862d08d8610de4e9|72f988bf86f141af91ab2d7cd011db47|1|0|637366060120388634&sdata=1du44oCOzChFspguawCG4tIMYkJS0j2Ua9PpdatmFsc%3D&reserved=0
Please fix all problems and resubmit a fixed version via the webform.
If you are not sure how to fix the problems shown, please ask for help on the R-package-devel mailing list:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstat.ethz.ch%2Fmailman%2Flistinfo%2Fr-package-devel&data=02|01|guolin.ke%40microsoft.com|e80d55ea1ac044b2862d08d8610de4e9|72f988bf86f141af91ab2d7cd011db47|1|0|637366060120398591&sdata=hcH6d5xAE%2BEF21v7l%2F6pOliuyftCEmHUgtH2VW%2FWwv4%3D&reserved=0
If you are fairly certain the rejection is a false positive, please reply-all to this message and explain.
More details are given in the directory:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_3.0.0_20200925_042449%2F&data=02|01|guolin.ke%40microsoft.com|e80d55ea1ac044b2862d08d8610de4e9|72f988bf86f141af91ab2d7cd011db47|1|0|637366060120398591&sdata=G1ZTY6UklXtux1IGgE%2FBfBc2sUNRx50P4xLnWfK%2B67w%3D&reserved=0
The files will be removed after roughly 7 days.
No strong reverse dependencies to be checked.
Best regards,
CRAN teams' auto-check service

oy ok so we do have to bump the version. Can I make it <IP_ADDRESS>?
And then we'll have 3-part version numbers for all future releases

yes, I think we can

yes, I think we can

ok thanks! I just don't think it's worth the work of doing a full LightGBM release (of all components) for this, since this branch is so old already.
You can use this package with the version bumped to <IP_ADDRESS>:
lightgbm_<IP_ADDRESS>.tar.gz

also hey! It's great that we got a response from CRAN within a couple hours. A really nice benefit of getting out of the "new submission" group 😎

@jameslamb

I am going to work now on adding a define -DON_SOLARIS and the code in configure.ac to detect that, so this patch is only used on Solaris.

Maybe we can avoid adding new custom flags and use built-in ones?
https://sourceforge.net/p/predef/wiki/OperatingSystems/
https://gist.github.com/ByteProject/320483/f31b9524c367aad857fd1d4496af3c7e73e14291#file-ptarget-h-L137-L143

@jameslamb

I am going to work now on adding a define -DON_SOLARIS and the code in configure.ac to detect that, so this patch is only used on Solaris.

Maybe we can avoid adding new custom flags and use built-in ones?
https://sourceforge.net/p/predef/wiki/OperatingSystems/
https://gist.github.com/ByteProject/320483/f31b9524c367aad857fd1d4496af3c7e73e14291#file-ptarget-h-L137-L143

oooo yeah that would be great, and it would make this change portable beyond just the CRAN R package. I'll try that tonight!
But if our current submission is accepted and passes checks on Solaris, I think we should update the version on this PR back to <IP_ADDRESS>, and then merge this PR. And that change can be a separate PR towards the next LightGBM release (which will be 3.1.0 I think, based on the PRs that have been merged recently)

@guolinke did you get an email from CRAN? It looks like we are in "pending" right now on https://lockedata.github.io/cransays/articles/dashboard.html, which they say means this:

the CRAN maintainers are waiting for an action on your side. You should check your emails

There is one email, but it is related to 3.0.0

Dear package maintainer,
this notification has been generated automatically.
Your package lightgbm_3.0.0.tar.gz has been built for Windows and will be published within 24 hours in the corresponding CRAN directory.
R version 4.0.2 (2020-06-22)
All the best,
Uwe Ligges
(Maintainer of binary packages for Windows)

There is one email, but it is related to 3.0.0
Dear package maintainer,
this notification has been generated automatically.
Your package lightgbm_3.0.0.tar.gz has been built for Windows and will be published within 24 hours in the corresponding CRAN directory.
R version 4.0.2 (2020-06-22)
All the best,
Uwe Ligges
(Maintainer of binary packages for Windows)

sad, ok! Well at least from https://lockedata.github.io/cransays/articles/dashboard.html  that CRAN is slowly catching up from the backlog they built up during their break in August. In late August there 150 submissions in progress, now there are 101.
Can you please reply-all to the message from https://github.com/microsoft/LightGBM/pull/3338#issuecomment-698723866 with something like this?

We submitted a new version of 'lightgbm' (<IP_ADDRESS>) 2 days ago, with fixes for these issues. I see that that submission is in "pending" (ftp://cran.r-project.org/incoming/pending/), which we understand to mean that you are waiting for something from us. Can you please tell us what other action or information you need?

Thanks, we see:
Still lots of alignment errors, such as
lightgbm.Rcheck/tests/testthat.Rout:io/dataset_loader.cpp:340:59:
runtime error: reference binding to misaligned address 0x7f51fefad81e for type 'const value_type', which requires 4 byte alignment
lightgbm.Rcheck/tests/testthat.Rout:/usr/include/c++/10/bits/stl_vector.h:1198:21:
runtime error: reference binding to misaligned address 0x7f51fefad81e for type 'const int', which requires 4 byte alignment lightgbm.Rcheck/tests/testthat.Rout:/usr/include/c++/10/bits/vector.tcc:449:28:runtime
error: reference binding to misaligned address 0x7f51fefad81e for type 'const type', which requires 4 byte alignment
lightgbm.Rcheck/tests/testthat.Rout:/usr/include/c++/10/bits/move.h:77:36:
runtime error: reference binding to misaligned address 0x7f51fefad81e for type 'const int', which requires 4 byte alignment
lightgbm.Rcheck/tests/testthat.Rout:/usr/include/c++/10/bits/alloc_traits.h:512:17:
runtime error: reference binding to misaligned address 0x7f51fefad81e for type 'const type', which requires 4 byte alignment
Please fix and resubmit.
Best,
Uwe Ligges

@jameslamb do we have the detailed logs for this check?

ugh this is so frustrating. I am annoyed that their response didn't include a link to logs (to tell us if the solaris patch worked on their setup) or any other information about which operating system they see errors on!
Can you reply and ask for a link to logs?
From the R Hub checks I ran a while ago, I remember seeing such misalignment errors on only one environment: Debian Linux, R-devel, GCC ASAN/UBSAN.

https://github.com/microsoft/LightGBM/issues/629#issuecomment-664714892
https://github.com/microsoft/LightGBM/issues/629#issuecomment-665418697

@guolinke I remember you had a theory on this, https://github.com/microsoft/LightGBM/issues/629#issuecomment-664806996, but at the time I didn't think it was what happened (https://github.com/microsoft/LightGBM/issues/629#issuecomment-665075114)

just got this from cran. https://www.stats.ox.ac.uk/pub/bdr/memtests/gcc-UBSAN/lightgbm/tests/testthat.Rout

oh wait! I do see now some detailed logs from the checks on our 3.0.0 submission, that show this error. I didn't see them before because they only show up in the unit test logs, and the tests did not fail (which is strange).
https://www.stats.ox.ac.uk/pub/bdr/memtests/gcc-UBSAN/lightgbm/tests/testthat.Rout

just got this from cran. https://www.stats.ox.ac.uk/pub/bdr/memtests/gcc-UBSAN/lightgbm/tests/testthat.Rout

ha ok, yes I just found that! Under "lightgbm checks" link on https://cran.r-project.org/web/packages/lightgbm/index.html (https://cran.r-project.org/web/checks/check_results_lightgbm.html). I hadn't noticed it before.

There are also errors in the tests they've labeled "valgrind", with things I've never seen before
==1188372== 1,008 bytes in 1 blocks are definitely lost in loss record 210 of 2,756
==1188372==    at 0x483AE7D: operator new(unsigned long) (/builddir/build/BUILD/valgrind-3.16.1/coregrind/m_replacemalloc/vg_replace_malloc.c:342)
==1188372==    by 0x1DD8615E: allocate (/usr/include/c++/10/ext/new_allocator.h:115)
==1188372==    by 0x1DD8615E: allocate (/usr/include/c++/10/bits/alloc_traits.h:460)
==1188372==    by 0x1DD8615E: _M_allocate (/usr/include/c++/10/bits/stl_vector.h:346)
==1188372==    by 0x1DD8615E: _M_create_storage (/usr/include/c++/10/bits/stl_vector.h:361)
==1188372==    by 0x1DD8615E: _Vector_base (/usr/include/c++/10/bits/stl_vector.h:305)
==1188372==    by 0x1DD8615E: vector (/usr/include/c++/10/bits/stl_vector.h:524)
==1188372==    by 0x1DD8615E: LightGBM::GBDT::FeatureImportance(int, int) const (packages/tests-vg/lightgbm/src/boosting/gbdt_model_text.cpp:595)
==1188372==    by 0x1DD8B024: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (packages/tests-vg/lightgbm/src/boosting/gbdt_model_text.cpp:368)
==1188372==    by 0x1DEF965D: SaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:717)
==1188372==    by 0x1DEF965D: LGBM_BoosterSaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:2185)
==1188372==    by 0x1DF0B8D7: LGBM_BoosterSaveModelToString_R (packages/tests-vg/lightgbm/src/lightgbm_R.cpp:657)
==1188372==    by 0x49CD21: R_doDotCall (svn/R-devel/src/main/dotcode.c:624)
==1188372==    by 0x49D2A3: do_dotcall (svn/R-devel/src/main/dotcode.c:1281)
==1188372==    by 0x4E14B1: bcEval (svn/R-devel/src/main/eval.c:7078)
==1188372==    by 0x4EF6F7: Rf_eval (svn/R-devel/src/main/eval.c:727)
==1188372==    by 0x4F110D: R_execClosure (svn/R-devel/src/main/eval.c:1895)
==1188372==    by 0x4F1E03: Rf_applyClosure (svn/R-devel/src/main/eval.c:1821)
==1188372==    by 0x4DEC9D: bcEval (svn/R-devel/src/main/eval.c:7089)
==1188372==
==1188372== 1,166 (64 direct, 1,102 indirect) bytes in 1 blocks are definitely lost in loss record 222 of 2,756
==1188372==    at 0x483AE7D: operator new(unsigned long) (/builddir/build/BUILD/valgrind-3.16.1/coregrind/m_replacemalloc/vg_replace_malloc.c:342)
==1188372==    by 0x1DD8A9E4: allocate (/usr/include/c++/10/ext/new_allocator.h:115)
==1188372==    by 0x1DD8A9E4: allocate (/usr/include/c++/10/bits/alloc_traits.h:460)
==1188372==    by 0x1DD8A9E4: _M_allocate (/usr/include/c++/10/bits/stl_vector.h:346)
==1188372==    by 0x1DD8A9E4: _M_create_storage (/usr/include/c++/10/bits/stl_vector.h:361)
==1188372==    by 0x1DD8A9E4: _Vector_base (/usr/include/c++/10/bits/stl_vector.h:305)
==1188372==    by 0x1DD8A9E4: vector (/usr/include/c++/10/bits/stl_vector.h:511)
==1188372==    by 0x1DD8A9E4: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (packages/tests-vg/lightgbm/src/boosting/gbdt_model_text.cpp:348)
==1188372==    by 0x1DEF965D: SaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:717)
==1188372==    by 0x1DEF965D: LGBM_BoosterSaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:2185)
==1188372==    by 0x1DF0B8D7: LGBM_BoosterSaveModelToString_R (packages/tests-vg/lightgbm/src/lightgbm_R.cpp:657)
==1188372==    by 0x49CD21: R_doDotCall (svn/R-devel/src/main/dotcode.c:624)
==1188372==    by 0x49D2A3: do_dotcall (svn/R-devel/src/main/dotcode.c:1281)
==1188372==    by 0x4E14B1: bcEval (svn/R-devel/src/main/eval.c:7078)
==1188372==    by 0x4EF6F7: Rf_eval (svn/R-devel/src/main/eval.c:727)
==1188372==    by 0x4F110D: R_execClosure (svn/R-devel/src/main/eval.c:1895)
==1188372==    by 0x4F1E03: Rf_applyClosure (svn/R-devel/src/main/eval.c:1821)
==1188372==    by 0x4DEC9D: bcEval (svn/R-devel/src/main/eval.c:7089)
==1188372==    by 0x4EF6F7: Rf_eval (svn/R-devel/src/main/eval.c:727)
==1188372==
==1188372== 8,193 bytes in 1 blocks are definitely lost in loss record 1,325 of 2,756
==1188372==    at 0x483AE7D: operator new(unsigned long) (/builddir/build/BUILD/valgrind-3.16.1/coregrind/m_replacemalloc/vg_replace_malloc.c:342)
==1188372==    by 0x73852AF: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::reserve(unsigned long) (in /usr/lib64/libstdc++.so.6.0.28)
==1188372==    by 0x737ADEB: std::__cxx11::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::overflow(int) (in /usr/lib64/libstdc++.so.6.0.28)
==1188372==    by 0x7383639: std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) (in /usr/lib64/libstdc++.so.6.0.28)
==1188372==    by 0x7375963: std::basic_ostream<char, std::char_traits<char> >& std::__ostream_insert<char, std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*, long) (in /usr/lib64/libstdc++.so.6.0.28)
==1188372==    by 0x1DD8AFB8: operator<< <char, std::char_traits<char>, std::allocator<char> > (/usr/include/c++/10/bits/basic_string.h:6463)
==1188372==    by 0x1DD8AFB8: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (packages/tests-vg/lightgbm/src/boosting/gbdt_model_text.cpp:363)
==1188372==    by 0x1DEF965D: SaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:717)
==1188372==    by 0x1DEF965D: LGBM_BoosterSaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:2185)
==1188372==    by 0x1DF0B8D7: LGBM_BoosterSaveModelToString_R (packages/tests-vg/lightgbm/src/lightgbm_R.cpp:657)
==1188372==    by 0x49CD21: R_doDotCall (svn/R-devel/src/main/dotcode.c:624)
==1188372==    by 0x49D2A3: do_dotcall (svn/R-devel/src/main/dotcode.c:1281)
==1188372==    by 0x4E14B1: bcEval (svn/R-devel/src/main/eval.c:7078)
==1188372==    by 0x4EF6F7: Rf_eval (svn/R-devel/src/main/eval.c:727)
==1188372==
==1188372== 9,192 (624 direct, 8,568 indirect) bytes in 1 blocks are definitely lost in loss record 1,332 of 2,756
==1188372==    at 0x483AE7D: operator new(unsigned long) (/builddir/build/BUILD/valgrind-3.16.1/coregrind/m_replacemalloc/vg_replace_malloc.c:342)
==1188372==    by 0x1DDF5334: LightGBM::DatasetLoader::ConstructFromSampleData(double**, int**, int, int const*, unsigned long, int) (packages/tests-vg/lightgbm/src/io/dataset_loader.cpp:673)
==1188372==    by 0x1DF005A4: LGBM_DatasetCreateFromMats (packages/tests-vg/lightgbm/src/c_api.cpp:1072)
==1188372==    by 0x1DF008AD: LGBM_DatasetCreateFromMat (packages/tests-vg/lightgbm/src/c_api.cpp:1006)
==1188372==    by 0x1DF0A14F: LGBM_DatasetCreateFromMat_R (packages/tests-vg/lightgbm/src/lightgbm_R.cpp:110)
==1188372==    by 0x49CD21: R_doDotCall (svn/R-devel/src/main/dotcode.c:624)
==1188372==    by 0x49D2A3: do_dotcall (svn/R-devel/src/main/dotcode.c:1281)
==1188372==    by 0x4E14B1: bcEval (svn/R-devel/src/main/eval.c:7078)
==1188372==    by 0x4EF6F7: Rf_eval (svn/R-devel/src/main/eval.c:727)
==1188372==    by 0x4F110D: R_execClosure (svn/R-devel/src/main/eval.c:1895)
==1188372==    by 0x4F1E03: Rf_applyClosure (svn/R-devel/src/main/eval.c:1821)
==1188372==    by 0x4EF8C3: Rf_eval (svn/R-devel/src/main/eval.c:850)

https://www.stats.ox.ac.uk/pub/bdr/memtests/valgrind/lightgbm/tests/testthat.Rout

tonight I can at least try to figure out how to reproduce these, either in docker or on R Hub, so we have a way to test for them.

Thanks @jameslamb , it is necessary to reproduce them, so that we can ensure we fix them.
I will help on the cpp part.

it seems most errors are related to model saving/loading..

Thanks @jameslamb , it is necessary to reproduce them, so that we can ensure we fix them.
I will help on the cpp part.

I think I can do that tonight! Thank you so much for all the help with this

Ok I was able to reproduce the mis-alignment issues!
You can run the following from the root of the repo to get a shell in a docker container that replicates the environment for one of the failing checks
docker run \
    -v $(pwd):/opt/LightGBM \
    -it rhub/rocker-gcc-san \
    /bin/bash

then run the following in that shell
cd /opt/LightGBM
Rscript -e "install.packages(c('R6', 'data.table', 'jsonlite', 'testthat'), repos = 'http://cran.rstudio.com')"

Rdevel CMD install lightgbm_<IP_ADDRESS>.tar.gz

cd R-package/tests
Rscriptdevel testthat.R

References

https://cran.r-project.org/web/packages/rhub/vignettes/rhub.html
http://dirk.eddelbuettel.com/code/sanitizers.html (thanks @eddelbuettel !)

for more information, here are the first few lines of logs from R CMD INSTALL. You can see the result of the malloc check and the compiler flags used
checking location of R... /usr/lib/R
checking whether MM_PREFETCH works... yes
checking whether MM_MALLOC works... yes
configure: creating ./config.status
config.status: creating src/Makevars
** libs
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I./include -DMM_PREFETCH=1 -DMM_MALLOC=1 -DUSE_SOCKET -DLGB_R_BUILD    -fopenmp -pthread -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-OT058M/r-base-4.0.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c application/application.cpp -o application/application.o

and here is the output of g++ --version
g++ (Debian 10.1.0-6) 10.1.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Thanks @jameslamb , can we enable this in our CI?

BTW, if I want to fix the mis-alignment problem, which branch is better for start?

Thanks @jameslamb , can we enable this in our CI?

oooo I think so! Let me try that

BTW, if I want to fix the mis-alignment problem, which branch is better for start?

honestly, I think that at this point we should merge this PR, and then try to prepare a 3.1.0 release (of all of LightGBM) this week. Since this process is dragging on, it means that our users are not using all of the fixes on this branch.
What do you think?
If we do that, I could open a pull request off of master with the CI check, and you could push code changes to my pull request branch.

it seems we need to fix the cran problem before 5 OCT.
I will be in the vacation (very limited computer access) in 2 OCT to 8 OCT.
So I will fix the mis-alignment problem as possible (in today or tomorrow). But may cannot release the 3.1.0 in time.
So maybe I can fix it based on master branch, and use cherry-pick the fix commit to this PR?

So I will fix the mis-alignment problem as possible (in today or tomorrow). But may cannot release the 3.1.0 in time.
 I think you can safely branch off of master, then. Since we saw this issue before I opened this PR (on #629 a few months ago), I don't think it was introduced by my changes here.

it seems we need to fix the cran problem before 5 OCT.
I will be in the vacation (very limited computer access) in 2 OCT to 8 OCT.

I really hope we can fix it before October 5, but it's ok if we don't. CRAN has made this a very slow process by taking so long between responses.

@jameslamb can you try #3415 ? I think it should fix the dataset binary alignment problem.
BTW, for following:
==1188372== 1,008 bytes in 1 blocks are definitely lost in loss record 210 of 2,756
==1188372==    at 0x483AE7D: operator new(unsigned long) (/builddir/build/BUILD/valgrind-3.16.1/coregrind/m_replacemalloc/vg_replace_malloc.c:342)
==1188372==    by 0x1DD8615E: allocate (/usr/include/c++/10/ext/new_allocator.h:115)
==1188372==    by 0x1DD8615E: allocate (/usr/include/c++/10/bits/alloc_traits.h:460)
==1188372==    by 0x1DD8615E: _M_allocate (/usr/include/c++/10/bits/stl_vector.h:346)
==1188372==    by 0x1DD8615E: _M_create_storage (/usr/include/c++/10/bits/stl_vector.h:361)
==1188372==    by 0x1DD8615E: _Vector_base (/usr/include/c++/10/bits/stl_vector.h:305)
==1188372==    by 0x1DD8615E: vector (/usr/include/c++/10/bits/stl_vector.h:524)
==1188372==    by 0x1DD8615E: LightGBM::GBDT::FeatureImportance(int, int) const (packages/tests-vg/lightgbm/src/boosting/gbdt_model_text.cpp:595)
==1188372==    by 0x1DD8B024: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (packages/tests-vg/lightgbm/src/boosting/gbdt_model_text.cpp:368)
==1188372==    by 0x1DEF965D: SaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:717)
==1188372==    by 0x1DEF965D: LGBM_BoosterSaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:2185)
==1188372==    by 0x1DF0B8D7: LGBM_BoosterSaveModelToString_R (packages/tests-vg/lightgbm/src/lightgbm_R.cpp:657)
==1188372==    by 0x49CD21: R_doDotCall (svn/R-devel/src/main/dotcode.c:624)
==1188372==    by 0x49D2A3: do_dotcall (svn/R-devel/src/main/dotcode.c:1281)
==1188372==    by 0x4E14B1: bcEval (svn/R-devel/src/main/eval.c:7078)
==1188372==    by 0x4EF6F7: Rf_eval (svn/R-devel/src/main/eval.c:727)
==1188372==    by 0x4F110D: R_execClosure (svn/R-devel/src/main/eval.c:1895)
==1188372==    by 0x4F1E03: Rf_applyClosure (svn/R-devel/src/main/eval.c:1821)
==1188372==    by 0x4DEC9D: bcEval (svn/R-devel/src/main/eval.c:7089)
==1188372==
==1188372== 1,166 (64 direct, 1,102 indirect) bytes in 1 blocks are definitely lost in loss record 222 of 2,756
==1188372==    at 0x483AE7D: operator new(unsigned long) (/builddir/build/BUILD/valgrind-3.16.1/coregrind/m_replacemalloc/vg_replace_malloc.c:342)
==1188372==    by 0x1DD8A9E4: allocate (/usr/include/c++/10/ext/new_allocator.h:115)
==1188372==    by 0x1DD8A9E4: allocate (/usr/include/c++/10/bits/alloc_traits.h:460)
==1188372==    by 0x1DD8A9E4: _M_allocate (/usr/include/c++/10/bits/stl_vector.h:346)
==1188372==    by 0x1DD8A9E4: _M_create_storage (/usr/include/c++/10/bits/stl_vector.h:361)
==1188372==    by 0x1DD8A9E4: _Vector_base (/usr/include/c++/10/bits/stl_vector.h:305)
==1188372==    by 0x1DD8A9E4: vector (/usr/include/c++/10/bits/stl_vector.h:511)
==1188372==    by 0x1DD8A9E4: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (packages/tests-vg/lightgbm/src/boosting/gbdt_model_text.cpp:348)
==1188372==    by 0x1DEF965D: SaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:717)
==1188372==    by 0x1DEF965D: LGBM_BoosterSaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:2185)
==1188372==    by 0x1DF0B8D7: LGBM_BoosterSaveModelToString_R (packages/tests-vg/lightgbm/src/lightgbm_R.cpp:657)
==1188372==    by 0x49CD21: R_doDotCall (svn/R-devel/src/main/dotcode.c:624)
==1188372==    by 0x49D2A3: do_dotcall (svn/R-devel/src/main/dotcode.c:1281)
==1188372==    by 0x4E14B1: bcEval (svn/R-devel/src/main/eval.c:7078)
==1188372==    by 0x4EF6F7: Rf_eval (svn/R-devel/src/main/eval.c:727)
==1188372==    by 0x4F110D: R_execClosure (svn/R-devel/src/main/eval.c:1895)
==1188372==    by 0x4F1E03: Rf_applyClosure (svn/R-devel/src/main/eval.c:1821)
==1188372==    by 0x4DEC9D: bcEval (svn/R-devel/src/main/eval.c:7089)
==1188372==    by 0x4EF6F7: Rf_eval (svn/R-devel/src/main/eval.c:727)
==1188372==
==1188372== 8,193 bytes in 1 blocks are definitely lost in loss record 1,325 of 2,756
==1188372==    at 0x483AE7D: operator new(unsigned long) (/builddir/build/BUILD/valgrind-3.16.1/coregrind/m_replacemalloc/vg_replace_malloc.c:342)
==1188372==    by 0x73852AF: std::__cxx11::basic_string<char, std::char_traits, std::allocator >::reserve(unsigned long) (in /usr/lib64/libstdc++.so.6.0.28)
==1188372==    by 0x737ADEB: std::__cxx11::basic_stringbuf<char, std::char_traits, std::allocator >::overflow(int) (in /usr/lib64/libstdc++.so.6.0.28)
==1188372==    by 0x7383639: std::basic_streambuf<char, std::char_traits >::xsputn(char const*, long) (in /usr/lib64/libstdc++.so.6.0.28)
==1188372==    by 0x7375963: std::basic_ostream<char, std::char_traits >& std::__ostream_insert<char, std::char_traits >(std::basic_ostream<char, std::char_traits >&, char const*, long) (in /usr/lib64/libstdc++.so.6.0.28)
==1188372==    by 0x1DD8AFB8: operator<< <char, std::char_traits, std::allocator > (/usr/include/c++/10/bits/basic_string.h:6463)
==1188372==    by 0x1DD8AFB8: LightGBM::GBDT::SaveModelToString[abi:cxx11](int, int, int) const (packages/tests-vg/lightgbm/src/boosting/gbdt_model_text.cpp:363)
==1188372==    by 0x1DEF965D: SaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:717)
==1188372==    by 0x1DEF965D: LGBM_BoosterSaveModelToString (packages/tests-vg/lightgbm/src/c_api.cpp:2185)
==1188372==    by 0x1DF0B8D7: LGBM_BoosterSaveModelToString_R (packages/tests-vg/lightgbm/src/lightgbm_R.cpp:657)
==1188372==    by 0x49CD21: R_doDotCall (svn/R-devel/src/main/dotcode.c:624)
==1188372==    by 0x49D2A3: do_dotcall (svn/R-devel/src/main/dotcode.c:1281)
==1188372==    by 0x4E14B1: bcEval (svn/R-devel/src/main/eval.c:7078)
==1188372==    by 0x4EF6F7: Rf_eval (svn/R-devel/src/main/eval.c:727)
==1188372==
==1188372== 9,192 (624 direct, 8,568 indirect) bytes in 1 blocks are definitely lost in loss record 1,332 of 2,756
==1188372==    at 0x483AE7D: operator new(unsigned long) (/builddir/build/BUILD/valgrind-3.16.1/coregrind/m_replacemalloc/vg_replace_malloc.c:342)
==1188372==    by 0x1DDF5334: LightGBM::DatasetLoader::ConstructFromSampleData(double**, int**, int, int const*, unsigned long, int) (packages/tests-vg/lightgbm/src/io/dataset_loader.cpp:673)
==1188372==    by 0x1DF005A4: LGBM_DatasetCreateFromMats (packages/tests-vg/lightgbm/src/c_api.cpp:1072)
==1188372==    by 0x1DF008AD: LGBM_DatasetCreateFromMat (packages/tests-vg/lightgbm/src/c_api.cpp:1006)
==1188372==    by 0x1DF0A14F: LGBM_DatasetCreateFromMat_R (packages/tests-vg/lightgbm/src/lightgbm_R.cpp:110)
==1188372==    by 0x49CD21: R_doDotCall (svn/R-devel/src/main/dotcode.c:624)
==1188372==    by 0x49D2A3: do_dotcall (svn/R-devel/src/main/dotcode.c:1281)
==1188372==    by 0x4E14B1: bcEval (svn/R-devel/src/main/eval.c:7078)
==1188372==    by 0x4EF6F7: Rf_eval (svn/R-devel/src/main/eval.c:727)
==1188372==    by 0x4F110D: R_execClosure (svn/R-devel/src/main/eval.c:1895)
==1188372==    by 0x4F1E03: Rf_applyClosure (svn/R-devel/src/main/eval.c:1821)
==1188372==    by 0x4EF8C3: Rf_eval (svn/R-devel/src/main/eval.c:850)
are they the errors need to be fixed too?

can you try #3415 ? I think it should fix the dataset binary alignment problem.

wow that was fast!  Yep, I can try right now.
I have a PR open into my fork to test a CI job so we can catch this before CRAN in the future: https://github.com/jameslamb/LightGBM/pull/35

are they the errors need to be fixed too?

I'm not sure, since Uwe didn't say anything about it (https://github.com/microsoft/LightGBM/pull/3338#issuecomment-700003126). But maybe it is one of those things that CRAN just assumes that we know to fix.
It's also possible that the mis-alignment fix could fix those too. I haven't been able to reproduce those valgrind tests yet (https://www.stats.ox.ac.uk/pub/bdr/memtests/valgrind/lightgbm/tests/testthat.Rout)

@jameslamb I just check these valgrind errors, which seems to indicate the memory leakage.
However, I check the code, all these memories are allocated by std::vector or std::unique_ptr. Compliers should ensure to free the memories.

@jameslamb I just check these valgrind errors, which seems to indicate the memory leakage.
However, I check the code, all these memories are allocated by std::vector or std::unique_ptr. Compliers should ensure to free the memories.

hmm ok, maybe they are false positives then. Hopefully CRAN will not care about them.

@jameslamb
It seems following error is due to the missing of support int64 in R.
==1188372== Invalid write of size 8
==1188372==    at 0x1DF0AEB4: LGBM_BoosterGetNumPredict_R (packages/tests-vg/lightgbm/src/lightgbm_R.cpp:509)
==1188372==    by 0x49CDA3: R_doDotCall (svn/R-devel/src/main/dotcode.c:607)
==1188372==    by 0x49D2A3: do_dotcall (svn/R-devel/src/main/dotcode.c:1281)
==1188372==    by 0x4E14B1: bcEval (svn/R-devel/src/main/eval.c:7078)
==1188372==    by 0x4EF6F7: Rf_eval (svn/R-devel/src/main/eval.c:727)
==1188372==    by 0x4F110D: R_execClosure (svn/R-devel/src/main/eval.c:1895)
==1188372==    by 0x4F1E03: Rf_applyClosure (svn/R-devel/src/main/eval.c:1821)
==1188372==    by 0x4EF8C3: Rf_eval (svn/R-devel/src/main/eval.c:850)
==1188372==    by 0x4F3B99: do_set (svn/R-devel/src/main/eval.c:2967)
==1188372==    by 0x4EFB44: Rf_eval (svn/R-devel/src/main/eval.c:802)
==1188372==    by 0x4F27B7: do_begin (svn/R-devel/src/main/eval.c:2515)
==1188372==    by 0x4EFB44: Rf_eval (svn/R-devel/src/main/eval.c:802)
==1188372==  Address 0x1e070cb0 is 4,368 bytes inside a block of size 7,960 alloc'd
==1188372==    at 0x483A809: malloc (/builddir/build/BUILD/valgrind-3.16.1/coregrind/m_replacemalloc/vg_replace_malloc.c:307)
==1188372==    by 0x52A8C0: GetNewPage (svn/R-devel/src/main/memory.c:946)
==1188372==    by 0x52C4FB: Rf_allocVector3 (svn/R-devel/src/main/memory.c:2784)
==1188372==    by 0x587CB2: Rf_allocVector (svn/R-devel/src/include/Rinlinedfuns.h:593)
==1188372==    by 0x587CB2: ReadItem (svn/R-devel/src/main/serialize.c:1948)
==1188372==    by 0x586F84: ReadItem (svn/R-devel/src/main/serialize.c:1872)
==1188372==    by 0x5880B2: ReadItem (svn/R-devel/src/main/serialize.c:2018)
==1188372==    by 0x588BD1: ReadBCConsts (svn/R-devel/src/main/serialize.c:2102)
==1188372==    by 0x588BD1: ReadBC1 (svn/R-devel/src/main/serialize.c:2119)
==1188372==    by 0x588CB5: ReadBCConsts (svn/R-devel/src/main/serialize.c:2088)
==1188372==    by 0x588CB5: ReadBC1 (svn/R-devel/src/main/serialize.c:2119)
==1188372==    by 0x587F69: ReadBC (svn/R-devel/src/main/serialize.c:2130)
==1188372==    by 0x587F69: ReadItem (svn/R-devel/src/main/serialize.c:1967)
==1188372==    by 0x586FA1: ReadItem (svn/R-devel/src/main/serialize.c:1874)
==1188372==    by 0x587712: ReadItem (svn/R-devel/src/main/serialize.c:1962)
==1188372==    by 0x587712: ReadItem (svn/R-devel/src/main/serialize.c:1962)
https://github.com/microsoft/LightGBM/blob/73d65c79898702cfc085712499e76b3acc414db6/R-package/src/lightgbm_R.cpp#L509
There are many int64 in this file too. any workarounds ?

for the remaining memory leaks, it seems to use the OpenMP with a new allocated vector in the omp loop will throw the error. like
https://github.com/microsoft/LightGBM/blob/73d65c79898702cfc085712499e76b3acc414db6/R-package/src/lightgbm_R.cpp#L202-L228

@jameslamb
I try some fixes in branch https://github.com/microsoft/LightGBM/tree/fix-memory-leakage , could you help to test it for the valgrind ?

@jameslamb
I try some fixes in branch https://github.com/microsoft/LightGBM/tree/fix-memory-leakage , could you help to test it for the valgrind ?

yep I can try to figure that out!

There are many int64 in this file too. any workarounds ?

There is an R package that provides integer64 support in R, but it is GPL-2 licensed, and until recently was not very actively maintained 😢
https://cran.r-project.org/web/packages/bit64/index.html
So if we are not comfortable taking on that dependency (I think we should not add it), maybe we have to cast to int32? Sorry, this is a little bit outside of my range of knowledge.

integer64 support in R

That is bit64, the author is a friend (and yes, a bit of a funny chap) and "we all" use it for 64-bit support:   data.table, nanotime, reticulate.  And after a long fight with CRAN it just got updated.

integer64 support in R

That is bit64, the author is a friend (and yes, a bit of a funny chap) and "we all" use it for 64-bit support: data.table, nanotime, ~reticulate~, dplyr, arrow, various DB packages, .... And after a long fight with CRAN it just got updated.

oh great, thank you! I saw there were no releases in 2018 or 2019, and then 3 recent ones (July and August of this year)

@jameslamb I see, let us convert it to int32 currently, until better solutions are out.

I just cherry-picked https://github.com/microsoft/LightGBM/commit/f30dbe8717177da2b1745ac3b2a5d5fd7475de55 onto this branch (changes from #3415)
I'm  trying now to replicate the valgrind checks

I have tried to replicate the valgrind tests, but so far can't reproduce the errors we see on CRAN. Right now, they're giving me this
==8179==
==8179== HEAP SUMMARY:
==8179==     in use at exit: 181,824,575 bytes in 37,649 blocks
==8179==   total heap usage: 76,635 allocs, 38,986 frees, 290,853,756 bytes allocated
==8179==
==8179== LEAK SUMMARY:
==8179==    definitely lost: 0 bytes in 0 blocks
==8179==    indirectly lost: 0 bytes in 0 blocks
==8179==      possibly lost: 0 bytes in 0 blocks
==8179==    still reachable: 181,824,575 bytes in 37,649 blocks
==8179==                       of which reachable via heuristic:
==8179==                         newarray           : 4,264 bytes in 1 blocks
==8179==         suppressed: 0 bytes in 0 blocks
==8179== Reachable blocks (those to which a pointer was found) are not shown.
==8179== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==8179==
==8179== For lists of detected and suppressed errors, rerun with: -s
==8179== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

I've been trying something like this.
cd /opt/LightGBM

sh build-cran-package.sh

Rscript -e "install.packages(c('R6', 'data.table', 'jsonlite', 'testthat'), repos = 'http://cran.rstudio.com')"

echo "--leak-check=full" > ~/.valgrindrc
echo "--track-origins=yes" >> ~/.valgrindrc
echo "--dsymutil=yes" >> ~/.valgrindrc

R CMD check \
    --use-valgrind \
    --install-args="--dsym" \
        lightgbm_<IP_ADDRESS>.tar.gz

I might need to try building R from source, based on "Writing R Extensions"

On platforms where valgrind is installed you can build a version of R with extra instrumentation to help valgrind detect errors in the use of memory allocated from the R heap. The configure option is --with-valgrind-instrumentation=level, where level is 0, 1 or 2. Level 0 is the default and does not add anything. Level 1 will detect some uses115 of uninitialised memory and has little impact on speed (compared to level 0). Level 2 will detect many other memory-use bugs116 but make R much slower when running under valgrind. Using this in conjunction with gctorture can be even more effective (and even slower).

References

https://cran.r-project.org/doc/manuals/r-release/R-exts.html#Using-valgrind
https://francoismichonneau.net/2014/03/how-to-do-ubsan-tests-r-package/
https://github.com/rocker-org/rocker/issues/237

I had to fix  the DESCRIPTION conflicts from #3409 because apparently GitHub Actions won't run if the branch has conflicts with the target

@jameslamb which branch you used for test?
maybe due to the errors are false positive due to compiler/platform.

@jameslamb which branch you used for test?
maybe the errors are false positives due to compiler/platform.

I used the current state of this branch. I'm testing a slightly different command, but then I think we should try submitting to CRAN to again. Since we know that can take a few days, and since I'm pretty confident we've fixed the misalignment problems, at least.

ugh I can't figure out how to resolve this conflict without merging all of master into this branch. Cherry-picking the change from #3409 on to this branch did not work.

I shouldn't have made #3409 haha. I never thought that the merge conflicts would prevent CI from running, lesson learned 😫

Ok I wanted CI to pass so we wouldn't be wasting our time with CRAN, but this merge conflict is preventing that and I don't know how to fix it without merging master into this branch.
@guolinke I propose that we submit this new package (which has the fixes from #3415) and see if it makes CRAN happy. I propose submitting that as <IP_ADDRESS>. What do you think?
lightgbm_<IP_ADDRESS>.tar.gz

@jameslamb oh, sorry, it seems I merge the master branch to this again...

@jameslamb oh, sorry, it seems I merge the master branch to this again...

ha see! It's difficult! I did that accidentally before, trying to fix merge conflicts.

okay, I will submit <IP_ADDRESS> first. Meanwhile, we can try to fix valgrind tests.

ok thank you! I will work on trying to reproduce the valgrind results

Dear maintainer,
package lightgbm_<IP_ADDRESS>.tar.gz has been auto-processed.
The auto-check found additional issues for the last version released on CRAN:
gcc-UBSAN https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.stats.ox.ac.uk%2Fpub%2Fbdr%2Fmemtests%2Fgcc-UBSAN%2Flightgbm&data=02|01|guolin.ke%40microsoft.com|21ef1228b76c4d89973008d864fe461c|72f988bf86f141af91ab2d7cd011db47|1|0|637370390995846409&sdata=6KQbOC5p0GNAWjCWrN0t8B2uXfC0p7NIVw8Sjrudyak%3D&reserved=0
valgrind https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.stats.ox.ac.uk%2Fpub%2Fbdr%2Fmemtests%2Fvalgrind%2Flightgbm&data=02|01|guolin.ke%40microsoft.com|21ef1228b76c4d89973008d864fe461c|72f988bf86f141af91ab2d7cd011db47|1|0|637370390995846409&sdata=O0%2BnbI40KURl5OEmwUgl3jNIgv%2BHHXd5vjiHMg5q%2F%2FA%3D&reserved=0
CRAN incoming checks do not test for these additional issues and you will need an appropriately instrumented build of R to reproduce these.
Hence please reply-all and explain: Have these been fixed?
Log dir: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_<IP_ADDRESS>_20200930_060414%2F&data=02|01|guolin.ke%40microsoft.com|21ef1228b76c4d89973008d864fe461c|72f988bf86f141af91ab2d7cd011db47|1|0|637370390995846409&sdata=ctigGHTKKRhEYZqZnUA9gpQggwYCwXWOTuo9%2FYZhcbA%3D&reserved=0
The files will be removed after roughly 7 days.
Installation time in seconds: 644
Check time in seconds: 237
R Under development (unstable) (2020-09-28 r79268)
Pretests results:
Windows: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_<IP_ADDRESS>_20200930_060414%2FWindows%2F00check.log&data=02|01|guolin.ke%40microsoft.com|21ef1228b76c4d89973008d864fe461c|72f988bf86f141af91ab2d7cd011db47|1|0|637370390995846409&sdata=CQxEYHhtCssHa%2FQPCyFKPAsa09VNVU3Jseqtxd9rwCk%3D&reserved=0
Status: OK
Debian: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_<IP_ADDRESS>_20200930_060414%2FDebian%2F00check.log&data=02|01|guolin.ke%40microsoft.com|21ef1228b76c4d89973008d864fe461c|72f988bf86f141af91ab2d7cd011db47|1|0|637370390995846409&sdata=izymsb9IGzZBY10jcgktF6EzdlfmK3fFjgYspdIsDeE%3D&reserved=0
Status: OK
Last released version's CRAN status: ERROR: 1, NOTE: 6, OK: 5
See: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fweb%2Fchecks%2Fcheck_results_lightgbm.html&data=02|01|guolin.ke%40microsoft.com|21ef1228b76c4d89973008d864fe461c|72f988bf86f141af91ab2d7cd011db47|1|0|637370390995846409&sdata=CtTc78XyPrNKI2wy%2BO0lW17znjd41sZLPD8PpYd1dfA%3D&reserved=0
Last released version's additional issues:
gcc-UBSAN https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.stats.ox.ac.uk%2Fpub%2Fbdr%2Fmemtests%2Fgcc-UBSAN%2Flightgbm&data=02|01|guolin.ke%40microsoft.com|21ef1228b76c4d89973008d864fe461c|72f988bf86f141af91ab2d7cd011db47|1|0|637370390995846409&sdata=6KQbOC5p0GNAWjCWrN0t8B2uXfC0p7NIVw8Sjrudyak%3D&reserved=0
valgrind https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.stats.ox.ac.uk%2Fpub%2Fbdr%2Fmemtests%2Fvalgrind%2Flightgbm&data=02|01|guolin.ke%40microsoft.com|21ef1228b76c4d89973008d864fe461c|72f988bf86f141af91ab2d7cd011db47|1|0|637370390995856403&sdata=Kj1aZIBlrrhKacc0mxdN%2FUOEtFuumRpVh%2Bbz8RFHLcY%3D&reserved=0
CRAN Web: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fpackage%3Dlightgbm&data=02|01|guolin.ke%40microsoft.com|21ef1228b76c4d89973008d864fe461c|72f988bf86f141af91ab2d7cd011db47|1|0|637370390995856403&sdata=Jf7qfxGvYgdmSIdUMqZTH0NCpiBTTrbsrbfePlajDtA%3D&reserved=0
No strong reverse dependencies to be checked.
Best regards,
CRAN teams' auto-check service

it seems the solaris fix is not included.

@guolinke those logs are referring to previous submission. For example, the Solaris log they linked to is from 9-25-2020

So when they say this:

CRAN incoming checks do not test for these additional issues and you will need an appropriately instrumented build of R to reproduce these.
Hence please reply-all and explain: Have these been fixed?

I guess we are supposed to also reply-all to the email and say that we think it's fixed. Can you reply-all with something like this?

We believe that lightgbm <IP_ADDRESS> has addressed the issues on Solaris and with UBSAN. We have been unable to reproduce the valgrind errors using 'R CMD CHECK --use-valgrind', so believe these have been fixed as well.

On 21/09/2020 13:29, Prof Brian Ripley wrote:

Dear maintainer,
Please see the problems shown on
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fweb%2Fchecks%2Fcheck_results_lightgbm.html&data=02|01|guolin.ke%40microsoft.com|2bd4c472015e4ca5517208d86629a85a|72f988bf86f141af91ab2d7cd011db47|1|0|637371676863723804&sdata=hYFG%2B8YGYmDuPm3hIJQ0RfUaUVWg5X7788ODk7j0%2B8M%3D&reserved=0.
Please correct before 2020-10-05 to safely retain your package on CRAN.
The CRAN Team

There is still a valgrind error. This did not happen when tested on
submission, but the tests did run until timeout at 4 hours. When you
write illegally, corruption is common.
Illegal writes are serious errors.
--
Brian D. Ripley<EMAIL_ADDRESS>Emeritus Professor of Applied Statistics, University of Oxford

The new
https://www.stats.ox.ac.uk/pub/bdr/memtests/valgrind/lightgbm/tests/testthat.Rout
Maybe the test is too costly, and the timeout forcely the program exit, and then memory leakage ?

hey overall, this is very good news! The UBSAN and Solaris errors are gone (https://cran.r-project.org/web/checks/check_results_lightgbm.html)

@guolinke I'm still trying to reproduce the valgrind failures, but it will take me a while I think.
Since you will be on vacation for the next week, I think we'll have to let CRAN kick us off for now. Can you please reply to their email and say something like this?

We are struggling to reproduce the errors show in CRAN's valgrind tests. We will probably not be able to fix them before October 5th. We will submit a new version that fixes these issues as soon as possible, but we understand that {lighgbm} may be removed until we do so.

If the errors can be reproduced, I think maybe I can take some time fix it. And my attempted fixes above, can you also test it?  The difference between it and master(this pr) will be very useful.

Sure! As soon as I figure out how to reproduce the valgrind tests, I'll try the fix-memory-leakage branch you mentioned in https://github.com/microsoft/LightGBM/pull/3338#issuecomment-700656706.
I'll keep posting updates on this PR, but please don't let it interrupt your vacation! CRAN will always be there 😀

ok I made some progress! Thanks to this excellent blog post from @richfitz , I'm able to reproduce the valgrind errors!
docker run \
    -v $(pwd):/opt/LightGBM \
    -it \
        wch1/r-debug

RDscriptvalgrind -e "install.packages(c('R6', 'data.table', 'jsonlite', 'testthat'), repos = 'http://cran.rstudio.com')"

cd /opt/LightGBM
sh build-cran-package.sh

RDvalgrind CMD INSTALL \
    --preclean \
    --install-tests \
        lightgbm_*.tar.gz

cd R-package/tests

RDvalgrind \
    --no-readline \
    --vanilla \
    -d valgrind \
        -f testthat.R \
2>&1 \
| tee out.log \
| cat

These tests are not identical to those from https://cran.r-project.org/web/checks/check_results_lightgbm.html, but I see some of the same things. Here are all the errors I see:
[LightGBM] [Warning] Using self-defined objective function
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.588622 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Warning] Using self-defined objective function
==1989== Invalid write of size 8
==1989==    at 0x1538022F: LGBM_BoosterGetNumPredict_R (lightgbm_R.cpp:509)
==1989==    by 0x4940EED: R_doDotCall (dotcode.c:607)
==1989==    by 0x494C019: do_dotcall (dotcode.c:1281)
==1989==    by 0x499E133: bcEval (eval.c:7078)
==1989==    by 0x4989CEC: Rf_eval (eval.c:727)
==1989==    by 0x498CA79: R_execClosure (eval.c:1895)
==1989==    by 0x498C729: Rf_applyClosure (eval.c:1821)
==1989==    by 0x498A4D7: Rf_eval (eval.c:850)
==1989==    by 0x4990748: do_set (eval.c:2967)
==1989==    by 0x498A13D: Rf_eval (eval.c:802)
==1989==    by 0x498F1C3: do_begin (eval.c:2515)
==1989==    by 0x498A13D: Rf_eval (eval.c:802)
==1989==  Address 0xf4fbb50 is 7,952 bytes inside a block of size 7,960 alloc'd
==1989==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1989==    by 0x49E0321: GetNewPage (memory.c:946)
==1989==    by 0x49EF4E3: Rf_allocVector3 (memory.c:2784)
==1989==    by 0x49D35A0: Rf_allocVector (Rinlinedfuns.h:593)
==1989==    by 0x4963315: duplicate1 (duplicate.c:352)
==1989==    by 0x4961B9D: duplicate_child (duplicate.c:206)
==1989==    by 0x4962592: duplicate1 (duplicate.c:338)
==1989==    by 0x4961B9D: duplicate_child (duplicate.c:206)
==1989==    by 0x4961E18: duplicate_list (duplicate.c:268)
==1989==    by 0x4962244: duplicate1 (duplicate.c:312)
==1989==    by 0x49620E4: duplicate1 (duplicate.c:305)
==1989==    by 0x4961ACD: Rf_duplicate (duplicate.c:139)
==1989==

==1989== Conditional jump or move depends on uninitialised value(s)
==1989==    at 0x49CD81E: gregexpr_Regexc (grep.c:2439)
==1989==    by 0x49D060C: do_regexpr (grep.c:3100)
==1989==    by 0x499E679: bcEval (eval.c:7121)
==1989==    by 0x4989CEC: Rf_eval (eval.c:727)
==1989==    by 0x498CA79: R_execClosure (eval.c:1895)
==1989==    by 0x498C729: Rf_applyClosure (eval.c:1821)
==1989==    by 0x499E2BC: bcEval (eval.c:7089)
==1989==    by 0x4989CEC: Rf_eval (eval.c:727)
==1989==    by 0x4989835: forcePromise (eval.c:555)
==1989==    by 0x4994A3C: FORCE_PROMISE (eval.c:5142)
==1989==    by 0x4994BF7: getvar (eval.c:5183)
==1989==    by 0x499B801: bcEval (eval.c:6873)
==1989==

I do see that the LEAK SUMMARY is identical to what's on CRAN!
in my tests
==1989== LEAK SUMMARY:
==1989==    definitely lost: 1,058,501 bytes in 7 blocks
==1989==    indirectly lost: 9,670 bytes in 83 blocks
==1989==      possibly lost: 336 bytes in 1 blocks
==1989==    still reachable: 273,381,429 bytes in 46,860 blocks
==1989==                       of which reachable via heuristic:
==1989==                         newarray           : 4,264 bytes in 1 blocks
==1989==         suppressed: 0 bytes in 0 blocks
==1989== Rerun with --leak-check=full to see details of leaked memory

on CRAN (link)
==417180== LEAK SUMMARY:
==417180==    definitely lost: 1,058,501 bytes in 7 blocks
==417180==    indirectly lost: 9,670 bytes in 83 blocks
==417180==      possibly lost: 368 bytes in 1 blocks
==417180==    still reachable: 235,062,504 bytes in 43,509 blocks
==417180==         suppressed: 1,352 bytes in 1 blocks
==417180== Reachable blocks (those to which a pointer was found) are not

full logs for `r/description-fixes` branch (click me)
==1989== Memcheck, a memory error detector
==1989== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1989== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==1989== Command: /usr/local/RDvalgrind/lib/R/bin/exec/R --no-readline --vanilla -f testthat.R
==1989==

R Under development (unstable) (2020-09-30 r79283) -- "Unsuffered Consequences"
Copyright (C) 2020 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(testthat)
> library(lightgbm)
Loading required package: R6
>
> test_check(
+     package = "lightgbm"
+     , stop_on_failure = TRUE
+     , stop_on_warning = FALSE
+ )
[LightGBM] [Info] Number of positive: 3140, number of negative: 3373
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.770279 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.482113 -> initscore=-0.071580
[LightGBM] [Info] Start training from score -0.071580
[1] "[1]:  train's binary_error:0.0222632"
[1] "[2]:  train's binary_error:0.0222632"
[1] "[3]:  train's binary_error:0.0222632"
[1] "[4]:  train's binary_error:0.0109013"
[1] "[5]:  train's binary_error:0.0141256"
[1] "[6]:  train's binary_error:0.0141256"
[1] "[7]:  train's binary_error:0.0141256"
[1] "[8]:  train's binary_error:0.0141256"
[1] "[9]:  train's binary_error:0.00598802"
[1] "[10]:  train's binary_error:0.00598802"
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.041739 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 98
[LightGBM] [Info] Number of data points in the train set: 150, number of used features: 4
[LightGBM] [Info] Start training from score -1.098612
[LightGBM] [Info] Start training from score -1.098612
[LightGBM] [Info] Start training from score -1.098612
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  train's multi_error:0.0466667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  train's multi_error:0.0466667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  train's multi_error:0.0466667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  train's multi_error:0.0466667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  train's multi_error:0.0466667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  train's multi_error:0.0466667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[7]:  train's multi_error:0.0466667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[8]:  train's multi_error:0.0466667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[9]:  train's multi_error:0.0466667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[10]:  train's multi_error:0.0466667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[11]:  train's multi_error:0.0333333"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[12]:  train's multi_error:0.0266667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[13]:  train's multi_error:0.0266667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[14]:  train's multi_error:0.0266667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[15]:  train's multi_error:0.0266667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[16]:  train's multi_error:0.0333333"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[17]:  train's multi_error:0.0266667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[18]:  train's multi_error:0.0333333"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[19]:  train's multi_error:0.0333333"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[20]:  train's multi_error:0.0333333"
[LightGBM] [Info] Number of positive: 3140, number of negative: 3373
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.612952 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.482113 -> initscore=-0.071580
[LightGBM] [Info] Start training from score -0.071580
[1] "[1]:  train's binary_error:0.0304007  train's auc:0.972508  train's binary_logloss:0.198597"
[1] "[2]:  train's binary_error:0.0222632  train's auc:0.995075  train's binary_logloss:0.111535"
[1] "[3]:  train's binary_error:0.00598802  train's auc:0.997845  train's binary_logloss:0.0480659"
[1] "[4]:  train's binary_error:0.00122831  train's auc:0.998433  train's binary_logloss:0.0279151"
[1] "[5]:  train's binary_error:0.00122831  train's auc:0.999354  train's binary_logloss:0.0190479"
[1] "[6]:  train's binary_error:0.00537387  train's auc:0.98965  train's binary_logloss:0.16706"
[1] "[7]:  train's binary_error:0  train's auc:1  train's binary_logloss:0.0128449"
[1] "[8]:  train's binary_error:0  train's auc:1  train's binary_logloss:0.00774702"
[1] "[9]:  train's binary_error:0  train's auc:1  train's binary_logloss:0.00472108"
[1] "[10]:  train's binary_error:0  train's auc:1  train's binary_logloss:0.00208929"
[LightGBM] [Info] Number of positive: 3140, number of negative: 3373
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.587755 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.482113 -> initscore=-0.071580
[LightGBM] [Info] Start training from score -0.071580
[1] "[1]:  train's binary_error:0.0222632"
[1] "[2]:  train's binary_error:0.0222632"
[1] "[3]:  train's binary_error:0.0222632"
[1] "[4]:  train's binary_error:0.0109013"
[1] "[5]:  train's binary_error:0.0141256"
[1] "[6]:  train's binary_error:0.0141256"
[1] "[7]:  train's binary_error:0.0141256"
[1] "[8]:  train's binary_error:0.0141256"
[1] "[9]:  train's binary_error:0.00598802"
[1] "[10]:  train's binary_error:0.00598802"
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.649783 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Info] Start training from score 0.482113
[1] "[1]:  train's l2:0.206337"
[1] "[2]:  train's l2:0.171229"
[1] "[3]:  train's l2:0.140871"
[1] "[4]:  train's l2:0.116282"
[1] "[5]:  train's l2:0.096364"
[1] "[6]:  train's l2:0.0802308"
[1] "[7]:  train's l2:0.0675595"
[1] "[8]:  train's l2:0.0567154"
[1] "[9]:  train's l2:0.0482086"
[1] "[10]:  train's l2:0.0402694"
[LightGBM] [Info] Number of positive: 3140, number of negative: 3373
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.579872 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.482113 -> initscore=-0.071580
[LightGBM] [Info] Start training from score -0.071580
[1] "[1]:  train's binary_error:0.0222632  train's auc:0.981784  valid1's binary_error:0.0222632  valid1's auc:0.981784  valid2's binary_error:0.0222632  valid2's auc:0.981784"
[1] "[2]:  train's binary_error:0.0222632  train's auc:0.981784  valid1's binary_error:0.0222632  valid1's auc:0.981784  valid2's binary_error:0.0222632  valid2's auc:0.981784"
[1] "[3]:  train's binary_error:0.0222632  train's auc:0.992951  valid1's binary_error:0.0222632  valid1's auc:0.992951  valid2's binary_error:0.0222632  valid2's auc:0.992951"
[1] "[4]:  train's binary_error:0.0109013  train's auc:0.992951  valid1's binary_error:0.0109013  valid1's auc:0.992951  valid2's binary_error:0.0109013  valid2's auc:0.992951"
[1] "[5]:  train's binary_error:0.0141256  train's auc:0.994714  valid1's binary_error:0.0141256  valid1's auc:0.994714  valid2's binary_error:0.0141256  valid2's auc:0.994714"
[1] "[6]:  train's binary_error:0.0141256  train's auc:0.994714  valid1's binary_error:0.0141256  valid1's auc:0.994714  valid2's binary_error:0.0141256  valid2's auc:0.994714"
[1] "[7]:  train's binary_error:0.0141256  train's auc:0.994714  valid1's binary_error:0.0141256  valid1's auc:0.994714  valid2's binary_error:0.0141256  valid2's auc:0.994714"
[1] "[8]:  train's binary_error:0.0141256  train's auc:0.994714  valid1's binary_error:0.0141256  valid1's auc:0.994714  valid2's binary_error:0.0141256  valid2's auc:0.994714"
[1] "[9]:  train's binary_error:0.00598802  train's auc:0.993175  valid1's binary_error:0.00598802  valid1's auc:0.993175  valid2's binary_error:0.00598802  valid2's auc:0.993175"
[1] "[10]:  train's binary_error:0.00598802  train's auc:0.998242  valid1's binary_error:0.00598802  valid1's auc:0.998242  valid2's binary_error:0.00598802  valid2's auc:0.998242"
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.616474 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 232
[LightGBM] [Info] Number of data points in the train set: 5211, number of used features: 116
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.606981 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 232
[LightGBM] [Info] Number of data points in the train set: 5211, number of used features: 116
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.545706 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 232
[LightGBM] [Info] Number of data points in the train set: 5210, number of used features: 116
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.603822 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 232
[LightGBM] [Info] Number of data points in the train set: 5210, number of used features: 116
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.530754 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 232
[LightGBM] [Info] Number of data points in the train set: 5210, number of used features: 116
[LightGBM] [Info] Start training from score 0.483976
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Info] Start training from score 0.480906
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Info] Start training from score 0.481574
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Info] Start training from score 0.482342
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Info] Start training from score 0.481766
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid's l2:0.000306984+0.000613968  valid's l1:0.000306994+0.00061397"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid's l2:0.000306984+0.000613968  valid's l1:0.000306984+0.000613968"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid's l2:0.000306984+0.000613968  valid's l1:0.000306984+0.000613968"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[1] "[4]:  valid's l2:0.000306984+0.000613968  valid's l1:0.000306984+0.000613968"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[1] "[5]:  valid's l2:0.000306984+0.000613968  valid's l1:0.000306984+0.000613968"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[1] "[6]:  valid's l2:0.000306984+0.000613968  valid's l1:0.000306984+0.000613968"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[1] "[7]:  valid's l2:0.000306984+0.000613968  valid's l1:0.000306984+0.000613968"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[1] "[8]:  valid's l2:0.000306984+0.000613968  valid's l1:0.000306984+0.000613968"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[1] "[9]:  valid's l2:0.000306984+0.000613968  valid's l1:0.000306984+0.000613968"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] Stopped training because there are no more leaves that meet the split requirements
[1] "[10]:  valid's l2:0.000306984+0.000613968  valid's l1:0.000306984+0.000613968"
[LightGBM] [Info] Number of positive: 198, number of negative: 202
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.039497 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 167
[LightGBM] [Info] Number of data points in the train set: 400, number of used features: 1
[LightGBM] [Info] Number of positive: 196, number of negative: 204
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.017922 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 167
[LightGBM] [Info] Number of data points in the train set: 400, number of used features: 1
[LightGBM] [Info] Number of positive: 207, number of negative: 193
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.020078 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 167
[LightGBM] [Info] Number of data points in the train set: 400, number of used features: 1
[LightGBM] [Info] Number of positive: 207, number of negative: 193
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.008228 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 167
[LightGBM] [Info] Number of data points in the train set: 400, number of used features: 1
[LightGBM] [Info] Number of positive: 192, number of negative: 208
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.020154 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 167
[LightGBM] [Info] Number of data points in the train set: 400, number of used features: 1
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.495000 -> initscore=-0.020001
[LightGBM] [Info] Start training from score -0.020001
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.490000 -> initscore=-0.040005
[LightGBM] [Info] Start training from score -0.040005
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.517500 -> initscore=0.070029
[LightGBM] [Info] Start training from score 0.070029
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.517500 -> initscore=0.070029
[LightGBM] [Info] Start training from score 0.070029
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.480000 -> initscore=-0.080043
[LightGBM] [Info] Start training from score -0.080043
[1] "[1]:  valid's auc:0.476662+0.0622898  valid's binary_error:0.5+0.0593296"
[1] "[2]:  valid's auc:0.477476+0.0393392  valid's binary_error:0.554+0.0372022"
[1] "[3]:  valid's auc:0.456927+0.042898  valid's binary_error:0.526+0.0361109"
[1] "[4]:  valid's auc:0.419531+0.0344972  valid's binary_error:0.54+0.0289828"
[1] "[5]:  valid's auc:0.459109+0.0862237  valid's binary_error:0.52+0.0489898"
[1] "[6]:  valid's auc:0.460522+0.0911246  valid's binary_error:0.528+0.0231517"
[1] "[7]:  valid's auc:0.456328+0.0540445  valid's binary_error:0.532+0.0386782"
[1] "[8]:  valid's auc:0.463653+0.0660907  valid's binary_error:0.514+0.0488262"
[1] "[9]:  valid's auc:0.443017+0.0549965  valid's binary_error:0.55+0.0303315"
[1] "[10]:  valid's auc:0.477483+0.0763283  valid's binary_error:0.488+0.0549181"
[LightGBM] [Info] Number of positive: 3140, number of negative: 3373
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.563600 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.482113 -> initscore=-0.071580
[LightGBM] [Info] Start training from score -0.071580
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  train's binary_error:0.00307078  train's auc:0.99996  train's binary_logloss:0.132074"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  train's binary_error:0.00153539  train's auc:1  train's binary_logloss:0.0444372"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  train's binary_error:0  train's auc:1  train's binary_logloss:0.0159408"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  train's binary_error:0  train's auc:1  train's binary_logloss:0.00590065"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  train's binary_error:0  train's auc:1  train's binary_logloss:0.00230167"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  train's binary_error:0  train's auc:1  train's binary_logloss:0.00084253"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[7]:  train's binary_error:0  train's auc:1  train's binary_logloss:0.000309409"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[8]:  train's binary_error:0  train's auc:1  train's binary_logloss:0.000113754"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[9]:  train's binary_error:0  train's auc:1  train's binary_logloss:4.1838e-05"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[10]:  train's binary_error:0  train's auc:1  train's binary_logloss:1.539e-05"
[LightGBM] [Info] Number of positive: 3140, number of negative: 3373
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.482113 -> initscore=-0.071580
[LightGBM] [Info] Start training from score -0.071580
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Info] Number of positive: 3140, number of negative: 3373
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.482113 -> initscore=-0.071580
[LightGBM] [Info] Start training from score -0.071580
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Info] Number of positive: 35110, number of negative: 34890
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.929984 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 12
[LightGBM] [Info] Number of data points in the train set: 70000, number of used features: 1
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.501571 -> initscore=0.006286
[LightGBM] [Info] Start training from score 0.006286
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Info] Number of positive: 500, number of negative: 500
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.030553 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 3
[LightGBM] [Info] Number of data points in the train set: 1000, number of used features: 1
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[7]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[8]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[9]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[10]:  valid1's binary_error:0"
[LightGBM] [Info] Number of positive: 500, number of negative: 500
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.026274 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 3
[LightGBM] [Info] Number of data points in the train set: 1000, number of used features: 1
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  valid1's binary_error:0"
[LightGBM] [Info] Number of positive: 500, number of negative: 500
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.022607 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 3
[LightGBM] [Info] Number of data points in the train set: 1000, number of used features: 1
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's binary_error:0"
[LightGBM] [Info] Number of positive: 500, number of negative: 500
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.023750 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 3
[LightGBM] [Info] Number of data points in the train set: 1000, number of used features: 1
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's binary_error:0"
[LightGBM] [Info] Number of positive: 500, number of negative: 500
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.019646 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 3
[LightGBM] [Info] Number of data points in the train set: 1000, number of used features: 1
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's binary_error:0"
[LightGBM] [Info] Number of positive: 500, number of negative: 500
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.030110 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 3
[LightGBM] [Info] Number of data points in the train set: 1000, number of used features: 1
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.500000 -> initscore=0.000000
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's binary_error:0"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's binary_error:0"
[LightGBM] [Info] Number of positive: 3140, number of negative: 3373
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.591415 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.482113 -> initscore=-0.071580
[LightGBM] [Info] Start training from score -0.071580
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's auc:0.987036"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's auc:0.987036"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's auc:0.998699"
[1] "[4]:  valid1's auc:0.998699"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's auc:0.998699"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  valid1's auc:0.999667"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[7]:  valid1's auc:0.999806"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[8]:  valid1's auc:0.999978"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[9]:  valid1's auc:0.999997"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[10]:  valid1's auc:0.999997"
[LightGBM] [Info] Number of positive: 3140, number of negative: 3373
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.621401 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.482113 -> initscore=-0.071580
[LightGBM] [Info] Start training from score -0.071580
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's binary_error:0.016139"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's binary_error:0.016139"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's binary_error:0.016139"
[1] "[4]:  valid1's binary_error:0.016139"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's binary_error:0.016139"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  valid1's binary_error:0.016139"
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.014909 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 3
[LightGBM] [Info] Number of data points in the train set: 1000, number of used features: 1
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's rmse:55"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's rmse:59.5"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's rmse:63.55"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's rmse:67.195"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's rmse:70.4755"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  valid1's rmse:73.428"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[7]:  valid1's rmse:76.0852"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[8]:  valid1's rmse:78.4766"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[9]:  valid1's rmse:80.629"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[10]:  valid1's rmse:82.5661"
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.054524 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 3
[LightGBM] [Info] Number of data points in the train set: 1000, number of used features: 1
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's rmse:55"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's rmse:59.5"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's rmse:63.55"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's rmse:67.195"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's rmse:70.4755"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  valid1's rmse:73.428"
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.007496 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 35
[LightGBM] [Info] Number of data points in the train set: 100, number of used features: 1
[LightGBM] [Info] Start training from score 0.045019
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's constant_metric:0.2  valid1's increasing_metric:0.1"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's constant_metric:0.2  valid1's increasing_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's constant_metric:0.2  valid1's increasing_metric:0.3"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's constant_metric:0.2  valid1's increasing_metric:0.4"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's constant_metric:0.2  valid1's increasing_metric:0.5"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  valid1's constant_metric:0.2  valid1's increasing_metric:0.6"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[7]:  valid1's constant_metric:0.2  valid1's increasing_metric:0.7"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[8]:  valid1's constant_metric:0.2  valid1's increasing_metric:0.8"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[9]:  valid1's constant_metric:0.2  valid1's increasing_metric:0.9"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[10]:  valid1's constant_metric:0.2  valid1's increasing_metric:1"
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.066944 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 35
[LightGBM] [Info] Number of data points in the train set: 100, number of used features: 1
[LightGBM] [Info] Start training from score 0.045019
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's increasing_metric:1.1  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's increasing_metric:1.2  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's increasing_metric:1.3  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's increasing_metric:1.4  valid1's constant_metric:0.2"
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.060709 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 35
[LightGBM] [Info] Number of data points in the train set: 100, number of used features: 1
[LightGBM] [Info] Start training from score 0.045019
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's increasing_metric:1.5  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's increasing_metric:1.6  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's increasing_metric:1.7  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's increasing_metric:1.8  valid1's constant_metric:0.2"
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.036956 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 35
[LightGBM] [Info] Number of data points in the train set: 100, number of used features: 1
[LightGBM] [Info] Start training from score 0.045019
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's increasing_metric:1.9  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's increasing_metric:2  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's increasing_metric:2.1  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's increasing_metric:2.2  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's increasing_metric:2.3  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  valid1's increasing_metric:2.4  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[7]:  valid1's increasing_metric:2.5  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[8]:  valid1's increasing_metric:2.6  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[9]:  valid1's increasing_metric:2.7  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[10]:  valid1's increasing_metric:2.8  valid1's constant_metric:0.2"
[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.024168 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 35
[LightGBM] [Info] Number of data points in the train set: 100, number of used features: 1
[LightGBM] [Info] Start training from score 0.045019
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's rmse:1.10501  valid1's l2:1.22105  valid1's increasing_metric:2.9  valid1's rmse:1.10501  valid1's l2:1.22105  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's rmse:1.10335  valid1's l2:1.21738  valid1's increasing_metric:3  valid1's rmse:1.10335  valid1's l2:1.21738  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's rmse:1.10199  valid1's l2:1.21438  valid1's increasing_metric:3.1  valid1's rmse:1.10199  valid1's l2:1.21438  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's rmse:1.10198  valid1's l2:1.21436  valid1's increasing_metric:3.2  valid1's rmse:1.10198  valid1's l2:1.21436  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's rmse:1.10128  valid1's l2:1.21282  valid1's increasing_metric:3.3  valid1's rmse:1.10128  valid1's l2:1.21282  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  valid1's rmse:1.10101  valid1's l2:1.21222  valid1's increasing_metric:3.4  valid1's rmse:1.10101  valid1's l2:1.21222  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[7]:  valid1's rmse:1.10065  valid1's l2:1.21143  valid1's increasing_metric:3.5  valid1's rmse:1.10065  valid1's l2:1.21143  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[8]:  valid1's rmse:1.10011  valid1's l2:1.21025  valid1's increasing_metric:3.6  valid1's rmse:1.10011  valid1's l2:1.21025  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[9]:  valid1's rmse:1.09999  valid1's l2:1.20997  valid1's increasing_metric:3.7  valid1's rmse:1.09999  valid1's l2:1.20997  valid1's constant_metric:0.2"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[10]:  valid1's rmse:1.09954  valid1's l2:1.20898  valid1's increasing_metric:3.8  valid1's rmse:1.09954  valid1's l2:1.20898  valid1's constant_metric:0.2"
[LightGBM] [Info] Number of positive: 66, number of negative: 54
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.026149 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 42
[LightGBM] [Info] Number of data points in the train set: 120, number of used features: 1
[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.550000 -> initscore=0.200671
[LightGBM] [Info] Start training from score 0.200671
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[1]:  valid1's binary_error:0.486486  valid1's binary_logloss:0.693255"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[2]:  valid1's binary_error:0.486486  valid1's binary_logloss:0.691495"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[3]:  valid1's binary_error:0.486486  valid1's binary_logloss:0.69009"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[4]:  valid1's binary_error:0.432432  valid1's binary_logloss:0.688968"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[5]:  valid1's binary_error:0.432432  valid1's binary_logloss:0.688534"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[6]:  valid1's binary_error:0.432432  valid1's binary_logloss:0.689883"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[7]:  valid1's binary_error:0.432432  valid1's binary_logloss:0.689641"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[8]:  valid1's binary_error:0.432432  valid1's binary_logloss:0.689532"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[9]:  valid1's binary_error:0.432432  valid1's binary_logloss:0.691066"
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] "[10]:  valid1's binary_error:0.432432  valid1's binary_logloss:0.690653"
[LightGBM] [Info] Number of positive: 66, number of negative: 54
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.018168 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 42
[LightGBM] [Info] Number of data points in the train set: 120, number of used features: 1