Early stopping for LightGBM

Currently we can set a value for the 'earlyStoppingRound' parameter of 'LightGBMClassifier'. However, how can we actually train the model on a training set and monitoring the performance/performing early stopping using a validation set? LightGBMClassifier.fit() only takes two parameters: dataset and params. I've checked that dataset can only take in the training set.

hi @gyz0807 , after fitting a model you can call score and then use the spark ML evaluators to do validation.  Since LightGBM is in spark, it works like all other estimators in the spark ecosystem, and is compatible with the Spark ML evaluators.  Please see:
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
or the mllib one:
https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
You can also use the ComputeModelStatistics from mmlspark as well, you will need to specify the label and score columns unless you are using TrainClassifier, in which case we add metadata to the dataframe so that you won't have to specify those columns (it will be handled automatically).
I hope this answers your question, I am guessing you are not as familiar with the spark ecosystem and you are more familiar with lightgbm which is why you are having this issue.
Please let me know if you have any questions.

Hi Ilya, thanks for the fast response. I think these are the ways to validate the model after I finish training it. Is it possible to validate the model while training? For example, for each boosting round, use the model performance on the validation set to determine whether to perform early stop or not. If the performance does not improve for like 10 rounds, training would be stopped.

@gyz0807 this sounds like a new enhancement, but it is different from how other spark ML estimators work.  Currently other spark ML learners do not support this kind of validation with early stopping during training workflow.  I think there needs to be more design on how this could be implemented.  As a workaround, however, you could just train for k iterations, save the learner, evaluate on the evaluation dataset, and then decide whether to continue to train by loading and using the saved learner (we support retraining scenario by passing in the lightgbm native learner string).

Good idea. Thanks for the help.

So are you saying that https://github.com/Azure/mmlspark/pull/344 effectively does nothing as it's only working on the train data set?... Will try the approach you suggested. Thanks for the help as always.

@chibberto good point, maybe @terrytangyuan may give more info on how this param can be used.  I'm not sure how it is possible without being able to provide a validation dataset hmm.  @terrytangyuan is there an example of how the current early stopping round param is used?  If there is, maybe we should add a test case to verify the param.
@chibberto I think this may be something that we need to call to add the validation data:
https://github.com/Microsoft/LightGBM/blob/master/include/LightGBM/c_api.h#L394
I wonder if having a param which splits the given dataset into train/validation might be useful, or if we can have another param that takes in an evaluation dataset (we won't be able to easily do it on fit however, since it only takes a single dataset via the apache spark API that we have to conform to).

@chibberto Yes, it's correct that this param is not working yet. Additional work is needed to support validation data and to expose the early stopping logic. You could try the temporary workaround @imatiach-msft suggested.

Thanks for the responses. I thought that was the case but just wanted to double check. I see your point about usage with fit being tricky with ML... Will try and use the work around for now.

Also looking for a working early stopping against an evaluation dataset for LightGBMClassifier, as per native LightGBM.  Something like this:
model = LightGBMClassifier(featuresCol = 'features', labelCol = 'label',
learningRate = 0.05, numIterations = 100)
model.fit(train, eval_set=test, eval_metric="auc", early_stopping_rounds=50, verbose = True)

Any idea if this is being worked on?  I know continual updates come through for the mml library, so just checking if this is on the radar?

@givenx sure, I can look into it when I have some spare time.  I like your idea of passing in the evaluation dataset as a param, that's probably the only way to make it work well with the spark interface.  BTW this feedback is very useful, I try to prioritize the most commented issues first.  Also, if you need this functionality right away, MMLSpark is open source so feel free to send out a PR.

@imatiach-msft sadly this is not my idea but how xgboost works. It would be really great to get this working as early stopping against an evaluation set is very effective. Not sure how to send out a PR though :-/

@imatiach-msft
Early stopping is another important feature to reach feature parity with LightGBM. Maybe we could open a meta-issue to track LightGBM-parity issues (I see that categorical features support and multi-class classification support have been added... we still lack: early stopping support, SHAP values support, NaNs pass through #304 )

@ekerazha the nans issue is with the TrainClassifier/TrainRegressor being used with LightGBM (which are an AutoML featurizer) - if you use LightGBM directly it should be fine.  It's not an issue in LightGBM.
I agree early stopping, LGBMRanker and shap TreeExplainer support should be added eventually.  I would prefer to have separate issues though and fix them one at a time.  Having everything in one place might be a bit confusing and could create very long message threads, if that's ok.

@ekerazha my only concern with the fit(train, val) api is that it won't work with something like Spark ML's cross validator, which would be a key scenario that we would probably want to support.  Probably using the the param is the best way to enable the broadest set of scenarios and usage, although the fit(train, val) interface is much nicer.

@imatiach-msft it looks like in Spark they are going to implement it this way: https://github.com/apache/spark/pull/21129/files They use an indicator column which is False for rows of the training set, and True for rows of the validation set

@ekerazha actually, that resolves a lot of issues for me.  I was just starting to implement this last night and I couldn't figure out how to pass the validation set to all of the workers (eg do a second map partitions somehow?).  This paradigm makes everything a lot easier since I just need to generate the validation set within the mapPartitions calls just like the lightgbm training set is already created.

closing as PR has been merged

Great job... kudos

Fantastic, thanks a million...

Sorry, guys, correct me if I am wrong but it seems that this feature is not available in the Azure/mmlspark LightGBMClassifier class. In fact, these are the only parameters I can set in the constructor:
LightGBMClassifier(
    baggingFraction=1.0,
    baggingFreq=0,
    baggingSeed=3,
    boostFromAverage=True,
    boostingType='gbdt',
    categoricalSlotIndexes=None,
    categoricalSlotNames=None,
    defaultListenPort=12400,
    earlyStoppingRound=0,
    featureFraction=1.0,
    featuresCol='features',
    initScoreCol=None,
    isProvideTrainingMetric=False,
    isUnbalance=False,
    labelCol='label',
    lambdaL1=0.0,
    lambdaL2=0.0,
    learningRate=0.1,
    maxBin=255,
    maxDepth=-1,
    metric='',
    minSumHessianInLeaf=0.001,
    modelString='',
    numBatches=0,
    numIterations=100,
    numLeaves=31,
    objective='binary',
    parallelism='data_parallel',
    predictionCol='prediction',
    probabilityCol='probability',
    rawPredictionCol='rawPrediction',
    thresholds=None,
    timeout=1200.0,
    useBarrierExecutionMode=False,
    validationIndicatorCol=None,
    verbosity=1,
    weightCol=None,
)

while the fit method only accepts 2 parameters:
lgb.fit(dataset, params=None)

Hi, Can you please elaborate how we should use validation set with lightgbm from mmlspark?
TypeError: init() got an unexpected keyword argument 'eval_sets'

Hi, Can you please elaborate how we should use validation set with lightgbm from mmlspark?
TypeError: init() got an unexpected keyword argument 'eval_sets'

in order to use the validation set you should NOT create a separate df but, instead, you simply have to add an additional column (boolean) where you put True or 1 for all the rows you'd like to use as validation set.
Personally I use this function to add this column:
    def _add_validation_column(self, dataframe, validation_percentage):
        # true only if random 0 <= random < p

        N = 10
        i = 0
        exit = 0
        while exit==0 and i<N:
            i += 1
            print(f"adding validation column (attempt: {i}/{N})")
            dataframe = dataframe.withColumn('validation', F.udf(lambda: random() < validation_percentage, 'boolean')())

            temp_df = dataframe.cube(['validation', 'to_hit']).count().toPandas()
            tot = temp_df.query('validation!=validation and to_hit!=to_hit')['count'].values[0]
            tot_to_hit_1 = temp_df.query('validation!=validation and to_hit==1')['count'].values[0]
            val = temp_df.query('validation==True and to_hit!=to_hit')['count'].values[0]
            val_to_hit_1 = temp_df.query('validation==True and to_hit==1')['count'].values[0]

            if tot_to_hit_1/tot > val_to_hit_1/val - 0.025 and tot_to_hit_1/tot < val_to_hit_1/val + 0.025:
                return dataframe
            else:
                dataframe.drop('validation')

        raise MaxIterationReached("Max number of iterations for _add_validation_column reached")

after this, the following lines should be enough:
lgb = LightGBMClassifier(
            featuresCol='features',
            labelCol='to_hit',
            isUnbalance=True,
            boostingType='dart',
            earlyStoppingRound=25,
            metric='binary_logloss',
            validationIndicatorCol='validation',
            **params
)
model = lgb.fit(train)
pred = model.transform(test)


🌱 cmd/kcp: use structural logging

Summary
Switch to structural logging in cmd/kcp
Related issue(s)
Part of #1694

/approve

#2207
/retest

/lgtm