[python-package] Using lgb.Booster(model_file=model_file) and joblib.load() loading model to predict the same dataset, get different result.

Description
[python-package] Using lgb.Booster(model_file=model_file) and joblib.load() loading model to predict the same dataset, get different result.
Using joblib.load() can get expectant result. But using lgb.Booster(model_file=model_file)  get a result,  recall
Reproducible example
Not very convenient to display the code. But if any version change code lgb.Booster(model_file=model_file) , i can rollback to that version.
Environment info
LightGBM version or commit hash: python package version 3.3.5
Command(s) you used to install LightGBM
pip install lightgbm

Additional Comments

Thanks for using LightGBM.
This is not enough information for us to be able do more for you than just guess at what might be happening.
Can you please share a minimal, reproducible example (link), some code we can run which shows exactly what you're trying to do? If you can't share your actual code, that's understandable... you could create a small example using publicly-available data like the datasets from sklearn.datasets.
Here's a good example: https://github.com/microsoft/LightGBM/issues/5617#issuecomment-1338655747

lightgbm demo.zip
I use model_train method to get model.pkl. Remove unreadable codes get model.txt. Use train data to predict, model.pkl get expectant result. But model.txt get error result.
lightgbm python version: 3.3.5

I've uploaded the code in plaintext here, and attached only the CSV dataset. In the future, please do that! So that others can see the code without needing to download and unzip a file, and so it'll show up in search results.
from sklearn.metrics import *
from sklearn.model_selection import StratifiedKFold, KFold, train_test_split
import lightgbm as lgb
import pandas as pd
import time
import calendar
import joblib

random_state = 1

def eval(y_test, y_pred, threshold):
    # statistics(y_test, y_pred)

    for i in range(y_pred.__len__()):
        if y_pred[i] > threshold:
            y_pred[i] = 1
        else:
            y_pred[i] = 0

    # eval
    rmse_test = mean_squared_error(y_test, y_pred) ** 0.5
    print(f'The RMSE of prediction is: {rmse_test}')

    # 混淆矩阵
    tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()
    print(f'tn, fp, fn, tp : {tn}, {fp}, {fn}, {tp}')

    # accuracy
    accuracy = accuracy_score(y_test, y_pred)
    print(f'accuracy score: {accuracy}')

    # precision
    precision = precision_score(y_test, y_pred)
    print(f'precision score: {precision}')

    # recall
    recall = recall_score(y_test, y_pred)
    print(f'recall score: {recall}')

    # f1
    f1 = f1_score(y_test, y_pred)
    print(f'f1 score: {f1}')

    # auc
    auc_scoure = roc_auc_score(y_test, y_pred)
    print(f'auc score: {auc_scoure}')

def set_type(df):
    continuous_columns = ['auditor_hired_date', 'auditor_quality_7d', 'auditor_quality_30d',
                          'auditor_valid_time_rate_7d', 'auditor_valid_time_rate_30d',
                          'auditor_pure_time_rate_7d', 'auditor_pure_time_rate_30d',
                          'audit_time', 'video_duration', 'time_level', 'case_duration',
                          'summary_length',
                          'quality_score']

    category_columns = ['auditor_title_id', 'auditor_status',
                        'audit_origin_status', 'content_biz_line', 'biz_code', 'ugc_scene', 'content_type',
                        'is_multi_pics', 'has_item',
                        'grade_machine_value', 'grade_manual_value', 'cert_type', 'creator_g_level', 'creator_role',
                        'is_second_edit']

    df[continuous_columns] = df[continuous_columns].astype('float')
    df[category_columns] = df[category_columns].astype('category')
    print(df.info())
    print(df["check_result"].value_counts())
    return df

def load(file_name):
    df = pd.read_csv(f"target_file/{file_name}.csv", header=0, sep=',', low_memory=False, encoding='utf-8')
    df = df.loc[:, ~df.columns.str.contains('Unnamed')]
    df = set_type(df)
    return df

def spilt_data(df, label_column, random_state):
    df_train, df_test = train_test_split(df, test_size=0.2, stratify=df[label_column], random_state=random_state)
    df_train, df_valid = train_test_split(df_train, test_size=0.2, stratify=df_train[label_column],
                                          random_state=random_state)

    y_train = df_train[label_column]
    y_valid = df_valid[label_column]
    y_test = df_test[label_column]
    x_train = df_train.drop(label_column, axis=1)
    x_valid = df_valid.drop(label_column, axis=1)
    x_test = df_test.drop(label_column, axis=1)
    return y_train, y_valid, y_test, x_train, x_valid, x_test

def model_train(file_name):
    ts = calendar.timegm(time.gmtime())

    print('Starting training...')
    print('Starting loading df...')
    df = load(file_name)

    y_train, y_valid, y_test, x_train, x_valid, x_test = spilt_data(df, "check_result", random_state)

    print('Starting training...')
    gbm = lgb.LGBMClassifier(num_leaves=8,
                             max_depth=3,
                             # min_child_samples=50,
                             learning_rate=0.05,
                             n_estimators=1000,
                             is_unbalance=True)
    gbm.fit(x_train, y_train,
            eval_set=[(x_valid, y_valid)],
            # eval_metric='l2',
            eval_metric='recall',
            # callbacks=[lgb.early_stopping(10000)],
            verbose=100)

    # save model to file
    model_file = f'model/model'
    # gbm.booster_.save_model(model_file + ".txt")
    joblib.dump(gbm, model_file + ".pkl")

    # feature importances
    print(f'Feature importances: {list(gbm.feature_importances_)}')
    print(gbm.feature_name_)

    print('Starting test training data...')
    # predict
    y_pred = gbm.predict(x_train, num_iteration=gbm.best_iteration_)
    eval(y_train, y_pred, 0.9)

    print('Starting predicting...')
    # predict
    y_pred = gbm.predict(x_valid, num_iteration=gbm.best_iteration_)
    eval(y_valid, y_pred, 0.9)

    print('Starting testing...')
    y_pred = gbm.predict(x_test, num_iteration=gbm.booster_.best_iteration)
    eval(y_test, y_pred, 0.9)

    print('Starting testing all...')
    y_all = df["check_result"]
    x_all = df.drop("check_result", axis=1)
    y_pred = gbm.predict(x_all, num_iteration=gbm.booster_.best_iteration)
    eval(y_all, y_pred, 0.9)

    return model_file

def model_test(file_name, model_file, model_type):
    print('Starting testing...')

    if "txt" == model_type:
        bst = lgb.Booster(model_file=f"model/{model_file}")
    elif "pkl" == model_type:
        bst = joblib.load(f"model/{model_file}")

    print('Starting loading df...')
    df = load(file_name)
    y_test = df["check_result"]
    X_test = df.drop("check_result", axis=1)

    # predict
    # print(f'best iteration: {bst.best_iteration}')
    y_pred = bst.predict(X_test)

    print(y_pred)

    eval(y_test, y_pred, 0.9)

if __name__ == "__main__":
    # model_file = model_train("dataset")
    # model_test("dataset", "model.pkl", "pkl")
    model_test("dataset", "model.txt", "txt")

    pass

dataset.csv
Before we spend time on this, could you please help us some more?

try to minimize this to the smallest amount of code that reproduces the issue (for example, please replace that * import with specific imports, remove unnecessary computation of metrics)
clarify.... is the problem that you get a "different" result or an "error"? If it's an "error", please provide the exact text of the error message

from sklearn.model_selection import train_test_split
import lightgbm as lgb
import pandas as pd
import time
import calendar
import joblib

random_state = 1

def eval(y_test, y_pred, threshold):
    for i in range(y_pred.__len__()):
        if y_pred[i] > threshold:
            y_pred[i] = 1
        else:
            y_pred[i] = 0

    tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()
    print(f'tn, fp, fn, tp : {tn}, {fp}, {fn}, {tp}')

    # recall
    recall = recall_score(y_test, y_pred)
    print(f'recall score: {recall}')

def set_type(df):
    continuous_columns = ['auditor_hired_date', 'auditor_quality_7d', 'auditor_quality_30d',
                          'auditor_valid_time_rate_7d', 'auditor_valid_time_rate_30d',
                          'auditor_pure_time_rate_7d', 'auditor_pure_time_rate_30d',
                          'audit_time', 'video_duration', 'time_level', 'case_duration',
                          'summary_length',
                          'quality_score']

    category_columns = ['auditor_title_id', 'auditor_status',
                        'audit_origin_status', 'content_biz_line', 'biz_code', 'ugc_scene', 'content_type',
                        'is_multi_pics', 'has_item',
                        'grade_machine_value', 'grade_manual_value', 'cert_type', 'creator_g_level', 'creator_role',
                        'is_second_edit']

    df[continuous_columns] = df[continuous_columns].astype('float')
    df[category_columns] = df[category_columns].astype('category')
    print(df.info())
    print(df["check_result"].value_counts())
    return df

def load(file_name):
    df = pd.read_csv(f"target_file/{file_name}.csv", header=0, sep=',', low_memory=False, encoding='utf-8')
    df = df.loc[:, ~df.columns.str.contains('Unnamed')]
    df = set_type(df)
    return df

def spilt_data(df, label_column, random_state):
    df_train, df_test = train_test_split(df, test_size=0.2, stratify=df[label_column], random_state=random_state)
    df_train, df_valid = train_test_split(df_train, test_size=0.2, stratify=df_train[label_column],
                                          random_state=random_state)

    y_train = df_train[label_column]
    y_valid = df_valid[label_column]
    y_test = df_test[label_column]
    x_train = df_train.drop(label_column, axis=1)
    x_valid = df_valid.drop(label_column, axis=1)
    x_test = df_test.drop(label_column, axis=1)
    return y_train, y_valid, y_test, x_train, x_valid, x_test

def model_train(file_name):
    ts = calendar.timegm(time.gmtime())

    print('Starting training...')
    print('Starting loading df...')
    df = load(file_name)

    y_train, y_valid, y_test, x_train, x_valid, x_test = spilt_data(df, "check_result", random_state)

    print('Starting training...')
    gbm = lgb.LGBMClassifier(num_leaves=8,
                             max_depth=3,
                             # min_child_samples=50,
                             learning_rate=0.05,
                             n_estimators=1000,
                             is_unbalance=True)
    gbm.fit(x_train, y_train,
            eval_set=[(x_valid, y_valid)],
            # eval_metric='l2',
            eval_metric='recall',
            # callbacks=[lgb.early_stopping(10000)],
            verbose=100)

    # save model to file
    model_file = f'model/model'
    # gbm.booster_.save_model(model_file + ".txt")
    joblib.dump(gbm, model_file + ".pkl")

    # feature importances
    print(f'Feature importances: {list(gbm.feature_importances_)}')
    print(gbm.feature_name_)

    print('Starting testing all...')
    y_all = df["check_result"]
    x_all = df.drop("check_result", axis=1)
    y_pred = gbm.predict(x_all, num_iteration=gbm.booster_.best_iteration)
    eval(y_all, y_pred, 0.9)

    return model_file

def model_test(file_name, model_file, model_type):
    print('Starting testing...')

    if "txt" == model_type:
        bst = lgb.Booster(model_file=f"model/{model_file}")
    elif "pkl" == model_type:
        bst = joblib.load(f"model/{model_file}")

    print('Starting loading df...')
    df = load(file_name)
    y_test = df["check_result"]
    X_test = df.drop("check_result", axis=1)

    # predict
    y_pred = bst.predict(X_test)

    print(y_pred)

    eval(y_test, y_pred, 0.9)

if __name__ == "__main__":
    model_file = model_train("dataset")
    # model_test("dataset", "model.pkl", "pkl")
    # model_test("dataset", "model.txt", "txt")

    pass

Using complete data to test has representativeness. Method model_file = model_train("dataset") get result:
tn, fp, fn, tp : 45458, 20051, 2540, 8507
recall score: 0.7700733230741378

Method model_test("dataset", "model.pkl", "pkl") get result:
tn, fp, fn, tp : 45458, 20051, 2540, 8507
recall score: 0.7700733230741378

Method model_test("dataset", "model.txt", "txt") get result:
tn, fp, fn, tp : 65490, 19, 11007, 40
recall score: 0.003620892550013578

We can see that, 'confusion_matrix' is different from the result by different type of model file. Format of 'pkl' is correct, format of 'txt' is incorrect.
Although not all datasets will yield different results, there is a problem with this case.

Additionally, sometimes the model results have a probability of 0 to 1, and sometimes only 0 and 1 are the two results. Is there any configuration can specify ?

from sklearn.model_selection import train_test_split
import lightgbm as lgb
import pandas as pd
import time
import calendar
import joblib

random_state = 1

def eval(y_test, y_pred, threshold):
    for i in range(y_pred.__len__()):
        if y_pred[i] > threshold:
            y_pred[i] = 1
        else:
            y_pred[i] = 0

    tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel()
    print(f'tn, fp, fn, tp : {tn}, {fp}, {fn}, {tp}')

    # recall
    recall = recall_score(y_test, y_pred)
    print(f'recall score: {recall}')

def set_type(df):
    continuous_columns = ['auditor_hired_date', 'auditor_quality_7d', 'auditor_quality_30d',
                          'auditor_valid_time_rate_7d', 'auditor_valid_time_rate_30d',
                          'auditor_pure_time_rate_7d', 'auditor_pure_time_rate_30d',
                          'audit_time', 'video_duration', 'time_level', 'case_duration',
                          'summary_length',
                          'quality_score']

    category_columns = ['auditor_title_id', 'auditor_status',
                        'audit_origin_status', 'content_biz_line', 'biz_code', 'ugc_scene', 'content_type',
                        'is_multi_pics', 'has_item',
                        'grade_machine_value', 'grade_manual_value', 'cert_type', 'creator_g_level', 'creator_role',
                        'is_second_edit']

    df[continuous_columns] = df[continuous_columns].astype('float')
    df[category_columns] = df[category_columns].astype('category')
    print(df.info())
    print(df["check_result"].value_counts())
    return df

def load(file_name):
    df = pd.read_csv(f"target_file/{file_name}.csv", header=0, sep=',', low_memory=False, encoding='utf-8')
    df = df.loc[:, ~df.columns.str.contains('Unnamed')]
    df = set_type(df)
    return df

def spilt_data(df, label_column, random_state):
    df_train, df_test = train_test_split(df, test_size=0.2, stratify=df[label_column], random_state=random_state)
    df_train, df_valid = train_test_split(df_train, test_size=0.2, stratify=df_train[label_column],
                                          random_state=random_state)

    y_train = df_train[label_column]
    y_valid = df_valid[label_column]
    y_test = df_test[label_column]
    x_train = df_train.drop(label_column, axis=1)
    x_valid = df_valid.drop(label_column, axis=1)
    x_test = df_test.drop(label_column, axis=1)
    return y_train, y_valid, y_test, x_train, x_valid, x_test

def model_train(file_name):
    ts = calendar.timegm(time.gmtime())

    print('Starting training...')
    print('Starting loading df...')
    df = load(file_name)

    y_train, y_valid, y_test, x_train, x_valid, x_test = spilt_data(df, "check_result", random_state)

    print('Starting training...')
    gbm = lgb.LGBMClassifier(num_leaves=8,
                             max_depth=3,
                             # min_child_samples=50,
                             learning_rate=0.05,
                             n_estimators=1000,
                             is_unbalance=True)
    gbm.fit(x_train, y_train,
            eval_set=[(x_valid, y_valid)],
            # eval_metric='l2',
            eval_metric='recall',
            # callbacks=[lgb.early_stopping(10000)],
            verbose=100)

    # save model to file
    model_file = f'model/model'
    # gbm.booster_.save_model(model_file + ".txt")
    joblib.dump(gbm, model_file + ".pkl")

    # feature importances
    print(f'Feature importances: {list(gbm.feature_importances_)}')
    print(gbm.feature_name_)

    print('Starting testing all...')
    y_all = df["check_result"]
    x_all = df.drop("check_result", axis=1)
    y_pred = gbm.predict(x_all, num_iteration=gbm.booster_.best_iteration)
    eval(y_all, y_pred, 0.9)

    return model_file

def model_test(file_name, model_file, model_type):
    print('Starting testing...')

    if "txt" == model_type:
        bst = lgb.Booster(model_file=f"model/{model_file}")
    elif "pkl" == model_type:
        bst = joblib.load(f"model/{model_file}")

    print('Starting loading df...')
    df = load(file_name)
    y_test = df["check_result"]
    X_test = df.drop("check_result", axis=1)

    # predict
    y_pred = bst.predict(X_test)

    print(y_pred)

    eval(y_test, y_pred, 0.9)

if __name__ == "__main__":
    model_file = model_train("dataset")
    # model_test("dataset", "model.pkl", "pkl")
    # model_test("dataset", "model.txt", "txt")

    pass

Using complete data to test has representativeness. Method model_file = model_train("dataset") get result:
tn, fp, fn, tp : 45458, 20051, 2540, 8507
recall score: 0.7700733230741378

Method model_test("dataset", "model.pkl", "pkl") get result:
tn, fp, fn, tp : 45458, 20051, 2540, 8507
recall score: 0.7700733230741378

Method model_test("dataset", "model.txt", "txt") get result:
tn, fp, fn, tp : 65490, 19, 11007, 40
recall score: 0.003620892550013578

We can see that, 'confusion_matrix' is different from the result by different type of model file. Format of 'pkl' is correct, format of 'txt' is incorrect. Although not all datasets will yield different results, there is a problem with this case.

I need results from the 'txt' model to be right. Because I need convert it to 'pmml' file, so then get rest interface.

Thanks for that. This is still quite a lot of code.
Can you please try to reduce it to a minimal example showing the issue you're reporting?

no commented-out code
no if __name___ == "__main__" block (just assume the whole thing will be run as a script)
no unnecessary statements like ts = calendar.timegm(time.gmtime()) or corresponding unnecessary
n_estimators reduced to a smaller value like 10 (assuming that still reproduces the issue)
just use something like numpy.testing.assert_array_equal() or similar instead of that custom eval() function to show that the predictions are different

Doing the will reduce the effort required to identify the exact cause of the issue you're reporting.

import lightgbm as lgb
import pandas as pd
import numpy as np
import joblib

def set_type(df):
    continuous_columns = ['auditor_hired_date', 'auditor_quality_7d', 'auditor_quality_30d',
                          'auditor_valid_time_rate_7d', 'auditor_valid_time_rate_30d',
                          'auditor_pure_time_rate_7d', 'auditor_pure_time_rate_30d',
                          'audit_time', 'video_duration', 'time_level', 'case_duration',
                          'summary_length',
                          'quality_score']

    category_columns = ['auditor_title_id', 'auditor_status',
                        'audit_origin_status', 'content_biz_line', 'biz_code', 'ugc_scene', 'content_type',
                        'is_multi_pics', 'has_item',
                        'grade_machine_value', 'grade_manual_value', 'cert_type', 'creator_g_level', 'creator_role',
                        'is_second_edit']

    df[continuous_columns] = df[continuous_columns].astype('float')
    df[category_columns] = df[category_columns].astype('category')
    return df

def load(file_name):
    df = pd.read_csv(f"target_file/{file_name}.csv", header=0, sep=',', low_memory=False, encoding='utf-8')
    df = df.loc[:, ~df.columns.str.contains('Unnamed')]
    df = set_type(df)
    return df

def model_test(file_name, model_file, model_type):
    print('Starting testing...')

    if "txt" == model_type:
        bst = lgb.Booster(model_file=f"model/{model_file}")
    elif "pkl" == model_type:
        bst = joblib.load(f"model/{model_file}")

    print('Starting loading df...')
    df = load(file_name)
    X_test = df.drop("check_result", axis=1)

    # predict
    y_pred = bst.predict(X_test)

    return y_pred

pkl_result = model_test("dataset", "model.pkl", "pkl")
print(pkl_result)
txt_result = model_test("dataset", "model.txt", "txt")
print(txt_result)
np.testing.assert_array_equal(pkl_result, txt_result)

Using the bst.predict method. 'pkl' model get 0 and 1. 'txt' model get 0 to 1. I didn't use 'predict_proba' method.

'pkl' model get 0 and 1. 'txt' model get 0 to 1.

Oh I see! Thanks for trying to minimize the example so it's easier to understand what you're reporting. What you've provided still isn't a reproducible example (you've now omitted the code for training models, so the code in https://github.com/microsoft/LightGBM/issues/5951#issuecomment-1617180504 wouldn't work by itself), but I think I understand.
The difference you're seeing is because what you're storing to text and what you're storing with joblib are different objects.
What you've stored in a text file is a lightgbm.Booster object
What you've stored with joblib is an instance of lightgbm.sklearn.LGBMClassifier, which wraps a lightgbm.Booster in a scikit-learn-compatible interface.
By default for binary classification, LGBMClassifier.predict() returns predicted classes (to be consistent with the guidance at https://scikit-learn.org/stable/developers/develop.html).
lightgbm.Booster.predict(), on the other hand, defaults to returning probabilities of each case being the positive class (y=1).
See the example code below, which will work with both the latest released version of lightgbm (v3.3.5) and the latest development version of lightgbm here on GitHub.
import lightgbm as lgb
import numpy as np
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

model = lgb.LGBMClassifier(n_estimators=10, min_child_samples=1)
model.fit(X, y)

# LGBMClassifier.predict() returns classes by default
sklearn_preds = model.predict(X)
sklearn_preds[:20]
# array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1])

# Booster.predict() returns probability of y_i=1
booster_preds = model.booster_.predict(X)
# array([0.22205477, 0.22205477, 0.22205477, 0.22205477, 0.22205477,
#        0.22205477, 0.22205477, 0.22205477, 0.22205477, 0.22205477,
#        0.22205477, 0.22205477, 0.22205477, 0.22205477, 0.22205477,
#        0.22205477, 0.22205477, 0.22205477, 0.22205477, 0.86518328])

# confirm that Booster.predict() produces probabilities
sklearn_probabilities = model.predict_proba(X)[:,1]
np.testing.assert_array_equal(booster_preds, sklearn_probabilities)

# confirm that Booster.predict()'s probabilities exactly correspond to
# LGBMClassifier.predict()'s classes
booster_classes = np.argmax(
    a=model.predict_proba(X),
    axis=1
)
np.testing.assert_array_equal(sklearn_preds, booster_classes)

Because I need convert it to 'pmml' file

There isn't an official LightGBM-maintained converter for PMML in this project currently. You could try these projects:

https://github.com/jpmml/sklearn2pmml
https://github.com/jpmml/jpmml-sklearn
https://github.com/jpmml/jpmml-lightgbm

I haven't personally used those so can't say for sure if they'll work well for you, but that's where I'd start if I wanted to convert a LightGBM model to PMML.

Thanks for explaining my problem.


Port Eris maintenance airlock sprites.

Gonna need all of these for Saltern so if anybody's up for this that'd be appreciated.

I'm on it!


remove push main step

Please refer to our contributing docs for any questions on submitting a pull request
Issues are required for both bug fixes and features.
Resolves 138

Before the change?

After the change?

Pull request checklist

[x] Tests for the changes have been added (for bug fixes / features)
[x] Docs have been reviewed and added / updated if needed (for bug fixes / features)

Does this introduce a breaking change?

[ ] Yes
[x] No

Merging.


Split SupportsBackend into SupportsBackend and SupportsBackendExecutor

Closes #4

@euri10 so the reason a connection was being passed around was to make sure everything ran inside a transaction. Your feedback makes me think the root of the issue is that SupportsBackend is doing too many things. I think splitting it into two interfaces like this makes it a lot easier to understand what's going on and how it should be used. Now we have 2 interfaces:

SupportsBackend which is responsible for (1) creating a SupportsBackendExecutor (i.e. connecting to the db and/or creating a transaction) and (2) creating an executable function from .sql file.
SupportsBackendExecutor which is responsible for actually executing things and should internally have a connection wrapped in a transaction. Since this connection is now always wrapped in a transaction (or whatever the backend needs) it can use self.connection instead of having the connection as a parameter to each method, which is what you had originally proposed for SupportsBackend.

How does this sound to you?

this is way cleaner yep !


Update description of Clearing House

The CLearing is specified in a dedicated document, this should be linked (to IDS-G) relevant parts for RAM should be updated and integrated:

System Layer
Process Layer

Closed by #193


Debug logging in main pshtt exception cases

A slightly more thought out attempt at pull request #62. I did a bit of copy-editing to the existing warning messages throughout the main exception cases and added a debug level logging line for the actual exception messages.

lLooks Ggood!


TypeError: command.transformArguments is not a function when executing zRange command

I am trying to execute zRange command as follows
try {
  const res = await client.zRange("test", 0, -1, {
    REV: true,
  });
  console.log("res: ", res);
} catch (error) {
  console.log("tes2: ", error);
}

Look for the REV: true here. When I execute the above command with 'REV' argument, am getting following error.
Note: It is working well on Ubuntu 20 and 22 with same version of node-redis, node and redis-server. Not sure why this is only occurring in Windows.
2022-08-04T09:35:50.180Z  { FIRST_KEY_INDEX: 1, IS_READ_ONLY: true, transformArguments: '' } [ 'test', 0, -1 ] zRange
2022-08-04T09:35:50.186Z  tes2:  TypeError: command.transformArguments is not a function
at transformCommandArguments (E:\zoga-backend\node_modules@redis\client\dist\lib\commander.js:72:23)
at Commander.commandsExecutor (E:\zoga-backend\node_modules@redis\client\dist\lib\client\index.js:169:88)
at Commander.BaseClass. [as zRange] (E:\zoga-backend\node_modules@redis\client\dist\lib\commander.js:9:29)
at Commander. (E:\zoga-backend\database\redis\index.js:42:32)
at Commander.emit (node:events:527:28)
at Commander.emit (node:domain:475:12)
at RedisSocket. (E:\zoga-backend\node_modules@redis\client\dist\lib\client\index.js:352:35)
at RedisSocket.emit (node:events:527:28)
at RedisSocket.emit (node:domain:475:12)
at RedisSocket._RedisSocket_connect (E:\zoga-backend\node_modules@redis\client\dist\lib\client\socket.js:114:14)

![image](https://user-images.githubusercontent.com/25585634/182815815-c3db5347-f1b0-451e-9371-277b93d9210e.png)

**Environment:**
 - **Node.js Version**: 16.15.0
 - **Redis Server Version**: 5.0.7 and 7.0.4
 - **Node Redis Version**: 4.2.0
 - **Platform**: Windows 11

Not sure what was the exact reason out of the following two, but they somehow fixed the above issue.

Shifted terminal from 'powershell' to 'git bash'
Removed, node_modules
Did npm i
Ran it again and issue was gone :)


docs(node): Add instructions on how to use the node

Overview

This PR adds docs that explain how to start the node and connect it to Cardano mainnet and testnet, as well as which dependencies it requires.
Checklist
My PR contains...

[x] No code changes (changes to documentation, CI, metadata, etc.)
[ ] Bug fixes (non-breaking change which fixes an issue)
[ ] Improvements (misc. changes to existing features)
[ ] Features (non-breaking change which adds functionality)

My changes...

[x] are breaking changes
[ ] are not breaking changes
[ ] If yes to above: I have updated the documentation accordingly

Documentation

[x] My changes do not require a change to the project documentation
[ ] My changes require a change to the project documentation
[ ] If yes to above: I have updated the documentation accordingly

Tests

[x] My changes can not or do not need to be tested
[ ] My changes can and should be tested by unit and/or integration tests
[ ] If yes to above: I have added tests to cover my changes
[ ] If yes to above: I have taken care to cover edge cases in my tests

@petevielhaber @essbante-io , could you review the PR as well?


MinOffsetToPull for compact height

Smaller devices (i.e. iPhones, especially 5s) in landscape barely give enough height to pull to refresh.
I changed the MinOffsetToPull constant to 50 to be suitable for all devices, but it might be preferable to have it dynamic.
What are your thoughts on this value?

Definitely, I will change it to be a variable in a pull to refresh class so people can change that. :+1:


License change

I'm proposing to change the license from the current MIT license to a dual license MIT and Apache License v2.
This is just because almost all Rust crates (and Rust itself) use these licenses and thus will make things more uniform.
This is a minor change and will not effect the project much, but just makes it more convenient.
@fmeringdal if you agree I'll update this in my fork and will be changed once everything is merged.

Good idea. Go ahead @ralpha :)


Bump git to 2.14.2

The shipped git is out of date, and possibly insecure. I'd like to bump the shipped version to 2.14.2. I'll be submitting a PR with that change, but I wanted to check the process to recompile git.
I've performed the following steps:
Setup

Bring up an Amazon Linux instance
sudo yum groupinstall "Development Tools"
sudo yum install git
mkdir ~/install
mkdir ~/discard

openssh

git clone https://github.com/openssh/openssh-portable.git
Comment sshconnect.h per https://github.com/lambci/lambci/issues/26#issuecomment-274876986
sudo yum install openssl-devel
sudo yum install perl-devel
run autoreconf
./configure --prefix=$HOME/install --sysconfdir=$HOME/discard --sbindir=$HOME/discard --mandir=$HOME/discard
make
make install
cd

Git

wget https://www.kernel.org/pub/software/scm/git/git-2.14.2.tar.xz
tar xf git-2.14.2.tar.xz
cd git-2.14.2
./configure --prefix=$HOME/install --mandir=$HOME/discard --htmldir=$HOME/discard --infodir=$HOME/discard --without-tcltk
make
make install

Then , tar up ~/install and drop it into the vendor directory.
Any comments, things I missed, or extra things that should be done?

(once you figure it out, this might be a cool thing to document or have a script ready to do it for you)

On a side note... how does one use the git tarball on a Lambda instance? Download the tarball, extract, and run? I'm getting a read only filesystem error...  It’s autoexpanded at https://github.com/lambci/lambci/blob/master/utils/config.js#L88 and used in the clone function https://github.com/lambci/lambci/blob/master/actions/build.js#L135

Sorry, I mean outside of LambCI; I'm attempting to use the git tarball on a vanilla Lambda instance.

@fxfitz you can only write to /tmp, so you've got to do everything out of there. Aside from that there are a number of env vars you need to ensure are setup. Bit too many to remember off the top of my head, but if you copy all the env vars that LambCI sets, you should be good

Thanks for the info! I'll give it a shot!

Thanks for documenting this @cjs
Aside from bumping up the git version number, is there anything you'd do differently, or are doing differently these days?
@mhart, I recall reading somewhere that you were able to keep the size of the tarball really small. Did you have a different method than this?

@WhiteHotLoveTiger I am still waiting on documentation on how @mhart built his version. The size differential means there's some difference I don't know about and that troubles me.

@cjs are you still looking into an updated git version?
I am working on an approach for faster load time of lambda-git which I believe we will be able to use later on to update the git version.
Just let me know.
BTW, thanks a bunch for documenting this process!

Hi @marcelobern I think it would be useful for this project to be able to update the version, so I'm eager to see what you can do.

You can get git 2.13.5 in the version-0.10.0 branch: https://github.com/lambci/lambci/tree/version-0.10.0/vendor

You can see how I built it here: https://github.com/lambci/lambci/tree/version-0.10.0/build/git

@mhart thanks for sharing git 2.13.5 and the corresponding instructions!
FYI, I came across a few permission issues by using git-2.13.5.tar which were not there in git-2.4.3.tar. Not sure if these will lead to any issues for you, so just in case, here they are:

bin/git/usr/bin/ssh-agent - no read/write (o2111), it used to be o711
bin/git/usr/libexec/openssh/ssh-keysign - no read/write (o111), it used to be o711

@marcelobern oh thanks for the heads up – I'll double check these and fix if necessary

@mhart and @cjs with the release of AWS Lambda layers, I just packaged the git version provided as a Lambda layer to avoid uploading all binaries with each new code change.
It is available at npm as lambda-layer-git.
I would love to hear any feedback you might have on this being useful to you (and others).

I'll defer to @mhart, I haven't even looked at Lambda layers yet.

@marcelobern I did the same thing a few weeks ago – with an updated git version: https://github.com/lambci/git-lambda-layer 👍

Now that https://github.com/lambci/git-lambda-layer I think this issue can be closed.


Iox #542 sigaction abstraction

Pre-Review Checklist for the PR Author

[x] Branch follows the naming format (iox-#123-this-is-a-branch)
[x] Commits messages are according to this guideline

[x] Commit messages have the issue ID (iox-#123 commit text)
[x] Commit messages are signed (git commit -s)
[x] Commit author matches Eclipse Contributor Agreement (and ECA is signed)

[x] Update the PR title

Follow the same conventions as for commit messages
Link to the relevant issue

[x] Relevant issues are linked
[x] Add sensible notes for the reviewer
[x] All checks have passed (except task-list-completed)
[x] Assign PR to reviewer

Notes for Reviewer

Checklist for the PR Reviewer

[x] Commits are properly organized and messages are according to the guideline
[x] Code according to our coding style and naming conventions
[x] Unit tests have been written for new behavior
[x] Public API changes are documented via doxygen
[x] Copyright owner are updated in the changed files
[x] PR title describes the changes

Post-review Checklist for the PR Author

[ ] All open points are addressed and tracked via issues

Post-review Checklist for the Eclipse Committer

[ ] All checkboxes in the PR checklist are checked or crossed out
[ ] Merge

References

Partially closes #542

I did not yet review the tests.
Regarding the code. Let's assume I want to register SIGINT and SIGTERM in a function. An because I don't like two variables I create a cxx::vector and do an emplace. In which order are they destroyed and which sigaction will be restored at last?

Great question. Since they are two different signals both will be restored to their original state!

I did not yet review the tests.
Regarding the code. Let's assume I want to register SIGINT and SIGTERM in a function. An because I don't like two variables I create a cxx::vector and do an emplace. In which order are they destroyed and which sigaction will be restored at last?

Great question. Since they are two different signals both will be restored to their original state!

Of course :D
What I was curious about was in which order the cxx::vector destroys it's elements. Let's assume you register SIGINT twice and store it in a cxx::vector. Are they destroyed in reverse order of creation? It's a misuse in this case, but what about other classes stored in a cxx::vector? If it is not destroyed in reverse order, is it okay?

@elBoberido  According to https://stackoverflow.com/questions/6169125/order-of-destruction-of-elements-of-an-stdvector which refers to the standard the elements are destroyed in reverse order.
I would consider this a misuse by the user, she needs to consider destruction order of the container when using such scope guards. (It might get much less predictable with structures using trees for internal representation, e.g. maps, the user needs to consider this)
TLDR: not the responsibility of this class.


Revert Backport 1.x PR #2191

Description
The alerting plugin build started to fail with this commit. This is now blocking the opensearch-build for 1.3
This reverts the backport commit to unblock the opensearch-build
Issues Resolved
https://github.com/opensearch-project/alerting/issues/312
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

start gradle check


CollectionDirector.Event/TableDirector.Event collection/collectionSize - table/tableSize are nil

Hello,
the itemSize event Context has always collectionView and and collectionSize properties nil which is annoying if you want to make layout with even rows or columns.

Hello,
the itemSize event Context has always collectionView and and collectionSize properties nil which is annoying if you want to make layout with even rows or columns.

Hi, have you found a solution for this ?

Hi, just upgraded from FlowKit and still resolving issues.
One of my cells requires height calculation based on parent UITableView, however the rowHeight callback receives nil in table property of Event. It receives cell, but superview of cell is nil somehow.
As I understand, FlowKit handled container property in InternalContext from the start of cell initialisation and passed it to Event separately, thus we had cell and table in the Event. I can't find commit about this change and its purpose and however it dramaticaly affects current height calculation, cause rowHeight callback does not receives parent tableView through cell or adapter and there is no other way to access to it.
Any suggestions?


Add a dedicated NetavarkError type

Subsequently, plumb this error through most of Netavark. The interface creation logic is still mostly on std::io::Error, but that natively converts into NetavarkError via From so I'm not touching that right now.
Still to do: error equality, and possibly dedicated types for iptables and firewalld so we can better track where in the code errors occur.
Also add a NetavarkResult and convert most of our code to use it.

I'll ask around to see if there's a better way, but if the answer is macros, I'll probably leave the todos - I still don't fully grok the way Rust macros are defined.

Dropping WIP

I think this will go green now. Accidentally removed a ! in the iptables logic.

LGTM but I would like to change all the network errors too, there is no reason why they should be io errors.

Yes looks like they are showing up as IO error as of now. Maybe a new type for them in a folloup up PR ?

/lgtm


⚠ Bofrost.at has degraded performance

In eed8bc0, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1239 ms

Resolved: Bofrost.at performance has improved in 0410fb0.


Build OpenThread code base for Host system.

A host system, one that communicates to an OpenThread NCP over a bus (UART, SPI), may have need for a modest portion of the OpenThread libraries.  These would include various header files {openthread-types.h, spinel.h, hdlc.hpp}  along with libraries that can be used to unpack spinel frames {spinel.c, hdlc.cpp}.
The request would be to configure Openthread to build for this alternative environment.  In the future one might add additional modules that would further ease the porting/development effort of a host system.

Isn't this what we do for Windows already? :)

I don't know is it?  how do you configure OpenThread for Windows?

All the VS project files build OpenThread into a Window Driver Static Library. That is then built into the otLwf.sys Windows Filter Driver. You can find more info here.

Yes, it does appear that you have achieved the stated goal for windows.  I would like to achieve the same goal for other platforms as well.

Is your goal to completely duplicate the code we have for Windows on other platforms? What platforms specifically?

We have linux based and RTOS based platforms which act as hosts to ncp devices here at Nest.  These platforms would benefit from the ability to build the small portion of OpenThread for a host based system.  That is spinel.c, hdlc.cpp and perhaps some future ncp_host.cpp to compliment ncp_base.cpp.


Create google.yml

Why:

What's being changed:

Check off the following:

[ ] I have reviewed my changes in staging. (look for the deploy-to-heroku link in your pull request, then click View deployment)
[ ] For content changes, I have reviewed the localization checklist
[ ] For content changes, I have reviewed the Content style guide for GitHub Docs.

We don't need a google.yml. I'm going to close this.


Wrong decompiler output

Binary Ninja Version: 2.1.2263 Personal, 4d2bf25a
Platform: Fedora 31 (Workstation Edition)
http://static.3k.ctf.to/pwn/9673cbce50ea53b390bbddf4f7f1850c7a346dc8.zip
Wrong decompiler output in graph mode in function notepad, specifically in Case 4. Disassembly output works well.

Thanks for the report! Though I do not see how it is wrong. Could please explain a little bit?

Sorry if I wasn't clear. In the Case 0x4, disassembler shows correctly, that there is a jump back to the block with login call (this also happens when calling the binary). The decompiler omits that block and jumps straight to the while loop.

 :-)

Fixed in build 2293


feat req: Interactive config generator

It would be nice for Wag to have a CLI command to generate a configuration file interactively.
For example
What is the public address your devices will connect to? <IP_ADDRESS>:4321
What help email would you like displayed on the auth page<EMAIL_ADDRESS>.. etc ..

Initially, I thought the same. I would be nice to regenerate a config file at any point. Then I realised that the only point at which the user private key is available is at the point of registration. Once they collect the config during the registration process, it's gone from the server.
This makes sense as it keeps the keys separate and prevents even admins from having access to both of the users keys.
That said, I carry out the user registrations and send the users the config myself. I have a repository of keys (but not on the wag server). I also modify the config to user /24 addresses in the AllowedIps= rather than lots of individual IP's. That way, I can update an ACL to add a user to allow them access to another system without them having to edit their config.

Hi @paulb-opusvl,
This issue is specifically opened for generating the server side config rather than the results of the client registration.
Little bit confused by what you mean about modifying the client config to use a /24, you should be able to do this in the Acls section by Allow-ing or Mfa-ing a subnet such as:
 "group:nerds": {
                "Mfa": [
                    "<IP_ADDRESS>/24"
                ],
                "Allow": [
                    "<IP_ADDRESS>/32"
                ]
            },

Which would allow access to the <IP_ADDRESS>/24 network when Mfa'd. I.e hopefully you can just use the registration process to do that for you, so no manual editing required!
To tangent off this tangent, I've been thinking about your request for centralised management of wag, what does that look like in your dream world?

My bad, I should learn to read without diving in.
I have lots of acls that have only /32, and I don't want all clients to access everything. If I put /24 in the AllowedIPs= I don't have to change it if I add another /32 in the ACL. The client will already have the route, but not the access - until I add them to the ACL.
A management GUI would be very nice. We have relatively few users and their configs can be managed with Ansible. I'm not necessarily uncomfortable with .json, but others would benefit from having a simple management interface.


[BUG] Request received log not logged

I have been using nestjs pino in my application and I the log for request received is not printed automatically when the request is received on a route.I have even implemented the customReceivedMessage function but the log is still not getting printed. Anything Im missing?

closing

@iamolegga Any reason for closing this issue?

there is a link in . Don't delete the template for bug report, it's impossible to check such bug description


DROPDOWN - Issues around focus border

Please verify the version of auro-form you have installed
1.5.0
Please describe the bug

Open dropdown locally
Navigate to API page
Click on basic dropdown
See error

Reproducing the error
This issue is reproducible on the Auro docsite
Expected behavior
There should be a focus border around the trigger when the dropdown or its children have focus
What browsers are you seeing the problem on?
Chrome
Additional context
There also seems to be issues around the error use cases. We should be seeing the same 2px blue border when the trigger has focus, but we only see 1px borders or no change at all:

Exit criteria
All border colors are applied in correct scenarios

focus in error state: https://alaskaair-my.sharepoint.com/:v:/p/michelle_noelke/EY-edr3aV_pOio5zs6prRQABiFkqtv--Ar4kPHRx9dS9BQ?e=tqp6uD


add CITATION.cff

This is as a first draft. I am not sure about a few things.
I've seen different approaches to the citation/authors some of them have thwo section, which almost always are the same:
authors:
- family-names: "A"
  given-names: "B"
...
preferred-citation:
  authors:
    - family-names: "A"
      given-names: "B"

Github generates a readable citation and a BibTex from this looking like this:
Demuzere, M., Argüeso, D., Zonato, A., & Kitnner, J. (2021). A WUDAPT-to-WRF python tool that injects WUDAPT's Local Climate Zone information into WRF. (Version v0.1.1) [Computer software]. https://github.com/matthiasdemuzere/w2w

@software{Demuzere_A_WUDAPT-to-WRF_python_2021,
author = {Demuzere, Mattias and Argüeso, Daniel and Zonato, Andrea and Kitnner, Jonas},
license = {MIT},
month = {11},
title = {{A WUDAPT-to-WRF python tool that injects WUDAPT's Local Climate Zone information into WRF.}},
url = {https://github.com/matthiasdemuzere/w2w},
version = {v0.1.1},
year = {2021}
}

also is the PyPi URL relevant in any kind?
the title here does not 100% match the Paper's title
there is url and repository-code not sure if they should be the same or one of them is supposed to point to PyPi, the other to github

https://github.com/citation-file-format/citation-file-format/blob/main/schema-guide.md#url
https://github.com/citation-file-format/citation-file-format/blob/main/schema-guide.md#repository-code
Happy to hear your opinions and interpretation on this.

Hi Jonas,
My two cents, but perhaps Matthias can confirm.
I think the citation looks fine as it is above.
I don't think the PyPi URL must be included here.
I prefer the title in the bibtex output you show, but have no strong opinion here.
cheers,
Dani

I think it is up to us to decide, so this is what I suggest:

authors: keep as is
title: use the JOSS one: W2W: A Python package that injects WUDAPT’s Local Climate Zone information in WRF
url: https://www.wudapt.org/wudapt-to-wrf/, since we consider the wudapt website as a hub to re-direct users to all available data and tools.
repository-code: https://github.com/matthiasdemuzere/w2w

Can you adjust accordingly @theendlessriver13, if you agree, and merge PR?

Note: I'll adjust the wudapt page then to make sure all is consistent!

Sounds good to me.

@theendlessriver13: I now see I also made a mistake in your name: Kitnner -> Kittner ... oops. Sorry!

already fixed :) -- should be ready to go now I guess?


Add LCAD-LBTC market + remove mac script

Add LCAD-LBTC market in addition to USDt-LBTC.
Remove Mac script since the same script now works on all platforms.
Please review @tiero

you need now to update the integration test to support the new fixture model


pico: include config in led.cpp

Otherwise the LED is never going to get enabled (broken by #827 as this file still compiles without its #defines, it just does nothing)

Me, merge a broken PR? noooooo :laughing:
Thank you!


Add products taxonomy to improve findability

Adds docs products taxonomy to the repo to improve findability experiences (onsite search, recommendations and others)

Product Label
Products

SharePoint
https://authoring-docs-microsoft.poolparty.biz/devrel/9d7be3ef-f27c-4c7f-9eba-67c3cd429995

Docs Build status updates of commit 49ff25b:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

SP-Framework/docfx.json
:bulb:Suggestion

Details

SP-Framework/docfx.json

Line 48, Column 31: [Suggestion-ms-author-invalid] Invalid value for 'ms.author', 'spdevdocs' is not a valid Microsoft alias.

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Race condition in resync when a deregister occurs?

What version of docker are you running?

root@ip-10-202-10-253:~# docker version
Client:
 Version:      18.05.0-ce
...
Server:
 Engine:
  Version:      18.05.0-ce

What version of registrator are you running?

2018/05/18 06:33:06 Starting registrator v7 ...
2018/05/18 06:33:06 Forcing host IP to <IP_ADDRESS>
2018/05/18 06:33:06 Using consul adapter: consul://<IP_ADDRESS>:8500

Did you build a custom version of registrator? If so, what is that image?

Using gliderlabs/registrator:v7 with a bash wrapper to start it, the relevant parts at the bottom being:
# Pass on SIGTERM to registrator when received. Sleep 2 seconds after the SIGTERM to allow time
# to perform any cleanups.
trap 'echo "SIGTERM received, initiate graceful shutdown."; kill -TERM $PID; sleep 2' SIGTERM

full_command="/bin/registrator -retry-attempts 5 -retry-interval 1000 -resync 30 -ip ${get_private_ip} -tags 'az-${az_suffix}' consul://${consul_ip}:8500"
echo "$(date) : Registrator Start Command - ${full_command}"
eval "${full_command}"
PID=$!
wait $PID

What is the exact command you are running registrator with?

See above

What is the exact command you are running your container with?

/registrator_start.sh, the bash wrapper.

A log capture of all the docker events before, during, and after the issue

Tried to get them but they seem to be rotated out unfortunately :(

If relevant, Dockerfile for application that is having issues.

Description of the problem:

It seems like the resync behaviour is racy, and possibly needs a mutex/lock between unregister events + resync events?
Looking at the consul logs, it looks as though a deregister occurred, but then a resync occurred and readded a recently deregistered service instance, within the same second?
This then leads to a stale/orphaned entry with a zero ServicePort which is never cleaned up, including on a resync

How reproducible:

Rare, feels racy. Hit it 4~ times in the past week (that we know of), with low hundred services and high hundreds of containers across 4 clusters.

Steps to Reproduce:

Run a ton of services, use resync, shorter interval = more regular = more occurrences?

Actual Results:
Note the Synced response after the Deregistered in this (grepped by a088d8cbf79df4f9d001):
    2018/06/12 13:04:06 [INFO] agent: Synced service "0f3bbd5258ff:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-a088d8cbf79df4f9d001:80"
    2018/06/12 13:04:36 [INFO] agent: Synced service "0f3bbd5258ff:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-a088d8cbf79df4f9d001:80"
    2018/06/12 13:05:06 [INFO] agent: Synced service "0f3bbd5258ff:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-a088d8cbf79df4f9d001:80"
    2018/06/12 13:05:36 [INFO] agent: Deregistered service "0f3bbd5258ff:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-a088d8cbf79df4f9d001:80"
    2018/06/12 13:05:36 [INFO] agent: Synced service "0f3bbd5258ff:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-a088d8cbf79df4f9d001:80"

And the consul service list for the IP in question. The first result is the stale entry (note no ServicePort), the second is a valid replacement service that was spawned by ECS during a rolling deploy, and is correctly registered:
  {
    "ID": "1fee1507-19a7-0d06-e41c-31d32167b999",
    "Node": "a68f43f934db",
    "Address": "<IP_ADDRESS>",
    "Datacenter": "aws-us-e1-production",
    "TaggedAddresses": {
      "lan": "<IP_ADDRESS>",
      "wan": "<IP_ADDRESS>"
    },
    "NodeMeta": {
      "consul-network-segment": ""
    },
    "ServiceID": "0f3bbd5258ff:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-a088d8cbf79df4f9d001:80",
    "ServiceName": "recommendation-engine-api-node",
    "ServiceTags": [
      "web-backend",
      "az-1d"
    ],
    "ServiceAddress": "<IP_ADDRESS>",
    "ServicePort": 0,
    "ServiceEnableTagOverride": false,
    "CreateIndex": 31357420,
    "ModifyIndex": 31357420
  },
  {
    "ID": "1fee1507-19a7-0d06-e41c-31d32167b999",
    "Node": "a68f43f934db",
    "Address": "<IP_ADDRESS>",
    "Datacenter": "aws-us-e1-production",
    "TaggedAddresses": {
      "lan": "<IP_ADDRESS>",
      "wan": "<IP_ADDRESS>"
    },
    "NodeMeta": {
      "consul-network-segment": ""
    },
    "ServiceID": "0f3bbd5258ff:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-f6e7fbc9dacee7db0200:80",
    "ServiceName": "recommendation-engine-api-node",
    "ServiceTags": [
      "web-backend",
      "az-1d"
    ],
    "ServiceAddress": "<IP_ADDRESS>",
    "ServicePort": 32929,
    "ServiceEnableTagOverride": false,
    "CreateIndex": 31429652,
    "ModifyIndex": 31429652
  },

Expected Results:

Let the service deregister properly.

Confirmed same consul log output on a second instance (a day earlier):
    2018/06/11 18:58:34 [INFO] agent: Synced service "38b2f9af21ea:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-acc5bbcebdbaeadad301:80"
    2018/06/11 18:59:04 [INFO] agent: Synced service "38b2f9af21ea:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-acc5bbcebdbaeadad301:80"
    2018/06/11 18:59:34 [INFO] agent: Synced service "38b2f9af21ea:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-acc5bbcebdbaeadad301:80"
    2018/06/11 19:00:04 [INFO] agent: Deregistered service "38b2f9af21ea:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-acc5bbcebdbaeadad301:80"
    2018/06/11 19:00:04 [INFO] agent: Synced service "38b2f9af21ea:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-acc5bbcebdbaeadad301:80"

Third instance, an hour ago:
    2018/06/12 20:07:31 [INFO] agent: Synced service "b371b03b211c:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-8ac7b7cde09ef3fe2800:80"
    2018/06/12 20:08:01 [INFO] agent: Synced service "b371b03b211c:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-8ac7b7cde09ef3fe2800:80"
    2018/06/12 20:08:31 [INFO] agent: Deregistered service "b371b03b211c:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-8ac7b7cde09ef3fe2800:80"
    2018/06/12 20:08:31 [INFO] agent: Synced service "b371b03b211c:ecs-recommendation-engine-api-node-production-16-recommendation-engine-api-node-nginx-8ac7b7cde09ef3fe2800:80"

Pattern: all 3 Deregister + Synced operations are on the same second, backing up my theory re the race condition. Running in parallel goroutines most likely?

Hit this again today...

Note: we now use a 300sec resync interval, and still managed to hit it.

Do you happen to have a PR to fix this? I don't have a bunch of time to investigate/write a fix, but would be happy to review/merge something :)

Do you happen to have a PR to fix this? I don't have a bunch of time to investigate/write a fix, but would be happy to review/merge something :)

I don't as yet, it's on my list to try and find time to a) write a test that repro's it and b) fix the underlying issue.

Note: doing further testing on master, as I realised v7 is quite old and a lot of new fixes have been applied. Will report back once I have any definitive results.


Arrays with both minItems and default misbehaves

When providing both default (as a list)  and minItems to a field of type array
the defaults are treated as a single value that ends up in each field.
Steps to Reproduce
example
{
  "type": "object",
  "properties": {
    "listOfStrings": {
      "type": "array",
      "title": "A few strings, weird when minItems is provided",
      "items": {
        "type": "string"
      },
      "default": [
        "hello",
        "world"
      ],
      "minItems": 3
    }
  }
}

(try removing either minItems or default to see expected behavior)
Expected behavior
defaults should be treated as a list and be given one entry each, i.e.:
When minItems > default.length provide empty fields after the fields with the default values.
When minItems <= default.length provide one field per value in default list.
Actual behavior
The default items gets mashed together into each field
Version
0.49.0

#579 would solve this.

I think this can be closed now, with #663 merged.

Yeap, thanks!


Getting error "Can't find variable: exports" in WebUI after updating to 0.7.2

After updating to the latest version, I'm getting the following JS console error when opening up a WebUI view:
ReferenceError: Can't find variable: exports

I've gone through a few of the previous commits, and the last commit that successfully compiles is be946f4f656cb83c9e6387978b1ca40670d292af.
Should I change how my Resource files are getting compiled? Would appreciate some direction on this so I can use the latest code.

Should be fixed with 0.7.3. Can you confirm?

It seems that's done the trick after reinstalling all npm dependencies.
Many thanks!


Multilanguage support

Hi,
The docker version filebot cannot support Chinese properly. Is it possible to develop the multilanguage function in this project?

What is the issue exactly ?  Text is not displaying correctly ?
You can try to set the variable ENABLE_CJK_FONT to 1.
https://github.com/jlesage/docker-filebot#environment-variables

Closing issue.  Please re-open if needed.


Add the method predict() for all models

What does this PR do?

Fixing #198
Before submitting

[x] This PR is made to fix a typo or improve the docs (you can dismiss the other checks if this is the case).
[x] Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have written necessary tests and already run them locally.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

214 of 224 (95.54%)  changed or added relevant lines in 29 files are covered.
8 unchanged lines in 4 files lost coverage.
Overall coverage increased (+0.7%) to 84.547%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pypots/classification/base.py
2
3
66.67%

pypots/clustering/base.py
2
3
66.67%

pypots/clustering/crli/model.py
11
12
91.67%

pypots/clustering/vader/model.py
11
12
91.67%

pypots/forecasting/base.py
2
3
66.67%

pypots/imputation/base.py
2
3
66.67%

pypots/base.py
18
22
81.82%

Files with Coverage Reduction
New Missed Lines
%

pypots/classification/base.py
2
78.72%

pypots/clustering/base.py
2
38.37%

pypots/forecasting/base.py
2
35.16%

pypots/imputation/base.py
2
80.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.7%

Covered Lines:
3923

Relevant Lines:
4640

💛  - Coveralls


Move deployment logic from travis to github actions

Just a reminder for us. The github actions PR only moved the tests but not the deployment so this is atm still handled by travis.

I see release.yaml workflow, and I don't see any travis config. Is this issue still relevant?

Looks like it can be closed: https://github.com/gnosis/ethcontract-rs/actions/workflows/release.yaml

We have successful deployments with it already.


Not getting any response from fortigate using fortios drtiver

In [1]: from napalm import get_network_driver
In [2]: driver = get_network_driver('fortios')
In [3]: device = driver('<IP_ADDRESS>', 'admin', None)
In [4]: device.open()

But ssh to the device works just fine
$ ssh -l admin <IP_ADDRESS> get sys status
FGVM010000054408 # Version: FortiGate-VM64 v5.2.6,build0711,160129 (GA)
Virus-DB: 16.00560(2012-10-19 08:31)
Extended DB: 1.00000(2012-10-17 15:46)
IPS-DB: 5.00555(2014-10-07 01:21)
IPS-ETDB: 0.00000(2001-01-01 00:00)
Serial-Number: FGVM010000012345
Botnet DB: 1.00000(2012-05-28 22:51)
License Status: Valid
VM Resources: 1 CPU/1 allowed, 970 MB RAM/2048 MB allowed
BIOS version: 04000002
Log hard disk: Available
Hostname: FGVM010000012345
Operation Mode: NAT
Current virtual domain: root
Max number of virtual domains: 10
Virtual domains status: 1 in NAT mode, 0 in TP mode
Virtual domain configuration: disable
FIPS-CC mode: disable
Current HA mode: standalone
Branch point: 711
Release Version Information: GA
FortiOS x86-64: Yes
System time: Tue Sep  6 09:19:23 2016

Could you paste the traceback?

I was able to replicate this issue. Here's my traceback:
`>>> from napalm import get_network_driver

driver = get_network_driver('fortios')
device = driver('<IP_ADDRESS>', 'admin', '')
device.open()
Traceback (most recent call last):
File "", line 1, in
File "/Users/ebeahan/Envs/napalm/lib/python2.7/site-packages/napalm_fortios/fortios.py", line 37, in open
self.device.open()
File "/Users/ebeahan/Envs/napalm/lib/python2.7/site-packages/pyFG/fortios.py", line 94, in open
self.ssh.connect(**cfg)
File "/Users/ebeahan/Envs/napalm/lib/python2.7/site-packages/paramiko/client.py", line 338, in connect
t.start_client()
File "/Users/ebeahan/Envs/napalm/lib/python2.7/site-packages/paramiko/transport.py", line 493, in start_client
raise e
EOFError`

Digging into this a bit, I found a paramiko issue describing this exact issue witih Fortinet:
https://github.com/paramiko/paramiko/issues/687
After following the listed workaround of editing the preferred key exchange algorithms in transport.py, the issue was resolved for me.
`>>> from napalm import get_network_driver

driver = get_network_driver('fortios')
device = driver('<IP_ADDRESS>', 'admin', '')
device.open()
`

I can check if the issue is resolved in a more recent version of FortiOS (was also testing with v5.26), but it appears the bug lies on the Fortinet side not napalm or paramiko.

Great, marking this as invalid as it's a bug somewhere else.
For reference, I am adding a link here with the comment mentioning the workaround you found: https://github.com/paramiko/paramiko/issues/687#issuecomment-196577317
I might also try to take a look to paramiko and see if there is some patch we can provide to fix the issue but I am afraid I don't have access to any fg device right now : (
In the meantime, if would be nice to add a reference here: https://github.com/napalm-automation/napalm/blob/master/docs/support/fortios.rst
You seem to know better which version seem to be affected so feel free to add as many details as you want about the issue there and how to work around it. Otherewise I will try to add them myself later next week.
Thanks!

Appears to affect FortiOS v5.2.x and beyond (tested up to 5.4.0 - 5.4.1 VM image is throwing fits but will try to test against that version as well). Will work on submitting a PR to updating the docs noting issue and workaround this weekend.

Great! Thanks : )

I am closing this one, feel free to reopen if it's still an issue.


docs and example updated

Coverage remained the same at 92.872% when pulling 23457f3162d491d9d343e8924a206bbc1e446834 on development into 1d4150d8b81f27e0c73136f675191b3ca19f666a on master.


Problems with a parameter checkpoint after training llama2-7b

I’ve trained llama2-7b model with int8 quantization. The resulting checkpoints dir has the following structure:
- 0/:
-- commit_success.txt

-- default/:
--- _METADATA
--- _sharding
--- checkpoint
--- commit_success.txt
--- d/:
---- cacfeed1cb58f5...
---- ed63ba6da8ada7...
--- manifest.ocdbt
--- ocdbt.process_0/:
---- d/:
----- 028478f3dcd5b9...
----- 551917150ca2a8...
----- 8111cad0633e85...
---- manifest.ocdbt
--- ocdbt.process_1/
--- ...

-- metrics/

-10000/

-20000/

- ...

Following this example, I’m trying to generate a parameter checkpoint for further decoding:
export CHECKPOINT_PATH=gs://.../checkpoints/10000/default
export PARAMETER_CHECKPOINT_RUN=<MY_RUN_NAME>
python3 MaxText/generate_param_only_checkpoint.py MaxText/configs/base.yml load_full_state_path=$CHECKPOINT_PATH run_name=${PARAMETER_CHECKPOINT_RUN} model_name='llama2-7b' force_unroll=true

I get this output:
...
I0214 21:09:09.466111<PHONE_NUMBER>61856 checkpointer.py:167] Finished restoring checkpoint from gs://.../.../.../.../checkpoints/10000/default.
In input checkpoint Number of model params=6.738 billion
Save decode checkpoint at: gs://.../.../.../param_checkpoint/checkpoints/
I0214 21:09:10.367621<PHONE_NUMBER>61856 async_checkpointer.py:210] Commit thread error check finished successfully
I0214 21:09:12.452971<PHONE_NUMBER>61856 async_checkpointer.py:293] Async saving item to gs://.../.../.../param_checkpoint/checkpoints/0.
W0214 21:09:13.010293<PHONE_NUMBER>61856 type_handlers.py:399] SaveArgs.aggregate is deprecated, please use custom TypeHandler (https://orbax.readthedocs.io/en/latest/custom_handlers.html#typehandler) or contact Orbax team to migrate before May 1st, 2024.
I0214 21:09:23.297378<PHONE_NUMBER>66688 async_checkpointer.py:150] Starting commit to storage layer by process: 0
I0214 21:09:23.300883<PHONE_NUMBER>61856 checkpoint_manager.py:811] Beginning async checkpoint finalize.
saved an decode checkpoint at gs://.../.../.../param_checkpoint/checkpoints/
Successfully generated decode checkpoint at: gs://.../.../.../param_checkpoint/checkpoints/0/default
I0214 21:10:41.916716<PHONE_NUMBER>66688 async_checkpointer.py:158] Finished committing to storage layer by process: 0
I0214 21:10:41.917720<PHONE_NUMBER>69376 async_checkpointer.py:207] Commit thread joined successfully

The resulting dir gs://.../.../.../param_checkpoint/checkpoints/0/default with a parameter checkpoint has the following structure:
- _METADATA
- _sharding
- checkpoint
- ocdbt.process_0/:
-- d/:
--- 14159648ab13db...
--- 6bb79329658hjl...
--- caa39c6cfcajsk...
--- d852816bl798dj...
-- manifest.ocdbt

Following this example further, I try to load a parameter checkpoint into decode.py:
python3 MaxText/decode.py MaxText/configs/base.yml load_parameters_path=gs://.../.../.../param_checkpoint/checkpoints/0/default run_name=<MY_RUN_NAME> per_device_batch_size=1 model_name='llama2-7b' ici_tensor_parallelism=4 max_prefill_predict_length=4  max_target_length=16 prompt="I love to" autoregressive_decode_assert="read. I love to write. I love to share." attention=dot_product scan_layers=false

I get this error:
restoring params from load_parameters_from_path='gs://.../.../.../param_checkpoint/checkpoints/0/default'
Traceback (most recent call last):
  File "/home/.../2024-02-14-19-59-41/MaxText/decode.py", line 278, in <module>
    app.run(main)
  File "/home/.../.local/lib/python3.10/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/.../.local/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/home/.../2024-02-14-19-59-41/MaxText/decode.py", line 275, in main
    decode_loop(pyconfig.config)
  File "/home/.../2024-02-14-19-59-41/MaxText/decode.py", line 188, in decode_loop
    state, state_mesh_annotations = max_utils.setup_decode_state(
  File "/home/.../2024-02-14-19-59-41/MaxText/max_utils.py", line 334, in setup_decode_state
    return setup_initial_state(model, None, config, rng, mesh, checkpoint_manager, is_training)
  File "/home/.../2024-02-14-19-59-41/MaxText/max_utils.py", line 362, in setup_initial_state
    state, raw_params = checkpointing.load_state_if_possible(checkpoint_manager,
  File "/home/.../2024-02-14-19-59-41/MaxText/checkpointing.py", line 116, in load_state_if_possible
    full_restored_state = checkpointer.restore(p, item = abstract_param_train_state,\
  File "/home/.../.local/lib/python3.10/site-packages/orbax/checkpoint/checkpointer.py", line 163, in restore
    raise ValueError(f'Found incomplete checkpoint at {directory}.')
ValueError: Found incomplete checkpoint at gs://.../.../.../param_checkpoint/checkpoints/0/default.

Diving into the code, I realize that this error appears when there is no commit_success.txt file in the checkpoint dir. This file hasn’t been generated automatically by generate_param_only_checkpoint.py, despite the status Successfully generated decode checkpoint . I tried to add this file manually — I got this error:
I0216 11:10:42.486244<PHONE_NUMBER>28960 checkpointer.py:164] Restoring item from gs://.../.../param_checkpoint/checkpoints/0/default.
Traceback (most recent call last):
  File "/home/.../2024-02-14-19-59-41/MaxText/decode.py", line 278, in <module>
    app.run(main)
  File "/home/.../.local/lib/python3.10/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/.../.local/lib/python3.10/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/home/.../2024-02-14-19-59-41/MaxText/decode.py", line 275, in main
    decode_loop(pyconfig.config)
  File "/home/.../2024-02-14-19-59-41/MaxText/decode.py", line 188, in decode_loop
    state, state_mesh_annotations = max_utils.setup_decode_state(
  File "/home/.../2024-02-14-19-59-41/MaxText/max_utils.py", line 334, in setup_decode_state
    return setup_initial_state(model, None, config, rng, mesh, checkpoint_manager, is_training)
  File "/home/.../2024-02-14-19-59-41/MaxText/max_utils.py", line 362, in setup_initial_state
    state, raw_params = checkpointing.load_state_if_possible(checkpoint_manager,
  File "/home/.../2024-02-14-19-59-41/MaxText/checkpointing.py", line 116, in load_state_if_possible
    full_restored_state = checkpointer.restore(p, item = abstract_param_train_state,\
  File "/home/.../.local/lib/python3.10/site-packages/orbax/checkpoint/checkpointer.py", line 166, in restore
    restored = self._handler.restore(directory, args=ckpt_args)
  File "/home/.../.local/lib/python3.10/site-packages/orbax/checkpoint/pytree_checkpoint_handler.py", line 1073, in restore
    restored_item = asyncio.run(
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/.../.local/lib/python3.10/site-packages/orbax/checkpoint/pytree_checkpoint_handler.py", line 903, in _maybe_deserialize
    deserialized_batches += await asyncio.gather(*deserialized_batches_ops)
  File "/home/.../.local/lib/python3.10/site-packages/orbax/checkpoint/type_handlers.py", line 1179, in deserialize
    results = await super().deserialize(infos, args)
  File "/home/.../.local/lib/python3.10/site-packages/orbax/checkpoint/type_handlers.py", line 1117, in deserialize
    await _assert_parameter_files_exist(
  File "/home/.../.local/lib/python3.10/site-packages/orbax/checkpoint/type_handlers.py", line 260, in _assert_parameter_files_exist
    raise FileNotFoundError(
FileNotFoundError: Individual parameter subdirectory not found at path: gs://.../.../param_checkpoint/checkpoints/0/default/step

Thus, it’s trying to find gs://.../.../param_checkpoint/checkpoints/0/default/step  dir, whereas there is no such.
Passing the initial checkpoint path gs://.../checkpoints/10000/default to the decode.py did not lead to the desired result:
ValueError: Dict key mismatch; expected keys: ['decoder_norm', 'layers', 'logits_dense']; dict: {'decoder_norm': {'scale': ArrayRestoreArgs(restore_type=None, dtype=dtype('float32'), mesh=Mesh(device_ids=array([[[[[0],
          [2],
          [1],
          [3]]]]]), axis_names=('data', 'fsdp', 'sequence', 'tensor', 'autoregressive')), mesh_axes=PartitionSpec(('fsdp', 'sequence'),), sharding=None, global_shape=None)}, 'layers_0': {'mlp': {'wi_0': {'kernel': ArrayRestoreArgs(restore_type=None, dtype=dtype('float32'), mesh=Mesh(device_ids=array([[[[[0],
          [2],
          [1],
          [3]]]]]), axis_names=('data', 'fsdp', 'sequence', 'tensor', 'autoregressive')), mesh_axes=PartitionSpec(('fsdp', 'sequence'), ('tensor', 'autoregressive')), sharding=None, global_shape=None)}, 'wi_1': {'kernel': ArrayRestoreArgs(restore_type=None, dtype=dtype('float32'), mesh=Mesh(device_ids=array([[[[[0],
...

I also tried to train a model async_checkpointing: False , thinking that there could be a problem with asynchronous checkpointing, but got the same result. What is the problem? Should I pass the number of training steps of the checkpoint?
I'd be grateful for any advice on how to resolve the issue.

This was fixed by an upgraded MaxText.


Bump up minimum android version to 9

Already there as https://github.com/igniterealtime/jbosh/commit/8370db39eb74e04144a6bafb93fa222dfa92c21e


[WARNING RSZ-0075] makeBufferedNet failed for driver core/rob/core/rob/99958/CON

Describe the bug
(can also be reproduced with make DESIGN_CONFIG=designs/asap7/aes/config.mk floorplan)
untar https://drive.google.com/file/d/1Yhg6RdbrmuNTAFz9-th3IIC2ZCWkrhNi/view?usp=sharing
./run-me-BoomTile-asap7-2.sh

OpenROAD v2.0-16664-g9ba039016
[deleted]
Running floorplan.tcl, stage 2_1_floorplan
[INFO ODB-0227] LEF file: /home/oyvind/OpenROAD-flow-scripts/flow/platforms/asap7/lef/asap7_tech_1x_201209.lef, created 24 layers, 9 vias
[INFO ODB-0227] LEF file: /home/oyvind/OpenROAD-flow-scripts/flow/platforms/asap7/lef/asap7sc7p5t_28_R_1x_220121a.lef, created 212 library cells
[INFO ODB-0227] LEF file: results/asap7/array_256x128/base/array_256x128.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/btb_128x56/base/btb_128x56.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/dataArrayB_256x64/base/dataArrayB_256x64.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/data_2048x8/base/data_2048x8.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/ebtb_128x40/base/ebtb_128x40.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/lb_32x128/base/lb_32x128.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/mem_256x4/base/mem_256x4.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/meta_128x120/base/meta_128x120.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/meta_40x240/base/meta_40x240.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/sdq_17x64/base/sdq_17x64.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/table_128x44/base/table_128x44.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/table_128x52/base/table_128x52.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/table_256x48/base/table_256x48.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/tag_array_64x168/base/tag_array_64x168.lef, created 5 vias, 1 library cells
[INFO ODB-0227] LEF file: results/asap7/tag_array_64x184/base/tag_array_64x184.lef, created 5 vias, 1 library cells

==========================================================================
Floorplan check_setup
--------------------------------------------------------------------------
Warning: There are 148 input ports missing set_input_delay.
Warning: There are 238 output ports missing set_output_delay.
Warning: There are 684 unconstrained endpoints.
number instances in verilog is 2043016
[WARNING IFP-0028] Core area lower left (2.000, 2.000) snapped to (2.052, 2.160).
[INFO IFP-0001] Added 5540 rows of 27702 site asap7sc7p5t.
repair_timing -verbose -repair_tns 0 -skip_last_gasp
[WARNING RSZ-0021] no estimated parasitics. Using wire load models.
[INFO RSZ-0094] Found 165887 endpoints with setup violations.
[INFO RSZ-0099] Repairing 1 out of 165887 (0.00%) violating endpoints...
   Iter   | Removed | Resized | Inserted | Cloned |  Pin  |    WNS   |   TNS      |  Viol  | Worst
          | Buffers |  Gates  | Buffers  |  Gates | Swaps |          |            | Endpts | Endpt
---------------------------------------------------------------------------------------------------
        0 |       0 |       0 |        0 |      0 |     0 | -2195.436 | -114376600.0 | 165887 | core/int_issue_unit/core/int_issue_unit/io_dis_uops_0_ready_REG$_DFF_P_/D
       10 |       0 |       7 |        0 |      1 |     1 | -1346.351 | -80619768.0 | 165887 | core/int_issue_unit/core/int_issue_unit/io_dis_uops_0_ready_REG$_DFF_P_/D
[WARNING RSZ-0075] makeBufferedNet failed for driver core/rob/core/rob/_99958_/CON
[WARNING RSZ-0075] makeBufferedNet failed for driver core/rob/core/rob/_99958_/CON
[WARNING RSZ-0075] makeBufferedNet failed for driver core/rob/core/rob/_99958_/CON
[deleted]

Expected Behavior
No warning
Environment
OpenROAD v2.0-16664-g9ba039016

To Reproduce
See above
Relevant log output
No response
Screenshots
No response
Additional Context
No response

@maliberty Is this just a warning that should be fixed or does it have any detrimental effect relevant to megaboom?

Mostly a warning.  In short it is saying it can't insert buffers into nets pre-placement.  Eventually that will be good to fix but for now we will just disable the attempt and therefor the warning.


not crawlable by google

Hi guys,
I deployed my app with mupx.
I've got the default settings in the mup.json, having also the
"setupPhantom": true
I also installed the spiderable package to the app, to make it crawlable by google.
While with the mup deployment google could read my page entirely, with the mupx deployment google just reads the link, and cannot give back any of the meta tags I have on the webpage.
I can't deploy my app anymore with the mup tecnhique, because it gives me a lot of troubles and errors during the deployment.
What could I do to make it work again, since I really need the SEO working on the website?

I'm not sure (as usual) but have a thought to explore.
Is it possible that with SSL turned on and using mupx it would become SEO enabled again?
Because it runs from a different container aka server maybe?

got mup working with this:
https://github.com/arunoda/meteor-up/issues/721
updating nodejs for meteor 1.2
so will try to see what google has to say now :D


Keycloak Integration

+1 !

+1

This was included in the alpha4 release.
myDeployment.as(Secured.class)
will find a keycloak.json from your classpath and setup keycloak (and munge up a web.xml with some security-contraints).
You can further call combinations of .protect("/path/to/whatever").withMethod("GET").withRoles("admin")
Might need additional improvement, but it works.


RLS config does not work for bool options

#2332 does not work for bool options, either with 'true' or 1. It works with LanguageClient-neovim which uses a settings.json.
let g:ale_rust_rls_config = {'rust': {'clippy_preference': 'on', 'all_features': 'true'}}

let g:ale_rust_rls_config = {'rust': {'clippy_preference': 'on', 'all_features': 1}}

Use v:true for Boolean values in Vim. See :help v:true.

Wow, didn‘t know—sorry for issue!
(ALE is by far the most ergonomic and well thought-out LSP plugin around. Thank you for your work!)


russia-blacklist.txt misspells Exhentai's domain

Operating system / операционная система
Windows 10
Running as service / Запуск программы как сервис
I installed it as a service / Установил как сервис Windows
Describe the bug / Опишите ошибку программы
The russia-blacklist.txt file mistakenly uses "exhentai.me" instead of "exhentai.org". I can only access the site with GoodbyeDPI after editing the entry in that file to the latter.
Additional information / Дополнительная информация
No response

There's no such entry in the blacklist. Extentai.org is not blocked in Russia.

There's no such entry in the blacklist. Extentai.org is not blocked in Russia.

Sir, you just open the file and search "exhentai.me" to find the misspelled entry; it's even in the latest version that I can download. I don't know what else to tell you.
PS: I'm not from Russia or even that close to it (culturally or geographically), but much of the sites blocked there are also blocked in my region.

russia-blacklist.txt contains the list of websites blocked in Russia. exhentai.org is not blocked in Russia.

russia-blacklist.txt contains the list of websites blocked in Russia. exhentai.org is not blocked in Russia.

Well exhentai.org is inaccessible to me unless I modify that entry in the file to change ".me" to ".org". What other solution do you suggest?

You can construct your own blacklist file or run the program without one.


Add support spark plugin in python-3.11 and remove requirement.txt

TL;DR
pyspark already supports python 3.11. we should run spark unit test with python 3.11 as well.
https://github.com/apache/spark/blob/370870b7a0303e4a2c4b3dea1b479b4fcbc93f8d/python/setup.py#L335-L344
Also remove requirements.txt from each plugin directory.
Type

[ ] Bug Fix
[x] Feature
[ ] Plugin

Are all requirements met?

[x] Code completed
[ ] Smoke tested
[ ] Unit tests added
[ ] Code documentation added
[ ] Any pending items have an associated Issue

Complete description
^^^
Tracking Issue
NA
Follow-up issue
NA

cc @wild-endeavor @eapolinario


alpha_ and lambda_ are not necessary for contextual linear bandit algorithms

Currently, contextual linear and logistic bandit algorithms share the same superclass BaseContextualPolicy.
The constructor of BaseContextualPolicy has alpha_ and lambda_ as arguments:
https://github.com/st-tech/zr-obp/blob/c9ad20cc89db55c85e1a502e923e9c41ab1635f2/obp/policy/base.py#L93-L129
These arguments are used to initialize self.alpha_list and self.lambda_list, which are used by LogisticEpsilonGreedy, LogisticTS, and LogisticUCB but not used by LinearEpsilonGreedy, LinTS, and LinUCB.
I suggest moving alpha_, lambda_,  self.alpha_list, and self.lambda_list to another class, BaseLogisticPolicy for example, and making logistic policies inherit this new class.

@Kurorororo Can you fix this issue by yourself after finishing https://github.com/st-tech/zr-obp/pull/78 ?

Sure!


Argument error on binary_part

I'm using this function in my code
Timex.format!(Timex.now, "{YYYY}-{0M}-{0D}-{h24}-{m}-{s}{ss}")

however sometimes I get this exception
** (ArgumentError) argument error
    :erlang.binary_part("95416", {0, 6})
    lib/format/datetime/formatter.ex:523: Timex.Format.DateTime.Formatter.format_token/6
    lib/format/datetime/formatters/default.ex:182: Timex.Format.DateTime.Formatters.Default.do_format/4
    lib/format/datetime/formatter.ex:54: Timex.Format.DateTime.Formatter.lformat!/4

I've tried to manually run the function may times on iex console and after 10-15 tries I got the same exception.
What could be the issues here?

The issue seems with the microseconds, you just need to run Timex.format!(Timex.now, "{ss}") that it generates that exception

The issue is that binary_part takes 3 arguments, the 2nd is the start index, the 3rd is the number of bytes to take. It looks like something is passing a tuple to it instead of the correct arguments. That tuple is a microsecond tuple I'm almost positive, which means there is a bug in the {ss} formatter. I'll take a look!

What I've noticed is that every time it happens, the microseconds has 5 digits instead of 6

Pushed in 3.1.9


unable to evaluate symlinks in Dockerfile path

I have a Dockerfile in the root of my repo but I am still getting
unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /home/runner/work/REPONAME/REPONAME/Dockerfile: no such file or directory ##[error]Process completed with exit code 1. when building.

Could you share a snip of your workflow, like how you're building it? And the preceding steps

I have the same error.
name: BUILD_AND_PUSH_DOCKER

on:
  push:
    branches:
      - preciz/phx

jobs:
  build_and_push_docker:
    runs-on: ubuntu-18.04
    steps:
      - uses: azure/docker-login@v1
        with:
          login-server: *****.azurecr.io
          username: ${{ secrets.REGISTRY_USERNAME }}
          password: ${{ secrets.REGISTRY_PASSWORD }}
      - run: |
          docker build --build-arg DATABASE_USER="${{ secrets.DATABASE_USER }}" \
                       --build-arg DATABASE_PASS="${{ secrets.DATABASE_PASS }}" \
                       --build-arg DATABASE_HOST="${{ secrets.DATABASE_HOST }}" \
                       --build-arg DATABASE="${{ secrets.DATABASE }}" \
                       --build-arg SECRET_KEY_BASE="${{ secrets.SECRET_KEY_BASE }}" . -t *****.azurecr.io/docs-elixir:v1
          docker push ******.azurecr.io/docs-elixir:v1

Looks like I forgot to add the checkout action:
https://help.github.com/en/actions/configuring-and-managing-workflows/configuring-a-workflow#using-the-checkout-action
     steps:
       - uses: actions/checkout@v2

@preciz, that makes sense. I probably made a similar mistake.

It seems Docker images when built from a Dockerfile by default assume that the Dockerfile in the working directory is named Dockerfile.Dockerfile.
Hence it issues the error "unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /home/xxx/yyy/Dockerfile: no such file or directory.
It is looking for a Dockerfile with firstname 'Dockerfile' instead of a Dockerfile with surname '.Dockerfile' which is what we would ideally want.
To build the image correctly or as a workaround to docker's weird image building simply specify the correct name of the Dockerfile as follows:
$ docker build -t <container-name> . -f whateverthename.Dockerfile.

That should do the trick


Cannot Find Symbol build errors for clients

Hello, I am looking to use the client in a project of mine, but have run into a build problem. it looks like the last commit removed the Class file, but not the references and calls to them. Any advice on getting this to build?
[INFO] Scanning for projects... [INFO] [INFO] -----------------< edu.bcm.hgsc.fhir:hgsc-fhir-client >----------------- [INFO] Building hgsc-fhir-client Webapp 0.0.1 [INFO] --------------------------------[ war ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ hgsc-fhir-client --- [INFO] Deleting /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/target [INFO] [INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ hgsc-fhir-client --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ hgsc-fhir-client --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 44 source files to /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/target/classes [INFO] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/services/FileUploadServiceImpl.java: /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/services/FileUploadServiceImpl.java uses unchecked or unsafe operations. [INFO] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/services/FileUploadServiceImpl.java: Recompile with -Xlint:unchecked for details. [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/utils/FhirResourcesMappingUtils.java:[63,57] cannot find symbol   symbol:   class DxCNVVariantsValueMapper   location: class edu.bcm.hgsc.fhir.utils.FhirResourcesMappingUtils [ERROR] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/services/FileUploadServiceImpl.java:[669,93] cannot find symbol   symbol:   variable orgName   location: class edu.bcm.hgsc.fhir.services.FileUploadServiceImpl [INFO] 2 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time:  1.898 s [INFO] Finished at: 2020-11-09T13:07:11-05:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hgsc-fhir-client: Compilation failure: Compilation failure: [ERROR] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/utils/FhirResourcesMappingUtils.java:[63,57] cannot find symbol [ERROR]   symbol:   class DxCNVVariantsValueMapper [ERROR]   location: class edu.bcm.hgsc.fhir.utils.FhirResourcesMappingUtils [ERROR] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/services/FileUploadServiceImpl.java:[669,93] cannot find symbol [ERROR]   symbol:   variable orgName [ERROR]   location: class edu.bcm.hgsc.fhir.services.FileUploadServiceImpl [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Added Fei, the primary developer to this thread.
HI Fei,
Can you please help with this request?
Thanks,
Mullai

From: p-oshea<EMAIL_ADDRESS>Sent: Monday, November 9, 2020 12:42 PM
To: emerge-ehri/FHIRGenomicsImplementation<EMAIL_ADDRESS>Cc: Subscribed<EMAIL_ADDRESS>Subject: [emerge-ehri/FHIRGenomicsImplementation] Cannot Find Symbol build errors for clients (#1)
CAUTION: This email is not from a BCM Source. Only click links or open attachments you know are safe.

Hello, I am looking to use the client in a project of mine, but have run into a build problem. it looks like the last commit removed the Class file, but not the references and calls to them. Any advice on getting this to build?
[INFO] Scanning for projects... [INFO] [INFO] -----------------< edu.bcm.hgsc.fhir:hgsc-fhir-client >----------------- [INFO] Building hgsc-fhir-client Webapp 0.0.1 [INFO] --------------------------------[ war ]--------------------------------- [INFO] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ hgsc-fhir-client --- [INFO] Deleting /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/target [INFO] [INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ hgsc-fhir-client --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 3 resources [INFO] [INFO] --- maven-compiler-plugin:3.8.0:compile (default-compile) @ hgsc-fhir-client --- [INFO] Changes detected - recompiling the module! [INFO] Compiling 44 source files to /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/target/classes [INFO] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/services/FileUploadServiceImpl.java: /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/services/FileUploadServiceImpl.java uses unchecked or unsafe operations. [INFO] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/services/FileUploadServiceImpl.java: Recompile with -Xlint:unchecked for details. [INFO] ------------------------------------------------------------- [ERROR] COMPILATION ERROR : [INFO] ------------------------------------------------------------- [ERROR] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/utils/FhirResourcesMappingUtils.java:[63,57] cannot find symbol symbol: class DxCNVVariantsValueMapper location: class edu.bcm.hgsc.fhir.utils.FhirResourcesMappingUtils [ERROR] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/services/FileUploadServiceImpl.java:[669,93] cannot find symbol symbol: variable orgName location: class edu.bcm.hgsc.fhir.services.FileUploadServiceImpl [INFO] 2 errors [INFO] ------------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.898 s [INFO] Finished at: 2020-11-09T13:07:11-05:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hgsc-fhir-client: Compilation failure: Compilation failure: [ERROR] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/utils/FhirResourcesMappingUtils.java:[63,57] cannot find symbol [ERROR] symbol: class DxCNVVariantsValueMapper [ERROR] location: class edu.bcm.hgsc.fhir.utils.FhirResourcesMappingUtils [ERROR] /Users/poshea/Documents/GitHub/FHIRGenomicsImplementation/hgsc-fhir-client/src/main/java/edu/bcm/hgsc/fhir/services/FileUploadServiceImpl.java:[669,93] cannot find symbol [ERROR] symbol: variable orgName [ERROR] location: class edu.bcm.hgsc.fhir.services.FileUploadServiceImpl [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_emerge-2Dehri_FHIRGenomicsImplementation_issues_1&d=DwMFaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=ftaHWW4W6kxrCO3UuP4oAg&m=UYPRXnYQ5GOww0y71NOKdf9ClrHhB_-djj-PLnGHx0M&s=runTItw_meIui6YVLZ--amo10FhVteW99X1FqHKt1MA&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ABZHIPIOKVEGYGH7SI6LI6TSPAZYNANCNFSM4TPW6MSQ&d=DwMFaQ&c=ZQs-KZ8oxEw0p81sqgiaRA&r=ftaHWW4W6kxrCO3UuP4oAg&m=UYPRXnYQ5GOww0y71NOKdf9ClrHhB_-djj-PLnGHx0M&s=Z7olz5ECPtq3amWig7sCdH40EN3Uhc01oAkW0cZhCbA&e=.

@p-oshea can you please try now?

hi @feiyanbcm I have successfully built it now, thanks!