Unsupported data type for inplace predict.

Dask API I guess changed some from Dec 2020 till now.  I'm trying to understand this one.
I'm still trying to construct repro, but I can't tell what the problem is.  It sounds like dask API changed so that only inplace prediction is allowed, and somehow that means some types of more flexible inputs are no longer allowed?
I'm just passing normal dask from, like I have been, to the predict.  The failure comes when doing preds.compute() after getting preds from xgboost.
@trivialfis any ideas?
Exception: XGBoostError('[14:17:33] /workspace/xgboost/src/gbm/gbtree.h:294: Check failed: success: Unsupported data type for inplace predict.\nStack trace:\n  [bt] (0) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x2e57f9) [0x7fd1112147f9]\n  [bt] (1) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(xgboost::gbm::GBTree::InplacePredict(dmlc::any const&, std::shared_ptr<xgboost::DMatrix>, float, xgboost::PredictionCacheEntry*, unsigned int, unsigned int) const+0x392) [0x7fd111223822]\n  [bt] (2) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(xgboost::LearnerImpl::InplacePredict(dmlc::any const&, std::shared_ptr<xgboost::DMatrix>, xgboost::PredictionType, float, xgboost::HostDeviceVector<float>**, unsigned int, unsigned int)+0x1e6) [0x7fd111254c36]\n  [bt] (3) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(void InplacePredictImpl<xgboost::data::ArrayAdapter>(std::shared_ptr<xgboost::data::ArrayAdapter>, std::shared_ptr<xgboost::DMatrix>, char const*, xgboost::Learner*, unsigned long, unsigned long, unsigned long const**, unsigned long*, float const**)+0x2ef) [0x7fd11114ba3f]\n  [bt] (4) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(XGBoosterPredictFromDense+0x2dd) [0x7fd11113779d]\n  [bt] (5) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x69dd) [0x7fd2329949dd]\n  [bt] (6) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x6067) [0x7fd232994067]\n  [bt] (7) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x10da8) [0x7fd2329aada8]\n  [bt] (8) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x1108c) [0x7fd2329ab08c]\n\n')

distributed.worker - WARNING - Compute Failed
Function:  subgraph_callable
args:      ('mapped_predict-ccba8a6f74a8fba018972d7b5d295e4c', <xgboost.core.Booster object at 0x7fd190409640>, True, (0, 1, 2, 3), None, 'drop_by_shallow_copy-f5652ef883146e8e6dc4e5ffcb4dfa51',      ChangeTempDir  ChangeTempMag  ...   I tried constructing a repro with same exact model parameters and data, but it doesn't fail.  But inside our code it always fails 100% same way.  Probably some other issue, but if I knew what the error meant I might know how to proceed.
from dask.distributed import Client
import dask.dataframe as dd
from dask_cuda import LocalCUDACluster

import numpy as np
nan = np.nan
import pandas as pd

import xgboost as xgb

def run():
    with LocalCUDACluster() as cluster:
        with Client(cluster) as client:
            params = {'base_score': 0.5, 'booster': 'gbtree', 'colsample_bylevel': 1, 'colsample_bynode': 1,
                      'colsample_bytree': 0.8, 'gamma': 0.0, 'gpu_id': 0, 'importance_type': 'gain',
                      'interaction_constraints': '', 'learning_rate': 0.05, 'max_delta_step': 0.0, 'max_depth': 6,
                      'min_child_weight': 1, 'missing': nan, 'monotone_constraints': '()', 'n_estimators': 100,
                      'n_jobs': 4, 'num_parallel_tree': 1, 'objective': 'multi:softprob', 'random_state': 0,
                      'reg_alpha': 0.0, 'reg_lambda': 1.0, 'scale_pos_weight': 1.0, 'subsample': 0.7,
                      'tree_method': 'gpu_hist', 'validate_parameters': 1, 'verbosity': None,
                      'early_stopping_rounds': 30, 'num_class': 4, 'max_bin': 256, 'grow_policy': 'depthwise',
                      'max_leaves': 64, 'enable_early_stopping_rounds': True, 'eval_metric': 'mlogloss',
                      'early_stopping_threshold': 0, 'monotonicity_constraints': False, 'silent': 0, 'debug_verbose': 2}

            model = xgb.dask.DaskXGBClassifier(**params)

            train_X = pd.read_csv("train_X.csv")
            train_y = train_X['target']
            train_X = train_X.drop('target', axis=1)
            test_X = pd.read_csv("test_X.csv")
            sample_weight = pd.Series(np.ones(len(train_y)))
            sample_weight_eval_set = [pd.Series(np.ones(len(train_y)))]

            chunksize = 177
            train_X = dd.from_pandas(train_X, chunksize=chunksize).persist()
            train_y = dd.from_pandas(train_y, chunksize=chunksize).persist()
            sample_weight = dd.from_pandas(sample_weight, chunksize=chunksize).persist()
            sample_weight_eval_set = [dd.from_pandas(sample_weight_eval_set[0], chunksize=chunksize).persist()]
            test_X = dd.from_pandas(test_X, chunksize=chunksize).persist()

            model.fit(train_X, train_y,
                           sample_weight=sample_weight,
                           eval_set=[(train_X.copy(), train_y)],
                           sample_weight_eval_set=sample_weight_eval_set,
                           )
            preds = model.predict_proba(test_X)
            print(preds.shape)
            preds = preds.compute()
            print(preds.shape)

if __name__ == "__main__":
    run()

daskissue.zip

Your example doesn't reproduce the issue.  The error in your backtrace should happen only when you are calling inplace_predict directly, and have set predictor to something that doesn't match your input data (cpu vs gpu).  This should not happen in sklearn interface since inplace_predict wouldn't be used if you have set the predictor parameter.

Your example doesn't reproduce the issue. The error in your backtrace should happen only when you are calling inplace_predict directly, and have set predictor to something that doesn't match your input data (cpu vs gpu). This should not happen in sklearn interface since inplace_predict wouldn't be used if you have set the predictor parameter.

Yes, I specifically said I can't reproduce.  I said I gave example that has same parameters and data and choices.
I'm not calling inplace_predict directly.  I'm only using sklearn API.

Could you please dump out the model.get_booster().save_config() right before running the problematic predict?

Will try to reproduce it today.

https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/xgboost-1.5.0_SNAPSHOT%2B2320aa0da23ed29610a3bd58cba9a1fe481e443a-py3-none-manylinux2010_x86_64.whl
Using this snapshot.
model.get_booster().save_config()

{"learner":{"generic_param":{"fail_on_invalid_gpu_id":"0","gpu_id":"0","gpu_page_size":"0","n_jobs":"4","nthread":"4","random_state":"0","seed":"0","seed_per_iteration":"0","validate_parameters":"1"},"gradient_booster":{"gbtree_train_param":{"num_parallel_tree":"1","predictor":"gpu_predictor","process_type":"default","tree_method":"gpu_hist","updater":"grow_gpu_hist","updater_seq":"grow_gpu_hist"},"name":"gbtree","specified_updater":false,"updater":{"grow_gpu_hist":{"gpu_hist_train_param":{"debug_synchronize":"0","deterministic_histogram":"1","single_precision_histogram":"0"},"train_param":{"alpha":"0","cache_opt":"1","colsample_bylevel":"1","colsample_bynode":"1","colsample_bytree":"0.800000012","default_direction":"learn","enable_feature_grouping":"0","eta":"0.0500000007","gamma":"0","grow_policy":"depthwise","interaction_constraints":"","lambda":"1","learning_rate":"0.0500000007","max_bin":"256","max_conflict_rate":"0","max_delta_step":"0","max_depth":"6","max_leaves":"64","max_search_group":"100","min_child_weight":"1","min_split_loss":"0","monotone_constraints":"()","refresh_leaf":"1","reg_alpha":"0","reg_lambda":"1","sampling_method":"uniform","sketch_eps":"0.0299999993","sketch_ratio":"2","sparse_threshold":"0.20000000000000001","split_evaluator":"elastic_net,monotonic","subsample":"0.699999988"}}}},"learner_model_param":{"base_score":"5E-1","num_class":"4","num_feature":"22"},"learner_train_param":{"booster":"gbtree","disable_default_eval_metric":"0","dsplit":"row","objective":"multi:softprob"},"metrics":["mlogloss"],"objective":{"name":"multi:softprob","softmax_multiclass_param":{"num_class":"4"}}},"version":[1,5,0]}

Then doing the .compute() leads to:
[22:56:40] WARNING: ../src/learner.cc:572:
Parameters: { "debug_verbose", "early_stopping_rounds", "early_stopping_threshold", "enable_early_stopping_rounds", "monotonicity_constraints", "scale_pos_weight", "silent" } might not be used.

  This may not be accurate due to some parameters are only used in language bindings but
  passed down to XGBoost core.  Or some parameters are not used but slip through this
  verification. Please open an issue if you find above cases.

distributed.worker - WARNING - Compute Failed
Function:  subgraph_callable
args:      ('mapped_predict-ee8f58e1bdad9911c6335de5c5432a14', <xgboost.core.Booster object at 0x7f60400f2ac0>, True, (0, 1, 2, 3), None, 'drop_by_shallow_copy-f5652ef883146e8e6dc4e5ffcb4dfa51',      ChangeTempDir  ChangeTempMag  ...

Thanks.  I will try to see which hash caused it.

BAD: https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/xgboost-1.5.0_SNAPSHOT%2B2320aa0da23ed29610a3bd58cba9a1fe481e443a-py3-none-manylinux2010_x86_64.whl
GOOD: https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/xgboost-1.4.0_SNAPSHOT%2Bd8ec7aad5a9a3eb580c55680aee8ad1a975cba20-py3-none-manylinux2010_x86_64.whl
BAD: https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/xgboost-1.5.0_SNAPSHOT%2B20f34d9776d705348264b3175418eb522a775f3a-py3-none-manylinux2010_x86_64.whl
BAD: https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/xgboost-1.4.0_SNAPSHOT%2Bea7a6a0321d23ecf8d35238bfa777589cd7f3332-py3-none-manylinux2010_x86_64.whl
BAD: https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/xgboost-1.4.0_SNAPSHOT%2Be2d8a994134d25be8318f3e6316a144230c41e5e-py3-none-manylinux2010_x86_64.whl
BAD: https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/xgboost-1.4.0_SNAPSHOT%2Bdc97b5f19f99c17b1277981d96ea87fbed176fcb-py3-none-manylinux2010_x86_64.whl
BAD: https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/xgboost-1.4.0_SNAPSHOT%2Bdbf7e9d3cbc0adc1c8cea5f1002fcc01fc6e121b-py3-none-manylinux2010_x86_64.whl
GOOD : https://s3-us-west-2.amazonaws.com/xgboost-nightly-builds/xgboost-1.4.0_SNAPSHOT%2Bdbb5208a0a11136d44451eab52b00f4bde2e3ad7-py3-none-manylinux2010_x86_64.whl
So the error is in:
git diff dbb5208a0a11136d44451eab52b00f4bde2e3ad7 dbf7e9d3cbc0adc1c8cea5f1002fcc01fc6e121b

This is same as the other issue: https://github.com/dmlc/xgboost/issues/6973#issuecomment-844811730

Thanks for finding the diff.  That PR modifies XGBoost to use inplace predict by default on sklearn interface.  I see the predictor is set to gpu_predictor, in which case XGBoost should use normal prediction.  I'm not entirely sure why yet.

Thanks for finding the diff. That PR modifies XGBoost to use inplace predict by default on sklearn interface. I see the predictor is set to gpu_predictor, in which case XGBoost should use normal prediction. I'm not entirely sure why is the error happening yet.

Did the output of model.get_booster().save_config() above help?  Also, the error literally makes it seem the frame type is wrong.  But what frame type (at that point) is allowed vs. not allowed?  I'm just asking to see if I can understand what is happening to better find a repro.
Thanks!

I managed to reproduce it in a weird way:
            # ... same as your code above

            model.fit(
                train_X,
                train_y,
                sample_weight=sample_weight,
                eval_set=[(train_X.copy(), train_y)],
                sample_weight_eval_set=sample_weight_eval_set,
            )

            # Bypass the sklearn interface to set the parameter.
            model.get_booster().set_param({"predictor": "gpu_predictor"})
            preds = model.predict_proba(test_X)
            print(preds.shape)
            preds = preds.compute()
            print(preds.shape)

if __name__ == "__main__":
    run()

See the line before running predict_proba.  I'm not sure if that's your case.

Aside from the above reproducible example, I suspect that this is related to https://github.com/dmlc/xgboost/issues/6991 .  There might be some mixups in your code that somehow native Booster and sklearn.XGBClassifier are loading each other.

It's possible.  We fit in one fork, and can sometimes predict in another fork.  So we have to pickle between.  And we tend to set some parameters of the booster that usually get lost (kinda related to n_classes_ going missing).
However, in this case, this is a direct dask use.  I'm not converting from dask to non-dask, so that #6991 is not directly relevant.

N classes should be preserved as mentioned in the other issue.  See the test I posted there.
There's is no problem on setting it with the standard set_params function from skl interface.  Also setting it in constructor is also fine.  Pickle is also tested.
I reproduced the issue only when one tries to set the parameter bypassing skl interface.  Could you confirm that's the case?

The only possible explanation I have got so far, is you have trained a model using xgb.dask.train, then load it with xgb.dask.DaskXGBClassifier.  Then the behavior is undefined.

No, I don't use xgb.dask.train, or load models with xgb.dask..
I do similar to what you showed and what I showed.

fork
fit with dask
pickle the model to file
complete fork
fork (or be on new system entirely, maybe with no GPUs anymore)
load the pickle
set some things that got lost or got changed since not stored in pickle.  In past this has been things like predictor, tree_method for correct new CPU prediction,  n_jobs if changed because we have more or less resources in new fork/system.

I'll try to make my example that didn't repro and add those booster setters like you did.  See if that repros.

Thanks for debugging on this.  I think at this point it would be great to focus on the other issue first.  We run extensive tests to ensure nothing is lost during pickle.  I'm quite surprised that's not the case in your environment.

Yes, super, so this repros:
trains.zip
from dask.distributed import Client
import dask.dataframe as dd
from dask_cuda import LocalCUDACluster

import numpy as np
nan = np.nan
import pandas as pd

import xgboost as xgb

def run():
    with LocalCUDACluster() as cluster:
        with Client(cluster) as client:
            params = {'base_score': 0.5, 'booster': 'gbtree', 'colsample_bylevel': 1, 'colsample_bynode': 1,
                      'colsample_bytree': 0.8, 'gamma': 0.0, 'gpu_id': 0, 'importance_type': 'gain',
                      'interaction_constraints': '', 'learning_rate': 0.05, 'max_delta_step': 0.0, 'max_depth': 6,
                      'min_child_weight': 1, 'missing': nan, 'monotone_constraints': '()', 'n_estimators': 100,
                      'n_jobs': 4, 'num_parallel_tree': 1, 'objective': 'multi:softprob', 'random_state': 0,
                      'reg_alpha': 0.0, 'reg_lambda': 1.0, 'scale_pos_weight': 1.0, 'subsample': 0.7,
                      'tree_method': 'gpu_hist', 'validate_parameters': 1, 'verbosity': None,
                      'early_stopping_rounds': 30, 'num_class': 4, 'max_bin': 256, 'grow_policy': 'depthwise',
                      'max_leaves': 64, 'enable_early_stopping_rounds': True, 'eval_metric': 'mlogloss',
                      'early_stopping_threshold': 0, 'monotonicity_constraints': False, 'silent': 0, 'debug_verbose': 2}

            model = xgb.dask.DaskXGBClassifier(**params)

            train_X = pd.read_csv("train_X.csv")
            train_y = train_X['target']
            train_X = train_X.drop('target', axis=1)
            sample_weight = pd.Series(np.ones(len(train_y)))
            sample_weight_eval_set = [pd.Series(np.ones(len(train_y)))]

            chunksize = 177
            train_X = dd.from_pandas(train_X, chunksize=chunksize).persist()
            train_y = dd.from_pandas(train_y, chunksize=chunksize).persist()
            sample_weight = dd.from_pandas(sample_weight, chunksize=chunksize).persist()
            sample_weight_eval_set = [dd.from_pandas(sample_weight_eval_set[0], chunksize=chunksize).persist()]

            model.fit(train_X, train_y,
                           sample_weight=sample_weight,
                           eval_set=[(train_X.copy(), train_y)],
                           sample_weight_eval_set=sample_weight_eval_set,
                           )
            import pickle

            pickle.dump(model, open("mm.pkl", "wb"))

    with LocalCUDACluster() as cluster:
        with Client(cluster) as client:
            model = pickle.load(open("mm.pkl", "rb"))

            # Bypass the sklearn interface to set the parameter.
            model.get_booster().set_param({"predictor": "gpu_predictor"})

            test_X = pd.read_csv("test_X.csv")
            test_X = dd.from_pandas(test_X, chunksize=chunksize).persist()
            preds = model.predict_proba(test_X)
            print(preds.shape)
            preds = preds.compute()
            print(preds.shape)

if __name__ == "__main__":
    run()

(base) jon@pseudotensor:~/h2oai$ python daskissue_pickleversion.py
[12:50:02] task [xgboost.dask]:tcp://<IP_ADDRESS>:46743 got new rank 0
[12:50:02] WARNING: /workspace/xgboost/src/learner.cc:572:
Parameters: { "debug_verbose", "early_stopping_rounds", "early_stopping_threshold", "enable_early_stopping_rounds", "monotonicity_constraints", "scale_pos_weight", "silent" } might not be used.

  This may not be accurate due to some parameters are only used in language bindings but
  passed down to XGBoost core.  Or some parameters are not used but slip through this
  verification. Please open an issue if you find above cases.

  ChangeTempDir  ChangeTempMag  MaxWindPeriod  RainToday  TempRange  PressureChange  RISK_MM  RainTomorrow
0             0   21.0     5.8       8.2          44.0     10.9         70.0       7.0  ...            1.0            1.0            3.0        0.0       10.7             0.0      0.2           0.0
1             1   22.0     4.2       8.4          43.0     12.4         63.0       4.0  ...            1.0            1.0            2.0        0.0       12.1             0.0      0.0           0.0
2             2   23.0     5.6       4.6          41.0     12.1         65.0       6.0  ...            1.0            1.0            2.0        0.0        8.7             0.0      0.0           0.0
3             3   24.0     4.0       4.1          48.0     14.1         70.0       7.0  ...            1.0
kwargs:    {}
Exception: XGBoostError('[12:50:05] /workspace/xgboost/src/gbm/gbtree.h:294: Check failed: success: Unsupported data type for inplace predict.\nStack trace:\n  [bt] (0) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x2e5109) [0x7fcd9745e109]\n  [bt] (1) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(xgboost::gbm::GBTree::InplacePredict(dmlc::any const&, std::shared_ptr<xgboost::DMatrix>, float, xgboost::PredictionCacheEntry*, unsigned int, unsigned int) const+0x392) [0x7fcd9746d132]\n  [bt] (2) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(xgboost::LearnerImpl::InplacePredict(dmlc::any const&, std::shared_ptr<xgboost::DMatrix>, xgboost::PredictionType, float, xgboost::HostDeviceVector<float>**, unsigned int, unsigned int)+0x1e6) [0x7fcd9749e546]\n  [bt] (3) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(void InplacePredictImpl<xgboost::data::ArrayAdapter>(std::shared_ptr<xgboost::data::ArrayAdapter>, std::shared_ptr<xgboost::DMatrix>, char const*, xgboost::Learner*, unsigned long, unsigned long, unsigned long const**, unsigned long*, float const**)+0x2ef) [0x7fcd9739535f]\n  [bt] (4) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(XGBoosterPredictFromDense+0x2cd) [0x7fcd973812dd]\n  [bt] (5) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x69dd) [0x7fce6e5799dd]\n  [bt] (6) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x6067) [0x7fce6e579067]\n  [bt] (7) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x10da8) [0x7fce6e58fda8]\n  [bt] (8) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x1108c) [0x7fce6e59008c]\n\n')

distributed.worker - WARNING - Compute Failed
Function:  subgraph_callable
args:      ('mapped_predict-c83c00f498802590271f95b70c5943ca', <xgboost.core.Booster object at 0x7fcd94518d30>,      Unnamed: 0   Date  EvapMM  Sunshine  MaxWindSpeed  Temp9am  RelHumid9am  Cloud9am  ...  ChangeTempDir  ChangeTempMag  MaxWindPeriod  RainToday  TempRange  PressureChange  RISK_MM  RainTomorrow
177         177  173.0     2.2       8.9          22.0      4.7         84.0       2.0  ...            1.0            0.0            2.0        0.0       19.4             0.0      0.0           0.0
178         178  199.0     1.6       9.0          22.0      9.6         74.0       1.0  ...            1.0            1.0            3.0        0.0       12.3             0.0      0.0           0.0
179         179  200.0     1.4       9.0          28.0      4.9         86.0       1.0  ...            1.0            0.0            2.0        0.0       15.7             0.0      0.0           0.0
180         180  201.0     2.0       3.0          30.0      7.9         76.0       7.0  ...            1.0
kwargs:    {}
Exception: XGBoostError('[12:50:05] /workspace/xgboost/src/gbm/gbtree.h:294: Check failed: success: Unsupported data type for inplace predict.\nStack trace:\n  [bt] (0) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x2e5109) [0x7fcd9745e109]\n  [bt] (1) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(xgboost::gbm::GBTree::InplacePredict(dmlc::any const&, std::shared_ptr<xgboost::DMatrix>, float, xgboost::PredictionCacheEntry*, unsigned int, unsigned int) const+0x392) [0x7fcd9746d132]\n  [bt] (2) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(xgboost::LearnerImpl::InplacePredict(dmlc::any const&, std::shared_ptr<xgboost::DMatrix>, xgboost::PredictionType, float, xgboost::HostDeviceVector<float>**, unsigned int, unsigned int)+0x1e6) [0x7fcd9749e546]\n  [bt] (3) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(void InplacePredictImpl<xgboost::data::ArrayAdapter>(std::shared_ptr<xgboost::data::ArrayAdapter>, std::shared_ptr<xgboost::DMatrix>, char const*, xgboost::Learner*, unsigned long, unsigned long, unsigned long const**, unsigned long*, float const**)+0x2ef) [0x7fcd9739535f]\n  [bt] (4) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(XGBoosterPredictFromDense+0x2cd) [0x7fcd973812dd]\n  [bt] (5) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x69dd) [0x7fce6e5799dd]\n  [bt] (6) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x6067) [0x7fce6e579067]\n  [bt] (7) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x10da8) [0x7fce6e58fda8]\n  [bt] (8) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x1108c) [0x7fce6e59008c]\n\n')

Traceback (most recent call last):
  File "daskissue_pickleversion.py", line 67, in <module>
    run()
  File "daskissue_pickleversion.py", line 62, in run
    preds = preds.compute()
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/dask/base.py", line 284, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/dask/base.py", line 566, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/distributed/client.py", line 2666, in get
    results = self.gather(packed, asynchronous=asynchronous, direct=direct)
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/distributed/client.py", line 1975, in gather
    return self.sync(
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/distributed/client.py", line 843, in sync
    return sync(
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/distributed/utils.py", line 353, in sync
    raise exc.with_traceback(tb)
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/distributed/utils.py", line 336, in f
    result[0] = yield future
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/distributed/client.py", line 1840, in _gather
    raise exception.with_traceback(traceback)
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/dask/optimization.py", line 963, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/dask/core.py", line 151, in get
    result = _execute_task(task, cache)
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/dask/core.py", line 121, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/dask/core.py", line 115, in _execute_task
    return [_execute_task(a, cache) for a in arg]
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/dask/core.py", line 115, in <listcomp>
    return [_execute_task(a, cache) for a in arg]
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/dask/utils.py", line 34, in apply
    return func(*args, **kwargs)
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/dask/dataframe/core.py", line 5665, in apply_and_enforce
    df = func(*args, **kwargs)
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/dask.py", line 1338, in mapped_predict
    prediction = booster.inplace_predict(
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/core.py", line 1919, in inplace_predict
    _check_call(
  File "/home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/core.py", line 214, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: [12:50:05] /workspace/xgboost/src/gbm/gbtree.h:294: Check failed: success: Unsupported data type for inplace predict.
Stack trace:
  [bt] (0) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(+0x2e5109) [0x7fcd9745e109]
  [bt] (1) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(xgboost::gbm::GBTree::InplacePredict(dmlc::any const&, std::shared_ptr<xgboost::DMatrix>, float, xgboost::PredictionCacheEntry*, unsigned int, unsigned int) const+0x392) [0x7fcd9746d132]
  [bt] (2) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(xgboost::LearnerImpl::InplacePredict(dmlc::any const&, std::shared_ptr<xgboost::DMatrix>, xgboost::PredictionType, float, xgboost::HostDeviceVector<float>**, unsigned int, unsigned int)+0x1e6) [0x7fcd9749e546]
  [bt] (3) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(void InplacePredictImpl<xgboost::data::ArrayAdapter>(std::shared_ptr<xgboost::data::ArrayAdapter>, std::shared_ptr<xgboost::DMatrix>, char const*, xgboost::Learner*, unsigned long, unsigned long, unsigned long const**, unsigned long*, float const**)+0x2ef) [0x7fcd9739535f]
  [bt] (4) /home/jon/minicondadai_py38/lib/python3.8/site-packages/xgboost/lib/libxgboost.so(XGBoosterPredictFromDense+0x2cd) [0x7fcd973812dd]
  [bt] (5) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x69dd) [0x7fce6e5799dd]
  [bt] (6) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/../../libffi.so.7(+0x6067) [0x7fce6e579067]
  [bt] (7) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x10da8) [0x7fce6e58fda8]
  [bt] (8) /home/jon/minicondadai_py38/lib/python3.8/lib-dynload/_ctypes.cpython-38-x86_64-linux-gnu.so(+0x1108c) [0x7fce6e59008c]

Thanks for confirming.  I can fix it in a PR.  But for general usage, I would suggest avoiding interleaving calls to different interfaces.
The set_params from sklearn is quite complicated, it inspects Python constructor with inspect using runtime reflection and other dart arts ...  The set_params/get_params in XGBoost wraps around the one from sklearn and does some more magic to ensure consistency between Python object attributes and internal parameters defined in C++.

As for the reason why we set things.

At least in past, setting the model state directly had no effect, it didn't change what booster had.
We haven't had issues with setting booster properties for non-dask models
The origin of the problem was the use of the pickled model can be on a system with change in GPUs or CPUs.  An example original issue: ttps://github.com/dmlc/xgboost/issues/4494

But over time we have added a few other things that change of systems have had issues with.  And the code basically looks like:
        if system_has_gpus:
            booster.set_param({'gpu_id': gpu_id})
            booster.set_param({'predictor': 'gpu_predictor'})
            model.kwargs['tree_method'] = 'gpu_hist'
            booster.set_param({'n_jobs': n_jobs})
            booster.set_param({'tree_method': 'gpu_hist'})
            model.n_jobs = n_jobs
            model.tree_method = 'gpu_hist'
        else:
            os.environ['CUDA_VISIBLE_DEVICES'] = ""
            self.params_base['n_gpus'] = 0
            booster.set_param({'predictor': 'cpu_predictor'})
            booster.set_param({'n_gpus': 0})
            booster.set_param({'tree_method': 'hist'})
            booster.set_param({'n_jobs': n_jobs})
            model.kwargs['n_gpus'] = 0
            model.kwargs.pop('gpu_id', None)
            model.kwargs['tree_method'] = 'hist'
            model.n_gpus = 0
            model.gpu_id = None
            model.tree_method = 'hist'
            model.n_jobs = n_jobs

These kinds of things are required because certainly long ago such aspects of the state were not all saved.  And there was duplication across model, model.kwargs, and booster.  So it wasn't clear which to set, so for sanity we set them all.
The problem stems from needing the pickle state from one system to be used for another.  However, even if one predicts with n_jobs=10 that doesn't mean one wants to predict with same n_jobs, so we have to change that too even if same system.

Got it.  I will see if I can make sure of the consistency between internal parameters and sklearn object attributes.

So it wasn't clear which to set, so for sanity we set them all.

In current version, use XGBClassifier.set_params or constructor.  These 2 are the only valid ways to set parameters.

Got it. I will see if I can make sure of the consistency between internal parameters and sklearn object attributes.

So it wasn't clear which to set, so for sanity we set them all.

In current version, use XGBClassifier.set_params or constructor. These 2 are the only valid ways to set skl model parameters.

Will those funnel down to the booster assignment?

Yes, both go to Booster.set_param.  But as mentioned, before that we have to go through some other things to ensure consistency.


wrap require('buffer') in a try/catch for better browserify --no-builtins support

because i would like to use this library in lieu of Buffer and not include the entire Buffer module in my browserify package. right now, it throws, crashing everything.

an alternative is to just check for typeof Buffer !== 'undefined'.

release? :D

Released as 2.1.9


format docuement html with script tags

VSCode Version: 1.30.1
OS Version: win7

Steps to Reproduce:
1.create a html file with script tag
<!DOCTYPE html>

  <meta name=charset content=utf-8>
  <meta name=viewport content="width=device-width,initial-scale=1">

2.format document( shift + alt + f)
the formated code appears as above.  tags haven't been once a line.
the html is generated by webpack4 plugin HtmlWebpackPlugin

Does this issue occur when all extensions are disabled?: Yes

Well, we use js-beautify, and it's the same result: https://beautifier.io
Please report issue to them: https://github.com/beautify-web/js-beautify


Your demo have a bug

I have found your demo has a little bug which maybe not important.when I turn 'Outside Touch' Off and turn 'Hide Done Button' On at the same time，your alertView which showed can't close.

Thanks for catching that! This was one of the things I was thinking on how to approach fixing. Best thing that came to mind is to create a safety mechanism such that incase user is locked out of closing the alert (like when you hide the buttons), it will automatically force 'Outside Touch' to be On. Note that it might not show in the Example app that it's on, but if you try it out, it will be on in the background. Check out the latest commit, cheers!


🐛 [firebase_app_distribution] In-app tester feedback feature from Firebase App Distribution unable to take screenshots in Flutter apps

Bug report
In-app testers' feedback feature is unable to take a screenshot of the Flutter app. Instead, a black rectangle is shown. Attaching an image from the gallery works as expected.
Here is the video of the problem:
https://github.com/firebase/flutterfire/assets/1845144/382a9af5-fb72-40f1-8e67-d1523815daf7
The same results are delivered to the Firebase App Distribution dashboard:

Steps to reproduce
Check the sample project for the exact steps.
Expected behavior
I guess product owners would know better, but I assume either the screenshot should be taken immediately or this feature should be disabled.
Sample project
I prepared a repository where this problem is reproducible. The README file contains all the steps performed.

Additional context
I opened a similar issue in the firebase-android-sdk repository. Even though only native android firebase sdk is used in the sample, it was closed with the suggestion to open it here.

Flutter doctor
Run flutter doctor and paste the output below:
Click To Expand
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.13.6, on macOS 13.5.2 22G91 darwin-arm64, locale en-UA)
[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 15.0)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.3)
[✓] IntelliJ IDEA Ultimate Edition (version 2023.2.2)
[✓] VS Code (version 1.79.2)
[✓] Connected device (4 available)
[✓] Network resources

• No issues found!

Flutter dependencies
Run flutter pub deps -- --style=compact and paste the output below:
Click To Expand
Dart SDK 3.1.3
Flutter SDK 3.13.6
flutter_app_firebase_feedback_demo 1.0.0+1

dependencies:
- cupertino_icons 1.0.6
- firebase_core 2.17.0 [firebase_core_platform_interface firebase_core_web flutter meta]
- flutter 0.0.0 [characters collection material_color_utilities meta vector_math web sky_engine]

dev dependencies:
- flutter_lints 2.0.3 [lints]
- flutter_test 0.0.0 [flutter test_api matcher path fake_async clock stack_trace vector_math async boolean_selector characters collection material_color_utilities meta source_span stream_channel string_scanner term_glyph web]

transitive dependencies:
- async 2.11.0 [collection meta]
- boolean_selector 2.1.1 [source_span string_scanner]
- characters 1.3.0
- clock 1.1.1
- collection 1.17.2
- fake_async 1.3.1 [clock collection]
- firebase_core_platform_interface 4.8.0 [collection flutter flutter_test meta plugin_platform_interface]
- firebase_core_web 2.8.0 [firebase_core_platform_interface flutter flutter_web_plugins js meta]
- flutter_web_plugins 0.0.0 [flutter characters collection material_color_utilities meta vector_math web]
- js 0.6.7 [meta]
- lints 2.1.1
- matcher 0.12.16 [async meta stack_trace term_glyph test_api]
- material_color_utilities 0.5.0 [collection]
- meta 1.9.1
- path 1.8.3
- plugin_platform_interface 2.1.6 [meta]
- sky_engine 0.0.99
- source_span 1.10.0 [collection path term_glyph]
- stack_trace 1.11.0 [path]
- stream_channel 2.1.1 [async]
- string_scanner 1.2.0 [source_span]
- term_glyph 1.2.1
- test_api 0.6.0 [async boolean_selector collection meta source_span stack_trace stream_channel string_scanner term_glyph]
- vector_math 2.1.4
- web 0.1.4-beta

Thanks for the detailed report and repro steps @foxanna
It seems firebase_app_distributon feature is not yet supported by Flutterfire per https://github.com/firebase/flutterfire/issues/11607
I'll keep this issue open for team's input / attention and for better tracking.
/cc @russellwheatley


Switching quality or enabling subtitles restarts playback

When in the middle of watching something, if I switch the quality or change the subtitle track, the video restarts from the beginning. It would be a better experience if the player remembered my position when I change settings.

I second this. Not only this is a UX issue but also a performance one. Sometimes when you switch subtitles and the video restarts the sync breaks and it won't fix until the tab is refreshed.

Thirding this.

...

This issue should be fixed in the current release of synclounge. Are you still experiencing this issue @evaera ?

Fixed!


raftstore: set sync hint on ingest sst commands (#7151)

What have you changed?
After ingest sst, sst files are deleted quickly. As a result, ingest sst command can not be handled again and must be synced.
Set sync hint on ingest sst commands to make sure they are synced properly.
Check List
Tests

Unit test

Related changes

Need to update the documentation

Release note

Set sync hint on ingest sst commands to make sure they are synced properly.

/merge

@overvenus Failed to build.

/merge


Apparent error in string conversion

I have a struct with
impl<'env,'borrow> MyStruct<'env,'borrow> {
  pub extern "java" fn message(&self,_env:&'borrow JNIEnv<'env>,_s:String) -> JniResult<bool> {}
}

...but when calling this function, I get an Error:
Error(JavaException, State { next:error:None, bracktrace: InternalBacktrace { backtrace: Some(
  0: error_chain::backtrace::imp::InternalBacktrace::new
  1: <error_chain::State as core::default::Default>::default
  2: <T as core::convert::Into<U>>::into
  3: jni::wrapper::jnienv::JNIEnv::new_string
  4: <alloc::string::String as robusta_jni::convert::safe::TryIntoJavaValue>::try_into
  5: ...::message
  ...
}}

...which looks to me like a bug in the string conversion to java. It's difficult to find the precise cause, since I can't set break points in rust when running the program from java (or, actually, Scala, but that shouldn't make a difference).
Any help would be very much appreciated :)

(also: I noticed that java booleans seem to be the opposite from the ones in rust: Passing the java value true to a rust function prints false and vice versa)

Also, I'm getting a similar error when calling a java function that takes an Integer array as input. When converting the corresponding Vec, it throws:
panicked at 'called Result::unwrap() on an Err value: "unexpected parseInput: java/lang/Integer"' in <i32 as robusta_jni::convert::JavaValue>::autobox...

Unfortunately I can't check the code right now, I'll get back to you later this week. From the stack trace it seems that it's the string conversion that fails from Rust as you suggested.

(also: I noticed that java booleans seem to be the opposite from the ones in rust: Passing the java value true to a rust function prints false and vice versa)

:facepalm:  Right here. To get to 1.0 I'm going to need an extensive test suite, that's for sure.
Could you provide a minimal example? Or try switching to a different call_type?

Hi, I'm really sorry I can't be more informative. The string bug I tried to narrow down to something simpler, but in all my toy examples it worked - only in the actual setting where I needed it to work did it not, and I have no idea what was different (the strings were definitely equal on an as_bytes()-level). So all I can say is that it's not a specific string that caused the problem, and in the setting where it happened, it seemed to happen with every string.
safe/unsafe call types I also tried, didn't make a difference.
I switched to the vanilla jni implementation for now, because the parts where I need jni are rather small and self contained anyway,

When converting the corresponding Vec, it throws

This is definitely related to signature parsing in jni-rs

@Jazzpirate I know it's been a while since you posted this, but can you provide a full traceback  of the failed string conversion? Or a link to the whole project, if it's possible.

BTW, can't a pending exception be the case?
At the point the issue was created, robusta was v0.2, and it used jni-rs v0.17.0.

Hi, I appreciate you looking into this, but I unfortunately can't be of much help (anymore). I never pushed the code anywhere since I couldn't get it to work, and I switched to jni-rs directly soon after...

But may be you can remember if it's possible that you had a java exception thrown before you call MyStruct::message

Hmm, I'm fairly certain I didn't - the idea is that I wrote a TeX-to-HTML converter in rust, which I want to use from java and use functions such as the one sketched above to pass output messages back. So all the java code did was basically something like compile("foo.tex",mystruct), the rust converter would get going, and occasionally call mystruct.message("(/.../article.sty") or whatever. So there was little to throw errors on the java side in the first place, and little on the rust side to even get to the point where message would be called if anything failed before, if that makes sense...


Update merge rule for Tag_RISCV_priv_spec*

The merge rule is too restrict for now, relax the merge rule to
accept different version, but linker require to report error if
incompatible.

We decide deprecate that (#339) instead of update the merge rule, which decide in last psABI call.


[Event Request] Codeunit 12 Gen. Jnl.-Post Line - OnBeforeCalcPmtDiscPossible

Please add a new publisher event to the function CalcPmtDiscPossible as follows:
LOCAL CalcPmtDiscPossible(GenJnlLine : Record "Gen. Journal Line";VAR CVLedgEntryBuf : Record "CV Ledger Entry Buffer")
//>> Start New Publisher Call
OnBeforeCalcPmtDiscPossible(GenJnlLine, CVLedgEntryBuf, AmountRoundingPrecision, IsHandled);
IF IsHandled THEN
  EXIT;
//<< End New Publisher Call
WITH GenJnlLine DO
  IF "Amount (LCY)" <> 0 THEN BEGIN
    IF (CVLedgEntryBuf."Pmt. Discount Date" >= CVLedgEntryBuf."Posting Date") OR
.
.

New Publisher Event:
LOCAL [IntegrationEvent] OnBeforeCalcPmtDiscPossible(GenJournalLine : Record "Gen. Journal Line";VAR CVLedgerEntryBuffer : Record "CV Ledger Entry Buffer";RoundingPrecision : Decimal;VAR IsHandled : Boolean)

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


[Storage] [DataMovement] Update JobPartPlanFile to support FileShare related objects

Thanks

Closing issue wrong place.


Applications: Apply Filters to Service Detail Panel

User Story
When using the filter bar, we want to only show matching services on the right side detail panel. This should help users focus on only the services they are actually interested in; for example, when filtering on site, users only wish to see services from the given site.
Definition of Done

A service group where all services match the given criteria is shown unchanged compared to the unfiltered view, and all of its services appear in the right-side panel;
A service group where some of the services match is shown in the left side table, but the content is modified to include only the services matching the filter criteria; that is the Health, Service Group and Release columns reflect only the services that match the filter criteria. The right side services list should show only the services that matched the filter criteria.

Add @tarablack01 as assignee to track the front end development.


Merge nose options into Django 1.8 argparse-based options

When django.core.management.base.BaseCommand includes 'use_argparse',
then nose's optparse options are merged using argparse's
parser.add_argument in BaseCommand's overriden add_arguments method.
For Django 1.7 and earlier, the current .options method is used to set
the options.
Fixes #178.  Includes #194, to prove it works on Django 1.8.  Replaces #195.

@jbalogh @jsocol  Any chance of official 1.8 support?

:+1: This looks like exactly what is needed.  @jsocol please take a look when you have a moment.

Manually applying this patch fixes #178 for me. Running Django 1.8 on Ubuntu 14.04. :+1:

+1

I just tested locally on my project and everything seems to work. Any ETA on when this will be merged and made into a release?

+1

I'm getting the error: argparse.ArgumentError: argument -l/--debug: conflicting option string(s): --debug when using this branch with django-jenkins. See: https://github.com/kmmbvnr/django-jenkins/issues/282

@nikolas, it looks like you resolved https://github.com/kmmbvnr/django-jenkins/issues/282 by using the standard Django test runner.
Here's another way.  In a file like myapp/test_runner.py
from django_nose.runner import NoseTestSuiteRunner

class MyTestSuiteRunner:
    _argparse_remove_options = NoseTestSuiteRunner._argparse_remove_options + ['--debug']

and then in settings:
TEST_RUNNER = 'myapp.test_runner.MyTestSuiteRunner'

It's also possible that setting parser.conflict_handler = 'resolve' would do this automagically.  I may update the pull request.

@jwhitlock cool, thanks.

@jsocol Hey buddy, there are lots of people whining about Django 1.8 on #178, you think we could get this merged and get a new pypi release, so we can close #149 and #178? thanks!

@conrado Believe me, I know, and I wish, I wish, I wish I had more time every day to be more on top of these things, but work has just been taking up too much time for the past month.
I can't make promises about my calendar right now, but I do know. If folks are really unhappy and champing at the bit, and maybe we need to find another person to join the maintainer team, since so few of us have time day to day.

If you want to add me, I'll try not to break anything :pray:

@jsocol I would happily hop on the maintainer team as well if you need more people.  Let me know how I can help.

@conrado @adepue I've been added to the maintainers list.  Will get 1.4 out today.  Any other PRs that you've tested as ready to go, or good candidates for 1.4.1?


Fixing LaunchSpecifications should be an array

Fixing TypeError: <class 'troposphere.ec2.SpotFleetRequestConfigData'>: None.LaunchSpecifications is <type 'list'>, expected <class 'troposphere.ec2.LaunchSpecifications'>
Since LaunchSpecifications should be an array.
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-spotfleet-spotfleetrequestconfigdata.html#cfn-ec2-spotfleet-spotfleetrequestconfigdata-launchspecifications

This is a dup of #338 which just got merged (along with another fix). Thank you for the PR.


Support setting tedge configuration via environment variables

Is your feature improvement request related to a problem? Please describe.

It is difficult to configure thin-edge.io when using within a container (multi-process or single process container setups) as the configuration is file based, which normally requires users to create a custom init script, which writes the values to file before starting the service.
Describe the solution you'd like

Support controlling values from the tedge.toml using environment variables.
The environment variable name translation should:

Convert setting name to uppercase
Replacing all dots (.) with underscore (_) or double underscore __?
Use a common prefix TEDGE to namespace each variable

For example the mapping between setting key and environment variable COULD look like this:

Setting
Environment variable

c8y.url
TEDGE_C8Y_URL

device.key.path
TEDGE_DEVICE_KEY_PATH

device.cert.path
TEDGE_DEVICE_CERT_PATH

Configuration layering precedence
Since the settings can be set via multiple sources, the following details the precedence of settings, and the the first non-empty value should be used.

Read value from Environment variables (treat an empty value as not-set)
Read value from tedge.toml
Read from code default

User facing example
Once this is supported it would make it possible building a container image which can set all of the required values when starting the container, and the thin-edge component binary can be called directly (without requiring an intermediate step):
docker run \
    --env TEDGE_C8Y_URL=myurl.cumulocity.com \
    --env TEDGE_DEVICE_KEY_PATH=/cert-volume/tedge.key \
    --env TEDGE_DEVICE_CERT_PATH=/cert-shared-volume/tedge.pem \
    myimage \
    tedge-agent

Describe alternatives you've considered

Additional context

This pattern is very common and is supported by various RUST libraries such as:

https://github.com/mehcode/config-rs
https://github.com/SergioBenitez/Figment

Using _ as the separator is easy for simple cases such as C8Y_URL, but not fore DEVICE_KEY_PATH, which would translate to device.key.path, whereas we want device.key_path. Both of the libraries resolve this ambiguity (by default) by using . instead of _ as the separator. In general, they're following a two step process: convert the variables to some sensible default data structure, such as {c8y: {url: "myurl.cumulocity.com"}} and then attempt to convert this value into our configuration struct. If we want to use _ as the separator, but also have _ in the configuration keys, we'd basically just have to either:
(a): keep guessing until we stumble across a valid combination of . and _
(b): hardcode each of these variables in
If we were to use C8Y.URL and DEVICE.KEY_PATH as the variable names, that would make everything a lot easier (as we can just use the library rather than coming up with the logic ourselves).
Another thing to bear in mind is whether we want to prefix the variables (e.g. TEDGE_C8Y.URL or TEDGE.C8Y.URL). It probably doesn't matter in the context of a container, but it might make things easier if there's a use case for this configuration elsewhere, where we don't necessarily want to accidentally collide with an existing variable (although the use of . makes that unlikely).
I also find the idea of a config file overriding environment variables to be surprising, I'd expect it to be the exact opposite as I don't have to persist environment variable changes, but I do have to persist config changes.

We can use a double underscore separator or just forbid people to use underscores in a property name

(a): keep guessing until we stumble across a valid combination of . and _

Environment variables should not use . in them. Whilst technically it might work in some occassions, it is considered bad practice or not against linux spec. Reference

When working on #1773, I was also surprised to find that some keys like mqtt.external.port are saved into /etc/tedge/tedge.toml like this:
[mqtt]
external_port = 1234

instead of something like:
[mqtt.external]
port = 1234

or
[mqtt]
external.port = 1234

which I think is a little misleading. The first . denoted a table, why wouldn't the second? Not to mention TOML supports using dots inside keys (dotted keys). So we could theoretically replace all the _ in keys with ., which would make all keys trivially translatable to env variables, but we would need to add an alias for every field in DTO object that contains underscores.
It seems to me like we can't decide on an config structure and want to have it both ways. The major goal of TOML is to map unambiguously to a deep hash table, so we created structs to deserialize to, but then we made an API for accessing this config as a flat typemap, where you have to add trivial getters/setters for every field, and manually define a key that is used in CLI, but has no relation to how it will actually be saved in a config file. Because of that, adding new config settings is a pain, and I believe it could be made a bit simpler.

This pattern is very common and is supported by various RUST libraries such as:

https://github.com/mehcode/config-rs
https://github.com/SergioBenitez/Figment

I'd be for looking into if we could make use of some dedicated config crate, so we could eliminate some of the boilerplate that we have wrt. configuration options. But this needs to be considered in the context of our architecture and our past design decisions, keeping in mind who can write which configuration options, who can read which options, etc. and if these common solutions will fit our needs, so this should probably be a subject of another ticket.

I believe it could be made a bit simpler

Unfortunately true.
This crate has been implemented at a time when the requirements were full of constraints (as exposing this file to the user only with the  tedge config command - hiding the file and even its location). We added on purpose, but then abused, this ability to change our mind on a name exposed to the user without breaking a former configuration file. Meantime most of these constraints have been removed, but the abstractions are still here adding a lot of burden to add a config parameter or override those with environment variables as proposed here.

I'd be for looking into if we could make use of some dedicated config crate, so we could eliminate some of the boilerplate that we have wrt. configuration options. But this needs to be considered in the context of our architecture and our past design decisions, keeping in mind who can write which configuration options, who can read which options, etc. and if these common solutions will fit our needs, so this should probably be a subject of another ticket.

I would support such an effort. We need to stay calm though; and take some time to assess what we really need and what has to be done to convert old configuration files.

I'd be for looking into if we could make use of some dedicated config crate, so we could eliminate some of the boilerplate that we have wrt. configuration options. But this needs to be considered in the context of our architecture and our past design decisions, keeping in mind who can write which configuration options, who can read which options, etc. and if these common solutions will fit our needs, so this should probably be a subject of another ticket.

Beyond the boilerplate to add the settings for MQTT clients address and port, what bothers me in this PR
https://github.com/thin-edge/thin-edge.io/pull/1789, is the number of time we repeat ourselves to extract the MQTT connection settings from tedge.conf! This should be done in a single place!

I agree that this needs considering quite carefully before we attempt to fix the more structural issues. After all, it's near enough entirely accidental complexity in this case. The library I used for the environment variable handling, figment, is only really designed for reading the configuration files, and that's all I've tried to do with it here. At a minimum, we need to support derived/inferred "configuration" (such as the device id from the certificate) and writing to variables using tedge config set.
I still need to add documentation for the environment variables, but there is a lot of potential confusion as to the arbitrary relationship between config keys and dto keys. IMO it would make sense to preserve the structure of the underlying configuration, but when using tedge config, deprecate the use of e.g. mqtt.external.port in favour of mqtt.external_port.


Any comments on the processing overhead ? "Assigning queues, and queuing additional DPCs requires additional processing overhead."

Interested to see if there are any additional time bounds that can be incurred based on the additional processing overhead needed for assigning queues and queueing additional DPCs.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6b976194-de35-a0ee-7ea3-c7c1b1bf2e1f
Version Independent ID: 081748f7-b9e4-2929-9619-da94ed518946
Content: RSS with a Single Hardware Receive Queue - Windows drivers
Content Source: windows-driver-docs-pr/network/rss-with-a-single-hardware-receive-queue.md
Product: windows-hardware
Technology: windows-devices
GitHub Login: @duncanmacmichael
Microsoft Alias: dumacmic

Good question @jagdishgopalakrishnan, let me ask the NDIS team to see what they think.

Thanks, appreciate your help.
Sent from my iPhone
On May 30, 2019, at 4:50 PM, Duncan MacMichael <notifications<EMAIL_ADDRESS>wrote:
Good question @jagdishgopalakrishnanhttps://github.com/jagdishgopalakrishnan, let me ask the NDIS team to see what they think.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/windows-driver-docs/issues/1578?email_source=notifications&email_token=AMFFRU7AUAQZCDL4K5SPBCDPYBR5PA5CNFSM4HPTOPGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWT2IAA#issuecomment-497525760, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMFFRU44DDWBV7OV4A5XHILPYBR5PANCNFSM4HPTOPGA.

The docs say:

Assigning queues, and queuing additional DPCs requires additional processing overhead. To achieve improved system performance, this overhead must be offset by better utilization of available CPUs.

This remark means that RSS doesn't magically make things faster.  Actually, RSS only adds overhead, since now you're paying for the overhead (ISRs, DPCs, function calls, etc) on multiple processors.
The purpose of RSS is to spread load across multiple CPUs, when 1 CPU isn't able to handle all the workload.  The docs are saying that, if 1 CPU is enough to handle your workload, then don't enable RSS; you'll only spend more cycles getting the same work done.  As with any scale-out situation in any technology, scaling isn't perfectly linear.  If you add more CPU cores, you increase the maximum amount of workload that can get done, but at the expense of slightly higher overhead.

Thanks, understood your comment about RSS purpose and the workload spread.
I had a slightly different question about "Assigning queues, and queuing additional DPCs requires additional processing overhead" which I will try and rephrase.
It sounds like there are 3 different approaches according to my understanding:

RSS with single HW queue: Main ISR spawns off DPC, This reassigns buffers to CPU associated Receiver Queues and spawns yet another DPC on the CPU which has the receiver queue.
RSS with NIC supported HW queues but with single interrupt: Here NIC already copies to CPU associated receiver queue. But since there is only one interrupt source, the main ISR needs to spawn a DPC on the CPU whose queue is non-empty.
RSS with NIC supported HW queues but with MSI directly associated with CPU. Here directly the CPU of choice gets interrupted and a corresponding DPC on that CPU.

Approach 3 seems like the most desirable in terms of performance and efficiency as this saves the most time in terms of "Assigning queues, and queuing additional DPCs requires additional processing overhead". So just wanted to understand if you had any ballpark time savings that one might end up in approach 3, getting specifically from queueing and spawning additional DPCs.

Ah, gotcha.  First, your taxonomy is correct: those are the 3 different ways a NIC driver can do RSS.  (You could add an "approach 0" to your taxonomy: a NIC that only provides the RSS hash alongside each packet, but doesn't support any queues or multiple interrupts.)  Your conclusion is also correct: approach 3 is the most desirable for the OS.  (And between the first two, approach 2 is more desirable than approach 1.)
The drawback of these more desirable approaches is that they demand more of the hardware.  Approach 3 requires managing a collection of interrupt vectors and hardware queues, so it requires quite a bit more silicon than approach 1.
It's difficult to give you a one-size-fits-all benchmark, since every workload is different.  It'd be far more meaningful for you to identify the workload that's meaningful to you, and have you run the benchmark yourself.  But to give you some vague rules-of-thumb:

Approach 1 means you have serious scalability bottleneck on CPU0 (or whichever CPU dispatches the ISR).  This will limit the maximum throughput you can achieve to something like 100k packets/sec.
Approaches 1 and 2 add extra latency in scheduling the cross-processor DPC.  This varies wildly: if the system is only moderately loaded, the cross-processor DPC might only incur a few microseconds of delay.  But if the system is more heavily loaded, then you can start seeing an extra few milliseconds of wasted time.  This isn't the end of the world, but remember that TCP is fairly sensitive to its RTT estimation: if TCP thinks the RTT goes up, it'll clamp down on your throughput.

I don't want to paint a gloomy picture: there are commercial NICs that use all three approaches, and having any RSS support is better than nothing.  But if you have choices, approach 3 gives you better latency and better scalability.  The top-tier server-grade NICs all use approach 3.
In any case though, the ultimate proof is in the measurement.  A NIC driver might, on paper, use the fanciest RSS approach, but still might have inferior performance because it was implemented carelessly.   At the end of the day, you can use theory to guide your measurement strategy, but the ultimate decision-maker has to be real measurements on your actual workloads.
If you fire up Windows Performance Toolkit (wprui.exe / wpa.exe), you can see exactly how much time is spent between an ISR and DPC, or between one DPC and another DPC.  For example, on one of my test systems, there's no contention for processor 0, so the DPC scheduling delay is a fairly minimal 30 microseconds.  But if multiple devices are trying to put DPCs on that processor, you'd see that number go way up, since one DPC gets queued behind another.  (This particular NIC uses approach 3, so there's no 2nd DPC on a different processor.)

Thank you very much for a very detailed response,
For more specific questions on approach 3, I think its probably better if we can directly correspond with you (NDIS Driver Team) (jagdish.gopalakrishnan@intel.com). It would be great if we can internally facilitate a discussion.
Thanks in advance.

Great, thanks @jtippet for the great info. @jagdishgopalakrishnan I'll go ahead and close this issue since you are taking this offline with the NDIS team. Thanks!

Hello Duncan,
I know you closed this issue, but would appreciate contact info for direct correspondence<EMAIL_ADDRESS>with jtippet or other NDIS driver team members for further offline discussion on option 3.

Hi @jagdishgopalakrishnan,
I'm not a member of the NDIS team, but I'll forward your request to them for further discussion here.
Thanks!

Thanks very much.
From: Duncan MacMichael<EMAIL_ADDRESS>Sent: Tuesday, June 4, 2019 5:04 PM
To: MicrosoftDocs/windows-driver-docs<EMAIL_ADDRESS>Cc: Gopalakrishnan, Jagdish<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/windows-driver-docs] Any comments on the processing overhead ? "Assigning queues, and queuing additional DPCs requires additional processing overhead." (#1578)
Hi @jagdishgopalakrishnanhttps://github.com/jagdishgopalakrishnan,
I'm not a member of the NDIS team, but I'll forward your request to them for further discussion here.
Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/MicrosoftDocs/windows-driver-docs/issues/1578?email_source=notifications&email_token=AMFFRU5YSVJQBVF5DIA45X3PY37GJA5CNFSM4HPTOPGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW6GWSY#issuecomment-498887499, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMFFRU225XYYWMN7LUQDHWDPY37GJANCNFSM4HPTOPGA.


Bug fixes and fish

Hi. This is one more PR with some more important bug fixes. I also added Fish completion for read-shell-command and added myself as a second maintainer (😉).

Closing this one in favor of this one: https://github.com/LuigiPiucco/doom-emacs/pull/7.


Add ElasticsearchModel.elasticsearchable when elasticsearch is disabled

When elasticsearch module is disabled, tests return errors like:
AttributeError: "elasticsearchable" is not a valid field for <AssetGroup(guid=56618dc2-ab0d-46c5-aee1-c01fea174ce0, )>.

The easiest way is just to add elasticsearchable to the dummy
ElasticsearchModel class.

I need this fix on main.  One line change, looks great!


Microsoft.Extensions.Caching.StackExchangeRedis SetString BUG

My project uses Microsoft.Extensions.Caching.StackExchangeRedis as the implementation of distributed cache. When connected to Garnet, the Set action will return an ERR unknown command exception.
IDistributedCache distributedCache
distributedCache.SetString(key, value, new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = absoluteExpirationRelativeToNow });
ERR unknown command

Currently, garnet does not fully support all redis commands.

What commands are unsupported for this scenario?

See https://github.com/microsoft/garnet/issues/85

What commands are unsupported for this scenario?

You can see api-compatibility.

Can you provide a simple repro - it should be possible to detect what variant of SET is being sent and add that relatively easily.

Or, just run the Garnet server in Debug mode with these three lines uncommented and with --logger-level Trace, then it will tell you what was received and unable to handle:
https://github.com/microsoft/garnet/blob/main/libs/server/Resp/RespServerSession.cs#L241

All the details are on #85

The Redis IDistributedCache implementation depends on Lua
It doesn't need to, the area owner has raised an issue and PR https://github.com/dotnet/aspnetcore/issues/54685

Closing as the error will resolve once the above PR eliminating Lua merges.


please use lower case artifact id

http://search.maven.org/#search|ga|1|Agrona
capital "A" interferes with various build pipelines

Fixed with this commit. https://github.com/real-logic/Agrona/commit/f4f66e36e9efb68bc4acc052a4ecb509907b33e7

thank you


use tokio 1.33 which seems to build on windows.

Fix #45
Not sure if correct way to fix. And why it was resolved something else than latest?

Hey!
Thanks for the PR. Before I merge it in, I'll do a bit of research and ask around to figure out why this actually happens. My assumption would be that version requirement "1" would install the latest so there might be something else happening under the hood.
If you try to build it with version 1 (ie. current main branch), which version of tokio does it actually use?

Apparently the issue is that I haven't updated my Cargo.lock file in ages.
Could you try this:

Revert this commit (ie. make sure it says version = "1" for Tokio
Run cargo update
Try to build again.

Yes, this does the trick! I am able to build after upate.

Great! I'll close this PR and make a Cargo.lock update directly.
Thanks for reporting and helping me figure this out, much appreciated! Your contribution made me learn new things about Rust.


Update finagle-core, util-core to 19.2.0

Updates

com.twitter:finagle-core
com.twitter:util-core

from 19.1.0 to 19.2.0.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention @scala-steward in the comments below.
Have a nice day!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [{ groupId = "com.twitter" }]

Included in #98 (needed some code changes as well).


-P CLI argument does not work as expected

code-server version: Development
OS Version: MAC OS X 10.13.6

Description
According to code-server --help, this is the expected format for passing a password argument to the CLI:
-P, --password <value>   Specify a password for authentication.
Currently, the --password argument works. But -P does not.
Steps to Reproduce

Attempt to run code-server with a -P CLI argument

UPDATE: This appears to be a quirk of the commander package we're using for the CLI. All arguments that are set using a single dash and require CLI input cannot handle = symbols. E.G
code-server -P abcdefg  (This works)
code-server -P=abcdefg (This throws errors)
Which is a bit odd because
code-server --password abcdegf (works)
code-server --password=abcdegf (also works)

This quirk is consistent and repeatable with the -h (for host) and -p (for port) flags as well.


Small fixes for wasm article

CC: @petele

Whoops!
There were 5 warnings that will prevent this PR from being merged. Please take a look, and either fix, or provide a justification for why they can't be fixed.
WARNINGS
src/content/en/updates/2018/03/emscripting-a-c-library.md#L130 - '

:tada: This has been pushed live to https://developers.google.com/web/


Create command to update a PHAR file

Closes #1

Coverage remained the same at 100.0% when pulling 7df5385011afe7bcba7e2e25f63354709d71ef61 on ricardogobbosouza:issue-1/new-feature into 692ea32ce251951e353c7c6454ecbb0163eae575 on cknow:master.


Can't Import my Image in the polygon configuration

Hey @Wufe , love the library, I saw the deer example, and tried to convert my image into a .svg file for my personal website, but it didn't work, any advice or tips upon how to make a working svg from jpeg, or any tools where I can draw up a vector image of myself and convert it to .svg?

Hi @shrey
Inkscape (https://inkscape.org/release/inkscape-1.0/) is a good tool to create SVG if you want, I don't know if it converts other images too.
About the Polygon Mask feature your image should be composed with <path> tags, other tags would not be used, you can check the svg code to be sure


Make it impossible to accidentally publish without .js files

Change-type: patch

Merged #10 instead


Quotes variables

try assert cases
TEST="erert
TEST='erert

first case return null even if commented
second case return case as value with ' at end

Thanks for the report. That example is invalid syntax, and should be caught by the parser. This will be fixed in v3.6.8, v4.2.0, v5.2.1, when they are released later today.

Thanks for the report. That example is invalid syntax, and should be caught by the parser. This will be fixed in v3.6.8, v4.2.0, v5.2.1, when they are released later today.


Error when using alternative outputFileSystem (e.g. memory-fs)

Version
3.5.5
Reproduction link
https://github.com/clemyan/vue-cli-fs-repro
Environment info
  System:
    OS: Windows 10
    CPU: (8) x64 Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
  Binaries:
    Node: 11.7.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.12.3 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 6.5.0 - C:\Program Files\nodejs\npm.CMD
  Browsers:
    Edge: 42.17134.1.0
  npmPackages:
    @vue/cli-overlay:  3.5.1
    @vue/cli-service: ^3.5.0 => 3.5.3
    @vue/cli-shared-utils:  3.5.1
    @vue/component-compiler-utils:  2.6.0
    @vue/preload-webpack-plugin:  1.1.0
    @vue/web-component-wrapper:  1.2.0
    vue: ^2.6.6 => 2.6.10
    vue-hot-reload-api:  2.3.3
    vue-loader:  15.7.0
    vue-style-loader:  4.1.2
    vue-template-compiler: ^2.5.21 => 2.6.10
    vue-template-es2015-compiler:  1.9.1
  npmGlobalPackages:
    @vue/cli: Not Found

Steps to reproduce
yarn build
What is expected?
Build finishes without error
What is actually happening?
An error is throw after assets are emitted. All assets are emitted properly.

The error is thrown from @vue\cli-service\lib\commands\build\formatStats.js while trying to print emitted assets info, because it attempts to read the emitted files from the real fs while the Webpack compiler could be using an outputFileSystem that does not write to real fs.
Since formatStats is already passed a Stats instance, in Webpack 4 it can avoid reading the fs by reading the emitted code from the stats object directly.
I have a use case in which I want to use webpack plugins to generate alternative assets (e.g. zip files) and avoid the cost of writing the app to the fs by using a custom outputFileSystem.

Upon further investigation, the assets prop is considered internal and not part of the public API of Stats. So reading the emitted code from the stats object directly is not compliant.
I guess the "proper" way is to hook into emit using a Webpack plugin then.


Update references for operator v2.2.0

NOTE: This should NOT be merged until after we cut the new release. If you'd like to test out the steps, you can swap v2.2.0 for v2.2.0-beta.2 which is currently on master.

We've made a few changes for the v2.2.0 release of the public operator including the location of the pre-generated manifests. Both the CRD and the operator yaml files now live in the install directory on master.
This PR updates the version to v2.2.0 and updates the docs to point to the new locations.

This change is

Rebased and updated kubernetes-stop-cluster.md files to use the new paths as well


trying to fix mingw issues in CI

I wonder if this change was to blame for this?
https://github.com/microsoft/LightGBM/commit/3454698efe92ee57ea7afbcb3578f8850d132adb#diff-1e7de1ae2d059d21e1dd75d5812d5a34b0222cef273b7c3a2af62eb747f9d20a
(almost certainly not)


wagtail-4.1 documentation entry

Please check the following:

[ ] Do the tests still pass?[^1]
[ ] Does the code comply with the style guide?

[ ] Run make lint from the Wagtail root.

[ ] For Python changes: Have you added tests to cover the new/fixed behaviour?
[ ] For front-end changes: Did you test on all of Wagtail’s supported environments?[^2]

[ ] Please list the exact browser and operating system versions you tested:
[ ] Please list which assistive technologies [^3] you tested:

[ ] For new features: Has the documentation been updated accordingly?

Please describe additional details for testing this change.
[^1]: Development Testing
[^2]: Browser and device support
[^3]: Accessibility Target

Thanks @Dev-Liz  - merged in via a8abb58ccf69581c0a69091d37eba609bfcbdf48
Really appreciate you taking the time to get your head around the external docs approach and thanks for asking for help from others in the community!
Congrats on making your first contribution to Wagtail.
Just a note for future - it's good to create a new branch for each set of changes you are making - otherwise your PRs will contain work from other things (e.g. the docs reworking branch).

Thank you for the opportunity @lb- and thank you for the corrections too. They are all well noted.


Docs5

@alanvgreen , why is HTML better for images here?

@alanvgreen , why is HTML better for images here?

Oh, because they link to external images.


1.8

Closes #
Briefly describe your proposed changes:

[ ] Rebased/mergeable
[ ] Tests pass
[ ] http/swagger.yml updated (if modified Go structs or API)
[ ] Sign CLA (if not already signed)

Looks like this was opened on accident. We're not merging the 1.8 branch into master.


Fix: Flow reporting ID's were reversed in network SynAck

--> Issue: https://github.com/aporeto-inc/aporeto/issues/961

/build - automatically fired by gogo with following PRs and commit SHAs v1.0.0
[
  {
    "project": "fix-fr",
    "component": "enforcerd",
    "pr-id": "1028",
    "commit-sha": "eb5f4b5f031c3296464dc9f2570a92955010305e"
  },
  {
    "project": "fix-fr",
    "component": "trireme-lib",
    "pr-id": "737",
    "commit-sha": "f25d89e73f284625c97984c46a5e88744f20070e"
  }
]

/build - automatically fired by gogo with following PRs and commit SHAs v1.0.0
[
  {
    "project": "fix-fr",
    "component": "enforcerd",
    "pr-id": "1028",
    "commit-sha": "eb5f4b5f031c3296464dc9f2570a92955010305e"
  },
  {
    "project": "fix-fr",
    "component": "trireme-lib",
    "pr-id": "737",
    "commit-sha": "a367b213ba8fd9c832bc951b573e2e71c0db5ff8"
  }
]


add Erf operator

Hi @joddiy  https://github.com/apache/singa/issues/776 will this work for you?

@dcslin  thanks, shicong, let me check.

checking warnings

it seems that LGTM warnings and CheckLicense errors are not introduced by this PR

@dcslin Thanks shicong, it works fine.


Make post result as a shared method to support independent repository PR Gate

Support post result according to pull request info without manifest.
Because there is no manifest file for independent pull request.

BUILD on-build-config #784 : SUCCESS

This works for me :+1:


Minor error in the code's comment?

Thanks for releasing this great work! I think there is a typo here. It should be
alphas_cumprod = cumprod((1 - betas))

instead of
alphas_cumprod = cumprod(betas)

Thank you for pointing it out! fixed in 6620984


COMP: Remove circular include dependency

Reduce the number of files that need to be read by the pre-processor and remove the circular include dependancies.
The itkExceptionObject.h file was allowed to be included only by itkMacro.h which always included the file.
Moving the contents of itkExceptionObject.h to
itkMacro.h simplifies the dependency complexity.
itk::ExceptionObject is different from other classes because it depends on macros at the top of itkMacro.h for it's definition, but also is needed for many macros at the bottom of itkMacro.h.
ITK/Modules/Core/Common/include/itkExceptionObject.h:22:12: warning: circular header file dependency detected while including 'itkMacro.h', please check the include path [misc-header-include-cycle]
22 | #  include "itkMacro.h"
|            ^
PR Checklist

[x] No API changes were made (or the changes have been approved)
[x] No major design changes were made (or the changes have been approved)

I think smashing the file into Macro.h will make it less manageable. The circular dependency had been addresses with preprocessor commands. I am not clear what issue this is trying to address.  What effect will it have on Doxygen documentation?

@blowekamp I respectfully disagree.  I have fought with those header guards and managed the convoluted circular many times.  They break static analysis tools that just give up when the circular dependency is encountered.   IDE's can fail to identify the circular inclusions, or how to handle them.
It's probably not much of a speed increase, but it requires 3 file reads by the pre-processor rather than 1. (for every object file compilation that uses itkMacro.h).
To me this is the better choice.

@blowekamp I will try a different approach, but I think it will likely not work out.


[DONOTMERGE] byteliterals + escape sequence for starlark!

Closing in favor of https://github.com/verygoodsecurity/starlarky/pull/115


Error: Ring contains less than 4 coordinates and will not be imported.

Hello,
when i try import my gml file, i get this error

but when i upload my file cesium https://cesium.com/ion , it work properly

This is not a 3DCityDB issue, but a data issue. Check the definition of LinearRing from GML:

A LinearRing is defined by four or more coordinate tuples, with linear interpolation between them; the first and last coordinates shall be coincident. The number of direct positions in the list shall be at least four.

To define a surface, you need at least three coordinate tuples (triangle). Since the first and last coordinates must be coincident for a LinearRing, you need at at least four. At least some LinearRing elements in your data violate this rule. The Importer/Exporter automatically tries and fixes LinearRing elements for which the first and last coordinates are not coincident. But even that does not help in your dataset. So, please fix you LinearRing elements.
Since those elements are not valid surfaces, you do not see them in Cesium Ion. You only see the valid surfaces.

i try this file
FZK-Haus-LoD3-KIT-IAI-KHH-B36-V1.zip
but i get this result

i try in cesium, this is the result

is my import/export setting wrong?

Yes, the file works fine for me:

Note that the building has height values, and these height values are written to the visualization model by default. Thus, if you  do not use a matching terrain in the viewer, the builing might fly over or lie below the earth surface. You can use the "VIS Export - Elevation" preferences on the "Preferences" tab to influence the exported height values. In my above screenshot, I did not load a terrain, so I checked the option "Move every object on the globe (zero elevation)". With this option, the building is nicely located on the flat globe.
More information can be found in the user manual: https://3dcitydb-docs.readthedocs.io/en/release-v4.3.0/impexp/export-vis.html


Consider Citron for parsing

I found this to be a pleasure to use and it
reduced the amount of code in my project considerably over the corresponding
hand-coded parser.

The current implementation uses a custom data structure as it parses declarations to configure scopes (i.e., assign Decl.parentDeclSpace). Would it be possible to reproduce that behavior with Citron?
I'm also a little concerned about error messages. The current implementation purposely parses a superset of Val's actual grammar so that the parser can identify common errors (e.g., missing annotation in a parameter declaration, method qualifier on free function, etc.). How would you handle this kind of errors with Citron?


Site conditional parameters in yt-dlp.conf

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm asking a question and not reporting a bug or requesting a feature
[X] I've looked through the README
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've searched known issues and the bugtracker for similar questions including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Please make sure the question is worded well enough to be understood
In my yt-dlp.conf I want to automatically sign into youtube with oauth2:
--username oauth2 --password ''
But this will then cause links from other video hosting sites to print the error:
Login using username and password is not currently supported.
How can I setup my conf file so it only applies these parameters when the url is youtube?
Provide verbose output that clearly demonstrates the problem

[ ] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[ ] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
No response

Use a netrc file: https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#authentication-with-netrc

Thanks!


About the duplicated device

What seems to be 'duplicated devices' on our frontend is actually somewhat an extended feature from Linux Kernel.

Because the user can't use (I'm assuming this based on my tests) both videos from an unique bus, and the commands from one affects the others, I think we should deal with it, and use it in a more concise way.
My initial thought on that is that maybe we could merge all the /dev/video* that shares the same bus, but probably we can't simply merge the controls/capabilities and expect it to works using the devices interchangeably - lets say video0 has JPEG encode, and to the video2 has H264, but if we merge both and keep only the video0 reference, can we use video0 as H264? Probably not, then we would need to add an abstraction layer just to map each capability to its original device (in this example, map JPEG to video0 and H264 to video2).
Maybe we could change our model to something that encapsulates each device inside each bus:
{
'': {
    'bus': 'usb-0000:09:00.3-2.4',
    'devices': [
      {
        'name': 'Video Capture 6',
        'device_path': '/dev/video2',
        ...
      },
      {
        'name': 'Video Capture 5',
        'device_path': '/dev/video0',
        ...
      }
    ]
    'controls': ...,
    'formats': ...,
    ...
  }
}

Do we want to deal with it, or we'll let it to the user? Any ideas?

Usually each device has different capabilities, as video stream types and etc.


Fix: (node:73) Warning: a promise was rejected with a non-error: [object String]

[x] Fixed warning about rejecting with non-error (was spamming in console)
(node:73) Warning: a promise was rejected with a non-error: [object String]
[x] Fixed package.json indentation (tabs -> spaces), so now it looks better even in GitHub source viewer
[x] Regenerated package-lock.json just for better freshness

I'm going to send another PRs soon (maybe few days or even weeks), because I think we should remove bloated in size request package from dependencies, standardize indentation style and code itself.
But for now, just waiting for your review or merging @sonicsnes

Cherry-picked the incorrect error throwing part of this. Thanks for the contribution!


Duolingo Pro doesnt press the start lesson button, had to press it manually.

https://media.discordapp.net/attachments/995904492988006531/1214123557328519209/v64AM6e.png?ex=65f7f7ad&is=65e582ad&hm=5d2c2c5d28abc74edc5a0aa55a8261c68e5dd6d9b496dc273fe7ad2d29b02a49&=&width=490&height=350
No idea why I cant submit images, but its the button beside the Solve button

try updating to 9.6.1, should have a fix for the issue
btw thanks for reporting it


this brings the package up to the GHC 8.4 level

Thanks @ababkin !  This PR is no longer in sync as changes happened to the released version.  Maybe parts can be salvaged.


Help to identify the ploidy

Hi,
Thanks for developing this great tool.
I'm trying to use GenomeScope 2 to identify the ploidy of two plant samples.
The result of sample A indicates that it's a tetraploid. But the result of sample B is not that typical. If sample B is a tetraploid, is the reads duplication rate of these two samples that cause the difference in their GenomeScope plots? (The sequencing data of the two samples is the same, about 20G. ）

I agree sample A looks tetraploid. B is unusual. Is it from a single plant
or a mixture? If it is mixed that can really complicate the analysis. Have
you tried smudgepairs? That can sometimes help as well
Good luck!
Mike
On Mon, Feb 27, 2023 at 4:58 AM Huyuxi08 @.***> wrote:

Hi,
Thanks for developing this great tool.
I'm trying to use GenomeScope 2 to identify the ploidy of two plant
samples.
The result of sample A indicates that it's a tetraploid. But the result of
sample B is not that typical. If sample B is a tetraploid, is the reads
duplication rate of these two samples that cause the difference in their
GenomeScope plots? (The sequencing data of the two samples is the same,
about 20G. And the genome size is 950 Mb)
[image: 5F6FA29D-34B3-4C53-A5EE-ED42772C88B4]
https://user-images.githubusercontent.com/60483748/221532396-eb22b465-4890-4e96-8dd7-75e10da7619c.jpeg
—
Reply to this email directly, view it on GitHub
https://github.com/schatzlab/genomescope/issues/92, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABP347SARZS7CGJF5XI46LWZR3DVANCNFSM6AAAAAAVJFVKAA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Server Crash

When I right click with a hoe on a crop (Copper) the server crash.
I did'nt tested with other crops

And your version is?

1.7.10

Unsuppurted.


Race condition when using config store backed by config maps

Expected Behavior
When I try to use a config store backed by a config map immediately after the process using it starts up, when the config map exists before the process starts, I'd like to be able to guarantee that I am reading the configured value.
I think we could do this if one of the following was true:

When the reconciler starts, it could wait for the config store to be initialized, i.e. wait for the goroutines that handle events to fire at least once (see https://github.com/knative/pkg/issues/1907#issuecomment-733254238 for a description of this code) before returning
Provide a way for code using this reconciler to wait for this state (e.g. maybe don't do (1) by default, but at least provide a way for a caller to do this, e.g. via the Reconciler object)
Export the config store so that I can look at the state

/kind bug
Actual Behavior
When I use a generated reconciler, which has an unexported configstore, if I try to back that config store with configmaps, there is a race condition where if I try to use that config store immediately after start up, the informed watcher's goroutines which read from the backing config maps and update the config store may not have fired yet, which means I might read something from the config store other than what I've configured.
Steps to Reproduce the Problem

I believe https://github.com/knative/pkg/issues/1907 was actually catching this bug (explanation at https://github.com/knative/pkg/issues/1907#issuecomment-733211228)
I repro-ed this on the Tekton side with these horrible hacks: https://github.com/bobcatfish/pipeline/commit/1fbc3889b8ab1fbdcb2f0510a8c0674b436c3bf7 (trying to more reliably reproduce https://github.com/tektoncd/pipeline/issues/2815)

Additional Info

More detail from a Tekton perspective in https://github.com/tektoncd/pipeline/issues/2815#issuecomment-733207368

@dprotaso @mattmoor @runzexia do you have any thoughts on what the best approach is to take here? Are you open to the knative/pkg controller code doing something to wait and make sure the configstores are initialized correctly? (This seems like the best solution to me cuz this bug would impact all users of this lib)
(some other ideas listed above)

@dprotaso @mattmoor @runzexia do you have any thoughts on what the best approach is to take here? Are you open to the knative/pkg controller code doing something to wait and make sure the configstores are initialized correctly? (This seems like the best solution to me cuz this bug would impact all users of this lib)
(some other ideas listed above)

Friendly ping, I also noticed this in https://github.com/imjasonh/tektoncd-concurrency

Based on the discussion in the linked issue, I'd change this check to not be based on the informer cache, but based on a positive hand-off signal from the callback: https://github.com/knative/pkg/blob/c5296cd61e9f34e183f8010a40dc866f895252b9/configmap/informed_watcher.go#L175-L176
I suspect we could do this just by instrumenting add here: https://github.com/knative/pkg/blob/c5296cd61e9f34e183f8010a40dc866f895252b9/configmap/informed_watcher.go#L160

@mattmoor sounds like it makes sense to resolve this in knative/pkg 👍
I'd be happy to make PR but I need a bit more detail about what you're envisioning (sounds like we'd be creating an interface where the callbacks can signal, maybe via a channel, that they're ready?) - lemme know if it makes more sense to schedule some time to discuss or if it makes sense for me to bring up in a knative WG

IDK that we need to change the interface, since we are responsible for calling each callback.  I was just thinking about some internal bookkeeping to check that all of the callbacks had been called...? 🤷
I'm happy to chat, or we can throw it on an API WG agenda.

@mattmoor I was gonna join a working group but I think I figured out what you meant - lemme know if https://github.com/knative/pkg/pull/2036 is the kind of thing you had in mind :D ❤


updating master

Codecov Report

Merging #66 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master      #66      +/-   ##
==========================================
+ Coverage   99.52%   99.53%   +0.01%
==========================================
  Files          28       28
  Lines         629      644      +15
  Branches       77       78       +1
==========================================
+ Hits          626      641      +15
  Misses          3        3

Impacted Files
Coverage Δ

src/components/DatatableCore/Body/BodyRow.js
100% <100%> (ø)
:arrow_up:

...c/components/DatatableCore/Body/BodyActionsCell.js
100% <100%> (ø)
:arrow_up:

src/proptypes.js
100% <100%> (ø)
:arrow_up:

src/redux/reducers/datatableReducer.js
99.45% <100%> (+0.02%)
:arrow_up:

src/redux/actions/datatableActions.js
100% <100%> (ø)
:arrow_up:

src/components/DatatableCore/Body/Body.js
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17c3105...9df4d6a. Read the comment docs.


[SPARK-35102][SQL] Make spark.sql.hive.version read-only, not deprecated and meaningful

What changes were proposed in this pull request?

Firstly let's take a look at the definition and comment.
// A fake config which is only here for backward compatibility reasons. This config has no effect
// to Spark, just for reporting the builtin Hive version of Spark to existing applications that
// already rely on this config.
val FAKE_HIVE_VERSION = buildConf("spark.sql.hive.version")
  .doc(s"deprecated, please use ${HIVE_METASTORE_VERSION.key} to get the Hive version in Spark.")
  .version("1.1.1")
  .fallbackConf(HIVE_METASTORE_VERSION)

It is used for reporting the built-in Hive version but the current status is unsatisfactory, as it is could be changed in many ways e.g. --conf/SET syntax.
It is marked as deprecated but kept a long way until now. I guess it is hard for us to remove it and not even necessary.
On second thought, it's actually good for us to keep it to work with the spark.sql.hive.metastore.version. As when spark.sql.hive.metastore.version is changed, it could be used to report the compiled hive version statically, it's useful when an error occurs in this case. So this parameter should be fixed to compiled hive version.
Why are the changes needed?

spark.sql.hive.version is useful in certain cases and should be read-only
Does this PR introduce any user-facing change?

spark.sql.hive.version now is read-only
How was this patch tested?

new test cases

Test build #137458 has started for PR 32200 at commit 639c998.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42034/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42034/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42034/

Test build #137462 has started for PR 32200 at commit f4fd608.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42037/

Test build #137458 has finished for PR 32200 at commit 639c998.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42037/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42037/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137458/

Test build #137462 has finished for PR 32200 at commit f4fd608.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137462/

cc @cloud-fan @maropu @dongjoon-hyun @HyukjinKwon thanks

Test build #137543 has started for PR 32200 at commit 40f5aae.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42118/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42118/

It is marked as deprecated but kept a long way until now. I guess it is hard for us to remove it and not even necessary.

Moving it into a static config looks fine. But, we cannot really remove it, e.g., in the next major update (Spark v4.0)?

IMHO, it might be better to keep it. Once the metastore.version is changed, this can be still used to report the built-in Hive version

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42118/

Test build #137575 has started for PR 32200 at commit f4fd608.

Test build #137543 has finished for PR 32200 at commit 40f5aae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137543/

Test build #137575 has finished for PR 32200 at commit f4fd608.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137575/

Test build #137599 has started for PR 32200 at commit 40f5aae.

Test build #137599 has finished for PR 32200 at commit 40f5aae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137599/

thanks, merging to master!


feat: deploy front-end in Vercel

I will provide access to the Vercel account for the one willing to help configure and deploy the app into Vercel.

For now it is deployed on my forked repository.
Deployment file for Vercel was created.

URL do ambiente UAT (dev):
https://web3task-front-uat.vercel.app/


Release version 0.2.0.

Use this issue to ask questions or give feedback on the the Node UDT pre-release. Follow this issue for changes.

How do we use this library?