high computing time with categorical features

Hi,
i'm using new cat parameter feature_types in xgboost 1.7,
training a xgbclassifier in a dataset of ~600k rows, >200 columns, with all numeric features, and tree_method = 'hist', takes about ~30min of training time in cpu.
Adding a single categorical feature, with 10 categories, training time goes up to 2h 30min.
Adding 10 features, with ~10 categories each, training time went up >> 12h, so i have to stop the machine.
IS this normal behavior or am i doing something wrong?
code
    # X : pd.DataFrame ~ 600k x 200, y ~ 600k x 1

    pipeline = []
    args_ordinal_encoder = dict(handle_unknown='use_encoded_value',
                                unknown_value=np.nan)
    transformers  = []
    transformers += [('num', "passthrough", variables_num)]
    transformers += [('str', OrdinalEncoder(**args_ordinal_encoder ), variables_str]

    preprocess = ColumnTransformer(transformers)
    pipeline += [('preprocess', preprocess)]
    # len(variables_num)~200, len(variables_str) =1
    feature_types = ['float']*len(variables_num) + ['c']*len(variables_str)
    learner = xgb.XGBClassifier(
        eval_metric="auc",
        tree_method='hist',
        enable_categorical=True,
        max_cat_to_onehot=4,
        feature_types=feature_types
    )

    pipeline += [('learner', learner)]
    pipeline = Pipeline(pipeline)

    pipeline.fit(X, y)

machine
machine_type: "e2-highmem-8"

interesting. I will try to generate a similar dataset to reproduce.

Initial run with 50/50 split of categorical data and numeric data generated by  https://github.com/dmlc/xgboost/blob/bf8de227a9014013ab973af5706ccf2d2efc4989/python-package/xgboost/testing/init.py#L482 doesn't show any significant performance issue. I finished training 100 estimators with 10 classes in 160 seconds on my laptop. Could you please check whether the time is spent on the ordinal encoder with your dataset?

Hi,
playing around, long execution times problem seems to appear when the categorical feature is 100% np.nan.
Basically, running code above for examples 1 & 2, execution time is ~2 min (like your time).
But for example 3, execution time is ~1 hour.
Questions

do you experience the same issue?
in case yes
should xgboost raise any warning?
even if it's not good practice to give dummy all nan features, why execution time goes up? shouldn't be the same?
example 1

x, y = make_classification(int(7e5), 200,  n_informative=10, n_classes=3)
x = pd.DataFrame(x)
x.columns = [f"x{v}" for v in range(200)]
y = pd.Series(y)
x['cat'] = np.random.randint(10, len(x))

example 2
example 1 (...)
loc_= int(len(x)*0.8)
 x.loc[0:loc_, 'cat'] = np.nan

example 3
example 1 (...)
 x['cat'] = np.nan

When you assign nan to the dataframe, wouldn't it make the column as numeric type instead of categorical data type?

also, I tried to do the same and the time seems normal to me.

see below end-to-end code to reproduce results.
I see,
execution time without nan: 7s
execution time with >99% nan: 1 min
import datetime as dt

import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.compose import ColumnTransformer
from sklearn.datasets import make_classification
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OrdinalEncoder

def xgb_cat():
    print('xgb cat')
    num_rows = int(1e5)
    num_cols = 200
    x, y = make_classification(num_rows, num_cols,
                               n_informative=180, n_classes=3)
    x = pd.DataFrame(x)
    print(f"{x.shape=}")
    num_vars = [f"x{v}" for v in range(num_cols)]
    x.columns = num_vars
    y = pd.Series(y)

    pipeline = []
    transformers = [('num', "passthrough", num_vars)]
    transformers += [('str', OrdinalEncoder(), ['cat'])]

    preprocess = ColumnTransformer(transformers)
    pipeline += [('preprocess', preprocess)]
    feature_types = ['float'] * num_cols + ['c']
    learner = xgb.XGBClassifier(
        eval_metric="auc",
        tree_method='hist',
        enable_categorical=True,
        max_cat_to_onehot=3,
        feature_types=feature_types
    )

    pipeline += [('learner', learner)]
    pipeline = Pipeline(pipeline)

    x['cat'] = [f'cat_{v}' for v in np.random.randint(10, size=len(x))]
    start_ = dt.datetime.now()
    print('cat without nan')
    print('start fit')
    pipeline.fit(x, y)
    end_ = dt.datetime.now()
    print(f"execution time: {end_ - start_}")
    # execution time: 7s

    x['cat'] = np.nan
    x.loc[0:99, 'cat'] = [f'cat_{v}' for v in np.random.randint(10, size=100)]
    start_ = dt.datetime.now()
    print('cat with >99% nan')
    print('start fit')
    pipeline.fit(x, y)
    end_ = dt.datetime.now()
    print(f"execution time: {end_ - start_}")
    # execution time: 10min
xgb_cat()

Unfortunately, I'm still not able to reproduce the significant slow down. Indeed there's a 6 seconds gap, which is likely caused by XGBoost training a different model to handle smaller number of useful features, but nothing abnormal.
xgb cat
x.shape=(100000, 200)
cat without nan
start fit
execution time: 0:00:20.080911
cat with >99% nan
start fit
execution time: 0:00:26.207889

numpy: 1.23.1
xgboost: 1.7.1
sklearn: 1.1.1

On 11/5/22 2:08 PM, iuiu moflets @.***> wrote:

see below end-to-end code to reproduce results.
I see,
execution time without nan: 7s
execution time with >99% nan: 1 min
|import datetime as dt import numpy as np import pandas as pd import
xgboost as xgb from sklearn.compose import ColumnTransformer from
sklearn.datasets import make_classification from sklearn.pipeline import
Pipeline from sklearn.preprocessing import OrdinalEncoder def xgb_cat():
print('xgb cat') num_rows = int(1e5) num_cols = 200 x, y =
make_classification(num_rows, num_cols, n_informative=180, n_classes=3)
x = pd.DataFrame(x) print(f"{x.shape=}") num_vars = [f"x{v}" for v in
range(num_cols)] x.columns = num_vars y = pd.Series(y) pipeline = []
transformers = [('num', "passthrough", num_vars)] transformers +=
[('str', OrdinalEncoder(), ['cat'])] preprocess =
ColumnTransformer(transformers) pipeline += [('preprocess', preprocess)]
feature_types = ['float'] * num_cols + ['c'] learner =
xgb.XGBClassifier( eval_metric="auc", tree_method='hist',
enable_categorical=True, max_cat_to_onehot=3,
feature_types=feature_types ) pipeline += [('learner', learner)]
pipeline = Pipeline(pipeline) x['cat'] = [f'cat_{v}' for v in
np.random.randint(10, size=len(x))] start_ = dt.datetime.now()
print('cat without nan') print('start fit') pipeline.fit(x, y) end_ =
dt.datetime.now() print(f"execution time: {end_ - start_}") # execution
time: 7s x['cat'] = np.nan x.loc[0:99, 'cat'] = [f'cat_{v}' for v in
np.random.randint(10, size=100)] start_ = dt.datetime.now() print('cat
with >99% nan') print('start fit') pipeline.fit(x, y) end_ =
dt.datetime.now() print(f"execution time: {end_ - start_}") # execution
time: 10min xgb_cat() |
—
Reply to this email directly, view it on GitHub
https://github.com/dmlc/xgboost/issues/8416#issuecomment-1304418980,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AD7YPKM23MCJXSNEN723OWTWGX2XDANCNFSM6AAAAAARV53VKA.
You are receiving this because you commented.Message ID:
@.***>

Other than the result from above comment, I also put a callback into xgboost to check the fit time, maybe you can try it as well:
import datetime as dt

import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.compose import ColumnTransformer
from sklearn.datasets import make_classification
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OrdinalEncoder
import sklearn

class TimeReport(xgb.callback.TrainingCallback):
    def __init__(self):
        self.start = None
        self.end = None

    def before_training(self, model):
        self.start = dt.datetime.now()
        return model

    def after_training(self, model):
        self.end = dt.datetime.now()
        print("XGBoost fit Duration:", self.end - self.start)
        return model

def xgb_cat():
    print("xgb cat")
    num_rows = int(1e5)
    num_cols = 200
    x, y = make_classification(num_rows, num_cols, n_informative=180, n_classes=3)
    x = pd.DataFrame(x)
    print(f"{x.shape=}")
    num_vars = [f"x{v}" for v in range(num_cols)]
    x.columns = num_vars
    y = pd.Series(y)

    pipeline = []
    transformers = [("num", "passthrough", num_vars)]
    transformers += [("str", OrdinalEncoder(), ["cat"])]

    preprocess = ColumnTransformer(transformers)
    pipeline += [("preprocess", preprocess)]
    feature_types = ["float"] * num_cols + ["c"]
    learner = xgb.XGBClassifier(
        eval_metric="auc",
        tree_method="hist",
        enable_categorical=True,
        max_cat_to_onehot=3,
        feature_types=feature_types,
        callbacks=[TimeReport()],
    )

    pipeline += [("learner", learner)]
    pipeline = Pipeline(pipeline)

    x["cat"] = [f"cat_{v}" for v in np.random.randint(10, size=len(x))]
    start_ = dt.datetime.now()
    print("cat without nan")
    print("start fit")
    pipeline.fit(x, y)
    end_ = dt.datetime.now()
    print(f"execution time: {end_ - start_}")
    # execution time: 7s

    x["cat"] = np.nan
    x.loc[0:99, "cat"] = [f"cat_{v}" for v in np.random.randint(10, size=100)]
    start_ = dt.datetime.now()
    print("cat with >99% nan")
    print("start fit")
    pipeline.fit(x, y)
    end_ = dt.datetime.now()
    print(f"execution time: {end_ - start_}")
    # execution time: 10min

xgb_cat()
print(
    "numpy:",
    np.__version__,
    "\nxgboost:",
    xgb.__version__,
    "\nsklearn:",
    sklearn.__version__,
)

adding the callback, i see that execution time comes from the mapping from string to int in the ordinalencoder.
This is sklearn stuff and not xgboost, so i'll open the ticket there.
If i use cat as numeric, this doesn't happen either
xgb cat
x.shape=(100000, 200)

cat without nan
start fit
XGBoost fit Duration: 0:00:07.707660
execution time: 0:00:08.859463

cat with >99% nan
start fit
XGBoost fit Duration: 0:00:09.214314
execution time: 0:00:11.692530
numpy: 1.23.4
xgboost: 1.7.1
sklearn: 1.1.3

@iuiu34 It's probably a good idea to use separate estimators instead of a pipeline to pinpoint the overhead. There are many data conversions happening there, ordinal encoding, type change, and xgboost also need to iterate through the columns to obtain categorical codes. You can try to python profiler if necessary. (assuming that the ratio of missing > 99% is normal in your use case instead of a data engineering bug).

yepp, you're right.
thanks for the support & quick responses.
In case anyone faces same problem i.e. encode categorical features very sparsed in big datasets.
Here is the issue in sklearn#24840
And as fix, withcategory_encoders.ordinal.OrdinalEncoder problem doesn't happen

Thank you for sharing! Hope that the issue will be resolved soon. ;-)


All Creative+ Non-Inventory Buttons Delete Items

The crafting slots, armour slots, and even the Curio button registers as an item deletion slot:

This feels similar to what causes #7.

This is because the JEI favorites menu is overlapping the left side of the GUI. Has nothing to do with me; I'm properly reporting my GUI's regions to JEI.

I'll double-check the JEI integ to be sure.

Huh, I didn't even notice that lol. At the very least could the UI be 'layered above' such that you can't click through to the favourites menu underneath?

No, I have no control over where JEI decides its regions are, and they're always "above" the inventory.


Properly parse struct definition by rejecting invalid public identifiers, Fixes: #787

Before, the struct definitions like:
pub struct Bar {
    pub x: i32
pub}

were not rejected by the parser. (as mentioned in #787 ). This pull request fixes this issue.
Also tests are written to validate this.
@Y-Nak

Also, could you add fix #787 to the PR comments so that the issue is automatically closed when the PR is merged?

@Y-Nak , I have updated the changes you suggested

Thanks!


expert mode

@janmazak I signed a few txs in expert and non-expert mode to make sure that everything looks as intended. I identified a few problems:

The app crashes when signing the "Sign tx with datum hash in output" Ledgerjs test case in expert mode.
When signing a Catalyst registration tx as non-expert, the initial Register Catalyst voting key? prompt is missing, and the aux data hash is shown despite it shouldn't be shown to non-experts.
Required signers should be allowed in ordinary txs in Ledgerjs, too.

Merged by Ledger, additional fixes are in https://github.com/vacuumlabs/ledger-app-cardano-shelley/pull/147


VM: Allow trailing commas in parameter and argument lists.

This is the VM-specific issue for #26644. That issue has the details.
See https://codereview.chromium.org/1991043004/ for a proof-of-concept implementation.

Landed.


Consider supporting bmp format

I noticed that some of my images do not appear in the gallery... Looking closer it seems to be because they are all bmp format.
The list of supported formats from the readme does not include bmp, however you have mentioned here and here that they should be supported.
I am not sure whether this is a bug or by design, but would you consider supporting bmp images for future versions?

Sharp (the used image manipulation lib) does not support it out of the box:
https://github.com/lovell/sharp/issues/806
https://github.com/lovell/sharp/issues/543
You can tune the Docker file to build libvips with BMP support. If you
manage to make it work, feel free to send a PR.
-- Sorry for being brief, sent from my phone.
On Thu, 31 Aug 2023, 10:44 mrtnbr, @.***> wrote:

Did you add the BMP format to the Supported photo formats option in the
Settings/Admin page?
—
Reply to this email directly, view it on GitHub
https://github.com/bpatrik/pigallery2/issues/705#issuecomment-1700616903,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABZKA5QJIWTE44DB2XMVNKTXYBFFPANCNFSM6AAAAAA4E6756M
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


Fixed context menu and added missing items

This fixes the context menu so that it works again.
It's editor only. We can add it to the tree view, but I think if we do that we should change the options. Showing a "go to definition" doesn't make sense in the tree view.
Closes #109

Yeah, the label is what's shown. Omnisharp Atom makes sense. Or even just Omnisharp. What do the other editors use?


local operator catalog raw file based format

what

Locally built custom OLM catalogs use (raw) File Based Catalog format instead of the deprecated SQLITE format.
From SQLite-based catalogs,

The SQLite database format for Operator catalogs is a deprecated feature. Deprecated functionality is still included in OpenShift Container Platform and continues to be supported; however, it will be removed in a future release of this product and is not recommended for new deployments.

Update github actions to push file based catalog format images
Makefile

target renamed: deploy-olm -> deploy-catalog
target renamed: undeploy-olm -> undeploy-catalog
function renamed: go-get-tool -> go-install-tool
bundle-dependencies target removed. Functionality moved to the bundle target.
replace envsubst operations to update yaml objects with external env variarbles with the yq tool
moved all catalog related logic to make/catalog.mk
New: catalog target to build catalog manifests

Documentation oriented to contributors doc/development.md:

New section: Deploy kuadrant operator using OLM
New section: "Build custom OLM catalog`
Added Table of Contents.

TODO

[x] Doc
[x] github actions

Verification steps
Build local catalog. It will include kuadrant operator, limitador operator and authorino operator
make catalog
make catalog-build CATALOG_IMG=quay.io/kuadrant/kuadrant-operator-catalog:local-testing

Push the image to Quay.io. Currently it is required because the catalogsource sets the image pull policy to Always, so even if the image is loaded in the kind cluster, it will fail if the image is not in quay.
make catalog-push CATALOG_IMG=quay.io/kuadrant/kuadrant-operator-catalog:local-testing

Create kind cluster
make kind-create-cluster

Deploy OLM
make install-olm

Deploy kuadrant operator using OLM
make deploy-catalog CATALOG_IMG=quay.io/kuadrant/kuadrant-operator-catalog:local-testing

Check the kuadrant operator, limitador operator and authorino operator are up and running
k get pods -n kuadrant-system
NAME                                                              READY   STATUS      RESTARTS   AGE
43539917290efc2e6f33b021e25da1f7a9f150d8d276acf92fd09b--1-wqggr   0/1     Completed   0          34s
6427bb5aac2edc4750e505a33be4eb4b56923d3aed4213abecdc70--1-v4lk5   0/1     Completed   0          34s
aecaf21d9bde4883921cefdd0ba3a142f079e897473c5d40fd8a1a--1-2249v   0/1     Completed   0          34s
authorino-operator-7d9968586c-x4kbl                               1/1     Running     0          16s
kuadrant-operator-catalog-rkmdn                                   1/1     Running     0          79s
kuadrant-operator-controller-manager-6688b8bfdf-m4rwh             1/1     Running     0          13s
limitador-operator-controller-manager-7f5dcdd976-66bcs            1/1     Running     0          11s

sorry for the bit long PR :pray:

It looks good, minor things I commented. However, I've still haven't successfully tried it out, having network issues:
dd@pomodoro:~/Projects/RedHat/kuadrant-operator|local-catalog-file-based⚡ ⇒  make install-olm
./utils/install-operator-sdk.sh /Users/dd/Projects/RedHat/kuadrant-operator/bin/operator-sdk v1.22.0
Downloading https://github.com/operator-framework/operator-sdk/releases/download/v1.22.0/operator-sdk_darwin_arm64
gpg: connecting dirmngr at '/Users/dd/.gnupg/S.dirmngr' failed: IPC connect call failed
gpg: keyserver receive failed: No dirmngr
make: *** [/Users/dd/Projects/RedHat/kuadrant-operator/bin/operator-sdk] Error 2
dd@pomodoro:~/Projects/RedHat/kuadrant-operator|local-catalog-file-based⚡ ⇒  make install-olm
./utils/install-operator-sdk.sh /Users/dd/Projects/RedHat/kuadrant-operator/bin/operator-sdk v1.22.0
Downloading https://github.com/operator-framework/operator-sdk/releases/download/v1.22.0/operator-sdk_darwin_arm64
make: *** [/Users/dd/Projects/RedHat/kuadrant-operator/bin/operator-sdk] Error 92
dd@pomodoro:~/Projects/RedHat/kuadrant-operator|local-catalog-file-based⚡ ⇒  make install-olm
./utils/install-operator-sdk.sh /Users/dd/Projects/RedHat/kuadrant-operator/bin/operator-sdk v1.22.0
Downloading https://github.com/operator-framework/operator-sdk/releases/download/v1.22.0/operator-sdk_darwin_arm64
gpg: key 052996E2A20B5C7E: "Operator SDK (release<EMAIL_ADDRESS>2 new signatures
gpg: Total number processed: 1
gpg:         new signatures: 2
gpg: assuming signed data in 'checksums.txt'
gpg: Signature made Tue Jun 14 22:57:45 2022 CEST
gpg:                using RSA key 8613DB87A5BA825EF3FD0EBE2A859D08BF9886DB
gpg: Good signature from "Operator SDK (release<EMAIL_ADDRESS>[unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 3B2F 1481 D146 2380 80B3  46BB 0529 96E2 A20B 5C7E
     Subkey fingerprint: 8613 DB87 A5BA 825E F3FD  0EBE 2A85 9D08 BF98 86DB
operator-sdk_darwin_arm64: OK
/Users/dd/Projects/RedHat/kuadrant-operator/bin/operator-sdk olm install
INFO[0000] Fetching CRDs for version "latest"
INFO[0000] Fetching resources for resolved version "latest"
I1207 18:09:07.865914   53853 request.go:601] Waited for 1.047390292s due to client-side throttling, not priority and fairness, request: GET:https://<IP_ADDRESS>:61648/apis/certificates.k8s.io/v1?timeout=32s
INFO[0018] Creating CRDs and resources
INFO[0018]   Creating CustomResourceDefinition "catalogsources.operators.coreos.com"
INFO[0018]   Creating CustomResourceDefinition "clusterserviceversions.operators.coreos.com"
INFO[0018]   Creating CustomResourceDefinition "installplans.operators.coreos.com"
INFO[0018]   Creating CustomResourceDefinition "olmconfigs.operators.coreos.com"
INFO[0018]   Creating CustomResourceDefinition "operatorconditions.operators.coreos.com"
INFO[0018]   Creating CustomResourceDefinition "operatorgroups.operators.coreos.com"
INFO[0018]   Creating CustomResourceDefinition "operators.operators.coreos.com"
INFO[0018]   Creating CustomResourceDefinition "subscriptions.operators.coreos.com"
INFO[0018]   Creating Namespace "olm"
INFO[0018]   Creating Namespace "operators"
INFO[0018]   Creating ServiceAccount "olm/olm-operator-serviceaccount"
INFO[0018]   Creating ClusterRole "system:controller:operator-lifecycle-manager"
INFO[0018]   Creating ClusterRoleBinding "olm-operator-binding-olm"
INFO[0018]   Creating OLMConfig "cluster"
INFO[0022]   Creating Deployment "olm/olm-operator"
INFO[0022]   Creating Deployment "olm/catalog-operator"
INFO[0022]   Creating ClusterRole "aggregate-olm-edit"
INFO[0022]   Creating ClusterRole "aggregate-olm-view"
INFO[0022]   Creating OperatorGroup "operators/global-operators"
INFO[0022]   Creating OperatorGroup "olm/olm-operators"
INFO[0022]   Creating ClusterServiceVersion "olm/packageserver"
INFO[0022]   Creating CatalogSource "olm/operatorhubio-catalog"
INFO[0022] Waiting for deployment/olm-operator rollout to complete
INFO[0022]   Waiting for Deployment "olm/olm-operator" to rollout: 0 of 1 updated replicas are available
FATA[0120] Failed to install OLM version "latest": deployment/olm-operator failed to rollout: timed out waiting for the condition

Ok, finally managed to try it out and it works like a charm!
dd@pomodoro:~/Projects/RedHat/kuadrant-operator|local-catalog-file-based⚡ ⇒  k get pods -n kuadrant-system
NAME                                                              READY   STATUS      RESTARTS   AGE
43539917290efc2e6f33b021e25da1f7a9f150d8d276acf92fd09b--1-f9dvv   0/1     Completed   0          4m57s
6427bb5aac2edc4750e505a33be4eb4b56923d3aed4213abecdc70--1-rjxdt   0/1     Completed   0          4m57s
aecaf21d9bde4883921cefdd0ba3a142f079e897473c5d40fd8a1a--1-rsdzb   0/1     Completed   0          4m57s
authorino-operator-7d9968586c-r658g                               1/1     Running     0          3m38s
kuadrant-operator-catalog-krz9t                                   1/1     Running     0          5m40s
kuadrant-operator-controller-manager-6688b8bfdf-h9h6g             1/1     Running     0          3m36s
limitador-operator-controller-manager-7f5dcdd976-zcl5w            1/1     Running     0          3m41s


feature(main): replace render charts dir  to scripts dir

Signed-off-by: cuisongliu<EMAIL_ADDRESS>

Fix a bug?

Fix a bug?

No. In fact, it doesn't make much sense to render the charts directory. Helm itself already has the function of rendering. Optimized to the directory of rendering scripts.


test(Dropdown): correct a test with open on Space key

Codecov Report

Merging #3427 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3427   +/-   ##
=======================================
  Coverage   99.92%   99.92%
=======================================
  Files         172      172
  Lines        2838     2838
=======================================
  Hits         2836     2836
  Misses          2        2

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0d28f12...91fba52. Read the comment docs.

Released in<EMAIL_ADDRESS>


DynamoDB values are overwritten each time the app runs

Each invocation of the app will overwrite all records in the two DynamoDB tables with the default values (new records will no be altered).

I have a question about this issue, just to make sure I am understanding correctly. Every time the app lambda was invoked, this would overwrite all the default values like TTL, regions targeted, permanent whitelist items etc., but would keep new whitelist items that were added the same?
Just wanted to make sure since I am interested in overwriting the default values every time the lambda is invoked (but keeping new whitelist items the same) so this way these default values have to be managed through the code.

@mcueva10 the auto-cleanup-settings.json file contains a version key. That key is compared on the application run and the data is inserted if the key in the file is greater than in the table. This generally only occurs when I add support for a new service.
The whitelist is inserted each time.

Oh ok thanks


C++ Optimization Question

Hello I<3LA inventors,
we asked us why I<3LA does not support C++ code generation for optimization problems? Aren't there generic libraries for C++ like there are for Python and Matlab?
For a seminar where I chose this language as topic, one possible task could be to extend the optimization problem functionality for C++ like recognizing specific optimization problems and generating better performing optimization code.
I am also creating a VSCode extension. I start with syntax highlighting before I try more interesting features like execution, WYSIWYG-style (which I thought would help the usability a lot).
Is there a way, how you discuss things about the language other than GitHub issues? For example you could activate GitHub discussions?
Greetings from Germany

Thanks for your interest. We didn't add C++ code generation for optimization originally since Eigen/C++ didn't have builtin support.
There are some libraries for C++ such as LBFGS++ and NLOpt. We added local function definition and tried to add LBFGS++ on local_func branch (not finished yet).
We are working on other directions of the language and may revisit the optimization later. I've activated GitHub discussions.


How to depend on SDL2 (libGL error)

Issue description
I hope opening an issue for this is appropriate. I'm using pysdl2 (meaning I have pysdl2 in my propagatedBuildInputs), which depends on SDL2, of course. Initialization of SDL2 fails because it cannot find libGL.so.1.
Looking at SDL2's source code, I see that it tries to dlopen either "libGL.so.1" (hard-coded) or whatever is inside SDL_OPENGL_LIBRARY. To properly package my application, am I to create a wrapper script for my application that sets this environment variable? Or what's the recommended approach when depending on libGL? I'm aware of PR 37369 (glvnd), but I'm not sure how to react to it.

Are you on non-NixOS Linux?

Nope, I'm on NixOS.

Then your LD_LIBRARY_PATH environment variable should contain /run/opengl-driver/lib:/run/opengl-driver-32/lib with libGL.so.1 in both of them. Is that so?

No, and it shouldn't contain that, see here.

Oops, I was not up to date on #37369. @abbradar, could you help?

maybe related https://github.com/NixOS/nixpkgs/issues/41730

SDL2 was not properly adapted for #37369, I'll fix it.

This broke glfw and/or libepoxy, whichever one tries to find libGL.so.1. It also broke zsnes.


Question about training

Dear Victor,
Again super cool work and it's awesome that your work online is so well documented and put together.
I have extended this model for multiple treatments T={t1,t2,t3..} (i.e. multinomial propensity prediction and binary outcome with |T| heads) and oversampling of minority treatment groups heavily changes causal estimates downstream than when I do not oversample minority classes.
I was wondering: Is there something theoretically incorrect about oversampling while looking at causality? or can I keep doing it because the results are more balanced?
Warm wishes and thank you for your time,
Shishir

Is your extension to allow for multiple possible contrasts or for multiple treatments applied over time? That is, are you considering treatments \in {0,1,2} and contrasts such as E[Y | do(T=2)] - E[Y | do(T=0)]? Or is it the case you'll apply multiple treatments to each unit?
In the former case, it might an issue of ATT vs ATE. Comping the former shouldn't be too sensitive to downsampling. The ATE might be though, because it'll change the population of people you're taking the expectation over.
Btw, we also ran into a use case where we needed to allow multiple treatments (and outcome missingness). The code for that is here: https://github.com/vveitch/causal-text-embeddings-tf2 You may find it helpful.
Best,
Victor

Dear Victor,
Sorry I might have misunderstood you but I think to answer your q: not over time, but after a static point in time, I have multiple potential outcomes. So in the original dragonnet scenario it's 2 treatments and 2 potential outcomes (so 2 cond outcome heads), and I'm doing 5 treatments and 5 potential outcomes (5 cond outcome heads). i believe it's what you're saying with multiple possible contrasts.
I thought about it after sharing as well; the data imbalance is a mess and some minority classes (treatments) have about 1% occurrence in the data. Anyway, I run the ATE on the entire test set - i just train with the minority oversampled batches. I'm thinking the latent space before propensity prediction is heavily biased by majority treatment/(s) if I do not do oversampling, but if I do, and I might be wrong here, but I feel it's a bit like a poor version of batch-wise, propensity matching. And so the causal estimates are quite stable - as in, low variance on tmle/naive and if i run 10 epochs/20 epochs of training the causal estimates don't move a lot.
And interesting, didn't know you had done a sim with multiple treatments! Thanks for sharing.
Warm wishes,
Shishir


Fragments stopped working after graphql update

Describe the issue
I updated graphql package from 3.1.0 to 4.0.1 and now GraphQL responses with fragments are not operated correctly. Here is an example on StackOverflow: https://stackoverflow.com/questions/70393062/json-fails-to-parse-graphql-response-after-flutter-libs-upgrade
I'm not sure if it's related directly to graphql update or is there something wrong with parsing the response with json libraries, but I hope anyone here knows what's going wrong. I checked all the changelogs and docs and didn't find any info about that.
To Reproduce
Steps to reproduce the behavior:

Update graphql from 3.1.0 to 4.0.1
Make a fresh build after flutter clean & updating the pods

Expected behavior
Old documents are working correctly and everything works as well as on 3.1.0
Actual behavior
Documents with fragments don't parse correctly anymore
device / execution context
iOS 14/15, also reproducible in Android
additional context
All the additional context is available in my StackOverflow post, please check it for code samples and screenshots
additional notes
I like everything but I hope it will work for me again :-)

I checked all the changelogs and docs and didn't find any info about that.

Maybe because the version that you are using it is the only one, there is a beta version available here https://pub.dev/packages/graphql/versions/5.0.1-beta.1 can you check with it?

@vincenzopalazzo I've just checked and 5.0.1-beta.1 also doesn't work

Mh, interesting. Any tips to reproduce the error in a local env?
Thanks

@vincenzopalazzo unfortunately the API I use is private and pretty hard to reproduce since there's authorisation needed. If there's any API you know where you can use fragments in queries you can try that.

Some app example contains fragments but I need to update them. I will put in my to-do list, but not sure that we can reproduce your error with another API. But we will see

I am also getting null when I use fragments.

@alitnk
Can you provide an small example? maybe we have a bug somewhere?

@vincenzopalazzo
This is the repository in which I'm facing the issue: https://github.com/matnbaz/matnbaz-mobile, look for FetchPolicy.noCache occurrences and try to remove them.
Let me know if this example has too much unrelated code or is just isn't good for testing the problem, I'd gladly make a small example.
And thank you for your time. the library is working perfectly, except for this issue.

@alitnk Thanks, this noCache workaround helped me as well

Let me know if this example has too much unrelated code or is just isn't good for testing the problem, I'd gladly make a small example.

If you can provide a small example will be perfect to me, and we can try to accelerate the debugging of it :) thanks

If you can provide a small example will be perfect to me, and we can try to accelerate the debugging of it :) thanks

There you go: https://github.com/alitnk/graphql_flutter_fragments_example
Worth mentioning that I'm using Artemis in this example, So the query document is auto generated.
Please mention me if there's an issue with the example repo.

Amazin, I will prioritize this issue in my queue to prove my gratitude to build this example.
Thanks!

The newer cache uses __typename to intelligently deconstruct and reconstruct the data using the normalize library, but it shouldn't try and do anything fancy without it.

Add __typename such as is supplied by the gql` helper
Try supplying possibleTypes to the cache (i.e. {"Role": {"Client"}})
See if normalize is 0.5.5. This issue maybe caused by some issue with fragments resolved in 0.5.5 (https://pub.dev/packages/normalize/changelog):
https://github.com/zino-hofmann/graphql-flutter/blob/f0793fd35279bb7948c53ce7204a343dca883705/packages/graphql/pubspec.yaml#L16

Ultimately the cache shouldn't do anything to query objects without __typename available to work with or customizations

@vincenzopalazzo, I've worked on improving how normalize parses fragments (see latest https://github.com/gql-dart/ferry/pull/300). As @micimize suggests, I expect this issue to be caused by this. In the coming versions of normalize, users will need to supply possibleTypes to the client. We'll need to update the documentation to reflect this. I'm using graphql_codegen to generate these types myself.

@asmodeoux Please try the following:

Add the normalize dependency to version 0.5.3, exactly.
Add __typename to all your query objects
Provide possibleTypes map. You can generate this with GraphQL codegen (https://pub.dev/packages/graphql_codegen)

I'm facing the same issue I didnt have it before 5.0. possibleTypes is null for all my types. the noCache workaround seems to work but i'd prefer to keep the cache. I'm using artemis btw
I'd appreciate any guidance. Thanks for all your hard work :)

Hi @msal4,
another solution can be to fix the version  normalize: 0.5.3, with dependeces override as suggested here

Add the normalize dependency to version 0.5.3, exactly.

We are working on a solution because normalize as a breaking change in the last version, so we are deciding if we need to work with an old solution or we need to update the doc to support the new one.
Sorry for that

Hey @vincenzopalazzo, I've already added normalize: 0.5.3, still facing the same issue.

I'm planning on working on this this weekend

@budde377 great!
yeah i know. I only use artemis for codegen I dont use their client. plus graphql_flutter works better with flutter.

I've created a PR outlining the steps necessary to get proper fragment resolution with the new version of normalize.
@alitnk Regarding your example, as far as I can see you're not providing the possibleTypes (here: https://github.com/alitnk/graphql_flutter_fragments_example/blob/master/lib/main.dart#L17) or introspection upon fragment spread (e.g. here https://github.com/alitnk/graphql_flutter_fragments_example/blob/master/graphql/queries/get-repositories.graphql#L21).
@msal4 Please check the documentation to see if it makes sense.

The new release is out https://github.com/zino-hofmann/graphql-flutter/releases/tag/v5.0.1-beta.3

@budde377 hey, sorry for the late response. I tried doing what you mentioned but still i couldn't get it to work unless i disable cache. I dont have any interfaces defined in my graphql schema and I already have __typename in my queries. I'm quite confused tbh.

@msal4, can you provide a reproducible example of this?
A repo with your queries, schema, pubspec, and how you're using the client.

@budde377 Apparently artemis generator doesn't seem to be working with graphql-flutter anymore, haven't checked why exactly but it's something related to the generated queries. I switched to graphql_codegen and everything seems to work just fine.
thanks :)

We have a problem with interface fragments that might also be related to this issue.
We have an interface for the user fields called UserInterface, and a type called Self, which is derived from the UserInterface. When we define the fragment as fragment UserFields on UserInterface and spread it in another fragment as fragment SelfFields on Self { ...UserFields }, on web (using Apollo client) it can correctly return the fields from both the user and self types. On mobile it correctly spreads the fragments and the server returns the fields, but graphql_flutter doesn't return it on the result object.
Any ideas on how to solve this?

As mentioned earlier in this thread, please try and update graphql flutter to the latest version (possibly beta), add __typename if it's not already provided by your query transformer, and supply a possibleTypes map to the client


background estimator 'underfits' background on negative contrast images

Background estimation by fitting a surface 'underfits' the background on negative contrast images.
Following is the result of drawing a line profile along horizontal at y=400. This problem is easily solved by inverting the input image.
Image

Profile

Even after inverting images, the surface fit approach does not report accurate background distribution.
Image

Background

Line profiles

Did you try use higher order function? get_background has a fit_order parameter for this. I haven't tuned it yet

Same issue at higher orders too:
Image

background with order 3:

background with order 4:

Looks like the fit is biased by the spots. I'll try robust fit or top-hat to fix this

Looks like sampling less pixels from the image for fitting is a quick fix:
Sample with grid size 32

Sample with grid size 128


Breadcrumb

There is a color contrast level issue on current breadcrumb :

Here is the proposal of the brand design team :

Local headers with breadcrumb at the breakpoints targeted by design team :
320

768

960

1200

Could be possible to have the hover and focus specification for activable elements?

Related to #167


Generate unhandled exception when I added Init function

(This is template for issue. Please choose from feature request or bug report or others then write.
If you need realtime communication then you can join our gitter room.)

FEATURE REQUEST / SUGGESTION
SUMMARY
DETAILS
PLATFORMS
which platform do you want?

[x] Android
[ ] iOS
[ ] UWP

BUG / PROBREM REPORT
SUMMARY
I tried to add FormsGoogleMaps.Init on MainActivity.cs in Android, but it kept failed.
VERSIONS
Hi I've used Xamarin.Forms.GoogleMaps ver 2.1.0-beta6 and updated Xamarin forms.
I was going to make sure to having the same property and development environment, but because of other packages that I have to use, only difference is the targeting.
My targetFramework is
Anyway, as you nocticed, I tried to add FormsGoogleMaps.Init on MainActivity.cs in Android. But it kept occur unhandled exception. Also the weird thing is I don't have Myapp. cs too.
I'm suppose to make IOS app as well, but Ive tested only Android app just now by using emulator.
Actually can I modify or add some Android code directly? How can I solve this problem?

Xamarin.Forms.GoogleMaps -2.1.0-beta6
Xamarin.Forms -<IP_ADDRESS>

PLATFORMS

[x ] Android
[] iOS
[ ] UWP

HOW TO REPRODUCE
(Let me show how to reproduce with steps or codes)

// code for reproduce

ACTUAL IMAGES
(Let me show your actual problem images/stacktrace. If you want to capture simulator screen to animation GIF, I recomend to LICEcap.)

// paste stacktrace

OTHERS
(free format)

Sorry I'm late.
Could you reproduce this yet?

We have not received the requested information.
If you are still experiencing this issue please provide all the requested information and re-open the issue report.
Thanks!


Support or change to lowercase hex encoding?

encoding the hex-str hashes as uppercase strings (5A2A20C8D80DF7C21016A56D65A0E296) instead of lowercase (5a2a20c8d80df7c21016a56d65a0e296) seem/feel pretty non-standard in that most hashing things that can give you a hex string will give you the hex string as lowercase.
Is there a reason that the hex string(s) are encoded uppercase?
FWIW/IMO: I personally find the upper case version harder to read. I also know that squealite has lower() so it's not a huge deal. mostly just curious over here.

@jessekrubin I was thinking about it too - at first i had it lowercase but changed it to upper case to be consistent with the built-in hex() function.  I agree it is not as readable in some cases, but consistency should take precedence here, esp because the documentation states that it is similar to hex(md5(...))

P.S. supporting both cases is a bit tricky because all hashing functions take any number of parameters, and hash all of them.  We could introduce alternative functions like md5hex_lower, but this is no different than lower(md5hex(...)). Also we could use some sort of a global settings mechanism if it is available (I am not too familiar with it).


Preds probs

Standardize names of label types: golds, preds, and probs (instead of gold, pred, prob).
Remove an extra kwarg in metric_score (had both prob and probs, which collided after the refactor)

Test plan:
All tests still pass after refactor

All tests still pass after refactor

but do they tho?

They do indeed. Ready for review!

@bhancock8 can we close?


readme - incorrect heading

Hi,
readme contains:
git status## Configure uberAgent for ElasticSearch/OpenSearch
I think the "git status" slipped in.

Dear Patrick,
thanks for reporting.
I fixed it.


Python 3.5/3.6

Hi,
First of all thanks for this amazing plugin!
I was wondering if you plan support for python 3 version anytime soon? It would open a lot of possibilities to certain areas.
Kind regards,
Theodore.

Unfortunately, no.
IronPython stopped development at 2.7.7, the version shipping with Rhino is currently 2.7.5, and that's the only interpreter that will let you run import Rhino. So anything that works with the Rhino Python libraries needs to be Python 2 or be able to translate on the fly between Python 2 and 3. RPyC, the remote computing library behind GhPythonRemote, does not and will not support this.
Things do not look much better with Rhino 6, since it is still using IronPython to interface with their .NET framework code.
If you absolutely need to call out to Python 3 from GH, you can have a look at GH CPython, but then you cannot import Rhino libraries in these scripts.
Out of curiosity, what libraries did you want to use that are Python 3 only?

Hi Pierre,
Thanks for the update, sounds good.
The main thing I am testing currently is tensorflow, which is only python3 for Windows.
In my case, doing machine learning, I don't really need (at least not yet) many Rhino specific libraries since I'm not currently developing components but only importing python libraries after all the GH definition has done its work in order to train models on my data.
I can still do vanilla ML with this and it can still be used to create ML components with scikit-learn for example. So it's cool. I won't be doing much DL inside GH anyways!
Kind regards,
Theodore.

+1 to this, despite being a "Not a chance" closed issue.
Right now, the library that I would like to use on python 3 is the async/await support. I know there's a backport for 2.7, but python 2.x is a dead end as it's officially End of Life and won't be supported anymore after 2020. It feels so wrong to not be able to use python 3 in 2017 😢


CMS text: 'Find a Store’ improperly named “Storefinder”

Description:  The Find a Store feature is improperly named as ‘Storefinder’ (one word) in the breadcrumb page titles of the feature.
How to duplicate: Visit the ‘Find a Store’ feature – any page

Done in #4666


fix(osx/#3698): Crash on open-with

Pick up https://github.com/revery-ui/revery/pull/1087 to fix #3698

CI failure is due to an open-vsx outage, overriding -


wp-cli

I can't find wp-cli in vendor/bin after creating the project with composre (based of bedrock).
Do I have to install wp-cli as separate tool globally on system?
Related: https://github.com/roots/bedrock/issues/14

Yes. See their installation docs: http://wp-cli.org/#installing

Thanks! I thought it is installed automatically or as project-local dependency.
It would be nice if this is hinted more in README.

WP-CLI recommends using the global phar installation method which is why we removed it. Bedrock isn't dependent on it at all, we just include a config file to make integration easier.


google groups new topic hangs

I've been trying this app recently and when I tried to post a new topic in google groups it hung at loading the Window to create the post.
It could be my setup but maybe a bug if anyone else can reproduce.

@nutpantz did you unblock the required 3rd party domains e.g. gstatic.com? Also, for Google-related webapps, I highly recommend using https://github.com/tobykurien/GoogleApps instead

Yes my bad, it was g static being blocked.but I love gapp browser,I  Will switch to that for google sites. Thank you


As a user I want to be able to seee my personal information

Add feature: User information page
After Jakob's lecture on GDPR, you learned that you have to be able to provide users information about their personal data that your application stores about them.
In the navigation bar on top, in between links to timelines and login, create a link to a new page with the link text "about me". That link appears only if a user is authenticated in your Chirp! application. If authenticated, a user should be able to see all information that your Chirp! application stores about her on that page. That is all personal information, like name, email address, links to other users that this user is following, etc. should be presented Additionally, all cheeps of this user are listed.
Optionally, you can implement functionality to download all this information as a zipped archive of text files. These might be CSV files, plain text files, etc.
Acceptance criteria:

A link at the navigation bar that takes a user to a personal-information page.
Besides user-information, all cheeps of this user should be listed on this page.
The link should only be visible if a user is authenticated.

all criteria are fulfilled.


Switch from rand to mt_rand

In reference to #16, while mt_rand is still not crypographically secure, it is much better than rand for applications like this. Without using a polyfilled random_int (or dropping all but the latest PHP support), I think this is the best your going to get.
Regarding your comment about a proof-of-concept, I will be releasing a tool at the end of the month which would make predicting the current implementation fairly trivial. Using mt_rand would make this attack unfeasible.

Fair enough. Thanks for this :)


1.12.2 ERROR help

What is the output link of /viaversion dump?
https://pastebin.com/THkWpxEa
Are you using any additional software like BungeeCord? If so, what software and version? (Please list your plugins as well)
Spigot 1.12
Plugins (32): LuckPerms, HealthBar, ArmorStandEditor, CrispyAntiWDL, DecoHeads, Multiverse-Core, RogueParkour, SkinsRestorer, NoPlugins, FastAsyncWorldEdit, React, RandomTeleport, CustomJoinItems, Vault, ServerListPlus, MoneyMobs, WorldEdit, Essentials, PerWorldInventory, Scavenger, HolographicDisplays, WorldGuard, DonatCase, ChestCommands, EssentialsProtect, EssentialsChat, EssentialsAntiBuild, AuctionGUIPlus, ViaVersion, EssentialsSpawn, TitleManager, AuthMe
How does this error happen? login? Using an item?
start server
Is there an error in the console? Use pastebin.com. Is there a kick message?
https://pastebin.com/jJJjpYfE

The error does not concern ViaVersion, this is down to your download corrupting etc.


Dockerhub Description sync

README only
https://github.com/peter-evans/dockerhub-description
Description and README
https://github.com/marketplace/actions/docker-hub-readme-description-sync

Perhaps just create a Netlify documentation site if this is important


Documentation📄: Add Possible Errors that users can face.

What is the current documentation state?
Add all the Possible errors that users can face while contributing or setting up a project locally on their machines by adding a separate section in the readme.MD file with the respected solutions. It will help newcomers to resolve possible errors by reading the docs we have.
Why do you want to improve the statement?
It is important as sometimes new contributors and users are not able to find any particular solutions to some errors and are not able to get help from the community as well, but if we mention all the errors that other community members and contributors have come across during the setup of the project or running it locally then it will very helpful for them !!
Proposed Statement
we can achieve this by making a separate section in the readme.MD under contributing sections by naming it "possible errors" or "errors you may come across"
Additional context.
I think that multiple PR may get open related to this issue so at the time of opening PR use update followed by issue number instead of fixes and I request reviewers not to close this issue after merging a PR as other contributors  may also open PR related to this

I am adding solution for this error for now
Please add the following to your Gemfile to avoid polling for changes:
    gem 'wdm', '>= 0.1.0' if [Gem.win](http://gem.win/)_platform?
 Auto-regeneration: enabled for 'C:/Users/abhij/Desktop/open-horizon/[open-horizon.github.io](http://open-horizon.github.io/)'
C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/jekyll-3.9.0/lib/jekyll/commands/serve/servlet.rb:3:in `require': cannot load such file -- webrick (LoadError)
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/jekyll-3.9.0/lib/jekyll/commands/serve/servlet.rb:3:in `<top (required)>'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/jekyll-3.9.0/lib/jekyll/commands/serve.rb:184:in `require_relative'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/jekyll-3.9.0/lib/jekyll/commands/serve.rb:184:in `setup'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/jekyll-3.9.0/lib/jekyll/commands/serve.rb:102:in `process'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/jekyll-3.9.0/lib/jekyll/commands/serve.rb:93:in `block in start'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/jekyll-3.9.0/lib/jekyll/commands/serve.rb:93:in `each'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/jekyll-3.9.0/lib/jekyll/commands/serve.rb:93:in `start'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/jekyll-3.9.0/lib/jekyll/commands/serve.rb:75:in `block (2 levels) in init_with_program'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `block in execute'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `each'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/mercenary-0.3.6/lib/mercenary/command.rb:220:in `execute'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/mercenary-0.3.6/lib/mercenary/program.rb:42:in `go'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/mercenary-0.3.6/lib/mercenary.rb:19:in `program'
        from C:/Ruby30-x64/lib/ruby/gems/3.0.0/gems/jekyll-3.9.0/exe/jekyll:15:in `<top (required)>'
        from C:/Ruby30-x64/bin/jekyll:25:in `load'
        from C:/Ruby30-x64/bin/jekyll:25:in `<main>'

        ```

This should be complete


Firestore analyze issues

At least the bottom one looks like an issue in Firestore itself:
bundle exec pod lib lint --sources=https://cdn.cocoapods.org/ --analyze --include-podspecs={FirebaseAuthInterop.podspec,FirebaseCore.podspec,GoogleUtilities.podspec,FirebaseCoreDiagnosticsInterop.podspec,FirebaseCoreDiagnostics.podspec,GoogleDataTransportCCTSupport.podspec,GoogleDataTransport.podspec} FirebaseFirestore.podspec --platforms=ios --allow-warnings --no-subspecs
Adding spec repo trunk with CDN https://cdn.cocoapods.org/
Still working, running for 1min.
Still working, running for 2min.
Still working, running for 3min.
Still working, running for 4min.
Still working, running for 5min.
Still working, running for 6min.
Still working, running for 7min.
Still working, running for 8min.
Still working, running for 9min.
Still working, running for 10min.
Still working, running for 11min.
Still working, running for 12min.
Still working, running for 13min.
Still working, running for 14min.
Still working, running for 15min.
Still working, running for 16min.
Still working, running for 17min.
Still working, running for 18min.
Still working, running for 19min.
Still working, running for 20min.
Still working, running for 21min.
Still working, running for 22min.
Still working, running for 23min.
Still working, running for 24min.
Still working, running for 25min.
-> FirebaseFirestore (1.6.1)
- WARN  | compiler_flags: Warnings must not be disabled(-Wno compiler flags).
- ERROR | [iOS] build_pod: Static Analysis failed. You can use --verbose for more information. You can use --no-clean to save a reproducible buid environment.
- NOTE  | xcodebuild:  note: Using new build system
- NOTE  | [iOS] xcodebuild:  note: Planning build
- NOTE  | [iOS] xcodebuild:  note: Constructing build description
- NOTE  | [iOS] xcodebuild:  warning: Capabilities for Signing & Capabilities may not function correctly because its entitlements use a placeholder team ID. To resolve this, select a development team in the App editor. (in target 'App' from project 'App')
- NOTE  | [iOS] xcodebuild:  warning: Skipping code signing because the target does not have an Info.plist file and one is not being generated automatically. (in target 'App' from project 'App')
- WARN  | xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/Protos/objc/google/firestore/v1/Write.pbobjc.m:33:10: warning: non-portable path to file '<protobuf/Timestamp.pbobjc.h>'; specified path differs in case from file name on disk [-Wnonportable-include-path]
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/third_party/abseil-cpp/absl/time/duration.cc:743:21: warning: The left operand of '==' is a garbage value
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/third_party/abseil-cpp/absl/strings/numbers.cc:208:5: warning: Value stored to 'digits' is never read
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/third_party/abseil-cpp/absl/strings/numbers.cc:373:34: warning: The result of the left shift is undefined due to shifting by '64', which is greater or equal to the width of type 'unsigned long long'
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/third_party/abseil-cpp/absl/strings/internal/str_format/float_conversion.cc:430:27: warning: Assigned value is garbage or undefined
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/third_party/abseil-cpp/absl/strings/escaping.cc:888:9: warning: Value stored to 'szdest' is never read
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/third_party/abseil-cpp/absl/strings/escaping.cc:908:9: warning: Value stored to 'szdest' is never read
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/third_party/abseil-cpp/absl/strings/escaping.cc:926:7: warning: Value stored to 'szdest' is never read
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/third_party/abseil-cpp/absl/numeric/int128.h:644:45: warning: The result of the right shift is undefined because the right operand is negative
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/third_party/abseil-cpp/absl/base/internal/low_level_alloc.cc:163:42: warning: Dereference of undefined pointer value
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/third_party/abseil-cpp/absl/base/internal/low_level_alloc.cc:188:37: warning: Dereference of undefined pointer value
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/third_party/abseil-cpp/absl/base/internal/low_level_alloc.cc:499:3: warning: 1st function call argument is an uninitialized value
- WARN  | [iOS] xcodebuild:  /Users/travis/build/firebase/firebase-ios-sdk/Firestore/core/src/firebase/firestore/nanopb/writer.h:115:27: warning: The right operand of '+' is a garbage value

This was fixed awhile ago. We're not currently disabling the analyzer on any pod lib lint tests.  (Discovered when the analyzer caught an issue in #9486


Weird regex search causing database 100%

Link to reproduction
Not for now
To Reproduce
No reproduce for now.
Describe the Bug
We found a weird search query in database log. It takes about <100ms to run but called a large amount of times
{"$match":{"$and":[{"$or":[{"version.name":{"$regex":{"$regularExpression":{"pattern":"(?=.*(?:(?:[^\\p{L}\\p{N}])|^).*illinois.*(?=[^\\p{L}\\p{N}]|$)).+","options":"i"}}}}]}]}}

Is this regex pattern looks familiar to you guys? I tried to search in payload source code but got no luck.
Payload Version
1.9.5

This is actually recently fixed. Just update to latest and you should be good to go!


[bugfix] cannot add options to FilterBox

closes https://github.com/apache/incubator-superset/issues/6227

Codecov Report

Merging #6231 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6231   +/-   ##
=======================================
  Coverage   76.85%   76.85%
=======================================
  Files          47       47
  Lines        9393     9393
=======================================
  Hits         7219     7219
  Misses       2174     2174

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e1a3a68...acd1535. Read the comment docs.


Added an option that makes an option vscode_cmakekit_generator:vscode_folder

Added an option that makes an option vscode_cmakekit_generator:vscode_folder mandatory when using this package.
Given this folder, the cmake-kits.json file is saved to a .vscode folder.
Validates on the basename in the configure step, which must be .vscode.
os.environ was used to pass the parameter to the Generator, because self.conanfile.options appears to be empty in the Generator content step.

Could you increment the version?

Done


test: HeadlessEnvironment loads modules from classpath

Tests using engine-test's HeadlessEnvironment have a lot in common with the issues in https://github.com/Terasology/ModuleTestingEnvironment/pull/41, so they could use the same sort of treatment, which is "turn on classpath module loading until we have a better idea."
This change to HeadlessEnvironment is a more general solution than that in https://github.com/Terasology/Pathfinding/pull/57, possible because HeadlessEnvironment directly creates its own ModuleManager so we don't need to try to indirectly influence ModuleManager creation by something in the global environment.
This PR is built on #4573 because they both modify ModuleManagerFactory.
How to test
Run tests involving HeadlessEnvironment or WorldProvidingHeadlessEnvironment. Check the Unhealthy Modules list on Jenkins, try running those tests under gradle locally, and see if this branch makes the number of failures go down or change in useful ways.

Some self-review notes:
ModuleManager constructor proliferation and ModuleManagerFactory: Wow, languages that don't have default values for function arguments are kind of a mess, aren't they?
I think we literally have more constructors than callers at this point.
I should know better than to commit anything saying "this is temporary," so I'll say this instead: I'm not dedicated to this interface for ModuleManager. I hope we can do some pruning during or after migrating to gestalt-v7 and we know what we want from it.
doLoadModulesFromClasspath's ridiculous number of ways of setting a flag: ditto on the "not committed to this." The environment variable was my first idea, as it'd be easy to set when running the entire game, not just for tests. But if nobody's needed to use that, we can drop that part.
and if we're lucky, we get to remove the option entirely if our gestalt-v7 implementation finds a way to load module dependencies without this everything-in-the-classpath approach.


Bosun version information missing

From the Bosun dashboard, there appears to be no means of obtaining a version number.  When reporting issues it would be nice to be able to reference the version being run.
I installed this to run on a local docker instance using: stackexchange/bosun:latest, and therefore have no idea what version I'm running.
Additionally, it would be good to be able to display the versions of the collectors being used by this instance of Bosun, but that may be harder to achieve.

Closing assuming above information was sufficient. If you need further help please re-open


How to evaluate & render?

[ ] I have marked all applicable categories:

[ ] exception-raising bug
[ ] RL algorithm bug
[ ] system worker bug
[ ] system utils bug
[ ] code design/refactor
[x] documentation request
[ ] new feature request

[x] I have visited the readme and doc

[x] I have searched through the issue tracker and pr tracker

[x] I have mentioned version numbers, operating system and environment, where applicable:
import ding, torch, sys
print(ding.__version__, torch.__version__, sys.version, sys.platform)

v0.4.3 1.12.1+cu113 3.7.13 (default, Mar 29 2022, 02:18:16)
[GCC 7.5.0] linux

How to evaluate the trained networks?
Is rendering supported? For instance, to save a gif while evaluating in MPE.

For evaluating the trained networks, you can refer to this discussion, we prepare eval and deploy examples.

For rendering, you can implement and call the enable_save_replay method to save replay video, like this demo, related code can also be found here.


Feat/reset.d.ts

Codecov Report

Merging #292 (75e3499) into feat/servers.url (d1bcd7b) will decrease coverage by 0.58%.
The diff coverage is 62.06%.

@@                 Coverage Diff                  @@
##           feat/servers.url     #292      +/-   ##
====================================================
- Coverage             88.79%   88.21%   -0.58%
====================================================
  Files                    68       69       +1
  Lines                  4034     4100      +66
  Branches                524      528       +4
====================================================
+ Hits                   3582     3617      +35
- Misses                  451      482      +31
  Partials                  1        1

Impacted Files
Coverage Δ

packages/core/src/managers/pluginManager/index.ts
100.00% <ø> (ø)

packages/core/src/utils/SummaryError.ts
100.00% <ø> (ø)

packages/swagger/src/parsers/oasParser.ts
84.28% <14.28%> (-6.49%)
:arrow_down:

.../src/managers/pluginManager/ParallelPluginError.ts
62.06% <21.42%> (-37.94%)
:arrow_down:

packages/cli/src/utils/getPlugins.ts
83.33% <50.00%> (-2.04%)
:arrow_down:

...ackages/swagger/src/generators/ImportsGenerator.ts
86.20% <50.00%> (ø)

...s/core/src/managers/pluginManager/PluginManager.ts
90.26% <58.33%> (-1.71%)
:arrow_down:

packages/core/src/build.ts
89.09% <66.66%> (-1.48%)
:arrow_down:

packages/core/src/utils/isPromise.ts
84.61% <75.00%> (-15.39%)
:arrow_down:

packages/core/src/managers/fileManager/utils.ts
87.87% <100.00%> (ø)

... and 8 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


x/website: missing " in code snippet

What version of Go are you using (go version)?

$ go version
# Doesn't matter.

Does this issue reproduce with the latest release?
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
# Doesn't matter.

What did you do?

Find typo when reading reference.
https://golang.org/ref/mod#environment-variables

What did you expect to see?
GOSUMDB="sum.golang.org+<publickey> https://sum.golang.org"

What did you see instead?

cc @jayconrod


Use prevHash from bitcoind block index

Changed method getChainWork into getBlockIndex
Added prevHash to getBlockIndex result
Closes https://github.com/bitpay/bitcoind.js/issues/39

This looks great


Can i use my own new images to train the model?

Can i use my own new images to train the model? I am working on cnn head pose estimation. your work is excellent. I was stuck with a question that "Can i use my own new images to train the model? "  whether this deepgaze will help me so in doing? Thank you.

Dear @shashi19081994
At the moment there is not a file that you can use to train the model. However there is an implementation of a trainer for aDeep MLP here in the example folder. You can slightly modify this example including the model and the weights used in the Deepgaze CNN head_pose_estimation.py package. If you have familiarity with Tensorflow it should be straightforward.
Once you have the file with the correct model and weights then it is necessary to pre-process your images. In the same example folder you can find some parser used on the Prima and AFLW datasets. You should modify one of the parser in order to resize and crop your images and then store them in a pickle file, which is the format used by the trainer.
At the end of the training you need to save the Tensorflow meta-files which can be loaded in Deepgaze or in your personal model.
Best, Massimiliano


Do not store JWT in HTML5 Local Storage

The HTML Local Storage can be compromised because it can be read by JavaScript and the same Domain. (XSS) Use Cookies instead. They are not accessible by JavaScript.
Ref.: https://stormpath.com/blog/where-to-store-your-jwts-cookies-vs-html5-web-storage

Or just don't commit them to local storage and set them in the Vuex store in memory:

https://auth0.com/docs/security/store-tokens#don-t-store-tokens-in-local-storage
https://community.auth0.com/t/why-is-storing-tokens-in-memory-recommended/17742


Enter key acts like pressing OK for Alert, Confirm dialog

Fix #9122

not sure this should be only with editor or also with game by default.

Why not just set the focus on the "Ok" button ?

@groud ok button will lose focus on many cases, like attach script, create new node, when typing something on LineEdit on that popup window.

In that case I wouldn't make the option global to the Dialog class, but depending on the use case.
Or at least it should be configurable I think.

@groud hm... I was wrong. the LineEdit eats Enter key, so it doesn't work as I expected.
need to dig it deeper.

Maybe this is a bug ? I'm not sure the LineEdit should catch the event if it has not nothing to do with it.

@groud I'm not sure either if it's considered as a bug or not...

Well it's not a bug, it's a feature: http://docs.godotengine.org/en/latest/classes/class_lineedit.html#signals
How else would users confirm their input in a LineEdit?

Yeah sure. But you might also need the LineEdit not to catch the Enter key.
Maybe this could be made configurable ?

@akien-mga text_entered signal should be emitted but need to eat enter key event also?
of course, we can connect text_entered signal to each LineEdit though.

of course, we can connect text_entered signal to each LineEdit though.

To be honest, I think this should be the cleanest way to do so, a little bit redundant but it may avoid unexpected situations in other cases.

yeah. I'm starting to think in that way.


basic rbac support

replace #1351
some of the works are from @rajiteh
changes summary:

system components use their own certs, identified as user/group document here:
https://kubernetes.io/docs/admin/authorization/rbac/#default-roles-and-role-bindings
changed components:  kubelet/kube-proxy/controller-manager/scheduler

addon run as ServiceAccount and changed related ClusterRole/ClusterRoleBinding,
patch existing Binding if necessary.
changed addons:  kube-dns/helm

add an authorization_modes var to control authorization modes

messed it up, create a new one. sorry

Is this supposed to replace #1351? Seems like there are a number of missing files.

@bradbeam yes, this replace #1351 . forgot -a, added.

Two more asks --
I just merged in an update to kube-dns and it looks like that caused some conflicts. Would you be able to address?
Could you add a test case for RBAC in ci?

@bradbeam rebased on latest master
i am not sure how to add unit tests. should i add RBAC for all existing test ? there seem to be a lots combinations.

I think using cluseterrolebinding for serviceaccount and node is more simple.
There is a apiserver_custom_flags in roles/kubernetes/master/defaults/main.yml file.
If you want, put the "--authorization-mode=RBAC" parameters there.

@seungkyua thanks for replying. can you explain it in more detail ? i didn't catch you.

I think using cluseterrolebinding for serviceaccount and node is more simple.
i am just following the RBAC docs for builtin user/group/rolebinding, i am not sure what you mean
here. :)

There is a apiserver_custom_flags in roles/kubernetes/master/defaults/main.yml file.
If you want, put the "--authorization-mode=RBAC" parameters there.
we added the authorization-modes to control the whole RBAC things, *apiserver_custom_flags * is only one of them. you can find the detailed commit comment in #1350 .

We have just tested this patch, everything was OK except kube-dns. To fix it we had to:

upgrade kube-dns version to 1.14.4
add kube-dns config map
add configmaps to system:kube-dns cluster role resources

@hzamani kube-dns is recently updated, i haven't got the time to update and test against it.
could you make a PR and i will merge it.
@bradbeam could you give me some hint on how to write ci test so we can push this forward?
@rajiteh unfortunately, i didn't have enough time for this. if you or some one want to do this, could take over from here.

For CI

Add in a new variable to control authorization modes
I think https://github.com/kubernetes-incubator/kubespray/blob/master/.gitlab-ci.yml#L64 we could add in AUTHORIZATION_MODES='{ "authorization_modes": ["AlwaysAllow"] }' ( to set the default )
Add in the new variable to the ansible command
https://github.com/kubernetes-incubator/kubespray/blob/master/.gitlab-ci.yml#L133 something like -e authorization_modes=${AUTHORIZATION_MODES}
Add in a new set of variables for rbac test https://github.com/kubernetes-incubator/kubespray/blob/master/.gitlab-ci.yml#L371

.ubuntu_calico_rbac_variables: &ubuntu_calico_rbac_variables
# stage: deploy-gce-special
  AUTHORIZATION_MODES: '{ "authorization_modes":  [ "RBAC" ] }'
  KUBE_NETWORK_PLUGIN: calico
  CLOUD_IMAGE: ubuntu-1604-xenial
  CLOUD_REGION: europe-west1-b
  CLUSTER_MODE: separate
  STARTUP_SCRIPT: ""

Actual test https://github.com/kubernetes-incubator/kubespray/blob/master/.gitlab-ci.yml#L596

coreos-alpha-weave-ha:
ubuntu-calico-rbac-sep:
  stage: deploy-gce-special
  <<: *job
  <<: *gce
  variables:
    <<: *gce_variables
    <<: *ubuntu_calico_rbac_variables
  when: manual
  except: ['triggers']
  only: ['master', /^pr-.*$/]

i think that should do it

@bradbeam thanks for detailed guide, i can just copy & paste from it. test added.
@hzamani with d5516a4ca9c42cc6cfb4c1aef692e0ef323abfcc kube-dns works without configmap.
and create a ConfigMap kube-dns in kube-system, kube-dns will pick up the new config as expected.
so i thin we don't need to do anything regarding ConfigMap.

@rajiteh unfortunately, i didn't have enough time for this. if you or some one want to do this, could take over from here.

@jwfang could you clarify? i can help out

@rajiteh i have some other tasks from work, and may not have enough time to keep track on this.
i guess you can create a new PR from this. when you've done that, i will close this one.

@bradbeam anything we can do to make the tested and get this merged ? thanks.

ci check this

Calico controller requires RBAC rules to access pods and namespaces and create network polices.

@hzamani thanks for your advice.
i am not familiary with calico, found https://github.com/projectcalico/calico/issues/402 and https://github.com/projectcalico/calico/blob/master/master/getting-started/kubernetes/installation/rbac.yaml , but not sure if they works.
anyway, PR are welcome.
@bradbeam i rebase for the lastest master, should we add some RBAC & calico checks for now
and do RBAC for calico later ? and change the test for non-calico-RBAC ?
if this is acceptable, i will submit another commit.

@jwfang The rbac.yaml from project calico is just what's needed.

@hzamani thanks.
i am new with calico. there seem to be serval components for calico, and i never get calico works in my environment ( i am using flannel).
we have calico and calico-rr roles, and there seems to be lots of features/components for calico itself.
can you give me some guide on where should i put the rbac.yaml from calico ?

I'm a bit unclear on where we stand on this PR. What's needed to get to a working state for this? Anyone tried it yet?

@jwfang I found it confusing to not have calico in kubernetes-apps/network_plugin, and it's a good idea to place calico_node daemon set there.
@rsmitty I tried it before calico related stuff and except for calico (which I had to add RBAC rules by hand to make it work) everything was OK.

@jwfang  sorry for late comment.
I think this is so complicated.
To use a rbac, as I know, we just make a ClusterRoleBinding about "admin users", "k8s nodes" and "default service account in {{system-namespaces}}" reference to ClusterRole about "cluster-admin".
It's good worked to my environment.

@rsmitty @hzamani
currently i have trouble to get calico-RBAC working.
from https://github.com/projectcalico/calico/blob/master/master/getting-started/kubernetes/installation/rbac.yaml, we need do two parts: calico-policy-controller and calico-node. i have do the calico-policy-controller part, but not sure how to do the calico-node part.
after a second thought, i guess we may not need to change calico-node to run as DamonSet. instead, we can just add certs for the current systemd service. i found api-server related stuff in https://github.com/kubernetes-incubator/kubespray/blob/992023288f4a42ce6960d611dc92c8ee798efd81/roles/network_plugin/calico/templates/cni-calico.conf.j2#L24, i think that's it.
i am new to cni/calico and not work full-time on this, so may not finish the calico stuff soon unless there is some guide/help.
@rsmitty if the other parts are fine, should we finish this PR and leave calico-RBAC to another PR ?

@jwfang
What network plugin have you been testing with? I think I'm fine with updating the CI tests with whatever plugin is working so we can get a successful CI run.
If you have time, we've got #1271 logged to track the necessary work to uplift calico, but I'm not sure I want that to be a blocker for this PR. Would you be able to update the docs to reflect that RBAC is in experimental phase and note that we currently do not support RBAC with either calico or vault.
I think once we get CI passing, I'm fine with getting this merged in.

@bradbeam we have successfully deployed Kubernetes using this patch. Thanks to @jwfang.

@jwfang @bradbeam
I suggest the rbac mode.
As I know, There are two ways to implement.

Using cluster-admin
Only default serviceaccount in kube-system, admin user (root, kube) in k8s-cluster.yml and kube-node-{{ hostname }} user are have cluster-admin role.
All of them are system user. So I think It's not a risk about security.

Using each user and group is binding to it's role.
Clusterrolebinding                                 Name                                                Type
system:kube-controller-manager  -->  system:kube-controller-manager  (User)
system:kube-scheduler                  -->  system:kube-scheduler                  (User)
system:node-proxier                       -->  system:kube-proxy                         (User)
system:node                                    -->  system:nodes                                  (Group)
system:kube-dns                            -->  kube-dns (only kube-system)        (ServiceAccount)
cluster-admin   (tiller, calico etc)   --> default   (only kube-system)           (ServiceAccount)
cluster-admin                                  --> root, kube                                         (Admin user)

I think we don't need to make a kube-config.yml and new cert name.

@bradbeam i am using flannel in my test environment.
i would like to do additonal calico/RBAC related works, but currently i think i am not qualified since i am not familiar with calico/cni and it seems would take a while to learn from scratch.
all the cni modes are in my to-learn list, but i can't make any promise for the time frame since i am doing this part-time.
for this PR, i did the following additional changes to make it mergable in the current state:

revert all calico related changes,
update docs say calico & vault not supported.
change gce test to use flannel instead of calico
add authorization_modes vars to all cluster.yml deployment

@seungkyua
i am not sure i understanding you corrently and completely.
currently the user/role/rolebinding are take directly from the official k8s RBAC docs. i just try to stick with
the docs without taking short-cuts. :)
the reason for multiple kube-config is: a single kube-config file can have only one current-context, i could find a way to share it between different components.

Thanks, kicking off the rbac ci test.

@bradbeam i see delete gce instances failed from ci, i guess that not related to this PR ?

I'm curious if the failure is actually a bit above that with running the netchecker test
TASK [debug] *******************************************************************
task path: /builds/kargo-ci/kubernetes-incubator__kubespray/tests/testcases/040_check-network-adv.yml:70
Wednesday 12 July 2017  17:05:27 +0000 (0:00:00.425)       0:00:01.880 ********
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ValueError: No JSON object could be decoded
fatal: [k8s-9725808-22224346-2]: FAILED! => {"failed": true, "msg": "Unexpected failure during module execution.", "stdout": ""}

In looking at the netchecker deployment, I don't think there's anything that should break when rbac is enabled. Would you be able to deploy netchecker to your environment and run through that testcases/040_check-network-adv.yml and see if you get different results?

@bradbeam sorry for the late, fixed the 040_check-network-adv.yml test and rebased.
it turns out netchecker-server needs list pods privilege.
could you rerun the test for me ？

@bradbeam sorry, commit my local changes for #1381. fix it in 3d87f23

@bradbeam hi, what's needed for this PR to get merged ? seems the tests passed.

I don't recall the issue, but I was not able to "upgrade" an existing cluster when I tried.

@Starefossen no, i didn't test the upgrade process much. :(
i guess for RBAC -> RBAC upgrade, it will works as other parts of kubespray;
for no-RBAC -> RBAC, it would be a two-steps process: no-RBAC -> RBAC+AlwaysAllow -> RBAC-only, and there should be problems as noted by @whereismyjetpack.
currently i am hoping to get this merged as it's for the following reason:

i have not enough time for this and i am not work full-time on this, as noted above;
it's already a bit complex for this PR;
other no-trival tasks, such as calico support, upgrade etc, could be done by others based on this.

I'm curious what problems with calico? I'm running a merged in copy of this PR with calico and it ssems to work fine.

@whereismyjetpack see https://github.com/projectcalico/calico/blob/master/master/getting-started/kubernetes/installation/rbac.yaml and https://github.com/kubernetes-incubator/kubespray/issues/1271, i guess calico also need proper role/rolebinding.
PS: i am not familiar with calico and never get calico works in my environment (maybe due to environment issue which i didn't figure out yet).
my original test for RBAC was using calico, and it failed in the CI test. so i switch to flannel for the CI in https://github.com/kubernetes-incubator/kubespray/pull/1382/commits/d24520161416cc586cfe0968fe66afc4722c4c0b

Thanks for that information, in my setup calico is seemingly deployed as a systemd unit file, and not a deployment within kube. i'm not sure under which conditions it chooses one over the other, or maybe that PR hasn't made it in yet? I don't want to hijack this PR with calico related convos, but wanted to thank you for replying

Okay, so it looks like we're passing CI with this PR. Awesome! I want to verify my understanding a bit. The way it's setup right now, the RBAC options actually won't be active by default, right? Asking, because I don't want to affect existing clusters with this PR if possible. I also would love to see some research on the upgrade path with this. How we might support adding RBAC to an existing cluster.
Other than that, I think I'm cool with merging this. We can follow it up with any other necessary changes.

the authorization_modes variable needs to be set, it defaults to an empty list. leaving it unset will setup the cluster as it does today.

@rsmitty as @whereismyjetpack pointed out, it default with no-RBAC and keeps the current behaviour.
for upgrade from non-RBAC to RBAC, unfortunately i haven't got enough time to work on. :(

where are we with this PR? anything i can do to help?

Really need this PR for our clusters. Anything I can do to get this PR merged? Thanks!

cc @bradbeam @rsmitty
@whereismyjetpack @joisthof i think you can provide feedback, such as successful/failure/problem encountered etc.
anyone willing to take over this is welcome. :)

Proceeding with merge. Will follow up with additional fixes as needed.

@jwfang  thanks for this awesome pr, appreciate your patience


ScopeManager.delactivate misspelled

See https://github.com/jukylin/jaeger-php/blob/51b6f403b410c3e7c14d7ee613d065e6c08190f9/src/Jaeger/ScopeManager.php#L42
It is specified as deactivate in the PHP OpenTracing library here:
https://github.com/opentracing/opentracing-php/blob/1.0.0-beta5/src/OpenTracing/Mock/MockScopeManager.php#L38

yes，beacuse I think deactivate is indistinct. decode activate or delete activate ?

It is undoing the work of activate technically so deactivate makes sense. However I suppose removeActive or deleteActive makes sense as well.

done #53


Got 520 error using proxies in Instaloader

Hi, I want to use proxies in Instaloader to prevent 429 error. But when I set the proxy using the below function, I get 520 error.
Function :
def SwitchProxy():
    '''This function switch proxy'''
    # Set verify=False
    os.environ['CURL_CA_BUNDLE'] = ''
    os.environ['HTTP_PROXY'] = 'I use proxycrawl.com smartproxy '
    os.environ['HTTPS_PROXY'] = 'I use proxycrawl.com smartproxy'

I'm not sure if the problem is the smart proxies or not, but I didn't get a 520 error when I simply requested to Instagram using this proxy and the below code.
import requests
import os

url = 'https://instagram.com/accounts/login'

os.environ['CURL_CA_BUNDLE'] = ''
os.environ['HTTP_PROXY'] = 'I use proxycrawl.com smartproxy'
os.environ['HTTPS_PROXY'] = 'I use proxycrawl.com smartproxy'

req = requests.get(url)
print(req.text)

Full error :
JSONDecodeError                           Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/instaloader/instaloadercontext.py](https://localhost:8080/#) in login(self, user, passwd)
    224         try:
--> 225             resp_json = login.json()
    226         except json.decoder.JSONDecodeError as err:

6 frames
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

ConnectionException                       Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/instaloader/instaloadercontext.py](https://localhost:8080/#) in login(self, user, passwd)
    227             raise ConnectionException(
    228                 "Login error: JSON decode fail, {} - {}.".format(login.status_code, login.reason)
--> 229             ) from err
    230         if resp_json.get('two_factor_required'):
    231             two_factor_session = copy_session(session, self.request_timeout)

ConnectionException: Login error: JSON decode fail, 520 - .

Does anyone have a solution?

I've just found an answer for my issue.
Right now, you can't use proxycrawl.com to scrape Instagram.
You should use other proxy services such as webshare.

there is no way to verify ssl false or add pem file in cert bundle
os.environ['HTTPS_PROXY'] = 'http://myproxy:22225'
https://requests.readthedocs.io/en/latest/user/advanced/#proxies
from requests.utils import DEFAULT_CA_BUNDLE_PATH
print(DEFAULT_CA_BUNDLE_PATH)
You override this default certificate bundle by setting the REQUESTS_CA_BUNDLE (or CURL_CA_BUNDLE) environment variable to another file path:
$ export REQUESTS_CA_BUNDLE="/usr/local/myproxy_info/cacert.pem"
$ export https_proxy="http://<IP_ADDRESS>:1080"
$ python

import requests
requests.get('https://example.org')   this is not working can you add to verify ssl or i can give path of my pem file in library for each method?


Remove "Proposals" language cross-platform

Remove "Proposal" from the name of all four of these actions. The same goes for Emergency Upgrade.
The distinction is that "Proposals" are voted on by the Token Assembly. If we wanted/needed to add another word, I would call them "Approvals" because certain signers approve them on a particular multisig/s (no voting).

Completed.


SNOW-1842464 Fix the stored procedure build break

Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
<!---
In this section, please add a Snowflake Jira issue number.
<p>Note that if a corresponding GitHub issue exists, you should still include
the Snowflake Jira issue number. For example, for GitHub issue
https://github.com/snowflakedb/snowpark-python/issues/1400, you should
add "SNOW-1335071" here.
--->
Fixes SNOW-1842464

Fill out the following pre-review checklist:

[ ] I am adding a new automated test(s) to verify correctness of my new code

[ ] If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing

[ ] I am adding new logging messages
[ ] I am adding a new telemetry message
[ ] I am adding new credentials
[x] I am adding a new dependency
[ ] If this is a new feature/behavior, I'm adding the Local Testing parity changes.
[x] I acknowledge that I have ensured my changes to be thread-safe. Follow the link for more information: Thread-safe Developer Guidelines

Please describe how your code solves the related issue.
Add explicit dependencies on python-dateutil to setup.py and recipe/meta.yaml.

we also need to fix setup.py: https://ci-dev-142.int.snowflakecomputing.com/job/RT-Precommit-Tests/71/console
that fix is probably different and needs investigation.

I'm restarting https://ci-dev-142.int.snowflakecomputing.com/job/PythonStoredProcBuildPrecommitTest/1484/ since the failures could be due to branch name change

@sfc-gh-oplaton let's update changelog to say we are adding new dependency

Done.

let's update changelog too.

Done.


Remove ErrorNotifier

What does this change?

Removed ErrorNotifier

What is the value of this and can you measure success?

Be able to handle errors.

Codecov Report

Merging #39 (f84c6f1) into master (f7cf96a) will increase coverage by 2.28%.
The diff coverage is 73.46%.

@@            Coverage Diff             @@
##           master      #39      +/-   ##
==========================================
+ Coverage   51.07%   53.36%   +2.28%
==========================================
  Files          34       33       -1
  Lines         466      461       -5
==========================================
+ Hits          238      246       +8
+ Misses        228      215      -13

Impacted Files
Coverage Δ

lib/ui/app_change_notifier.dart
0.00% <0.00%> (ø)

lib/data/app_error.dart
87.50% <71.42%> (-9.28%)
:arrow_down:

lib/ui/home/home_page.dart
70.58% <71.42%> (-1.29%)
:arrow_down:

lib/app.dart
83.33% <75.00%> (+44.87%)
:arrow_up:

lib/data/remote/news_data_source_impl.dart
100.00% <100.00%> (ø)

lib/ui/app_theme.dart
86.20% <100.00%> (+32.87%)
:arrow_up:

lib/ui/home/home_view_model.dart
75.00% <100.00%> (-5.00%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f7cf96a...f84c6f1. Read the comment docs.


Highlight button when touching not round

When you keep touching the button, the TouchableHighlight doesn't fit the button.
My code:
<FloatActionButton
              buttonColor={primaryColor}
              degrees={0}
              offsetX={40}
              offsetY={40}
              onPress={() => onOpenModal(true)} >
              <IconMaterial name="plus" />
</FloatActionButton>

hey @amtBurgos  , this can be fixed by using fixNativeFeedbackRadius prop .
so it looks something like .
<ActionButton         buttonColor='#00B89B' 	fixNativeFeedbackRadius={true} 	onPress={()=>{console.log('Tap')}} />

Thank you :)

This does only fix the problem, if the ActionButton doesn't have any child buttons. If there are childbuttons then the style is bugged.


Proposal: Add support for iio mount matrix

Task Description
Please support iio mount matrix in soletta, I need this feature for the soletta sensor application(fbp or c api) can read out the data that following android orientation.
iio mount matrix is merge into mainline kernel recently (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=dfc57732ad38f93ae6232a3b4e64fd077383a0f1).
The mount matrix is store in iio mount matrix entry, it will give "0, 1, 0; 1, 0, 0; 0, 0, -1". Store this into a float array and use the following method to process.
https://github.com/01org/android-iio-sensors-hal/blob/master/transform.c#L200 is the example of handling mount matrix.
Thanks.
Dependencies
N/A

I'm thinking about this and would likely solve it outside of the IIO:

sol-util.h to provide a matrix transformation code, for use in C code;
introduce a node type to convert matrix, not specific about IIO, but any direction vector

Is this enough? For sure we could add the matrix transformation to the IIO node using that sol-util.h function to be introduced, I'm just wondering if that is going to help if we provide a more generic direction-vector/transform node that can be used for all inputs, like other kinds of sensors, data from the internet, etc.

But this is iio mount matrix, is in iio driver. I think is better to put under iio layer.

@barbieri
Can include this feature in V1?
@fulong82 @lblim

ok, then it makes sense to have both the option and the node type to transform. The option will write directly to the kernel options.
As for v1, I don't want to commit to that in order to avoid delaying the v1 even further. However, if for bureaucracy purposes the v1 is delayed and we have the code ready, then it could go in v1.

FYI. The iio mount matrix sysfs entry is READ only.

@laykuanloon please close the ticket


chore: folderize tryit

Fixes: https://github.com/stoplightio/elements/issues/1100
Reducing entropy

What are we going to swap now? 😢

Babel warnings 😉


TorXakis - SMT - Regex - Parse \xhh

Also parse escape chars with hexadecimal code: \xhh
Note \xhh is NOT supported by XSD regexes
How to handle it?
Charsets in XML 1.1 standard
References
Support character references in regex:
&# [0-9]+ ;
&#x [0-9a-fA-F]+ ;

related to #358

Leave a comment


When typing the cursor and entered text gets centered on the page.

Hey, I love the theme and plan on using it for some RPG stuff I'm writing.
However, When typing the cursor and entered text gets centered on the page, and then once you click out of the area, the text is left-aligned like you'd expect. While the text ends up ok in basic paragraphs, it makes it hard to edit and see where things are going.
You can see the starting line is inserted in the center of the page.

Once you select somewhere else, it snaps to left-justified

Unfortunately, I think this behaviour is just how CSS handles columns - when you've got barely any content, it will start off in the second column. If anyone who knows more about CSS than I do knows a way around it, please feel free to reopen this issue & then put in a pull request to fix it!


Issue273: Added FlowDefinition for handling supplied flows

Thanks for your pull request.  Please review the following guidelines.

[x] Title includes issue id.
[x] Description of the change added.
[ ] Commits are squashed.
[x] Tests added.
[x] Documentation added/updated.
[x] Also please review CONTRIBUTING.md.

This is the initial pull request with separate commits for code, test, and doc updates. Will squash when ready to merge. Addressed some comments but not all.


Perma-pending resource when using Kafka CRD (b/c of CRD race condition?)

Problem
Got report of a CRD use case where the custom resource was stuck in "pending" forever, even though apparently all of its pods were ready. See Tiltfile for repro, provided by @jfancher.
Repro steps:

curl https://gist.githubusercontent.com/maiamcc/4336a0ef15ff3cb77faa2cdeb6872d5c/raw/d12b22875d6d3689de74bae39b2927bc0f391482/CRD-race-condition-repro.Tiltfile > Tiltfile
tilt up, wait for things to spin up
cluster resource will remain pending, even though according to kubectl -n kafka get pods, it has created multiple pods (cluster-kafka-0, cluster-zookeeper-[0-2]) and all are Running)
ctrl+c to exit Tilt, and tilt up again; this time, cluster eventually goes green (as does its dependent resource, busybox)

Expected: cluster and busybox go green on the first tilt up
Investigation so far

when cluster is stuck in pending, it's because Tilt doesn't know about any of its pods

in Tilt runs where cluster doesn't come up, we see a lot of log spam of:

Handling pod update (“cluster-zookeeper-0”): error mapping kafka.strimzi.io/Kafka: no matches for kind “Kafka” in group “kafka.strimzi.io”.

The above error is thrown here, when the Dynamic REST Mapper fails to look up an object by group+kind.

here's the call chain when we get an error for the one of the problematic pods:
goroutine 2918 [running]:
github.com/windmilleng/tilt/internal/k8s.K8sClient.GetByReference(0x5e5deb9, 0x12, 0x62085e0, 0xc00072f900, 0x6285ec0, 0xc000200460, 0xc0004c4600, 0x61e7fc0, 0xc00072e880, 0x5e4a657, ...)
    /Users/maia/code/go/src/github.com/windmilleng/tilt/internal/k8s/client.go:396 +0x519
github.com/windmilleng/tilt/internal/k8s.OwnerFetcher.ownersOfMeta(0x6283620, 0xc0005d61e0, 0xc00025a870, 0xc0001a9688, 0x6242f20, 0xc0010c2a80, 0x6258de0, 0xc000010758, 0x0, 0x0, ...)
    /Users/maia/code/go/src/github.com/windmilleng/tilt/internal/k8s/owner_fetcher.go:149 +0x298
github.com/windmilleng/tilt/internal/k8s.OwnerFetcher.ownerTreeOfHelper(0x6283620, 0xc0005d61e0, 0xc00025a870, 0xc0001a9688, 0x6242f20, 0xc0010c2a80, 0xc001478230, 0xb, 0xc001478279, 0x5, ...)
    /Users/maia/code/go/src/github.com/windmilleng/tilt/internal/k8s/owner_fetcher.go:130 +0x112
github.com/windmilleng/tilt/internal/k8s.OwnerFetcher.OwnerTreeOf(0x6283620, 0xc0005d61e0, 0xc00025a870, 0xc0001a9688, 0x6242f20, 0xc0010c2a80, 0x62032e0, 0xc000010758, 0x0, 0x0, ...)
    /Users/maia/code/go/src/github.com/windmilleng/tilt/internal/k8s/owner_fetcher.go:125 +0x236
github.com/windmilleng/tilt/internal/k8s.OwnerFetcher.ownerTreeOfHelper(0x6283620, 0xc0005d61e0, 0xc00025a870, 0xc0001a9688, 0x6242f20, 0xc0010c2a80, 0x58a3dab, 0x3, 0xc00151d3f9, 0x5, ...)
    /Users/maia/code/go/src/github.com/windmilleng/tilt/internal/k8s/owner_fetcher.go:135 +0x1c3
github.com/windmilleng/tilt/internal/k8s.OwnerFetcher.OwnerTreeOf(0x6283620, 0xc0005d61e0, 0xc00025a870, 0xc0001a9688, 0x6242f20, 0xc0010c2a80, 0x6201120, 0xc000807400, 0x0, 0x0, ...)
    /Users/maia/code/go/src/github.com/windmilleng/tilt/internal/k8s/owner_fetcher.go:125 +0x236
github.com/windmilleng/tilt/internal/engine/k8swatch.(*PodWatcher).dispatchPodChange(0xc000730a80, 0x6242f20, 0xc0010c2a80, 0xc000807400, 0x6243720, 0xc0000ce0a0)
    /Users/maia/code/go/src/github.com/windmilleng/tilt/internal/engine/k8swatch/pod_watch.go:257 +0xad
created by github.com/windmilleng/tilt/internal/engine/k8swatch.(*PodWatcher).dispatchPodChangesLoop
    /Users/maia/code/go/src/github.com/windmilleng/tilt/internal/engine/k8swatch/pod_watch.go:291 +0x1a4

Possible workarounds

brute force: add a timeout (by means of a local_resource?) after uncategorized applies, to give the CRD time to be... discoverable?? Properly mapped?? This seems to work at least sometimes 😬

local_resource('sleep', 'sleep 15', resource_deps=["uncategorized"])
...
k8s_resource("strimzi-cluster-operator", resource_deps=["sleep"])

find a way to actually check if the CRD is ready/discoverable/whatever:

local_resource('check-crd', '...wait for crd to be mapped...', resource_deps=["uncategorized"])
...
k8s_resource("strimzi-cluster-operator", resource_deps=["check-crd"])

something else?

good investigation!
If the the pod watcher sees a pod, and it doesn't know the type of that pod's parent CRD, then it seems like we can safely assume that the CRD is ready. (since the CRD controller is what created the pod in the first place!) Can we just reset the REST mapper and retry?


Acrolinx quality improvements: modify-root-navigation.md

@robinharwood, @Xelu86
Can you review the proposed changes? IMPORTANT: When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

@gswashington, a few updates but looks good. Merging now.