[TF 2.0] categorical_column_with_vocabulary_list not usable in custom training loop

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): Source
TensorFlow version (use command below): v2.0.0-beta0-16-g1d91213fe7 2.0.0-beta1
Python version: 3.6.8
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

Describe the current behavior
Outside of fit, e.g in a custom training loop, categorical_column_with_vocabulary_list results in an error. I have provided a modified version of Classifying Structured data which demonstrates this.
The error is ValueError: Column dtype and SparseTensors dtype must be compatible. key: thal, column dtype: <dtype: 'string'>, tensor dtype: <dtype: 'int32'>
Describe the expected behavior
Code runs without causing an error
Code to reproduce the issue
It should be directly copy-paste-able
import pandas as pd
import tensorflow as tf
from sklearn.model_selection import train_test_split

def df_to_dataset(df, shuffle=True, batch_size=32):
    df = df.copy()
    labels = df.pop('target')
    ds = tf.data.Dataset.from_tensor_slices(
        (dict(df), labels)
    )
    if shuffle:
        ds = ds.shuffle(buffer_size=len(df))
    ds = ds.batch(batch_size)
    return ds

def generate_features():
    feature_columns = []
    feature_layer_inputs = {}

    thal = tf.feature_column.categorical_column_with_vocabulary_list(
          'thal', ['fixed', 'normal', 'reversible'])
    thal_one_hot = tf.feature_column.indicator_column(thal)
    feature_columns.append(thal_one_hot)
    feature_layer_inputs['thal'] = tf.keras.Input(shape=(1,), name='thal', dtype=tf.string)

    return feature_columns, feature_layer_inputs

def create_model(feature_columns, feature_layer_inputs):
    input_layer = tf.keras.layers.DenseFeatures(feature_columns)
    inputs = input_layer(feature_layer_inputs)

    l1 = tf.keras.layers.Dense(128, activation='relu')(inputs)
    l2 = tf.keras.layers.Dense(128, activation='relu')(l1)

    output = tf.keras.layers.Dense(1, activation='sigmoid')(l2)

    model = tf.keras.Model(
        inputs=[v for v in feature_layer_inputs.values()],
        outputs=[output]
    )
    return model

def make_loss(loss_object):
    def loss(model, x, y):
        y_pred = model(x)
        return loss_object(y_true=y, y_pred=y_pred)
    return loss

def grad(model, inputs, targets, loss):
    with tf.GradientTape() as tape:
        loss_value = loss(model, inputs, targets)
    return loss_value, tape.gradient(loss_value, model.trainable_variables)

def fit(epochs, train_ds, model, optimizer, loss_obj):
    loss = make_loss(loss_obj)
    for epoch in range(epochs):
        for i, (x, y) in enumerate(train_ds):
            loss_values, grad_values = grad(model, x, y, loss)
            optimizer.apply_gradients(zip(grad_values, model.trainable_variables))

if __name__ == '__main__':
    URL = 'https://storage.googleapis.com/applied-dl/heart.csv'
    df = pd.read_csv(URL)
    CUSTOM_TRAINING = True

    train, test = train_test_split(df, test_size=0.2)
    train, val = train_test_split(train, test_size=0.2)

    # hardcoded stuff
    batch_size = 32
    train_ds = df_to_dataset(train, batch_size=batch_size)

    # Create model and features
    feature_columns, feature_layer_inputs = generate_features()
    model = create_model(feature_columns, feature_layer_inputs)

    if CUSTOM_TRAINING:
        print('Trying custom training')
        bce = tf.keras.losses.BinaryCrossentropy()
        adam = tf.keras.optimizers.Adam()
        fit(epochs=5, train_ds=train_ds,
            model=model, optimizer=adam, loss_obj=bce)
    else:
        print('Using pre-defined fit')
        model.compile(optimizer='adam',
                      loss='binary_crossentropy',
                      metrics=['accuracy'])
        model.fit(train_ds, epochs=5)

If you flip the CUSTOM_TRAINING variable between True and False (line 72) you'll see what I mean.
Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

The complete (relevant) stacktrace is

  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1758, in <module>
    main()
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1752, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1147, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm CE.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/ian.quah/PycharmProjects/tf2/datasets/issues.py", line 90, in <module>
    model=model, optimizer=adam, loss_obj=bce)
  File "/Users/ian.quah/PycharmProjects/tf2/datasets/issues.py", line 65, in fit
    loss_values, grad_values = grad(model, x, y, loss)
  File "/Users/ian.quah/PycharmProjects/tf2/datasets/issues.py", line 57, in grad
    loss_value = loss(model, inputs, targets)
  File "/Users/ian.quah/PycharmProjects/tf2/datasets/issues.py", line 50, in loss
    y_pred = model(x)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 712, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 753, in call
    return self._run_internal_graph(inputs, training=training, mask=mask)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 895, in _run_internal_graph
    output_tensors = layer(computed_tensors, **kwargs)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 712, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column_v2.py", line 474, in call
    self._state_manager)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column_v2.py", line 4299, in get_dense_tensor
    return transformation_cache.get(self, state_manager)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column_v2.py", line 2562, in get
    transformed = column.transform_feature(self, state_manager)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column_v2.py", line 4238, in transform_feature
    transformation_cache, state_manager)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column_v2.py", line 3714, in get_sparse_tensors
    transformation_cache.get(self, state_manager), None)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column_v2.py", line 2562, in get
    transformed = column.transform_feature(self, state_manager)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column_v2.py", line 3692, in transform_feature
    return self._transform_input_tensor(input_tensor)
  File "/anaconda3/envs/mlpl/lib/python3.6/site-packages/tensorflow/python/feature_column/feature_column_v2.py", line 3668, in _transform_input_tensor
    self.key, self.dtype, input_tensor.dtype))
ValueError: Column dtype and SparseTensors dtype must be compatible. key: thal, column dtype: <dtype: 'string'>, tensor dtype: <dtype: 'int32'>

Placing a debugger on line 50 leads me to feature_column_v2.py specifically _transform_input_tensor

The input_tensor arg to _transform_input_tensor is
SparseTensor(indices=tf.Tensor(
[[ 0  0]
 [ 1  0]
 [ 2  0]
 [ 3  0]
 [ 4  0]
 [ 5  0]
 [ 6  0]
 [ 7  0]
 [ 8  0]
 [ 9  0]
 [10  0]
 [11  0]
 [12  0]
 [13  0]
 [14  0]
 [15  0]
 [16  0]
 [17  0]
 [18  0]
 [19  0]
 [20  0]
 [21  0]
 [22  0]
 [23  0]
 [24  0]
 [25  0]
 [26  0]
 [27  0]
 [28  0]
 [29  0]
 [30  0]
 [31  0]], shape=(32, 2), dtype=int64), values=tf.Tensor(
[49 34 58 46 59 47 55 58 41 68 62 51 61 46 39 48 37 41 51 59 51 70 60 57
 54 60 52 44 65 49 44 59], shape=(32,), dtype=int32), dense_shape=tf.Tensor([32  1], shape=(2,), dtype=int64))

which seems strange? It's like it forgot that it had transformed those variables?

Reproduced the error with TF Version 2.0.beta1

Reproduced the error with TF Version 2.0.beta1

Not to question your judgement (proceeds to question) but is this a performance bug?

Thanks!

Does anyone have an intuition as to why this happens in a custom loop but not in fit?

Does anyone have a quick fix?

New error for 'tf-nightly-2.0-preview           2.0.0.dev20190714 ' with @tf.function
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1758, in <module>
    main()
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1752, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1147, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/lqk/project/PycharmProjects/TF2/shared_embedding.py", line 44, in <module>
    df = shared_embedding_column_with_hash_bucket(dic_data)
  File "/Users/lqk/anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 429, in __call__
    return self._stateless_fn(*args, **kwds)
  File "/Users/lqk/anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1661, in __call__
    graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
  File "/Users/lqk/anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1991, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/Users/lqk/anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 1877, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/Users/lqk/anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py", line 791, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/Users/lqk/anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 310, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/Users/lqk/anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py", line 781, in wrapper
    raise e.ag_error_metadata.to_exception(type(e))
AttributeError: in converted code:
    relative to /Users/lqk:

    project/PycharmProjects/TF2/shared_embedding.py:36 shared_embedding_column_with_hash_bucket  *
        df = tf.keras.layers.DenseFeatures(color_column_embed)(dic_data)
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py:758 __call__
        outputs = self.call(inputs, *args, **kwargs)
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:508 call
        self._state_manager)
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:3418 get_dense_tensor
        return self._get_dense_tensor_internal(transformation_cache, state_manager)
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:3395 _get_dense_tensor_internal
        embedding_weights = self.shared_embedding_column_creator.embedding_weights
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/feature_column/feature_column_v2.py:3322 embedding_weights
        trainable=self._trainable)
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/ops/variable_scope.py:1500 get_variable
        aggregation=aggregation)
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/ops/variable_scope.py:1243 get_variable
        aggregation=aggregation)
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/ops/variable_scope.py:567 get_variable
        aggregation=aggregation)
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/ops/variable_scope.py:519 _true_getter
        aggregation=aggregation)
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/ops/variable_scope.py:861 _get_single_variable
        tb = var.op.traceback[::-1]
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:558 op
        return self._handle.op
    anaconda2/envs/p36t2/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:986 op
        "Tensor.op is meaningless when eager execution is enabled.")

    AttributeError: Tensor.op is meaningless when eager execution is enabled.

Adding @rohan100jain who is the expert on feature columns.

I think I've also reproduced this.
I'm trying to call predict_classes on a tf.keras.Sequential model that contains categorical_column_with_vocabulary_list, categorical_column_with_hash_bucket, and numeric_column. I'm passing predict_classes a list of NumPy arrays that contain the types numpy.int64, numpy.float64, and str.
I've reproduced the same error:
ValueError: Column dtype and SparseTensors dtype must be compatible. key: columnName, column dtype: <dtype: 'string'>, tensor dtype: <dtype: 'int64'>

columnName in the NumPy array is the correct type. It is a string in the form \d+, e.g. '1234' (it is categorical data that just so happens to be comprised of digits). The values in the NumPy array are in the correct order.
There are no problems with compile, fit, or evaluate from tf.keras.Sequential.
I was able to call predict when passing in a dataset instead of a list of NumPy arrays, but the output was a float. I was hoping for a binary classification, which is why I was trying predict_classes. (Any guidance here would also be tremendously appreciated).
System information

Have I written custom code: Yes
OS Platform and Distribution: macOS 10.14.6
Mobile device: N/A
TensorFlow installed from (source or binary): binary
TensorFlow version: v2.0.0-beta0-16-g1d91213fe7 2.0.0-beta1
Python version: 3.7.3
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source): N/A
CUDA/cuDNN version: N/A
GPU model and memory: N/A

Is this still an open issue?

facing same issue.

I am also facing the same issue. Any help...

This got resolved for me. The datatype of the vocabulary list passed to the method, the data in the column of the record and the data type that we specify in the model have to be the same. Was able to find the issue, it got resolved.

same issue with tensorflow 1.14
ValueError: Column dtype and SparseTensors dtype must be compatible. key: adt_sev_flag, column dtype: <dtype: 'int64'>, tensor dtype: <dtype: 'float64'>

This got resolved for me. The datatype of the vocabulary list passed to the method, the data in the column of the record and the data type that we specify in the model have to be the same. Was able to find the issue, it got resolved.

how did you solved it?

The have you checked if the datatype of the data in the vocab list, the datatype of the input data and the data type of the feature_column declaration are same. In my case, the vocab list that I passed had infered to a different datatype and I passed it into the categorical_column_with_vocabulary_list layer incorrectly.

I have tried this example in tf-nightly, which seems to have been resolved. It's probably a Keras issue instead of FeatureColumn. Can anyone help to verify?

I'm using 1.12 version and i have no problem. But today it just happen. I can't understand how and why im getting this error. From nowhere and no update no changes ? And what should be the fix?


publish fixes for old datasketches.github.io links

I hope you are also updating the source otherwise these changes will disappear.

yes, i submitted the PR for that against the old datasketches repo a couple days back, you already merged that.  i commit to that first before i generate and commit here.


Update vagrantfile

Raise RAM from 4096 to 5120 because of OOM with helloworld
Switch to jenkins-persistent-template to have persistant storage

@tnozicka Locally I'm actually going to 6GB for same reason.  Any issues with doing that instead?

Closed via #6 and #7


Remove UnbannedReflectiveBuildItem + policy extension ?

When a component uses the camel-api-component-maven-plugin (olingo4, box, fhir, zendesk etc...)  one has to register the Configuration classes for reflection using ReflectiveClassBuildItem.
With the Policy extension, one also has to register the same classes with UnbannedReflectiveBuildItem or else the build fails.
Since these are valid uses of ReflectiveClassBuildItem I really don't think the build should fail.

We currently ban classes annotated with @org.apache.camel.spi.UriParams that actually should not need reflection, because there are means on the Camel site how to avoid it. If those means do not work, one should file a Camel issue or even better fix it on the Camel side. Unbanning should be seen only as a temporary measure to make things work. Registering for reflection comes at some cost (image size, runtime efficiency) and we want to avoid it whereever possible.

I'd say that we should trust a little extensions developers and thus evaluation should be done mainly through the PR review process.

I'm not sure I understand, some classes annotated with @org.apache.camel.spi.UriParams do need reflection, so why make the build fail ?

Because on the camel side there were efforts to avoid reflection and there are multiple commits where this is visible on each component. It wasn't possible to do this on all the components btw, so I agree the build should not fail.

I agree with @lburgazzoli, @oscerd
Build failures should really only occur in critical situations. IMO this hurts the overall developer experience here

I am sorry for the inconvenience, but I still think that registering a class for reflection that belongs to a well known group that is expected not to need it, should not go without checking it properly.
Speaking for me, I do not feel able to eval during the review by pure sight whether some Camel class has some specific annotation. I think the policy does some useful work there. If we really want to tackle this, we should keep the policy.
Going through tens of extensions manually afterwards to check whether we register for reflection only what is necessary is a lot of work, that I did twice and I would not wish anybody to that again.
However, if we actually do not have the ambition to do this properly, I won't be against removing the policy.
By no means this inconvenience can impact Camel Quarkus application developers unless they consciously depend on the policy extension. This is only the case in our integration tests. Failing is the right thing to do there because what does not fail the build gets ignored.

As a compromise, can we not just log a message at WARN or DEBUG level if UriParams stuff is registered for reflection? At least that'll give an easy way to track stuff down for investigation.

I think the warning is the right thing as it also avoid to keep a list of white listed classes (that needs to be maintained)

OK, accepting that the policy is not seen as useful by the majority and thus also switching to passive mode for the solving of the underlying problem.

All the Camel components now do not use reflection when configuring themselves. We have those generated Configurer classes today. Today we fixed one (hopefully last) issue in the api components that @jamesnetherton discovered. Thats fixed in Camel 3.5 onwards (we can backport to 3.4.3)


how can I see/get the AST of a vulnerability

I want to konw how can I see/get the AST of a vulnerability?
I use the kaybee pull command to get the bugs of kb. However, I just get the source code about a cve vulnerability, in which the statement.yaml describes the fixes info about a vul.
So, how can I see/get the AST of a vulnerability, as I know steady extract the AST of the vulnerability code and fixed code.
Looking forward to your reply, thank you very much!

ASTs are computed by Eclipse Steady during the import of the vulnerabilities in the form of statements from project KB. They are then stored in the database and available from the rest endpoints of the rest-backend.
To see/get the ASTs you can set up the steady backend services and either wait for the initial import to be completed, or use the kb-importer module to analyze vulnerabilities starting from statements.
To set up the steady backend services you can follow the guide at https://eclipse.github.io/steady/admin/tutorials/docker/
The initial import of vulnerabilities starts as soon as the docker containers are started by takes hours to complete. You can check the vulnerabilities already imported using the endpoint http://localhost:8033/bugs To access ASTs you can use the endpoint http://localhost:8033/bugs/CVE-XXXX-YYYY, the ASTs are the vulnerable and fixed methods provided under the fields "buggyBody" and "fixedBody" respectively (for method and constructors whose change type is MOD).
Alterantively, to analyze statements explicitly, you can follow the guide at https://eclipse.github.io/steady/user/manuals/updating_vuln_data/

Thanks for your reply.


ARROW-403: [Java] Create transfer pairs for internal vectors in UnionVector transfer impl

@StevenMPhillips, @julienledem Could you please review the patch?

@julienledem: Updated the patch which removes Callback argument changes.

LGTM. Will merge when the build passes.


AmazonS3FileSystemListing get stuck on listObjects when bucket contains more than max response items

seams like the marker isn't being updated on the request so the object listing loop process infinitely.
suggested fix:
public class AmazonS3FileSystemListing implements FileSystemListing<AmazonS3Storage> {
...
    public Collection<FileObjectMeta> listObjects() {
....
                 if (marker != null) {
                    LOG.debug("Object listing is truncated, next marker is {}", marker);
                    request.setMarker(marker); // add this line
                }
  }
}

Hi @yairbit, thank you for opening this issue and for the fix suggestion :)


Add persistence for localfs datasets

What does this PR do?
Add persistency logic for localfs datasetio provider

[ ] Addresses issue (#issue)

Test Plan
Please describe:

tests you ran to verify your changes with result summaries.
provide instructions so it can be reproduced.

Sources
Please link relevant resources if necessary.
https://github.com/meta-llama/llama-stack/issues/539
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Ran pre-commit to handle lint / formatting issues.
[x] Read the contributor guideline,
Pull Request section?
[ ] Updated relevant documentation.
[ ] Wrote necessary unit or integration tests.

Hi @VladOS95-cyber , could you attach the tests you ran in the test plan?

Hi @VladOS95-cyber thanks for the PR, could you attach the tests you ran in the test plan?

Hi @yanxi0830! Do you mean unit test results I executed?

Hi @yanxi0830! My test results in attach:

I don't think that my changes is a root cause of failed one. I found that in def get_dataframe_from_url(url: URL): in url_utils.py, df = pandas.read_excel(data_bytes) line 41 tries to read data_bytes that was wrapped upon data of 'mimetype': 'application/vnd.ms-excel'. pandas.read_excel throws an expection trying to read it and it does not matter if we specify engines manually or not. But it could successfully read by df = pandas.read_csv(data_bytes) and does not throw any exceptions and all tests are green.
I could try to provide my fix for it if it is needed, or I miss something here?

Hi @yanxi0830! Just a kind reminder, please take a look when you have time, in order to not lose this PR.


⚠ TorrentLeech has degraded performance

In c05b117, TorrentLeech ($TL) experienced degraded performance:

HTTP code: 200
Response time: 1846 ms

Resolved: TorrentLeech performance has improved in 4933993 after 11 minutes.


Latex $$ fix

PR Checklist
If you are contributing to Javis.jl, please make sure you are able to check off each item on this list:

[x] Did I update CHANGELOG.md with whatever changes/features I added with this PR?
[x] Did I make sure to only change the part of the file where I introduced a new change/feature?
[x] Did I cover all corner cases to be close to 100% test coverage (if applicable)?
[x] Did I properly add Javis dependencies to the Project.toml + set an upper bound of the dependency (if applicable)?
[x] Did I properly add test dependencies to the test directory (if applicable)?
[x] Did I check relevant tutorials that may be affected by changes in this PR?
[x] Did I clearly articulate why this PR was made the way it was and how it was made?

Link to relevant issue(s)
Closes #306
How did you address these issues with this PR? What methods did you use?
Check if $ in the LaTeXString before removing.

Thanks a lot! Do you mind to change the version directly in the Project.toml? :wink:

Regarding the test failure:
You can only use LaTeX commands which are in the LaTeXSVG dict

Regarding the test failure:
You can only use LaTeX commands which are in the LaTeXSVG dict

This doesn't make sense to me by looking at the source. Tests pass locally. The CI logs seem to indicate that tex2svg is not installed.

Are you saying it needs to be included in the Dict constructor in latexsvgfile.jl? If, so it will end up just bypassing the code that needs tested. I can understand why in your other tests you want to bypass txt2svg and use a reference svg, but does that makes sense here?

Good point we might want to save the ts in the dict and not the text, right? Then you can do your step before checking if it is cached. It currently fails because it's not cached and then we try to render it but haven't installed the npm module on GitHub.

That would work, but will require changing the types used in LaTeXSVG as the key won't be LaTeXString anymore. I'm unclear on what the upstream affects of that would be.
Two other thoughts:

Why not just install the npm module?
I could bypass all this by moving to code to a function, e.g strip_eq(txt::LaTeXString) = txt[1] == '$' ? txt[2:(end-1)] : txt. The test will then just be a test of this function.

I like the first option but I have not enough knowledge how to do it. I'm fine with option 2 for now

OK, I will do 2. However, as a future note, from a cache perspective it would be better to cache based on the stripped string because now you can have two different keys that produce the same svg., e.g. \\sqrt{x} and $\\sqrt{x}$.

Totally agree. Maybe add a Todo: in the code 😊

done

Thanks a lot! Enjoy the holidays :wink:


Allow constant outputs in ForwardDiff

Fix #525

Codecov Report
Attention: Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Project coverage is 7.14%. Comparing base (665cc2c) to head (730db92).

Files with missing lines
Patch %
Lines

...xt/DifferentiationInterfaceForwardDiffExt/utils.jl
0.00%
3 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (665cc2c) and HEAD (730db92). Click for more details.
HEAD has 122 uploads less than BASE

Flag
BASE (665cc2c)
HEAD (730db92)

DIT
30
0

DI
93
1

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #526       +/-   ##
==========================================
- Coverage   98.60%   7.14%   -91.46%
==========================================
  Files         106      70       -36
  Lines        4593    2980     -1613
==========================================
- Hits         4529     213     -4316
- Misses         64    2767     +2703

Flag
Coverage Δ

DI
7.14% <0.00%> (-91.59%)
:arrow_down:

DIT
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


fix: return undefined from FetchCacher#load on NotFound

This makes FetchCacher#load return undefined when the file fetcher returns NotFound error in order to align it with Deno CLI's implementation:
let error_class_name = get_error_class_name(&err);
match error_class_name {
  "NotFound" => Ok(None),
  "NotCached" if options.cache_setting == LoaderCacheSetting::Only => Ok(None),
  _ => Err(err),
}

With this patch, deno_emit can bundle source code containing imports from JSR.
Let's say we have a bundle target script as follows:
import { add } from "jsr:@magurotuna/test-add";

console.log(add(1, 3));

Without this patch, running bundle function from deno_emit on this script gives:
error: Uncaught (in promise) Error: Unable to output during bundling: load_transformed failed: failed to analyze module: failed to resolve jsr:@magurotuna/test-add from file:///Users/yusuke/Repo/github.com/magurotuna/deno_emit/js/maguro_add.ts: Cannot resolve "jsr:@magurotuna/test-add" from "file:///Users/yusuke/Repo/github.com/magurotuna/deno_emit/js/maguro_add.ts".
      const ret = new Error(getStringFromWasm0(arg0, arg1));
                  ^
    at __wbg_new_28c511d9baebfa89 (file:///Users/yusuke/Repo/github.com/magurotuna/deno_emit/js/emit.generated.js:563:19)
    at <anonymous> (wasm://wasm/010ad402:1:3297307)
    at <anonymous> (wasm://wasm/010ad402:1:358125)
    at <anonymous> (wasm://wasm/010ad402:1:1823927)
    at <anonymous> (wasm://wasm/010ad402:1:2937828)
    at __wbg_adapter_46 (file:///Users/yusuke/Repo/github.com/magurotuna/deno_emit/js/emit.generated.js:247:6)
    at real (file:///Users/yusuke/Repo/github.com/magurotuna/deno_emit/js/emit.generated.js:231:14)
    at ext:core/01_core.js:291:9
    at eventLoopTick (ext:core/01_core.js:175:7)

With this patch, it can generate:
function add(a, b) {
    return a + b;
}
console.log(add(1, 3));

Can you add a test?

Done 74da9e6

That said, it looks like the code already handles this, but you are encountering an error. Where is this error surfacing from?

Not sure exactly, but the approach I took to finding where to fix is as follows:

Found the error message NotFound: Specifier not found in cache<EMAIL_ADDRESS>--cached-only is specified.
Spotted where this message comes from: https://github.com/denoland/deno_cache_dir/blob/ed6ea63e05b678e3e9f8834d45d1c911d485a9bb/file_fetcher.ts#L207-L211
Wondered why cacheSetting is set to only even though its default is use and I didn't specify it. Discovered that in deno_graph there is a place calling to load with CacheSetting::Only forcibly: https://github.com/denoland/deno_graph/blob/019728c7628d6b47222894c1356a3c9202d9204a/src/graph.rs#L3953
Also, according to the CacheSetting::Only doc comment, the intended behavior of load method when target file is not found in the cache is to return Ok(None), not an error. This can be verified by looking at FetchCacher load impl in Deno CLI, where it attempts to fetch a requested file, and then if NotFound error is returned it is converted into Ok(None).


Chdir() at launch

suggested by @samiebee43 to resolve #294
also trying to find a way to keep the console window suppressed on Win10 and Win11 without --windowed (which messes up other things) (see https://github.com/pyinstaller/pyinstaller/pull/7735)

Tested on Win11 laptop via installer. Chdir() works as expected, console window at least minimizes (doesn't appear at all on Win10).


Add fcontext resource to manage file contexts.

One more try. I think I have everything now. I have no idea why git deletes and inserts you as author and I don't care. Its the same line...

You can rebase your original PR, no need to create new ones :)
Close whichever you like and I'll look at the other one.
On Mar 3, 2015 11:08 AM, "Jörg Herzinger"<EMAIL_ADDRESS>wrote:

One more try. I think I have everything now. I have no idea why git
deletes and inserts you as author and I don't care. Its the same line...
You can view, comment on, or merge this pull request online at:
https://github.com/BackSlasher/chef-selinuxpolicy/pull/5
Commit Summary

Add fcontext resource to manage file contexts.

File Changes

M README.md
https://github.com/BackSlasher/chef-selinuxpolicy/pull/5/files#diff-0
(37)
A libraries/helper-restorecon.rb
https://github.com/BackSlasher/chef-selinuxpolicy/pull/5/files#diff-1
(19)
A providers/fcontext.rb
https://github.com/BackSlasher/chef-selinuxpolicy/pull/5/files#diff-2
(53)
A resources/fcontext.rb
https://github.com/BackSlasher/chef-selinuxpolicy/pull/5/files#diff-3
(7)

Patch Links:

https://github.com/BackSlasher/chef-selinuxpolicy/pull/5.patch
https://github.com/BackSlasher/chef-selinuxpolicy/pull/5.diff

—
Reply to this email directly or view it on GitHub
https://github.com/BackSlasher/chef-selinuxpolicy/pull/5.

I'm accepting this but I'll probably mess with your commit a bit. I don't want to do this chinese whispers thing with you because you worked hard and my edits are minute.
I'm told you'll still get the commit credit, and you'll forever be in the README of fame :)

Thats what pull requests are for. If I was not happy with you editing my code I would have forked your project. ;)
Btw: Opscode also showed some interest in selinux. Maybe they will fork this project some day. https://github.com/chef-cookbooks/mysql/issues/305

Didn't really merge to master, will merge privately

Solved the regex issue btw, you have to specify regex type before the regex expression because find was designed to annoy people.


Error occurs when importing metagraph that contains cudnnRNN cells

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS 7.4
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): pip install .whl
TensorFlow version (use command below): 1.12.0
Python version: 2.7.14
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source): c++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
CUDA/cuDNN version: cuda: 9.0 cuDNN: 7.1.3
GPU model and memory: GTX1050Ti/4GB
tf_env.txt

Describe the current behavior
I try to import graph from a MetaGraphDef proto that contains a CudnnRNN cell, the error below raises:
 Traceback (most recent call last):
  File "test.py", line 21, in <module>
    saver = tf.train.import_meta_graph(mgd)
  File "/root/anaconda2/envs/tf12/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1674, in import_meta_graph
    meta_graph_or_file, clear_devices, import_scope, **kwargs)[0]
  File "/root/anaconda2/envs/tf12/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1696, in _import_meta_graph_with_return_elements
    **kwargs))
  File "/root/anaconda2/envs/tf12/lib/python2.7/site-packages/tensorflow/python/framework/meta_graph.py", line 852, in import_scoped_meta_graph_with_return_elements
    ops.prepend_name_scope(value, scope_to_prepend_to_names))
  File "/root/anaconda2/envs/tf12/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3490, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/root/anaconda2/envs/tf12/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3550, in _as_graph_element_locked
    "graph." % repr(name))
KeyError: "The name 'cudnn_lstm/opaque_kernel_saveable' refers to an Operation not in the graph."

What I've done:
I've debuged the souce code and found out that a CudnnLSTMSaveable instance is created when cudnnRNN cell gets called and corresponding op was saved in tf.GraphKeys.SAVEABLE_OBJECTS collections with the name "cudnn_lstm/opaque_kernel_saveable".
when executing saver.import_meta_graph, in saver.save_op(line 189 of saver.py) function the saveable object mentioned above is somehow transfered into "cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel" and "cudnn_lstm/rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/bias" and gets saved. But the op named "cudnn_lstm/opaque_kernel_saveable" didn't gets saved. I'm wandering if this is a bug.
Describe the expected behavior
graph gets imported
Code to reproduce the issue
import tensorflow as tf
import numpy as np
num_layers = 1
num_units = 128
direction = "unidirectional"
inputs    = tf.placeholder(tf.float32, [None, None, 32], name="inputs")
convolved = tf.transpose(inputs, [1, 0, 2])

lstm      =  tf.contrib.cudnn_rnn.CudnnLSTM(num_layers, num_units, direction=direction)
outputs, output_states = lstm(convolved, training=True)
enc_output = tf.transpose(outputs, [1, 0, 2])
optimizer  = tf.train.AdamOptimizer(0.001)
optim = optimizer.minimize(enc_output)
saver     = tf.train.Saver()
init      = tf.global_variables_initializer()
with tf.Session() as session:
    session.run(init)
    # export MetaGraphDef proto to mgd
    mgd=saver.export_meta_graph()
with tf.Session(graph=tf.Graph()) as session:
    # import graph from mgd on another graph
    saver = tf.train.import_meta_graph(mgd)

Getting the same issue while loading SavedModel with cudnn_lstm layer inside. I thought maybe I need to prune these nodes, but when I inspected graph with tensorboard it couldn't find any! Moreover, due to the same issue it's impossible to launch Graph Transform Tool on it.
I think I'm close as never before to one more one-layer perceptron for university exam project...

I'm struggling with the same issue. I have saved my network using CudnnLSTM.
cudnn_lstm =tf.contrib.cudnn_rnn.CudnnLSTM(num_layers=2, num_units=36, dtype=tf.float32)
After that, using the code seen below, I'm trying to restore variables like input placeholder or accuracy.
`graph = tf.Graph()
with tf.Graph().as_default(), tf.device('/gpu:0'):
with tf.Session() as sess:
saver = tf.train.import_meta_graph('checkpoints/lstm.ckpt.meta')
saver.restore(sess, tf.train.latest_checkpoint('checkpoints/.'))
    input = graph.get_tensor_by_name('inputs:0')`

But I get this error, " The name 'cudnn_lstm/opaque_kernel_saveable' refers to an Operation not in the graph'.

I'm struggling with the same problem. I'have saved my network with using CudnnLstm ;
cudnn_lstm = tf.contrib.cudnn_rnn.CudnnLSTM(num_layers=2, num_units=36, dtype=tf.float32)
I saved it in a checkpoint file. After that, using the code seen below, I'm trying to restore variables like input placeholder or accuracy.
graph = tf.Graph()
with tf.Session(graph=graph) as sess:
    saver = tf.train.import_meta_graph('checkpoints-cnn/har.ckpt.meta')
    saver.restore(sess, tf.train.latest_checkpoint('checkpoints-cnn/.'))

    input = graph.get_tensor_by_name('inputs:0')

But I get this error " The name 'cudnn_lstm/opaque_kernel_saveable' refers to an Operation not in the graph'

@protoget Anything new? I'm getting this exact same error but with cudnn_gru,
" The name 'cudnn_gru/opaque_kernel_saveable' refers to an Operation not in the graph'

@protoget Anything new? I have the same problem

When does KeyError:The name 'something' refers to an Operation not in the graph occur. facing similar issue. What operation is being performed(possibly in ops.py) that leads to this error?
Also, kindly look this

As a temporary work around I used tf.keras.layers.CuDNNLSTM to build my RNN layers.

@shahinkl  I solved it, you should rebuild model and restore ckpt instead of using  import_meta_graph

@brennenhuang
We know it will work with restore ckpt. That is why we are able to do validation and testing while training the graph. We face the issue while exporting and deploying the saved_model

Anything new? I have the same problem

Same issue here. Any pointers?

Same issue here

Same issue

same issue

same issue, any update?

bump, same issue @drpngx @protoget

As a temporary work around I used tf.keras.layers.CuDNNLSTM to build my RNN layers.

how to do this with eager execution off??

same issue, any update?

same issue,solved?

Using tensorflow serving to deploy metagraphdef proto does not have this issue

same issue, any possible solution?

same issue when importing graph from .meta file by using tf.train.import_meta_graph？ tensorflow version==1.14.0, looking forward to solve.

same issue, any possible solution?

I find this problem is because one node named 'cudnn_lstm/opaque_kernel_saveable' was added to graph.collection['saveable_objects'] but this node not added to saved model.
when i build saved model, i run print(sess.graph.get_collection('saveable_objects')), then i got:
[<tensorflow.contrib.cudnn_rnn.python.ops.cudnn_rnn_ops.CudnnLSTMSaveable object at 0x7fd0881535d0>]
so, i try this before building saved model: sess.graph.clear_collection('saveable_objects'), then i can restore saved model.
However, i am not sure whether this is right, and i can only delete all nodes in graph.collection['saveable_objects'], how can i only delete the node 'cudnn_lstm/opaque_kernel_saveable'?

I find this problem is because one node named 'cudnn_lstm/opaque_kernel_saveable' was added to graph.collection['saveable_objects'] but this node not added to saved model.
when i build saved model, i run print(sess.graph.get_collection('saveable_objects')), then i got:
[<tensorflow.contrib.cudnn_rnn.python.ops.cudnn_rnn_ops.CudnnLSTMSaveable object at 0x7fd0881535d0>]
so, i try this before building saved model: sess.graph.clear_collection('saveable_objects'), then i can restore saved model.
However, i am not sure whether this is right, and i can only delete all nodes in graph.collection['saveable_objects'], how can i only delete the node 'cudnn_lstm/opaque_kernel_saveable'?

same issue, any possible solution?

i used keras.layers.CuDNNLSTM (tensorflow version:1.14) to avoid this issue

Update, i find the codes add node named 'cudnn_lstm/opaque_kernel_saveable' to graph.collection['saveable_objects'] in /usr/lib64/python2.7/site-packages/tensorflow_core/contrib/cudnn_rnn/python/layers/cudnn_rnn.py L372-L373(Tensorflow 1.15):
# Create saveable in the outer scope of the cudnn subgraph, such that
# alternative subgraph with platform-independent rnn cells can load the
# checkpoints directly.
if not (self.built or vs.get_variable_scope().reuse is True):
    self._create_saveable()

So i delete them to slove this problem.

same issue, any possible solution?

@sjtuldl !
It seems you are using older versions(1.x versions) of Tensorflow which is not supported any more. We recommend that you upgrade to 2.8.0 and let us know if the issue still persists in newer versions .Thanks!

@sjtuldl !
It seems you are using older versions(1.x versions) of Tensorflow which is not supported any more. We recommend that you upgrade to 2.8.0 and let us know if the issue still persists in newer versions .Thank you!


Feature to Tag issue to another issue

Can issues be linked to same issues on the next release?

It probably won't make it for the initial 2.5 release, but after that I'm sure we can squeeze this in easy enough.


BUG: Cursor Duplicates when changing location in editor

Description
Instead of the cursor moving when the user clicks on or moves to a new spot, a remnant of the cursor remains and keeps blinking like the user's cursor is there. This seems to happen if the cursor resides in a spot for more than about 1/4 a second, and if the user clicks anywhere in the editor to move the cursor. The number of cursors seems to just increase. Please see the following video:
https://github.com/AuroraEditor/AuroraEditor/assets/104732280/28f7a091-adee-47a0-9c5c-47a50cc916cd
To Reproduce

Open AuroraEditor
Open any file
Click around in the file, moving the cursor.
Notice the issue: phantom cursors do not disappear.

Expected behavior
The cursor moves where the user moves it and the other cursors disappear as soon as the user moves the cursor from where they are. There should only be one visible cursor - assuming the user does not activate multi cursor (e.g. Option-clicking in Xcode). I did not for this demo.
Version information
Aurora Editor: Latest (1.0?)
Commit: 75c6fe3
MacOS: 14.4.1
Xcode: 15.3
Additional context
There's another issue that I've discovered with the editor too. I'll link that here in a second.

Thanks for submitting a issue!
to assign this issue to yourself please comment @aurora-care-bear please assign me


added TCP session

to speed up

Hi, have you added this to all the files or just L530?

to every instance in PyL530.py

Would it be possible to attach it to the p100 class that the others inherit from?

Lgtm


chore(main): release 0.4.0

:robot: I have created a release beep boop
0.4.0 (2023-05-01)
🔖 Features

add support for play functions (#57) (e309234)

🧹 Miscellaneous

deps: update all non-major dependencies (#54) (4993db7)
deps: update all non-major dependencies (#56) (e3a3396)
use renovate widenPeerDependencies preset (f7798bd)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/tobiasdiez/storybook-vue-addon/releases/tag/v0.4.0 :sunflower:


How to output / print specific metadata

Checklist

[X] I'm asking a question and not reporting a bug/feature request
[X] I've looked through the README
[X] I've read the guidelines for opening an issue
[X] I've searched the bugtracker for similar questions including closed ones

Question
I am wondering how I can print or output (to a file) specific metadata like upload date in the same order as the playlist (respecting if --playlist-reverse is used or not).
I did try to use --write-info-json but I could not find the upload date of each videos in the json files. I don't mind making a script to extract upload dates from individual json files, but it would be easier if that can be printed in the console directly.
Perhaps would it be possible to run a command that would output this metadata without downloading the videos ? Essentially I would like to have a way to bulk extract upload dates from a channel or playlist in a convenient way to be used elsewhere.
Sorry if this is a duplicate, thanks !

--print

I did try to use --write-info-json but I could not find the upload date of each videos in the json files

Depending on the website, the upload date may not be available

Oh, thanks !
Is there is a way to output --print to a file like output.txt ?

Not directly. You can always redirect stdout to a file instead


Tag releases

Hey!
It would be convenient to tag releases (like for other bindings). People using "go get" don't care much, but people using other package managers may find this useful (and distribution packagers too).

I've not done any releases yet, hence no tags. The first release will coincide with libvirt 3.0.0 and will be tagged


How to act on output file changes?

So I wrote this plugin on Friday: https://www.npmjs.com/package/broccoli-splitter
The main reason for that is the fact I have to split my final index.html file into two files which are then integrated into the rest of pages in a relatively large portal. The problem is that as soon as I watch the dist folder (because index.html which I have to take is the final one), there will be a cyclic process which triggers the build in an infinity loop.
is there a way to tell broccoli only to watch one file (dist/index.html) and not the whole dir? I can not find any thing similar online. or maybe some other way to get my hands on the output of the process which produces the dist/index.html file, before this file is written into disk?

The dist directory is not intended to be watched by Broccoli - trying it will inevitably lead to chaos.
It seems that what you're trying to do is take (some of) the output of Ember CLI's default stack, and emit something new based on it, right?
I haven't used Ember CLI much, but I assume you can get at the output (root) tree object somehow, right? I'd then pass that tree to other plugins like broccoli-splitter, and e.g. merge the results to get a new output tree.

is there a way to tell broccoli only to watch one file (dist/index.html) and not the whole dir? I can not find any thing similar online. or maybe some other way to get my hands on the output of the process which produces the dist/index.html file, before this file is written into disk?

not really, file-watching is OS implemented on a per DIR or per tree of dir basis.

I haven't used Ember CLI much, but I assume you can get at the output (root) tree object somehow, right? I'd then pass that tree to other plugins like broccoli-splitter, and e.g. merge the results to get a new output tree.

the tree exported in the Brocfile would be this tree.


not working on Linux

Hello sir
I cannot login from Mac to Linux and Windows to Linux but from Linux to Mac and Linux to Windows is possible.I think its due to Virtual IP i have on Linux
Also there is a way to give full support using your software thanks

Hey @gvsurenderreddy
I'm sorry. I didn't saw your issue until now. Something strange with githubs notifications.
Using my Windows VM, I could connect easily to my linux host.
What do you mean with "Virtual IP"?
Are there any errors in the console?

sir its working takes longtime to Load but its a very good work.Sir full remote control app is possible please

full remote is beyond the scope of this project. Maybe you want to checkout this electron based project. Seems like it does what you are requesting: https://electronjs.org/apps/thismypc

Thank you so much for you help.SO kind of your sir.I cannot say in words how much it helps me thanks again

Glad I cloud help :)


[Bug] main page click on collective is confusing

When I click on the collective text it expands it, I expect it to go to the collective page.
User is not clear where he needs to press to see collective

When I click on the collective text it expands it, I expect it to go to the collective page. User is not clear where he needs to press to see collective

This behavior was explicitly requested by your team

@krisbitney that's why I didnt assign it to you

@sirpy thanks for catching this - we will review in the coming two weeks once the MVP development is wrapped.


test_server.py: error: too few arguments

Seems to have been caused by #8

Fixed for now by e84925cc925394210b1e3836bc5ce334454ddac5


Fix V2 Rest book API return funding result error

Description:
rest book api return different sort by trading and funding,
and now the SDK only support trading.
Breaking changes:
None
New features:
None
Fixes:

[x] NewBookUpdateFromRaw func data comment error
[x] process funding symbol corerectly

PR status:

[ ] Version bumped
[ ] Change-log updated

@fshiori Really appreciate the PR, bump into the exact same problem here.
@JacobPlaster Would you please take a look and merge the PR? Or how can I help to merge this PR?
Thanks


Idea for a Light Weight Edition

Would it be possible to produce a reload4j-le (light-weight) edition that removes all appenders apart from the file and console appenders?
This would be useful for reassuring security teams as whilst new appenders could be added by configuration and additional JARS/classes they don't exist "by default". Also for the resulting JAR should be smaller so slightly better for containers (containers usually only log to console anyway).
Hopefully this can use the exact same code base but exclude classes with a new reload4j-le build file.
This kind of ties into issue #5

There is balance to be had. While moving chainsaw should upset many people, moving SMTPAppender is a bit different.

There is balance to be had.

I agree. That is why I suggested leaving the reloadj4 JAR as is pretty much so it will work as is for anyone that does not want change.
Meanwhile this would be a new JAR that applications can use but they need to realise that it is much more limited and does not contain anything more that the basic file and console appenders.


Artman logging publishing python

[Current coverage][cc-pull] is 69.76%

Merging [#31][cc-pull] into [master][cc-base-branch] will increase coverage by 0.28%

@@             master        #31   diff @@
==========================================
  Files            26         27     +1
  Lines           793        817    +24
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits            551        570    +19
- Misses          242        247     +5
  Partials          0          0

Powered by Codecov. Last updated by [5d41544...36fbbd7][cc-compare]
[cc-base-branch]: https://codecov.io/gh/googleapis/artman/branch/master?src=pr
[cc-compare]: https://codecov.io/gh/googleapis/artman/compare/5d415443aa0b833c13477bf2ed2360b3b2585596...36fbbd728e0ec8ecffa44855a6c1c68001f354ad
[cc-pull]: https://codecov.io/gh/googleapis/artman/pull/31?src=pr


Update changelog

https://github.com/Leaflet/Leaflet/pull/2382 and other changes since then.

Reopening to update the CHANGELOG.md document itself, which is a separate task from updating API docs.

[ ] add a section on #2382 changes
[ ] add recent #3278 changes
[ ] cover older commits not covered by https://github.com/Leaflet/Leaflet/commit/45034155bdf8d958554b43ed8dbd7020df9ca6ab

Done.


Merge/main to develop

What changes are proposed in this pull request?
Merging main into develop

Closing in favor of https://github.com/voxel51/fiftyone/pull/4187


MacOS CI

Contributions: We are not currently taking public contributions - see our contributions policy. However, we are accepting issues and we do want your feedback.

Description
Enabling MacOS CI for Buildkite. Also fixing a small bug in how we copy the schema and make the pre-exit hook cross-platform compatible.
What is still missing and will be done in a separate PR: reporting of the test and integrating that into slack.
can be reviewed in commit order
Release note
still need to be done
Tests
running many builds via buildkite. Will update with build as soon as the next one gets kicked off

Out of date and still a draft. Can we change both :o ?

PR: reporting of the test and integrating that into slack.
Ticket and TODO?

done, rebased on top of master and opened up the PR @joshuahuburn


update dependency versions

update following dependency versions:

pyserini
tokenizers
transformers
tqdm
sentencepiece

@stephaniewhoo can you see if our MS MARCO Passage ranking (https://github.com/castorini/pygaggle/blob/master/docs/experiments-msmarco-passage-subset.md) experiments pass with this? Then I'll merge it! Thanks :)

@stephaniewhoo can you see if our MS MARCO Passage ranking (https://github.com/castorini/pygaggle/blob/master/docs/experiments-msmarco-passage-subset.md) experiments pass with this? Then I'll merge it! Thanks :)

Hi, sorry for the late reply. I was waiting on cc. Yes, I have tried with these versions on cc for the subset exp, it works.

Okay, awesome! Merging! Thanks @stephaniewhoo


Link Support

Links

Link dialog should have both the text being linked and the link itself
Should be able to link to any Outline
Should be able to search for Outlines in the link dialog
Implement a custom scheme so that external apps can also link to Outlines

When adding links to text (Command-K), the pre-selected field in the new window is “Text” instead of “Link”, which slows down the actual linking process by having to use the pointer or the Tab key to be able to paste the link and conclude the operation.
Would it be possible to have “Link” be the preselected field?

Good suggestion. I'll add it to the 1.1 release.


Publically expose QuantumCircuit._layout and add helper functions for working with embedded layout

What should we add?
As discussed in #8800 the way we're embedding the Layout from the transpiler into the output quantum circuit is a bit of a weird place right now. We have a private attribute _layout which is used widely across the qiskit code but we provide no public or supported interface for working with this. This came to be an issue for fixing #8800 because to fix the issue we changed what was stored in the _layout attribute, and while it is a private layout external users depend on it because it was the only way to reason about the permutation caused by initial layout in the transpiler. To address this gap in 0.23.0 we should introduce a public QuantumCircuit.layout attribute and/or also have methods to work with the embedded layout. Like something to return the qargs for the circuit.

I think QuantumCircuit.metadata is a more natural place to include this information (since a Layout itself isn't strictly a property of a single circuit, it's a mapping of (virtual) qubits in one circuit to (physical) qubits in another).
Another piece it would be good to discuss before we finalize this direction is whether or not to include pre/post Permutations explicitly as a result of the layout and routing passes (with an optimization pass to move them after of qubit initialization and before qubit measurement). This had been discussed in the past and had some advantages in e.g. transpiling subcircuits, but I don't think there was ever a concrete plan of implementation.

I'm not convinced that metadata is right, and I think some of that is that I don't entirely agree with the idea that the "physical" qubits are in a different circuit - they're specific to a machine, but then so are the operations that are contained within the circuit.  I do buy the idea that the mapping isn't really an property of the circuit in a vacuum, but a transpiled circuit already doesn't exist in one.
I'm nervous about us suddenly applying important meaning to metadata, because the getter documentation for it says:

The metadata for the circuit is a user provided dict of metadata for the circuit. It will not be used to influence the execution or operation of the circuit, but it is expected to be passed between all transforms of the circuit (ie transpilation) and that providers will associate any circuit metadata with the results it returns from execution of that circuit.

Putting the mapping to hardware qubits in that dictionary seems to me to be a violation of the idea that metadata doesn't influence execution or operation of the circuit.  We'd also be newly adding a restriction on the keys available for users to assign with arbitrary data, and I really don't like how using dict directly masks the typing information that IDEs / linters would be able to use if it's a regular attribute.
(Technically there's typing syntax to mitigate some of that last point, if you're prepared to reserve a key in metadata that users can never use, but I personally think that's a hack around bad coding practices in Python.)

All that's to say that my vote is to use a public .layout attribute.

I would like to add a comment here to explain an observation that I made a few weeks ago. Specifically, I was working with the primitives and wanted to skip transpilation on the server side. This meant, that I had to transpile my circuit locally and set the corresponding option in my primitive job.
However, this immediately led to the problem that my (now physical) circuit contained a different number of qubits than my operator resulting in errors (obviously). I could not find a builtin way of how to deal with this situation (that is to say, I did not find a transformation method which would map my qubit operator to the correct set of physical qubits) so I ended up using the private _layout attribute of my QuantumCircuit to build a simple expansion routine for my operator.
There must be some similar code running somewhere on the server side but I did not find that.
All that is to say:

I would also be in favor of some public layout information
I would also argue a builtin way for what I had to do above should be provided based on the public layout information

Max: can you make the operator form before the transpilation?  Seems like that'd save you both simulation cost, and solve your problem.

@mrossinek fwiw, this kind of thing is the intent of the Operator.from_circuit constructor method: https://qiskit.org/documentation/stubs/qiskit.quantum_info.Operator.from_circuit.html#qiskit.quantum_info.Operator.from_circuit which already factors in the initial layout and final, layout if it's present. Although, it currently does not factor in ancilla expansion reversal that you described, which is something is something @alexanderivrii brought up recently as well. We should open another issue to track that.
But, yeah I think we all agree that the point here is that the _layout attribute has been semi-public for a long time because it's the only way users have to reason about the permutation caused by compilation. So we should promote it to be an actual public documented interface

I'm also in favor of using a .layout attribute for the reasons @jakelishman outlined above along with two other reasons, for QPY metadata  gets serialized by default by passing it to the default stdlib json serializer and deserializer. The interface we've defined for that is if users have anything outside what stdlib json can handle they should provide their own JSONEncoder and JSONDecoder subclasses to the qpy dump and load functions to be able to handle custom types. If we put the layout information in metadata we'd have to update the default serializers in qpy to be able to handle Layout  and Qubit objects, but that would result in breaking any users that were using our interface as described, because if they expanded the metadata with their own custom types pre-transpilation then what previously worked for serializing a post-transpiled circuit would no longer work because their custom encoder and decoder would not handle the layout anymore unless they subclassed our custom encoder.
The other reason is having a separate attribute will let us clearly document the attribute. We can write a detailed description of the what the layout (as at this point it's really initial layout, input qubit mapping, and final layout) is and examples on how to use it. Having a place we can point people to to explain all of this I think will be quite valuable because explaining layout and routing as permutations can sometimes be confusing.

See my comment from May of last year that touches on the same issue:
https://github.com/Qiskit/qiskit-ibm-runtime/issues/338


Add pakrym/jab as a module and repo project

Jab today is the only solution to use AOT source generated dependency
injection which avoids Reflection and better supports NativeAOT. As the
package is a build only dependency, it won't be listed as a package
dependency anywhere but is a required build dependency.
Eventually in the future, when dotnet/runtime offers a solution for
source generated dependency injection, this repository can be removed.
Jab will be used in the Compatibility tooling in dotnet/sdk.

/azp run

For some reason CI is not getting triggered with this PR.  While I figure that out, I manually queued a build - https://dev.azure.com/dnceng/public/_build/results?buildId=1817613&view=results.  The following prebuilts have been added which will need to be addressed:
Microsoft.Build.Tasks.Git.1.1.1
Microsoft.CodeAnalysis.Analyzers.3.3.3
Microsoft.CodeAnalysis.Common.3.11.0
Microsoft.CodeAnalysis.Common.4.0.1
Microsoft.CodeAnalysis.CSharp.3.11.0
Microsoft.CodeAnalysis.CSharp.4.0.1
Microsoft.SourceLink.Common.1.1.1
Microsoft.SourceLink.GitHub.1.1.1
System.Text.Encoding.CodePages.4.5.1
There are a couple options to remove these:

Upgrade the versions referenced by the repo to a version already included in source-build.  Ideally this would dynamically pick up the source-build version.
Compile against reference assemblies which means adding these assemblies to https://github.com/dotnet/source-build-reference-packages.  Given it looks like the generator is getting packaged I'm not sure this is a viable option?

Upgrade the versions referenced by the repo to a version already included in source-build. Ideally this would dynamically pick up the source-build version.

We could easily remove the Microsoft.SourceLink package references via a patch but the Microsoft.CodeAnalysis references are strong dependencies. Does this repository know which versions of these packages are available in source build? If yes, then we could upgrade them.

Does this repository know which versions of these packages are available in source build? If yes, then we could upgrade them.

Yes it should using the standard package version property syntax.  You can see a couple patches already lift the source-link versions:
https://github.com/dotnet/source-build-externals/blob/main/patches/application-insights/0001-Eliminate-prebuilts.patch
https://github.com/dotnet/source-build-externals/blob/main/patches/MSBuildLocator/0001-Eliminate-prebuilts.patch

@MichaelSimons the MicrosoftSourceLinkVersion property is available because arcade provides a default for it in its DefaultVersions.Generated.props file: <MicrosoftSourceLinkVersion>1.2.0-beta-22179-02</MicrosoftSourceLinkVersion>.
The MicrosoftCodeAnalysisAnalyzersVersion and MicrosoftCodeAnalysisCSharpVersion properties that are required to override what's inside Jab.csproj aren't defined though by Arcade.
Should we add a subscription from this repository to roslyn and add Versions.props and Version.Details.xml entries so that these versions are available? Is there any downside with having such a subscription?

@ViktorHofer - you are right, I was thinking in context of the tarball build and not the repo build.  Yes we should add a subscription to Roslyn and add Versions.props and Version.Details.xml entries.
I can't think of any downsides of having the subscription, I can only think of downside of not having the subscription.  The downside would be the repo build and tarball source-build would be building with different versions.  Source-build by default overrides the Versions.props to pickup the latest.

Sorry for the delayed response. I finally need this so I'm looking into this again.

I can't think of any downsides of having the subscription, I can only think of downside of not having the subscription. The downside would be the repo build and tarball source-build would be building with different versions. Source-build by default overrides the Versions.props to pickup the latest.

@MichaelSimons I'm trying to add the subscription now. Do you know which channel we would need to use for it. I don't really understand how Roslyn versions.

@MichaelSimons @mmitche can you please answer the above?

I just added the subscriptions by mimicking existing subscriptions to roslyn and roslyn-analyzers.

@MichaelSimons PTAL


Fix full pipeline: filter out non-existing modalities

@rcannood This PR fixes an issue with processing h5mu files with this popeline, but the file does not contain all the modalities that the pipeline supports. Without filtering out these events from the channel, the popeline will raise the assertion error. I have added a vdj modality to the pipeline to test this filtering (the test data is only atac and RNA). I have not added the filtering for the rna modality because is made the assumption that it always needs to br present? Please edit this PR as you see fit (like removing the vdj or alsoalsof adding the filter to the rna modalitt.

Thanks! I made a few changes :)

I let the split modality component return a csv with the modality names.
The full pipeline doesn't list the split modality directory's contents but instead uses the csv to retrieve the output files and filter.
The obs names and .obs[sample_id] is modified / added using the input ids.


add a kubecon landing page

Hello @jazzdan, @yuindustries,
Please review the following commits I made in branch nicks/landing:
4e10cc77f703b060570bd36697e8a0dc7cc39e65 (2019-05-15 13:23:20 -0400)
add a kubecon landing page


Should default constructor arguments be moved to config?

First, I think this library is really amazing. You have great internal interfaces which allow one to nicely implement alternative implementations. E.g., I am using this library inside a Chrome extension so I implemented a storage class to use extension storage and specialized extension popup.
But I also want to run this library in a service worker of the extension. And things became complicated there: service workers do not have XMLHttpRequest, but only fetch. So I had to fork and change the JsonService code (again, great interface!) to use fetch (see https://github.com/IdentityModel/oidc-client-js/pull/1335 for PR because I think that should be default). The reason for fork is that it is not really easy to use a custom JsonService class. Currently, JsonService is passed as a default value for a parameter to a constructor in multiple places, but it is hard to change that default value or pass some other value to those parameters.
I would suggest that all those constructor parameters with default values are also configured through config parameter. For example, in MetadataService, the initialization could then be instead of:
    constructor(settings, JsonServiceCtor = JsonService) {
        ...
        this._jsonService = new JsonServiceCtor(['application/jwk-set+json']);
    }

This:
    constructor(settings, JsonServiceCtor = JsonService) {
        ...
        this._jsonService = new (settings.JsonServiceCtor || JsonServiceCtor)(['application/jwk-set+json']);
    }

Another approach could be to have Global.JsonServiceCtor. But that is more just a question where this config value is set.

I would suggest that all those constructor parameters with default values are also configured through config parameter.

Yep, this is a realization I came to a while back as well. I did build the first version of this thing like 7 years ago now (give or take?). Of course this would be a breaking change, so it's something on my TODO list if/when I ever get the cycles for a v2.

I do not think it is a breaking change if done like I described above (check if value is set in config, if not, use the value from the constructor).


infer error about visualizer

mmdetection/mmdet/visualization/local_visualizer.py", line 153, in _draw_instances
label_text = classes[
IndexError: tuple index out of range
https://github.com/open-mmlab/mmdetection/blob/44ebd17b145c2372c4b700bfb9cb20dbd28ab64a/mmdet/visualization/local_visualizer.py#L153-L154

I have tried the solution, but it still failed.


Fix "Select all" buttons

Closes https://github.com/rilldata/rill-private-issues/issues/981

@ericpgreen2 : Please resolve conflicts and merge.


Do not send onSlidingCompleted when value is changed programmatically

This pull request fixes #487
It prevents from sending the onSlidingCompleted event when value is not changed by manual dragging.
This change was implemented according to other platforms so that the behavior is constant across platforms: The controlledValue feature will not trigger any event. Events are reserved for manual dragging or tapping.
https://user-images.githubusercontent.com/70535775/214949193-52edc4fc-2275-4150-8d6b-e38bbde2095c.mov
This change has been also reflected in the documentation.

Isn't it a breaking change?

Isn't it a breaking change?

Hello @laurent22! No, not really. Same as on iOS and other platforms - the programmatic change of value prop should not trigger the onSlidingCompleted. Changing value programmatically is not a sliding so this event would be misleading. Moreover, if developer changes the value programmatically then it should be intentional hence well known that it happened, so no need to notify about it by sending any sliding or dragging-related event.
I hope it does answer your concerns.


DXCDT-576: Prevent themeID from getting sent to the web app

🔧 Changes
Prevent theme ID from getting sent to the web app and rework save logic.
📚 References

🔬 Testing

📝 Checklist

[x] All new/changed/fixed functionality is covered by tests (or N/A)
[x] I have added documentation for all new/changed functionality (or N/A)

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

:exclamation: No coverage uploaded for pull request base (feature/auth0-universal-login-customize@b958c52). Click here to learn what that means.

Additional details and impacted files
@@                            Coverage Diff                             @@
##             feature/auth0-universal-login-customize     #885   +/-   ##
==========================================================================
  Coverage                                           ?   72.42%
==========================================================================
  Files                                              ?       93
  Lines                                              ?    12298
  Branches                                           ?        0
==========================================================================
  Hits                                               ?     8907
  Misses                                             ?     2874
  Partials                                           ?      517

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Bug with paginator

Pages of types other than post is also being iterated over in .Paginator, because it is not properly filtered like the index / taxonomy list.
This result in some page having fewer posts than others (because they contain items of other types, which would be exclude by page listing itself). If the last page contains no post item, it would be an empty page.

There is a similar issue in the downstream fork here.


Update ha_deck1.yaml

error message on compile in ESPHome, adjusted code (as suggest by esphome interface) words.

Thank you for your contribution.
Please remove properties: version, source, platform_version, and variant as they are not required (the example should demonstrate integrations using the latest ESPHome version only).

I've updated the example YAML.


Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client

Hello guys!
When creating a user and then having him to login I got this error:

[dev.start] POST /api/users/login 200 207.582 ms - 198
[dev.start]   APIError: Not Found!
[dev.start]
[dev.start]   - error.js?:25 new ExtendableError
[dev.start]     webpack:///./src/services/error.js?:25:11
[dev.start]
[dev.start]   - error.js?:43 new APIError
[dev.start]     webpack:///./src/services/error.js?:43:5
[dev.start]
[dev.start]   - index.js?:52 routes.all
[dev.start]     webpack:///./src/routes/index.js?:52:42
[dev.start]
[dev.start]
[dev.start] Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
[dev.start]     at validateHeader (_http_outgoing.js:503:11)

Any ideas what's wrong?

@javaallah In Authentication controller in login function delete return next() or set return true


fixef: Implement Iterative Least Squares Solver

Context
Currently, pyfixest can compute fixed effect estimates in the fixef() method by using a sparse solver.
A potentially faster alternative is to implement an iterative solver. See section 3 in the lfe paper.

Hi Ben (@b-knight) -  I have been playing around with scipy's solver for linear systems and it's much faster than the sparse solver I have implemented by hand:
import numpy as np
from scipy.sparse import rand
from scipy.sparse.linalg import lsqr, spsolve
import time

m, n = 2_000_000, 500
A = rand(m, n, density=0.1, format='csr')
x_true = np.random.rand(n)
b = A @ x_true

tic = time.time()
result = lsqr(A, b)
toc = time.time()
print(f"Elapsed time: {toc - tic:.2f} seconds")
x_lsqr = result[0]
x_lsqr[0:5]
#Elapsed time: 1.35 seconds
#array([0.5335062 , 0.99837454, 0.69382102, 0.5373236 , 0.45653057])

tic = time.time()
alpha = spsolve(A.transpose() @ A, A.transpose() @ b)
toc = time.time()
print(f"Elapsed time: {toc - tic:.2f} seconds")
alpha[0:5]
# Elapsed time: 30.38 seconds
# array([0.53350756, 0.99837705, 0.69382127, 0.53732394, 0.45653019])

Hence to speed up the fixef and the predict method, we'd have to change this line https://github.com/py-econometrics/pyfixest/blob/405ba73787af6bac793afb5bdbb34d2e7ad6cd53/pyfixest/estimation/feols_.py#L1329 in .fixef() to using the lsqr solver.
Would you like to implement the change? Else I'm happy to do it myself =)
Best, Alex

@greenguy33 can I assign this to you? I think this should be a minimal change =)

@s3alfisc sure, I'll take a look asap (ideally later this week or early next)


Provide version flag for uninstall command in standalone mode

Provide --runtime-version flag for dapr uninstall command in standalone mode

Just trying to understand the use case. We can only have one daprd binary in a path so what is the use of this flag in standalone mode?

@pruthvidhodda , you are right. I opened this issue to track for standalone while we were making fix for Kubernetes mode.
But looks like this is not needed. Closing the issue.


Compile python requirements

Automated changes by create-pull-request GitHub action

Added dbt-core explicitly to requirements as recommended in 1.8.0 upgrade guide
Also @damonmcc - unit tests 👀

Sqlfluff currently isn't compatible with dbt 1.8.0, given that we don't have a burning need for it yet, i think best to just wait. I'll monitor sqlfluff releases this week


Update packages

Updated javascript packages
Moved node version to lts

@OlhaPeculiar Could you check it please here https://fortify-tools.vercel.app/
Thanks.

At the moment everything is working correctly on https://fortify-tools.vercel.app/

@OlhaPeculiar Could you check please here https://fortify-tools.vercel.app/

At the moment everything is working correctly on https://fortify-tools.vercel.app/


[local_auth] Found a number of compile-time errors due to deprecations.

Steps to Reproduce

Using "flutter create whatever-name", create the demo app.
Paste the code under https://pub.dev/packages/local_auth/example to lib/main.dart.
Add the local_auth dependency to pubspec.yaml. I did local_auth: ^1.1.5
Add the "NSFaceIDUsageDescription" key to Info.plist.
Connect an iphone to your mac. I used a usb cable.
Open the file Runner.xcworkspace using Xcode.
Change the device to your actual iphone, not a simulator.
Change the deployment target to iOS 12.
Run the app and see the compile-time errors.

Expected results:
I expected the app to install to my iphone.
Actual results:
Compile-time errors.
I fixed the issues and the app run perfectly with biometry and passcode.
All fixes refer to file local_auth/ios/Classes/FLTLocalAuthPlugin.m. Latest commit 08f8f21 on Apr 5, 2021.
See below how:

Logs
Line 85:
Replaced
[[UIApplication sharedApplication] openURL:url];

by
[[UIApplication sharedApplication] openURL:url options:@{} completionHandler:nil];

Line 113:
Replaced LAErrorTouchIDNotEnrolled by LAErrorBiometryNotEnrolled
Line 181:
Replaced LAErrorTouchIDNotAvailable by LAErrorBiometryNotAvailable
Line 182:
Replaced LAErrorTouchIDNotEnrolled by LAErrorBiometryNotEnrolled
Line 183:
Replaced LAErrorTouchIDLockout by LAErrorBiometryLockout
Line 203:
Replaced LAErrorTouchIDNotEnrolled by LAErrorBiometryNotEnrolled
Line 213:
Replaced LAErrorTouchIDLockout by LAErrorBiometryLockout

8. Change the deployment target to iOS 12.

The plugin supports back to iOS 8. If you change the minimum deployment target, you will indeed get deprecation warnings since that's how iOS deprecation warnings are gated, but your proposed changes would break everyone using the plugin for older OS versions.
At such time as the minimum supported OS version of the plugin is changed, API usage will be updated accordingly, but it cannot currently be done.


Managing permissions and configuration values?

Thanks for indulging a question that I wasn't able to answer from the documentation. I'm willing to PR an update to the docs if I figure it out.
Should we use execute() to grant permissions? That would be something like
$this->execute('GRANT SELECT ON db.new_table TO \'user\'@\'localhost\'');

Now if part of that is different in different environments, is there a way to use a configuration value in the migration? Say if localhost or some other value were different in each environment.
Thanks for any pointers or ideas!

I think the Environment is now being (or will be) presented to the migration, so you should be able to determine which (if you have many) database is being migrated.

Ah, I needed to search "environment" issues. Looks like PR #794 will address this question and implement #984.


Add a WebSocket plugin

To use websockets in jetty using JSR-356 annotations, you need to configure the context and add the relevant jar file.

Why this plugin is not into master yet?

Merged into master.

1.60 should be available on Maven Central now.


how can i change the feature importance type using a pipeline object?

i am trying to compare xgboost and light gradient boosting feature importance but both have different assessment criteria. so i am trying to change the feature importance type and see how i can do this.
e.g.
pipeline = make_pipeline(XGBClassifier())
param_grid =    {
                  'xgbclassifier__learning_rate': [0.01,0.005,0.001],

                  }

gini_scorer = make_scorer(normalized_gini, greater_is_better = True)

# Initialize Grid Search Modelg
model = GridSearchCV(pipeline,param_grid = param_grid,scoring = gini_scorer,
                                 verbose= 1,iid= True,
                                     refit = True,cv  = 3)

model.fit(x, y)
print("Best score: %0.3f" % model.best_score_)
print("Best parameters set:")
model = model.best_estimator_

then to get feature importances i:

feature_importances = model.steps[-1][1].feature_importances_
pd.DataFrame(feature_importances, index=feature_names,
                     columns=['Importance']).sort_values('Importance') \
            .plot(kind='barh', figsize=(15, 25))

however i am not sure how i can edit this code of mine to allow a change in feature importance e.g. what if i want it by gain etc..

Maybe importance_type parameter can help?

where do i put this?

The constructor of XGBClassifier.

for some reason, when i try plotting the feature importance two ways where:
pipeline = make_pipeline(XGBClassifier(importance_type='weight'))
1)```
features_xgboost = model.best_estimator_.named_steps["xgbclassifier"].get_booster().get_score()
keys = list(features_xgboost.keys())
values = list(features_xgboost.values())
data = pd.DataFrame(data=values, index=keys, columns=["score"]).sort_values(by = "score", ascending=True)
data.plot(kind='barh')

2 ) ```
features = x_train.columns
feature_names = features.tolist()
pd.DataFrame(feature_importances, index=feature_names,
                     columns=['Importance']).sort_values('Importance') \
            .plot(kind='barh', figsize=(15, 25))

i get different scales, how can this be? so for 1) i get the scale between 0-300 with different orders of features as well and in 2) it's between 0-0.04

XGBClassifier.feature_importances_ normalizes the feature importances. If you divide the importances in (1) by their sum you should get the same as in (2).


check_syntax: Cannot read property 'DD03P_TABLE' of undefined

abaplint 2.57.3
Issue:
./src/isu/#redwood#1isu.fugr.lfkkaktiv2dat.abap[144, 1] - Cannot read property 'DD03P_TABLE' of undefined (check_syntax) [E]
Source:

Note: Issue occurs with AbapLint and SCI.

Table Details:

also try running ZABAPLINT_DEBUG and check the contents of the XML files

try with 2.57.4, it might give a different error

After checking the DFKKOP table and it's includes, i found these includes:

These includes exist in the dictionary but are switched off

The dependency generate does not generate any fields for these includes, ergo causing the error.
I've reran abaplint with the new version and we now have a more clear message:
./src/isu/#redwood#1isu.fugr.lfkkaktiv2dat.abap[144, 1] - Structure does not contain any components (check_syntax) [E]
Could the include/table causing this be added to the message?

try again, 2.57.5

Message is improved. Thx.
./src/isu/#redwood#1isu.fugr.lfkkaktiv2dat.abap[144, 1] - Table/Structure SI_FKKOPR does not contain any components (check_syntax) [E]
List Dependencies for FKKOP:

Should this dependency be removed if the switch is not active?

hmm, that might be difficult and/or slow to determine

When running dependency with log/download...

The include structures are generated without fields. The alternative would be to generated the fields (they exist in DD03L).
It would anyway be good if these fields are generated as they could be referenced in switched-code.
By using state=M (call to DDIF_TABL_GET) the fields are retrieved.
Would that cause other issues?

having too many fields would be okay

Pull Request

The change works well for the Linter, but SCI still shows the issue.

The SCI error comes from the cache. After disabling the cache, the error is gone.
is there a way to force the cache to be rebuild?

Created issue SCI #212 for the clear cache.


Query refactor to use generic functions [k8s]

Description
Currently queries have a lot of code that could be re-used and refactored into auxiliary functions to be used in all relevant queries. The goal is to make use of the generic functions and refactor all reavant queries for k8s

Refactor has been achieved in various PRs made


(proper) Adding cli functionality to check strings in an adhoc manner

Exactly the same as https://github.com/Yelp/detect-secrets/pull/52, except will merge to master this time.

"into clearer-usage-commands from adhoc-line-scanning" sneaky sneaky lol :D


Link to correct extended integration test driver file in integration_test README

Updated link in integration_test readme to point to the new integration test driver.
List which issues are fixed by this PR. You must list at least one issue.
No issues that I could find
Pre-launch Checklist

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I read the Tree Hygiene wiki page, which explains my responsibilities.
[x] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I signed the CLA.
[x] I listed at least one issue that this PR fixes in the description above.
[x] I updated/added relevant documentation (doc comments with ///).
[x] I added new tests to check the change I am making or feature I am adding, or Hixie said the PR is test-exempt.
[x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel on Discord.

test-exempt: only affects .md files


Let configuration cache support java-gradle-plugin projects with dependencies

Fixes #18064

@bot-gradle test QFL

OK, I've already triggered QuickFeedbackLinux build for you.

@bot-gradle test and merge

Your PR is queued. See the queue page for details.

OK, I've already triggered a build for you.

Pre-tested commit build failed.

@bot-gradle test and merge

OK, I've already triggered a build for you.


update webhook (change admissionregistration to v1)

Signed-off-by: HaoJie Liu<EMAIL_ADDRESS>

admissionregistration.k8s.io/v1beta1 ValidatingWebhookConfiguration is deprecated in v1.16+, unavailable in v1.22+


update engineers teach physics

https://trello.com/c/UF2QP467
update engineers teach physics page with new components

If we have now decided to capitalise Engineers teach physics, we need to apply that change to the physics page as well
/life-as-a-teacher/explore-subjects/physics


Support for bitwise operators

Is it possible to add support for bitwise operators in WHERE clause.?
for example if I had the messages table with the field status  of type INT and the bits of the value of this field represent a specific status i.e delivered, seen, ..etc. Instead of having a separate BOOL field for each status.
supposing that the first bit is for the delivered status and the second one is for the seen status
If i want to retrieve only delivered messages that are not seen yet, the query would be something like:
SELECT `status` FROM `messages` WHERE (`status` & 1 = 1)

I have created pull request with this feature: https://github.com/kayak/pypika/pull/288

Fixed by #288

Anyone tried this one with the latest version? (ver 0.39.0 as of now)
I tried to use bitwiseand for where condition but it returns error because of this line https://github.com/kayak/pypika/blob/master/pypika/terms.py#L807

AttributeError: 'int' object has no attribute 'nodes_'

I wonder if it's a bug or I use the function incorrectly

@ekamuktia please can you provide us some example code, and if it fails, raise a new issue? Thanks!

Here's the sample code
table = Table('table_name')
Query.select(table.star).from_(table).where(table.category.bitwiseand(2))

This code is supposed to generate

SELECT * FROM table_name WHERE category & 2

But it returns error code as I mentioned before


Error when trying the hot bubble with GPU

I'm trying to run the hot bubble example but using a GPU. However I get an error when defining the model that I cannot understand (the caveat here being that I have zero experience working with GPUs). The minimum code to reproduce this is
using Oceananigans
model = BasicModel(N=(256, 1, 256), L=(2000, 1, 2000), architecture=GPU(), ν=4e-2, κ=4e-2)

This is the error that I get:
ERROR: LoadError: Device capability v2.1.0 not supported by available toolchain
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] supported_capability(::CUDAdrv.CuDevice) at /home/tomaschor/.julia/packages/CUDAnative/Lr0yj/src/utils.jl:7
 [3] macro expansion at /home/tomaschor/.julia/packages/CUDAnative/Lr0yj/src/execution.jl:388 [inlined]
 [4] #cufunction#176(::Nothing, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(CUDAnative.cufunction), ::getfield(GPUArrays, Symbol("##23#24")), ::Type{Tuple{CuArrays.CuKernelState,CUDAnative.CuDeviceArray{Float64,3,CUDAnative.AS.Global},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}},typeof(identity),Tuple{Int64}}}}) at /home/tomaschor/.julia/packages/CUDAnative/Lr0yj/src/execution.jl:357
 [5] cufunction(::Function, ::Type) at /home/tomaschor/.julia/packages/CUDAnative/Lr0yj/src/execution.jl:357
 [6] macro expansion at /home/tomaschor/.julia/packages/CUDAnative/Lr0yj/src/execution.jl:174 [inlined]
 [7] macro expansion at ./gcutils.jl:87 [inlined]
 [8] macro expansion at /home/tomaschor/.julia/packages/CUDAnative/Lr0yj/src/execution.jl:171 [inlined]
 [9] _gpu_call(::CuArrays.CuArrayBackend, ::Function, ::CuArrays.CuArray{Float64,3}, ::Tuple{CuArrays.CuArray{Float64,3},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}},typeof(identity),Tuple{Int64}}}, ::Tuple{Tuple{Int64},Tuple{Int64}}) at /home/tomaschor/.julia/packages/CuArrays/wXQp8/src/gpuarray_interface.jl:60
 [10] gpu_call(::Function, ::CuArrays.CuArray{Float64,3}, ::Tuple{CuArrays.CuArray{Float64,3},Base.Broadcast.Broadcasted{Nothing,Tuple{Base.OneTo{Int64},Base.OneTo{Int64},Base.OneTo{Int64}},typeof(identity),Tuple{Int64}}}, ::Int64) at /home/tomaschor/.julia/packages/GPUArrays/fLiQ1/src/abstract_gpu_interface.jl:151
 [11] gpu_call at /home/tomaschor/.julia/packages/GPUArrays/fLiQ1/src/abstract_gpu_interface.jl:128 [inlined]
 [12] copyto! at /home/tomaschor/.julia/packages/GPUArrays/fLiQ1/src/broadcast.jl:48 [inlined]
 [13] copyto! at /home/tomaschor/.julia/packages/GPUArrays/fLiQ1/src/broadcast.jl:60 [inlined]
 [14] materialize! at ./broadcast.jl:801 [inlined]
 [15] zeros at /home/tomaschor/.julia/packages/Oceananigans/s0kF6/src/utils.jl:92 [inlined]
 [16] zeros at /home/tomaschor/.julia/packages/Oceananigans/s0kF6/src/utils.jl:100 [inlined]
 [17] Field(::Tuple{DataType,DataType,DataType}, ::GPU, ::RegularCartesianGrid{Float64,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}) at /home/tomaschor/.julia/packages/Oceananigans/s0kF6/src/fields.jl:32
 [18] Type at /home/tomaschor/.julia/packages/Oceananigans/s0kF6/src/fields.jl:50 [inlined]
 [19] Type at /home/tomaschor/.julia/packages/Oceananigans/s0kF6/src/fields.jl:109 [inlined]
 [20] VelocityFields(::GPU, ::RegularCartesianGrid{Float64,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}}) at /home/tomaschor/.julia/packages/Oceananigans/s0kF6/src/models.jl:186
 [21] (::getfield(Core, Symbol("#kw#Type")))(::NamedTuple{(:float_type, :grid, :closure, :architecture),Tuple{DataType,RegularCartesianGrid{Float64,StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}},ConstantIsotropicDiffusivity{Float64},GPU}}, ::Type{Model}) at ./none:0
 [22] #BasicModel#23(::Tuple{Int64,Int64,Int64}, ::Tuple{Int64,Int64,Int64}, ::Float64, ::Float64, ::Type, ::Base.Iterators.Pairs{Symbol,GPU,Tuple{Symbol},NamedTuple{(:architecture,),Tuple{GPU}}}, ::typeof(BasicModel)) at /home/tomaschor/.julia/packages/Oceananigans/s0kF6/src/models.jl:125
 [23] (::getfield(Oceananigans, Symbol("#kw##BasicModel")))(::NamedTuple{(:N, :L, :architecture, :ν, :κ),Tuple{Tuple{Int64,Int64,Int64},Tuple{Int64,Int64,Int64},GPU,Float64,Float64}}, ::typeof(BasicModel)) at ./none:0
 [24] top-level scope at /home/tomaschor/oceantest/test1.jl:5
 [25] include at ./boot.jl:328 [inlined]
 [26] include_relative(::Module, ::String) at ./loading.jl:1094
 [27] include(::Module, ::String) at ./Base.jl:31
 [28] include(::String) at ./client.jl:431
 [29] top-level scope at REPL[1]:1
in expression starting at /home/tomaschor/oceantest/test1.jl:5

Here's the output of nvidia-smi:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.129                Driver Version: 390.129                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  NVS 310             Off  | 00000000:03:00.0 N/A |                  N/A |
| 30%   40C    P8    N/A /  N/A |    634MiB /   962MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0                    Not Supported                                       |
+-----------------------------------------------------------------------------+

I'm using the most recent version of both Julia and Oceananigans, and I have installed the CUDA toolkit with sudo apt-get install nvidia-cuda-toolkit. I'm running this on Linux Mint.
The only relevant info I could find was this issue, but my limited knowledge of GPUs makes it hard for me to take much useful from it.
Any thoughts?

Well!
It looks like the "compute capability" of an NVIDIA device determines the kind of computations it can do:
https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#compute-capabilities
Your device, which is an "NVS 310", has a compute capability of 2.1:
https://developer.nvidia.com/cuda-gpus
Apparently this compute capability is not supported by the julia CUDA tools. So I'm not sure you can resolve this issue, other than finding a different GPU. The issue you pasted briefly explains why this might be the case; it has something to do with the compiler used by CUDAdrv?
We typically use the Tesla V100 or P100, which apparently have compute capabilities of 7.0 or 6.0.
I'm not 100% sure the NVS 310 would accelerate computations relative to your CPU anyways. It only has 48 cores (compared to 5,120 for a Tesla V100), and 512 MB of memory?
https://www.nvidia.com/en-us/design-visualization/nvs-graphics-cards/
Do you have access to a cluster with GPUs?

If you just want to play around with a fast GPU @suyashbire1 said you get $300 of free credits when you sign up on Google Cloud. I think he was able to use the credits to spin up a Tesla V100 for a while (they're ~$2/hour).

Hi Tomas,
Google gives you a $300 credit when you first sign up for Google Compute but it can't be used for GPUs anymore. See https://cloud.google.com/free/docs/gcp-free-tier.
Other options are requesting allocations on Bridges (it has V100s) or Comet (it has P100s) through XSEDE (https://www.xsede.org/ecosystem/resources) but that might be an overkill if you just want to play around with a GPU.

Thanks guys! It sure would be nice to play around on COMET, but maybe I should wait until I have something a little more serious before I use their resources :)
In the meantime I think I'll change my set-up and try to run some 2D turbulence simulations on my desktop's GPU and see far I can get. Thanks!

@tomchor are there any more questions? I think we resolved this issue: the GPU you were attempting is not compatible with the julia CUDA tools (so this is not an issue with Oceananigans. If that's true please close the issue.

This is solved! I just forgot to close it. Thanks!