Analyze prebuilts for websdk

The following prebuilt packages are included as dependencies in source-build by websdk.  These need to be analyzed to determine how to avoid requiring packages that source-build doesn’t build.

Change to use the source-built version
Package is not critical to product and can be removed.  (i.e.test-specific, windows-specific, etc)
The package is used as reference only
The package should be built in source-build
Other – please elaborate on what you think should be done.

For more detailed usage information for these, please see the Usage Exploration tab of the SourceBuildPrebuiltReport.

PackageName
PackageVersion
SourceBuiltVersion
Category

Microsoft.Build.Framework
15.1.1012
16.0.0-preview.385

Microsoft.Build.Utilities.Core
15.1.1012
16.0.0-preview.385

Microsoft.DiaSymReader.Pdb2Pdb
1.1.0-beta1-62506-02

Microsoft.DotNet.SignTool
1.0.0-beta.19067.6

Microsoft.Web.Xdt
3.0.0

System.Collections.Specialized
4.3.0

System.Security.Cryptography.ProtectedData
4.3.0

System.Xml.XmlDocument
4.3.0

sn
1.0.0

vswhere
2.5.2

@dseefeld - The above dependencies list was generated before the arcade. Can we re-run this to see what are the pending source build dependencies?

@dseefeld - The xdt dependency to websdk should be fixed now
https://github.com/aspnet/websdk/blob/b603b295fd0d97bed95faea40ce564147baa68f7/eng/Version.Details.xml#L12

Package
Version
Status

Microsoft.Build.Framework
15.1.1012
Reference Only

Microsoft.Build.Utilities.Core
15.1.1012
Reference Only

Microsoft.Web.Xdt
3.0.0
https://github.com/aspnet/websdk/issues/664

Microsoft.AspNetCore.Mvc.Analyzers
3.0.0-preview4-19216-03
https://github.com/aspnet/websdk/issues/665

Microsoft.AspNetCore.Mvc.Api.Analyzers
3.0.0-preview4-19216-03
https://github.com/aspnet/websdk/issues/666

System.Security.Cryptography.ProtectedData
4.3.0
Reference Only

System.Xml.XmlDocument
4.3.0
Reference Only

All prebuilts removed.


Basic LineOfSight Observer class

We already have a base class for cameras. We need an appropriate base class for line of sight observers such as optical fibres.

Added basic LineOfSight class in commit 9412c5014bb546f5e32bc859a5500d46932ee3ae. More refinement and features will be tracked through more specific tickets.


[SPARK-39225][CORE] Support spark.history.fs.update.batchSize

What changes were proposed in this pull request?

Add a spark property, spark.history.fs.update.batchSize(default 1000), which specifies the batch size for updating new eventlog files.
For each eventlog scan/parse process, only a max of spark.history.fs.update.batchSize jobs are scanned/parsed. This allows the scan process ends within a reasonable time, so new eventlog files(appeared after the scan process starts) can be scanned sooner.

Why are the changes needed?
Current Spark History Server suffers when there are a large number of eventlog files under eventLog.dir: when a SHS starts, the initial scan may take a long time, and new eventlog files would not be scanned/parsed until the initial scan completes.
For example, if the initial scan takes 1-2 days(this is not uncommon in large environments), the newly finished spark jobs would not show up in SHS since their eventlog files are not scanned/parsed until the initial scan process finishes. This would result in a 1-2 days SHS malfunctioning since the newly finished spark jobs are most likely to be queried by users.
This PR adds a limit for each scan process to spark.history.fs.update.batchSize, so that each scan can finish within a reasonable time(default 1000 jobs can usually be scanned in ~10-15 mins but this can vary depending on the sizes of the eventlog files). This will prevent the long initial scan process from blocking the newly appeared eventlog files be scanned.

Does this PR introduce any user-facing change?
No.

How was this patch tested?
Tested locally:

Tested with unit tests in the core module and all passed.
Placed 10 eventlog files under eventLog.dir, and set spark.history.fs.update.batchSize to 2. When the SHS started, observed that the scan process parsed 2 files at each time(starting from the files with the latest modification time).
Added a new eventlog file after the first scan started, and observed that the 2nd scan process picked up and parsed the new eventlog file before parsing other remaining (older) eventlog files.

Thank you for updates, @hai-tao-1 . Yes, the only remaining comment is the test case.

We need a test case for the configuration. Please check the corner cases especially.

BTW, I revised the PR and JIRA title more precisely based on your PR content.

Can one of the admins verify this patch?

Thank you for updates, @hai-tao-1 . Yes, the only remaining comment is the test case.

We need a test case for the configuration. Please check the corner cases especially.

@dongjoon-hyun Added unit test  test("SPARK-39225: Support spark.history.fs.update.batchSize").

Merged to master. I added you to the Apache Spark contributor group and assigned SPARK-39225 to you, @hai-tao-1 .
Welcome to the Apache Spark community.


03-configure-data-model-in-power-bi-desktop.md

English:
and then in the Data Category dropdown list, select Country/Region.
Current Translation:
然后在“数据类别”下拉列表中，选择“Country/Region”。
Suggested Translation:
然后在“数据类别”下拉列表中，选择“国家/地区”。

Hello contributor,
Thank you for your contribution! We are processing your suggestions and we will update you shortly.
Kind regards,
Microsoft Worldwide Learning Team

No locfile in iCMS, So I Re-index the file, waiting for result. ICMS QUERY

fixed.


how to test the trained module?

i trained the movie review training set using this code. i got trained files in the path "runs/1458022294/summaries/train".  how can i test the module is there any API in python to test it?

Summaries are what you can visualize in Tensorboard. To load the trained model you need to load the checkpoint files. Check out the official documentation on that: https://www.tensorflow.org/versions/r0.7/how_tos/variables/index.html#variables-creation-initialization-saving-and-loading

@dennybritz
hi, there:
you use tf.train.Saver() to save all variables in train.py
but your W is defined in text_cnn.py
does the tensorflow save these variables
alse the vocabulary, vocabulary_inv that is not defined by using tf.Variable()
does the tensorflow save these?
thanks in advance~

@zishell Check out the Tensorflow documentation for storing and saving variables: https://www.tensorflow.org/versions/r0.7/how_tos/variables/index.html#variables-creation-initialization-saving-and-loading
In short, all the model parameters (W, etc) are saved by Tensorflow because they are part of the implicit graph, but the vocabulary isn't saved, you need to load it yourself. If it's not clear I recommend reading the Tensorflow documentation, it explains how variables and graphs are working.

@dennybritz thanks, I have understood of save and restore.
This is my script to classify one sentence:
if __name__ == '__main__':
    with tf.Session() as sess:

        # process the raw sentence
        new_review = "simplistic , silly and tedious . "
        new_review = new_review.strip()
        new_review = data_helpers.clean_str(new_review)
        new_review = new_review.split(" ")

        sentences, dump= data_helpers.load_data_and_labels()
        sequence_length = max(len(x) for x in sentences)
        sentences_padded = data_helpers.pad_sentences(sentences)
        vocabulary, vocabulary_inv = data_helpers.build_vocab(sentences_padded)

        num_padding = sequence_length - len(new_review)
        new_sentence = new_review + ["<PAD/>"] * num_padding

        #convert x
        x = np.array([vocabulary[word] for word in new_sentence])
        input_x = (x)

        sequence_length = x.shape[0]
        vocab_size = len(vocabulary)
        embedding_size = FLAGS.embedding_dim
        filter_sizes = map(int, FLAGS.filter_sizes.split(","))
        num_filters = FLAGS.num_filters
        l2_reg_lambda = FLAGS.l2_reg_lambda

        w_embedding = tf.Variable(tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0), name="embedding/W")
        embedded_chars = tf.nn.embedding_lookup(w_embedding, input_x)
        embedded_chars_expanded = tf.expand_dims(embedded_chars, -1)
        # Restore variables from disk.
        saver = tf.train.Saver()
        saver.restore(sess, "runs/1459166181/checkpoints/model-20000")
        print("Model restored.")

        # Create a convolution + maxpool layer for each filter size
        pooled_outputs = []
        for i, filter_size in enumerate(filter_sizes):
            with tf.name_scope("conv-maxpool-%s" % filter_size):
                # Convolution Layer
                filter_shape = [filter_size, embedding_size, 1, num_filters]
                W = tf.Variable(tf.truncated_normal(filter_shape, stddev=0.1), name="W")
                b = tf.Variable(tf.constant(0.1, shape=[num_filters]), name="b")
                # Restore variables from disk.
                saver = tf.train.Saver()
                saver.restore(sess, "runs/1459166181/checkpoints/model-20000")
                print("Model restored.")

                conv = tf.nn.conv2d(
                    embedded_chars_expanded,
                    W,
                    strides=[1, 1, 1, 1],
                    padding="VALID",
                    name="conv")
                # Apply nonlinearity
                h = tf.nn.relu(tf.nn.bias_add(conv, b), name="relu")
                # Maxpooling over the outputs
                pooled = tf.nn.max_pool(
                    h,
                    ksize=[1, sequence_length - filter_size + 1, 1, 1],
                    strides=[1, 1, 1, 1],
                    padding='VALID',
                    name="pool")
                pooled_outputs.append(pooled)

        # Combine all the pooled features
        num_filters_total = num_filters * len(filter_sizes)
        h_pool = tf.concat(3, pooled_outputs)
        h_pool_flat = tf.reshape(h_pool, [-1, num_filters_total])

        # Add dropout
        h_drop = tf.nn.dropout(h_pool_flat, 0.5)

        # Final (unnormalized) scores and predictions
        W = tf.Variable(tf.truncated_normal([num_filters_total, num_classes], stddev=0.1), name="output/W")
        b = tf.Variable(tf.constant(0.1, shape=[num_classes]), name="output/b")
        # Restore variables from disk.
        saver = tf.train.Saver()
        saver.restore(sess, "runs/1459166181/checkpoints/model-20000")
        print("Model restored.")

        scores = tf.nn.xw_plus_b(h_drop, W, b, name="scores")
        predictions = tf.argmax(scores, 1, name="predictions")
        print predictions

I have two questions:

How to convert x to input_x, when I debug the train.py, found that x_batch is tuple, so I convert the x to (x), is that right

w_embedding = tf.Variable(tf.random_uniform([vocab_size, embedding_size], -1.0, 1.0), name="embedding/W") embedded_chars = tf.nn.embedding_lookup(w_embedding, input_x) embedded_chars_expanded = tf.expand_dims(embedded_chars, -1)
the dimension of embedded_chars is [56,128] not like yours [?,56,128]
and the dimension of embedded_chars_expanded is [56,128,1] not like yours [?,56,128,1]
How can I convert it, thanks

@dennybritz I've read the Tensorflow documentation, but still couldn't quite get it right.
Since many people have asked similar question in the tutorial's comments section.
Is it possible for you to update the sample code with an eval example?

Yeah, maybe that's a good idea. I'll try to get it done this weekend.

@dennybritz  It will be very helpful if you share the work you doing now as soon as possible..thanks in advance
@zishell  I tried your code for testing that particular sentence, but i got an error like

NameError:

name 'FLAGS' is not defined

How can I resolve this?

@AAMIBhavya  I did not paste the head of my script, you should add the flags by yourself copy from the train.py

Okay, I added a sample eval.py. Also added a note to the README. Let me know if this works.

Thanks, when I run eval.py, I got output like this
`BATCH_SIZE=64
CHECKPOINT_DIR=/home/product/Downloads/cnn-text-classification-tf-master/runs/1459745478/checkpoints/
LOG_DEVICE_PLACEMENT=False
Loading data...
Vocabulary size: 18765
Test set size 10662
Evaluating...
Total number of test examples: 10662
Accuracy: 0
product@product:~/Downloads/cnn-text-classification-tf-master$ python eval.py
Parameters:
ALLOW_SOFT_PLACEMENT=True
BATCH_SIZE=64
CHECKPOINT_DIR=/home/product/Downloads/cnn-text-classification-tf-master/runs/1459745478/checkpoints/
LOG_DEVICE_PLACEMENT=False
Loading data...
[[   1  571    7 ...,    0    0    0]
[   1 3805 2181 ...,    0    0    0]
[ 718   13   44 ...,    0    0    0]
...,
[  12    9 1474 ...,    0    0    0]
[   1  166  439 ...,    0    0    0]
[3308    7   63 ...,    0    0    0]]
[1 1 1 ..., 0 0 0]
Vocabulary size: 18765
Test set size 10662
Evaluating...
Total number of test examples: 10662
Accuracy: 0`
Why I got accuracy as 0 (I just printed the values of x_test and y_test also). And one more thing, how can I test a sentence which is not in training sample, since it doesn't have any value for y_test know?

@zishell Thanks. I corrected that part. But I got error after restoring the trained model as
**Model restored.
Model restored.
Traceback (most recent call last):
File "test.py", line 90, in
name="conv")
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 211, in conv2d
use_cudnn_on_gpu=use_cudnn_on_gpu, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 655, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2042, in create_op
set_shapes_for_outputs(ret)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1528, in set_shapes_for_outputs
shapes = shape_func(op)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/common_shapes.py", line 176, in conv2d_shape
input_shape = op.inputs[0].get_shape().with_rank(4)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_shape.py", line 616, in with_rank
raise ValueError("Shape %s must have rank %d" % (self, rank))
ValueError: Shape (56, 128, 1) must have rank 4
What this value error corresponds to? How can I resolve this?

@AAMIBhavya  You may need to pull the updated code and retrain the model. The eval script I wrote only works with Tensorflow 0.7, so make sure you have the latest version.

@dennybritz Thanks. Actually the version of Tensorflow I am using is 0.7.1

Hello,
I tried running the eval.py giving the path to checkpoint folder, and getting the following error.
I am new at neural networks, could you please look into the error?
Traceback (most recent call last):
  File "eval.py", line 68, in <module>
    batch_predictions = sess.run(predictions, {input_x: x_test_batch, dropout_keep_prob: 1.0})
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 315, in run
    return self._run(None, fetches, feed_dict)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 511, in _run
    feed_dict_string)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 564, in _do_run
    target_list)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 586, in _do_call
    e.code)
tensorflow.python.framework.errors.InvalidArgumentError: computed output size would be negative
	 [[Node: conv-maxpool-5/pool = MaxPool[ksize=[1, 55, 1, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/cpu:0"](conv-maxpool-5/relu)]]
Caused by op u'conv-maxpool-5/pool', defined at:
  File "eval.py", line 50, in <module>
    saver = tf.train.import_meta_graph("{}.meta".format(checkpoint_file))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1285, in import_meta_graph
    return _import_meta_graph_def(_read_meta_graph_file(meta_graph_or_file))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1220, in _import_meta_graph_def
    importer.import_graph_def(meta_graph_def.graph_def, name="")
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/importer.py", line 238, in import_graph_def
    compute_shapes=False, compute_device=False)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2040, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1087, in __init__
    self._traceback = _extract_stack()

Hi guys! Anyone has understood how can use eval.py to test a single sentence ( present in training sample ).  I just change the data loading section to load one sentence.  but I feel confused when I try to print out the prediction.

after Load my sentence
after add pad to my sentence to the same length of the longest sentence in sample data
after  mapping sentence to vectors based on a vocabulary. ( data vocabulary from sample data )
after generate batches for one epoch

batches = data_helpers.batch_iter( data, FLAGS.batch_size, 1, shuffle=False) (data instead x_test ). I 'm not able to understand how print out prediction. This is my output
Loading data...
Vocabulary size: 18762
Test set size 10662
Evaluating...
[[1207    2  304    4  660    0    0    0    0    0    0    0    0    0
0    0    0    0    0    0    0    0    0    0    0    0    0    0
0    0    0    0    0    0    0    0    0    0    0    0    0    0
0    0    0    0    0    0    0    0    0    0    0    0    0    0]]
This is the aspect of data ( single sentence used ). It corresponds to "simplistic , silly and tedious ."
Now i don't know how use y_test fro prediction. Please help me

@AAMIBhavya It seems strange that you would get an accuracy of 0. I just tested this locally and it's working for me if I train and then point to the checkpoint dir. I'm using the latest version of Tensorflow and Python 3.
@kinarashah I haven't seen this before, but could you try train + eval with Python 3 and the latest version of Tensorflow?
@JamesRedfiled  You don't need the y_test for predictions. These are the labels and they are only used to calculate the accuracy. To get the predictions you can just print out the all_predictions variable.

@dennybritz Thanks a lot! I do not boring you with my 'silly' questions but I just tried to print 'all predictions'  with always the same input sentence..( exactly sentence pick up from sample data rt-polarity.neg ). The response is newer equal.. Sometimes return 1 or 0. I feel very 'silly'.. I wrote in http://www.wildml.com/2015/12/implementing-a-cnn-for-text-classification-in-tensorflow/ link  too.
The question is: If I use the same negative sentence from sample data, the response of all_prediction print , it'd be always 0 ??
Thank you for your time Denny!

@JamesRedfield Thanks, I'll look into it today and get back to you. The predictions should be equal, so something may be wrong there.

@JamesRedfield Thanks for finding this. It seems like most_common method does random sorting for words that occur the same number of times. So words may receive a different index during training and eval, resulting in wrong predictions.
I fixed this in https://github.com/dennybritz/cnn-text-classification-tf/commit/67272960d2e430f63cfe9886f3b849a0b8202c2b
Let me know if it works!

Thanks! I'll try!

@dennybritz When we use new test data, there may be new terms which is not in the vocabulary. How it can be handled? Can that new terms also include in that vocabulary?

Hi @AAMIBhavya. In my case, the  solution is to ignore the new terms because is not mapped. If you want to include the new terms, you 'll need to retrain the data set with new sentences with new terms.

@JamesRedfield Thanks..

@AAMIBhavya Yes, the most common way is to limit the vocabulary to a certain number of most common words (e.g. 10k or 20k) and replace all other tokens with a special UNK word/token. All words not in the vocab are then treated the same.
A more sophisticated approach is to use character-level models to "generalize" to new words, e.g. http://arxiv.org/abs/1602.00367

Hi. I also tried the eval script but I have the same problem of @kinarashah
`W tensorflow/core/kernels/pooling_ops_common.cc:64] Invalid argument: computed output size would be negative
W tensorflow/core/common_runtime/executor.cc:1102] 0x2bcda30 Compute status: Invalid argument: computed output size would be negative
[[Node: conv-maxpool-5/pool = MaxPoolksize=[1, 32, 1, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/kernels/pooling_ops_common.cc:64] Invalid argument: computed output size would be negative
W tensorflow/core/common_runtime/executor.cc:1102] 0x2bcda30 Compute status: Invalid argument: computed output size would be negative
[[Node: conv-maxpool-4/pool = MaxPoolksize=[1, 33, 1, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/kernels/pooling_ops_common.cc:64] Invalid argument: computed output size would be negative
W tensorflow/core/common_runtime/executor.cc:1102] 0x2bcda30 Compute status: Invalid argument: computed output size would be negative
[[Node: conv-maxpool-3/pool = MaxPoolksize=[1, 34, 1, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/cpu:0"]]
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 571, in _do_call
return fn(*args)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 555, in _run_fn
return tf_session.TF_Run(session, feed_dict, fetch_list, target_list)
tensorflow.python.pywrap_tensorflow.StatusNotOK: Invalid argument: computed output size would be negative
[[Node: conv-maxpool-5/pool = MaxPoolksize=[1, 32, 1, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/cpu:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "eval.py", line 69, in
batch_predictions = sess.run(predictions, {input_x: x_test_batch, dropout_keep_prob: 1.0})
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 315, in run
return self._run(None, fetches, feed_dict)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 511, in _run
feed_dict_string)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 564, in _do_run
target_list)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 586, in _do_call
e.code)
tensorflow.python.framework.errors.InvalidArgumentError: computed output size would be negative
[[Node: conv-maxpool-5/pool = MaxPoolksize=[1, 32, 1, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/cpu:0"]]
Caused by op 'conv-maxpool-5/pool', defined at:
File "eval.py", line 51, in
saver = tf.train.import_meta_graph("{}.meta".format(checkpoint_file))
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/training/saver.py", line 1285, in import_meta_graph
return _import_meta_graph_def(_read_meta_graph_file(meta_graph_or_file))
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/training/saver.py", line 1220, in _import_meta_graph_def
importer.import_graph_def(meta_graph_def.graph_def, name="")
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/importer.py", line 238, in import_graph_def
compute_shapes=False, compute_device=False)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py", line 2040, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py", line 1087, in init
self._traceback = _extract_stack()
`
I tried both with Python 2.7 and Python 3.4 and I have the same problem.
How I can fix?

@fmaglia I'm not sure what could be wrong there. I haven't seen this error before.
Are you using the latest version of Tensorflow? What are you running on (Linux/Windows/Mac)? Are you using your own data are only the code here?

I'm using the latest version of Tensorflow.
I'm running on Linux Ubuntu 14.04 LTS.
I'm using my own data (tweet of SemEval competition).
Thanks for the quick reply.
dataset.zip

Does the code work for you with the default data here? Are you sure you have the same vocabulary for both training and test data, and that the test data doesn't contain any out of vocabulary words?

I didn't tried the code with the default data.
I have different vocabulary for training and test data because the dataset are different. One is for the training of the classifier, the second is for the testing of the neural network.
Why the vocabulary should be the same?

The vocabulary must be the same or you can't index the embeddings. That's standard for almost all models (other than those that explicitly generalize to new terms, like character models). It has been discussed above, see https://github.com/dennybritz/cnn-text-classification-tf/issues/8#issuecomment-207511797
Limit the vocabulary to the N most common terms and use them for your train/test.

I was having issues with the eval script saying that accuracy was 0 as well, but I went in and changed correct_predictions and len(y_test) to floats instead of ints and that solved the problem for me.

@NickLevitt That makes sense, thanks for figuring that out. I changed it to float in the code now. Strange that it worked for me without though.

@dennybritz Thanks for your quick replies..I corrected all the errors and I also tested with my own test corpus..it gave good results. Good work..
@NickLevitt Thanks..I was having that accuracy 0 issue, but I discover that it was because of python 2. When I run that code using python3 I got the correct value for accuracy. That **accuracy 0 **problem comes only when you use python 2 (since there is a need for type casting). So try with python 3..

@dennybritz I'm also facing the same issue as @fmaglia . What I did to evaluate, was to keep just the first sentence in both the pos and neg files and run the eval.py for this data.
The code works  perfectly fine for the complete dataset and gives accuracy of 0.973551 as mentioned by you. Please let me know if you have any pointers for me. Thanks.

@nishantvishwamitra You must have the same vocabulary during training and test. In eval, you need to build the full vocabulary from the training data, and then map your test sentences in that vocabulary.

I try to solve my problem using the vocabulary of the training test also for the test set and removing words of test set that aren't in the training set but when I launch the eval script :
`W tensorflow/core/kernels/pooling_ops_common.cc:64] Invalid argument: computed output size would be negative
W tensorflow/core/common_runtime/executor.cc:1102] 0x18a1fb0 Compute status: Invalid argument: computed output size would be negative
[[Node: conv-maxpool-3/pool = MaxPoolksize=[1, 34, 1, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/kernels/pooling_ops_common.cc:64] Invalid argument: computed output size would be negative
W tensorflow/core/common_runtime/executor.cc:1102] 0x18a1fb0 Compute status: Invalid argument: computed output size would be negative
[[Node: conv-maxpool-5/pool = MaxPoolksize=[1, 32, 1, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/cpu:0"]]
W tensorflow/core/kernels/pooling_ops_common.cc:64] Invalid argument: computed output size would be negative
W tensorflow/core/common_runtime/executor.cc:1102] 0x18a1fb0 Compute status: Invalid argument: computed output size would be negative
[[Node: conv-maxpool-4/pool = MaxPoolksize=[1, 33, 1, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/cpu:0"]]
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 571, in _do_call
return fn(*args)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 555, in _run_fn
return tf_session.TF_Run(session, feed_dict, fetch_list, target_list)
tensorflow.python.pywrap_tensorflow.StatusNotOK: Invalid argument: computed output size would be negative
[[Node: conv-maxpool-3/pool = MaxPoolksize=[1, 34, 1, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/cpu:0"]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "eval.py", line 70, in
batch_predictions = sess.run(predictions, {input_x: x_test_batch, dropout_keep_prob: 1.0})
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 315, in run
return self._run(None, fetches, feed_dict)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 511, in _run
feed_dict_string)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 564, in _do_run
target_list)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/client/session.py", line 586, in _do_call
e.code)
tensorflow.python.framework.errors.InvalidArgumentError: computed output size would be negative
[[Node: conv-maxpool-3/pool = MaxPoolksize=[1, 34, 1, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/cpu:0"]]
Caused by op 'conv-maxpool-3/pool', defined at:
File "eval.py", line 51, in
saver = tf.train.import_meta_graph("{}.meta".format(checkpoint_file))
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/training/saver.py", line 1285, in import_meta_graph
return _import_meta_graph_def(_read_meta_graph_file(meta_graph_or_file))
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/training/saver.py", line 1220, in _import_meta_graph_def
importer.import_graph_def(meta_graph_def.graph_def, name="")
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/importer.py", line 238, in import_graph_def
compute_shapes=False, compute_device=False)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py", line 2040, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py", line 1087, in init
self._traceback = _extract_stack()
`
Have you any idea about what causing error?

Hi @dennybritz ! The embedding implementation doesn’t currently have GPU support and throws an error if placed on the GPU.
Is it means that I can't use GPU anyway? I tried your algorithm on AMI and I'm' trying to figure that out.. but the result is:
W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:211]
Ran out of memory trying to allocate 1.62MiB.  See logs for memory state
thanks

@JamesRedfield You can use the GPU, just not for the embeddings. Tensorflow is is missing  support for the necessary operations. Maybe they fixed it in the latest version so you cold try to upgrade.
The rest should work fine on the GPU, are you sure it's set up correctly? Do the other Tensorflow examples run on it? Running TF on AWS GPU instances is a bit tricky, but a lot has been written about it online.

I believe all is correct.
I had some problem to use tensorflow 0.8 with CUDA 7.5 and cudnn 5.0. In my case, It was very tricky run a simple example like this:
import tensorflow as tf
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
print sess.run(c)
to use GPU.
At the end, after I spent 3-4 days to read moooore forum I realized one configuration, tensor flow 0.7 - CUDA 7.0 - CUDNN 4.0. I run on python3.4.  This is my case..
Now I run your code after reboot the server but I see something not correct. Your experience, in this case, is very precious.
This is the process
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ubuntu    2177  143  2.4 39720844 383504 pts/0 Sl+  14:14   8:26 python3.4 train.py
Nvidia information
NVIDIA-SMI 346.46     Driver Version: 346.46         |
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                       Usage      |
|=============================================================================|
|    0      2177    C   python3.4                                     3819MiB |
but the training time is exactly the same. I'm using GPU device
with tf.device('/gpu:0'), tf.name_scope("embedding"):
Do you know where I wrong? Any ideas?

As I mentioned above, you can't use the GPU for the embeddings with tf.device('/gpu:0'), tf.name_scope("embedding"): - you need to use the CPU.
You can run the rest of the network on the GPU.

Oh sorry! I understand ! I believed  the 0.7 tensorflow is relative new to resolve this trouble but when you said "latest version" , you referred to 0.8. Thanks a lot!

@JamesRedfield  I'm not sure, according to https://github.com/tensorflow/tensorflow/issues/305 it seems like they added support for it. So I guess it should work...
Your error Ran out of memory trying to allocate 1.62MiB. See logs for memory state doesn't really make sense to me. Is there no memory left on the GPU? It seems like other people have had similar issues: https://github.com/tensorflow/tensorflow/issues/398
This seems like more of a Tensorflow issue, not an issue with this code. So maybe ask in the Tensorflow group...

I'll look around before changing my "safe" setting. :) Thank for you incredible  support @dennybritz

Hi @dennybritz.
If I have, for example,  positive rt-polarity data file with 5000 lines and negative rt-polarity file with 2000 lines... And if I duplicate the negative lines to reach 5000 lines ( like positive input ). How change the results or the quality of algorithm. Is it still reliable?

@JamesRedfield You are iterating through the training files multiple times anyway, just duplicating the data won't do anything.

Hello @dennybritz. I have another question and  I hope not to be tedious. :) The algorithm is not language specific? Is it right? I mean, the algorithm use a simple WORD to VECTOR without methods language oriented.
I trained more data in a different language. At the end the response is not good. I mean, If I submit a new set of negative sentences, not presente in the trained data  but similar to the negative model loaded, the result is statistically in the middle. Half is classified negative and the rest positive.
I can have your own opinion?

Yes, it's not language specific and should work as long as the language is not too different (like Chinese where you probably need to train on cahracters instead of words). Make sure that the vocabulary is big enough and that your test set doesn't use completely different vocabulary.
Btw, the convolutions are applied to the embedded vectors, not the one-hot vectors.

Hi @dennybritz
The vocabolary size is: 13K
The number of sentences is: 9000 ( 4500 are positive - 1200 are negative [4500 with duplication] )
Test set:
new negative sentences are 60
words: 1100 of which 290 are no mapped
So the average of words not mapped, in new sentences, is 30%.
But in some cases, the number of words not mapped in a new NEGATIVE sentence, is zero, so the response is the same POSITIVE. Is it makes sense for you?
The result of elaboration is half sentences classified like positive and half negative.I tried with an different size test set ( like 300 new negative sentences) but the result is exacly the same.
What do you think? Is the vocabulary is big enough?
ps. when I discover a new word not mapped, during eval.py elaboration I  put  a   character
thanks

@JamesRedfield This seems reasonable to me, not sure what could be wrong there.

Can someone share the syntax they used to test their own content (line 30 in eval.py)? I am curious about testing one sentence and also testing a batch of sentences.

A couple of basic questions from a newbie:

How long does the train.py run for and how long does eval.py run for approx?
Has anyone been able to implement the word2vec exercise mentioned at the end of the post?
What do you run your eval.py on? Has anyone been able to run eval with their own sentences?

Presenting this paper for a school presentation and need any help possible.
Thanks!

Hi, thanks for the code. Just one question: how to modify (if possible at all)  eval.py to get a precision/recall table?

@aigujin Easiest way is probably to use http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html with the predictions.

@dennybritz Hi, thanks for the suggestion; it worked. Now, I would like to clarify a couple of issues:

If I have out-of-sample sentences to predict classes, I ignore the variable y_test, right? (I don't know the classes and want a model to classify)
In the same set up (out-of-sample), do I need to build a new vocabulary or I can use the one from the model?

@aigujin

Yes
You must to use the same vocabulary you used during training because of the learned word mappings. You don't build a new one. If your examples are very different types of sentences than those used in training it may be better to re-train.

Hi @dennybritz I have a question for you!
I modify the code for training and evaluation the same data in every simulation because I would like to analyze the changing of the accuracy in the classification of the dev set. I also modified the eval.py, but the result of the eval script is different to the result of the dev accuracy in the train script. In "eval.py" the result about accuracy is better than "train.py".
I didn't modified the tensorflow code, but the only the operations in the data_helpers.py and the inizialization of the data in train.py and the eval.py.
Have you any ideas about this situation?

@fmaglia The eval.py file loads the training data by default. It's expected to get better accuracy on the training data than on dev/test data (obviously, since you're training on it). If you have a test set you should load that in eval.py.

I splitted the initial dataset in training set (67% of the dataset) and in dev/test data (33% of the dataset).
In train.py I used the training set for the training and the test set for the classification, instead in eval.py I used only the test set for the evaluation. The data set used are the same but not the results obtained!

That's very strange, it should be the same if you're using the same test set.

I found now the error. Thanks so much!

@fmaglia Hi, how did you solve the error in this comment. It seems, I am getting the same.
Thanks

@aigujin  In that case I was trying to evaluate a test set that are composed by the words, that aren't all in the vocabulary.
I solved it by changing the data_helpers.py.
I inserted the test data in the training set and I split the dataset for training and a part for testing.
But I encountered a problem: the data_helpers script inserted positive sentences and negative sentences. The latest sentences (used for test set) are all negative!
I solved with this code (in data_helpers.py):
`        # Generate labels
positive_labels = [[0, 1] for _ in positive_examples]
negative_labels = [[1, 0] for _ in negative_examples]
labels = []
totLabels = len(positive_examples) + len(negative_examples)
for j in range(totLabels):
if j%2==0:
labels.append([0,1])
else:
labels.append([1,0])
#ordering the dataset
x_text = []
totale = len(positive_examples) + len(negative_examples)
for i in range(totale):
    if i%2==0:
        x_text.append(positive_examples.pop(0))
    else:
        x_text.append(negative_examples.pop(0))`

P.S.
my dataset is balanced.

Hey,
I want to classify Q & A with 45 classes .Can i use this model ? I have already pre-processed the data into one-hot encoding as was done in this post.What else do i have to change in the code?
I trained without any errors.My vocab size is only 550 words.
Also how to evaluate a single Q's class?I have also made the changes for the batch-size and epoch parameters as my data-set is pretty small consisting of only 400 Q with 45 classes.
I tried to run the eval.py changing the data file(in data_helpers.py) to a file with only one Q.At present i am not getting any errors but i am unable to understand the result prodeuced by print(all_predictions).Can someone be more specific how to proceed?
Kindly reply as soon as possible.

want to classify Q & A with 45 classes .Can i use this model ?

Yes, it should be the same.
I can't tell much based on the error you posted. It may help to post properly formatted code/errors and a diff with your changes. Change the num_classes argument when instantiating the TextCNN during training (see train.py) and make sure your data preprocessing is right. Make sure training works first, not eval.

Hey,
Training is working fine but the error occurs in eval.py. Also there is nothing saved in  the checkpoins or summaries folders(empty).There is one more issue when i run the eval.py in the same jupyter notebook as  train.py (changing the data-set file to one containing only a single Q) , the eval file executes without any errors but i get absurd results.But when i try to execute it in another notebook it gives me the above mentioned errors.
I am posting the data_helpers.py file(changed only the function that processes data)   along with my data-set(part of it) here, kindly help me resolve the errors as soon as possible.
`import numpy as np
import re
import itertools
from collections import Counter
def clean_str(string):
string = string.replace("----"," ")
string = string.replace("?","")
string = string.replace(",","")
string = re.sub(r"[^A-Za-z0-9(),!?'`]", " ", string)
string = re.sub(r"'s", " 's", string)
string = re.sub(r"'ve", " 've", string)
string = re.sub(r"n't", " n't", string)
string = re.sub(r"'re", " 're", string)
string = re.sub(r"'d", " 'd", string)
string = re.sub(r"'ll", " 'll", string)
string = re.sub(r",", " , ", string)
string = re.sub(r"!", " ! ", string)
string = re.sub(r"(", " ( ", string)
string = re.sub(r")", " ) ", string)
string = re.sub(r"?", " ? ", string)
string = re.sub(r"\s{2,}", " ", string)
return string.strip().lower()
def load_data_and_labels():
"""
Loads  data from files, splits the data into words and generates labels.
Returns split sentences and labels.i am able to get one-hot vectors with 45 classes as [1,0,0,0,0.....,0]
"""
label=np.array([np.zeros(45)])
lab=np.array([np.zeros(45)])

#load data
class_no=0

x_texti=list(open("./data/file.txt","r").readlines())
x_texti = [s.strip() for s in x_texti]
length = len(x_texti)

#generate labels
lab[0][class_no]=1
label=lab
count=1

print(label)
for i in range(1,length):

    if(x_texti[i]!="----"):
        label=np.concatenate((label,lab),0)

        count=count+1

        #print(i)
        #print(label)

    else:
        lab[0][class_no]=0
        class_no=class_no+1
        lab[0][class_no]=1
        continue

#generate labels
#label=[lab for _ in x_texti]
#y=label
#x_text=np.array([[x_texti[0]]])
#split by words
x_text=[]
for sent in x_texti:
    if(sent!="----"):
        x_text=x_text+[clean_str(sent)]
#split by words
#x_text = [clean_str(sent) for sent in x_texti ]
x_text = [s.split(" ") for s in x_text]

return [x_text,label,count]

def pad_sentences(sentences, padding_word=""):
"""
Pads all sentences to the same length. The length is defined by the longest sentence.
Returns padded sentences.
"""
sequence_length = max(len(x) for x in sentences)
padded_sentences = []
for i in range(len(sentences)):
sentence = sentences[i]
num_padding = sequence_length - len(sentence)
new_sentence = sentence + [padding_word] * num_padding
padded_sentences.append(new_sentence)
return padded_sentences
def build_vocab(sentences):
"""
Builds a vocabulary mapping from word to index based on the sentences.
Returns vocabulary mapping and inverse vocabulary mapping.
"""
# Build vocabulary
word_counts = Counter(itertools.chain(*sentences))
# Mapping from index to word
vocabulary_inv = [x[0] for x in word_counts.most_common()]
vocabulary_inv = list(sorted(vocabulary_inv))
# Mapping from word to index
vocabulary = {x: i for i, x in enumerate(vocabulary_inv)}
return [vocabulary, vocabulary_inv]
def build_input_data(sentences, labels, vocabulary):
"""
Maps sentencs and labels to vectors based on a vocabulary.
"""
x = np.array([[vocabulary[word] for word in sentence] for sentence in sentences])
y = np.array(labels)
return [x, labels]

def load_data():
"""
Loads and preprocesses data for the dataset.
Returns input vectors, labels, vocabulary, and inverse vocabulary.
"""
# Load and preprocesses data
sentences, labels,count = load_data_and_labels()
sentences_padded = pad_sentences(sentences)
vocabulary, vocabulary_inv = build_vocab(sentences_padded)
x, y = build_input_data(sentences_padded, labels, vocabulary)
return [count,sentences, x, y, vocabulary, vocabulary_inv]
def batch_iter(data, batch_size, num_epochs, shuffle=True):
"""
Generates a batch iterator for a dataset.
"""
data = np.array(data)
data_size = len(data)
num_batches_per_epoch = int(len(data)/batch_size) + 1
for epoch in range(num_epochs):
# Shuffle the data at each epoch
if shuffle:
shuffle_indices = np.random.permutation(np.arange(data_size))
shuffled_data = data[shuffle_indices]
else:
shuffled_data = data
for batch_num in range(num_batches_per_epoch):
start_index = batch_num * batch_size
end_index = min((batch_num + 1) * batch_size, data_size)
yield shuffled_data[start_index:end_index]`
Data-set:
`Does the Natural Language Classifier train using all of the data, or does it partition it in some way?
What kind of preprocessing does the classifier perform on its input?
Does the classifier train using all of the data, or does it partition it in some way?
does the classifier train on all the data or does it hold some out?
does nlclassifier perform random sampling to the training datasets?
Where can I find documentation on the Natural Language Classifier API?
Where can I find the API documentation?
What is the API for the Classifier?
Where can I find documentation on the NL classifier API?
Where can I find REST API documentation for the NL classifier?
Where can I view documentation for the classifier API?
How do I access the classifier API?`
Here "----" implies new class.
Thanks in advance.

One more thing @dennybritz if i try to test it on another Q which has no word as in the trained vocablary , will it give me any errors while running or it will just reduce the accuracy?(right now i am just testing with the same words)

With the current code I believe it would throw an error. Here's how you usually deal with this:

Decide on a number of words in your vocabulary, e.g. 10k.
During data loading (in helpers.py), only consider the most common 10k words. Replace all other words with a special OOV (out of vocabulary) token.
During training you'll learn a representation for your 10k words and the OOV "word".
During testing do the same thing and replace all unknown words with the OOV token.

If that's too much manual work, check out the VocabularyProcessor that comes with Tensorflow/skflow. You can find an example here: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/skflow/text_classification.py

Hey @dennybritz  thanks for the replies.
I? finally resolved those errors with modifications in data_helpers.py. I am now building the vocab while training and while testing(eval.py) i load my own data from file (a single sentence) but also build the vocab from the training data and then mapping the new sentence from this vocab.
But now i am getting 0 accuracy and absurd output ( all_predictions=[14.] for one sentence). i have made one-hot vectors for the input as [1,0,0,0,......0] (here 1 for class 1) and similarly [0,1,0...0,0](here 1 for class 2) and so on. Shouldn't i get a better result or is there something wrong?
Can you just mail me @dennybritz  so that i can mail you back my files so that you can check?
my email<EMAIL_ADDRESS>

Hey,@dennybritz
Thanks for taking so much pains.I finally have a working model.
One last query how can we use word2vec model in this?
Thanks & Regards,
Raj Miglani.

To be more precise, since the function build_vocab() learns the vocabulary mapping based on the input data, both training/evaluation/test data should be indexed using the same vocabulary mapping when replacing the OOV. Otherwise, the index number of the same word could be different between training/testing.
Also, you need to make sure the max sentence length is the actual max sentence length in all training/evaluation/test data sets.

@rajmiglani See https://github.com/dennybritz/cnn-text-classification-tf/issues/10 and like zhang-jian said, make sure your vocabulary matches up.

PLEASE take me off this chain. Thanks.
On Monday, June 13, 2016, Denny Britz<EMAIL_ADDRESS>wrote:

@rajmiglani https://github.com/rajmiglani See #10
https://github.com/dennybritz/cnn-text-classification-tf/issues/10 and
like zhang-jian said, make sure your vocabulary matches up.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dennybritz/cnn-text-classification-tf/issues/8#issuecomment-225749224,
or mute the thread
https://github.com/notifications/unsubscribe/ALRpG44KMJRuCQXZRICRm7Iw1EKNoBNxks5qLfdsgaJpZM4HxCMG
.

--
Areeba Kamal
Mount Holyoke College '16

@areebak There's an unsubscribe button on the right side. I can't do anything about it.

Hey,
Sorry for the troubles guys but as my prev comment mentions i have solved the issues.
Just wanted to know how to use Word2vec in this model.

@zishell
Could you share the full script to predict one sentence? I've tried to use the eval.py, but failed to predict correct. Many thanks. :)

@dennybritz you should have a blog post or a video on your YouTube channel just explaining the process to use eval.py or maybe the entire process, which would be very helpful for many of us especially the newbies (including me). Thanks :+1: :smiley:

Yeah, I should probably write more extensive documentation and a word2vec example. I didn't expect this many people would use the code. I'll try to put something together on the weekend.

Okay, I refactored the code to make eval.py easier to use. You can now just load any string text data and don't need to deal with the vocabulary generation yourself. The vocabulary is saved during training and then automatically loaded during test. Note that you need to retrain for it to work.
See https://github.com/dennybritz/cnn-text-classification-tf/commit/9ba22d2563701c3fdc8f2707372035f32e40c98f and the new eval.py for an example.

If I run the standard model on the Rotten Tomatoes 5331/5331. Is there a way to run the eval.py on only the dev instances from the training? Or do I need to look at the summaries for that.


Remove SQL dependency and work on CSVs only

The current version of RKI-Data-Diff uses SQL for data storage. However, the storage of the overall dump with columns GueltigAb and GueltigBis could be similarly achieved by just using a CSV file, like the original data from RKI.
The original idea to use SQL was that at the end you want to query the data anyhow, as in our examples in Queries.md. The additional dependency on MySQL though makes it hard to use RKI-Data-Diff in the first place. This became clear when integrating it into GitHub Actions, as well as that we had to write a separate paragraph on common SQL errors in the README...
After all, we should just move away from the dependency on SQL and work with CSV only.

First steps are made in the 3-csv-instead-of-sql branch.


Current default MOTD causes errors

I get the following error with the default motd: error: \u characters in JSON strings are currently not supported
Probably the hammer
https://github.com/numtide/devshell/blob/827e007dd9ad7b670f05287d50247151dddd5ff0/options.nix#L107
I'm guessing this is a nix v2 limitation
Setting a custom motd fixed the issue.
Is this fix sufficient for now? Remove emoji until nix v3?

Ok there's already a PR for this issue. #12
Looks like it was the esc var rather than the hammer

Presumably fixed by #24


Edge - Docs

In the sidebar docs view, the bottom "Next" button is getting cut off by our new Toolbar.

Fixed by #1163


Convert Linux joystick code to using the evdev API

Convert Linux joystick code to using the evdev API
Add support for hats on Linux

I'd recently experimented with swapping over from SDL2 to glfw and ran into the fact that hats are currently unsupported on Linux. I found a few issues on here which seem to have came to the same conclusion as myself; there is no good way to support them without converting over to the evdev API. I've done this conversion, and added support for hats.
As a disclaimer, I have no prior experience with evdev, so please triple check this change.
One concern of mine is that I'm not at all honoring the flat member from struct input_absinfo. It seems while the fuzz member is honored by the kernel, flat is expected to be handled by userspace code. However, I don't really know how exactly it should be used.

Awesome! Will review.

Tried it out and it works beautifully. I'll make a proper review of it as well, as you suggested.

This is related to #906.

This has been merged as d1a2ec4d20d5757e6f0188476993ee8623f9cbaf with bug fixes in 206f9ca4bcebba1b232a0080d7051070e52a5e73 and 6da26c8d6c868e0864d2173cf46f32ce6238f1e2 and additional cleanup in d2952e4e92b4c06d0da6f7fe220d9095ba5028a2.
Thank you!


ActiveSupport check more accurate

Due some gems (ex. mail) define own ActiveSupport dummy class, we need more accurate check real ActiveSupport class existing.

Thank you for the submission and sorry about the long delay in response. The failures in our Travis builds have now been fixed, so rebasing on master should address the build failures in this PR.
git rebase upstream/master
Regarding this change... it would need some example code in this PR showing the problem and a test for the change before it could be considered.

Oh, it looks like there was an earlier PR for the same thing https://github.com/michaeldv/awesome_print/pull/200

I have merged https://github.com/michaeldv/awesome_print/pull/200
It also appears this was fixed by Mail v2.6.4, in the following commit:
https://github.com/mikel/mail/pull/830#discussion-diff-45785453L394

A fix for this from an earlier PR (#200) was released with version 1.7.0, but thank you for your contribution. 👍


[spiral/core] Add the ability to specify the scope of the dispatcher using an attribute

Q
A

Bugfix?
❌

Breaks BC?
❌

New feature?
✔

The ability to specify the scope name for the dispatcher using the Spiral\Attribute\DispatcherScope attribute has been added.
use Spiral\Attribute\DispatcherScope;
use Spiral\Boot\DispatcherInterface;
use Spiral\Framework\ScopeName;

#[DispatcherScope(scope: ScopeName::Console)]
final class ConsoleDispatcher implements DispatcherInterface
{
    public function canServe(): bool
    {
        // ...
    }
    public function serve(): int
    {
        // ...
    }
}

All good, but please hit me to check naming before release


Implement PGN export

https://en.wikipedia.org/wiki/Portable_Game_Notation

The export is implemented as a code block in the embed on the final game board


Query examples incorrect

[Enter feedback here]
From the text -

For example, the query string (search=/.*numeric*./) returns results on...

INCORRECT: search=/.*numeric*./
CORRECT:    search=/.*numeric.*/
Note - I have escaped the * in the markup

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 800a099f-6158-0bf9-965c-c9f05ea66d20
Version Independent ID: f15f1766-f5da-ea35-af1e-976aff808a41
Content: Partial terms, patterns, and special characters - Azure Cognitive Search
Content Source: articles/search/search-query-partial-matching.md
Service: cognitive-search
GitHub Login: @HeidiSteen
Microsoft Alias: heidist

@tingeypa Thanks for the feedback! We are currently investigating and will update you shortly.

@tingeypa Thanks again for the feedback! The PR has been created once reviewed by author then will be visible in doc.


usb: dfu: disable USB DFU uploading by default

usb: do not add ZLP if lenght to be transmitted is just zero
It is possible that the actual data stage length of
the control transfer is zero, in that case we do not
need an additional ZLP packet.
This fixes a problem with USB DFU, where after an upload
the device is no longer responsive if upload size is
multiple of control endpoint MPS.

usb: dfu: disable USB DFU uploading by default
Firmware uploading to the host may not always be desired,
disable it by default.
Enable it for existing USB DFU sample to keep
the usual behavior, add note about new option to README.rst.

@nvlsianpu can you please take a look?


HBASE-24051

If you have an inherited parent class that implements CanUnbuffer instead of directly implementing CanUnbuffer, you cannot get unbuffer

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 28s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+1 :green_heart:
mvninstall
5m 57s
master passed

+1 :green_heart:
compile
0m 58s
master passed

+1 :green_heart:
shadedjars
5m  3s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 38s
master passed

_ Patch Compile Tests _

-1 :x:
mvninstall
2m 51s
root in the patch failed.

-1 :x:
compile
0m 43s
hbase-server in the patch failed.

-0 :warning:
javac
0m 43s
hbase-server in the patch failed.

-1 :x:
shadedjars
3m 46s
patch has 16 errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 36s
the patch passed

_ Other Tests _

-1 :x:
unit
0m 39s
hbase-server in the patch failed.

22m 48s

Subsystem
Report/Notes

Docker
Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1355

JIRA Issue
HBASE-24051

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux aab50f7de711 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 76e3db6720

Default Java
1.8.0_232

mvninstall
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/1/artifact/yetus-jdk8-hadoop2-check/output/patch-mvninstall-root.txt

compile
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/1/artifact/yetus-jdk8-hadoop2-check/output/patch-compile-hbase-server.txt

javac
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/1/artifact/yetus-jdk8-hadoop2-check/output/patch-compile-hbase-server.txt

shadedjars
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/1/artifact/yetus-jdk8-hadoop2-check/output/patch-shadedjars.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/1/testReport/

Max. process+thread count
87 (vs. ulimit of 10000)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/1/console

versions
git=2.17.1 maven=2018-06-17T18:33:14Z)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 30s
Docker mode activated.

_ Patch Compile Tests _

-1 :x:
mvninstall
3m 36s
root in the patch failed.

-1 :x:
compile
0m 48s
hbase-server in the patch failed.

-0 :warning:
javac
0m 48s
hbase-server in the patch failed.

-1 :x:
shadedjars
5m  3s
patch has 16 errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 39s
hbase-server in the patch failed.

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 15s
Maven dependency ordering for patch

-1 :x:
mvninstall
3m 17s
root in the patch failed.

-1 :x:
compile
2m 16s
root in the patch failed.

-0 :warning:
javac
2m 16s
root in the patch failed.

-1 :x:
shadedjars
4m 35s
patch has 16 errors when building our shaded downstream artifacts.

-0 :warning:
javadoc
0m 39s
hbase-server in the patch failed.

-0 :warning:
javadoc
0m 19s
root in the patch failed.

_ Other Tests _

-0 :warning:
unit
6m 15s
root in the patch failed.

38m 15s

Subsystem
Report/Notes

Docker
Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1355

JIRA Issue
HBASE-24051

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 9ca5978cf8da 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 76e3db6720

Default Java
2020-01-14

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt

mvninstall
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk11-hadoop3-check/output/patch-mvninstall-root.txt

compile
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-root.txt

javac
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk11-hadoop3-check/output/patch-compile-root.txt

shadedjars
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk11-hadoop3-check/output/patch-shadedjars.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt

javadoc
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-root.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/testReport/

Max. process+thread count
563 (vs. ulimit of 10000)

modules
C: hbase-server . U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/console

versions
git=2.17.1 maven=2018-06-17T18:33:14Z)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 32s
Docker mode activated.

-0 :warning:
yetus
0m  3s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ master Compile Tests _

+0 :ok:
mvndep
0m 33s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
6m 19s
master passed

+1 :green_heart:
compile
3m 29s
master passed

+1 :green_heart:
shadedjars
5m 35s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
3m 28s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 13s
Maven dependency ordering for patch

-1 :x:
mvninstall
3m 14s
root in the patch failed.

-1 :x:
compile
2m 23s
root in the patch failed.

-0 :warning:
javac
2m 23s
root in the patch failed.

-1 :x:
shadedjars
4m 40s
patch has 16 errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
3m 57s
the patch passed

_ Other Tests _

-1 :x:
unit
6m 28s
root in the patch failed.

43m 32s

Subsystem
Report/Notes

Docker
Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1355

JIRA Issue
HBASE-24051

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 45c3b17d3846 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 76e3db6720

Default Java
1.8.0_232

mvninstall
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk8-hadoop2-check/output/patch-mvninstall-root.txt

compile
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk8-hadoop2-check/output/patch-compile-root.txt

javac
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk8-hadoop2-check/output/patch-compile-root.txt

shadedjars
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk8-hadoop2-check/output/patch-shadedjars.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-root.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/testReport/

Max. process+thread count
508 (vs. ulimit of 10000)

modules
C: hbase-server . U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/console

versions
git=2.17.1 maven=2018-06-17T18:33:14Z)

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
2m 34s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

+1 :green_heart:
@author
0m  1s
The patch does not contain any @author tags.

_ master Compile Tests _

+0 :ok:
mvndep
0m 47s
Maven dependency ordering for branch

+1 :green_heart:
mvninstall
9m 26s
master passed

+1 :green_heart:
checkstyle
3m 47s
master passed

+1 :green_heart:
spotbugs
23m 34s
master passed

_ Patch Compile Tests _

+0 :ok:
mvndep
0m 16s
Maven dependency ordering for patch

-1 :x:
mvninstall
4m  6s
root in the patch failed.

+1 :green_heart:
checkstyle
3m 43s
the patch passed

-0 :warning:
rubocop
0m  8s
The patch generated 2 new + 19 unchanged - 1 fixed = 21 total (was 20)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

-1 :x:
hadoopcheck
3m 15s
The patch causes 16 errors with Hadoop v2.10.0.

-1 :x:
hadoopcheck
6m 40s
The patch causes 16 errors with Hadoop v3.1.2.

-1 :x:
spotbugs
0m 57s
hbase-server in the patch failed.

-1 :x:
spotbugs
10m 39s
root in the patch failed.

_ Other Tests _

+1 :green_heart:
asflicense
0m 40s
The patch does not generate ASF License warnings.

69m 43s

Subsystem
Report/Notes

Docker
Client=19.03.8 Server=19.03.8 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/1355

JIRA Issue
HBASE-24051

Optional Tests
dupname asflicense rubocop spotbugs hadoopcheck hbaseanti checkstyle

uname
Linux 1c1a2993d52a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 76e3db6720

mvninstall
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-general-check/output/patch-mvninstall-root.txt

rubocop
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-general-check/output/diff-patch-rubocop.txt

hadoopcheck
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-general-check/output/patch-javac-2.10.0.txt

hadoopcheck
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-general-check/output/patch-javac-3.1.2.txt

spotbugs
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-general-check/output/patch-spotbugs-hbase-server.txt

spotbugs
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/artifact/yetus-general-check/output/patch-spotbugs-root.txt

Max. process+thread count
84 (vs. ulimit of 10000)

modules
C: hbase-server . U: .

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1355/2/console

versions
git=2.17.1 maven=2018-06-17T18:33:14Z) spotbugs=3.1.12 rubocop=0.80.0

Powered by
Apache Yetus 0.11.1 https://yetus.apache.org

This message was automatically generated.


[Bug] Lay on Hands as NonHolyPally - Spell not learned on click cast

Addon Version
r243
World of Warcraft Version
Retail
Describe the bug
I get a Spell not learned error when trying to cast Lay on Hands via click casting as a Ret or Prot Paladin. I do not have the issue when Holy. I am able to cast it normally with action bar casts. Attached is a screenshot for when LoH is bound to Shift-Left Click. Other spells seem to be fine on that bind, and moving LoH to another bind yields the same "Spell not learned" error.

Steps to reproduce
As a Ret/Prot pally, bind LoH to any click cast. You should get a "Spell not learned" error
BugSack
No response

This is a Cell bug and not a bug for this plugin, you should report it to them instead.


Implement/Fix padding truncation of each module

Codecov Report

Merging #172 (d93ded5) into dev (0f28ac4) will decrease coverage by 0.04%.
The diff coverage is 92.81%.

@@            Coverage Diff             @@
##              dev     #172      +/-   ##
==========================================
- Coverage   87.49%   87.45%   -0.04%
==========================================
  Files          60       60
  Lines        5125     5167      +42
==========================================
+ Hits         4484     4519      +35
- Misses        641      648       +7

Impacted Files
Coverage Δ

src/kwja/cli/cli.py
33.82% <0.00%> (ø)

src/kwja/datamodule/datasets/base.py
96.92% <ø> (-0.18%)
:arrow_down:

src/kwja/utils/constants.py
100.00% <ø> (ø)

src/kwja/datamodule/datasets/word.py
93.42% <82.60%> (-1.95%)
:arrow_down:

src/kwja/datamodule/datasets/senter.py
89.09% <83.33%> (-2.75%)
:arrow_down:

src/kwja/datamodule/datamodule.py
69.87% <90.47%> (+15.16%)
:arrow_up:

src/kwja/metrics/char.py
96.26% <92.30%> (-0.55%)
:arrow_down:

src/kwja/metrics/senter.py
98.57% <92.30%> (-1.43%)
:arrow_down:

src/kwja/metrics/typo.py
97.84% <92.30%> (-0.91%)
:arrow_down:

src/kwja/metrics/word.py
98.31% <94.44%> (-0.26%)
:arrow_down:

... and 18 more

... and 1 file with indirect coverage changes


Incorrect Density for HBM2_8Gb Preset in Memory Configuration

https://github.com/CMU-SAFARI/ramulator2/blob/e62c84a6f0e06566ba6e182d308434b4532068a5/src/dram/impl/HBM2.cpp#L14
Hello,
The HBM2_8Gb preset defines a density of 6<<10 (6144 Mb or 6 Gb) per channel. However, the preset name (HBM2_8Gb) suggests the density should be 8<<10 (8192 Mb or 8 Gb) per channel. This mismatch leads to errors when using this preset in configurations expecting 8 Gb density.
Steps to Reproduce

Configure Ramulator with the following setup:

DRAM:
  impl: HBM2
  org:
    preset: HBM2_8Gb
    channel: 8
    rank: 1
  timing:
    preset: HBM2_2000

Run a simulation with this configuration.
Observe the error indicating a mismatch between the calculated density and the preset density:
An error occurred: Calculated HBM2 channel density 8192 Mb does not equal the provided density 6144 Mb!

Expected Behavior
The HBM2_8Gb preset should define a density of 8<<10 (8192 Mb) per channel, matching the preset name.
Thank you!

Also, out of pure curiosity, can HBM be used with a closed-row policy? I realize it probably doesn't make much sense from a real-world use perspective, but I was wondering if it was at all possible.
For reference, the configuration errors out when you try to implement this since the closed-row policy requires a rank from the memory implementation (and HBM of course doesn't have one).
https://github.com/CMU-SAFARI/ramulator2/blob/e62c84a6f0e06566ba6e182d308434b4532068a5/src/dram_controller/impl/rowpolicy/basic_rowpolicies.cpp#L56


Iterator error after pip install

Hello,
Thanks so much for an awesome package!:-)
I recently re-pip installed plbolts on a google colab notebook. My code worked before. I'm trying to train a DQN.
On importing the DQN (from pl_bolts.models.rl import DQN), I get the following error:
/usr/local/lib/python3.7/dist-packages/torch/utils/data/_typing.py in _dp_init_subclass(sub_cls, *args, **kwargs)
    369                      return_hint.__origin__ == collections.abc.Iterator)):
    370                 raise TypeError("Expected 'Iterator' as the return annotation for `__iter__` of {}"
--> 371                                 ", but found {}".format(sub_cls.__name__, _type_repr(hints['return'])))
    372             data_type = return_hint.__args__[0]
    373             if not issubtype(data_type, sub_cls.type.param):

TypeError: Expected 'Iterator' as the return annotation for `__iter__` of ExperienceSourceDataset, but found typing.Iterable

Help would be appreciated!
Thank you:-)

hi, can you tell us how to reproduce, what code have you run?

Hey,
Thanks for the quick reply!
I ran:
!pip install atari-py
!pip install pytorch-lightning-bolts
!pip install pytorch-lightning
from pl_bolts.models.rl import DQN
import time
from pytorch_lightning import Trainer
start=time.time()
#dqn = DQN("DemonAttack-v0")
trainer = Trainer(gpus=1)
trainer.fit(dqn)
end=time.time()
print(end-start)
The code failed on the DQN import.

I encountered the same error while trying to simply import the pl_bolts library after successful installation. This was executed within the CoLab environment, and the same code used to work a couple of weeks ago.
!pip install pytorch-lightning
!pip install pytorch-lightning-bolts

import pytorch_lightning as pl
import pl_bolts

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-17-49491e23d5a9> in <module>()
      1 import pytorch_lightning as pl
----> 2 import pl_bolts
      3
      4 print(f"pl version: {pl.__version__}")
      5 print(f"pl_bolts version: {pl_bolts.__version__}")

5 frames
/usr/local/lib/python3.7/dist-packages/torch/utils/data/_typing.py in _dp_init_subclass(sub_cls, *args, **kwargs)
    369                      return_hint.__origin__ == collections.abc.Iterator)):
    370                 raise TypeError("Expected 'Iterator' as the return annotation for `__iter__` of {}"
--> 371                                 ", but found {}".format(sub_cls.__name__, _type_repr(hints['return'])))
    372             data_type = return_hint.__args__[0]
    373             if not issubtype(data_type, sub_cls.type.param):

TypeError: Expected 'Iterator' as the return annotation for `__iter__` of ExperienceSourceDataset, but found typing.Iterable

Yep, I was also working in Colab. The funny thing is that some time before the code worked.

I found that installing the latest version from the Master branch fixes the import problem (no guarantees, though, that it wouldn't break something else...):
pip install git+https://github.com/PytorchLightning/lightning-bolts.git@master --upgrade


Requires Python 3.11, not >=3.5 as the pypi docs say

A recent update uses list[str] as current PEP guidelines suggest (rpc.py, line 28). However, this breaks python installs prior to 3.11
https://stackoverflow.com/questions/75695843/is-liststr-an-iterable
(I just updated my VMs to Python 3.11, so it doesn't affect my projects personally, but recommend updating the pypi docs to specify this)

Ex: def get_blocks(self, blocks: list[str]):
Full error from Ubuntu 20.04 running python 3.8.10
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/bananopie/__init__.py", line 1, in <module>
    from .rpc import RPC
  File "/usr/local/lib/python3.8/dist-packages/bananopie/rpc.py", line 3, in <module>
    class RPC:
  File "/usr/local/lib/python3.8/dist-packages/bananopie/rpc.py", line 28, in RPC
    def get_blocks(self, blocks: list[str]):
TypeError: 'type' object is not subscriptable```

Oh and here are the pypi docs that say >=3.5
https://pypi.org/project/bananopie/

Good catch! I might remove those type definitions to support lower versions.

I would agree with that, probably want to support 3.6+

@elementalennui-dev Should be fixed, I removed the list type annotations, let me know if that is not the case

Goddamn I knew I forgot one

Ok, now it should be fixed. Augh.


Z-Index of modal__overlay

Hi,
I'm not sure if this package is still maintained but I'm finding it very useful - thanks!
In my testing, sometimes the overlay fails to cover all elements because these other elements have manually-set positive z-index. An easy fix which could just be addressed in the demo micromodal.css gist (which I'm happy to submit a PR for) is to set the z-index of the modal__overlay to be some large number.

Closing issue due to inactivity.

should do that, very common situation.

Thanks @A-Circle-Zhang , I forgot about this. Submitted the simple 1 line PR, am happy to discuss more
Thanks!


Web UX: Tabbing focuses content from another page

While tabbing through a wonder's photo gallery, artifacts, events views — a user loses sight of their cursor for several stops (~8) between a selection on the view they are on.
During this time, pressing space bar or enter reveals that content on the main wonder article view is being focused (youtube video, map) and can be opened overtop.
Expected result:
Only visible content should affect tab order and be reachable when navigating.
https://github.com/gskinnerTeam/flutter-wonderous-app/assets/7093485/cfebf9d4-fdd6-4687-ad82-93c8e7ef0fbf

Can I help in this issue


[FEA] Support batched window operations

Is your feature request related to a problem? Please describe.
This is a general request based off the specific issues seen in #1642
Currently window operations on the GPU require all of the data for a partition to be in a single GPU batch.  This limits the amount of data we can process because of limitations on the size of GPU batches.  This is actually not a requirement, but it made implementing the operator much simpler.
We should implement window processing in such a way that we don't have to have all of the data on the GPU at once, only what is required.
Describe the solution you'd like
There really are two different types of window operations that are defined by how the window is specified.
Row based window operations (look at a window that is N rows before the current row and M rows after it)
Range based window operations (look at window made up of all rows there the order by key is between the current rows value - X and the current row's value + Y)
Row based window operations also have an important sub-variant for rank operations which I will explain more on later.
Window operations are typically partitioned, and as such in the worst case we need to have all of the data for a single partition in memory at once. If there are many groups in the partition by columns we could look at the partition by columns for the last row and do a lower bounds check to figure out how much of the batch is complete and can be processed.  We then can wait until we have the rest of that group to process it. But in many cases we can actually do better than that, and in cases like #1642 there are only 3 groups total and we can run out of memory even in a single group.
Ultimately what we need is information about how much of the data (row number) is done and can be output as complete and how much of the data needs to be retained for the next batch so we can use it to get the correct answer for other rows.  In the worst case, UNBOUNDED preceding or UNBOUNDED following, this would end up splitting the data on group/partition by boundaries.
For row based queries we can calculate this ourselves.  If we look at the partition by keys for the last row in a batch we can then do a lower bound on it to find the starting point for the window.  We then add the number of rows following to that row to know where the cutoff would be for output (we have all the needed data for processing).  We then could subtract the number of rows preceding from the completed index to know how much data we have to keep around for the next batch.  In either case if it is unbounded then we should fall back to chunk it by the group by/partition by keys.
For rank queries we can do even better. Rank queries include rank, dense_range and row_number.  In this case we don't need all of the data from a window we just need the last row of the previous batch.  With that we can add any offsets needed to the new batch to pick up where we left off. For example a row_number query would only need the partition by keys and the final row number from the previous batch. After the new batch finishes processing it would look for all rows that had the same partition by keys in them (are a part of the same group) and then add the row_number from the last row of the previous batch to it.  Rank and dense rank can do similar tricks to make it work. This is important because technically these queries have a window that is unlimited preceding and 0 following. Without this optimization we would still need all of the data for a single group to be in memory at once.
For range based queries we will need help from cudf to be able to support this. We would need to be able to get back from running a window query the same information.  What row number has completed processing and has all of the data needed to produce a correct answer and what row number on needs to be retained from the current batch so we can calculate the correct results on the next batch.  Technically we cannot know the row number needed to be kept because we don't know what value will be in the next row, but we do know that it cannot be smaller than the current value, so we can keep enough data to support the last row in the current batch.
We probably want to maintain special cases for unlimited preceding/following where we just fall back to splitting on group boundaries because we know up front that processing partial groups is a waste of time.
Describe alternatives you've considered
There really isn't another option if we want to make this work.
Additional context
We could split this up into separate optimization.

[x]  Chunking/Splitting by partition keys. #1856
[x] Rank based optimization when these are all of the queries in an operation. #1859
[ ] Row based optimization when there are no unlimited window bounds. #1860
[ ] Range based optimization when there are no unlimited window bounds and once cudf can support this.

The rank/row number optimization ended up being the running window changes.  This ended up being a lot bigger than initially expected because for performance reasons this required us to switch to a scan/group by scan implementation instead.  The other memory optimizations are on hold until a customer ends up with issues because of them.


Support multi-threading

Fixes #36. Not everything works perfectly, in particular non-varobj local don't have a well-defined thread, but it's a good base to build on. I'm not 100% sure about the logic behind VAR_HANDLES_START, so the current array size increase may be overkills
Also includes some odds-and-ends fixing parsing and reference printing.

ok while quickly going over the changes I didn't notice any major issues so I will do more extensive testing later with all supported debuggers (not sure if mago supports multithreading, but it's just a side product anyway to allow D people on windows to debug if they really want to).
I don't really know anything about async in js so I need to lookup if any thread safety issues can occur there or if it's just fancy promises.
Otherwise I don't know how I feel about all the missing semicolons and using ' for strings longer than 1 character yet lol
But this seems like a good and useful PR, thanks

async in JS is effectively equivalent to promise then chains, there's still no actual parallelism or threading. I can fix the semicolons and 's.
I've only tested with gdb so YMMV, single-threaded functionality shouldn't change with other compilers though I think.

Gentle reminder.

sorry was kind of busy with other stuff and didn't really feel like reviewing something large, I can do it now

GDB

[x] single threaded still works locally
[x] multi threaded now works locally
[x] single threaded still works over SSH
[x] multi threaded now works over SSH
[ ] single threaded still works over gdbserver
[ ] multi threaded now works over gdbserver
[ ] single threaded still works when attaching
[ ] multi threaded now works when attaching
Exception:

TypeError: Cannot read property 'getThreads' of undefined
    at GDBDebugSession.threadsRequest (/tmp/code-debug/out/src/mibase.js:239:24)
    at GDBDebugSession.dispatchRequest (/tmp/code-debug/node_modules/vscode-debugadapter/lib/debugSession.js:398:22)
    at GDBDebugSession._handleData (/tmp/code-debug/node_modules/vscode-debugadapter/lib/protocol.js:97:38)
    at Socket.inStream.on (/tmp/code-debug/node_modules/vscode-debugadapter/lib/protocol.js:18:44)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:191:7)
    at readableAddChunk (_stream_readable.js:178:18)
    at Socket.Readable.push (_stream_readable.js:136:10)
    at Pipe.onread (net.js:560:20): Error: TypeError: Cannot read property 'getThreads' of undefined
    at GDBDebugSession.threadsRequest (/tmp/code-debug/out/src/mibase.js:239:24)
    at GDBDebugSession.dispatchRequest (/tmp/code-debug/node_modules/vscode-debugadapter/lib/debugSession.js:398:22)
    at GDBDebugSession._handleData (/tmp/code-debug/node_modules/vscode-debugadapter/lib/protocol.js:97:38)
    at Socket.inStream.on (/tmp/code-debug/node_modules/vscode-debugadapter/lib/protocol.js:18:44)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:191:7)
    at readableAddChunk (_stream_readable.js:178:18)
    at Socket.Readable.push (_stream_readable.js:136:10)
    at Pipe.onread (net.js:560:20)
    at file:///opt/visual-studio-code/resources/app/out/vs/workbench/workbench.main.js:2644:530
    at Object.v [as _notify] (file:///opt/visual-studio-code/resources/app/out/vs/workbench/workbench.main.js:140:302)
    at Object.enter (file:///opt/visual-studio-code/resources/app/out/vs/workbench/workbench.main.js:143:568)
    at n.Class.derive._oncancel._run (file:///opt/visual-studio-code/resources/app/out/vs/workbench/workbench.main.js:144:866)
    at n.Class.derive._oncancel._error (file:///opt/visual-studio-code/resources/app/out/vs/workbench/workbench.main.js:144:383)
    at file:///opt/visual-studio-code/resources/app/out/vs/workbench/workbench.main.js:2223:975
    at _.e.dispatch (file:///opt/visual-studio-code/resources/app/out/vs/workbench/workbench.main.js:2225:291)
    at _.e.handleData (file:///opt/visual-studio-code/resources/app/out/vs/workbench/workbench.main.js:2224:815)
    at Socket.<anonymous> (file:///opt/visual-studio-code/resources/app/out/vs/workbench/workbench.main.js:2223:835)
    at emitOne (events.js:96:13)

LLDB

[x] single threaded still works locally
[x] multi threaded now works locally.
minor issue: thread names say undefined
[ ] single threaded still works over SSH (breakpoints don't work)
[ ] multi threaded now works over SSH (breakpoints don't work)
[ ] single threaded still works when attaching
[ ] multi threaded now works when attaching

Mago-MI
~- [ ] single threaded still works locally~ (can't test because I can't run a VM because I disabled AMD-V in my bios because it locks my CPU after some random amount of time)
I don't think testing anything else here makes sense, because it is very limited
But I supposed most of these issues were present already, I am going to check for completeness though. The only thing I see when testing here right now is therefore that undefined in the thread name of lldb.

ok so the Thread exception when it fails to connect is new, but gdbserver still doesn't work.
I did find one issue though: breakpoints show as verified in the broken lldb ssh connection now, they were unverified like this  before

I can merge this as it is now because it is a good start, but I would prefer it if there were some // TODO comments where I just commented what behaviour changed

Thanks for the review, I've added some defensive ifs and TODOs. Regarding the breakpoints being verified, I think there's some semantic difference between breakpoints being verified, and being enabled. Since I get the "done" response back from lldb, I assume they're verified as far as lldb is concerned (modulo lldb mi bugs), but they might not actually be enabled because of various reasons. Regardless, there's a new callback-based breakpoint API coming, so that might do a better job of maintaining breakpoint state.


feature request: Add copy output plugin for Fluentd

Is your feature request related to a problem? Please describe.
The copy plugin for Fluentd is not available, so we cannot configure multiple outputs (See #874)
Describe the solution you'd like
Implementation of the copyoutput plugin for Fluentd
Additional context
No response

@benjaminhuo @wenchajun I would like to have your opinion about my proposal before working on the code

@benjaminhuo @wenchajun I would like to have your opinion about my proposal before working on the code

@antrema That's a big enhancement. Thanks for your proposal!
I'm ok with this proposal, @wanjunlei @wenchajun what do you think?