Big SparseTensor constant and Datasets either turn into a large graph or a slow execution.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Debian Testing/Sid
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.3.0
Python version: 3.8.5
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: 10.1.243/7.6.5
GPU model and memory: Quadro T2000

I am loading a sparse matrix via numpy.load with shape [16777216  5416537] and 335548396 values (this is the TFIDF matrix of DrQA). I want to use a dataset and use the matrix at some point of my computation. I have tried to different ways, one is simply super slow and the other one hits the 2GB limit (I assume the optimizer is kicking in and unfolding the constant matrix and overpopulation the graph, but I am not sure and my test with the config for Grappler did not work)
The slow version (I have 32GB and 16 cores available) is something like this:
indices, values, dense_shape = load_parts_of_sparse_matrix_as_np_arrays()

def f(x, indices, values, dense_shape):
  m = tf.SparseTensor(indices, values, dense_shape=dense_shape)
  return do_something(x, m)

ds = load_ds()

ds = tf.data.Dataset.zip((
  ds,
  tf.data.Dataset.from_tensors(indices).repeat(),
  tf.data.Dataset.from_tensors(values).repeat(),
  tf.data.Dataset.from_tensors(dense_shape).repeat(),
))

ds.map(f)

I guess memory is being copied at every iteration.
The following modification hits the 2gb limit
indices, values, dense_shape = load_parts_of_sparse_matrix_as_np_arrays()

m = tf.SparseTensor(indices, values, dense_shape=dense_shape)

def f(x):
  return do_something(x, m)

ds = load_ds()

ds.map(f)

This is a bit harder for me to understand. My only guess is that TF wraps the function passed to map by a tf.function then it trace the function, detects the constant and expands it overpopulating the graph somehow. I turned constant_folding off without success.
So far the only way I get good results is by disabling eager execution and use a placeholder for the parts of the sparse tensor, in any other way I hit the 2GB result. Sadly, I can't control this when using datasets.
I tried asking for help in Stack Overflow first, but I have received no answer yet.

Just to add, I am using tf.sparse.sparse_dense_matmul to multiply a dense tensor and a sparse one. I was previously using a py_function and the sparse representation of the matrix was kept in the Python side with scipy.sparse. This was fast (between 10-20 iterations per second.) I was expecting to have similar results in TF, but moving the constant matrix to the graph results in an almost 80-160x slower performance (between 4 to 8 seconds per iteration).
I can still use the py_function in my application, but I guess there has to be a way to make TF faster than scipy for this task.

Extra thoughts: I realized that the slow version is extra slow because I run the ds.map with the argument num_parallel_calls=32. This creates a memory bottleneck. If I reduce from 32 to 4 it runs faster (no memory bottleneck is noticeable), but still extremely slow in comparison.

Extra thoughts: I realized that the slow version is extra slow because I run the ds.map with the argument num_parallel_calls=32. This creates a memory bottleneck. If I reduce from 32 to 4 it runs faster (no memory bottleneck is noticeable), but still extremely slow in comparison.

Hi @jorgeecardona,
The second approach should work performantly as long as the sparse indices/values tensors can fit in the graph. The size of the dense matrix represented by the SparseTensor shouldn't matter, since only the indices/values/dense_shape are serialized into the graph.
If the indices/values are too large and exceed the 2GB graph size limit, I think you could work around the issue by providing the SparseTensor through a generator dataset:
SIZE = 1000000
indices, values, dense_shape = [[i, i] for i in range(SIZE)], list(range(SIZE)), [SIZE, SIZE]
m = tf.SparseTensor(indices, values, dense_shape=dense_shape)

def gen():
  yield m

def do_something(x, m):
  return m

ds = tf.data.Dataset.range(10).repeat()
ds_m = tf.data.Dataset.from_generator(gen, output_signature=tf.SparseTensorSpec.from_value(m)).repeat()

ds = tf.data.Dataset.zip((ds, ds_m))
ds = ds.map(do_something)

Hi @jorgeecardona,
The second approach should work performantly as long as the sparse indices/values tensors can fit in the graph. The size of the dense matrix represented by the SparseTensor shouldn't matter, since only the indices/values/dense_shape are serialized into the graph.
If the indices/values are too large and exceed the 2GB graph size limit, I think you could work around the issue by providing the SparseTensor through a generator dataset:
SIZE = 1000000
indices, values, dense_shape = [[i, i] for i in range(SIZE)], list(range(SIZE)), [SIZE, SIZE]
m = tf.SparseTensor(indices, values, dense_shape=dense_shape)

def gen():
  yield m

def do_something(x, m):
  return m

ds = tf.data.Dataset.range(10).repeat()
ds_m = tf.data.Dataset.from_generator(gen, output_signature=tf.SparseTensorSpec.from_value(m)).repeat()

ds = tf.data.Dataset.zip((ds, ds_m))
ds = ds.map(do_something)

Hi @aaudiber, thank you for your answer. I updated to TF 2.4 (in 2.3 I can't use the output_signature argument of from_generator) and I still get low performance from that. The numbers are very similar to the first implementation, and I feel there is a similar memory bottleneck when I set num_parallel_calls to 16.
Could this all be a bad implementation of the sparse multiplication? at this moment is basically 10 times slower than using py_function and scipy.sparse.
Best.

Hi @aaudiber, thank you for your answer. I updated to TF 2.4 (in 2.3 I can't use the output_signature argument of from_generator) and I still get low performance from that. The numbers are very similar to the first implementation, and I feel there is a similar memory bottleneck when I set num_parallel_calls to 16.
Could this all be a bad implementation of the sparse multiplication? at this moment is basically 10 times slower than using py_function and scipy.sparse.
Best.

Hm, each repeat() of the generator dataset will call the generator function again, which may cause the sparse tensor to be copied. Could you try adding a .cache() transformation to the from_generator dataset, right before the .repeat(). I just edited the example above to insert the .cache() in the right place.

Hm, each repeat() of the generator dataset will call the generator function again, which may cause the sparse tensor to be copied. Could you try adding a .cache() transformation to the from_generator dataset, right before the .repeat(). I just edited the example above to insert the .cache() in the right place.

Hi, I don't see an improvement with the cache. This is part of the code in case it helps:
def closest_docs_2(ds, indices, values, dense_shape, deterministic=True):

    b = tf.SparseTensor(indices, values, dense_shape=dense_shape)

    def b_gen():
        yield b

    def f(x, b):

        x = tf.sparse.reshape(x, [1, -1])

        x_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
            x.indices, x.values, x.dense_shape)

        b_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
            b.indices, b.values, b.dense_shape)

        c_sm = sparse_csr_matrix_ops.sparse_matrix_sparse_mat_mul(
            a=x_sm, b=b_sm, type=tf.float32)

        c = sparse_csr_matrix_ops.csr_sparse_matrix_to_sparse_tensor(
            c_sm, tf.float32
        )

        indices_ = tf.reshape(c.indices, [-1])

        return tf.gather(
            indices_, tf.argsort(c.values, direction='DESCENDING')
        )

        return tf.argsort(values, direction='DESCENDING')

    ds = tf.data.Dataset.zip((
        ds,
        tf.data.Dataset.from_generator(
            b_gen, output_signature=tf.SparseTensorSpec.from_value(b)
        ).cache().repeat()
    ))

    # Find the closests documents.
    return ds.map(f, num_parallel_calls=8, deterministic=deterministic)

The code with scipy is not different in spirit.

Hi, I don't see an improvement with the cache. This is part of the code in case it helps:
def closest_docs_2(ds, indices, values, dense_shape, deterministic=True):

    b = tf.SparseTensor(indices, values, dense_shape=dense_shape)

    def b_gen():
        yield b

    def f(x, b):

        x = tf.sparse.reshape(x, [1, -1])

        x_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
            x.indices, x.values, x.dense_shape)

        b_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
            b.indices, b.values, b.dense_shape)

        c_sm = sparse_csr_matrix_ops.sparse_matrix_sparse_mat_mul(
            a=x_sm, b=b_sm, type=tf.float32)

        c = sparse_csr_matrix_ops.csr_sparse_matrix_to_sparse_tensor(
            c_sm, tf.float32
        )

        indices_ = tf.reshape(c.indices, [-1])

        return tf.gather(
            indices_, tf.argsort(c.values, direction='DESCENDING')
        )

        return tf.argsort(values, direction='DESCENDING')

    ds = tf.data.Dataset.zip((
        ds,
        tf.data.Dataset.from_generator(
            b_gen, output_signature=tf.SparseTensorSpec.from_value(b)
        ).cache().repeat()
    ))

    # Find the closests documents.
    return ds.map(f, num_parallel_calls=8, deterministic=deterministic)

The code with scipy is not different in spirit.

Hi,
I am still running into this, I was able to multiply some sparse matrics with less values as the one previously tested and I get better performance in TF (both CPU and GPU) that scipy. But ones I try to use a similar matrix (similar in number of values as the one originally tested) I hit the protobuf limit:
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:504] CHECK failed: (value.size()) <= (kint32max):
Segmentation fault

I am now running TF 2.4.1.
It works still if I return the sparsetensor with the matrix from a py_function but this is called on each iteration of the dataset map and is very slow. If the  sparsetensor is created before the dataset.map I hit the limit (I assume the map traces the function and injects the eager sparse tensor into the graph.)

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.


Use adapter.quote() for get_table_types_sql

Describe the feature
Use adapter.quote() instead of double-quotes or backticks within macros/sql/get_table_types_sql.sql (as described here
https://github.com/dbt-labs/dbt-utils/pull/537#discussion_r845947638)
Who will this benefit?
This will make things cleaner and more simple when macros are migrated for https://github.com/dbt-labs/dbt-core/issues/4813

I am happy to pick this up!

Awesome @alla-bongard !
The update would go in the following places:

https://github.com/dbt-labs/dbt-utils/blob/77dcf2e609bd0528de51d2a30f1bd59110c1b12f/macros/sql/get_table_types_sql.sql#L11
https://github.com/dbt-labs/dbt-utils/blob/77dcf2e609bd0528de51d2a30f1bd59110c1b12f/macros/sql/get_table_types_sql.sql#L21
https://github.com/dbt-labs/dbt-utils/blob/77dcf2e609bd0528de51d2a30f1bd59110c1b12f/macros/sql/get_table_types_sql.sql#L30

For good measure, these could be optionally updated also:

https://github.com/dbt-labs/dbt-utils/blob/77dcf2e609bd0528de51d2a30f1bd59110c1b12f/macros/sql/get_tables_by_pattern_sql.sql#L9-L10

Note: If the bigquery__get_table_types_sql implementation becomes identical to the default__get_table_types_sql one, then it should be removed altogether 🙏

clear, and it was exactly what I did! a PR is on the way! :)

🙌


Fixing what looks to be a small bug

The operation passed into Execute/ExecuteWithRetries seems to be invoked 1 additional time than the configured circuit breaker threshold before going into the open circuit state. I don't think this is the desired behavior.
This PR should fix it.
The tests will also log some info to the console to help give context.

@tathamoddie Any progress on this one?

@tathamoddie any chance to merge this PR ?


Feature/extension method to test point inside control bounds

As discussed with @grokys this is not the correct approach, should use this.InputHitTest
see #1221


Error on arm64

+ node /srv/maverick/var/build/codeserver/scripts/nbin.js /srv/maverick/var/build/codeserver/build/code-serverproduction-vsc1.38.1-linux-aarch64-built linux code-serverproduction-vsc1.38.1-linux-aarch64
✔ Wrote 4514 files!
info  Fetching {"url":"https://nbin.cdr.sh/1.2.2/node-10.15.1-linux-arm64"}
Error: Forbidden
    at e.Binary.<anonymous> (/srv/maverick/var/build/codeserver/node_modules/@coder/nbin/out/api.js:1:43464)
    at Generator.next (<anonymous>)
    at s (/srv/maverick/var/build/codeserver/node_modules/@coder/nbin/out/api.js:1:40627)
    at process._tickCallback (internal/process/next_tick.js:68:7)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

This is because https://nbin.cdr.sh/1.2.2/node-10.15.1-linux-arm64 returns an 'Access Denied' error.

Ah, I see this is in progress: https://github.com/cdr/nbin/pull/23


Cluster support

Add support for replicated schema_versions table on clustered Clickhouse deployments

Also, please add cluster support to the README.md in Features sections.

thank you!


[Feature Request] User Friendly Reports

LinEnum finds lots of useful info but, it can some times be a bit hard to analyze when the reports big and there's lots their.
Perhaps, have an option to create an html report with hyperlinks and cullers as well as minimizing or expanding sections, this may make it easier to analyze.
Thanks

TBH this type of report has been suggested before. I agree, as the checks increase, the script output gets larger and this would be useful.

Long time user of LineEnum here. The need for a consistent and parseable report format has led me to refactor LinEnum and apply the following alternative approach: default output to json with optional export to html. You might want to have a look: enumHostLinux


Firefox extension popup window is displayed with full screen layout

A user, suspecting a Firefox browser configuration error, has the visual of the wallet looking like this:

Left normal, right "full screen" centered layout you would expect when opening the wallet on a new tab:

I would expect the centered image, large title and small explainer "enter the password you set..." on the full screen tab:

Is there anything we can do to prevent this from happening?

I suspect this is a question of just which viewport width cut-off we have for the two formatting types, and we might want to make it wider for the narrower formatting.
However, when testing in Firefox, the narrower formatting does appear for me:

So this user's popup seems to be wider than normal for some reason?

I suspect this is a question of just which viewport width cut-off we have for the two formatting types, and we might want to make it wider for the narrower formatting.
However, when testing in Firefox, the narrower formatting does appear for me:

Yes this is also what I see when using Firefox. A normal width popup. Actually it doesn't matter how small or big I make the Firefox window the popup always appears to have a fixed width.

So this user's popup seems to be wider than normal for some reason?
I checked the full screenshot again and the width of the popup for this user does look different. On the 1920x1080 screenshot he send me the wallet is approximately 760 pixels wide. When I resize the browser to HD resolution the wallet is always about a little less then 500 pixels wide on my computer.

The user was experiencing scaling issues (Windows or Firefox related, they weren't sure) so that may have caused such a difference?
If the wallet dictates the width of the popup normally (I assume that because it is always the same for me) why doesn't the wallet do that in this case? should it?
If we can fix this by changing the viewport width cut-off that sounds good to me.

Closing this off as it's outdated and we aren't supporting Firefox now anyway.
When I tested it is also working correctly with the latest design anyway:


Non inplace model editing

Improvement! Edits defined within an edit context don't affect the base model by default. This means that the edits are not in-place.
from nnsight import LanguageModel

gpt2 = LanguageModel("openai-community/gpt2", device_map="auto")

with gpt2.edit("") as gpt2_edited:
        gpt2.transformer.h[1].output[0][:] = 0

with gpt2.trace("Hello World"):
    l1_out = gpt2.transformer.h[1].output[0].save()

with gpt2_edited.trace("Hello World"):
    l1_out_edited = gpt2_edited.transformer.h[1].output[0].save()

print("L1 - Out: ", l1_out)
print("L1 - Out (edited): ", l1_out_edited)

"""
L1 - Out: tensor([[[ 1.0967, -1.8792,  1.0121,  ..., -1.0442, -0.5490, -1.1239],
         [-0.2300,  0.5204,  0.4756,  ..., -1.6795, -0.5285,  0.4190]]],
       device='mps:0', grad_fn=<AddBackward0>)

L1 - Out (edited): tensor([[[0., 0., 0.,  ..., 0., 0., 0.],
         [0., 0., 0.,  ..., 0., 0., 0.]]], device='mps:0',
       grad_fn=<CopySlices>)
"""

If you wish to make in-place edits, simply set inplace=True:
with model.edit(inplace=True):
    pass

New Feature! If you wish to access the Tracer context responsible for the edits, simply set return_context=True:
with model.edit(return_context=True) as (edited_model, editor):
    pass

Note that the return_context flag is only relevant when creating an edit context.
New feature! You can now clear edits from a model. Particularly useful if you run in-place edits on a model.
from nnsight import LanguageModel

gpt2 = LanguageModel("openai-community/gpt2", device_map="auto")

with gpt2.edit() as gpt2:
    gpt2.transformer.h[1].output[0][:] = 0

gpt2.clear_edits()

with gpt2.trace("Hello World"):
    l1_out_unedited = gpt2.transformer.h[1].output[0].save()

print("L1 - Out (unedited): ", l1_out_unedited)

"""
L1 - Out (unedited): tensor([[[ 1.0967, -1.8792,  1.0121,  ..., -1.0442, -0.5490, -1.1239],
         [-0.2300,  0.5204,  0.4756,  ..., -1.6795, -0.5285,  0.4190]]],
       device='mps:0', grad_fn=<AddBackward0>)
"""

@AdamBelfki3 does the editor object expose any methods besides .log()? .exit() does not seem to work.


Incorrect Expected Value Causes Re-execution

🐛 Bug Report
Hi, I'm new to Jest so this may be caused by ignorance on my part.  I'd be happy if it's me!
I'm testing an async function let's call doSomething() which returns a status code and a result via a callback.  The async function itself calls a library function mod.fun() which returns a promise.
I am attempting to mock mod.fun() to test doSomething().  The test validates that the right status and data is returned from the async function via the callback, using two expects:
expect(status).toBe(42);
expect(dataOut).toEqual({a: 1});

So long as the dataOut return object matches the expected value (here {a: 1}), the test succeeds as expected.  However, if the argument to toEqual() is changed to an incorrect value, the test fails; not because the dataOut is incorrect (this would be expected) but because the status code changed to -1.
Adding some diagnostic console.log()s illustrates that the callback is being interrupted and called twice.  I believe this is because the callback is called once for the .then() and once for the .catch() for the mocked function.
To Reproduce

clone https://github.com/hungryogre/jest-test-expect.git
run npm install
run npm test; the test succeeds.  All three test messages from the console.log() calls on lines 11, 13, and 15 execute exactly once.  This is expected.
in index.test.js line 14, change the expected object parameter in .toEqual() to something other than {a: 1}.  Look at the change for the last commit to see an example.
run npm test; the test fails, but because the expected status value is -1 instead of 42. Test message 1 is output twice, message 2 output once, and message 3 never.

Expected behavior
I would expect (ahem):

the doSomething() callback to be called only once,
the test to fail because Jest expected the .toEqual() matcher failed, and
I would not expect both the .then() and the .catch() methods to be called

Link to repl or repo (highly encouraged)
https://github.com/hungryogre/jest-test-expect.git
Run npx envinfo --preset jest
Paste the results here:
  System:
    OS: macOS High Sierra 10.13.6
    CPU: x64 Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
  Binaries:
    Node: 10.8.0 - /usr/local/bin/node
    npm: 6.4.1 - /usr/local/bin/npm
  npmPackages:
    jest: ^23.6.0 => 23.6.0

I have also duplicated the issue on Windows 10, node v8.11.4 and npm 6.4.1.
Thank you, and apologies in advance if this is a noob issue.

This is a problem with using callbacks and the way Jest uses assertions. The callback fails because the assertion fails. That's picked up by your promise chain, which then invokes its catch, calling the function again. When that fails, you have no catch, so it bubbles up to Jest for it to handle.
We recommend to use the promise API (either returning the promise, or using async-await). If you have to use callbacks, you must try-catch your assertions:
diff --git i/index.test.js w/index.test.js
index 1c467be..3203610 100644
--- i/index.test.js
+++ w/index.test.js
@@ -8,11 +8,15 @@ test('it should return 42', done => {
     mod.fun.mockReturnValueOnce(Promise.resolve({ a: 1 }))

     doSomething({ y: 7 }, (code, result) => {
-        console.log(`test1`)
-        expect(code).toBe(42)
-        console.log(`test2`)
-        expect(result).toEqual({ a: 1 })
-        console.log(`test3`)
-        done()
+        try {
+            console.log(`test1`)
+            expect(code).toBe(42)
+            console.log(`test2`)
+            expect(result).toEqual({a: 1})
+            console.log(`test3`)
+            done()
+        } catch (e) {
+            done(e)
+        }
     })
 })

If I have that test with empty toEqual, this is the result:

Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.

Aw dang.  I'm sorry for posting here.  Donation to Jest for atonement!

No worries! Happy to take PRs to our docs about this, if you feel up for it! 🙂 It should emphazise promises over callbacks, and probably include this gotcha


Credit and fraud detection

A python project which takes details of a transaction and predicts whether a given transaction is fraud or not.

Must Give a Star to repo for Merging Successfully. Thanks

Can you please merge and close the pull request.


client_channel connectivity state hangs for a period when resolver result changes.

What version of gRPC and what language are you using?
v1.58.0, python
What operating system (Linux, Windows,...) and version?
Linux
What runtime / compiler are you using (e.g. python version or version of gcc)
gcc
What did you do?
Our service structure look like this:
The client use a hostname which resolves to a IP of Nginx. Nginx proxies the request to backend server.

Set debug log environment variables to output trace log:

export GRPC_TRACE=handshaker,connectivity_state,client_channel,call_error,subchannel
export GRPC_VERBOSITY=debug

Then use python interactive terminal to create python client instance, connect and send a gRPC request to hostname. It should succeed.

Don't stop your python terminal. wait for about 2 minutes. by default, nginx server sends a HTTP2 GOAWAY FRAME to the idle client. Which causes
the connectivity state of client_transport to change: READY -> TRANSIENT_FAILURE, then TRANSIENT_FAILURE->SHUTDOWN
and the connectivity state of client_channel to change READY -> IDLE

Then start nginx in another host,  change the dns record to the new nginx's host.

Wait for the DNS record TTL to end(until the dig result is always the new one)

Use your python terminal to send a new request through previously created client instance.

Use netstat -netp |grep python to check your python client connection state.

At first the client should try previous IP.
The connectivity state of client_channel should change first from IDLE -> CONNECTING, then CONNECTING -> TRANSIENT_FAILURE
Then after reconnect backoff timeout, it should use the new IP to connect.

What did you expect to see?
Before connect to the new IP, connectivity state should change to CONNECTING first.
After connected to the new IP. the client should be able receive response from server.
What did you see instead?
The trace log shows the client_channel connectivity state is always TRANSIENT_FAILURE.
The connectivity state of client_channel tries to change TRANSIENT_FAILURE->IDLE and IDLE -> CONNECTING, but ConnectivityStateTracker failed to track the actual state changes. The state stucks at TRANSIENT_FAILURE.
After connected to the new IP, all client request still failed.
After about 2 minutes, the connectivity state recovered from  TRANSIENT_FAILURE -> READY directly.
Below are all the logs I got.
grpc.log
Anything else we should know about your project / environment?

@yijiem please help take a look, thank you very much!

@XuanWang-Amos, if you run an interactive python terminal like this, is there going to be a background thread doing polling for C-core, or is it starving the client of polling?

Assume this is not Async, Python client thread will be blocked waiting for next event from C-core.

I'm not asking about whether the python API is async.  I'm asking whether there is some other thread polling a CQ or whether the python interpretter sitting at a prompt is going to starve the client of polling.

We do have a daemon thread polling for connectivity state:https://github.com/grpc/grpc/blob/a100e90f9760a115722cec072d518bef080971a9/src/python/grpcio/grpc/_channel.py#L1901-L1908
Also want to note that it's possible to skip some intermediate state, but 2 minutes sounds a long time.

By the way, there's an interesting fact might help: We use TLS, but when client start using new IP for connection, after the tcp connection established, it won't lanch TLS handshake. No client hello is sent. It just waits and reports error of handshake timeout.

We do have a daemon thread polling for connectivity state:

@XuanWang-Amos It seems the the daemon polling thread is not started by default? It seems it will only be started when user calls the channel.subscribe to register a callback when connectivity changes?

Yeah if you're not calling channel.subscribe then we won't start the new thread.
We checked the log again, looks like we did resolve to the new IP address, but then the handshake failed with Handshake timed out error, it would be helpful if you can also provide some logs from the 2nd Nginx server.
I1227 06:13:17.381883608   20404 client_channel.cc:1248]               chand=0x55b3e7d23070: got resolver result
I1227 06:13:17.381905693   20404 client_channel.cc:1310]               chand=0x55b3e7d23070: resolver returned no service config. Using default service config for channel.
I1227 06:13:17.381919588   20404 client_channel.cc:1347]               chand=0x55b3e7d23070: service config not changed
I1227 06:13:17.381928561   20404 client_channel.cc:1426]               chand=0x55b3e7d23070: Updating child policy 0x7fa2d0014d50
I1227 06:13:17.381938134   20404 child_policy_handler.cc:246]          [child_policy_handler 0x7fa2d0014d50] updating child policy 0x7fa2d0002840
I1227 06:13:17.382020267   20404 subchannel.cc:512]                    subchannel 0x7fa2d0119830 {address=ipv4:<IP_ADDRESS>:443, args={grpc.client_channel_factory=0x55b3e7a26e90, grpc.default_authority=in01-shaoyue-test.az-westus3.vectordb-uat3.zillizcloud.com:443, grpc.enable_retries=1, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x55b3e7bb6160, grpc.internal.event_engine=0x7fa300015ed0, grpc.internal.security_connector=0x7fa2d012fef0, grpc.internal.subchannel_pool=0x55b3e7c49bd0, grpc.keepalive_time_ms=55000, grpc.max_receive_message_length=-1, grpc.max_send_message_length=-1, grpc.min_reconnect_backoff_ms=5000, grpc.primary_user_agent=grpc-python/1.58.0, grpc.resource_quota=0x55b3e7c23a70, grpc.server_uri=dns:///in01-shaoyue-test.az-westus3.vectordb-uat3.zillizcloud.com:443}}: throttling keepalive time to 55000
I1227 06:13:17.382030629   20404 client_channel.cc:499]                chand=0x55b3e7d23070: creating subchannel wrapper 0x55b3e78235c0 for subchannel 0x7fa2d0119830
I1227 06:13:17.382037561   20404 client_channel.cc:624]                chand=0x55b3e7d23070: connectivity change for subchannel wrapper 0x55b3e78235c0 subchannel 0x7fa2d0119830; hopping into work_serializer
I1227 06:13:17.382053081   20404 client_channel.cc:648]                chand=0x55b3e7d23070: processing connectivity change in work serializer for subchannel wrapper 0x55b3e78235c0 subchannel 0x7fa2d0119830 watcher=0x7fa300016100 state=IDLE status=OK
I1227 06:13:17.382072570   20404 handshaker.cc:75]                     handshake_manager 0x7fa3000011f0: adding handshaker tcp_connect [0x7fa2d0127350] at index 0
I1227 06:13:17.382081726   20404 handshaker.cc:75]                     handshake_manager 0x7fa3000011f0: adding handshaker http_connect [0x7fa300004030] at index 1
I1227 06:13:17.382156844   20404 handshaker.cc:75]                     handshake_manager 0x7fa3000011f0: adding handshaker security [0x7fa2d0130710] at index 2
I1227 06:13:17.382177579   20404 handshaker.cc:100]                    handshake_manager 0x7fa3000011f0: error=OK shutdown=0 index=0, args={endpoint=(nil), args={grpc.client_channel_factory=0x55b3e7a26e90, grpc.default_authority=in01-shaoyue-test.az-westus3.vectordb-uat3.zillizcloud.com:443, grpc.enable_retries=1, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x55b3e7bb6160, grpc.internal.event_engine=0x7fa300015ed0, grpc.internal.security_connector=0x7fa2d012fef0, grpc.internal.subchannel_pool=0x55b3e7c49bd0, grpc.internal.tcp_handshaker_bind_endpoint_to_pollset=1, grpc.internal.tcp_handshaker_resolved_address=ipv4:<IP_ADDRESS>:443, grpc.keepalive_time_ms=55000, grpc.max_receive_message_length=-1, grpc.max_send_message_length=-1, grpc.min_reconnect_backoff_ms=5000, grpc.primary_user_agent=grpc-python/1.58.0, grpc.resource_quota=0x55b3e7c23a70, grpc.server_uri=dns:///in01-shaoyue-test.az-westus3.vectordb-uat3.zillizcloud.com:443}, read_buffer=0x7fa300001050 (length=0), exit_early=0}
I1227 06:13:17.382189579   20404 handshaker.cc:146]                    handshake_manager 0x7fa3000011f0: calling handshaker tcp_connect [0x7fa2d0127350] at index 0
I1227 06:13:17.382340062   20404 client_channel.cc:624]                chand=0x55b3e7d23070: connectivity change for subchannel wrapper 0x55b3e78235c0 subchannel 0x7fa2d0119830; hopping into work_serializer
I1227 06:13:17.382356819   20404 client_channel.cc:522]                chand=0x55b3e7d23070: destroying subchannel wrapper 0x55b3e79b9540 for subchannel 0x55b3e7d2e530
I1227 06:13:17.382375990   20404 client_channel.cc:648]                chand=0x55b3e7d23070: processing connectivity change in work serializer for subchannel wrapper 0x55b3e78235c0 subchannel 0x7fa2d0119830 watcher=0x7fa300016100 state=CONNECTING status=OK
I1227 06:13:17.382402920   20404 handshaker.cc:100]                    handshake_manager 0x55b3e79ab780: error=UNKNOWN:tcp handshaker shutdown {created_time:"2023-12-27T06:13:17.38237152+00:00"} shutdown=1 index=1, args={endpoint=(nil), args={}, read_buffer=(nil) (length=0), exit_early=0}
I1227 06:13:17.382416349   20404 handshaker.cc:132]                    handshake_manager 0x55b3e79ab780: handshaking complete -- scheduling on_handshake_done with error=UNKNOWN:tcp handshaker shutdown {created_time:"2023-12-27T06:13:17.38237152+00:00"}
I1227 06:13:20.282889129   20402 handshaker.cc:100]                    handshake_manager 0x7fa3000011f0: error=OK shutdown=0 index=1, args={endpoint=0x7fa2d01286d0, args={grpc.client_channel_factory=0x55b3e7a26e90, grpc.default_authority=in01-shaoyue-test.az-westus3.vectordb-uat3.zillizcloud.com:443, grpc.enable_retries=1, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x55b3e7bb6160, grpc.internal.event_engine=0x7fa300015ed0, grpc.internal.security_connector=0x7fa2d012fef0, grpc.internal.subchannel_pool=0x55b3e7c49bd0, grpc.keepalive_time_ms=55000, grpc.max_receive_message_length=-1, grpc.max_send_message_length=-1, grpc.min_reconnect_backoff_ms=5000, grpc.primary_user_agent=grpc-python/1.58.0, grpc.resource_quota=0x55b3e7c23a70, grpc.server_uri=dns:///in01-shaoyue-test.az-westus3.vectordb-uat3.zillizcloud.com:443}, read_buffer=0x7fa300001050 (length=0), exit_early=0}
I1227 06:13:20.282918232   20402 handshaker.cc:146]                    handshake_manager 0x7fa3000011f0: calling handshaker http_connect [0x7fa300004030] at index 1
I1227 06:13:20.282929533   20402 handshaker.cc:100]                    handshake_manager 0x7fa3000011f0: error=OK shutdown=0 index=2, args={endpoint=0x7fa2d01286d0, args={grpc.client_channel_factory=0x55b3e7a26e90, grpc.default_authority=in01-shaoyue-test.az-westus3.vectordb-uat3.zillizcloud.com:443, grpc.enable_retries=1, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x55b3e7bb6160, grpc.internal.event_engine=0x7fa300015ed0, grpc.internal.security_connector=0x7fa2d012fef0, grpc.internal.subchannel_pool=0x55b3e7c49bd0, grpc.keepalive_time_ms=55000, grpc.max_receive_message_length=-1, grpc.max_send_message_length=-1, grpc.min_reconnect_backoff_ms=5000, grpc.primary_user_agent=grpc-python/1.58.0, grpc.resource_quota=0x55b3e7c23a70, grpc.server_uri=dns:///in01-shaoyue-test.az-westus3.vectordb-uat3.zillizcloud.com:443}, read_buffer=0x7fa300001050 (length=0), exit_early=0}
I1227 06:13:20.282936944   20402 handshaker.cc:146]                    handshake_manager 0x7fa3000011f0: calling handshaker security [0x7fa2d0130710] at index 2

D1227 06:13:22.383410837   20411 security_handshaker.cc:208]           Security handshake failed: UNKNOWN:Handshake read failed {created_time:"2023-12-27T06:13:22.383367605+00:00", children:[UNKNOWN:FD Shutdown {created_time:"2023-12-27T06:13:22.383297727+00:00", children:[UNKNOWN:Handshake timed out {created_time:"2023-12-27T06:13:22.383285445+00:00"}]}]}
I1227 06:13:22.383452807   20411 handshaker.cc:100]                    handshake_manager 0x7fa3000011f0: error=UNKNOWN:Handshake read failed {created_time:"2023-12-27T06:13:22.383367605+00:00", children:[UNKNOWN:FD Shutdown {created_time:"2023-12-27T06:13:22.383297727+00:00", children:[UNKNOWN:Handshake timed out {created_time:"2023-12-27T06:13:22.383285445+00:00"}]}]} shutdown=1 index=3, args={endpoint=(nil), args={}, read_buffer=(nil) (length=0), exit_early=0}
I1227 06:13:22.383475866   20411 handshaker.cc:132]                    handshake_manager 0x7fa3000011f0: handshaking complete -- scheduling on_handshake_done with error=UNKNOWN:Handshake read failed {created_time:"2023-12-27T06:13:22.383367605+00:00", children:[UNKNOWN:FD Shutdown {created_time:"2023-12-27T06:13:22.383297727+00:00", children:[UNKNOWN:Handshake timed out {created_time:"2023-12-27T06:13:22.383285445+00:00"}]}]}
I1227 06:13:22.383589856   20411 subchannel.cc:711]                    subchannel 0x7fa2d0119830 {address=ipv4:<IP_ADDRESS>:443, args={grpc.client_channel_factory=0x55b3e7a26e90, grpc.default_authority=in01-shaoyue-test.az-westus3.vectordb-uat3.zillizcloud.com:443, grpc.enable_retries=1, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x55b3e7bb6160, grpc.internal.event_engine=0x7fa300015ed0, grpc.internal.security_connector=0x7fa2d012fef0, grpc.internal.subchannel_pool=0x55b3e7c49bd0, grpc.keepalive_time_ms=55000, grpc.max_receive_message_length=-1, grpc.max_send_message_length=-1, grpc.min_reconnect_backoff_ms=5000, grpc.primary_user_agent=grpc-python/1.58.0, grpc.resource_quota=0x55b3e7c23a70, grpc.server_uri=dns:///in01-shaoyue-test.az-westus3.vectordb-uat3.zillizcloud.com:443}}: connect failed (UNKNOWN:Handshake read failed {created_time:"2023-12-27T06:13:22.383367605+00:00", children:[UNKNOWN:FD Shutdown {created_time:"2023-12-27T06:13:22.383297727+00:00", children:[UNKNOWN:Handshake timed out

@XuanWang-Amos I didn't open the debug log for nginx. I'll reproduce it again.

@XuanWang-Amos When I was reproducing it again I noticed that I used to the grpc.min_reconnect_backoff_ms from its default 20000 to 10000 to shorten recovery time.
This time I didn't set grpc.min_reconnect_backoff_ms, it looks much better, it took 38s to recover. better than last time, but still much longer than expected. Nginx's log seems not helpful, but I captured all traffics with tcpdump.
timeout20s-cli.log
timeout20s-grpc-client.pcap.tar.gz
Let me describe this in detail using the timeline of timeout20s-grpc-client.pcap:
frame 0 is at about D0104 06:34:12.003721114

Frame 0-23: Client TLS handshake with old nginx <IP_ADDRESS>:443

Frame 24-31: Client sends test grpc call, and receive response immediately  from sever.

Frame 32-39: old nginx sends GOAWAY to client, connection is closed.

Frame 40-47: client start another grpc call, the client_channel first trying old IP with a 20 seconds min backoff timeout.

Frame 48-57: Client start trying new nginx <IP_ADDRESS>:443, establish a tcp connection and start cipher exchanging.
Theclient hello is sent 3.5 seconds after tcp handshake finished. Which is very strange, because last time it was sent immediately.

6.Frame 58-67: Note frame 58 Change Cipher Spec is also 5 seconds later after client side received server hello. This is expected to happen immediately after server hello is received. Afterwards TLS handshake is finished successfully.
The TCP transmission frame shows the old socket for old nginx is still trying to reach it.

7.Frame 68-end: another 5 seconds later the TLS handshake finished, the client channel is established. another 2 seconds later the request is sent, and response is received.

@haorenfsa Thank you for the detailed timeline! I think we understand the issue quite a bit better now. It has to do with delays while establishing the second connection.
@yashykt Can you help delegate this to a C++ team member who can investigate?

I just want to point out that the fact that we're seeing 5-second delays here reinforces my earlier guess that this is polling starvation.  If the application is not polling the CQ, then it will fall back on the client channel backup poller, which runs every 5 seconds by default.
You can try confirming this by running with the env var GRPC_CLIENT_CHANNEL_BACKUP_POLL_INTERVAL_MS=1000.  If that results in the 5-second delays getting replaced with 1-second delays, then that's pretty clear confirmation that there's a polling problem here.

@makdharma Good to hear that! I'll test it later

Haven’t got time to test it yet. How cloud I reopen this?


Defaults

Hi Lukas,
please, remove Defaults from source code. Add it to docs only with suggestion to place it to app/_config. I can't disable your defaults now, both mine and your Defaults are populated as yml files are merged.

@silverstripesk you can "null" the defaults in your project's configuration, see https://docs.silverstripe.org/en/4/developer_guides/configuration/configuration/#configuration-values

There is an new version on the way with better defaults - just not finished yet :nerd_face: I've also reworked translations and probable 'll make the whole config available for admins only as json with a code editor in siteconfig.


Close proxycommand socket when its not needed anymore and resolve ssh zombies

Needs confirmation that this is written correctly from @eedgar (-- @punkrokk )
Paramiko has a bug that is not patched, that causes ssh proxy connected to not get closed.
eg. zombie ssh processes are left from the proxycommand
If I were to modify my st2.conf file as follows:
#/bin/bash
crudini --set /etc/st2/st2.conf ssh_runner use_ssh_config True
crudini --set /etc/st2/st2.conf ssh_runner ssh_config_file_path /root/.ssh/config
chown -R root:root /root/.ssh/*
chmod 600 /root/.ssh/config
chmod 600 /root/.ssh/id_rsa

  Host 10.1.*
    ProxyCommand ssh -o StrictHostKeyChecking=no bastion nc %h %p
    IdentityFile ~/.ssh/id_rsa
    User stanley

  Host bastion
    Hostname bastion.example.com
    IdentityFile ~/.ssh/id_rsa
    User stanley

We end up with zombie ssh connections. Paramiko has a bug, but it is not patched: https://github.com/paramiko/paramiko/issues/789
This PR checks for sockets that are left open and closes them.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@StackStorm/maintainers Can someone comment on how to update the Unit tests based on this find?
https://github.com/StackStorm/st2docs/pull/965

@eedgar are you able to help us update the CI tests for this?

Moving to 3.1 since I think this is a bug to get squashed.

Merging.
Thanks @eedgar for the contribution and @punkrokk @blag for the review and great effort in getting the PR reach the quality bar!

Really thanks to @eedgar for the 2nd effort here!


(Request) Oregairu My Youth Romantic Comedy Is Wrong, As I Expected Switch (NSP)

Yahari Game demo Ore no Seishun Love Come wa Machigatteiru.


Small fixups and re-organizations

Add disclaimer

Job VTB Documentation on 44b4352 wanted to post the following:
View the site here
This comment will be updated on new commits.


Add SNZB-02D support

You need to add id 1771 to the closed-source FE part too, obviously. I just hacked it in minified versions in this repo.

Tank you


Add checkout strategy to stash apply/pop methods

For a finer control over the unstash process, propogate checkout strategy to Objective-Git from the underlying libgit2 stash methods by adding an extra strategy argument to -[GTRepository applyStashAtIndex:…] and  -[GTRepository popStashAtIndex:…]

@tiennou , I'm going to flood you with these one by one now :D

@slavikus Could you rebase this branch onto master with #459 being merged?

@pietbrauer will do.

@pietbrauer there we go.

@slavikus So it's basically adding  (with a default checkout strategy) and two new lines or did something go wrong in the merging?

@pietbrauer In this PR, I've added checkout strategy as a separate parameter; in #459 , it was replaced by more generic (and more broad) GTCheckoutOptions class, so I've removed the more shallow variant that was previously used here.

Ok, so something went wrong. There are only 4 changed lines

Nah, not really. #459 basically superceeded this one entirely. :)

However, now when I think of it, the  "(with a default checkout strategy)" makes no sense anymore. :) I'll push a clarifying commit.

Ok, cool. Thanks. No need to wait for CI then.


certified images with operator-sdk build

Producing RedHat-certified images requires a license.txt in the licenses directory and also certain mandatory labels. Can't see any way to tell "operator-sdk build" that I want to set these. I've created a new image using the image built from "operator-sdk build" as a base image but not clear to me if that's the intended route?

@ryandawsonuk if by "set" a license.txt and labels you mean add them to the built image, you can do so by adding them to the build/Dockerfile with COPY and LABEL directives.

Just realised I didn't explain this very well. I am indeed adding in to the built image by building on top of it. What I was suggesting is that it would be good if the [operator-sdk build](https://github.com/SeldonIO/seldon-deploy-operator/blob/6c5bb4ac4e12191b908515ee52987c8d485f9a8d/Makefile#L10) command could support options to add at that point.

@ryandawsonuk I'm going to close this because the operator-sdk build command was removed in 1.0.0. Its replacement is the docker-build target in the Makefile that is scaffolded by operator-sdk init, which you can tweak with your desired image builder, flags, etc.


websocket return HTTP 400 when proxy pass

failed: Error during WebSocket handshake: Unexpected response code: 400
sometimes see the error (not always).
Add dollar sign

something wrong need to edit


gRPC - Istio 1.38 EnvoyFilter not working

Bug description
I am using the following Lua filter to log request/response data. But the Lua filter doesn't seem to be executed.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: lua-test
  namespace: staging-as
spec:
  workloadSelector:
    labels:
      app: route-guide
      component: route-server
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        #portNumber: 8980
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "mixer"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.lua
        config:
          inline_code: |
            function envoy_on_request(handle)
              handle:logWarn("DEBUG REQUEST")
              print ("**********************")
              handle:logInfo("Envoy request");
            end
            function envoy_on_response(handle)
              handle:logWarn("DEBUG RESPONSE")
              print ("**********************")
              handle:logInfo("Envoy response");
            end

I get only the following log message after the gRPC client completes receiving its response from gRPC service
[2020-07-31T16:43:28.410Z] "- - -" 0 - "-" "-" 1400 5804 8980 - "-" "-" "-" "-" "<IP_ADDRESS>:8980" inbound|8980||route-guide-service.staging-as.svc.cluster.local <IP_ADDRESS>:58936 <IP_ADDRESS>:8980 <IP_ADDRESS>:22380 - -
I tried adding the below
       typed_config:
         "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"

but got this error:
admission webhook "pilot.validation.istio.io" denied the request: configuration is invalid: envoy filter: can't unmarshal Any nested proto *v2.HttpConnectionManager: unknown field "inlineCode" in v2.HttpConnectionManager
I already raised issue #25981, but pre-maturely closed as duplicate.
gRPC service is running in port 8980 in Kubernetes. The following is the service+deployment info
apiVersion: v1
kind: Service
metadata:
  name: route-guide-service
  namespace: staging-as
  labels:
    app: route-guide
    component: route-server
spec:
  type: LoadBalancer
  ports:
  - port: 8980
    protocol: TCP
    targetPort: 8980
  selector:
    component: route-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
  namespace: staging-as
spec:
  replicas: 1
  selector:
    matchLabels:
      component: route-server
  template:
    metadata:
      labels:
        app: route-guide
        component: route-server
        namespace: staging-as
    spec:
      containers:
      - name: route-guide
        image: cubeiocorp/grpc-route-guide-server:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8980

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure
Expected behavior
The log messages in the envoy_on_request/envoy_on_response to be executed.
Steps to reproduce the bug
Apply the config shared and send a gRPC request.
Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)
Istio version - 1.3.8
kubectl version displays the following:
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-14T04:24:34Z", GoVersion:"go1.12.13", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.10", GitCommit:"37d169313237cb4ceb2cc4bef300f2ae3053c1a2", GitTreeState:"clean", BuildDate:"2019-08-19T10:44:49Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}
How was Istio installed?
Using docker image
docker.io/istio/proxyv2:1.3.8
Environment where bug was observed (cloud vendor, OS, etc)
AWS

It seems it is not executed because its rejected. You need to provide a valid EnvoyFilter. See https://github.com/istio/istio/wiki/EnvoyFilter-Samples#adding-a-http-lua-filter

Closing as this is working as intended and on an unsupported version: https://istio.io/latest/news/support/announcing-1.3-eol-final/. Thanks!


Sorting by CPU causes row column to expand

The sort caret gets pushed up, causing the row to expand. Just a minor visual bug, thought I might bring it up (:

This is probably the same table header issue as in #2586

Maybe a spacing issue with column width?

Yeah, probably.


Is there an official Language with protobuf?

The editor offers JS API to hook up a language. It does not dictate if the language is implemented on a web worker, on a server or even on the UI thread. The editor is not concerned with the transport protocol between the UI and the server.


Support utility VM changes in non-base layers

With this change, non-base layers can have utility VM changes. Since
Server 2016 does not support layered utility VMs, this works by cloning
the parent layer's utility VM and applying the changes directly. Layers
are assumed to be immutable, so hard links are used to make this cloning
operation fast.
@swernli PTAL

LGTM


OTP20037: Chojuk alignment preparation for experiments for Translation using LLm.

Description:
To do the experiments on translation using LLMs, I will be needing to prepare some data for the benchmarking as well as to give examples for the few-shot translation chain of thought.
Completion Criteria:
CSV with Tibetan Chojuk with alignments of translation and commentary for it.
Subtasks:

[x] Go through the already aligned chojuk data and create csv
[x] Create CSV for the Chojuk Commentary alignment.
[ ] look for other chojuk translations with alignment and create CSV

Reviewer:

[ ] @tenzin3

@tenzin3 I have send PR, kindly review it.

@tenzin3 I have send PR, kindly review it.

Merged


MM-20554 Migrate 'components/delete_post_modal' module and associated tests to TypeScript

Summary
This pull request migrates 'components/delete_post_modal' to TypeScript.
All tests pass locally, suggestions on improvement welcome.
Ticket Link
Fixes mattermost/mattermost-server#13687
JIRA https://mattermost.atlassian.net/browse/MM-20554

/update-branch


chore: pre-commit autoupdate

Codecov Report

Merging #292 (74f2b2d) into main (91f4652) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #292   +/-   ##
=======================================
  Coverage   58.81%   58.81%
=======================================
  Files          37       37
  Lines        2472     2472
=======================================
  Hits         1454     1454
  Misses       1018     1018

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 91f4652...74f2b2d. Read the comment docs.

:tada: This PR is included in version 5.5.8 :tada:
The release is available on:

v5.5.8
GitHub release

Your semantic-release bot :package::rocket:


Custom check before onNext event is fired

I, in the first step try to validate a user input by sending an AJAX request to server and retrieve the result and want to prevent user to go through next step if result contains an error.
I made my $.ajax() call right within onNext event function but wizard goes to next step very quickly after clicking next button without even waiting for $.ajax() to finish. So I'm not able to stop processing to next step on ajax error.
Is there any workarounds for this? If yes then can I have this not within onNext event and just in another place/file?
Thank you

According to https://github.com/VinceG/twitter-bootstrap-wizard/issues/41 solved. Thanks


Implement rules for The Conclave

Resolves #149

Thank you!

Thanks for merging and deploying!
I think you need to run the following to update the translation files client side (at least I did locally while developing). While the server side validation is working now on the site, the client side validation + card filtering is failing because the translation for Translator.trans('card.traits.maester') is missing.
php app/console bazinga:js-translation:dump

Nevermind, it seems like it got updated as I wrote this comment (or maybe something was aggressively cached on my end). Sorry for the hassle.
Thanks again!

Thanks!

Le 2 nov. 2017 à 20:20, Brian Upton<EMAIL_ADDRESS>a écrit :
Thanks for merging and deploying!
I think you need to run the following to update the translation files client side (at least I did locally while developing). While the server side validation is working now on the site, the client side validation + card filtering is failing because the translation for Translator.trans('card.traits.maester') is missing.
php app/console bazinga:js-translation:dump
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub https://github.com/Alsciende/thronesdb/pull/167#issuecomment-341530523, or mute the thread https://github.com/notifications/unsubscribe-auth/AAX7JqKLmcuVgful08ulXK9R1Z6WY5Lmks5syhYKgaJpZM4QFoMR.


'是因为' doesn't cut as expected

What's the best way to get jieba to cut '是因为' into '是' and '因为'?
I was processing 影子的出现是因为有光 to tag the sentence for rare words and it scored much rarer than expected because of the 是因为 token.
Cut for search on 是因为 gives ['因为', '是因为'] -- how often do the jieba cut functions duplicate the input like that? Is that by design? It was a little surprising, but maybe that's part of how that function is designed for search engines, I'm not sure.
Setting HMM to False gives ['影子', '的', '出现', '是因为', '有', '光']
Unsure if this is a bug or by design. Is the right approach here to use a custom user dictionary limited to the top 20k words or so?
Apologies if this is pure user error, I am new to jieba and still trying to figure out all the features. Thanks for any recommendations.

Since 是因为 is included in jieba dict, the default and HMM will both regard 是因为 as a conjunction (c).
jieba.del_word('是因为')

If you want to split the word, you could remove the word from jieba dict.

邮件已收到~


status mods of dirs no longer affect files

not sure if this actually works, I hope so - as the file/dir status indicators are useless for me as they are all green

actually, could be an issue with tree-view https://github.com/atom/tree-view/issues/935

@balupton this worked for me, thanks!


[Feature request] MAPS : Autozoom

Hi,
Like the "classic" maps in Kibana (and Opensearch Dashboards), would it be possible to add an autozoom function?
This way, if I only have a few points in, let's say, Europe, the map open and already fit in this geographical region and I don't have to scroll with my mouse to zoom in here.
Best regards,

Hello,
any news about this bug / enhancement?
Best regards,

Hi @vincent2mots Thanks for the request. That sounds a nice feature to have. Would you be willing to give it a try yourself?

Hi @heemin32 , I can help with the precise description / tests but I'm not sure I can help you with the technical implementation

Catch All Triage - 1 2 3 4 5 6

Hi @dblock , what do u mean by "Catch All Triage"?

Hello! How can I help to implement this feature?


⚠ SkyfallMC Minecraft Server has degraded performance

In d00ffdb, SkyfallMC Minecraft Server (play.skyfallmc.ga) experienced degraded performance:

HTTP code: 200
Response time: 1061 ms

Resolved: SkyfallMC Minecraft Server performance has improved in 04abd65.


Allow comparative chart download

Overview
Updates the comparative chart download to correctly format the comparative "out of ten" chart titles.
Connects #118
Demo

Testing Instructions

Select a combination of geographies for multiple years and geographies for one year, then select multiple years and a question of each type (stacked bar chart, grouped bar chart, and waffle / out-of-ten chart).
Download the image for each chart.
Download the CSV for each chart.
Download the CSV for the full page.
Repeat the above steps with only one year selected.
The images and CSVs should download correctly and without errors.

Taking a look now

When generating a CSV for a stacked bar chart, the results have 2021 rows even though there are no values in the visual chart. Also, the numeric output is under Gender rather than Value:

Actually, the above may be a problem in the configuration of some questions than a result of this PR. I see misquoted fields in the CSV here:

Thank you for reviewing! I have added a followup issue for the text issue here.


Waste of time following this page

It's a waste of time following this page as none of the file locations are relevant anymore and the UI screenshots no longer exist. You should either deprecate the documentation or remove it entirely, either way indicate that it is no longer relevent and will waste your time ...

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 22e20f91-d94c-419c-3a64-7616af5e7429
Version Independent ID: 0f4f0b1a-2506-ec3d-6bda-ccce7565cb64
Content: Get started with Docker - Visual Studio for Mac
Content Source: mac/docker-quickstart.md
Product: visual-studio-mac
Technology: vs-ide-general
GitHub Login: @heiligerdankgesang
Microsoft Alias: dominicn

There is a note on the page indicating that visual studio for MAc will be
done by aug 31 2024. Which pretty much wraps up the issue I raised. Cheers
On Thu, Nov 16, 2023 at 10:11 AM github-actions[bot] <
@.***> wrote:

Thanks for sharing your feedback. We apologize for our delayed response.
Due to the sheer volume of feedback in our repo, we have decided to
prioritize the newest issues and are automatically closing our oldest
issues with an Issues bot. If you believe your feedback is still
actionable, then either respond directly in this issue or open a new
documentation issue and we’ll review. If no activity occurs in the next 14
days, this issue will be closed.
—
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/visualstudio-docs/issues/9541#issuecomment-1813498387,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AATKYXUCOWJXY3LRLKE7HUTYEVK37AVCNFSM6AAAAAA3TSYN56VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJTGQ4TQMZYG4
.
You are receiving this because you authored the thread.Message ID:
@.***>


Add the ability to turn off the built-in wait for lazy loading for faster (but less stable) execution

fixed in 4.4.20200921


Configuring DHCP option type to any other value than 6 (Specifies list of DNS servers to the client) is not possible

Actual Behavior:

The default is 6,
empty field is not allowed
any other value has no impact (as I know)
it is not clear how to configure the device to act as a DNS relay

It is also not easy to know for unexperienced users to know what this option is good for.

@AndrewMargarit Handing to you, not sure if this is front or back

It's both I believe.
I'll look into it and come back.

@AndrewMargarit did you look at this ??

By default, we add option 6 to our DHCP configuration block.
list 'dhcp_option' '6,xx.xx.xx.xx,xx.xx.xx.xx'

If we remove this, the DHCP server will send the IP of the gateway and use the upstream DNS server or the one set up in the network config.


[Bug fix] fix jitwrapper check fail

Fix Issue#241

the applyFullConversion can only apply to moduleOp
remove unused void FILOMemoryPool::clear()

close this PR, as the fix is now in pr#249


adding upguard_node ansible module

Created base ansible readme framework.
Added 'upguard_node' ansible module.
Added 'play.yml' as a working example.

It seems the current method does some level of encoding auto-magically... With that said, I think I'll switch up to use the 'params' argument instead, which is probably the better direction anyhow.

a = 'some _string !with $special %chars &to *encode'
import requests
r = requests.get('http://www.python.org?name=' + a + '&other=' + a)
r.url
u'https://www.python.org/?name=some _string !with $special %25chars &to encode&other=some%20_string%20!with%20$special%20%25chars%20&to%20encode'
r = requests.get('http://www.python.org', params={'name': a, 'other': a})
r.url
u'https://www.python.org/?other=some+_string+!with+%24special+%25chars+%26to+*encode&name=some+_string+!with+%24special+%25chars+%26to+*encode'
import urllib
urllib.urlencode({'name': a, 'other': a})
'other=some+_string+%21with+%24special+%25chars+%26to+%2Aencode&name=some+_string+%21with+%24special+%25chars+%26to+%2Aencode'

Changes made and tested.


Add source maps generation to build/dev/dev:watch

This is a proposal based on this conversation: https://pre-release.mattermost.com/core/pl/ke489sx953f89jsmupittcwgrw

Changed! :)


Clarify behaviour of plusTime on invalid date(-time)s.

This fixes issue #129.
Also updated top-level README.

We can merge this one without notifying eHN (TechIOP) because it's only a clarification of the spec + some unit tests that don't even make it into the NPM package, I think.


Make env checker work for metaworld

Currently the environment checker for metaworld fails due to an missmatch between observation and observation space dtypes. That should be a relatively easy fix for the repo, but has not been solved yet. Progress
As soon as this works, we can enable the checker again in make_metaworld.

Fixed by the refactor.


text-shadow doesn't work

I'm using following default.css:
.text
{
color: #00ff33; /* green /
text-shadow: 10px 0 #ff0000; / dx dy red */
}
And what I see - text color became green, but text shadow not appeared.

addition:
Text shadow works on Android 4.4.2 (Samsung S4), but doesn't work on Android 5.1.1 (Nexus 5).

I found the problem (PXTextViewStyleAdapter.createStylers():97)
textView.setShadowLayer(0.5f, shadow.getHorizontalOffset(), shadow.getVerticalOffset(), shadow.getColor());
0.5f for radius is too small for Nexus devices.

So what value worked for you?

I add in PXShadow.java getter "getBlurDistance".
Then in PXTextViewStyleAdapter.createStylers():97 I replaced textView.setShadowLayer(shadow.getBlurDistance(), shadow.getHorizontalOffset(), shadow.getVerticalOffset(), shadow.getColor());
And in default.css I set text-shadow: 2 2 1 #000000;
It's working.
I think 1.0f is min value.

You are welcome to submit a PR and I'll take a look.


Its Possible Copy To Clipboard ?

[x ] I have read through the manual page (man fzf)
[ x] I have the latest version of fzf
[ x] I have searched through the existing issues

Info

OS

[x ] Linux
[x ] Mac OS X
[ ] Windows
[ ] Etc.

Shell

[ x] bash
[ x] zsh
[ x] fish

Problem / Steps to reproduce
I cant copy select line in fzf to my clipboard. I Just try a lot of combinations but not work. The default key CTRL+R never work. . I use this option
export FZF_DEFAULT_OPTS="
--layout=reverse
--info=inline
--height=80%
--multi
--preview-window=:hidden
--preview '([[ -f {} ]] && (bat --style=numbers --color=always {} || cat {})) || ([[ -d {} ]] && (tree -C {} | less)) || echo {} 2> /dev/null | head -200'
--color='hl:148,hl+:154,pointer:032,marker:010,bg+:237,gutter:008'
--prompt='∼ ' --pointer='▶' --marker='✓'
--bind '?:toggle-preview'
--bind 'ctrl-a:select-all'
--bind 'ctrl-y:execute-silent(echo {+} | xclip)'
--bind 'ctrl-e:execute(echo {+} | xargs -o vim)'
--bind 'ctrl-v:execute(code {+})'
"

By default, xclip copies to X primary selection, XA_PRIMARY which is not where most apps expect to find clipboard data. To copy to the clipboard, XA_CLIPBOARD, with Ctrl-Y,  use `--bind 'ctrl-y:execute-silent(echo {+} | xclip -se c)'.

@imambungo Thx for this tip but i mean here to high-lite desired text in terminal (terminator) and with key binding copy it or just ctrl + shift + c to copy it, all this by using only keyboard not mouse.

@Sy3Omda I don't know how to do that in terminator but you can use tmux to do that.

I tried several options and these worked out for me. This might help someone looking for these.

Press CTRL-Y to copy path of selected option to clipboard
Press CTRL-ALT-Y to copy contents of the selected option into clipboard'

--bind 'ctrl-y:execute(readlink -f {} | xclip -selection clipboard)'
--bind 'ctrl-alt-y:execute-silent(xclip -selection clipboard {})'

Just adding here; Thank for all this !
This is working for me with xclip --
--bind 'ctrl-y:execute(readlink -f {} | echo -n {2..} | xclip -selection clipboard)'


Indicate path stopped to caller

pathfinder#goto now throws if stopped.
The path hasn't been completed, and as such, the call should not indicate success.

did you check this doesn't break anything?
how did you test it?

I've been using this change in my bot for some time, it does not seem to break anything.


UserWarning: Failed to load image Python extension: Black Image

Windows 11 AMD 6800 xt Driver: 22.11.2

Need to use the appropriate driver


Updated start_connection_worker so there's a mutex around background thread starting.

Previously, in rare cases, it could be the case that two threads both checked running? and then both threads would create an agent background thread. This would result in two threads sharing a socket and trying to connect. Since SSL isn't threadsafe this could cause a segfault.

[ ] Investigate test-case: agent = Instrumental::Agent.new("bf5e8cbe60b436b104046a5023fa1fe0"); 10.times.map { Thread.new {agent.increment("thread_test.branch.1") }}.map(&:join) causing:
[2] pry(main)> E, [2016-06-10T12:45:44.249761 #63262] ERROR -- : Exception occurred: Bad Response "fail\n" to "authenticate bf5e8cbe60b436b104046a5023fa1fe0"

/Users/jqr/code/instrumental/ruby/lib/instrumental/agent.rb:390:in block in send_with_reply_timeout' /Users/jqr/code/instrumental/ruby/lib/instrumental/agent.rb:237:in block in with_timeout'

:+1:

Earlier today I was able to reproduce this by running a test like this ~20 times:
it "should be thread safe" do
  1000.times.map do |i|
    Thread.new do
      agent.increment("foo")
    end
  end.map(&:join)
end

Unfortunately I haven't been able to reproduce it in the past hours or so.

Seeing some odd behavior: E, [2016-05-16T22:28:39.422435 #92083] ERROR -- : Exception occurred: Bad Response "fail\n" to "authenticate YOUR KEY HERE"
When running agent = Instrumental::Agent.new("YOUR KEY HERE"); 100.times.map { Thread.new {agent.increment("thread_test.branch.1") }}.map(&:join)

Also could not reproduce on master :(

This PR is (hopefully) relevant to #56

:shipit:

@mediocretes @jqr I added https://github.com/Instrumental/instrumental_agent-ruby/pull/57/commits/f821de7d9c1efe8df1a474881de2fba730a75624 which should make it so only the locking thread blocks. It doesn't address https://github.com/Instrumental/instrumental_agent-ruby/issues/59 but it doesn't make it worse.
What do you guys think about this PR with that change?

👍

This PR is pending investigation/fix for the issue @mediocretes brought up.

Actually, digging deeper it appears I was mistaken and I am not receiving that error on this branch. Can you double check @mediocretes?

Yes, I can still produce the error on this branch essentially 100% of the time and not on master, with the following steps:
cd lib
irb
require_relative 'instrumental_agent'
agent = Instrumental::Agent.new("AN ACTUAL KEY"); 100.times.map { Thread.new {agent.increment("thread_test.branch.1") }}.map(&:join)

FWIW, the metrics do all appear to get through correctly, all my metrics have nice round multiples of 100 in them.

Hmm. I haven't been able to reproduce this still. Given we have some users already using this in production, @mediocretes should we proceed until we can get reliable reproduction of this uncommon edge case?

Sure.  I got James to run this on his machine and he has no problems in either branch.  I'll investigate later or when someone other than me hits the same issue.


Fix dot menu popover displayed under blocks

As reported on the following issue: #3309 When two blocks having the "DotMenu" are below each other, if you try to open the menu of the first block it will be shown below the second block.
The issue was solved by moving the classes that are defining the position of the element from the div that is setting its appearance to the Popover.Panel element.
Before:

After:

Awesome! Thanks @breizh24 👍


FPS Reported In Paper

Hello,
Many thanks for this excellent work! I was very impressed by your first Pseudo-LiDAR paper and now your team's idea to include a 4-beam LiDAR is really sensible .... just makes sense.
The runtime in the paper is reported at 90 ms/frame on a single GPU but the specs of the GPU are not reported.
Would you mind to share which GPU you used to obtain the 90 ms/frame reported in the paper?
You mentioned some CUDA improvements, did you run any experiments with CUDA improvements (not reported in the paper)?
I am curious what would be an acceptable frame rate for a real autonomous driving scenario?  A single GPU would get ca. 11 frames/second essentially, have you guys tried multiple GPU's?  Or anything similar not reported in the paper to keep things simple?
I re-produced the first PL paper using your AnyNet implementation for disp estimation and saw a dramatic improvement in processing speed with limited (although at times noticeable) difference.
I am looking forward to doing this same again here.  Hopefully the L# idea can really make up that difference for AnyNet and still be efficient for embedded platforms

the paper report is just GDC algorithm runs in 90 ms/frame using a single GPU.

Hi @tareeqav ,
It is good to know that you are using AnyNet on PL framework. As @Jinyong-Huang points, 90ms/frame is the GDC algorithm speed. The whole PL++ pipeline will take more than 500ms per frame. Looking forward to your amazing work.

Hello @mileyan and @tareeqav
I am trying to do a similar thing to reproduce a pipeline similar to pseudo lidar repo with Anynet. we now trained the anynet on the kitti object dataset but the resulted point cloud is not clean as either of this pseudo lidar or pseudo lidar ++
Anynet point cloud output:

Psmnet point cloud output:

as you see both are almost the same except for the noise around the car in Anynet point cloud, so do you have any idea how to improve anynet output? @tareeqav  did you make any changes to anynet model to fit in the pipeline? or do you know if there is any filtering algorithm that can help in removing this noise?


lethe compilation issue

There is a compilation issue for lethe; while making it shows that several header files and member  functions used are not found in dealii. May be the dealii version have changed these files. Is this an issue I only face.

Error logs:
fatal error: deal.II/fe/fe_simplex_p.h: No such file or directory
40 | #include <deal.II/fe/fe_simplex_p.h>
error: ‘convert_hypercube_to_simplex_mesh’ is not a member of ‘dealii::GridGenerator’
84 |           GridGenerator::convert_hypercube_to_simplex_mesh(

Hello, I'll try to reproduce the error. Which dealii version are you using?

I followed the Candi installation ; dealii 9.2
Thanks

I followed the Candi installation ; dealii 9.2
Thanks

Could you repeat the candi installation with the following branch: https://github.com/dealii/candi/tree/dealii-9.3? It seems that candi is still defaulted to version 9.2. However, you need at least 9.3 of deal.II.

You should update the required version in cmake.

Okay I will,
The error looks like it's going to be fixed by this. I will change this in Candi and let you know soon. If worked IL will close this issue.
Thanks very much.

You should update the required version in cmake.

That is true. @justAPhDStudent or @oguevremont Do you want to make the change in a PR?

I prefer anyone in the official team do the pull request.
I will make the changes in CMakeLists.txt too.
Thanks.

Yes I'll do it

Just an FYI, may be not so relevant. Lethe compiled and I could run some examples. Though the ctest utility failed; it showed tests as Failed. Anyhow I could run some examples manually, which is the major concern here. Just mentioning this for completeness.

Is this still an open issue? @oguevremont you can make a PR with the change in the deal.II required version. When Paul reworked the CMakeList he added a nice check for the dependencies, but clearly we forgot to change the most basic thing which is the require dversion.
@justAPhDStudent Could you please tell me which tests fail when you run Ctest ? Some tests are not very portable (they work on the CI and under most distros, but we they seem to fail on some of the machines we have access to, like the Compute Canada clusters). We'd be glad to improve the tests to make them more robust. :)

I just made the PR now that the CI tests have passed :)


learner.org

please add support for learner.org
eg:
http://www.learner.org/vod/vod_window.html?pid=197
thank you!

Kaltura embed. Currently working.


typed_mock library: _ArgumentMatcher_equals class should check for reference equality

Originally opened as https://github.com/dart-lang/sdk/issues/21945
This issue was originally filed by<EMAIL_ADDRESS>
The _ArgumentMatcher_equals class in the typed_mock library should check for reference equality before checking equivalence using '=='.   Currently, it only checks equivalence which means I have to implement == and hascode() on any object I want to pass as an argument to a mocked method.
If typed_mock lived on GitHub, I could just submit a PR :)

 Comment by lrhn

Added Pkg-mock, Area-Pkg, Triaged labels.


Upate custom cfme fact to allow the collection of other custom facts all under the 'cfme' name

@curt-matthews i assumed you were updating that as another pull request since you hadn't dont it with your first one :)


Generate settings/bg_targets.json for x12 models

settings/bg_targets.json is generated from settings/bg_targets_raw.json, adding the user-preferred units, and pulls no data from the pump.

How has this been working in go-explorer-hat with x12 pumps?  I haven't seen any regressions on other pump models...

It's working fine on my test rig, but I think n is still 1.

Cool.  If no objections I'll merge to dev soon to get n > 1 testing going.


GitHub Action Workflow fails due to denied permission

Setup
docker image used in workflow: skohub/skohub-vocabs-docker:latest
workflow file example: main.yml
github action: job error
:heavy_exclamation_mark: Error (job failed in stage)
name: build public dir with docker image
run: docker run [...]
> gatsby build --prefix-paths
>> `Error:EACCES: permission denied, mkdir '/app/public/static'`

:grey_question: Possibile reason

The user/uid or group permissions within the docker does not match the previous action steps (mkdir public <-> volume mount)

:grey_exclamation: Possible solutions

add (adduser/addgroup) and/or set user in docker?
chown/chmod/chgrp

:heavy_check_mark: Tested workaround

changed the folder permissions to be less restrictive in main.yml

- run: mkdir public
- run: chmod -R 777 public
- run: mkdir data
- run: chmod -R 777 data

⚠ Issue with build (probably unrelated / separate issue?)

The github pages did not include an "index.html" with a successful build using the workaround. The gh-pages branch contained only index.de.html and index.en.html. Besides the root, also the generation of concrete concept-index.html is affected, e.g. in a gh-pages: "vocabs/w3id.org/comein/vocabs/medienkompetenzrahmenNRW/" the file 52990101.de.html was generated but 52990101.html was not.

Hey Manuel,
you are totally right.
The user in the docker container is a node user with UID 1000. Therefore it is not allowed to interfere with root-created folders.
I added your suggestions to the workflow file and everything seems fine now.
Regarding the issue with the file extensions:
This is indeed they way it is supposed to work. The earlier versions, where this docker image also provided an .html without language tags was a hacky workaround. In order to align this docker image with the official skohub-vocabs image I removed this hack.
I also have to admit that I'm not totally happy with this. The reason for the language based extension is that skohub-vocabs is supposed to run with an apache webserver. The apache will automatically fetch the right language based on your browser settings.
So the suggested workaround for you would be to add .de.html to your index page when linking from it.
If this somehow is not suitable, feel free to open a ticket in the skohub-vocabs repo, so I have a reason to work on it.
Thanks!
I will close this issue for now, feel free to reopen, if the issue remains.

Thanks Steffen for clarifying the underlying issue.
I think a major drawback could be, that the id of concepts cannot be used as an URI directly (which previously was the case), so it might be an idea for another ticket to deal with that issue (e.g. htaccess-wise with defaulting to first/one language?)
Example:

https://w3id.org/comein/vocabs/DigCompEdu/3.2 leads to "You just hit a route that doesn't exist... the sadness."

I know that the concept ids main purpose is unique identification but from a users perspective some of them look like URIs where users might expect meaningful content.
I will open an issue where the relevance of that could be discussed.


fix: enable use of manylinux wheeltags once again

This boolean was filtering out the creation of manylinux wheel tags.

Can we test this in some way to prevent regressions? Maybe with Rubens new integration tests?

Sure we could test more, but this only adds more tags not less :)

I tested the installation of some PyPI packages on this PR.  There may be an upstream problem with uv.

Yes, i think adding some pypi dep to a end-to-end test that uses these tags would be nice

Thank you @liblaf!

Yes, i think adding some pypi dep to a end-to-end test that uses these tags would be nice

Added a single test, that should at least catch if we disable that boolean again :)


chatglm-6b can not use deepspeed inference[BUG]

Model * GPU size memory required for tensor parallel inference and it does not reduce latency
Are there any plans to support it?

Hi @guifeiwork , can you please provide more information about your setup and any errors you are seeing? And please provide a reproducer script.
ds_report output
Please run ds_report to give us details about your setup.
System info (please complete the following information):
OS: [e.g. Ubuntu 18.04]
GPU count and types [e.g. two machines with x8 A100s each]
(if applicable) Hugging Face Transformers/Accelerate/etc. versions
Python version
Any other relevant info about your setup


Different icons by item on MaterialRatingControl

Sample: https://mui.com/components/rating/#radio-group

We are excited to announce that we are working on a new generation of controls based on Material Design V3, which we will be releasing soon.
This ticket has been included/fixed in the upcoming version of this NuGet package, which will include both current and new generations of controls.


⚠ Text Filter API Endpoint has degraded performance

In 465fcc9, Text Filter API Endpoint (http://textfilter.roblox.com/) experienced degraded performance:

HTTP code: 200
Response time: 1198 ms

Resolved: Text Filter API Endpoint performance has improved in 049a12f.


Reverse Pixel Mapping throws Error.

I am having trouble mapping a large installation (24 fadecandys).  Mapping goes as planned, except when it comes to reverse mapping, which is key to our setup.
Using the example JSON config from the fadecandy/doc/fc_server_config.md:
{
    "listen": ["<IP_ADDRESS>", 7890],
    "verbose": true,

    "color": {
        "gamma": 2.5,
        "whitepoint": [0.98, 1.0, 1.0]
    },

    "devices": [
        {
            "type": "fadecandy",
            "serial": "XXXXXXXXXXXXXXX",
            "led": false,
            "map": [
                [ 0, 0, 0, 512 ]
            ]
        },
        {
            "type": "fadecandy",
            "serial": "XXXXXXXXXXXXXXX",
            "map": [
                [ 0, 512, 0, 64 ],
                [ 0, 576, 128, 32, "bgr" ],
                [ 0, 608, 191, -32 ]
            ]
        }
    ]
}

I'm running fcserver-osx on a mac mini with this config and two of our fadecandys.  When I attempt to connect from the strip64_flames sketch, I receive the error: Unsupported JSON mapping instrunction: [ 0, 608, 191, -32 ]

The issue is due to the fact that @j9brown's  https://github.com/scanlime/fadecandy/commit/9181cec46c4b402cfbba8c02d37f350c7857be16 commit added the functionality and documentation, but did not update the binaries.


Move existing account in mask on parent notify

When a mask is notified that a parent has set an account, it checks for an account at the same location, and prunes it if it's the same (according to Account.equal).
The underlying Merkle tree may have a different account at the same location by coincidence. In that case, the mask should move its account to a different location.
The existing Merkle tree implementations have a notion of "next available slot", which the mask could implement for this purpose.

I think this notion appears in Account_location in Merkle_ledger.Database.

Closing as stale.


v1beta4 instances.import for cloud SQL API not working

v1beta4 for cloud sql admin API- Python client I is not working and I am stumped.  The following code was deployed as a Cloud Functions using python 3.7
import base64
import logging,json
from httplib2 import Http
from googleapiclient import discovery
from googleapiclient.errors import HttpError
from oauth2client.client import GoogleCredentials

PROJECT = "test-project"
INSTANCE = "testdb-instance"
DATABASE = "testdb"

def main(data,context):

  credentials = GoogleCredentials.get_application_default()
  service = discovery.build('sqladmin', 'v1beta4', http=credentials.authorize(Http()), cache_discovery=False)
  uri = "gs://{0}/{1}".format(data['bucket'], data['name'])

  instances_import_request_body = {
      "importContext" :{
          "kind": "sql#importContext",
          "fileType": "SQL",
          "uri": uri,
          "database": DATABASE,
    }
  }

  try:
      request = service.instances().import(
         project=PROJECT,
         instance=INSTANCE,
         body=instances_import_request_body)
      response = request.execute()
      print(json.dumps(response, indent=2))
  except HttpError as err:
    logging.error("cannot import DB - Reason: {}".format(err))
  else:
    logging.info("success status: {}".format(response))

When I try to deploy i got the following error back
ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Function failed on loading user code. Error message: Code in file main.py can't be loaded.
Detailed stack trace:
Traceback (most recent call last):
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v1.py", line 315, in check_or_load_user_function
    _function_handler.load_user_function()
  File "/env/local/lib/python3.7/site-packages/google/cloud/functions/worker_v1.py", line 190, in load_user_function
    spec.loader.exec_module(main_module)
  File "<frozen importlib._bootstrap_external>", line 724, in exec_module
  File "<frozen importlib._bootstrap_external>", line 860, in get_code
  File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/user_code/main.py", line xx
    request = service.instances().import(
                                       ^
SyntaxError: invalid syntax

If I changed the above to other method such as export or list then the code would work fine with the correct body content.
Help. - I am stucked.

Hi,
The import method is named import_ instead of import since import is a reserved name in Python. See documentation.
In addition, please use google-auth. oauth2client was deprecated several years ago and is no longer actively maintained.
If you are in a cloud functions environment you can skip explicitly loading credentials by giving the default service account associated with the function the necessary permissions.
discovery.build('sqladmin', 'v1beta4') should be sufficient (you'll also need to add google-auth to your requirements.txt)
As for the error you're seeing,


BNS Model

Adds the infrastructure necessary to do training using Q-transforms, with the SingleQTransform module from `ml4gw.

Adds BNS prior based on R&P injections
Adds args for the Q-transform into the train config
Saving the best model now uses the dataloader to infer model input size
Adds ability to specify a checkpoint from which to restart training
For export, model input size is inferred from the training batch saved during training
Adds an augmentor to the BatchWhitener, which acts on the whitened data before the data is passed to the NN
Implements workaround for SV plotting for a single mass combination, and makes mass_combo an argument of the function

Closes #125
Closes #139
Closes #140
Closes #142

@EthanMarx Curious to get your thoughts here. I don't love the way that q gets handled, and more generally, how any future augmentor args would have to be passed. I'm also thinking that it would be a good idea to make a place to put different configs for different scenarios, so that they can be easily swapped out without needing to change each parameter manually. We could have a  set of base BBH configs and a set of base BNS configs that could be further modified for whatever experiment we're running.

Looking at the configuration changes, I think that multiple config idea is necessary now. Could you add a configs directory at the root of the train project? We can then have bns.yaml bbh.yaml and tune.yaml live there.

TODOs from our call:

Move qtransform module into the forward method of a pl.LightningModule so that it can be traced with the model at export time. Maybe generalize this to some augmentor = Optional[torch.nn.Module] and then it can be configured in the yaml config
Remove q parameter from luigi pipeline. This will now live in the training config.yaml
Export code no longer needs to infer input size from batch file, or include the qtransform in the BatchWhitener. I think there shouldn't be any changes to the export code.
Test out training pipeline for BBH and BNS to ensure that the model tracing during checkpointing is completely general.

@EthanMarx I was able to incorporate the augmentor into the model. It's a little less neat than I was hoping, but it works for both BNS and BBH cases. One thing I'm not loving is that during training, the log message is
  | Name      | Type           | Params
---------------------------------------------
0 | augmentor | Identity       | 0
1 | model     | Sequential     | 7.2 M
2 | metric    | TimeSlideAUROC | 0
---------------------------------------------

Rather than something more specific for the model. Not sure if there's a way to show the architecture there, rather than the model.
I'm realizing that this change kind of removes the need to have separate time and frequency domain data objects, so we may want to deprecate them in the future unless we find another need.
The only remaining issue is with exporting the Q-transform because of the FFTs. It looks like PyTorch has a beta dynamo_export that can do the conversion, but I need to play around with it to see if all the version changes will work.

Ah yeah for clarity sake, it be worth removing the torch.nn.Sequential and just calling the augmentor by hand.
Also, good point on the FrequencyDomainDataset, if it's not being used anywhere, I would say remove it, or open an issue to remove it down the line.
Is it not currently possible to export the ffts? If so, we should probably reconsider this setup

If I call the augmentor by hand, then the trace won't include it, and it won't get saved with the rest of the model.
I can't export as-is, but ONNX supports FFTs, it's just a matter of whether the converter between PyTorch and ONNX has those included. It looks like it's possible, but might require changing so much that it's not worth it. I'm going to see what it takes to get things working.

I think as long as you call the augmentor in the forward method it should get traced

I tried that, and I ran into issues loading the model from checkpoint to do the trace. But now that I'm thinking about it again, I think I know what I was doing wrong. I'll re-attempt.

Hoping this is as simple as replacing torch.onnx.export with torch.onnx.dynamo_export in hermes. It looks like dynamo_export is still in beta version, but it might be stable enough to support our few use cases.
We can add some flag dynamo boolean flag to the TorchOnnx exporter that will use whichever one is desired.

It also requires onnxscript, which, judging by the PyPI release history, is also very much in development. Again, might be stable enough for us.
It turns out that dynamo and jit are separate frameworks, so more changes would be necessary. It seems like dynamo will be what's used more going forward, so I think it'll be worth it to figure out. I'll make a separate PR for just those changes and then rebase this one.

@EthanMarx As we discussed, I've reverted back to having the Q-transform explicitly in export. Including as part of the model will be the right thing to do eventually, but that requires a possibly significant update to hermes, and that shouldn't hold up this PR.
I've removed the q parameter from the pipeline config and the tasks, so it just lives in the train and export config.yaml files. I think it's cleaner this way, even though there's nothing to enforce that the value in each project be the same.
I've tested both the bbh.yaml and the bns.yaml config files (with max_epochs = 2), and both are able to get through training and export. Let me know if you have further thoughts on this PR.

Ah okay, I just assumed that was set up already. I'm not sure how it was working if None was being passed for q, but maybe some default value was being used (or maybe I missed something in testing). Was there a reason that the config.yaml wasn't being used yet, or was it just still on the to-do list?

To use the config.yaml we would need to start calling the export cli directly from the luigi task. Currently, I just import and call the executable itself.
The main reason I did this was to avoid having to manage multiple configs that share variables, and to avoid too much confusion over which variables are being passed where. This works great when the train and export projects are entirely decoupled, but it seems that's no longer going to be the case.
I think there must be a way for the export config to import/share variables from the train config, in which case I think this problem could be solved. Let me look into this.

That would be great. I was wondering if it would ever make sense to combine the train and export projects, but being able to import the config file would be a much neater solution.

Yeah thats definitely an idea worth considering at this point

@EthanMarx I went back to having q specified in the sandbox config, and I realized that there's an OptionalFloatParameter that makes things neater, though still not perfect. I also found that things worked a little more smoothly if I had the Q-transform be instantiated in the setup function of the dataset, so I switched to doing things that way.
Let me know what you think.

Yeah that definitely makes sense for now. Looks good. Is this ready for a final once over?

Yeah, I'd say so.

@EthanMarx Looks like pushing additional commits dismissed your review, can you re-approve?


gzipPlugin: compress html files by default

@rollup/plugin-html creates index.html as asset.  It is reasonable when rollup-plugin-gz compresses also the html files by default, apart from .css and .js files.

Thanks for your PR!
Released in v2.5.0


Template idea: Design system tooling: Lit + storybook + vite

It should be easy to publish the components to npm once. And it's also possible to build a storybook as a static website, so I probably  should look into making it easy to deploy on cloudflare || gh-pages

https://leon.id/articles/web-components/2022-02-vite-lit-storybook


HBASE-22645: Backport HBASE-19893-restore_snapshot is broken in maste…

…r branch when region splits

@Apache9 This change reverts some parts of the addendum that was added to fix UT failures after HBASE-21310 & HBASE-21311.
Also, @brfrn169 can I get your review here!

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
172
Docker mode activated.

_ Prechecks _

+1
hbaseanti
0
Patch does not have any anti-patterns.

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 4 new or modified test files.

_ branch-2.1 Compile Tests _

+1
mvninstall
304
branch-2.1 passed

+1
compile
66
branch-2.1 passed

+1
checkstyle
93
branch-2.1 passed

+1
shadedjars
304
branch has no errors when building our shaded downstream artifacts.

+1
findbugs
169
branch-2.1 passed

+1
javadoc
40
branch-2.1 passed

_ Patch Compile Tests _

+1
mvninstall
294
the patch passed

+1
compile
68
the patch passed

+1
javac
68
the patch passed

+1
checkstyle
95
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
shadedjars
302
patch has no errors when building our shaded downstream artifacts.

+1
hadoopcheck
1199
Patch does not cause any errors with Hadoop 2.7.7 2.8.5 or 3.0.3 3.1.2.

+1
findbugs
171
the patch passed

+1
javadoc
40
the patch passed

_ Other Tests _

-1
unit
13750
hbase-server in the patch failed.

+1
asflicense
30
The patch does not generate ASF License warnings.

17473

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-349/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/349

JIRA Issue
HBASE-22645

Optional Tests
dupname  asflicense  javac  javadoc  unit  findbugs  shadedjars  hadoopcheck  hbaseanti  checkstyle  compile

uname
Linux af09c57dc0f3 4.4.0-145-generic #171-Ubuntu SMP Tue Mar 26 12:43:40 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/testptch/patchprocess/precommit/personality/provided.sh

git revision
branch-2.1 / 963cb45f1c

maven
version: Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z)

Default Java
1.8.0_181

findbugs
v3.1.11

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-349/1/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-349/1/testReport/

Max. process+thread count
4846 (vs. ulimit of 10000)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-349/1/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

+1 LGTM. And it looks like the failed tests in the last QA is unrelated to the patch.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
0
Docker mode activated.

-1
patch
7
https://github.com/apache/hbase/pull/349 does not apply to branch-2.1. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.

Subsystem
Report/Notes

GITHUB PR
https://github.com/apache/hbase/pull/349

JIRA Issue
HBASE-22645

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-349/2/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

The patch went in already in JIRA. Closing this PR


Miscellaneous cleanup of Events API

There were two issues with the Events API:

There were several places where we were using const primitive arguments to functions. https://clang.llvm.org/extra/clang-tidy/checks/readability-avoid-const-params-in-decls.html
There were several places where we marked a SetXXX function as const where we mutated internal data, breaking const-correctness.

These are split in two commits for easier review.

Test failures are flakes:

macOS action: PERFORMANCE_plugin_specialization
codecov dropped by 0.01%

I can't reproduce the PERFORMANCE_plugin_specialization test failure and I think it is unrelated to the changes in this PR


validator  is not called when value is specified on commandline

The validator action is not called when the parameter is specified with -d
I expect the validator to be be called regardless where the parameter value came from.

Inputs should be validated regardless how they were given
I have automated tests of template using -d to expand and just realized the validators were not tested.

Great.
I don't know if this should be left open until #962  is released.

Once fixed, we close the issues. Otherwise it's hard to know when it's time to release!
You can install from master if you need it.


Random artifacts appearing with strokeText()

I'm using strokeText() followed by a fillText() to create an effect similar to memes, however, given enough stroke width (>3), random spikes start appearing.
This happens a lot when using Impact:

However, it's also present when using NotoSans (see the spikes around the flower), and there is an inconsistent width in the r and n characters in the word "Grenade" at the top:

The fonts I tried are:

Impact-Medium, Impact-Unicode-Medium, NotoSans-Bold, NotoEmoji-Bold
NotoSans-Bold, NotoEmoji-Bold

If needed, the Impact font was downloaded from https://www.dafontfree.io/download/impact/, and the NotoSans font was downloaded a long time ago but iirc I downloaded it from https://fonts.google.com/noto/specimen/Noto+Sans.
This is being tested on Arch Linux x86-64 with Kernel 6.3.2-arch1-1.

Try adding a miterLimit
ctx.miterLimit = 2;


Starting foreground service crash (from crashlytics)

Fatal Exception: android.app.RemoteServiceException$ForegroundServiceDidNotStartInTimeException
Context.startForegroundService() did not then call Service.startForeground(): ServiceRecord{5bb0c29 u0 com.ruuvi.station/.bluetooth.BluetoothForegroundService}
and many similar crashes in logs.
Probably can be reproduced with Android 13 phone. But I can't do it with my phone for some reason.

@andreevdenis Asked feedback also from Lauri and we concluded that nobody's seen this crash occur. Moving the card to backlog, we will probably bump into it during later testing.

I was able to reproduce this a couple of times recently. I'll try to investigate this issue

Moving this to done and will follow up on Firebase logs. Apparently difficult issue to reproduce (seen only when restarting phone and launching app too fast?)