Nesting keras.Models makes tf.gradients() not update while training

Click to expand!
Issue Type
Bug
Source
binary
Tensorflow Version
2.10
Custom Code
No
OS Platform and Distribution
Linux Ubuntu 20.04
Mobile device
No response
Python version
3.8
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current Behaviour?
Nesting a keras.model  inside another one makes analytic gradients computed with tf.gradients() not update with training. On the toy example presented bellow, the inside network consists of a single multiplication with a trainable variable (i.e. Dense(1, use_bias=False)), computes derivative, and concatenas all 3 so  for input [x] output should be [x,a*x,a] (a is the traibale variable). Training this independently works as expected.
But if you nest that inside another keras.model(), even a trivial one that multiplies *2, devides /2 (i.e. no change) and then calls the inside model, the gradient computation is unchanged after training. so after you train, the output is [x,a*x,a0] where a0 is the initial weight (1.5 vs 2 in my toy example).
Computing gradients with tf.GradientTape as in here also has the exact same behaviour  https://www.tensorflow.org/guide/advanced_autodiff

My 'real world' project involved solving a differential equation using https://github.com/titu1994/tf_SIREN  and switching bewteen training inside and outside model. As gradients dont work properly this way, my workaround was to inline the code of inside model to outside, keep 2 model instances and manual copy weights from one to the other which feels very hacky and I am sure is not indeded.

Tested on tf 2.6,2.8,2.10 , both Ubuntu/Windows and the behaviour is identical everywhere.

Standalone code to reproduce the issue
import numpy as np
import tensorflow as tf
tf.random.set_seed(0)

from tensorflow.keras.optimizers import Adam,SGD
from tensorflow.keras import Input, Model,Sequential
from tensorflow.keras.initializers import Constant
from tensorflow.keras.layers import Layer, Dense
from tensorflow.keras.layers import Concatenate, concatenate

tf.compat.v1.disable_eager_execution()

#trivial network doing multiplication by a single trainable variable
def inside_network():
    x = Input(1)

    layer=Dense(1,activation='linear',use_bias=False, kernel_initializer=Constant(value=1.5))
    y=layer(x)

    dy=tf.gradients(y,[x])[0]
    #for weight a, output should be [x, ax, a]
    out= concatenate([x, y, dy],axis=-1)

    network = Model(inputs = x, outputs = out, name='inside_network')

    return network

def outside_network():
    x = Input((1))
    y=2*x
    y=y*0.5
    model=inside_network()
    y=model(y)

    outside_network = Model(inputs = x, outputs = y, name='outside_network')
    return outside_network

def get_first_weights(mod):
    for layer in mod.layers:
        w=layer.get_weights()
        if len(w)>0:
            return w

Ndata=1024
np.random.seed(0)
xdata=np.random.rand(Ndata,1)
xdata[0:4,0]=np.arange(0,4)
a=2.0

data_train_target=np.concatenate([xdata, a*xdata, a*np.ones((Ndata,1))],axis=-1)
print('input :4',xdata[:4,:])
print('target :4',data_train_target[:4,:])

model1=inside_network()

model1.compile(optimizer=Adam(1e-2),  loss='mae') #
print('dense layer weights before training',get_first_weights(model1))

model1.fit(xdata,data_train_target,verbose=1,epochs=2,shuffle=True) #,batch_size=bs

print('dense layer weights after training',get_first_weights(model1))

dataout=model1.predict(xdata)
print('dataout',dataout[0:4])
print('target',data_train_target[0:4,:])

model2=outside_network()

model2.compile(optimizer=Adam(1e-2),  loss='mae') #

print('dense layer weights before training',get_first_weights(model2))

model2.fit(xdata,data_train_target,verbose=1,epochs=2,shuffle=True)

print('dense layer weights after training',get_first_weights(model2))

dataout2=model2.predict(xdata)
print('dataout2',dataout2[0:4])
print(data_train_target[:4,:])

Relevant log output
input :4 [[0.]
 [1.]
 [2.]
 [3.]]
target :4 [[0. 0. 2.]
 [1. 2. 2.]
 [2. 4. 2.]
 [3. 6. 2.]]
2022-11-20 18:27:50.249131: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-11-20 18:27:50.295560: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-11-20 18:27:50.295870: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-11-20 18:27:50.298148: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-11-20 18:27:50.298388: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-11-20 18:27:50.298603: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-11-20 18:27:51.014096: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-11-20 18:27:51.014341: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-11-20 18:27:51.014531: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-11-20 18:27:51.014710: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8946 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:09:00.0, compute capability: 7.5
dense layer weights before training [array([[1.5]], dtype=float32)]
Train on 1024 samples
Epoch 1/2
1024/1024 [==============================] - 1s 927us/sample - loss: 0.1731
Epoch 2/2
1024/1024 [==============================] - 0s 43us/sample - loss: 0.0320
dense layer weights after training [array([[2.011962]], dtype=float32)]
/home/flogothetis/.local/lib/python3.8/site-packages/keras/engine/training_v1.py:2079: UserWarning: `Model.state_updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically.
  updates=self.state_updates,
dataout [[0.       0.       2.011962]
 [1.       2.011962 2.011962]
 [2.       4.023924 2.011962]
 [3.       6.035886 2.011962]]
target [[0. 0. 2.]
 [1. 2. 2.]
 [2. 4. 2.]
 [3. 6. 2.]]
dense layer weights before training [array([[1.5]], dtype=float32)]
Train on 1024 samples
Epoch 1/2
1024/1024 [==============================] - 0s 142us/sample - loss: 0.2240
Epoch 2/2
1024/1024 [==============================] - 0s 58us/sample - loss: 0.1774
dense layer weights after training [array([[2.0133402]], dtype=float32)]
dataout2 [[0.        0.        1.5      ]
 [1.        2.0133402 1.5      ]
 [2.        4.0266805 1.5      ]
 [3.        6.040021  1.5      ]]
[[0. 0. 2.]
 [1. 2. 2.]
 [2. 4. 2.]
 [3. 6. 2.]]

Your inside network is not connected to the outside bro, because you have 2 inputs.

tf.gradients is only valid in a graph context. In particular, it is valid in the context of a tf.function wrapper, where code is executing as a graph.

@fotlogo,
Please take a look at above comment and also xs and ys are each a Tensor or a list of tensors. grad_ys is a list of Tensor, holding the gradients received by the ys. The list must be the same length as ys.
gradients() adds ops to the graph to output the derivatives of ys with respect to xs. It returns a list of Tensor of length len(xs) where each tensor is the sum(dy/dx) for y in ys and for x in xs. Thank you!

@svobora thank for the very quick reply! I was under the impression that including layer in a keras.model()  and then calling model.compile() and model.fit() guarantees that the code will run in graph mode. and in any case I have included the 'tf.compat.v1.disable_eager_execution() '  to make sure I only run in graph mode right ?
@tilakrayal there should not be any sizing issues as the code runs fine (otherwise it will have generated a run time error) and indeed the fist part of the code training the inside_network() works exactly as expected  with the gradient being computed correctly (see the first part of my output)
the problem arises when you nest that code inside another model, and then the gradients are still computed but not being updated after the  training (final collumn of output being 1.5 were it should be 2)

I think you are making a mistake in treatment of a Network (a sequence of tf layers) and a Model (an object to handle fitting). Just use layer sequences to form networks, then build a single Model on top of the however interconnected networks.
import numpy as np
import tensorflow as tf

tf.random.set_seed(0)

from tensorflow.keras.optimizers import Adam, SGD
from tensorflow.keras import Input, Model, Sequential
from tensorflow.keras.initializers import Constant
from tensorflow.keras.layers import Layer, Dense
from tensorflow.keras.layers import Concatenate, concatenate

tf.compat.v1.disable_eager_execution()

# trivial network doing multiplication by a single trainable variable
def inside_network(x):

    layer = Dense(1, activation='linear', use_bias=False, kernel_initializer=Constant(value=1.5))
    y = layer(x)

    dy = tf.gradients(y, [x])[0]
    # for weight a, output should be [x, ax, a]
    out = concatenate([x, y, dy], axis=-1)

    return out

def outside_network():
    x = Input((1))
    y = 2 * x
    y = y * 0.5
    y = inside_network(y)

    outside_network = Model(inputs=x, outputs=y, name='outside_network')
    return outside_network

def get_first_weights(mod):
    for layer in mod.layers:
        w = layer.get_weights()
        if len(w) > 0:
            return w

Ndata = 1024
np.random.seed(0)
xdata = np.random.rand(Ndata, 1)
xdata[0:4, 0] = np.arange(0, 4)
a = 2.0

data_train_target = np.concatenate([xdata, a * xdata, a * np.ones((Ndata, 1))], axis=-1)
print('input :4', xdata[:4, :])
print('target :4', data_train_target[:4, :])

x = Input((1))
model1 = Model(x, inside_network(x), name='inside_network')

model1.compile(optimizer=Adam(1e-2), loss='mae')  #
print('dense layer weights before training', get_first_weights(model1))

model1.fit(xdata, data_train_target, verbose=1, epochs=2, shuffle=True)  # ,batch_size=bs

print('dense layer weights after training', get_first_weights(model1))

dataout = model1.predict(xdata)
print('dataout', dataout[0:4])
print('target', data_train_target[0:4, :])

model2 = outside_network()

model2.compile(optimizer=Adam(1e-2), loss='mae')  #

print('dense layer weights before training', get_first_weights(model2))

model2.fit(xdata, data_train_target, verbose=1, epochs=2, shuffle=True)

print('dense layer weights after training', get_first_weights(model2))

dataout2 = model2.predict(xdata)
print('dataout2', dataout2[0:4])
print(data_train_target[:4, :])

@fotlogo,
I was able to reproduce the issue on tensorflow v2.10, whereas on v2.11 I was facing a different issue. Kindly find the gist of it here. Thank you!

@tilakrayal Sorry for the late reply ! According to the changelog in https://github.com/tensorflow/tensorflow/releases 'old' optimisers on 2.11 are moved in tensorflow.keras.optimizers.legacy . so by changing the import as from tensorflow.keras.optimizers.legacy import Adam,SGD the exact same behaviour as in 2.10 is observed.
updated the gist here

Hi @fotlogo ,
Let me explain my understanding here. For model1 (inside_network) you have initialized weight as 1.5. Then you passed inputs = 'x' and outputs = [x,ax,a] where a is the derivative/weight.The value of 'a' before training is 1.5 here. After training model1 the 'a' value becomes 2.011962.
In second model i.e model2 (outside network) the inputs ='x' and the outputs=inside_network()(y), here inside_network() is not a trained model yet and the value of outputs is [x,ax,a]. Note here the last value 'a' is initialized as 1.5 but when passed it to second model 'a' is just a constant value only and the value 'ax' is the variable and here only the coefficient 'a' value getting trained and updated to 2.0133402 from 1.5.For model2 the last value in concat layer is just a fixed constant i.e. 1.5 which will not update.But the value a in 'ax' updated to 2.0133402 and you can see the predict values multiples of this except last column which is a constant '1.5'.
I hope this explanation will give some clarity on why the value 1.5 not getting updates in model2 .Please feel free to come back if still have questions.
Thanks!

Closing as stale. Please reopen if you'd like to work on this further. Thanks.


Some question about different implementation of SGC

Hi, I'm working on some GNNs recently. Thanks for your excellent work! I have some question about SGC, hope you can give some advice.

When I increase the arg "--degree",  suffered severe performance degradation. But, this did not happen when I use the code by pytorch geometric. What causes this difference?
I want to know if "message passing" in pytorch geometric is better than matrix multiplication in GNNs.
When can simply use the multiplication of adjacency matrix and feature matrix to achieve message passing and aggregation？

Looking forward to your reply， thanks!

Hi @Root970103 ,
Unfortunately, I am not very familiar with the pytorch geometric implementation. Can't be more helpful here. Sorry!


Vickel: Watch albertgonzalezcryptorecoverywi(?:@gmail.com)?

Vickel requests the watch of the watch_keyword albertgonzalezcryptorecoverywi(?:@gmail\.com)?. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

albertgonzalezcryptorecoverywi(?:@gmail\.com)? has been seen in 2 true positives, 0 false positives, and 0 NAAs.

Approved by Makyen in Charcoal HQ


ImportError: libcublas.so.9.0 [Tensorflow GPU with CUDA 10.0]

Hello,
As Tensorflow currently supports only Cuda 9.0 [I believe except the nightly builds] I have installed a working version by following the guide here:
https://www.pytorials.com/how-to-install-tensorflow-gpu-with-cuda-10-0-for-python-on-ubuntu/
Then after installing magenta-gpu via pip to this virtualenv, it  broke the tensorflow-gpu and had to re-configure and re-install from the .whl again. The error I got was:
libcublas.so.9.0: cannot open shared object file: No such file or directory
I'm wondering how can I fix this issue? Would "pip install magenta" work alone as tensorflow-gpu is already existent and fully functional in the virtualenv?
There are numerous topics regarding this error but none seemed relevant to my situation as the solution for most was to change the version of Tensorflow - which currently I can't to keep everything else stable.
System:
Ubuntu 18.04
Python 3.6.8
Tensorflow 1.12
CUDA 10.0
CuDNN 7.3.1
GPU RTX2080Ti
#1187 and #1407 [closed] seemed similar but due to various version differences I thought I could open a new issue.
Thanks very much.

I suspect that your tensorflow installation has maybe gotten messed up due to the different packages. The tensorflow and tensorflow-gpu packages end up installing many of the same files and if you install one after the other, it will just overwrite a lot of those files. The only difference between magenta and magenta-gpu is whether the package depends on tensorflow or tensorflow-gpu.
If you need to use a custom compiled version of TensorFlow, I'd recommend the following:

Install the magenta package.
pip uninstall tensorflow, pip uninstall tensorflow-gpu (to ensure that you've removed all tensorflow files from your virtualenv)
Install your custom built version of tensorflow.


Add state function on container to return state with network.

exemple: container.state().network['eth0']

lgtm


Support SDK deployment target ranges

In each SDK's SDKSettings.plist is information about the range of OS version the SDK can deploy to. For example, the tvOS 17.2 SDK can deploy from 12.0 ... 17.2.99.
This would be interesting information to collect and surface on the website.

References:

https://chaos.social/@ole/111845398248580914
https://mastodon.social/@easeout/111845722871919452

That would be great. Thanks for considering it.


Create staging and ceremony branches for root testing

This is an idea in conjunction with @jku
Description
Currently, we perform all repository actions on the main branch. Ceremony directories are created and then merged to a top-level repository/ folder. This makes it difficult to stage new types of configurations for new or staging/testing roots without affecting the "prod" level main branch configuration.
Proposal
In order to reach https://github.com/sigstore/root-signing/issues/416 for a staging test root (test a new configuration) and to separate out ceremony events from the main, we can do the following:

Create ceremony/DATE branches when performing new root-signing events.

[ ] Update init_repository.yml scripts to work off branches and create pull requests on new branches.

Create a staging branch where we replace keys with fake HSM keys (which I have support for), and we effectively have a mirrored repository there where we can run test operations on the roots.

Benefits

Code and workflows no longer needs to be configured on different $REPO's but instead different branch names. All repositories work off repository/.
We can stage a test root with fake HSM keys on a staging branch that can use staged configs, before we merge those configs to the main branch to use for actual ceremonies.

Open Questions
How to keep the non-repository code (workflows, scripts, binaries) in sync across the branches? Or should the branches only contain configuration and repositories?

As to not interfere with the current flow of ceremony root-signing, I can try to create the staging branch first.

it's also possible that the ceremony branch could be a single branch with a well-known name:

it makes sense that there is only one ceremony at a time
with branches all changes end up in the history of the main branch: there's no need to store specifically named branches (you could still create specific tags when merging to main to document things)
tool usage would be simpler -- no need to specify the ceremony name when the name is well known

Anyway, no need to rush this sort of change if the current ceremony process works: let's take our time.

Updates: ceremony branch support is now complete.
I have yet to set up a staging branch - which we can test here: https://github.com/sigstore/root-signing/issues/416
The nice part is that it can use the same reusable workflows, just with a different branch name, so we'll get testing from that. Work needs to be done to (1) ensure that the keys are substituted for test HSM and (2) all other metadata is synced


useless -instructions for adding snackbar to Maui Android / Windows

Type of issue
Typo
Description
there's no XML code in package.appxmanifest.  VS22 shows a graphical interface.  there's not direction on how to add support for snackbars as per the instructions
Page URL
https://learn.microsoft.com/en-us/dotnet/communitytoolkit/maui/alerts/toast?tabs=windows%2Candroid
Content source URL
https://github.com/MicrosoftDocs/CommunityToolkit/blob/main/docs/maui/alerts/toast.md
Document Version Independent Id
8536c827-5a35-ebdd-51fd-0a184bfe1d69
Article author
@VladislavAntonyuk
Metadata

ID: 8536c827-5a35-ebdd-51fd-0a184bfe1d69
Service: dotnet-communitytoolkit
Sub-service: dotnet-maui-community-toolkit

If you open the file in any editor other than Visual Studio then you will see XML. It is an XML file but Visual Studio tries to be helpful and provides a UI for editing the manifest.
You can also try right clicking on your package.appxmanifest file and choosing Open With -> Source Code Editor. Or something along those lines

Yes, thank you.  I realized that after sending the comment.
Thank you for the clarification.
On Tue, Aug 6, 2024 at 10:23 PM Shaun Lawrence @.***>
wrote:

If you open the file in any editor other than Visual Studio then you will
see XML. It is an XML file but Visual Studio tries to be helpful and
provides a UI for editing the manifest.
You can also try right clicking on your package.appxmanifest file and
choosing Open With -> Source Code Editor. Or something along those lines
—
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/CommunityToolkit/issues/449#issuecomment-2271986846,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AP2WCMCQSKEFPR3W5BQQ4YLZQEPDXAVCNFSM6AAAAABMAQXBDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZRHE4DMOBUGY
.
You are receiving this because you authored the thread.Message ID:
@.***>


[Enhancement] Supports querying a single topic and its corresponding consumer group's message request mode.

Before Creating the Enhancement Request

[X] I have confirmed that this should be classified as an enhancement rather than a bug/feature.

Summary
We didn't have a interface to get a single topic and its corresponding consumer groups' message request mode.
Motivation
Users should know what's their consumers' message request mode in some situations.
Describe the Solution You'd Like
Add a new mqadmin interface to get a topic-consumer message request mode.
Describe Alternatives You've Considered
Add a new mqadmin interface to get a topic-consumer message request mode.
Additional Context
No response

Hi, what are the specific usage scenarios?

Hi, what are the specific usage scenarios?
We want to display the consumption mode in console :)

ConsumerConnectionSubCommand should be able to meet this scenario.

@RongtongJin @lizhimins @ayanamist  Hello，plz help to review this issue.


Cannot relink proper binaries when they include assembly or complex linking invocations

@ianamason The fact that we forget about the build invocations is a problem. Usual workflow would be:

I build a binary using gllvm
I extract the bitcode from it using get-bc
I make transformations on the bitcode (making optimization, specialization, adding logging...)
I want to rebuild a binary out of it and... it fails because we don't keep track of the build invocations (in particular of fun like --start-group ... --end-group) and we did not store object files that don't have bitcode for it (because they were built from assembly)

It would be nice to store either assembly or compiled assembly in the store, along with the bitcode files, and also to keep track of build invocations into a more complex manifest file that could be read by a third tool, gllvm-relink that would rebuild the binaries.
What do you think about that?

I have discussed this with @HassenSaidiatSRI on several occasions.
I think it is something we should do. I think we should be careful to
capture as many of the use cases as we can.  I think there are examples
where the assembly is built on the fly, so we should design this with an
open mind. I think gmp is one such example. You no doubt will com across
more in your project.

Do we want two sections or just one?

A lot of invocations make use of things like linking groups. One solution would be to store a manifest in the bc store directory for each binary, that would keep track of the exact build invocation. get-bc could then extract the bitcode from the binary, and maybe generate a Makefile that can be used to relink.

In the ELF files you mean? I have no opinion about this. Maybe an external file stored into the store can do the trick.

so the store would become compulsory?
maybe keep it as simple as possible until forced to do otherwise.

Do you think that storing the exact linking command into an elf section would be okay? Some invocations are very long

Lets keep it out of the elf or mach-o until we know exactly what we are doing.
Earlier versions of wllvm
actually stored the bitcode in the elf, but I am not sure why they moved to just paths, apart from
the obvious simplicity @travitch?

@ianamason Think about static binaries like clang that are 80M stripped... I think that linking or something would fail if you were to include the bitcode directly into the elf.

Not to mention having to parse the section and separate out the modules for llvm-linking

Actually is it useful to NOT use the store @ianamason ?

So it sounds premature to settle on a where, when we don't seem to have settled on a what.
we have assembly, linking commands, what else?

Well I think it is enough. Each time you are building a binary, you check every source into the linking command: either it is an object file for which you have bitcode, or it is an object file that you have to store. Then you can keep track of the linking command.

I am confused. Are we going to keep the assembly, or just the object file it generates?
Better for analysis to have the former rather than the latter. Still Rome wasn't built in a day.
Sent from my iPad

On Jul 7, 2017, at 1:28 PM, loicgelle<EMAIL_ADDRESS>wrote:
Well I think it is enough. Each time you are building a binary, you check every source into the linking command: either it is an object file for which you have bitcode, or it is an object file that you have to store. Then you can keep track of the linking command.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I was thinking about the object because then you only have to re-link to get an executable back. If you store the assembly, then you also have to store its building invocation (although I agree it should be straightforward). Actually I don't really have a strong opinion about that.

wllvm used to store bitcode in ELF sections (which I did prefer), but it ran into trouble with some very large binaries.  The size of the bitcode (with debug information) exceeded the maximum size of ELF sections, which broke everything.  After finding that, I switched over to just storing paths in the ELF section.


confidence_threshold not accurately reflected in dice matcher

Test case:
#!/usr/bin/env ruby
require 'licensee'
Licensee.confidence_threshold = 85
p = Licensee.project(ARGV[0])
puts "#{p.license.key} #{p.matched_file.confidence}"

Running this against https://raw.githubusercontent.com/pgbouncer/pgbouncer/master/COPYRIGHT, for example, yields other. However, lowering the confidence threshold to 80 nets isc 90.41095890410958. This feels strange to me; a threshold of 90 should have even worked there, right? The highest integer threshold I'm able to use is 81.
This is not unique to that license file; it's just a good demonstration. I also saw this happening on another license that should have matched at 85% confidence, yet it refused to identify until confidence was lowered around 60 or so.

In fact, looking at those numbers again... 90% of 90 is 81 -- right where things stopped working. This makes me wonder if the default threshold of 98% is actually effectively 99%; even more strict. I'm poking around to see if this threshold is being over-calculated somewhere.

@jpeddicord sorry I never responded to you here. Did you ever track down the root cause? Looking into this myself now as well.

No worries, I got distracted from this issue myself. This is from memory of when I was debugging:
In content_helper's max_delta, the math there is set up to report a delta based on the length of a string. However, in dice matching (similarity method a few lines down), the strategy is double the length over the overlap size. This is a correct Sørensen-Dice approximation, and licensee generates correct similarities. The problem is that max_delta too aggressively filters out values.
So what's the fix? 2 characters:
@max_delta ||= (length * Licensee.inverse_confidence_threshold).to_i

becomes
@max_delta ||= (length * Licensee.inverse_confidence_threshold).to_i * 2

However, making this change leaves you with a problem:
A fair amount of unit tests and behaviors break, because licensee was tuned for this behavior. This is the hard part: you have a choice of making this fix and enabling more flexible thresholds, but then the default threshold needs tuning to make tests work again. This is where I recall stopping, as it seemed to be a more invasive change.
Slight tangent: max_delta is actually computed based on string (character) lengths, while similarity is computed based on counts of words. This will get you slightly different values for similarity and max_delta on the same strings. It's mostly only going to present itself as an issue if you get a license where someone replaced (e.g.) "owner" with "this-person-with-a-really-long-one-word-name" -- max_delta would filter this out while similarity may still match. Not critical and probably worth discussing outside this issue.

I played around with this and made #242 to explore how to address.
As far as I can tell, max_delta is serving two purposes: (1) avoid work of calculating similarity of licenses that just aren't going to be matches, and (2) filter out some licenses that are going to be matches if we don't filter them out, because they're long enough that a fair amount of text changes don't drive down similarity very much.
(2) is what is causing the unintuitive result documented in this issue, where you set a confidence level and don't get as even potential matches licenses that you know have similarity above the confidence level you've selected.
#242 avoids the necessity of (2) by scaling dissimilarity with content size. Just an idea. :smile:
Here are command line results for #242  with confidence threshold set to 90:
$ script/git-repo https://github.com/pgbouncer/pgbouncer
License: Other
Matched files: ["COPYRIGHT"]
COPYRIGHT:
  Content hash: 05d71d5a7b5a26dab27ff0f7eff7f3ae7e1a36c8
  License: Other
  Closest licenses:
    * ISC similarity: 80.40%
    * PostgreSQL similarity: 32.46%
    * MIT similarity: 27.55%

Set confidence threshold to 80, you get ISC detected.
Note ISC's similarity is 80% rather than 90% because it is being lightly penalized already by dissimilarity scaling.


multiple GPU usage

Seems there is only a review missing now. I can approve the pull request so we can merge it.

Codecov Report

Merging #195 (cd01fda) into master (90e44b2) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head cd01fda differs from pull request most recent head 6fb5671. Consider uploading reports for the commit 6fb5671 to get more accurate results

@@           Coverage Diff           @@
##           master     #195   +/-   ##
=======================================
  Coverage   95.28%   95.28%
=======================================
  Files         124      124
  Lines        4158     4158
=======================================
  Hits         3962     3962
  Misses        196      196

Flag
Coverage Δ

unittests
95.28% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


seq: different output for seq --format=d%030g 12

With our:
$ ./target/debug/coreutils seq --format=d%030g 15
d000000000000000000000000000001
[...]
d000000000000000000000000000009
d00000000000000000000000001e+01
d00000000000000000000000001e+01
d00000000000000000000000001e+01

with GNU:
/usr/bin/seq --format=d%030g 15
d000000000000000000000000000001
[...]
d000000000000000000000000000009
d000000000000000000000000000010
d000000000000000000000000000011
d000000000000000000000000000012
d000000000000000000000000000013
d000000000000000000000000000014
d000000000000000000000000000015

it causes tests/cp/link-heap to fail

Seems that it is coming from this function:
https://github.com/uutils/coreutils/blob/97cb2b2de6efa884a45af0ca255f85dcd135345e/src/uucore/src/lib/features/format/num_format.rs#L364


Logging bug

Description
Logging a project doesn't work, as it is supposed to, whenever the method Slackclient.rtm_connect() is called. logging.basicConfig() is changed back to default everytime.
What type of issue is this? (place an x in one of the [ ])

[x ] bug
[ ] enhancement (feature request)
[ ] question
[ ] documentation related
[ ] testing related
[ ] discussion

Requirements (place an x in each of the [ ])

[x ] I've read and understood the Contributing guidelines and have done my best effort to follow them.
[x ] I've read and agree to the Code of Conduct.
[x ] I've searched for any related issues and avoided creating a duplicate issue.

Bug Report
Filling out the following details about bugs will help us solve your issue sooner.
Reproducible in:
slackclient version: 1.3.0
python version: 3
OS version(s):
Steps to reproduce:

create a logger
add a FileHandler
call Slackclient.rtm_connect(with_team_state=False)
logg something

Expected result:
-logging message is put in the file by the FileHandler
Actual result:
-logging message is put in both the file and the console with default-configurations
This happens because Slackclient.rtm_connect() calls the method self.server.rtm_connect(). In class Server(), logging.debug() is called several times, which automatically changes the basic configurations for logging, if the root logger is not given a handler, which is not the case.
Attachments:

Came here to create this exact issue. Here is my repro, which is slightly different than described above, but is due to the same root cause. It makes going through logs very annoying.
import logging
import os

def create_logger():
        logger = logging.getLogger(__name__)
        logger.setLevel(logging.DEBUG)
        return logger

if __name__ == "__main__":
    token = os.environ['Slack_Token']

    logger = create_logger()
    logger.info('Connecting to Slack...')
    client = SlackClient(token)
    logger.info('Slack instantiated with token. Connecting to RTM...')
    client.rtm_connect(with_team_state=False, auto_reconnect=True)

    logger.info('Connected to RTM. This line will log twice.')

Thanks @rdunker for reporting this issue (and @mike-mo for confirming). I believe I know the root cause of this issue, I'll try and get this addressed soon.

What is the status of this issue?
Should the code above work?  If not, why not?
Would this logging connection be sync or async?

As the following sample shows, with a plain env, it should work. But I know logging package may have the issue described here with some environments (e.g., AWS Lambda).
In the case, try the following code before initializing your app. It should work for you.
import logging
root = logging.getLogger()
if root.handlers:
    for handler in root.handlers:
        root.removeHandler(handler)

Working Example
Prerequisites
$ python --version
Python 3.8.2
$ python -m venv env
$ source env/bin/activate
(env) $ pip install -U pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.1.1
(env) $ echo "slackclient==1.3.0" > requirements.txt
(env) $ pip install -r requirements.txt
Collecting slackclient==1.3.0
  Using cached slackclient-1.3.0-py2.py3-none-any.whl (18 kB)
Collecting websocket-client<1.0a0,>=0.35
  Using cached websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)
Collecting requests<3.0a0,>=2.11
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting six<2.0a0,>=1.10
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached urllib3-1.25.9-py2.py3-none-any.whl (126 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2<IP_ADDRESS>-py2.py3-none-any.whl (157 kB)
Installing collected packages: six, websocket-client, idna, chardet, urllib3, certifi, requests, slackclient
Successfully installed certifi-2<IP_ADDRESS> chardet-3.0.4 idna-2.9 requests-2.23.0 six-1.14.0 slackclient-1.3.0 urllib3-1.25.9 websocket-client-0.57.0

app.py
from slackclient import SlackClient
import logging
import os

import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

if __name__ == "__main__":
    logger.info('Connecting to Slack...')

    token = os.environ['SLACK_SDK_TEST_CLASSIC_APP_BOT_TOKEN']
    client = SlackClient(token)

    logger.info('Slack instantiated with token. Connecting to RTM...')
    client.rtm_connect()

    logger.info('Connected to RTM. This line will log twice.')

Run the script
(env) $ python app.py
INFO:__main__:Connecting to Slack...
INFO:__main__:Slack instantiated with token. Connecting to RTM...
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): slack.com:443
DEBUG:urllib3.connectionpool:https://slack.com:443 "POST /api/rtm.start HTTP/1.1" 200 None
DEBUG:root:RTM connected
INFO:__main__:Connected to RTM. This line will log twice.

I've already provided working example here. Let me close this issue now.


request FunctionTypeImpl._isInstantiated promotion to public interface

FunctionTypeImpl._isInstantiated being a public member of FunctionType would enable dartdoc to avoid fragile heuristics to guess whether a type is instantiated or not.  With dart-lang/dartdoc#1509 the heuristic is improved, but I'd prefer to get rid of it entirely.

That seems reasonable, but if it can wait I'd like to do that as part of re-working the analyzer APIs to account for Dart 2 semantics. That way we can be sure we have a simple and consistent API.

It can wait, I've found a workaround for now.


Validate options using Joi

The validation code was messy, so I've cleaned it up a little using Joi. All tests pass but the object being sent is incorrect (instead of pin.id, it's now pin.opts.id, etc.) so I'll need to add a .toJSON() method to each object.

Coverage decreased (-1.34%) to 98.66% when pulling<PHONE_NUMBER>51095c8c786ea9bb0b16d3c35a8a5c on expr:joi-all-the-things into faca07ea269664dc68c0bf71a8b5f1f07e5395ae on pebble:master.

Coverage decreased (-1.34%) to 98.66% when pulling<PHONE_NUMBER>51095c8c786ea9bb0b16d3c35a8a5c on expr:joi-all-the-things into faca07ea269664dc68c0bf71a8b5f1f07e5395ae on pebble:master.

nice! could you get code coverage back to 100%?

will do it this weekend.


Show only "tomorrow", "the day after tomorrow" etc.

Hello,
thanks for this great card. I am playing around with it and i am wondering if it's possible to show only the tomorrow date or the day after tomorrow? Maybe with card-mod?
So basically i would like to set "days: 2" but hiding the current day.
Thanks

@hAMMERlized You can set startingDay to tomorrow and days to 1. This will only show tomorrow. There is no option to start with the day after tomorrow though;.


Impossible to get the first day of any 2016 week with other locale than english

Description of the Issue and Steps to Reproduce:
console.log('2016 - 20 === ', moment().locale('fr').year(2016).week(20).startOf('week').format('dddd Do MMMM YYYY')); // 2016 - 20 ===  lundi 11 mai 2015
console.log('2016 - 20 === ', moment().locale('en').year(2016).week(20).startOf('week').format('dddd Do MMMM YYYY')); // 2016 - 20 ===  Sunday 8th May 2016
console.log('2015 - 20 === ', moment().locale('fr').year(2015).week(20).startOf('week').format('dddd Do MMMM YYYY')); // 2016 - 20 ===  lundi 11 mai 2015

I'm expecting the first line to give a date in 2016 (like the second one), not 2015.
I don't know if I am using the library the wrong way or if it is a bug, but I have the same issue with every week, it looks like the 2016 year does not exist.
I also reproduced the issue with other locales like it or de.
Environment:
Chrome 55 on Ubuntu 16.0.4 64 bits. I run my tests using the javascript console on my browser on https://momentjs.com
If you are reporting an issue, please run the following code in the environment you are using and include the output:
Mon Jan 02 2017 16:34:31 GMT+0100 (CET)
02/01/2017 à 16:34:31
-60
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
2.17.1

Same error here, same output

Same problems here. It's setting of the week that seems to mess it up. Using nl locale:
var date = moment().year(2016).week(50);
console.log(date);  // moment("2015-12-12T20:30:08.380")

Without setting the week:
var date = moment().year(2016);
console.log(date);  // moment("2016-01-02T20:31:26.205")

moment().locale('en', null).millisecond(0).second(0).minute(0).hour(0).year(2016).isoWeek(48).isoWeekday(1);
Gives me:
Mon Nov 23 2015 00:00:00 GMT-0800 (PST)

Same problem here:
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/moment.js/2.17.1/moment.min.js"></script>
</head>
<body>
    <script>
        console.log(moment().year(2015).isoWeek(40).format('YYYY MM DD'));
        console.log(moment().year(2016).isoWeek(40).format('YYYY MM DD'));
        console.log(moment().year(2017).isoWeek(40).format('YYYY MM DD'));
    </script>
</body>
</html>

Gives me:
2015 10 03
2015 10 04
2017 10 03

This is pretty urgent for us :cry:

This is because you are changing year and not date, so
var m = moment();  //creates current date object
m.year(2016); //sets year to 2016 but date is still January 3rd
m.isoWeek(44) // subtracts 63 days (current week (53 - 44)*7 = 63) which means it is now 2015

The solution would be to do
var m = moment().year(2016).endOf("year");
console.log(m.isoWeek(44).year());

or for momentjs to calculate the date of 44 isoweek for provided year instead of subtracting dats

So it is a missuses of the library, thanks. But do you understand why the dates seems to be correct if the locale is english?

No, actually I think it is bug in the library, trying figure out how to fix it :)

I am not familiar with internals, but if I am understanding it correctly, then maybe calculating isoweek would be better done by setting day to firstWeekOffset() + input*7
Something like
function getSetISOWeek (input) {
    if(input == null){
        return weekOfYear(this, 1, 4).week;
    }
    else{
        set$1(this, 'Date', firstWeekOffset(this.year(), 1, 4)+input*7);
        return this;
    }
}

I'm encoutering the same problem too. Your workaround did the trick though :
let failing = moment.utc().year(2016).week(1).weekday(0).startOf('day');
console.log(failing.format('YYYY-MM-DD')); // 2014-12-29

let working = moment.utc().endOf('year');
working.year(2016).week(1).weekday(0).startOf('day');
console.log(working.format('YYYY-MM-DD')); // 2016-01-04

It will do the trick to the year, but will fail for the date, I think correct implementation should be accounting for year change when calculating isoweek
function getSetISOWeek (input) {
    var woy = weekOfYear(this, 1, 4);
    return input == null ? woy.week : this.add((input - woy.week + (this.year()-woy.year)*52) * 7, 'd');
}

tldr use the following code to get the 1st week of 2016 according to ISO 8601 correctly:
moment().isoWeekYear(2016).isoWeek(1)

If you always use isoWeekYear() to set a year in which you want to get a isoWeek, and use year() when you want to set a year in a date, the issues stated above do not occur.
For the mentioned locales, the week system is ISO 8601, and therefore moment's week() function is (effectively) the same as the isoWeek() function. All of the following is therefore according to ISO 8601. North-american locales should not have this issue, but some others with weeks that overlap the year boundary may.
Week 01 of a given year is the week in which four or more days are in that year. The year that the ISO week corresponds to is called the week-year, which you can get and set with the isoWeekYear() function in moment. Therefore, the first three days of a year can be in a previous week-year and reversely, the last three days of a year may be in the next week-year. This may be hard to follow, but it is by design, the wiki article gives more explanation.
Where this gets confusing is that isoWeek(), and week() for certain locales, sets the week in the same week-year. So the following is true:
date = moment({year:2016, month:0, day:3})
console.log(date.format("YYYY-MM-DD")) //2016-01-03
console.log(date.isoWeekYear()) //2015

And any call to isoWeek(), or week() for certain locales, will set that week in 2015. I don't know whether this is by design or a bug, since the documentation is unclear on what year will be used for calls to week(). If this behavior is by design, it may be useful to mention this more clearly in the documentation.
Please note that using endOf('year') is not a good solution, because the last day of the year may be in week 01 of the next year, causing the same issue:
console.log(moment({year:2018}).endOf('year').isoWeekYear()) //2019

Couldn't have said better @kiliangrashoff -- that's absolutely right.

yes, but it does not make sense for end user to use isoWeekYear if he is using weeks and locale happens to be iso standard. How would I know it is iso standard?


Generating sunburst

Hello
I'm a bioinformatician and I like your interactive sunburst to explore reactome enrichment analysis. I have a lot of reactomeGSA results to analyse (10 comparisons) and I try to use your script (generate_sunburst.py) to draw interactive sunbursts but I failed, I have an error "G is not a tree" with my data.
My data is slightly different than yours and I think if I had your input file, I can understand how data is transformed by the script. And I hope I will debug my issue.
Can you send me your input files (metabric_path_survival.p and metabric_path_activities.p) ? Or example files ?
Thank you
Aurelie

Gustavo, would you be able to share the specified files?
On Fri, Jun 24, 2022 at 3:25 PM lireo @.***> wrote:

Hello
I'm a bioinformatician and I like your interactive sunburst to explore
reactome enrichment analysis. I have a lot of reactomeGSA results to
analyse (10 comparisons) and I try to use your script
(generate_sunburst.py) to draw interactive sunbursts but I failed, I have
an error "G is not a tree" with my data.
My data is slightly different than yours and I think if I had your input
file, I can understand how data is transformed by the script. And I hope I
will debug my issue.
Can you send me your input files (metabric_path_survival.p and
metabric_path_activities.p) ? Or example files ?
Thank you
Aurelie
—
Reply to this email directly, view it on GitHub
https://github.com/statisticalbiotechnology/metabric-pathway-survival/issues/1,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAXKAFD4VDKBG2SY6RUHBTVQWZMRANCNFSM5ZX3EDHA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***
com>

Hi Aurelie,
The function you want to use in generarate_sunburst.py is:
sunburst(in_df, outname = 'sun_tree.json')

It takes as input a pandas DataFrame in_df, it has as indexes the pathway reactome ID, and the columns 'value' as the colors to be plotted, 'ngenes' as the width (number of genes in the pathway), and 'descr' as the description of the pathway.
Be sure to have the ReactomePathwaysRelation.txt file (you can download it from the Reactome website) on the .data/ path.
It will output a json file on the path given to the outname parameter. This json file then has to be read by the d3.js implementation on the doc/ directory. The easiest way to do this is to replace the result.json file in that directory. Since d3.js is javascript, simply opening the doc/result.html on a browser probably won't work, you have to set up a local web server. The easiest way to do this is by
python3 -m http.server

You can find more info about this here
Let me know if this works for you.

I reload the new script with value instead of q but nothing changed.
My reactomeGSA result are quite old (last year) so I rerun it to use same version of reactome in my analysis and ReactomePathwaysRelation.txt (I download it today).
I have again the same key error but later (1985st key instead of 19th) :
Traceback (most recent call last):   File "/Users/aurelie/Documents/projets/host-RNAs_evs/analyse_aurelie/enrichment/reactomegsa/generate_sunburst.py", line 109, in <module>     sunburst(in_df, outname='sunburst/results.json')   File "/Users/aurelie/Documents/projets/host-RNAs_evs/analyse_aurelie/enrichment/reactomegsa/generate_sunburst.py", line 81, in sunburst     thirdDict['value'].update({key : topDict['value'][value]}) KeyError: 'R-HSA-162909'
This key is included in ReactomePathwaysRelation.txt but not in my input file. Previously I tried to ignore these keys by adding if value in topDict['value'] before the update but after I have the error 'G is not a tree'.
I think I'm improving but I miss something with dictionaries construction.
Thanks

Hi again Aurelie,
The "G is not a tree" error might be due to the fact that by excluding a node from graph G at random, it might not be a tree anymore. This would only work if the excluded node is one of the end leafs.
Since the hierarchy itself is a tree, we need a full tree in order to display the results correctly.
Would using a placeholder value for the missing results work for you?

Hi,
I think I found the problem. In my previous run, a lot of paths were missing (all non significative).
And when I rerun reactomeGSA, I use include_disease_pathways = FALSEbut I don't kow why, one disease path was always present (R-HSA-164952). Consequently the disease part of the tree is full of holes.
Right now, I rerun reactomeGSA with include_disease_pathways = TRUE and hope I will finally get my sunburst after.
Thanks for your help.
Have a great week-end

Hi,
Finally it works ! As I thought it was a problem with disease pathways. If diseases are included, some are missing, even with a new updated reactomegsa analysis. And if diseases are excluded, a pathway stay in the analysis.
So, I run without diseases and manually remove the pathway stuck (R-HSA-164952).
With your explanations about a local web server and the html file, I managed to generate my sunbursts.
Thank you


Generic data prims reth-chain-state

Describe the feature
- [ ] https://github.com/paradigmxyz/reth/issues/12665
- [ ] https://github.com/paradigmxyz/reth/issues/12817

Additional context
No response

hey @0xshikhar , seems like all subtasks here are assigned for now, thanks anyway :)

okk, no worries

https://github.com/paradigmxyz/reth/issues/13239 is out of scope for now


偶尔出现崩溃，会出现发不了请求

Your description here
Expected outcome
What should happen?
偶尔出现崩溃，会出现发不了请求
Actual outcome
What happened?
Versions
App: <IP_ADDRESS>8-stable
Chrome: 54.0.2840.99
Platform: Windows (10)
Steps to reproduce

open
input body
send

Hi,
Can you be more specific on this? What is happening exactly? Is there any error in logs (they are in the left navigation)? It is happening always or for specific request?
I'll be glad for more details :)

Error during fetch. Request aborted
Error: Request aborted
at SocketFetch.abort (chrome-extension://hgmloofddffdnphfgcellkdfbfbjeloo/elements/elements.js:34196:30)
at _timeout.timeoutId.window.setTimeout (chrome-extension://hgmloofddffdnphfgcellkdfbfbjeloo/elements/elements.js:34319:14)

Request aborted error happens then you actually abort the request using the abort button. Are you clicking the send button twice? It is instantly replaced by the abort button and if you clicking it twice you are aborting the request.
I can't see any other explanation since this is the only way to cause this error.
Also I'm not able to reproduce this.
I'm closing it as Work As Intended but feel free to reopen this issue if needed.


gfx12 - since compiler has wrong result with s_barrier, replace by s_…

…barrier_wait to check arch capability

locally verify on navi4 and gfx90a


ci: move to GitHub actions and Codecov

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@aac52bd). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff            @@
##             main       #17   +/-   ##
========================================
  Coverage        ?   100.00%
========================================
  Files           ?         2
  Lines           ?       305
  Branches        ?         0
========================================
  Hits            ?       305
  Misses          ?         0
  Partials        ?         0

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


"webtoons.com" extension

Source name
webtoon
Source link
https://www.webtoons.com/en/
Source language
English
Other details
I already made this extension and use it in my pc, but am not sure if I should pull request or not. Also, I don't know how to pull request.
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a title with source name.
[X] I have checked that the extension does not already exist by searching the GitHub repository and verified it does not appear in the code base.

How to make a pull request:

Fork the repo
clone your forked repo
add your Webtoons extension in the correct folder of your repo
git commit
git push
(Or instead of cloning, adding, committing, pushing, you can upload your extension directly in GitHub website)
Go to your repo's main page
Click on "Contribute"
Click on "Open pull request"

thanks for the help. I'm closing this issue now


Request for Assistance with MIPROv2 Optimization Issue

Dear Sirs,
I hope this message finds you well. I am reaching out to seek your guidance regarding an issue I am encountering with the MIPROv2 optimizer in DSPy.
When running the optimizer, the computation consistently gets stuck during the first trial and never progresses to completion. The compute process continues indefinitely without any indication of errors or progress. Despite using a minimal configuration to reduce the computational load, the problem persists. Another optimizer works quite well with this data.
Here are the key settings and context of the issue:
Datasets are not very large. 20 text files (40-70k each)
Parameters are as follows:
print("This is before teleprompt import MIPROv2")
from dspy.teleprompt import MIPROv2
metric = nonlinear_score
RCR_student_program = RCR_Classifier_Pipeline(prompt_model=lm)
print("This is before teleprompter = MIPROv2(")
teleprompter = MIPROv2(
prompt_model=lm,
task_model=RCR_student_program,
metric=metric_with_assertions_retry_5,
verbose=True,
auto="light",
max_bootstrapped_demos=2,  # Set a limit on bootstrapped demos
max_labeled_demos=2,  # Set a limit on labeled demos
num_candidates=3,  # Limit the number of candidates
num_threads=16,
metric_threshold=0.80,  # Set a metric threshold
)
print("This is after teleprompter = MIPROv2 before telepromter.compile")
final_optimized_program = teleprompter.compile(
student=RCR_student_program,
trainset=trainset,
valset=valset,
minibatch=True,  # Enable minibatch processing
minibatch_size=25,  # Set minibatch size
num_trials=1,  # Set the number of trials
minibatch_full_eval_steps=10  # Set the number of full evaluation steps
)
print("This is after teleprompter = MIPROv2 before telepromter.compile")
Models:Both prompt_model=gpt4ominiptu and task_model=mrtr-gpt-4-32k are lightweight to ensure computational efficiency. DSPy version is 2.5.43.
Environment: Azure Databricks spark cluster. 64 GB Memory, 16 Cores, no GPU. Another optimizer works with this config quite well. Everything gets stuck with teleprompter.complile.
Stderr file contains one warning message about ANTLR parser (reapets 3-4 times): ANTLR Tool version 4.8 used for code generation does not match the current runtime version 4.9.3
I was trying to get to the root cause of the problem, but the documentation for MIPRO is not up to date. I would greatly appreciate your help in diagnosing and resolving this problem. If additional information or logs would help, please let me know, and I would be happy to provide them.
Thank you for your time and support.
Best regards,
Igor Chizhov
<EMAIL_ADDRESS>

Are you using Ollama? Make sure your ollama run can handle multiple threads at once. There's a config for that in Ollama.
Alternatively, just replace num_threads=16 with num_threads=1.

Thank you for getting back to me. No, I am not using ollama. However, I will run it with num_threads=1.

Could you please advise how to debug MIPROv2? How to see where is it, what is it doing, and how to monitor its performance. At present, it is a black box for me.

You can set verbose=True
Or just paste the stage where it gets stuck here (MIPRO prints a lot of logs) and we can help

In stderr file of Azure Databricks, I am getting a message about ANTLR: appcds_setup elapsed time: 0.000
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
ANTLR Tool version 4.8 used for code generation does not match the current runtime version 4.9.3
ANTLR Tool version 4.8 used for code generation does not match the current runtime version 4.9.3
ANTLR Tool version 4.8 used for code generation does not match the current runtime version 4.9.3
ANTLR Tool version 4.8 used for code generation does not match the current runtime version 4.9.3
Sun Dec 29 22:59:23 2024 Connection to spark from PID  1779
Sun Dec 29 22:59:23 2024 Initialized gateway on port 40267
Sun Dec 29 22:59:24 2024 Connected to spark.
I tried to install  ANTLR locally on the cluster: pip install --force-reinstall "antlr4-python3-runtime==4.9.3". However, I am still getting the same message about ANTLR.
How critical is this for using MIPROv2?

Verbose was True from day one. The problem is that I do not see any logs except those posted in the previous comment. When MIPRO starts the first trial (num_trials=1), everything gets stuck. The logs are empty. There is nothing there. What else should I enable to get more visibility? Please advise.

Hi @GaryChizm , could you share the stack trace and printed console messages you see when running your program with MIPRO?
The ANTLR error you are getting is most likely unrelated to the MIPRO execution and potentially on the cluster end and not from the DSPy library, but it would be helpful to confirm if you can share the execution outputs from your end.
Feel free to blur / remove anything that can't be shared publicly!

Here is the cell that  has been running over 2 hours with practically no CPU activity on the cluster:
import os
import logging
import time
import numpy as np  # For dummy computation
print("This is before Configure Logging")
Step 1: Reconfigure existing loggers to capture all events
for handler in logging.root.handlers[:]:
logging.root.removeHandler(handler)
Force File Creation
log_file_path = os.path.join(os.getcwd(), "execution_logs.log")
try:
with open(log_file_path, "w") as log_file:
log_file.write("Log file initialized.\n")
print(f"Log file successfully created: {log_file_path}")
except Exception as e:
print(f"Error creating log file: {e}")
raise
Configure logging
logging.basicConfig(
filename=log_file_path,
level=logging.DEBUG,  # Capture all logs for debugging
format="%(asctime)s - %(levelname)s - %(message)s"
)
logging.debug("Reconfigured logging to capture all events.")
logging.info("Script execution started.")
Step 2: Granular logging for troubleshooting
try:
logging.debug("Before importing MIPROv2.")
from dspy.teleprompt import MIPROv2
logging.info("MIPROv2 imported successfully.")
except Exception as e:
logging.error(f"Error importing MIPROv2: {e}")
raise
print("This is before Initialize metric and student program")
try:
logging.debug("Before initializing metric.")
metric = nonlinear_score
logging.info("Metric initialized.")
logging.debug("Before creating RCR_Classifier_Pipeline.")
RCR_student_program = RCR_Classifier_Pipeline(prompt_model=lm)
logging.info("RCR_Classifier_Pipeline created.")

except Exception as e:
logging.error(f"Error initializing RCR_student_program or metric: {e}")
raise
print("This is before Initialize MIPROv2")
Step 3: Investigate resource bottlenecks
try:
logging.debug("Checking trainset and valset sizes.")
logging.debug(f"Trainset size: {len(trainset)}")
logging.debug(f"Valset size: {len(valset)}")
except Exception as e:
logging.error(f"Error accessing trainset or valset: {e}")
raise
Step 4: Monitor External Dependencies
try:
logging.debug("Validating external dependency: prompt_model (lm).")
# Test the accessibility and response of the external API/resource
if hasattr(lm, "ping"):  # Assuming the resource has a ping method
response = lm.ping()
if response != "pong":  # Replace with actual expected response
raise ValueError("Unexpected response from prompt_model.")
logging.info("Prompt model is accessible and responsive.")
else:
logging.warning("No 'ping' method available; skipping accessibility check.")
except Exception as e:
logging.error(f"Error validating prompt_model: {e}")
raise
Initialize MIPROv2
try:
logging.debug("Before initializing teleprompter.")
teleprompter = MIPROv2(
prompt_model=lm,
task_model=RCR_student_program,
metric=metric_with_assertions_retry_5,
verbose=True,
auto="light",
max_bootstrapped_demos=2,  # Set a limit on bootstrapped demos
max_labeled_demos=2,       # Set a limit on labeled demos
num_candidates=3,          # Limit the number of candidates
num_threads=16,
metric_threshold=0.80      # Set a metric threshold
)
logging.info("Teleprompter initialized successfully.")
except Exception as e:
logging.error(f"Error initializing teleprompter: {e}")
raise
print("This is after Initialize MIPROv2 before final optimized program")
Compile the final optimized program
try:
logging.info("Before compiling the teleprompter.")
final_optimized_program = teleprompter.compile(
student=RCR_student_program,
trainset=trainset,
valset=valset,
minibatch=True,            # Enable minibatch processing
minibatch_size=25,         # Set minibatch size
num_trials=1,              # Set the number of trials
minibatch_full_eval_steps=10  # Set the number of full evaluation steps
)
logging.info("Teleprompter compiled successfully.")
except Exception as e:
logging.error(f"Error compiling the teleprompter: {e}")
raise
Step 5: Test CPU utilization with dummy computation
try:
logging.debug("Testing CPU utilization with dummy computation.")
start_time = time.time()
np.dot(np.random.random((1000, 1000)), np.random.random((1000, 1000)))
logging.debug(f"Dummy computation completed in {time.time() - start_time} seconds.")
except Exception as e:
logging.error(f"Error during dummy computation: {e}")
raise
print("This is before Final logging statements")
Final logging statements
logging.info("Execution completed successfully.")
logging.info("DSPy version: 2.5.43")
logging.info("Prompt Model: gpt4ominiptu")
logging.info("Task Model: mrtr-gpt-4-32k")
I managed to create a log file: execution_logs.log, but there is nothing there except one comment #Log file initialized. Stderr file on Azure Databricks contains 0 bytes.
The output of this cell looks like this:
This is before Configure Logging 2024/12/31 22:22:55 INFO dspy.teleprompt.mipro_optimizer_v2: RUNNING WITH THE FOLLOWING LIGHT AUTO RUN SETTINGS: num_trials: 7 minibatch: False num_candidates: 1 valset size: 47 Log file successfully created<EMAIL_ADDRESS>This is before Initialize metric and student program This is before Initialize MIPROv2 This is after Initialize MIPROv2 before final optimized program
It is running, but it seems like it does not compute anything. CPU near 0.
I use gpt models: prompt_model=gpt4ominiptu and task_model=mrtr-gpt-4-32k are lightweight to ensure computational efficiency. I can reach them both in 2 separate cells:
lm("I am language model. How are you doing?") #lm is gpt4ominiptu
output: ["I'm just a program, so I don't have feelings, but I'm here and ready to help you! How can I assist you today?"]
gpt_4("I am language model. How are you doing?") # gpt_4 is mrtr-gpt-4-32k
output: ["As an artificial intelligence, I don't have feelings, but I'm here and ready to assist you. How can I help you today?"]
So, I can reach both models. API works. However, MIPRO got stuck on the first trial.
Do not know what to think.


More Languages? ReadME?

How about if we could add more languages to the asset? and update the readme with the current languages? : )

@danishsshaikh Do you mean the documentation language ?

@danishsshaikh Do you mean the documentation language ?
Yeah, well i meant to ask is could we add more languages to the asset? Like I don't think if it supports Hindi or Marathi (Indian Languages)
And, also update the documentation with how many current languages does the asset supports currently.

Thanks,
Dan

@danishsshaikh This system does the localization management only; it does not translates one language to another. It simply includes languages from Unity's SystemLanguage enum. In the future, I might implement adding custom languages as well as built-in ones.
As you see, there are lots of languages in SystemLanguage enum. Updating documentation for all languages is impossible at least for me :)

It all makes sense.
Thanks for replying back :)


Current Goals

Now that I believe I have the copilot plugins in a stable state, here are the goals I have planned for the future:

Customization of result frequency: Currently, completions are requested by cmp which is debounced. I would like to add an option for constant requests for results via the uv loop api, similar to how copilot.vim gets their completions. Depending on the frequency you set, you may have dramatic increases in CPU usage and editor latency, but I do think this is worth implementing.

Extend the api, make it easier to use: This one is pretty self explanatory, hooking into copilot and adding your own functionality and handlers should be made much simpler. By virtue of how cmp works, it was easier to just make the buf requests there to get a functional product, but this will need to be implemented as part of goal (1). Development of this will likely precede or occur alongside goal (1)

Fix copilot adding extraneous braces, brackets, parenthesis, etc: If characters like closing braces or parenthesis exist, copilot will insert additional ones, which kinda sucks. This isn't going to be easy and is an issue in copilot.vim as well, but if it's done well, it would be a major step forward. This one is done!!!

Creation of hosts/terms json files should be implemented as copilot.vim does

???: Provide your input! What else would you like to see? Please feel free to comment and discuss here, or create an issue with the enhancement/feature flags.

Hi Zack,
Thanks for your work on that plugin.
I would like to see the following things:

Async startup of copilot(is it possible?)
Specifying on which filetypes to start the plugin
Configurable line length

Something like https://github.com/tzachar/cmp-tabnine#setup would be nice

Hi @sbulav, thank you for the input, just a couple of comments:

Async startup of copilot(is it possible?)

I'm not certain of what you mean by this. The server itself is a rpc, so neovim just sends out the command to start it. If you want to have it start automatically rather than InsertEnter, and still not affect startup time, using vim.defer_fn in your packer use to wrap the config should work.  It may just take some time playing with the value to get the correct behavior. See the README.md for more details on this.

Specifying on which filetypes to start the plugin

This is a great idea and something that should be released soon.

Configurable line length

I'm not sure what you mean by this. If here, you mean ask for completions of a certain length, that's not supported by copilot to my knowledge. The copilot discussion forum has also been complaining over the last couple days about general copilot performance regarding its ability to generate full functions/only getting one line, so it's not just the plugins I wrote...
We are somewhat at the mercy of copilot's api here. I did some testing for https://github.com/zbirenbaum/copilot-cmp/issues/5 and it turns out that while I can force a request for completions in such cases, the server returns no results.
I have requested an openai codex key, and whenever I get off that waitlist, I'll use it to make my own minimal lsp for completions.

@sbulav
Checks have been written in to make sure copilot doesn't start in buffers like help or packer. ft_disable has been added to the params, so any languages you don't want to use copilot for can be input there. I think a blacklist makes more sense than a whitelist given the nature of copilot.

Hi @zbirenbaum
Thanks for your in-depth explanation!
I had significant input lag with vim.schedule on one of the previous copilot.lua versions, but with the latest version, the lag is gone.
As for the line length, I mean that sometimes completion, provided by the copilot, gets ridiculously long. Check out this example:

I'd like to have an option to limit line length to, let's say, 80 symbols.

Hi @zbirenbaum Thanks for your in-depth explanation! I had significant input lag with vim.schedule on one of the previous copilot.lua versions, but with the latest version, the lag is gone.
As for the line length, I mean that sometimes completion, provided by the copilot, gets ridiculously long. Check out this example:
I'd like to have an option to limit line length to, let's say, 80 symbols.

Glad to hear the lag issue is fixed!
I noticed that too, but I have cmp's max len limited so it never got that bad lol. I pushed an update to copilot-cmp just now, which caps it at 30 chars or abbreviates it with the first 20 + '...' + last 10 for the label.
I'd like the user to be able to customize it, but integrating setup options has been an ongoing debate with myself regarding how to design them best. For now, I hope the change helps.

Quick update on this. The other day I actually managed to get copilot to get a completion on a newline, but cmp refused to show it, stating something along the lines of 'no prior character to complete' even when I try to manually trigger the menu. I think that this functionality is just incompatible with the way cmp is written. Unfortunately this is just an upstream problem I don't really have control over.
I am debating attempting to make a somewhat sizable PR to cmp which would fix this, but due to the scope of what is involved it would likely sit in the development branch for ages if it gets merged at all. I'll try digging into it at some point and see if there isn't an easier way, but I really wouldn't get your hopes up.
I am working on a few mechanical things that would make this type of feature work more smoothly which will be part of a rather sizable update coming soon:tm: but I think that for the time being getting this to integrate with cmp like the other results do is simply unrealistic. I'll continue to leave this open so that I can address it with the solution once said update is out, and I will probably put it in the docs as well.
As for the example implementations, I am happy to take input. Currently I was thinking that I would create a preview window triggerable by some command with selections and code previews.

Thank you for great work you did with the plugins.
Here are a couple of things that would be good to have:

Some way to tell if there is an ongoing request to copilot servers. Being able to distinguish between "loading" state and "0 results" state would be a huge productivity boost for anyone using this plugin.
An alternative to original plugin's Copilot status. I have been using the tpope's version for a few month before I switched to yours and I really miss the ability to check the status of the server after not getting any suggestions for a few minutes.

@dklymenk :Copilot status should now work as expected.


MAINT: responses to first batch of code-review comments

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

81 of 81 (100.0%)  changed or added relevant lines in 1 file are covered.
44 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.08%) to 91.758%

Files with Coverage Reduction
New Missed Lines
%

src/cogent3/core/location.py
44
94.01%

Totals

Change from base Build<PHONE_NUMBER>:
0.08%

Covered Lines:
30070

Relevant Lines:
32771

💛  - Coveralls


How to get in touch about a security issue?

Hey there,
As there isn't a SECURITY.md with an email on your repository, I am unsure how to contact you regarding a potential security issue.
Would you kindly add a SECURITY.md file with an e-mail to your repository? GitHub recommends this as the best way to ensure security issues are responsibly disclosed, and it would massively help security researchers get in touch next time.
Thank you so much and I look forward to hearing from you!

Hi @zidingz, thank you for raising this.
You can report to<EMAIL_ADDRESS>or via Hacker one.
Keeping this issue open until we add a SECURITY.md.

Thank you for your quick response!
An email will be with you shortly.

Closing per https://github.com/netlify/netlify-cms/security/policy.
We've added an organization wide policy to our repos. Thank you for raising this!


useStdError=true can bring false positive test results

The problem
I am not quite sure that this is fixable on the reporter side, but I feel I need to create the ticket just to bring an attention to the others.
Teamcity has a known issue of output stream synchronizations
Which means the order of messages across stdOut and stdErr is not guaranteed.
It also appears that teamcity cares about the message order in regards to test reports (at least it cares in 2020.1)
With useStdError=true the reporter will throw 'testFailure' into stdErr, but 'testFinished' into stdOut. if you are unlucky, 'testFinished' will arrive earlier and the test will be marked as passed
Environment

What mocha-teamcity-reporter version are you using: 4.0.0
Version of NPM/yarn etc and Node.js: node 14.13.1, npm 6.14.10
Version of teamcity (If Applicable): 2020.1 (on premises)

Details
An example test or code sample
To see the effect, you don't need to install mocha and teamcity reporter plugin.
Just create a teamcity build configuration with one linux shell step:
echo "##teamcity[testSuiteStarted name='suite0' flowId='40792']"
echo "##teamcity[testStarted  name='test1' captureStandardOutput='true' flowId='40792']"
>&2 echo "##teamcity[testFailed   name='test1' message='horrible error' captureStandardOutput='true' flowId='40792']"
echo "##teamcity[testFinished name='test1' duration='1424' flowId='40792']"
echo "##teamcity[testStarted  name='test2' captureStandardOutput='true' flowId='40792']"
>&2 echo "##teamcity[testFailed   name='test2' message='horrible error' captureStandardOutput='true' flowId='40792']"
echo "##teamcity[testFinished name='test2' duration='1481' flowId='40792']"
echo "##teamcity[testStarted  name='test3' captureStandardOutput='true' flowId='40792']"
>&2 echo "##teamcity[testFailed   name='test3' message='horrible error' captureStandardOutput='true' flowId='40792']"
echo "##teamcity[testFinished name='test3' duration='1583' flowId='40792']"
echo "##teamcity[testStarted  name='test4' captureStandardOutput='true' flowId='40792']"
>&2 echo "##teamcity[testFailed   name='test4' message='horrible error' captureStandardOutput='true' flowId='40792']"
echo "##teamcity[testFinished name='test4' duration='1623' flowId='40792']"
echo "##teamcity[testStarted  name='test5' captureStandardOutput='true' flowId='40792']"
>&2 echo "##teamcity[testFailed   name='test5' message='horrible error' captureStandardOutput='true' flowId='40792']"
echo "##teamcity[testFinished name='test5' duration='1525' flowId='40792']"
echo "##teamcity[testStarted  name='test6' captureStandardOutput='true' flowId='40792']"
>&2 echo "##teamcity[testFailed   name='test6' message='horrible error' captureStandardOutput='true' flowId='40792']"
echo "##teamcity[testFinished name='test6' duration='1466' flowId='40792']"
echo "##teamcity[testSuiteFinished name='suite0' duration='12740' flowId='40792']"
echo "Executing global teardown"

Run the build several times and observe random results

Thanks @dmitriy-grablyov seems like a gnarly one, any ideas on how to reduce the impact? If not I will probably just amend the readme with the information you provided for others and close this off?

I might suspect that checking the test state in the 'test finished' hookand throwing the message into stdErr too if test was failed, might be a solution to this issue.
On the other hand I doubt, that this is a good solution at all

@dmitriy-grablyov did you face this issue only with useStdError=true?
It looks like I've faced the same with no useStdError flag. But I am not sure.
I can see in raw log the followiing:
e2e_1  | ##teamcity[testFinished name='"before each" hook: Clean search in "FtsUI - 1 Search Input "' flowId='19']
e2e_1  | ##teamcity[testFailed name='FtsUI - 1.2 Security selected #full #intfull' message='Test and Master SecuritySelected.png files should be equal: expected |'Screenshot /e2e/test-screenshots/FTS/SecuritySelected.png did not match with master screenshot /e2e/master-screenshots/FTS/SecuritySelected. DiffBounds: {"left":0,"top":0,"right":180,"bottom":31} Diff-image will be saved to /e2e/diff-screenshots/FTS/SecuritySelected.png.|' to equal true' details='AssertionError: Test and Master SecuritySelected.png files should be equal: expected |'Screenshot /e2e/test-screenshots/FTS/SecuritySelected.png did not match with master screenshot /e2e/master-screenshots/FTS/SecuritySelected. DiffBounds: {"left":0,"top":0,"right":180,"bottom":31} Diff-image will be saved to /e2e/diff-screenshots/FTS/SecuritySelected.png.|' to equal true|n    at Function.compareScreenshots (src/Tests/UIGeneric.ts:21:80)|n    at processTicksAndRejections (internal/process/task_queues.js:93:5)|n    at Context.<anonymous> (src/TestsPoMa/FTS/FTS.prl.ui.ts:45:7)' captureStandardOutput='true' flowId='19']
e2e_1  | ##teamcity[testFinished name='FtsUI - 1.2 Security selected #full #intfull' duration='6166' flowId='19']

But in TeamCity:

I am really confused because I did not set useStdError any value.

No, this I haven't experienced

@dmitriy-grablyov thanks. It looks like a separate issue and I guess I know what is happening here. Will create another one.


Cannot override default languaes

Hello! I cannot seem to override the default languages. My friend and I are working on a project and everything works perfect on his end, however, when I pull the project, I see these languages as default ones:
0 => "en"
1 => "fr"
"es"
0 => "MX"
1 => "CO"
In the translatable file, the languages are edited like this
'locales' => [
'bg' => 'Български',
'en' => 'Английски'
],

The first idea, if it works on your friends setup, do you have a cached config?
artisan config:clear


Add close and closed hooks for alert

Basically, I have alerts in my app that are contained within a fixed layout - when closing the alert I need a hook so that I can remove a class from my fixed layout to "shift" it up by 30px or so to account for the space created when the alert is no longer there. FYI, the actual listener I use is this:
$(".alert-message").bind("closed", function() {
    if ($(this).hasClass("flash")) {
        $(".with-flashes").each(function(i, item) {
            $(item).removeClass("with-flashes");
        });
    }
});

because close is async, it would need to be in the removeElement function. Also, could reopn this pull request against 2.0-wip - feature development is close on 1.* thanks!


Added meta description support on announcement pages

First proposed by @rebelholic in #1048. This PR introduces an initial support for meta descriptions. I honestly don't know why announcements didn't have meta description support before.
This also fixes the bootstrap theme displaying Lorem ipsum in announcement pages.
I'd like someone to make a deep review on the code base before merging this, I'm not really in a state of clean mind right now so I might have missed something. This also creates a new description column on the post table, so I mark it as a breaking change and change the updater script to behave accordingly:
https://github.com/boxbilling/boxbilling/blob/7300327ae444d4a53c32d029b94245ff389fe5ed/src/bb-update.php#L21-L23
Just to mention, this also creates a dummy placeholder description if the author didn't specify a custom description text:
https://github.com/boxbilling/boxbilling/blob/7300327ae444d4a53c32d029b94245ff389fe5ed/src/bb-modules/News/Service.php#L45-L52

one thing I forgot: change SQL template for new installations.
I'll do that tomorrow as it's getting late now.

Hi, after you changed it. I will review this code
Regards
On Sat, 2 Oct 2021 at 03.23 Yağızhan @.***> wrote:

one thing I forgot: change SQL template for new installations.
I'll do that tomorrow as it's getting late now.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/boxbilling/boxbilling/pull/1051#issuecomment-932526877,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABPXCHELBNL5653CCL3L3PDUEYKCPANCNFSM5FFNXQ2A
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--
[image: --]
Marda Faqruradi
[image: https://]about.me/mardafaqruradi
https://about.me/mardafaqruradi?promo=email_sig

Looks like the description meta data is working correctly

LGTM, I think it would be good to update it so instead of saying "Description" it says "SEO Description" or something of the like so people know it is for SEO data and that they won't see it on their website

Done.

Also, something is a bit off here, not sure if it's due to your changes or not though.
The initial default announcements that BB comes with both load just fine

 Also, something is a bit off here, not sure if it's due to your changes or not though. The initial default announcements that BB comes with both load just fine

Not related to this specific patch. It happened after I updated CKEditor to it's latest. I guess it used to print output as markdown before, now it outputs pure HTML. We need to tell Twig not to look for markdown in it and display raw HTML coming directly from CKEditor instead.

Missed this one until now and it's already merged. All looks good to me though :)


Chore/updates and stuff

Updates dependencies and some test changes

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (3f52fee) 97.81% compared to head (83ab82a) 97.81%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##             main      #13   +/-   ##
=======================================
  Coverage   97.81%   97.81%
=======================================
  Files           4        4
  Lines         137      137
  Branches       15       15
=======================================
  Hits          134      134
  Misses          3        3

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Add support for expressions

I'll implement more of the expressions() method, to support:
Functions

[x] Arithmetic expressions
[x] https://neo4j.com/docs/cypher-manual/current/functions/mathematical-numeric/
[x] https://neo4j.com/docs/cypher-manual/current/functions/mathematical-logarithmic/
[x] https://neo4j.com/docs/cypher-manual/current/functions/mathematical-trigonometric/
[x] https://neo4j.com/docs/cypher-manual/current/functions/string/

Note, there's still quite a few missing features in Cypher-DSL here

[ ] https://neo4j.com/docs/cypher-manual/current/functions/temporal/
[ ] https://neo4j.com/docs/cypher-manual/current/functions/temporal/duration/
[ ] https://neo4j.com/docs/cypher-manual/current/functions/spatial/ ~(won't be available with the jOOQ Open Source edition)~

While the parser and QOM API don't have a representation for these things in the jOOQ Open Source Edition, we can specify Settings.parseUnknownFunctions and try to translate things from there.

[x] https://neo4j.com/docs/cypher-manual/current/functions/scalar/

Query syntax

[x] CASE https://neo4j.com/docs/cypher-manual/current/syntax/expressions/#query-syntax-case

[x] CASE abbreviations
[x] CASE simple
[x] CASE searched

[ ] EXISTS (currently not supported in Cypher-DSL, with subqueries?)
[ ] COUNT (doesn't have the exact SQL equivalent)

Not yet possible:

[ ] https://neo4j.com/docs/cypher-manual/current/functions/aggregating/ (still missing jOOQ QOM support, will implement next week)

Really great work on the expressions so far and the TCK works really well.

Kamino closed and cloned this issue to neo4j/neo4j-jdbc


[Gluten-374] Add sql tests in the timestampNTZ and native folders for velox backend

What changes were proposed in this pull request?
Add sql tests in the timestampNTZ and native folders for velox backend
How was this patch tested?
mvn clean install -pl gluten-ut -Pspark-3.2 -Pspark-ut -Pbackends-velox -DargLine="-Dspark.test.home=/opt/spark" -Dsuites="org.apache.spark.sql.GlutenSQLQueryTestSuite" -Darrow.version=10.0.0-SNAPSHOT

@rui-mo cc

Thank you, will merge after CI pass.


What is the purpose of the library, if you have to download the whole dataset?

The imagenet example (https://github.com/tmbdev/pytorch-imagenet-wds) seems to demonstrate that the library is completely useless.
First of all, what is a "shard"? Why do I need to "shard" a dataset? What is this about, where is the documentation for that?
Second, how can a library called "WEB Dataset" demand that you download the dataset in order to "shard" it and only then use it?
I need to train a model with imagenet, but I do not have hard drive space to download it. It seems that webdataset doesn't help me at all. What does this says about multi-terabyte datasets? Either the example was horrible, or the library is very questionable in terms of achieving what it aims to achieve.

Here is a YouTube playlist that explains how to use WebDataset and what to use it for. It explains why/when you want to shard, etc.

The imagenet example (https://github.com/tmbdev/pytorch-imagenet-wds) seems to demonstrate that the library is completely useless.I need to train a model with imagenet, but I do not have hard drive space to download it.

We can't distribute pre-sharded versions of ImageNet because ImageNet is available only under license; that's why the example shows you how to shard it. The code example you found exists only as an illustration of how to modify the PyTorch ImageNet example, not as a good example of how to write ImageNet training code.
WebDataset can train without any local storage. We frequently use it that way when training in containers. We have pre-sharded data available for other datasets. For example, OpenImages is available at http://storage.googleapis.com/nvdata-openimages/openimages-train-{000000..000554}.tar  You can train against those shards directly over the Internet, or alternatively, you can copy them to a local webserver and then run one or more training jobs against that web server. Whether you should use remote servers, local servers, or local disk depends on how much storage and how much network bandwidth you have available; WebDataset (like web browsers) can use any such data source transparently.
Although there is a lot of documentation (in the docs/ subdirectory) and the library is fast and pretty mature, we have no simple turn-key examples available for end users. We'll try to add those over the next few months.


How to use the translation server?

Dear OpennMT Team,
Please excuse my asking. How can I actually run the server with a test model from the OpenNMT website and perform translations?
Thanks!

maybe you could have a try with my ugly code before they have done the whole job. @mehmedes

I have the same question. I'ts not clear in the documentation how you can use the REST server. Where is the API docs? How can I send a GET request to get translations returned?

You can consult http://opennmt.net/OpenNMT/tools/servers/


Suggestions

Can you add epoch times, refresh balance button, 2FA verification thanks for developing
epoch examle;

hi @malikdoksoz , thank you for your suggestions. They are very welcome and helps us to grow. Let me try to reply to each suggestion:

epoch times: Yes we already have this in the works and partially developed
refresh balance: That is also already developed a PR should be made soon (aka to be expected in the next releases)
2FA verification: This is something that requires a bit of thinking as the keys are not stored on the device, but we'll take it into consideration :)

Closing because both the epoch info and the refresh balance have been added on Clorio 🙏


CI VM settings in tests

#264 have failed pipeline.
Test_DefaultFont_MatchesOsExpectations() failed on MacOs VM.
Maybe its not correct, to check in tests something outside project code.
This test causes a false alarm and will have to be rewritten every time Azure changes the virtual machine setup.

Yeah this is SUPER annoying... it looks like "Times" was changed to "Times New Roman" 🙈
I'll probably leave it just to see how often this happens. If it happens frequently, I'll just disable the test.


Restore deploy-storybook job

Description
Restore the job that deploys to Storybook that was lost when migrating from CircleCI
From https://github.com/oddbird/Metecho/pull/2149#discussion_r865138094

 Deploy Storybook docs via GH action on pushes to main branch.

Hrm... Canceling since a higher priority waiting request for 'refs/heads/main' exists
https://github.com/oddbird/Metecho/actions/runs/2277924304

Looks like this is the issue (possibly in addition to needing a token or PAT?): https://github.com/storybookjs/storybook-deployer/issues/77
https://github.com/oddbird/Metecho/runs/6313167269?check_suite_focus=true


Defined number of loop in the path navigator mode

Short Description
When we use the path navigator loop mode the bot do an infinity loop.
to reducing the bot detection, we can defined a number of loop and stop the bot after
Possible solution
We can defined a number of loop and stop the bot after
How it would help others
Reducing the bot detection

I'm thinking limiting the total distance travelled would be a better option? Since everyone using paths may have difference sized loops. In this case this issue is related to #3922 and #4019 but limit with distance travelled.


python setup.py egg_info does very strange things

I get the error below when running a pip install . in an environment that tox has set up. I have absolutely no idea why pgen is being run when python setup.py egg_info is run in pyOCD, but I'm hoping someone here does?
Collecting pyyaml<4,>=3 (from valinor<1.0,>=0.0.0->yotta==0.7.1)
  Using cached PyYAML-3.11.tar.gz
Collecting pyOCD<1.0,>=0.3 (from valinor<1.0,>=0.0.0->yotta==0.7.1)
  Using cached pyOCD-0.5.1.zip
    Complete output from command python setup.py egg_info:
    usage: pgen [-h] [-v] [-q] [--version]
                {clean,build,init,import,export,list} ...
    pgen: error: invalid choice: '/var/folders/k_/yq4_rh2d05gbzxls2wr9r8mw0000gp/T/easy_install-amRspV/Cython-0.23.2/Cython/Parser/Grammar' (choose from 'clean', 'build', 'init', 'import', 'export', 'list')
    Traceback (most recent call last):
      File "<string>", line 20, in <module>
      File "/private/var/folders/k_/yq4_rh2d05gbzxls2wr9r8mw0000gp/T/pip-build-itqBec/pyOCD/setup.py", line 70, in <module>
        include_package_data=True,  # include files from MANIFEST.in
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 111, in setup
        _setup_distribution = dist = klass(attrs)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/dist.py", line 268, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/dist.py", line 313, in fetch_build_eggs
        replace_conflicting=True,
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/pkg_resources/__init__.py", line 836, in resolve
        dist = best[req.key] = env.best_match(req, ws, installer)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1081, in best_match
        return self.obtain(req, installer)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1093, in obtain
        return installer(requirement)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/dist.py", line 380, in fetch_build_egg
        return cmd.easy_install(req)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 629, in easy_install
        return self.install_item(spec, dist.location, tmpdir, deps)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 659, in install_item
        dists = self.install_eggs(spec, download, tmpdir)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 842, in install_eggs
        return self.build_and_install(setup_script, setup_base)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1070, in build_and_install
        self.run_setup(setup_script, setup_base, args)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1056, in run_setup
        run_setup(setup_script, args)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/sandbox.py", line 240, in run_setup
        raise
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/sandbox.py", line 193, in setup_context
        yield
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 35, in __exit__
        self.gen.throw(type, value, traceback)
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/sandbox.py", line 164, in save_modules
        saved_exc.resume()
      File "/Volumes/Work/synced/Dev/autopulated/yotta/.tox/py27/lib/python2.7/site-packages/setuptools/sandbox.py", line 138, in resume
        type, exc = map(pickle.loads, self._saved)
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1382, in loads
        return Unpickler(file).load()
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 858, in load
        dispatch[key](self)
      File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1133, in load_reduce
        value = func(*args)
    TypeError: __init__() takes at least 3 arguments (1 given)

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/k_/yq4_rh2d05gbzxls2wr9r8mw0000gp/T/pip-build-itqBec/pyOCD

Is it trying to install cython? Uninstall project_generator and try again.
I recall I found this there recently, created a tracking issue: https://github.com/project-generator/project_generator/issues/271, we might need to come-up with another name :(

yup if I change the installation order so that cython comes first things work :)

I'll close this as it does not relate to pyOCD.


Add config entry support to person tracker

Description:
Configuration.yaml will always override person config entry.
I don't think we should support unload here.
Pull request in home-assistant.io with documentation (if applicable): home-assistant/home-assistant.io#<home-assistant.io PR number goes here>
Checklist:

[x] The code change is tested and works locally.
[ ] Local tests pass with tox. Your PR cannot be merged unless tests pass
[ ] There is no commented out code in this PR.

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

Ballob: Core piece and shouldn't be disabled.