missing checkpoints

Hi, the README says:

FinMA v0.1 (NLP 7B version)
FinMA v0.1 (Full 7B version)

but none of those checkpoints exist. Only TheFinAI/finma-7b-full repo is there but it's missing the actual checkpoint.
Thank you.

The links in the readme were changed, but the new repos are missing the model checkpoints as you said.
Old repos with checkpoints are still up at:
ChanceFocus/finma-7b-full
ChanceFocus/finma-7b-nlp

thank you for that hint, aloy99 - found the olds ones now
let me know if you want me to close this issue or perhaps it's better to do so once the README.md has been matched to the hub contents?

No worries!
I'm not a contributor, I think someone on the team will update the new repositories soon.

Are you able to use these older models though? If I try the eval as explained here:
https://github.com/The-FinAI/PIXIU?tab=readme-ov-file#automated-task-assessment
it works with llama2-7b model, but if I try any of ChanceFocus/finma-7b-full, ChanceFocus/finma-7b-nlp I get a bunch of:
../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [82,0,0], thread: [0,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [82,0,0], thread: [1,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1292: indexSelectLargeIndex: block: [82,0,0], thread: [2,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

which usually indicates that the tokenizer has higher indices than the embedding - i.e. the model or the tokenizer files are possibly borked?

I encountered the same issue last night, are you using the same arguments as in the readme?
I got it to run by loading the model as a VLLM model with the below arguments, but benchmark results were much worse than what is presented in the PIXIU paper. This might be expected, as VLLM model output in HF is inconsistent with what the same model would produce when loaded as AutoLlamaCausalLM/
python3 src/eval.py \     --model hf-causal-vllm\     --tasks flare_fpb,flare_fiqasa,flare_headlines,flare_ner,flare_finqa,flare_convfinqa,flare_sm_bigdata,flare_sm_acl,flare_sm_cikm \     --model_args use_accelerate=True,pretrained=ChanceFocus/finma-7b-full,tokenizer=ChanceFocus/finma-7b-full,max_gen_toks=2048,use_fast=False \     --no_cache \     --batch_size 8 \     --model_prompt 'finma_prompt' \     --num_fewshot 0 \     --write_out

Thank you! We will upload the mentioned checkpoints soon.

@Me1oyy, how can we reproduce the numbers in the paper - as you can see from the comments above it doesn't work.

Thank you for sharing the working code, @aloy99! I was able to reproduce it.
So you're saying it's lower than the paper, I run just the flare_finqa task and got acc=0.0785, whereas in paper it's actually the lower 0.04 - unless FinMA in the paper refers to ChanceFocus/finma-7b-nlp perhaps?
I get 0.061 w/ finma-7b-nlp
Also it's bound not produce very good results because many of the prompts are cut off:
 85%|███████████████████████████████████████████████████████████████████████████████████▍              | 976/1147 [00:46<00:07, 21.96it/sWARNING 03-27 23:10:31 scheduler.py:149] Input prompt (2445 tokens) is too long and exceeds limit of 2048           | 0/16 [00:00<?, ?it/s]

Also any idea why if I use normal llama I get acc=0? surely it should at least a bit of it right, no?
 --model "hf-causal-llama"     --model_args "use_accelerate=True,pretrained=meta-llama/Llama-2-7b-hf,tokenizer=meta-llama/Llama-2-7b-hf,use_fast=False"     --tasks "flare_finqa" --batch_size 16  --model_prompt 'finma_prompt'     --num_fewshot 0     --write_out

@stas00 You're right, I think the poor performance is largely because prompts are truncated to the maximum context length, and there is no response as a result. This shows up as 'missing' in the results report of the evaluation script.
I'm comparing my results to this paper.
It mentions on page 7 that:

The maximum length of input texts is 2048.

which does not seem to be the case with the versions of the evaluation script we can access at the moment
The paper also contains the following results:

Here are the results I got:

Task
Version
Metric
Value

Stderr

flare_convfinqa
1
acc
0.0007
±
0.0007

flare_finqa
1
acc
0.0000
±
0.0000

flare_fiqasa
1
acc
0.0000
±
0.0000

missing
1.0000
±
0.0000

f1
0.0000

macro_f1
0.0000

mcc
0.0000

flare_fpb
1
acc
0.0474
±
0.0068

missing
0.8598
±
0.0112

f1
0.0765

macro_f1
0.1025

mcc
0.0281

flare_headlines
1
avg_f1
0.6201

flare_ner
1
entity_f1
0.0000

flare_sm_acl
1
acc
0.5024
±
0.0082

missing
0.0000
±
0.0000

f1
0.4993

macro_f1
0.5002

mcc
0.0091

flare_sm_bigdata
1
acc
0.5041
±
0.0130

missing
0.0000
±
0.0000

f1
0.5053

macro_f1
0.5034

mcc
0.0115

flare_sm_cikm
1
acc
0.5074
±
0.0148

missing
0.0000
±
0.0000

f1
0.5097

macro_f1
0.5068

mcc
0.0297

I was able to reproduce the F1 score and accuracy alone when running the FPB task alone in a previous run: I'll find the arguments for that run again and update you when I'm able to get compute again.

Shouldn't you compare it against their latest paper which in theory should match the source code on this repo? https://arxiv.org/abs/2402.12659
Please clarify - is the table you shared against llama or finma-7b-nlp or finma-7b-full?
So you too are getting acc=0 on flare_finqa and flare_fiqasa as I got with llama2-7b, which is suspicious

@stas00
The latest paper expands the set of tasks that models are benchmarked on, I wanted a subset of tasks that covered most of the categories that I could run within a reasonable amount of time (thus I referred to the earlier paper's selection of tasks). But yes, I think it would make sense to compare to the results from the newer paper instead.
The table I shared above was evaluated with finma-7b-full, with the snippet I shared previously:
    --model hf-causal-vllm\
    --tasks flare_fpb,flare_fiqasa,flare_headlines,flare_ner,flare_finqa,flare_convfinqa,flare_sm_bigdata,flare_sm_acl,flare_sm_cikm \
    --model_args use_accelerate=True,pretrained=ChanceFocus/finma-7b-full,tokenizer=ChanceFocus/finma-7b-full,max_gen_toks=2048,use_fast=False \
    --no_cache \
    --batch_size 8 \
    --model_prompt 'finma_prompt' \
    --num_fewshot 0 \
    --write_out

I spent last night re-running the benchmarks with a shell script that runs one task at a time, and the results I got are very close to those presented in both papers. I'm puzzled by why I no longer have the issue with missing responses from the model, given that the only change I've made is to run the tasks separately (hardware was also changed, from RTX4090 to A6000).
declare -a tasks=("flare_fpb" "flare_fiqasa" "flare_headlines" "flare_ner" "flare_finqa" "flare_convfinqa" "flare_sm_bigdata" "flare_sm_acl" "flare_sm_cikm")

now="$(date +'%Y-%m-%d-%T')"
start=$(date +%s)

for TASK in "${tasks[@]}"
do
    python3 src/eval.py \
        --model hf-causal-vllm\
        --tasks $TASK \
        --model_args use_accelerate=True,pretrained=chancefocus/finma-7b-full,tokenizer=chancefocus/finma-7b-full,use_fast=False \
        --no_cache \
        --batch_size 8 \
        --model_prompt 'finma_prompt' \
        --num_fewshot 0  >> output_"$now".txt
done
end=$(date +%s)
seconds=$(echo "$end - $start" | bc)
echo $seconds' sec'
awk -v t=$seconds 'BEGIN{t=int(t*1000); printf "%d:%02d:%02d\n", t/3600000, t/60000%60, t/1000%60}' >> output_"$now".txt

Keep in mind that if you're using any non-finma model, you should run flare_headlines and flare_ner with --num_fewshot 5 as mentioned in the PIXIU paper (the older one):

Unfortunately, the newer FinBen paper mentions that some tasks are zero-shot and some tasks are few-shot, but does not mention which tasks fall into which category.

@stas00 The latest paper expands the set of tasks that models are benchmarked on, I wanted a subset of tasks that covered most of the categories that I could run within a reasonable amount of time (thus I referred to the earlier paper's selection of tasks). But yes, I think it would make sense to compare to the results from the newer paper instead.
The table I shared above was evaluated with finma-7b-full, with the snippet I shared previously:
    --model hf-causal-vllm\
    --tasks flare_fpb,flare_fiqasa,flare_headlines,flare_ner,flare_finqa,flare_convfinqa,flare_sm_bigdata,flare_sm_acl,flare_sm_cikm \
    --model_args use_accelerate=True,pretrained=ChanceFocus/finma-7b-full,tokenizer=ChanceFocus/finma-7b-full,max_gen_toks=2048,use_fast=False \
    --no_cache \
    --batch_size 8 \
    --model_prompt 'finma_prompt' \
    --num_fewshot 0 \
    --write_out

I spent last night re-running the benchmarks with a shell script that runs one task at a time, and the results I got are very close to those presented in both papers. I'm puzzled by why I no longer have the issue with missing responses from the model, given that the only change I've made is to run the tasks separately (I did also change hardware from RTX4090 to A6000, but I don't think it would be a vram issue).
declare -a tasks=("flare_fpb" "flare_fiqasa" "flare_headlines" "flare_ner" "flare_finqa" "flare_convfinqa" "flare_sm_bigdata" "flare_sm_acl" "flare_sm_cikm")

now="$(date +'%Y-%m-%d-%T')"
start=$(date +%s)

for TASK in "${tasks[@]}"
do
    python3 src/eval.py \
        --model hf-causal-vllm\
        --tasks $TASK \
        --model_args use_accelerate=True,pretrained=chancefocus/finma-7b-full,tokenizer=chancefocus/finma-7b-full,use_fast=False \
        --no_cache \
        --batch_size 8 \
        --model_prompt 'finma_prompt' \
        --num_fewshot 0  >> output_"$now".txt
done
end=$(date +%s)
seconds=$(echo "$end - $start" | bc)
echo $seconds' sec'
awk -v t=$seconds 'BEGIN{t=int(t*1000); printf "%d:%02d:%02d\n", t/3600000, t/60000%60, t/1000%60}' >> output_"$now".txt

Keep in mind that if you're using any non-finma model, you should run flare_headlines and flare_ner with --num_fewshot 5 as mentioned in the PIXIU paper (the older one):

Unfortunately, the newer FinBen paper mentions that some tasks are zero-shot and some tasks are few-shot, but does not mention which tasks fall into which category.
I benchmarked llama-2-7b-chat (to match the FinBen paper), and got an accuracy of 0.766 on flare_fiqasa and 0 on flare_finqa. I think it's not surprising to get a 0 for flare_finqa, given that the metric used is exact match accuracy. flare_fiqasa on the other hand is a normal sentiment analysis task and your result of 0 accuracy on that is surprising.
I've also dropped the authors an email requesting the arguments/scripts that were used for the benchmark results presented in the papers.

Hi @aloy99 and @stas00,
I really appreciate your interest and efforts in reproducing our results. We are grateful for your engagement with our work.
I must apologize for the delayed response. Our team is currently immersed in preparing a rebuttal for the ARR of the FinBen paper, which has demanded our full attention.
We recognize the importance of transparency and reproducibility in research. To address this, we are committed to providing detailed instructions on how to reproduce all our results, including all necessary parameters and inference choices. This will ensure that anyone interested can accurately benchmark and understand the nuances of our models.
Please accept our apologies for not being able to respond sooner. We are dedicated to addressing all your issues and questions as promptly as possible. Thank you again for your patience and for bringing these matters to our attention.

@jiminHuang
Your response is much appreciated, I hope to hear from the team regarding the release of instructions for reproducing your results when possible. I have reached out via email as well regarding a paper I am working on and the information I may need.
All the best for your rebuttal!

Please check our latest notebook https://colab.research.google.com/drive/1ogcCmhMc5lPhUamCk6512H3PJwPEaBZN?usp=sharing. All replication issues should be addressed.


Error Massage: tensorflow.python.framework.errors_impl.UnknownError:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.

python version: 3.6.7
tensorflow-gpu version: 2.0.0
keras version: 2.3.1
cuDNN version:10.0
CUDA version:10.0

mnist_mlp.py (https://github.com/keras-team/keras/blob/master/examples/mnist_mlp.py) works perfectly but code which is given below gives me this error:
    tensorflow.python.framework.errors_impl.UnknownError:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
             [[node conv2d_7/convolution (defined at C:\Users\ACSECKIN\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow_core\python\framework\ops.py:1751) ]] [Op:__inference_keras_scratch_graph_4815]

    Function call stack:
    keras_scratch_graph

Project code derived from https://machinelearningmastery.com/how-to-develop-a-pix2pix-gan-for-image-to-image-translation/
code:
```
from numpy import load
from numpy import zeros
from numpy import ones
from numpy.random import randint
from keras.optimizers import Adam
from keras.initializers import RandomNormal
from keras.models import Model
from keras.models import Input
from keras.layers import Conv2D
from keras.layers import Conv2DTranspose
from keras.layers import LeakyReLU
from keras.layers import Activation
from keras.layers import Concatenate
from keras.layers import Dropout
from keras.layers import BatchNormalization
from keras.layers import LeakyReLU
from matplotlib import pyplot

# define the discriminator model
def define_discriminator(image_shape):
	# weight initialization
	init = RandomNormal(stddev=0.02)
	# source image input
	in_src_image = Input(shape=image_shape)
	# target image input
	in_target_image = Input(shape=image_shape)
	# concatenate images channel-wise
	merged = Concatenate()([in_src_image, in_target_image])
	# C64
	d = Conv2D(64, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(merged)
	d = LeakyReLU(alpha=0.2)(d)
	# C128
	d = Conv2D(128, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(d)
	d = BatchNormalization()(d)
	d = LeakyReLU(alpha=0.2)(d)
	# C256
	d = Conv2D(256, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(d)
	d = BatchNormalization()(d)
	d = LeakyReLU(alpha=0.2)(d)
	# C512
	d = Conv2D(512, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(d)
	d = BatchNormalization()(d)
	d = LeakyReLU(alpha=0.2)(d)
	# second last output layer
	d = Conv2D(512, (4,4), padding='same', kernel_initializer=init)(d)
	d = BatchNormalization()(d)
	d = LeakyReLU(alpha=0.2)(d)
	# patch output
	d = Conv2D(1, (4,4), padding='same', kernel_initializer=init)(d)
	patch_out = Activation('sigmoid')(d)
	# define model
	model = Model([in_src_image, in_target_image], patch_out)
	# compile model
	opt = Adam(lr=0.0002, beta_1=0.5)
	model.compile(loss='binary_crossentropy', optimizer=opt, loss_weights=[0.5])
	return model

# define an encoder block
def define_encoder_block(layer_in, n_filters, batchnorm=True):
	# weight initialization
	init = RandomNormal(stddev=0.02)
	# add downsampling layer
	g = Conv2D(n_filters, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(layer_in)
	# conditionally add batch normalization
	if batchnorm:
		g = BatchNormalization()(g, training=True)
	# leaky relu activation
	g = LeakyReLU(alpha=0.2)(g)
	return g

# define a decoder block
def decoder_block(layer_in, skip_in, n_filters, dropout=True):
	# weight initialization
	init = RandomNormal(stddev=0.02)
	# add upsampling layer
	g = Conv2DTranspose(n_filters, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(layer_in)
	# add batch normalization
	g = BatchNormalization()(g, training=True)
	# conditionally add dropout
	if dropout:
		g = Dropout(0.5)(g, training=True)
	# merge with skip connection
	g = Concatenate()([g, skip_in])
	# relu activation
	g = Activation('relu')(g)
	return g

# define the standalone generator model
def define_generator(image_shape=(256,256,3)):
	# weight initialization
	init = RandomNormal(stddev=0.02)
	# image input
	in_image = Input(shape=image_shape)
	# encoder model
	e1 = define_encoder_block(in_image, 64, batchnorm=False)
	e2 = define_encoder_block(e1, 128)
	e3 = define_encoder_block(e2, 256)
	e4 = define_encoder_block(e3, 512)
	e5 = define_encoder_block(e4, 512)
	e6 = define_encoder_block(e5, 512)
	e7 = define_encoder_block(e6, 512)
	# bottleneck, no batch norm and relu
	b = Conv2D(512, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(e7)
	b = Activation('relu')(b)
	# decoder model
	d1 = decoder_block(b, e7, 512)
	d2 = decoder_block(d1, e6, 512)
	d3 = decoder_block(d2, e5, 512)
	d4 = decoder_block(d3, e4, 512, dropout=False)
	d5 = decoder_block(d4, e3, 256, dropout=False)
	d6 = decoder_block(d5, e2, 128, dropout=False)
	d7 = decoder_block(d6, e1, 64, dropout=False)
	# output
	g = Conv2DTranspose(3, (4,4), strides=(2,2), padding='same', kernel_initializer=init)(d7)
	out_image = Activation('tanh')(g)
	# define model
	model = Model(in_image, out_image)
	return model

# define the combined generator and discriminator model, for updating the generator
def define_gan(g_model, d_model, image_shape):
	# make weights in the discriminator not trainable
	d_model.trainable = False
	# define the source image
	in_src = Input(shape=image_shape)
	# connect the source image to the generator input
	gen_out = g_model(in_src)
	# connect the source input and generator output to the discriminator input
	dis_out = d_model([in_src, gen_out])
	# src image as input, generated image and classification output
	model = Model(in_src, [dis_out, gen_out])
	# compile model
	opt = Adam(lr=0.0002, beta_1=0.5)
	model.compile(loss=['binary_crossentropy', 'mae'], optimizer=opt, loss_weights=[1,100])
	return model

# load and prepare training images
def load_real_samples(filename):
	# load compressed arrays
	data = load(filename)
	# unpack arrays
	X1, X2 = data['arr_0'], data['arr_1']
	# scale from [0,255] to [-1,1]
	X1 = (X1 - 127.5) / 127.5
	X2 = (X2 - 127.5) / 127.5
	return [X1, X2]

# select a batch of random samples, returns images and target
def generate_real_samples(dataset, n_samples, patch_shape):
	# unpack dataset
	trainA, trainB = dataset
	# choose random instances
	ix = randint(0, trainA.shape[0], n_samples)
	# retrieve selected images
	X1, X2 = trainA[ix], trainB[ix]
	# generate 'real' class labels (1)
	y = ones((n_samples, patch_shape, patch_shape, 1))
	return [X1, X2], y

# generate a batch of images, returns images and targets
def generate_fake_samples(g_model, samples, patch_shape):
	# generate fake instance
	X = g_model.predict(samples)
	# create 'fake' class labels (0)
	y = zeros((len(X), patch_shape, patch_shape, 1))
	return X, y

# generate samples and save as a plot and save the model
def summarize_performance(step, g_model, dataset, n_samples=3):
	# select a sample of input images
	[X_realA, X_realB], _ = generate_real_samples(dataset, n_samples, 1)
	# generate a batch of fake samples
	X_fakeB, _ = generate_fake_samples(g_model, X_realA, 1)
	# scale all pixels from [-1,1] to [0,1]
	X_realA = (X_realA + 1) / 2.0
	X_realB = (X_realB + 1) / 2.0
	X_fakeB = (X_fakeB + 1) / 2.0
	# plot real source images
	for i in range(n_samples):
		pyplot.subplot(3, n_samples, 1 + i)
		pyplot.axis('off')
		pyplot.imshow(X_realA[i])
	# plot generated target image
	for i in range(n_samples):
		pyplot.subplot(3, n_samples, 1 + n_samples + i)
		pyplot.axis('off')
		pyplot.imshow(X_fakeB[i])
	# plot real target image
	for i in range(n_samples):
		pyplot.subplot(3, n_samples, 1 + n_samples*2 + i)
		pyplot.axis('off')
		pyplot.imshow(X_realB[i])
	# save plot to file
	filename1 = 'plot_%06d.png' % (step+1)
	pyplot.savefig(filename1)
	pyplot.close()
	# save the generator model
	filename2 = 'model_%06d.h5' % (step+1)
	g_model.save(filename2)
	print('>Saved: %s and %s' % (filename1, filename2))

# train pix2pix models
def train(d_model, g_model, gan_model, dataset, n_epochs=50, n_batch=1):
	# determine the output square shape of the discriminator
	n_patch = d_model.output_shape[1]
	# unpack dataset
	trainA, trainB = dataset
	# calculate the number of batches per training epoch
	bat_per_epo = int(len(trainA) / n_batch)
	# calculate the number of training iterations
	n_steps = bat_per_epo * n_epochs
	# manually enumerate epochs
	for i in range(n_steps):
		# select a batch of real samples
		[X_realA, X_realB], y_real = generate_real_samples(dataset, n_batch, n_patch)
		# generate a batch of fake samples
		X_fakeB, y_fake = generate_fake_samples(g_model, X_realA, n_patch)
		# update discriminator for real samples
		d_loss1 = d_model.train_on_batch([X_realA, X_realB], y_real)
		# update discriminator for generated samples
		d_loss2 = d_model.train_on_batch([X_realA, X_fakeB], y_fake)
		# update the generator
		g_loss, _, _ = gan_model.train_on_batch(X_realA, [y_real, X_realB])
		# summarize performance
		print('>%d, d1[%.3f] d2[%.3f] g[%.3f]' % (i+1, d_loss1, d_loss2, g_loss))
		# summarize model performance
		if (i+1) % (bat_per_epo * 10) == 0:
			summarize_performance(i, g_model, dataset)

# load image data
dataset = load_real_samples('fabric_256.npz')
print('Loaded', dataset[0].shape, dataset[1].shape)
# define input shape based on the loaded dataset
image_shape = dataset[0].shape[1:]
# define the models
d_model = define_discriminator(image_shape)
g_model = define_generator(image_shape)
# define the composite model
gan_model = define_gan(g_model, d_model, image_shape)
# train model
train(d_model, g_model, gan_model, dataset)

@acseckin
Can you please go through TensorFlow GPU instructions page and see if it helps you.I guess the problem with the compatibility.
Upgrade libcudnn packages to 7.6.2 and Initialize your code with following code and let us know how it progresses.
import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config=config)

Thanks!

@acseckin
Any update on this issue please? Thanks!

Still same. Can you share a working/ compatible versions. I tried a lot of combinations (tensorflow, keras cuda and cudnn) but no response for gpu. code works fine for cpu.

Hope this helps. I was facing the similar issue with:
tensorlfow 2.0
CUDA 10.0
cuDNN 7.6.5
I downgraded to cuDNN 7.6.2 and my model is training now. I tried cuDNN 7.5.1 before cuDNN 7.6.2, but that complained that source was built with 7.6 +

I was having this issue when running with TensorFlow v1.15, CUDA 10.0.130,  CUDNN <IP_ADDRESS> in eager mode. Adding allow_growth as @acseckin suggested fixed the problem.

@acseckin Can I close this issue as it has been solved. Thanks!

Also just a note, I saw the error again when running another piece of code. Upgrading CUDNN solved it:
sudo apt-get install --no-install-recommends libcudnn7=<IP_ADDRESS>-1+cuda10.0

I guess this should be updated in the documentation especially here so  there is no confusion progressing furthur.

For TF-GPU , CUDA, CUDNN compatibility refer tested build configchart.
Closing this issue since it's resolved. Feel free to reopen if necessary. Thanks!


Adaptation for BSD on amd64 arch. & Fixed typo for BSD on 32 bit arch.

In my machine uname -m returns amd64 so I fixed logic for determining architecture in plugins/go-build/bin/go-build especially in BSD-related function. It may be suggested to add similar architecture determining code for other OS.
In addition I fixed trivial typo.

Thank you for the PR, I'll review it within today (European Central Time)!

I took my sweet time, sorry for the wait!

@windymelt https://github.com/syndbg/goenv/releases/tag/1.14.0 released!


activitymanager error (due to no data ?)

Describe the bug
  File "C:\Users\Athan\AppData\Local\EDMarketConnector\plugins\BGS-Tally\bgstally\activitymanager.py", line 42, in save
    activity.save(path.join(self.bgstally.plugin_dir, FOLDER_ACTIVITYDATA, activity.tick_id + FILE_SUFFIX))
                                                                           ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

To Reproduce
Steps to reproduce the behavior:

I've not actually been playing, so have zero data any time recently.
Run EDMC with BGS-Tally
See error on console.

Expected behavior
No errors.
Desktop (please complete the following information):

OS: Windows 10 (latest build)
Plugin Version: 2.2.1 git tag, not from .zip file
EDMC Version: Current develop ( c350782a67713565ae15a4201569932f158bea36  )

Additional context
I'd guess that activity.tick_id is None, so you need a check on that before invoking this code, possibly more of it above too.

Could I get a little more context:

Had you just upgraded to the 2.2.1 codebase, and if so do you recall what version you previously ran?
Does the 'Previous BGS Tallies' dropdown have anything in it?
What happens if you click the 'Latest BGS Tally' button - any exceptions thrown?

From the 2.2.0 tag

Yes, lots.  The most recent right now is from 2023-01-10, there are more than just this, but the image shows I have gaps.

No exceptions

Nor did I get the 'original' exception this time.  Possibly because that first run of 2.2.1 left some data behind that avoids it ?

No, no other BGS-Tally exceptions.  Although here's the full version of it:
2023-01-12 14:39:19.994 UTC - ERROR - 21164:24668:24668 plug.notify_journal_entry:319: Plugin "BGS-Tally" failed
Traceback (most recent call last):
  File "C:\Users\Athan\Documents\Devel\EDMarketConnector\plug.py", line 316, in notify_journal_entry
    newerror = journal_entry(cmdr, is_beta, system, station, dict(entry), dict(state))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Athan\AppData\Local\EDMarketConnector\plugins\BGS-Tally\load.py", line 59, in journal_entry
    this.journal_entry(cmdr, is_beta, system, station, entry, state)
  File "C:\Users\Athan\AppData\Local\EDMarketConnector\plugins\BGS-Tally\bgstally\bgstally.py", line 167, in journal_entry
    if dirty: self.save_data()
              ^^^^^^^^^^^^^^^^
  File "C:\Users\Athan\AppData\Local\EDMarketConnector\plugins\BGS-Tally\bgstally\bgstally.py", line 231, in save_data
    self.activity_manager.save()
  File "C:\Users\Athan\AppData\Local\EDMarketConnector\plugins\BGS-Tally\bgstally\activitymanager.py", line 42, in save
    activity.save(path.join(self.bgstally.plugin_dir, FOLDER_ACTIVITYDATA, activity.tick_id + FILE_SUFFIX))
                                                                           ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

Thanks Ath, I'll try to work out how it got into a 'no tick ID' situation, and as a last resort also put some error-detection in place as you suggest.


ReactiveTextField.maxLines cannot be null

Thank you for migrating to null-safety! However I found an issue.
TextField.maxLines can be null but ReactiveTextField.maxLines can't be.
https://github.com/joanpablo/reactive_forms/blob/master/lib/src/widgets/reactive_text_field.dart#L114

Thanks @limenote135
I will fix it in next release very soon

A temporary workaround is simply set large number such as 999.
ReactiveTextField<String>(
            formControlName: 'something',
            minLines: 5,
            maxLines: 99, // instead of null
          )

In the current released version, this issue is solved. Thanks

@joanpablo Hmm... Which version do you mean the current release is? v10.0.2 has this issue.

Hi @limenote135,
Sorry My misunderstood, you are right is not yet solved, I will quickly fix it and by the end of the day I will release a new version.
Thankd

Hi @limenote135,
I have just release v10.0.3 the maxLines fixed. Please check it and re-open the issue or create e new one if you find something else wrong.
Thanks, have a nice evening.

@joanpablo Thank you for your quick fix!


Migrate to clean JAX-RS 2.0 API and remove proprietary dependencies

(related to #107 and #122)
The DockerClient still has dependencies on Jersey that make it really hard (nearly impossible) to use the docker client library in environments that depend on other REST implementations, e.g. Wildfly that is built on RESTeasy, checked sources for version 3.6.8 (fortunately it's all in one place,-):

DefaultDockerClient.java:import org.glassfish.jersey.apache.connector.ApacheClientProperties;
DefaultDockerClient.java:import org.glassfish.jersey.apache.connector.ApacheConnectorProvider;
DefaultDockerClient.java:import org.glassfish.jersey.client.ClientConfig;
DefaultDockerClient.java:import org.glassfish.jersey.internal.util.Base64;
DefaultDockerClient.java:import org.glassfish.jersey.jackson.JacksonFeature;

Maven side dependencies on Jersey:
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-client</artifactId>
<version>2.19</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.connectors</groupId>
<artifactId>jersey-apache-connector</artifactId>
<version>2.19</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.media</groupId>
<artifactId>jersey-media-json-jackson</artifactId>
<version>2.19</version>
</dependency>

Interestingly, the client works for some commands (like looking up, starting or stopping containers), but fails for others (like reading logs or executing commands), I guess only the failing commands use some Jersey specific calls.
Please think about fixing this in the library and only target standard JAX-RS 2 API, as this would be the most straight forward solution and should be really doable (I can support the effort...), as bending widlfly or EAR deployments is really a hassle I want to avoid.
related Stackoverflow Thread

To be more concrete, of course you need a minimal client implementation as Java does not provide a default implementation for JAX-RS 2, sadly.
I am currently evaluating:

UniREST or
https://github.com/Netflix/feign

My hope would be that this does not clash with server side RESTeasy...

Sadly it doesn't quite work like that, as you always need a specific implementation included in the project.
I will therefore create a fork with RESTeasy, but not use any provider specific extensions.
Then I'd only need to modify the POM and provide a variant for RESTeasy using a classifier, much like the shaded variant.

I have been thinking about attempting to change the library to use a non-Jersey library, since Jersey drags in so so many dependencies and causes headaches for people that use different versions (or at least add an option or alternate DockerClient). For example something like Apache HttpClient or OkHttp.
If you have an environment that uses some other Jersey implementation and docker-client used something non-Jersey, would that also alleviate the issue?

sorry missed the comment/update - I'm working on updating the old commit by @JWGmeligMeyling, strip Jersey special code and replace it with REST-impl-agnostic code from Apache HttpClient.
When I'm done, I'll be happy to provide a pull request, but sadly had no time last week, will revisit this week as it's anyway needed for my work.

I implemented a simple factory that made it easy to exclude the jersey dependencies in this PR: https://github.com/spotify/docker-client/pull/340. The alternate proposal to no longer use jersey would also work.

Thanks @jskjons this would also be a promising route to go.
I am actually pretty far with the clean-room JAX-RS 2.0 approach, but hit a snag:
There is no support for custom protocols like unix:// (used for socket connection) in the JAX-RS 2.0 client API, so for that you'd still need Jersey or RESTeasy, below for Jersey:
https://jersey.java.net/documentation/latest/client.html#d0e4844
Also, timeout parameters are not standardized and also implementation dependent.
Maybe we can merge our work @jskjons ?
I'll push my changes to my fork for review.

I've finally come back to this issue and was able to update your PR #340 @jskjons - many thanks for the excellent work!
Only had to fix some dependency issues with jersey pulling in an outdated hk2 lib and some conflicting libs in my own project, but the main issue that kept me busy was a conflict between docker-client's httpclient library and the one seen on Wildfly's classpath.
In the end, a clean manifest entry in the EAR plugin to define a module dependency to httpclient was all it needed, always easy in hindsight,-)
Last issue is that when pulling an image, it is downloaded completely and verified OK but then I still get an IOException, although I see some RESTeasy related fixes in the code of the PR.
@jskjons you said events are not working, but the ProgressStream should work also for RESTeasy?
java.io.IOException: Stream closed
        at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:291)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._loadMore(UTF8StreamJsonParser.java:204)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd2(UTF8StreamJsonParser.java:2978)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:2973)
        at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:731)
        at com.fasterxml.jackson.databind.MappingIterator.hasNextValue(MappingIterator.java:240)
        at com.spotify.docker.client.ProgressStream.hasNextMessage(ProgressStream.java:53)

Another open issue is related to Tests covering Docker swam, all related tests fail with this error using ResteasyClientFactory:
javax.ws.rs.NotAcceptableException: HTTP 406 Not Acceptable
This node is not a swarm manager. Use \"docker swarm init\" or \"docker swarm join\" to connect this node to swarm and try again.

created PR #691 based on my work on PR #340 - please have a look (already needs merging again...:-/ )


Fix stray character with 'smoothscroll' marker and double-width char

This now also fixes the whole first line becoming completely wrong when using DBCS, but I cannot find a way to test that.

This now also fixes the whole first line becoming completely wrong when using DBCS, but I cannot find a way to test that.

Functions like screenstring() don't seem to support DBCS, and it is worthy to add DBCS support to these functions just to test this?

Functions like screenstring() don't seem to support DBCS,

I think it is supported. See Test_screenchar_utf8() in test_utf8.vim.

Ah, I wasn't aware that this supports UTF-8 but doesn't support DBCS.

DBCS is stored differently, this was implemented before Unicode support
and makes use of the property that with this encoding a double-wide
character uses two bytes.  These two bytes are then stored in two
positions in ScreenLines[].  Except for DBCS_JPNU which uses
ScreenLines2[] for single-width double-byte characters.
It should be fairly easy to implement.  Search for ScreenLines2[] in
src/drawline.c to see how bytes are stored.  f_screenstring() then needs
to do the reverse.
--
% cat /usr/include/real_life.h
void life(void);
/// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net   \
///                                                                      \
\\        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///

Do you plan to add DBCS support to screenstring()?  Or are you waiting for someone to implement this?

I'll try to do that soon.

I changed functions like screenstring() to use screen_getbytes().

I changed functions like screenstring() to use screen_getbytes().

Let me make a patch for this first, the code and test for it as separate
from fixing the original problem.
--
From "know your smileys":
:.-(	Crying
/// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net   \
///                                                                      \
\\        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///


deadlock in org.fusesource.lmdbjni.JNI.mdb_txn_begin?

I'm trying to convert from my old db to lmdb, using 8 threads in parallel.  There's about 20000 entries, but after about 6000 org.fusesource.lmdbjni.Database#put(byte[], byte[])s, I either hit a SIGSEGV or the threads lock up all with this stack trace:
	  at org.fusesource.lmdbjni.JNI.mdb_txn_begin(JNI.java:-1)
	  at org.fusesource.lmdbjni.Env.createTransaction(Env.java:453)
	  at org.fusesource.lmdbjni.Env.createWriteTransaction(Env.java:411)
	  at org.fusesource.lmdbjni.Database.put(Database.java:394)
	  at org.fusesource.lmdbjni.Database.put(Database.java:386)

I'm using my fork of 0.4.7-SNAPSHOT.  Increasing the map size works, but I would have expected a MDB_MAP_FULL instead of a hang.

Try one thread and use an external transaction instead, commit when done.
Example:
 try (Transaction tx = env.createWriteTransaction()) {
   db.put(tx, key, val);
   ...
   tx.commit();
 }

LMDB is single-writer so other writing threads would block anyway.
If your application SIGSEGV you're probably using the API incorrectly. Show me the code and I may be able to help you.

Yes, single threaded, it gives an MDB_MAP_FULL.  I haven't tried with a single transaction, but I imagine that would not hang/SIGSEGV.
I got trigger happy and deleted my old db, but here's a small testcase with the same symptoms.
  @Test
  public void testStress() {
    Collections.nCopies(8, null).parallelStream().forEach(new Consumer<Object>() {
      @Override
      public void accept(Object ignored) {
        Random random = new Random();
        for (int i = 0; i < 15000; i++) {
          db.put(bytes(Long.toString(random.nextLong())), bytes(Long.toString(random.nextLong())));
        }
      }
    });
  }

Hmm. Yes I get the same SIGSEGV. The test runs fine with a ExecutorService though.
    ExecutorService service = Executors.newCachedThreadPool();
    service.execute(() -> {
      Random random = new Random();
      for (int i = 0; i < 15000; i++) {
        db.put(bytes(Long.toString(random.nextLong())), bytes(Long.toString(random.nextLong())));
      }
    });

MDB_MAP_FULL means the database is full. You can increase the size by calling Env.setMapSize before opening the environment.

The SIGSEGV happens when the transaction aborts which is strange. A put should always succeed or block-then-succeed when the write-lock is released.
I can also see the hang now sometimes. Sometimes a thread seems hanging in mdb_put and sometimes all threads block at mdb_txn_begin.
And all this only happens with ForkJoin. Weird.

It may well be a bug in LMDB.

I'm not sure I understand your ExecutorService example.  Doesn't that only execute the loop once and not 8 times concurrently?
Yeah, I got it all working (serial and parallel) with Env.setMapSize.  Sorry I didn't mention that earlier.
I haven't observed it hang in mdb_put, but I suppose at this point you've run it more times than I have.  Hopefully all three symptoms will have the same fix.

Sorry my bad about the ExecutorService example.
The problem only seems to manifest when the database is too small. Do you see this as well?

env.setMapSize(50, ByteUnit.MEBIBYTES); - OK
env.setMapSize(30, ByteUnit.MEBIBYTES); - OK
env.setMapSize(20, ByteUnit.MEBIBYTES); - OK
env.setMapSize(15, ByteUnit.MEBIBYTES); - OK
env.setMapSize(10, ByteUnit.MEBIBYTES); - OK
env.setMapSize(5, ByteUnit.MEBIBYTES); - SIGSEGV
env.setMapSize(8, ByteUnit.MEBIBYTES); - hang in mdb_txn_begin
env.setMapSize(9, ByteUnit.MEBIBYTES); - OK
I thought 10 was the default, so not sure why it has issues without setting the map size.


dump all data without sync binlog

Sometimes we only need dump data and ignore sync the binlog such as fetching old data.
Only move the code from tryDump to dump and export a method named Dump.

Add a flag to mark whether is dumped.
We can call Run after called Dump,
but we can't call Dump after called Run for that data will be dump twice.


page.getSize() with box MediaBox and CropBox values

We encountered a situation where pdf dimensions didn't seem to be accurately represented. It only seemed to happen with a couple of specific PDFs which, unfortunately,  I can't share here - but I can tell you how to recreate it.
In our situation, we are applying content onto a pdf and saving it, using some coordinates given by a frontend UI (think drag and drop).
The PDFs we were seeing issues with were putting content far offscreen (we thought at first it wasn't being applied at all). When looking at the PDF dimensions in Mac's file info util, we'd see dimensions of 659 × 790. When using an Apache tool to inspect the document, we found that the document showed a MediaBox value of 1217.01 x 790.729, and a CropBox width of 659.121 - which lined up with what Mac's file inspector was saying. Now, of course, we know the width that was being computed in our script was the 1217 width, but it was being displayed in that 659 width.
What we ended up doing to address this was using the page.getCropBox() method to see if it had a width, then subtracting that from the MediaBox width (page.getSize()) to get an offset to apply to our xPos for our content overlay.
To write tests for this, we're using the page.setMediaBox() and page.setCropBox() methods to reproduce this condition in the pdf metadata.
This is working, but it definitely makes me nervous. So my question is, have you encountered this issue, and if so, what's the best way to handle it?
Thanks again!

Just adding some additional thoughts here:
In viewers like pdf.js, CropBox is commonly used to display the contents of a PDF - so in our case, those are the dimension I expect to work from for the page.
I'm wondering if we should consider introducing a new method like: page.getCropSize() - or perhaps a flag on page.getSize() - where your intentions are to get visible working dimension for a given page... just a thought, and again, this may be handled in some other way that I'm missing.
Thanks!

In addition to https://pdf-lib.js.org/docs/api/classes/pdfpage#getsize the following methods are available:

https://pdf-lib.js.org/docs/api/classes/pdfpage#gettrimbox
https://pdf-lib.js.org/docs/api/classes/pdfpage#getartbox
https://pdf-lib.js.org/docs/api/classes/pdfpage#getbleedbox
https://pdf-lib.js.org/docs/api/classes/pdfpage#getcropbox

The https://pdf-lib.js.org/docs/api/classes/pdfpage#setsize API docs explain a bit about the various boxes. It also has some internal logic that attempts to set the correct boxes automagically: https://github.com/Hopding/pdf-lib/blob/e10290ac8503fd5b21f2d895252835e00a1f16b6/src/api/PDFPage.ts#L189-L218
Perhaps additional logic could be added to https://pdf-lib.js.org/docs/api/classes/pdfpage#getsize to make it do the expected thing automatically (in most cases) and provide better docs for when it does not. Added it to the roadmap: https://github.com/Hopding/pdf-lib/discussions/998.

I know this issue was closed a long time ago, but I brought up a case of inconsistency when retrieving MediaBox and CropBox information from a PDF page.
I have a PDF whose first page has different box information than the pages. However, when I retrieve this information using pdfinfo, I get information that differs from the information that pdf-lib gives me.

I drew some circles using the CropBox information and this is how the 2 tests turned out. The first printout was using the pdfinfo information. The second printout was using the information that pdf-lib gives me through the getCropBox() method.

How is this MediaBox and CropBox information obtained in pdf-lib?
The example PDF is below:
input2.pdf


Problem with Anaconda

Hi. I am trying to install Anaconda. I have Windows 10. Here are the steps I have taken:
Login to https://www.anaconda.com/products/individual
I download https://repo.anaconda.com/archive/Anaconda3-2020.02-Windows-x86_64.exe
I install Anaconda (run) as Administrator
Installation steps

Install for: Just me
destination Folder: C:\anaconda3.
Advanced Options: Register Anaconda3 as my default Python 3.7

I run the Anaconda Prompt (with Administrator permissions) the following: anaconda-navigator
Errors that appear to me:

ModuleNotFoundError: No module named 'PySide'
PythonQtError ('No Qt bindings could be found')
qtpy.PythonQtError: No Qt bindings could be found

I proceed to install PySide: pip install PySide2
Successfully installed: Successfully installed PySide2-<IP_ADDRESS> shiboken2-<IP_ADDRESS>
Within the environment variables, in system variables I have configured the following:
Variable: Path
Value: C: \ anaconda3 \ Lib \ site-packages \ PySide2 \
Variable: QT_PLUGIN_PATH
Value: C: \ anaconda3 \ Lib \ site-packages \ PySide2 \ plugins
I run anaconda-navigator again
I see the Anaconda icon when it starts to load the applications it appears to me:
Message: (base) C:\WINDOWS\system32>anaconda-navigator
Traceback (most recent call last):
File "C:\anaconda3\lib\site-packages\anaconda_navigator\widgets\main_window.py", line 541, in setup
self.post_setup(conda_data=conda_data)
File "C:\anaconda3\lib\site-packages\anaconda_navigator\widgets\main_window.py", line 578, in post_setup
self.tab_home.setup(conda_data)
File "C:\anaconda3\lib\site-packages\anaconda_navigator\widgets\tabs\home.py", line 168, in setup
self.set_environments(environments)
File "C:\anaconda3\lib\site-packages\anaconda_navigator\widgets\tabs\home.py", line 175, in set_environments
self.combo.currentIndexChanged.disconnect()
RuntimeError: Failed to disconnect signal currentIndexChanged(int).
I hope you can help me.
Regards.

I don't support the Anaconda packages.
They were produced @aknuds1 I think, but it's not clear to me what package you're installing with anaconda.
I really need to release the new version to PyPI which doesn't include the module itself in setup.py, and therefore does not need a version of Qt installed during setup. That being said the packages in PyPI now are wheels (or I think there are wheels and sdist), if you install the wheel the setup.py problem goes away.
I highly recommend migrating to poetry for dependency management, it's standardized and supports pep517 and is generally pleasant to use in ways that I found anaconda never was, but I suppose your mileage may vary.
try:

installing from the tests-2019 branch which has the pyproject.toml + new setup.py that does not import quamash
installing the wheel directly (instead of pip install quamash) download and install https://files.pythonhosted.org/packages/89/41/ba1396ebed0dcfd09854addea45323c93f498dc7c45d855943ab31791194/Quamash-0.6.1-py3-none-any.whl

I think there's a flag to pip to force it to use the .whl and not the sdist, but I forget it


comments.visible seemingly has no effect

Testing #167403
Steps:

Open vscode
Checkout PR with comments
Set comments.visible to false

:bug: => Comments view, comment ranges, plus icon, peek comment widget all seem to be present, even after reloading

I am seeing similar behavior, when I add "comments.visable": false, to my settings and hover over it, it says Unknown Configuration Setting so I am not sure if I am doing it incorrectly or if it is what SteVen is seeing.

@eleanorjboyd it looks like you have a typo in "visible" making it unknown

yep that was my mistake 🤦♀
Thanks for the fix, I got the setting working and found some different behavior from you. The comments disappeared for me after I switched the setting and so it hid PR comments I had in a python file.

@sbatten I'm not able to reproduce this. Are there any other details of your setup that might be relevant?
Worth noting: "comments.visible": false does not disable the Comments view. If you open the comments view then click on a comment then comment visibility will be toggled back on.

@alexr00 thanks for the explanation, that was my confusion. can we repurpose this issue to address the confusion in the setting description. Perhaps an additional sentence: You can still access comments via the comments view and this setting will be ignored for the remainder of the session if so.


util: set fetch_directory to $HOME

ceph-ansible tries to write to fetch_directory. It defaults this variable to the current working directory, and when we're running ansible in prodution, we don't want to grant the UID write access to cwd.
Se fetch_directory to $HOME, since we know that will be writable.

This latest version passes the tests.


Missing filename= in Content-Disposition ignores the received file

When posting a file to the server with Content-Type: multipart/form-data; boundary=.... and a Content-Disposion: that is missing filename the file is ignored and does not get bound.
[HttpPost]
public async Task<IActionResult> Upload(IFormFile file)

If the posted content has Content-Disposition: form-data; name=\"file\"; filename=\"latest.zip\" the file is received correctly.
If the posted content has Content-Disposition: form-data; name=\"file\" the file is not received.
RFC2388 4.4 specifies that The sending application MAY supply a file name.
The issue seems to be in https://github.com/aspnet/HttpAbstractions/blob/master/src/Microsoft.AspNetCore.Http/Features/FormFeature.cs#L301
which expects FileName to be present.
I found this as I was trying to post data with (node-)fetch using FormData which does not automatically add a filename parameter.
This seems related to #446.

Multipart forms can contain form data and files. The file name is the only hint which it is. Regular data goes into the form collection.
https://github.com/aspnet/HttpAbstractions/blob/master/src/Microsoft.AspNetCore.Http/Features/FormFeature.cs#L292

I found this as I was trying to post data with (node-)fetch using FormData which does not automatically add a filename parameter.

So you used FormData to post a file without the file name parameter, is that common? What do other frameworks do?
@Tratcher do you know what ASP.NET 4.x did?
@corneliutusnea do you have a runnable sample or some sample code we could use to repro this?


Video call flashes native UI

My understanding is that for video calls you route to your video components when answering an incoming call. Whenever I do that, the native UI flashes for a second before showing my component. Is there a way to prevent that?

Sorry in my project I only need voices at the moment so I haven't played with video yet. Feel free to post your solutions here if you figure it out. Cheers!


[azservicebus] Enable distributed tracing

[ ] The purpose of this PR is explained in this or a referenced issue.
[ ] The PR does not update generated files.

These files are managed by the codegen framework at Azure/autorest.go.

[x] Tests are included and/or updated for code changes.
[ ] Updates to module CHANGELOG.md are included.
[ ] MIT license headers are included in each file.

Hi @lmolkova ! I had a small question regarding diagnostic-id in https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-end-to-end-tracing?tabs=net-standard-sdk-2
The .NET SDK seems to be hooking them up to a ReceiveMessages trace when the users set the diagnostic-id and linking the list of diagnostic ids from the messages to the Receive trace (code here). I might be misunderstanding how .NET is doing it, but I am wondering what we enable with the diagnostic-ids?


Bug: Pressing Up/Down with a decorator node selection skips immediate siblings

When a decorator node is selected, pressing Arrow Up/Arrow Down moves the cursor off of the decorator node but skips the immediate sibling placing the cursor further away than expected.
Lexical version: 0.5.0
Steps To Reproduce

Open the playground and add a HR node then a couple of paragraphs
Select the HR node by using Left/Right arrow keys
Press Down arrow key

The current behavior
https://user-images.githubusercontent.com/415/198145317-5dc1d9d4-6530-410a-9c07-5f6e2b47d755.mp4
The expected behavior
Cursor is placed on the immediate sibling, same as when pressing Arrow Left/Arrow Right

This is intentional, unfortunately handling up/down is tricky and so we leave it to the browser for default behavior. It's tricky as knowing what is visually above means having to calculate the rect of the current selection.

Closing this for the moment as there doesn't seem like an action we can take, but feel free to open otherwise @trueadm.

Our use-case was pretty basic, we wanted a consistent behaviour of ArrowUp moving caret to end of previous node and  ArrowDown moving caret to beginning of next node. It was possible to handle this simply via overriding the corresponding key commands - implementation in https://github.com/TryGhost/Koenig/commit/a2a23debc760f6b53d2264895d851d8d28450f7d if anyone else finds this 🙂


Enable session specific logging

From @martindsouza on January 26, 2013 14:31
Ability to set the configuration specific to a session. Odds are use client identifier so APEX sessions can leverage it.
Copied from original issue: tmuth/Logger---A-PL-SQL-Logging-Utility#4

From @tmuth on January 26, 2013 15:26
This is something a lot of people have asked for and a great addition.  I'd like to benchmark the impact and potentially have the ability to completely disable it if there are negative performance implications.
Thanks,
Tyler
On Jan 26, 2013, at 9:31 AM, giffyd<EMAIL_ADDRESS>wrote:

Ability to set the configuration specific to a session. Odds are use client identifier so APEX sessions can leverage it.
—
Reply to this email directly or view it on GitHub.

One idea I have is to create another table to store the sessions that you want logged. What it could look like is client_identifier, level, expiry_date. In the ok_to_log we can also reference this table. Since function result_cache is enabled it could have a minimal impact on performance provided not too many changes are made to the client_id table.

From @tmuth on January 28, 2013 15:47
Possibly add new flag to enable it, but not as a new logger_level. Similar to SCN. client_identifier, PL/SQL call.


Should not call new Date

The whole point of moment is that it normalizes new Date() calls. Instead replace the calls with moment(). All operations should be moment operations.
For example new Date('January 1, 1900') becomes moment( [1900, 1, 1] ) or moment( '1900-01-01', 'YYYY-MM-DD');

Mm unfortunately this would place limitations on the date formats you could use when instantiating the calendar but I could and will do better on not using new Date when I don't need to.

https://github.com/Baremetrics/calendar/commit/ef9bd8cedca66ffb268f6776a2820d46995ba03a

On second thought I think I'd like to limit the date input to that format for consistency's sake.
https://github.com/Baremetrics/calendar/commit/22e0256e0b76d6f49085eb59ee1b20474ef29e55

You should probably limit to a format, not necc. that format, with regards to #26 #2

Opening this back up.. I didn't like my implementation. It works for me as is but if someone wants to knock it out that's fine by me.

I'll look into it :)
On Fri, 4 Sep 2015 13:45 Tyler van der Hoeven<EMAIL_ADDRESS>wrote:

Opening this back up.. I didn't like my implementation. It works for me as
is but if someone wants to knock it out that's fine by me.
—
Reply to this email directly or view it on GitHub
https://github.com/Baremetrics/calendar/issues/28#issuecomment-137803067
.


Frontend: text not centered in mobile

On this @Bonnee very secure site, if you are using mobile, some values are centered and not left aligned.

We want to give hackers the best user experience when they breach our systems

Hahaha


Features and improvements

Hi @amir1376 and contributors,
Great app, finally an open source download manager. There aren't many good ones left.
Here are some feature and improvement suggestions for future versions, let me know what you think:
Localization
The app is only available in English at the moment as it's a new project. But localization should be considered for the future as it will bring in more users and popularize ABDM. You can set up a Crowdin or Weblate translation panel and the growing community can contribute in the translation of the app and extensions in many languages. Both platforms are free for open source projects, just need to submit.
Auto-download option
With browser extension integration, clicking on a download link always shows an "add download" window first. Adding an option to hide add download window and directly start download with download progress window would be great. Also auto-centering those windows would be nicer visually.
Browser extension
The browser extension allows to add specific file extensions that will automatically be downloaded with AB. Which I think is not very efficient as you have to manually add formats you want. I think changing that option to enter the file extensions you DON'T want AB to download is better. As there will be fewer formats you don't want and all the rest will be intercepted by AB.
Also adding a website blacklist would be great, the extension will not intercept any downloads on that custom website list.
Some websites block download managers (1fichier, rapidgator for example) so a website blacklist is needed to add them in.
Remember window size and position
The app always open in the same spot regardless if you moved it previously. Adding a feature that remembers window size and position would be great. It will always open in the last size and position you left the app window in.
File categories (left panel)
Not really important but adding an option to hide the side panel with "All, FInished and unfinished + file categories" would be nice. I never use those and it adds a customization option for people that don't need it and want to simplify the interface. If hidden, the main window always shows all downloads (in progress, finished and unfinished).
You can slide the main window left to hide it, but it's not really hidden and comes back at every app launch.
Auto-updater
Last but not least, an in-app auto-update feature. You can choose for the app to automatically check for new updates or manually only if the user goes to Help > About for example.

Thank you very much I will consider your suggestions in the future

Great, some features/improvements I mentioned are easier to implement than others. I look forward to the project's future.

Hi @amir1376, now that localization is successfully implemented I edited the issue and ajusted some other improvements. I tried to regroup the best overall improvements in this issue for easier tracking on my part but also for you.


Possibly: additional backends

Currently we support gam and bam. In my mind these are the big two. Potential additional options are mgcv::ginla() or brms::brm(), but I don't have a clear vision for these yet.
The questions to resolve here are:

Who would the additional backend be for? What's the use-case? What workflows would we try to support?
How would they mechanically be implemented? The lift would be pretty light for an additional mgcv function but brms would likely take a bit more rejiggering. Is it enough to switch to an internal S3 class structure? Do we need to think more about a public model fitting function? I'm resistant to it, but would listen if we could open up some important missing functionality. Obviously the answer to (2) is downstream of (1).

In a world where you wanted to generalise this to more complex models then I think brms support would be good but if we are happy shutting the door on that (without major reworking) i think its fine it make a statement that this is a mgcv extension package and work from there

I'm definitely not opposed in principle, but I think we'd need a clear vision of the scenarios we want to enable and what it means for functionality vis a vis EpiNow2/Epinowcast. If we go full bayes, why not implement it as a forward renewal model? If we do, how is this different than a re-implementation of EpiNow2? If we don't, we aren't we implementing the current state of the art?
Basically, I think we should have a clear idea of what we're doing here and when we want to hand things off to the Sam Abbott Epi* universe. I think there could be a compelling case for joint delay estimation and forward convolution, which will be a major weakness here. I don't know where to draw the line though, and that makes me uneasy.

An additional thought here: it may be worth exploring gamm and/or gamm4. I imagine the many thousands of timeseries case will be challenging and computationally intensive and we could squeeze out some better performance with the lme4 backend vs the standard nlme.

Dropping this here for more thought on gamm/gamm4

Chiming in:
I think the main goal here is to have something we could possibly pull off the shelf and/or share with partners, ideally before the end of the calendar year. I think it's important to focus on one thing that works, rather than generality and flexibility right now.
With that in mind...

Who would the additional backend be for? What's the use-case? What workflows would we try to support?

I think the main use case should be to support alternative backends if they help maintain stability and reasonable speed across many signals, including for large counts. I think that major extensions/generalizations of the methods (e.g. moving away from spline-based methods) is out of scope here.
Following on that, I could see trying to support gamm4() for the lme4 backend and maybe mgcv::ginla(), as these are backends that might help with speed or convergence without much of a lift. From the `gamm' documentation, it's not clear to me that the lme backend would really help with speed or stability unless we stick entirely to identity link functions.

How would they mechanically be implemented? The lift would be pretty light for an additional mgcv function but brms would likely take a bit more rejiggering. Is it enough to switch to an internal S3 class structure? Do we need to think more about a public model fitting function? I'm resistant to it, but would listen if we could open up some important missing functionality. Obviously the answer to (2) is downstream of (1).

I think we should skip brms. Do you still need to talk through implementation for the other two? If it's going to be a big lift there, I could even see sticking with gam and bam and calling it a day.

I think we should skip brms. Do you still need to talk through implementation for the other two?

To focus this issue I don't think it's about which backends to eventually include. I think it's about should the package be designed now to handle backends from different packages (with a shared interface) or just limit to mgcv.
For the reasons already given I think it's compelling to design now for multiple backends but not actually implement any of them any time soon

including for large counts.

I don't think the size of counts interacts with the current proposed model methodology so any backend should scale well here.****

convergence

I'm not sure they will help with this but they will offer other modelling options and maybe more stable than mgcv in some circumstances (noting stable != Converged or reliable imo).

convergence

I'm not sure they will help with this but they will offer other modelling options and maybe more stable than mgcv in some circumstances (noting stable != Converged or reliable imo).

. I think it's important to focus on one thing that works, rather than generality and flexibility right now.

I think this is the conversation to have. If you want at any point to think about other package backends I think will save a lot of pain to do the tooling to allow that seamlessly now (otherwise many breaking changes . In most instances we are taking about things like allowing users to pass ... args or overwrite hard coded internal settings.
If you don't want to support beyond mgcv, which I think is a totally reasonable place to set the scope then as is will work quite well (with a little more user options imo). The only issue is changing from this position has the most negative impact of available choices once a lot of work has been done

For the question of which backends maybe they need their own issues and potential schedule. I do think there are very compelling reasons to think about brms here but also see the counter argument about scope creep.

Seee #34 and #35 for specific backend issues.


[11.x] Add the ability to bind keys together in translation 🚀

This PR introduces a significant enhancement to Laravel's translation capabilities by facilitating the linking of keys within an array of language files. For instance, if you have multiple keys with the same text, this feature allows for concise representation.
Previously, you would handle such scenarios like this:
// Resources/lang/en/messages.php

return [
    'example' => [
        'hello' => 'Hello',
        'world' => 'World!',

        // Or, with a duplicate text with the same meaning
        'hello_world' => 'Hello World!'
    ]
];

// Resources/views/example.blade.php

{{ trans('messages.example.hello').trans('messages.example.world') }}

// Or
{{ trans('messages.hello_world') }}

With this enhancement, the process becomes more streamlined:
// Resources/lang/en/messages.php

return [
    'example' => [
        'hello' => 'Hello',
        'world' => 'World!',
        'hello_world' => '@{hello} @{world}'
    ]
];

// Resources/views/example.blade.php

{{ trans('example.hello_world') }}

This improvement not only simplifies the code but also enhances readability and maintainability.

Thanks for your pull request to Laravel!
Unfortunately, I'm going to delay merging this code for now. To preserve our ability to adequately maintain the framework, we need to be very careful regarding the amount of code we include.
If possible, please consider releasing your code as a package so that the community can still take advantage of your contributions!
If you feel absolutely certain that this code corrects a bug in the framework, please "@" mention me in a follow-up comment with further explanation so that GitHub will send me a notification of your response.


Paste HTML in editor seems not working

Describe the bug
When i paste HTML content in tui.editor, the content is raw text.
To Reproduce
Steps to reproduce the behavior:

Copy some HTML content from a web page
Go to https://nhn.github.io/tui.editor/latest/tutorial-example15-editor-with-all-plugins
Paste content in editor
you see no markdown translation, only raw text.

Expected behavior
I expect HTML content to be parse and transform into markdown syntax as README.md says :

Copy and Paste : Paste anything from browser, screenshot, excel, powerpoint, etc.

Desktop (please complete the following information):

OS: Windows 10
Browser Chrome
Version 81

@StephaneRavet I think this is normal behavior, not a bug. If you copy and paste the html data, it is correct that the html comes out as the WYSIWYG editor handles the html. However, the Markdown editor treats the Markdown syntax as a string and converts the Markdown text to Html to show in the preview. So, I think that the data pasted into the Markdown editor comes out as raw text.

Ok, i founded a work-around for my need : use a browser extension.
https://chrome.google.com/webstore/detail/copy-as-markdown/dgoenpnkphkichnohepecnmpmihnabdg
Thanks for reply and thanks for your great work ! 👏


Ran npm run format

Correcting the files so they don't clutter new repos with unnecessary commits #430

@prototypa, I would advise against merging this pull request, but would more than welcome the formatting of all files. It would be more convenient to manually execute the command npm run format on your end, rather than having to inspect 700 changes individually for any potential malicious alterations. As per #415

Concur with your security concerns!
Hence why I kept the pull request as just the formatting.

Hi @ShoGinn
As @JuliusBairaktaris mentioned, there are a lot of files to review. I'm going to close this PR and run the npm run format myself.
I am also going to work on automating that process. If you have any suggestions, please let me know.
Thank you.


"Hole" in hour-of-the-day on tab 1 data

We can see that we have datapoint for hour 17 and hour 16, but they are not connected.
This could be because the data begins at hour 16 of the first day, and ends at hour 17 of the last day.
Perhaps we could Clip/remove the first "partial" day in the dataset so that our plots don't appear to have holes and loops.

We could look into adding in the floor plan of the house here too.


disableBack in $ionicHistory.nextViewOptions fails sometimes.

Hello,
I created the below view by using the 'stripe styled tabs'.
I wanted to mimic the sub tab functionality of other android apps. but couldnt find any docs on it in ionic so I created a three different routes for the tabs and just change the 'active' class for the selected tab.
What I noticed is that since I am just simulating sub-tabs the history should not change and the back button shouldn't be visible.
So, I added the following in all the tab routes controllers
$ionicHistory.nextViewOptions({
    disableBack: true,
    disableAnimate: true
});

While interchanging/navigating between the tabs, the above code works for 4-5 times, but then, the normal history enabled behavior starts to take place, the view starts animating and the back button starts appearing.

Greetings @hussainb!
My sensors indicate that you need to update your issue through our custom issue form. We are now requiring all issues to be submitted this way, to ensure that we have all of the information necessary to fix them as quickly as possible.
Click Here To Update Your Issue
I will have no choice but to close this issue if it is not resubmitted through the form. Please fill out the rest of the form, so that I may use my friendly robot powers to assist you.
Thank you!

Hi,
I have a simple login based application in which user can access About menu with or without login, issue came with android hardware back button,
Login screen > About page > From Menu go to Login > Press hardware back button > Application Should get closed rather go to About page.
Below is the code for navigation to login screen.
$scope.login = function() {
            $ionicHistory.nextViewOptions({
                disableBack: true,
                disableAnimate: true,
                historyRoot: true
            });
            $ionicHistory.clearCache();
            $ionicHistory.clearHistory();
            $ionicSideMenuDelegate.toggleLeft();
            $state.go('login');
        };

I am using ionic rc release 4 currently.

Could you put these in a codepen or something we can inspect?

Greetings @hussainb!
My sensors indicate a reply was requested, but we have not received one yet, I am using my robot powers to close this issue. However, if you are still experiencing this issue, please feel free to reopen it by creating a new issue, and include any examples or necessary information.
Thank you for allowing me to assist you.

Greetings @hussainb!
My sensors indicate a reply was requested, but we have not received one yet, I am using my robot powers to close this issue. However, if you are still experiencing this issue, please feel free to reopen it by creating a new issue, and include any examples or necessary information.
Thank you for allowing me to assist you.

I recently encountered this issue, the way I solved it was changing historyRoot to false:
$ionicHistory.nextViewOptions({ disableBack: true, disableAnimate: true, historyRoot: false })
to
$ionicHistory.nextViewOptions({ disableBack: true, disableAnimate: true, historyRoot: true })


Autocomplete not completing with mouse click on touchscreen on Chrome

This is a (multiple allowed):

[x] bug

[ ] enhancement

[ ] feature-discussion (RFC)

Framework7 Version: v1.5.2

Platform and Target: Issue present on Chrome on touchscreen PCs

What you did
When using an autocomplete on a touchscreen PC, clicking on the desired item does not complete the field, rather it leaves only what was typed in the input. The feature functions as expected when the item is selected on the touchscreen however.
The issue is not present on devices without touchscreens, and is not present on Firefox either way. Not tested on Safari or other browsers.
Expected Behavior
Using the example here:
if the user types “ban”, the option “Banana” is in the dropdown. If the user clicks "Banana" with the mouse or touches it on the touchscreen, the field should complete with "Banana".
Actual Behavior
If the user clicks “Banana” with the cursor, the field seems to complete with “ban”, and the dropdown disappears. If the user touches “Banana” on the touchscreen, it is completed as desired.

Hi, works for me using Chrome on iOS. Are you on Android?

Works for me (Chrome on Win10 32Bit)

@vanlub @ZanderBrown
This was on Linux, but my team members reported it on Windows as well. Again, only on machines with touchscreens when using the mouse.

Ah so clicking not tapping. Okay you had me confused their.
Okay so clicking doesnt work.

Mouseclicks not working on a touchscreen. Weirdest thing I've ever head of :D

I have the same issue, but not with a touchscreen. Windows 10 desktop.

Issue is closed because of outdated/irrelevant/not actual
If this issue is still actual and reproducible for latest version of Framework7, please create new issue and fill the issue template correctly:

Clearly describe the issue including steps to reproduce when it is a bug.
Make sure you fill in the earliest version that you know has the issue.
Provide live link or JSFiddle/Codepen or website with issue

I am also facing the same issue now.  Can you please clearly mention in which version it is fixed.


Links in a collection - require a link to license

License for usage of the data.
Open question: should we require unified links to known licenses (for example CC), or allow links to any random license?

To make it easier to understand this issue. This is how UKMO test server is displaying license in links section of collection.
        {             "href": "https://creativecommons.org/licenses/by-nc-nd/4.0/",             "hreflang": "en",             "rel": "licence",             "type": "text/html"         },

@mrauhala What do you think? Is it ok to include public domain there? Are there other licenses that should be there?
Yes public domain is ok, I guess other variants of cc by are needed as well.

@mrauhala Are there any constraints through EU Data directive and/or EUMETNET and WMO relevant to this list of licenses?

@mrauhala Are there any constraints through EU Data directive and/or EUMETNET and WMO relevant to this list of licenses?

HVD Implementation Act states about license:

under the conditions of the Creative Commons BY 4.0 licence or any equivalent or less restrictive open licence;

Reference: https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=uriserv:OJ.L_.2<IP_ADDRESS>3.01.ENG

Great! I interpret that as it will be enough to list those two licenses in our profile as well?

Our data are actually governmentally required to be dual-licenced (currently CC BY 4.0 plus NLOD 2.0). Not sure if you can specify more than one licence in EDR?


use AssocQueryString instead of directly querying the registry

Pull request checklist:

[x] CHANGELOG.md was updated, if applicable

The existing getCommand() in IncognitoBrowser.cpp directly queries undocumented registry keys which could break if Windows decides to change the registry layout (as they have in the past.)
This PR changes getCommand() to instead call AssocQueryString, which smooths out differences between versions of Windows.

Overall the functionality of this change looks good, just some refactoring to leave the windows-specific implementation in its own file would be nice.

I also plan to implement linux support (well, freedesktop support) for this in a separate PR. Should I create a new XDGHelper or LinuxHelper file for that?

Overall the functionality of this change looks good, just some refactoring to leave the windows-specific implementation in its own file would be nice.

I also plan to implement linux support (well, freedesktop support) for this in a separate PR. Should I create a new XDGHelper or LinuxHelper file for that?

Yes, for that sort of functionality additional OS-specific helper files in the same directory is fine (i.e. LinuxHelpers.cpp/hpp)

@sheybey I pushed two commits, are you ok with a merge after the changes I made?

@sheybey I pushed two commits, are you ok with a merge after the changes I made?

Looks good to me


remove accidental duplication

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 81.61%. Comparing base (340dbeb) to head (97fbbf9).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #271      +/-   ##
==========================================
+ Coverage   81.48%   81.61%   +0.13%
==========================================
  Files          77       77
  Lines        2355     2350       -5
==========================================
- Hits         1919     1918       -1
+ Misses        436      432       -4

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Expose serverSapienAI (see description)

This time just one local function which could be exposed: createGeneralOrder

Exposed in <IP_ADDRESS>


The Client hasn't been generated yet

Bug description
Running 'prisma generate' or 'prisma db push' works fine but whenever I try to run my code in PyCharm I get the following output:
Traceback (most recent call last): File "D:\Users\badet\Desktop\cloudbot\main.py", line 9, in <module> from prisma import Prisma File "<frozen importlib._bootstrap>", line 1075, in _handle_fromlist File "D:\Users\badet\PycharmProjects\1\venv\lib\site-packages\prisma\__init__.py", line 50, in __getattr__ raise RuntimeError( RuntimeError: The Client hasn't been generated yet, you must run prisma generate before you can use the client. See https://prisma-client-py.readthedocs.io/en/stable/reference/troubleshooting/#client-has-not-been-generated-yet Process finished with exit code 1
my main.py:
import datetime
import os

from termcolor import colored
import ts3
import mariadb
import sys
import asyncio
from prisma import Prisma
from prisma import Client

# Initialize Prisma client
prisma = Client()

# Authenticate your bot with Prisma
bot_email =<EMAIL_ADDRESS># Replace with your bot's email address
bot_password = "GnbacoapJQ5MQ43#"   # Replace with your bot's password

# Assuming you have a User model in your Prisma schema
authenticated_bot = prisma.user.find_first(where={"email": bot_email, "password": bot_password})

if authenticated_bot:
    bot_id = authenticated_bot.id
    print(f"Bot authenticated with ID: {bot_id}")

How to reproduce
Steps to reproduce the behavior:

prisma generate or prisma push db
run code

Expected behavior
I expected my code to start like usually.
Prisma information

datasource db {
  provider          = "postgresql"
  url               = env("DATABASE_URL")
}

generator client {
  provider = "prisma-client-py"
  recursive_type_depth = 5
  interface  = "asyncio"
}

model users {
  id            Int     @id @default(autoincrement())
  uid           String
  createdAt     DateTime @default(now())
  nickname      String
  trustfactor   Int
}

Environment & setup

OS: Windows 11
Database: Supabase PostgreSQL
Python version: 3.10.4
Prisma version: 5.4.2

For some reason changing the Python interpreter fixed the issue...


canary not working with custom domain.

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[x] This is not a question about how to use Istio

Bug Description
In our setup we use flagger with gateway api for canary rollout. so fagger canary object with create an httproute object for traffic split.
`
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
annotations:
helm.toolkit.fluxcd.io/driftDetection: disabled
ingress.kubernetes.io/service-upstream: "true"
kustomize.toolkit.fluxcd.io/reconcile: disabled
creationTimestamp: "2024-11-01T06:09:28Z"
generation: 61
labels:
app: echo
app.kubernetes.io/instance: echo-dev-sre
heritage: Helm
release: echo
name: echo
namespace: echo
ownerReferences:

apiVersion: flagger.app/v1beta1
blockOwnerDeletion: true
controller: true
kind: Canary
name: echo
uid: 1a7d73f1-8943-4c78-ac1a-59f71229a4c6
resourceVersion: "945980622"
uid: 0c0736b9-0d0b-4a1e-9d4e-246008dea28a
spec:
parentRefs:
group: ""
kind: Service
name: echo
rules:
backendRefs:

group: ""
kind: Service
name: echo-primary
port: 8080
weight: 50
group: ""
kind: Service
name: echo-canary
port: 8080
weight: 50
matches:
path:
type: PathPrefix
value: /
timeouts:
request: 5s
status:
parents:

conditions:

lastTransitionTime: "2024-11-01T06:09:28Z"
message: Route was valid
observedGeneration: 61
reason: Accepted
status: "True"
type: Accepted
lastTransitionTime: "2024-12-05T07:32:43Z"
message: All references resolved
observedGeneration: 61
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
controllerName: istio.io/gateway-controller
parentRef:
group: ""
kind: Service
name: echo

conditions:

lastTransitionTime: "2024-12-09T10:53:39Z"
message: ""
reason: Accepted
status: "True"
type: Accepted
parentRef:
group: core
kind: Service
name: echo
namespace: echo
`

calling this service with echo.echo.svc.cluster.local:8080 working as expected.. traffic split 50-50 between primary and canary.
But we want to use custom domain like echo.dev-services.com . so we created a serviceenty for it:
`apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
name: echo-serviceentry
namespace: echo
spec:
hosts:

"echo.dev-services.com"
location: MESH_INTERNAL
ports:
number: 80
name: http
protocol: HTTP
resolution: DNS
endpoints:
address: echo.echo.svc.cluster.local
ports:
http: 8080`

calling echo.dev-services.com always sending traffic to primary and canary is not working in this case. I envoy config the weights are configure in route_config as:
      - domains:           - echo.echo.svc.cluster.local           - echo.echo           - echo.echo.svc           - <IP_ADDRESS>         include_request_attempt_count: true         name: echo.echo.svc.cluster.local:8080         routes:         - decorator:             operation: echo-0-istio-autogenerated-k8s-gateway:8080/*           match:             case_sensitive: true             prefix: /           metadata:             filter_metadata:               istio:                 config: /apis/networking.istio.io/v1alpha3/namespaces/echo/virtual-service/echo-0-istio-autogenerated-k8s-gateway           name: echo.echo.0           route:             cluster_not_found_response_code: INTERNAL_SERVER_ERROR             max_grpc_timeout: 5s             retry_policy:               host_selection_retry_max_attempts: "5"               num_retries: 2               retriable_status_codes:               - 503               retry_host_predicate:               - name: envoy.retry_host_predicates.previous_hosts                 typed_config:                   '@type': type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate               retry_on: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes             timeout: 5s             weighted_clusters:               clusters:               - name: outbound|8080||echo-primary.echo.svc.cluster.local                 weight: 50               - name: outbound|8080||echo-canary.echo.svc.cluster.local                 weight: 50
We want to have canary (httproute) respected for custom domains incluster as well as cross-cluster.  how can we achive it and whats missing here.
Version
client version: 1.23.2
control plane version: 1.23.2
data plane version: 1.23.2-tetrate2 (20 proxies)

kubectl:
Client Version: v1.31.0
Kustomize Version: v5.4.2
Server Version: v1.31.2-eks-7f9249a

Additional Information
No response

The flagger docs indicate that flagger adjusts HTTPRoutes attached to a Gateway object, but the route you shared is attached to a service.  Was this route created by flagger?  If so, can you share the flagger canary object?
https://docs.flagger.app/tutorials/gatewayapi-progressive-delivery


[DS-3783] Change http return code when required parameter is missing

https://jira.duraspace.org/browse/DS-3783
Note: My intention was to not include the stack trace in the return message, but it is still being returned.
Questions: rather than 422, should we use a code known to HttpServletResponse?
Are we comfortable having all IllegalArgumentException instances generating this return code?

From the RFC:
The 422 (Unprocessable Entity) status code means the server
understands the content type of the request entity (hence a
415(Unsupported Media Type) status code is inappropriate), and the
syntax of the request entity is correct (thus a 400 (Bad Request)
status code is inappropriate) but was unable to process the contained
instructions.  For example, this error condition may occur if an XML
request body contains well-formed (i.e., syntactically correct), but
semantically erroneous, XML instructions.
My initial instinct would have been to go straight for 400 as 422 wasn't on my radar, but if a missing required parameter meets the description of "well formed by semantically incorrect", then it should be fine. I think I'd interpret that as eg. all parameters are there, but one of them refers to a string instead of an int, or something?

I prefer 422 as it is a bit more specific than 400. The same question on stacktrace also show a similar preference https://stackoverflow.com/questions/3050518/what-http-status-response-code-should-i-use-if-the-request-is-missing-a-required

FWIW, this implementation looks reasonable to me, @terrywbrady.  I'd look to @abollini though to provide his feedback as well.
I do see @abollini's point about possibly creating an extension of @Param to avoid having to do if (param==null} checks throughout the code. It would be cleaner/easier overall. But, I'm not certain how frequently we'll be using @Param (this seems to be the first usage in the codebase).
As for your question about @RequestParam('name=[my-param], required=true'), it would be nice if @Param supported a required setting by default, but it doesn't :(  https://docs.spring.io/spring-data/jpa/docs/1.4.3.RELEASE/reference/html/jpa.repositories.html#jpa.named-parameters And, if it isn't clear, @RequestParam and @Param are unrelated (despite the similar names).
In case it helps, @terrywbrady, we do have another custom annotation that can help you learn/understand how this might be done via a custom annotation. In fact, that @SearchRestMethod(name = "subCommunities") annotation on that same method is custom.  Here's the basics of how it works:

The entire @SearchRestMethod annotation is defined here: https://github.com/DSpace/DSpace/blob/master/dspace-spring-rest/src/main/java/org/dspace/app/rest/SearchRestMethod.java

This annotation is likely as clear as mud by itself. But, if you look at a guide for creating custom annotations, it might make more sense: https://dzone.com/articles/creating-custom-annotations-in-java
One of the key points here is that the annotation on it's own does nothing.  It's basically like a "note" attached to this method that says "this method defines the SearchRestMethod's 'name' property to be subCommunities.

The "magic" of where this annotation is used is in our RestResourceController.executeSearchMethods() method (which is triggered whenever the /search path is called): https://github.com/DSpace/DSpace/blob/master/dspace-spring-rest/src/main/java/org/dspace/app/rest/RestResourceController.java#L805

Specifically, look at this line: https://github.com/DSpace/DSpace/blob/master/dspace-spring-rest/src/main/java/org/dspace/app/rest/RestResourceController.java#L819
That line takes in the passed in searchMethodName (from the path), and calls this method: https://github.com/DSpace/DSpace/blob/master/dspace-spring-rest/src/main/java/org/dspace/app/rest/utils/RestRepositoryUtils.java#L94
And the RestRepositoryUtils.getSearchMethod() class literally looks for the method (in our API) that contains the @SearchRestMethod annotation with the given name set to the value of searchMethodName.

Annotations are a bit tricky to get used to...but, once created, they make new development a ton easier.  For instance, we could have created a method to parse the searchMethodName and call the proper classes/methods based on a giant if statement. But, instead, this annotation lets us more easily note which method to call for specific values of searchMethodName.

The use of the annotation provides a further benefit. It will allow us to "know" what the method expect and using this information we will be able to create the right link (a template link). It will be used also in the implementation of the ALPS support.
We already have a couple of custom annotation implemented in the sourcecode, see for instance org.dspace.app.rest.model.LinkRest

This should fix the above concerns https://github.com/4Science/DSpace/tree/Georgetown-University-Libraries-ds3783a
I will wait tomorrow to check the Travis result


Allow Sylius 1.9

Use MYSQL on our plugin to avoid migrations diff due to Maria DB : https://github.com/Sylius/Sylius/pull/12413
Allow Sylius ^1.8

Thanks a lot 🌯


Ability to pass in version argument to ElasticsearchTypeMixin.bulk_index method

The idea is to pass in a version argument in order to alter the results of get_document, get_es, & possibly get_type_name. The best way I see this working out is to add the argument with default set to None.
Adding such an enhancement would allow anyone to facilitate an ES transition/migration as they needed it.

Sorry, but I don't understand the use case for this? Can you elaborate?
(Apologies for the late reply!)

@jaddison Specifying a version allows for the flexibility to sync to potentially more than one endpoint with different mappings. The goal is to reduce boilerplate needed facilitate such a feature.

@avelis I still don't understand the feature - but feel free to submit a PR demonstrating the feature and its use case. The PR wouldn't need to be 'complete' or fleshed out, but enough to demonstrate your intentions. Hopefully it wouldn't take more than a few minutes to put it together. If it makes sense (ie we can agree on a path forward) then it can be fleshed out more.
As you can tell, I'm hesitant to add anything that increases complexity or falls out of the 'natural' scope for the project. If it is indeed within scope and is (or could be?) a common usage, then by all means I would merge as long as it had appropriate docs and tests.

@jaddison I can see what I can do. To better illustrate where I am coming from I am linking a blog post that highlights their similar approach to re-indexing with zero downtime. This is what is driving the inspiration in this ticket.
ES almost always requires a re-index if a mapping change removes any fields in a given document. This proves challenging if your index is large and can impact a user experience if down. The article above essentially suggested a parallel syncing strategy. The DSE library provides the structure for get_mapping and the serialization method get_document. The challenge is that if two indexes have two different mappings how do I know which document dictionary to generate for which mapping without having some indicator as to what endpoint the document is going to. That is at the point of my suggestion for a version method argument.
However, I do understand that it moves out of the territory of "simple". Thanks again for making this library.

Ok, so I'll paraphrase what I think you're suggesting.
Even though DSE supports alias-based 'zero' downtime (indexing all data to a new index while the old one is running, then switching the alias to the new index name), there are still problems.

new codebase is running with the old index mappings/data (while the new index mappings/data are being indexed)

this is not really something that can be solved within DSE as it's much too application specific

data might change (created/changed/deleted) after reindexing data into a new index has started; currently it is possible for these changes to get missed

a crude/simplistic attempt to mitigate this exists, but a) it puts the onus on the developer to order their results accordingly and b), it won't catch deletes unless they're 'soft' deletes

You see the ability to write a single Model's queryset in parallel to two separate indexes (types, maybe) (through parameter-versioned get_mapping/get_document methods) as a way around these issues. I think I get it.
I think parameter-versioning get_mapping/get_document is one way to do it, but I'm not entirely sure how the system would know to reference both when indexing data normally.
In the long run, I think maybe a better approach is to use a 'router' paradigm:

a central registry (ie. the router) of what ElasticsearchTypeMixin-based class handlers map to which Django model

it would replace the ELASTICSEARCH_TYPE_CLASSES simplistic setting (maybe the logic around this setting can simply be modified...)
this would allow us to register multiple class handlers for each Django model

In a complex project migration scenario:

make model specific changes (additions/changes, only - no deletes)
a. no functional template/view/etc related changes would be included at this point that rely on these changes
add new ElasticsearchTypeMixin-based class handler that maps/indexes data in the new way
a. might need to modify the original ElasticsearchTypeMixin class handler depending on the model changes to maintain original index mapping/data consistency
b. add the new ElasticsearchTypeMixin class handler to the DSE router for the model in question
probably need to manually create the new index/mapping
deploy this
a. at this point, the old and new ElasticsearchTypeMixin class handlers should be adding, updating and deleting as necessary
run the reindex operation, targeting ONLY the index storing data for the new ElasticsearchTypeMixin class handler
update the rest of the code base to use the data/structure stored from new ElasticsearchTypeMixin class handler mapping
deploy this (only when the reindex is complete, obviously)
remove the old ElasticsearchTypeMixin based class handler from the router
make any final model changes to remove fields or other destructive changes that couldn't be done because the old ElasticsearchTypeMixin class handler required it
deploy this

I think between the blog article you linked and your descriptions, this covers it?

@jaddison Looks pretty thorough to me. I really appreciate you taking the time to digest the article and hear out my scenario. Not many libraries in the Django+ES space are tackling the project migration lifecycle for Python+ES.
With that said I realize this is not a small undertaking. I have greatly benefited from DSE for it's simplicity and continued upkeep. Thanks so much!

You're welcome, although I'll admit time doesn't always allow for timely responsiveness.
It's kind of a tough problem to solve in such a generic fashion - migration processes are usually quite specific to the platform/app/system. So while what is suggested above could be implemented, it's a big investment of time. I'm not certain of how much it would be used.
If you want to take a stab at a revamp to support this, I'd be game to include it, particularly if it was backward compatible.


Extra function arguments to scipy.integrate.quad_vec

Hello, thanks for the work on scipy!
I was looking at the docs/source for scipy.integrate.quad_vec and I noticed that there isn't a provision to provide arguments for the callable function we wish to integrate. For the scipy.integrate.quadrature, we have the args argument that we can pass to the function for such a behaviour. Is there a way to do that with quad_vec?
Thanks!

Thanks for the suggestion. In the meantime, you can pass the parameters by using functools.partial for instance.

Ah thanks! I forgot about partial, for some reason. I'll do that for now. But it'll be great to have this feature built-in too in future!

@tupui I can send a PR to add this argument. Should be straightforward.


change I/O to take one input CSV

CSV should include 'SMILES' and 'Name' cols. All other cols to be ignored but preserved for output.

https://github.com/adelenelai/classify_homologues/commit/1f5fbb1794536b6124a1f5dd6850729f43746c52


Allow Alternative Shell Image for the Pipeline Controller

Feature request
As it works now the shell images for the Tekton pipeline controller is hard coded (https://github.com/tektoncd/pipeline/blob/main/config/controller.yaml#L76C29-L76C54) and cannot be changed when using the script action. Ideally this image location could be overwritten to point to any image desired. This would be especially valuable in egress restricted environments where the type of external requests are restricted.
Use case
Instead of using the hard coded shell image (cgr.dev/chainguard/busybox@sha256:19f02276bf8dbdd62f069b922f10c65262cc34b710eea26ff928129a736be791) I would like to point to a different shell image that I am in control of on my local network.

If you use tekton operator to install, you can use a different image.

@chriscannon I am not sure I entirely follow. You can update the deployment (or the release.yaml) to use a different shell image. But no matter what, it doesn't have anything to do with the script feature. The script feature depend on a shell being available in the image used by the step where it is defined, it doesn't use the shell image.

@vdemeester I guess what's confusing to me then is that whewn I use the script feature I see it pull this image cgr.dev/chainguard/busybox@sha256:19f02276bf8dbdd62f069b922f10c65262cc34b710eea26ff928129a736be791. When I stop using the script feature it does not try to pull that image. Why does this happen if it's using the shell on the image I specify in the step?

@chriscannon yes, it uses the shell image to actually "write" the script (from the spec to the container "filesystem") in a place (/tekton/scripts) that is mounted in a given step. It is used as an init container to "prepare" things for the execution, but not for the step itself. Is that a bit clearer ?

I'm just going to update the image in the release.yaml.

@vdemeester, I'm having a similar issue. What's the best way to update the shell-image's to point at a different registry (works as a proxy, no access to public registry directly)?


⚠ Samsung has degraded performance

In 557b128, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2333 ms

Resolved: Samsung performance has improved in f03ad7c.