train & test

Hello and thank you for your effort
I want to train and test a model using the embedded code
But in both cases, it gives me the following error:

Can you guide me?

Thank you for your interest in our work.
To resolve the error, please verify the number of .jpg files in your dataset folder of Market-1501. Ensure there are exactly 12936 files in "bounding_box_train", 19732 in "bounding_box_test", and 3368 in "query". Alternatively, download the dataset from here.

Thank you for your quick and effective reply
I fixed the previous problem, but now I have a problem with the model
Thank you for your guidance

raise ValueError(
ValueError: Layer count mismatch when loading weights from file. Model expected 0 layers, found 2 saved layers.

It might be a compatibility issue between your setup (Python 3.12 and likely the latest TensorFlow) and the code's original environment (TensorFlow 2.2.3 and Python 3.8). Here are two approaches to address this:

Use the code's original environment. Refer to the instructions here. However, this approach might not be feasible for newer GPUs.
Update the repository dependencies. Consider updating the repository to support the latest TensorFlow. You may train your own models as long as the final performance reflects the reported results.

thank you very much.
I installed tensorflow 2.14 and python 3.10.12.
The program has been executed up to the following amount
But it also gave an error that I don't understand where the problem comes from
I also think that the network parameters are a bit illogical, don't you think so?
And please advise about the error
Model: "training_model"

Layer (type)                Output Shape                 Param #   Connected to
input_11 (InputLayer)       [(None, 384, 128, 3)]        0         []
inference_model (Functiona  [(None, 2048),               7629920   ['input_11[0][0]',
l)                           (None, 1024),               0          'tf.image.flip_left_right[0][
(None, 1024)]                         0]']
tf.image.flip_left_right (  (None, 384, 128, 3)          0         ['input_11[0][0]']
TFOpLambda)
tf.operators.add (TFOp  (None, 2048)                 0         ['inference_model[0][0]']
Lambda)
tf.operators.add_2 (TF  (None, 1024)                 0         ['inference_model[0][1]']
OpLambda)
tf.operators.add_4 (TF  (None, 1024)                 0         ['inference_model[0][2]']
OpLambda)
tf.operators.add_1 (TF  (None, 2048)                 0         ['tf.operators.add[0][0]',
OpLambda)                                                           'inference_model[1][0]']
tf.operators.add_3 (TF  (None, 1024)                 0         ['tf.operators.add_2[0][0]
OpLambda)                                                          ',
'inference_model[1][1]']
tf.operators.add_5 (TF  (None, 1024)                 0         ['tf.operators.add_4[0][0]
OpLambda)                                                          ',
'inference_model[1][2]']
tf.math.truediv_2 (TFOpLam  (None, 2048)                 0         ['tf.operators.add_1[0][0]
bda)                                                               ']
tf.math.truediv_3 (TFOpLam  (None, 1024)                 0         ['tf.operators.add_3[0][0]
bda)                                                               ']
tf.math.truediv_4 (TFOpLam  (None, 1024)                 0         ['tf.operators.add_5[0][0]
bda)                                                               ']
classification_model (Func  [(None, 751),                3092480   ['tf.math.truediv_2[0][0]',
tional)                      (None, 751),                           'tf.math.truediv_3[0][0]',
(None, 751)]                           'tf.math.truediv_4[0][0]']
tf.convert_to_tensor (TFOp  (None, 2048)                 0         ['inference_model[1][0]']
Lambda)
tf.cast (TFOpLambda)        (None, 2048)                 0         ['inference_model[0][0]']
tf.math.squared_difference  (None, 2048)                 0         ['tf.convert_to_tensor[0][0]',
(TFOpLambda)                                                       'tf.cast[0][0]']
tf.convert_to_tensor_1 (TF  (None, 1024)                 0         ['inference_model[1][1]']
OpLambda)
tf.cast_1 (TFOpLambda)      (None, 1024)                 0         ['inference_model[0][1]']
tf.math.reduce_mean_3 (TFO  (None,)                      0         ['tf.math.squared_difference[0
pLambda)                                                           ][0]']
tf.math.squared_difference  (None, 1024)                 0         ['tf.convert_to_tensor_1[0][0]
_1 (TFOpLambda)                                                    ',
'tf.cast_1[0][0]']
tf.convert_to_tensor_2 (TF  (None, 1024)                 0         ['inference_model[1][2]']
OpLambda)
tf.cast_2 (TFOpLambda)      (None, 1024)                 0         ['inference_model[0][2]']
tf.math.reduce_mean_4 (TFO  ()                           0         ['tf.math.reduce_mean_3[0][0]'
pLambda)                                                           ]
tf.math.reduce_mean_5 (TFO  (None,)                      0         ['tf.math.squared_difference_1
pLambda)                                                           [0][0]']
tf.math.squared_difference  (None, 1024)                 0         ['tf.convert_to_tensor_2[0][0]
_2 (TFOpLambda)                                                    ',
'tf.cast_2[0][0]']
tf.operators.add_6 (TF  ()                           0         ['tf.math.reduce_mean_4[0][0]'
OpLambda)                                                          ]
tf.math.reduce_mean_6 (TFO  ()                           0         ['tf.math.reduce_mean_5[0][0]'
pLambda)                                                           ]
tf.math.reduce_mean_7 (TFO  (None,)                      0         ['tf.math.squared_difference_2
pLambda)                                                           [0][0]']
tf.operators.add_7 (TF  ()                           0         ['tf.operators.add_6[0][0]
OpLambda)                                                          ',
'tf.math.reduce_mean_6[0][0]'
]
tf.math.reduce_mean_8 (TFO  ()                           0         ['tf.math.reduce_mean_7[0][0]'
pLambda)                                                           ]
tf.operators.add_8 (TF  ()                           0         ['tf.operators.add_7[0][0]
OpLambda)                                                          ',
'tf.math.reduce_mean_8[0][0]'
]
add_metric (AddMetric)      ()                           0         ['tf.operators.add_8[0][0]
']
tf.math.multiply (TFOpLamb  ()                           0         ['tf.operators.add_8[0][0]
da)                                                                ']
add_loss (AddLoss)          ()                           0         ['tf.math.multiply[0][0]']
==================================================================================================
Total params: 79391680 (302.86 MB)
Trainable params: 40835072 (155.77 MB)
Non-trainable params: 38556608 (147.08 MB)

Summarizing inference_model_132322870669664 ...
Model: "inference_model"

Layer (type)                Output Shape                 Param #   Connected to
input_1 (InputLayer)        [(None, 384, 128, 3)]        0         []
preprocess_input (Function  (None, 384, 128, 3)          0         ['input_1[0][0]']
al)
features/init_block (Funct  (None, 96, 32, 64)           9664      ['preprocess_input[0][0]']
ional)
features/stage1 (Functiona  (None, 96, 32, 256)          218624    ['features/init_block[0][0]']
l)
features/stage2 (Functiona  (None, 48, 16,<PHONE_NUMBER>   ['features/stage1[0][0]']
l)
features/stage3 (Functiona  (None, 24, 8, 1024)          7118848   ['features/stage2[0][0]']
l)
features/stage4_regional_b  (None, 24, 8, 2048)          1498726   ['features/stage3[0][0]']
ranch (Functional)                                       4
lambda_1 (Lambda)           (None, 12, 8, 2048)          0         ['features/stage4_regional_bra
nch[0][0]']
lambda_3 (Lambda)           (None, 12, 8, 2048)          0         ['features/stage4_regional_bra
nch[0][0]']
conv2d (Conv2D)             (None, 12, 8, 1024)          1887539   ['lambda_1[0][0]']
2
conv2d_1 (Conv2D)           (None, 12, 8, 1024)          1887539   ['lambda_3[0][0]']
2
features/stage4_global_bra  (None, 24, 8, 2048)          1498726   ['features/stage3[0][0]']
nch (Functional)                                         4
activation (Activation)     (None, 12, 8, 1024)          0         ['conv2d[0][0]']
activation_1 (Activation)   (None, 12, 8, 1024)          0         ['conv2d_1[0][0]']
tf.math.maximum (TFOpLambd  (None, 24, 8, 2048)          0         ['features/stage4_global_branc
a)                                                                 h[0][0]']
tf.math.maximum_1 (TFOpLam  (None, 12, 8, 1024)          0         ['activation[0][0]']
bda)
tf.math.maximum_2 (TFOpLam  (None, 12, 8, 1024)          0         ['activation_1[0][0]']
bda)
tf.math.pow (TFOpLambda)    (None, 24, 8, 2048)          0         ['tf.math.maximum[0][0]']
tf.math.pow_2 (TFOpLambda)  (None, 12, 8, 1024)          0         ['tf.math.maximum_1[0][0]']
tf.math.pow_4 (TFOpLambda)  (None, 12, 8, 1024)          0         ['tf.math.maximum_2[0][0]']
tf.math.reduce_mean (TFOpL  (None, 2048)                 0         ['tf.math.pow[0][0]']
ambda)
tf.math.reduce_mean_1 (TFO  (None, 1024)                 0         ['tf.math.pow_2[0][0]']
pLambda)
tf.math.reduce_mean_2 (TFO  (None, 1024)                 0         ['tf.math.pow_4[0][0]']
pLambda)
tf.math.pow_1 (TFOpLambda)  (None, 2048)                 0         ['tf.math.reduce_mean[0][0]']
tf.math.pow_3 (TFOpLambda)  (None, 1024)                 0         ['tf.math.reduce_mean_1[0][0]'
]
tf.math.pow_5 (TFOpLambda)  (None, 1024)                 0         ['tf.math.reduce_mean_2[0][0]'
]
lambda (Lambda)             (None, 2048)                 0         ['tf.math.pow_1[0][0]']
lambda_2 (Lambda)           (None, 1024)                 0         ['tf.math.pow_3[0][0]']
lambda_4 (Lambda)           (None, 1024)                 0         ['tf.math.pow_5[0][0]']
==================================================================================================
Total params: 76299200 (291.06 MB)
Trainable params: 37750784 (144.01 MB)
Non-trainable params: 38548416 (147.05 MB)

Summarizing preprocess_input_132323024335632 ...
Model: "preprocess_input"

Layer (type)                Output Shape              Param #
input_2 (InputLayer)        [(None, 384, 128, 3)]     0
tf.math.truediv (TFOpLambd  (None, 384, 128, 3)       0
a)
tf.nn.bias_add (TFOpLambda  (None, 384, 128, 3)       0
)
tf.math.truediv_1 (TFOpLam  (None, 384, 128, 3)       0
bda)
=================================================================
Total params: 0 (0.00 Byte)
Trainable params: 0 (0.00 Byte)
Non-trainable params: 0 (0.00 Byte)

Summarizing features/init_block_132323154296304 ...
Model: "features/init_block"

Layer (type)                Output Shape              Param #
input_3 (InputLayer)        [(None, 384, 128, 3)]     0
conv (ConvBlock)            (None, 192, 64, 64)       9664
pool (MaxPool2d)            (None, 96, 32, 64)        0
=================================================================
Total params: 9664 (37.75 KB)
Trainable params: 0 (0.00 Byte)
Non-trainable params: 9664 (37.75 KB)

Summarizing features/stage1_132323035711712 ...
Model: "features/stage1"

Layer (type)                Output Shape              Param #
input_4 (InputLayer)        [(None, 96, 32, 64)]      0
stage1/unit_0_1 (ResUnit)   (None, 96, 32, 256)       76288
stage1/unit_0_2 (ResUnit)   (None, 96, 32, 256)       71168
stage1/unit_0_3 (ResUnit)   (None, 96, 32, 256)       71168
=================================================================
Total params: 218624 (854.00 KB)
Trainable params: 0 (0.00 Byte)
Non-trainable params: 218624 (854.00 KB)

Summarizing features/stage2_132323040862960 ...
Model: "features/stage2"

Layer (type)                Output Shape              Param #
input_5 (InputLayer)        [(None, 96, 32, 256)]     0
stage2/unit_1_1 (ResUnit)   (None, 48, 16, 512)       381952
stage2/unit_1_2 (ResUnit)   (None, 48, 16, 512)       281600
stage2/unit_1_3 (ResUnit)   (None, 48, 16, 512)       281600
stage2/unit_1_4 (ResUnit)   (None, 48, 16, 512)       281600
=================================================================
Total params: 1226752 (4.68 MB)
Trainable params: 0 (0.00 Byte)
Non-trainable params: 1226752 (4.68 MB)

Summarizing features/stage3_132323037229984 ...
Model: "features/stage3"

Layer (type)                Output Shape              Param #
input_6 (InputLayer)        [(None, 48, 16, 512)]     0
stage3/unit_2_1 (ResUnit)   (None, 24, 8, 1024)       1517568
stage3/unit_2_2 (ResUnit)   (None, 24, 8, 1024)       1120256
stage3/unit_2_3 (ResUnit)   (None, 24, 8, 1024)       1120256
stage3/unit_2_4 (ResUnit)   (None, 24, 8, 1024)       1120256
stage3/unit_2_5 (ResUnit)   (None, 24, 8, 1024)       1120256
stage3/unit_2_6 (ResUnit)   (None, 24, 8, 1024)       1120256
=================================================================
Total params: 7118848 (27.16 MB)
Trainable params: 0 (0.00 Byte)
Non-trainable params: 7118848 (27.16 MB)

Summarizing features/stage4_regional_branch_132323156030704 ...
Model: "features/stage4_regional_branch"

Layer (type)                Output Shape              Param #
input_7 (InputLayer)        [(None, 24, 8, 1024)]     0
unit_3_1 (ResUnit)          (None, 24, 8, 2048)       6049792
unit_3_2 (ResUnit)          (None, 24, 8, 2048)       4468736
unit_3_3 (ResUnit)          (None, 24, 8, 2048)       4468736
=================================================================
Total params: 14987264 (57.17 MB)
Trainable params: 0 (0.00 Byte)
Non-trainable params: 14987264 (57.17 MB)

Summarizing features/stage4_global_branch_132323034999696 ...
Model: "features/stage4_global_branch"

Layer (type)                Output Shape              Param #
input_7 (InputLayer)        [(None, 24, 8, 1024)]     0
unit_3_1 (ResUnit)          (None, 24, 8, 2048)       6049792
unit_3_2 (ResUnit)          (None, 24, 8, 2048)       4468736
unit_3_3 (ResUnit)          (None, 24, 8, 2048)       4468736
=================================================================
Total params: 14987264 (57.17 MB)
Trainable params: 0 (0.00 Byte)
Non-trainable params: 14987264 (57.17 MB)

Summarizing classification_model_132322843232480 ...
Model: "classification_model"

Layer (type)                Output Shape                 Param #   Connected to
input_8 (InputLayer)        [(None, 2048)]               0         []
input_9 (InputLayer)        [(None, 1024)]               0         []
input_10 (InputLayer)       [(None, 1024)]               0         []
batch_normalization (Batch  (None, 2048)                 8192      ['input_8[0][0]']
Normalization)
batch_normalization_1 (Bat  (None, 1024)                 4096      ['input_9[0][0]']
chNormalization)
batch_normalization_2 (Bat  (None, 1024)                 4096      ['input_10[0][0]']
chNormalization)
dense (Dense)               (None,<PHONE_NUMBER>   ['batch_normalization[0][0]']
dense_1 (Dense)             (None, 751)                  769024    ['batch_normalization_1[0][0]'
]
dense_2 (Dense)             (None, 751)                  769024    ['batch_normalization_2[0][0]'
]
activation_2 (Activation)   (None, 751)                  0         ['dense[0][0]']
activation_3 (Activation)   (None, 751)                  0         ['dense_1[0][0]']
activation_4 (Activation)   (None, 751)                  0         ['dense_2[0][0]']
==================================================================================================
Total params: 3092480 (11.80 MB)
Trainable params: 3084288 (11.77 MB)
Non-trainable params: 8192 (32.00 KB)

Traceback (most recent call last):
File "/content/FlipReID/solution.py", line 925, in
app.run(main)
File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/content/FlipReID/solution.py", line 789, in main
visualize_model(model=training_model, output_folder_path=output_folder_path)
File "/content/FlipReID/utils/vis_utils.py", line 23, in visualize_model
item for item in model._layers  # pylint: disable=protected-access
AttributeError: 'Functional' object has no attribute '_layers'. Did you mean: 'layers'?

AttributeError: 'Functional' object has no attribute '_layers'. Did you mean: 'layers'? is caused by the differences in TensorFlow. The provided code uses TensorFlow 2.2.3, rather than TensorFlow 2.14. Could you try using an environment the same as this?
visualize_model would plot the model, and it is not essential. You may comment out this line and check whether the program runs afterward.

I did what you said
It is running for the first epoch, but it is only running for the first epoch for about 1 hour and 40 minutes
without any results
Is it normal?

1 hour and 40 minutes sounds too long if steps_per_epoch is set to 200.
I remember that the training procedure is very efficient, and the GPU utilization rate should be around 100% most of the time. You may check the output of nvidia-smi.

I checked
It doesn't seem to use gpu, although it can recognize that gpu is available.
torch.cuda.is_available():True
torch.device:cuda
Initiating the image augmentor ...
Perform training ...
Freeze layers in the backbone model for 20 epochs.
Epoch 1: LearningRateScheduler setting learning rate to 2e-06.
Epoch 1/20
WARNING:tensorflow:From /usr/local/lib/python3.10/site-packages/tensorflow/python/util/deprecation.py:660: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
W0827 09:33:01.223699<PHONE_NUMBER>11296 deprecation.py:50] From /usr/local/lib/python3.10/site-packages/tensorflow/python/util/deprecation.py:660: calling map_fn_v2 (from tensorflow.python.ops.map_fn) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Use fn_output_signature instead
2024-08-27 09:33:13.213882: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 201326592 exceeds 10% of free system memory.
2024-08-27 09:33:13.590387: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 201326592 exceeds 10% of free system memory.
2024-08-27 09:33:14.429463: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 209780736 exceeds 10% of free system memory.
2024-08-27 09:33:14.468940: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 209780736 exceeds 10% of free system memory.
2024-08-27 09:33:14.759357: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 201326592 exceeds 10% of free system memory.

Can you provide compatibility in updated versions for the installed packages?

Due to current workload constraints, I'm unable to update the repository for the latest TensorFlow version at this time. The repository is provided in its current state. However, I'm committed to assisting you with any questions or issues you may encounter.
Before starting experiments with FlipReID, please verify that your environment is set up correctly. A straightforward way to do this is to try a simpler example like MNIST. This will help confirm that your GPU is working and TensorFlow is installed properly.
Once your environment is ready, you have two options:

Use the recommended environment. The repository should work out of the box. Keep in mind that older TensorFlow versions may have compatibility limitations with newer GPUs due to dependencies like CUDA.
Update the repository and use the latest TensorFlow. This shouldn't be too complicated. However, you might not be able to load the pre-trained weights I've provided. If this is the case, you can train the models from scratch.

Hello, dear engineer
I came again with a new question
I managed to run the code as far as traning A and traning B, but when saving the best model I get the following error:
Epoch 1: test_cosine_False_mAP_score improved from -inf to 0.03133, saving model to /content/FlipReID/output/Market1501_resnet50/training_model.h5
/usr/local/lib/python3.10/dist-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via model.save(). This file format is considered legacy. We recommend using instead the native Keras format, e.g. model.save('my_model.keras').
saving_api.save_model(
Traceback (most recent call last):
File "/content/FlipReID/solution.py", line 925, in
app.run(main)
File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 308, in run
_run_main(main, args)
File "/usr/local/lib/python3.10/dist-packages/absl/app.py", line 254, in _run_main
sys.exit(main(argv))
File "/content/FlipReID/solution.py", line 903, in main
training_model.fit(x=train_generator,
File "/usr/local/lib/python3.10/dist-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
ValueError: Unable to serialize VariableSpec(shape=(), dtype=tf.float32, trainable=True, alias_id=None) to JSON, because the TypeSpec class <class 'tensorflow.python.ops.resource_variable_ops.VariableSpec'> has not been registered.

You can follow the instructions in the log and try model.save('my_model.keras') instead of model.save().
You can search for TypeSpec class <class 'tensorflow.python.ops.resource_variable_ops.VariableSpec'> has not been registered in the TensorFlow repository. This issue is again due to the newer version of TensorFlow.


Implement Monitoring Tools to Ensure JWT Authentication Works Correctly

Discovery:
With the implementation of JWT authentication in LibreChat, it's important to set up monitoring tools to ensure the system functions as expected. Monitoring should be in place to track authentication successes and failures, performance bottlenecks, and potential security vulnerabilities related to JWT tokens. By setting up comprehensive monitoring, any issues with authentication or performance can be detected early and addressed before they impact the user experience.
Problem Definition:
Without proper monitoring, it is difficult to detect issues related to JWT authentication, such as token validation failures, performance degradation, or security vulnerabilities. Real-time logging and alerting are necessary to monitor the health of the system, especially under load, and to ensure that JWT tokens are handled securely and efficiently.
Motivations:
By implementing monitoring tools for JWT authentication, we can:

Detect and respond to issues in the authentication process in real time.
Ensure the performance and scalability of the system during peak loads.
Track potential security threats related to token misuse or validation failures.
Improve the overall stability and reliability of the authentication flow in LibreChat.

Proposed Solution:

Set Up CloudWatch Logs for ECS and JWT Authentication:

Enable Amazon CloudWatch Logs for the ECS service running LibreChat to capture detailed logs for each ECS task.
Ensure that logs include key information such as:

Successful JWT authentication attempts.
Failed authentication attempts (e.g., invalid tokens, expired tokens).
User creation events when a user.id is missing in the database.

Set up structured logging in the application code to output meaningful log messages related to JWT authentication.

Configure CloudWatch Metrics:

Define custom metrics in Amazon CloudWatch to track the performance of JWT authentication, including:

Number of successful logins via JWT.
Number of failed authentication attempts.
Token validation time (average time to validate a JWT token).
ECS task CPU and memory usage during authentication processes.

Create dashboards in CloudWatch to visualize key metrics and monitor trends over time.

Set Up Alarms and Alerts:

Create CloudWatch Alarms to trigger alerts when specific thresholds are exceeded, such as:

High number of failed authentication attempts (e.g., more than 10 failed attempts within 5 minutes, which could indicate a potential security threat).
Elevated ECS task CPU or memory usage, which could indicate performance bottlenecks in token validation.
High latency in JWT validation requests.

Integrate alarms with AWS SNS or other notification services to send real-time alerts via email or Slack when issues arise.

Monitor Security and Token Misuse:

Implement logging for security-related events, including:

Attempts to authenticate with invalid or expired tokens.
Replay attacks or suspicious behavior related to token usage.
Token renewal requests (if applicable) and the number of requests per user.

Set up CloudWatch alarms to notify the team of any unusual activity, such as a sudden increase in expired token usage or multiple invalid token submissions.

Integrate Application Performance Monitoring (APM):

Use a performance monitoring tool such as AWS X-Ray to trace authentication requests and monitor the entire flow of JWT token validation.
Trace the interactions between the front-end and the back-end to detect any performance issues or slowdowns in token validation.
Use X-Ray to identify bottlenecks in ECS service scaling or latency in user authentication.

Test Monitoring Setup:

Perform a series of tests to ensure that the monitoring setup is capturing relevant data, such as:

Triggering successful and failed authentication events to check if they are properly logged and counted in CloudWatch.
Stress-testing the ECS tasks to ensure the CPU and memory usage is being monitored accurately.
Simulating a high volume of authentication requests to verify that alarms are triggered correctly when thresholds are breached.

Definition of Done:

[ ] CloudWatch Logs are enabled for ECS, and detailed logging is implemented for JWT authentication events.
[ ] CloudWatch Metrics are set up to track key performance indicators for JWT authentication, including success/failure rates, ECS performance, and token validation times.
[ ] CloudWatch Alarms are configured to trigger alerts when authentication or performance thresholds are exceeded.
[ ] Monitoring and alerting systems are tested to ensure they work as expected under various scenarios, including both normal and peak traffic.
[ ] AWS X-Ray (or an equivalent tool) is integrated to trace JWT authentication requests and monitor the overall performance of the authentication process.

Additional Information:

Security Considerations: Ensure that logs do not expose sensitive information such as JWT tokens or user credentials. Only relevant metadata (e.g., token expiration time, validation status) should be logged.
Monitoring Tools: Consider additional monitoring tools like Datadog, New Relic, or Prometheus if the scope of monitoring expands beyond AWS CloudWatch and X-Ray.

#time 1
including authentication logs

#time 4
Logs, monitoring, alarms, metrics


[Bug]: GSheets - on creating a new query, the dropdowns size has increased and there is an overlap

Is there an existing issue for this?

[X] I have searched the existing issues

Description
The dropdowns size [Operation, Entity, etc ] has increased, and there is an overlap in the Spread Sheet, Sheet name, Table heading row index dropdowns
Steps To Reproduce
Create a new GSheets ds, and create a new query on it.
Note that the dropdowns sizes have increased.
Also there is an overlap in the Spread Sheet, Sheet name, Table heading row index drop-downs

Public Sample App
No response
Environment
Release
Issue video log
No response
Version
Cloud

DS page of S3, Mongo etc has also been impacted.

Please refer to the screenshot misplaced close icon


[FLINK-34401][docs-zh] Translate "Flame Graphs" page into Chinese

What is the purpose of the change
Translate Flame Graphs to Chinese
Brief change log
The page is located at "docs/content.zh/docs/ops/debugging/flame_graphs.md"
Verifying this change
Docs only change.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no) no
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no) no
The serializers: (yes / no / don't know) no
The runtime per-record code paths (performance sensitive): (yes / no / don't know) no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) no
The S3 file system connector: (yes / no / don't know) bo

Documentation

Does this pull request introduce a new feature? (yes / no) no
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@flinkbot run azure

@flinkbot run azure

Hi @JingGe , thank you for your advice, it is my first time participating in the Flink project. i have  added 2 commits with your suggestion, Please review it when you have free time, thank you.

Thanks @lxliyou001 for your contribution. I just left some comments.
Hi @JingGe , thank you for your advice, it is my first time participating in the Flink project. i have added 2 commits with your suggestion, Please review it when you have free time, thank you.

Hi @JingGe , thank you for your advice, it is my first time participating in the Flink project. i have added 2 commits with your suggestion, Please review it when you have free time, thank you.

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure

@flinkbot run azure

Could you rebase your branch?

Could you rebase your branch?

done, it that OK for you to merge this PR

hi @JingGe , I have done with it, is that OK for you to merge this PR? thanks a lot.

hi @JingGe , I have done with it, is that OK for you to merge this PR? thanks a lot.

sure, let's wait until the CI passed

@Zakelly 谢谢建议，还有这个pr没合入之前，每次master变更都要merge，然后merge又要过CI，CI耗时又很长，很是麻烦，我想问问CI在跑的过程中能合入吗

@Zakelly 谢谢建议，还有这个pr没合入之前，每次master变更都要merge，然后merge又要过CI，CI耗时又很长，很是麻烦，我想问问CI在跑的过程中能合入吗

如果是文档类的更改，确认不对CI造成影响的可以，committer可以合。master变更的话不用merge回你的更改分支哈，如果有更改可以继续提交commit即可。如果因为有冲突合不进去那需要进行rebase。

@Zakelly 谢谢建议，还有这个pr没合入之前，每次master变更都要merge，然后merge又要过CI，CI耗时又很长，很是麻烦，我想问问CI在跑的过程中能合入吗

如果是文档类的更改，确认不对CI造成影响的可以，committer可以合。master变更的话不用merge回你的更改分支哈，如果有更改可以继续提交commit即可。如果因为有冲突合不进去那需要进行rebase。

好的，非常感谢

@flinkbot run azure

@JingGe could you please merge this PR, thanks

@lxliyou001 sure, would you like to squash your commits?

@JingGe sure,could you help me do the "Squash and merge" action? thanks.


Cut a new release?

Looks like there have been a couple improvements that haven't yet been released

Just published 0.6.0. Thanks fo the reminder!


Map the limitations of lakeFSFS without multipart uploads

What are the implications of implementing #5355 without implementing multipart uploads?
What is lakeFSFS without multipart uploads, in each of the cloud providers?

Limitation for a single object upload:

For AWS and Azure: 5GB.
For GCP: 5TB (==no limit, although they also recommend chunking for big objects).


bare logs, recipe fix

Allow running codemod --logs which opens directory with logs
Fixes logic for downloading recipe codemods

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Parser Error: Can't use function '(' in math mode

Hey, thank you for forking this library!
I'm having this issue while rendering, any idea why?

That is a command for inline equations and is used in text mode. Do you intend any other command?

@Aniruddh-0701 I'm experiencing the same issue with inline math.
Math.tex(
          '\( M V E \)',
         mathStyle: MathStyle.text,
),

@Aniruddh-0701 I'm experiencing the same issue with inline math.
Math.tex(
          '\( M V E \)',
         mathStyle: MathStyle.text,
),

(...) group commands mean that the code enclosed by them should be rendered in math mode. It is used in text mode from inline equations. Its equivalent is $...$.
Math.tex() command only works on math mode. Hence, the command will not be recognized

still i got same essue on The value of \( x \) that satisfies the equation \( 5^{x-1} + 5^{x-1} + 5^{x-1} = 15 \) is \( x = 2 \). line how can solve it i'm use  RegExp(r'(|\((.*?)\))', dotAll: true); on that i'm display math equetion but got Parser Error: Can't use function '(' in math mode

If you wish to generate the following text:
The value of $x$ that satisfies the equation $5^{x-1} + 5^{x-1} + 5^{x-1} = 15$ is $x = 2$.
Try the following code:
\text{The value of} ~x~ \text{that satisfies the equation} 5^{x-1}~ + 5^{x-1} + 5^{x-1} = 15 ~\rm{is}~ x = 2.


react-native-permissions not compatible with react-native 0.66.2 (IOS)

Bug summary
Hi all,
When trying to adding camera permissions to my react native app, everything builds fine on Android and IOS. On android this functionality seems to work fine, however when asking permissions on IOS, i made some code that opens the settings:
Permissions.openSettings();
There is no checkbox for giving camera permissions. I tried to reset my privacy settings on the IOS but i still see no camera icon.
Is it correct that react-native-permissions is not yet ready for this react native version? What is the last react-native-version that has been succesfully tested with this package?
If more info is needed please do reply here :)
Happy Coding,
Kees Til
Library version
3.2.0
Environment info
I Just downgraded to another version, if this is really necessary i wil post it. For now downgrading to 0.63.3 and then:

System:
    OS: macOS 11.6
    CPU: (8) arm64 Apple M1
    Memory: 128.16 MB / 8.00 GB
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 17.3.0 - /opt/homebrew/bin/node
    Yarn: 1.22.17 - /opt/homebrew/bin/yarn
    npm: 8.3.0 - /opt/homebrew/bin/npm
    Watchman: 2<IP_ADDRESS> - /opt/homebrew/bin/watchman
  Managers:
    CocoaPods: 1.11.2 - /opt/homebrew/bin/pod
  SDKs:
    iOS SDK:
      Platforms: DriverKit 21.2, iOS 15.2, macOS 12.1, tvOS 15.2, watchOS 8.3
    Android SDK:
      API Levels: 28, 32
      Build Tools: 27.0.3, 28.0.3, 29.0.2, 30.0.2, 32.0.0
      System Images: android-28 | Intel x86 Atom_64, android-32 | Google APIs ARM 64 v8a
      Android NDK: Not Found
  IDEs:
    Android Studio: 2020.3 AI-203.7717.56.2031.7935034
    Xcode: 13.2.1/13C100 - /usr/bin/xcodebuild
  Languages:
    Java: 1.8.0_292 - /usr/bin/javac
    Python: 2.7.16 - /usr/bin/python
  npmPackages:
    @react-native-community/cli: Not Found
    react: 16.13.1 => 16.13.1
    react-native: 0.63.4 => 0.63.4
    react-native-macos: Not Found
  npmGlobalPackages:
    *react-native*: Not Found

Steps to reproduce
see bug summary
Reproducible sample code
async function requestDirectPermissionsPlugin(permissionType) {
  const status = await pluginPermissionsCheck(permissionType);

  if (status === true) {
    return status;
  }

  if (Platform.OS === 'ios' && status === false) {
    Permissions.openSettings();
    return null;
  }

  return pluginPermissionsRequest(permissionType);
}

react-native-permissions is fully compatible with RN 0.66.x. In order to have the permission switch visible in settings, you have to request it. More infos here https://stackoverflow.com/questions/32332313/ios-camera-permissions-doesnt-appear-under-settings-in-some-devices
async function requestDirectPermissionsPlugin(permissionType) {
  const status = await request(permissionType);

  if (status === RESULTS.GRANTED) {
    return true;
  } else {
    Permissions.openSettings();
  }
}

When removing the following code for the sake of purpose:
Permissions.openSettings()
I see that the call for a request: Permissions.request(permissionType) works. So thats good news!
However i want the switch to be present. Since the Permissions.openSettings exist within this package, maybe expanding the documentation on how to add these switches when using this function is a great idea?
I am not the greatest with native code (objective-C/SWIFT), If you have any idea on how to do this i would be very happy, but if you have no time i will try figuring it out by myself. Thanks for the fast reply!

That's an iOS issue. The switch cannot be present in some cases (first app run, permission not requested, parental control etc).
We cannot do something about it in our side, sorry!


fix(types): Configuration miss devServer Type

https://github.com/webpack/webpack/issues/13621

What kind of change does this PR introduce?
bugfix

Did you add tests for your changes?

Does this PR introduce a breaking change?
no

What needs to be documented once your changes are merged?
no

It's intentionally missing, because @types/webpack-dev-server adds them. If webpack would have them in the types, that would break as types are conflicting


Move Primer::Navigation::TabComponent to Primer::Alpha::Navigation::Tab

Description
Move Primer::Navigation::TabComponent to Primer::Alpha::Navigation::Tab, and deprecate the original
Integration

Does this change require any updates to code in production?

not immediately, but deprecations need to be cleaned up
Merge checklist

[ ] Added/updated tests
[ ] Added/updated documentation
[ ] Added/updated previews

bump

@jonrohan should we try to revive this PR, or would it best to close it and handle fresh? cc @hrs as this is related to the component renaming effort you've been getting up to speed on

It seems the only issue is that the tag needs to be enclosed in an  or  tag, or the test needs to be disabled for this  element: Tab


Fix edge cases in search input

The search queries sent to Elastic Search are interpreted as Lucene syntax, which means that some characters will result in undefined behavior. For example, + must be between words, - removes terms, etc etc.
In this commit we start stripping these values so they dont result in unexpected behavior (users cannot be expected to know or understand (or guess that we use) Lucene).
Specifically, we replace them with spaces, so we dont end up
concatenating words that should not be concatenated. E.g. magasin+okay
will result in magasin okay rather than magasinokay.

Det virker helt fint i min ende :). Godt arbejde!

Men skal det i master branch eller accessibility-enhancements branch?


rebase against master

Hi Hervé
This PR is just a rebase against master. It should be helpful to rebuilt your own PR.
I hope I properly understand how to do it.

This may not be a proper rebase which might have require a git push --force . It is more like a git merge I presume.

Not needed anymore


Possible to provide minimal code for Figure 2?

Thanks for the awesome paper!
Is it possible to provide minimal code for reproducing some parts of Figure 2? I believe that would greatly benefit the community of researchers that are working on similar things.

As we say in the fig caption, we follow the great work of Garipov et al., 2018 in creating this figure. Their repository is here: https://github.com/timgaripov/dnn-mode-connectivity with plane.py and plane_plot.py being the important files.

Thanks.


COLLECTIONS-814 CollectionUtils.removeAll() not throwing NPE

COLLECTIONS-814
CollectionUtils.removeAll() not throwing proper NullPointerException(NPE) if the first parameter is empty
Add null check in ListUtil.removeAll()

Codecov Report

Merging #340 (a3490d3) into master (474713f) will increase coverage by 0.05%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##             master     #340      +/-   ##
============================================
+ Coverage     85.93%   85.98%   +0.05%
- Complexity     4669     4671       +2
============================================
  Files           289      289
  Lines         13445    13447       +2
  Branches       1977     1977
============================================
+ Hits          11554    11563       +9
+ Misses         1327     1323       -4
+ Partials        564      561       -3

Impacted Files
Coverage Δ

...ava/org/apache/commons/collections4/ListUtils.java
87.30% <100.00%> (+0.20%)
:arrow_up:

...commons/collections4/map/AbstractReferenceMap.java
91.48% <0.00%> (+2.59%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

ping?

Merged. Thanks @angusdev !


rf: ex introduces double-qualified pkg.pkg.Name identifiers

I expect the test case below should succeed, but it currently fails because ex instead tries to substitute "sub1.sub1.X" in place of "m.X" in sub2/2.go.
/cc @rsc
ex . ./sub2 {
  import "m/sub1"
  X -> sub1.X
}
-- m.go --
package m

import "m/sub1"

var X = 42

func init() { sub1.X = X }

var _ = X
-- sub1/1.go --
package sub1

var X int
-- sub2/2.go --
package sub2

import "m"

var _ = m.X
-- stdout --
diff old/m.go new/m.go
--- old/m.go
+++ new/m.go
@@ -4,6 +4,6 @@

 var X = 42

-func init() { sub1.X = X }
+func init() { sub1.X = sub1.X }

-var _ = X
+var _ = sub1.X
diff old/sub2/2.go new/sub2/2.go
--- old/sub2/2.go
+++ new/sub2/2.go
@@ -1,5 +1,7 @@
 package sub2

-import "m"
+import (
+	"m/sub1"
+)

-var _ = m.X
+var _ = sub1.X

Similar problem seems to affect inline, but it generates m.sub1.X instead.

Test now passes, so it seems like this issue has been fixed at some point.


Error restoring database after about 10 GB

This might mean that we have some sort of weird condition in our async pipelines that we use to wire up pg_restore. This is all happening locally, so the chance of networking issues being to blame is slim to none.

SIGINT also doesn't work


Supported operating systems

Is Windows Server 2019 supported?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 0dbd38fa-89f0-320a-b517-b440be6927f9
Version Independent ID: 8b2eeee3-fb49-37e5-917d-f36bcd60ad29
Content: Prerequisites - Azure Disk Encryption for IaaS VMs
Content Source: articles/security/azure-security-disk-encryption-prerequisites.md
Service: security
GitHub Login: @msmbaldwin
Microsoft Alias: mbaldwin

@hulltr1 Thanks for your feedback! We will investigate and update as appropriate.

@hulltr1 I am not sure about windows 2019, so I am assigning this to document author to comment on the same.
@msmbaldwin Can you please help and update the documentation if Windows Server 2019 is supported.

Any update?

@pixelrobots: Windows Server 2019 is supported. We are updating the documentation to reflect this.

Hello again, @PixelRobots.  Azure Disk Encryption prerequisites now states:

Windows client: Windows 8 and later.
Windows Server: Windows Server 2008 R2 and later.
Note
Windows Server 2008 R2 requires the .NET Framework 4.5 to be installed for encryption; install it from Windows Update with the optional update Microsoft .NET Framework 4.5.2 for Windows Server 2008 R2 x64-based systems (KB2901983).
Windows Server 2012 R2 Core and Windows Server 2016 Core requires the bdehdcfg component to be installed on the VM for encryption.

Thanks for the response.

#please-close


This colorizes the lines leading up to a nonmonophyletic node

I added a loop that classifies the paths as either leading to a nonmonophyletic node or not. If you run hybsearch, you see that the red lines do not go all the way back to the origin- if you want them to, I can try to change it as such.
Closes #41

I'll update this on top of master tomorrow morning


simplefs: add a debug command for path deobfuscation

This can show multiple paths per obfuscated path, since conflict
suffixes and symlinks make this possible.
Sample output:
$ keybase fs debug deobfuscate /keybase/private/strib/entry-voyage/physical-exit.pkg
/keybase/private/strib/tmp/trezor-bridge-2.0.13.pkg

$ keybase fs debug deobfuscate /keybase/private/strib/entry-voyage/physical-exit-85487.pkg /keybase/private/strib/entry-voyage/sign-reduce-2
/keybase/private/strib/entry-voyage/physical-exit-85487.pkg:
  /keybase/private/strib/tmp/trezor-bridge-2.0.13.pkg

/keybase/private/strib/entry-voyage/sign-reduce-2:
  /keybase/private/strib/tmp/z9

Issue: HOTPOT-66

Would be nice to sneak this into the release, to help debug any issues that come up where I need to interact with a user about specific obfuscated filenames.  Should be pretty low risk, if anyone has time to review today, I'd appreciate it!

asdfglkjh github ate my most important comment :(

@jakob223:

I think it would be more user-friendly if the CLI tool gave an error message and a non-zero return code when it can't find a match, instead of silently printing nothing.

What should it do in the case where multiple paths are given, but only one of them has no match?  Fully error and non-0 exit code, or just print the error under the path with no matches and still return 0?

I'd prefer the latter, but I don't have strong feelings as long as there's an acknowledgement that something had no results.

Addressed feedback, except for the || !fi.IsDir() thing, which I think is correct.


_\flatten not expand empty array inside passed array

when I pass _\flatten an array which contains empty array, _\flatten does not expand the empty array.
ex.
use function _\flatten;

$arr = [
    [1, 2, 3],
    [],
    [4, 5, 6],
];
$result = flatten($arr);

I expected that $result is [1, 2, 3, 4, 5, 6], but actually [1, 2, 3, [], 4, 5, 6].
this behavior is expected?

Thanks @tooppoo, this is now fixed in the master branch


Fix transparency when OIT is enabled.

Fixes #2572.

Looks good!

Also, could you update CHANGES.md in master?


respect SUPPRESS_EXCEPTIONS configuration

Even when using the SUPPRESS_EXCEPTIONS configuration option, PathNotFoundExceptions are thrown when paths don't match the JSON. Since #751 hasn't been released yet this has a significant performance impact.

@kallestenflo would you mind taking a look please?


Tip Calculator

Is your feature request related to a problem? Please describe.
Whenever we go outside for a lunch/dinner in a restaurant. We always pay a tip to the waiter after paying the bill.
Describe the solution you'd like
This calculator will help us to calculate the tip to be paid based on our bill price.

@Sulagna-Dutta-Roy  Plz assign this issue to me!


App crashes while scrolling through messages

Expected behavior
While going back and forth between messages application should not crash.
Actual behavior
While going through messages in Sent folder, either just to read and press the next arrow or to delete (maybe other folders as well - I did not test others) before app goes to next message (or while trying to load it) application crashes.
Steps to reproduce

Go to an email account and select the Sent folder either through the folder icon or menu.
Open any message to read it.
Delete the message or just try to hit the next arrow.
It will crash app.  Sometimes not on the first or second message but it will crash eventually.

Environment
K-9 Mail version:
5.201
Android version:
6.0
Account type (IMAP, POP3, WebDAV/Exchange):
POP3 and IMAP

Hopefully there is a Google Play crashlog (assuming you installed via Google Play) and sent a report for this.

I did.

Hi all, not sure if this is the correct thread but I got a similar issue. The app freeze and soon crash itself. There's no particular situation for this. It can happens soon as I open K-9, or after opening an email. Btw, reopen the app for the second or the third time it usually works without further freeze. I use Xprivacy but i totally disable it for K-9 (and anyway I was previously give ALL permission, just to be sure).
I try to download the log file from ADB but for some reason inside the txt file i got this message:
/sbin/sh: exec: line 1: logcat: not found
Anyway, by enabling the log option I found 5 logs as emails soon as I start K-9. This are related to different email account that I signed on K-9 so I zipped 5 file simple txt file (that I upload here) where, the first account (mentioned in the log text file name) it's the most used; the second is sometimes used and the third very rare used...
Hope can help. Thank you in advance for any support.
I'm on HTC One M9 - ViperoneM9 6.2.0 (no Magisk) SuperSu


Code modernization and an optimization

code modernization
removed unnecessary O(n) generation of const set

@bp-kelley I have grouped all the cosmetic changes in this PR. It would be super-cool if you could review and merge this in, so then I can move on to the serious stuff that you raised comments on. Thanks!

@ptosco Thanks for doing breaking up the changes, it will make the review easier.
I wish we had been using C++11 when I started this, it makes a lot of things easier.  I probably don't have time today, but it should go quickly tomorrow morning.

@bp-kelley Great, thanks. This review should be a easy - most changes are trivial, the only relevant one is the once-and-for-all initialization of the core_atoms_with_user_labels set - but that is also quite straightforward. Have a nice weekend!

@ptosco - once this is =fully approved let's do a quick PR where we run clang-format before we do the algorithm review.


Fixed authentication when userInfo provided with resource URL

In a features file, when the "userInfo" (username:password) is provided with the URL, the authentication is not getting processed.
This patch takes the userInfo and creates an HTTP header for authentication.
Note: I am not sure if the userInfo should be removed from the URL or not before the request is made, so I just left it intact.

 Thanks !

@dleangen I saw your directly commit your changes. Please, next time, consider my comments, thanks ;)

Hi @jbonofre, which comments are you referring to? Did I not do the PR correctly?
Sorry, I'm not sure what you mean. It was not my intent to do anything improperly. Could you please explain a little more what you mean?

Hi David,
ideally you should create a jira issue describing the problem before doing a PR. Then in the commit(s) you should use the jira id like [KARAF-123] Your message...
You can then also name the branch like the jira issue id. (Not mandatory as the branch is lost after the merge anyway).
So the jira issue is part of the next release notes and the commit is linked to the jira issue.
Christian
We can also do that by amending the commit message of course but it is easier if you already provide this information.

Ok, got it! Thanks for explaining.

I will add this information to the community docs so we can refer to it and you have a nice reference to look up. See https://karaf.apache.org/community.html

:-)


After installing dBeaver 22.1.5, I get "Datasource config parse error"

Description
This is similar to https://github.com/dbeaver/dbeaver/issues/17300
I installed dBeaver 22.1.5 this week.  I was prompted to migrate my configuration to the new format and I choose 'Migrate' and my existing connections are imported.
But then I need to create a new connection.  The new connection is type Postgre.  I already have 2 existing postgre connections as well.
After I create the new connection and exit I get the following error
Datasource config parse error
Document parse error (line 1, pos 1)
Document parse error (line 1, pos 1)
org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
Content is not allowed in prolog.

So I can rename the DbeaverData folder at /users//AppData/Roaming to something like DbeaerData.last and then I can access my existing connections again.  But my new connection is no longer there, so I have to recreate it again and then of course if I exit at the end of the day, the next time I open DBeaver, I will get the error again.
I believe my last version of dBeaver was 6.0.5

DBeaver Version
Dbeaver 22.1.5
Operating System
Windows 10
Database and driver
PostGre SQL, creating a new connection
Steps to reproduce

Install dBeaver 22.1.5
Add a new database connection.
Exist dBeaver
Open dBeaver

Additional context
None

Please, send the log files. You can find instructions here: https://dbeaver.com/docs/wiki/Log-files/

Also could you send your data-sources file (see https://github.com/dbeaver/dbeaver/wiki/Admin-Manage-Connections) to my email<EMAIL_ADDRESS>It would be very helpful.

Here is my log file
.log

Thank you

+1. All saved connections / driver configurations disappearing:
$ tree ~/.local/share/DBeaverData/
~/.local/share/DBeaverData/
├── drivers
│   └── maven
│       └── maven-central
│           └── com.h2database
│               ├── h2-2.1.214.jar
│               └── h2-2.1.214.pom
├── workspace6
│   └── General
│       └── Scripts
│           └── Script.sql

(I was downloading h2-2.1.214 driver in the latest session - only maven repo cache is present)

Log says that file C:\Users\LJohnsto\AppData\Roaming\DBeaverData\workspace6\General.dbeaver-data-sources.xml is corrupted.
It is a file where older DBeaver versions kepp datasources configuration.

Reproducing:
https://user-images.githubusercontent.com/265872/187096793-f2d8c7dd-f90c-4906-a4b3-fe011994f3de.mp4

verified

Same problem, but I don't want to delete .dbeaver. I need to keep my previous data source configuration.
How can I fix it?


Revert to compiling webpack configs in sequence

This turn offs multi-compiler mode in webpack as it seems unreliable.
From docs :
webpack will not run the multiple configurations in parallel. Each configuration is only processed after the previous one has finished processing. To have webpack process them in parallel, you can use a third-party solution like parallel-webpack.

Which is untrue (we've had webpack([ssrConfig, clientConfig]) previously and clientConfig was processed before ssrConfig - now it seems random)

Although we can explore other options like - paralle-webpack, not sure if that fits the purpose or not.


cannot stop beacon node with CTRL+C: UnhandledPromiseRejectionWarning - Cannot read property 'unsubscribeFromNewEpoch'

Describe the bug
CTRL+C does not allow me to exit the running beacon node
ict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 228)
(node:52111) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'unsubscribeFromNewEpoch' of undefined
    at ArrayDagLMDGHOST.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/chain/forkChoice/arrayDag/lmdGhost.js:218:16)
    at BeaconChain.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/chain/chain.js:167:27)
    at BeaconNode.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/node/nodejs.js:168:22)
    at async process.cleanup (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar-cli/lib/cmds/beacon/cmds/run/run.js:65:5)
(node:52111) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 230)
(node:52111) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'unsubscribeFromNewEpoch' of undefined
    at ArrayDagLMDGHOST.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/chain/forkChoice/arrayDag/lmdGhost.js:218:16)
    at BeaconChain.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/chain/chain.js:167:27)
    at BeaconNode.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/node/nodejs.js:168:22)
    at async process.cleanup (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar-cli/lib/cmds/beacon/cmds/run/run.js:65:5)
(node:52111) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 232)
(node:52111) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'unsubscribeFromNewEpoch' of undefined
    at ArrayDagLMDGHOST.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/chain/forkChoice/arrayDag/lmdGhost.js:218:16)
    at BeaconChain.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/chain/chain.js:167:27)
    at BeaconNode.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/node/nodejs.js:168:22)
    at async process.cleanup (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar-cli/lib/cmds/beacon/cmds/run/run.js:65:5)
(node:52111) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 234)
(node:52111) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'unsubscribeFromNewEpoch' of undefined
    at ArrayDagLMDGHOST.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/chain/forkChoice/arrayDag/lmdGhost.js:218:16)
    at BeaconChain.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/chain/chain.js:167:27)
    at BeaconNode.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/node/nodejs.js:168:22)
    at async process.cleanup (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar-cli/lib/cmds/beacon/cmds/run/run.js:65:5)
(node:52111) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 236)
(node:52111) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'unsubscribeFromNewEpoch' of undefined
    at ArrayDagLMDGHOST.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/chain/forkChoice/arrayDag/lmdGhost.js:218:16)
    at BeaconChain.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/chain/chain.js:167:27)
    at BeaconNode.stop (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar/lib/node/nodejs.js:168:22)
    at async process.cleanup (/usr/local/share/.config/yarn/global/node_modules/@chainsafe/lodestar-cli/lib/cmds/beacon/cmds/run/run.js:65:5)
(node:52111) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 238)

Expected behavior
CTRL+C should shut down the node
Steps to Reproduce

node /usr/local/bin/lodestar beacon run --testnet medalla --eth1.enabled true --eth1.provider.url http://localhost:8545 --eth1.provider.network 5 --eth1.depositContract.deployedAt 3085928 --eth1.depositContract.address 0x07b39F4fDE4A38bACe212b546dAc87C58DfE3fDC
CTRL+C

OS: Archlinux
Branch: master
Commit hash: 02d82b2d550c596fa91ced65194ccb9c33f58175

Only usage of unsubscribeFromNewEpoch related to the stack trace above is this (below). But clock is checked before called, how can it be that is called despite being undefined?
export class ArrayDagLMDGHOST {
  public async stop(): Promise<void> {
    this.clock && this.clock.unsubscribeFromNewEpoch(this.onTick);
  }
  ...

resolved via #1440
It should allow force exit while we take some time to ensure everything is properly stopped


Provide shell.nix for easier compiling with nix-shell

Created shell.nix under root, as mentioned in #481
Dependencies are the same as the ones in docs/linux_dependencies.md

nix-shell -p pkgconfig x11 xorg.libXcursor xorg.libXrandr xorg.libXi vulkan-tools lutris vulkan-headers vulkan-loader vulkan-validation-layers alsaLib

I would prefer it if we moved this to docs/linux_dependencies.md under the NixOS section. I want to keep the root as flat / legible as possible and Nix config only applies to a small subset of the population.

I would prefer it if we moved this to docs/linux_dependencies.md under the NixOS section. I want to keep the root as flat / legible as possible and Nix config only applies to a small subset of the population.

Would you like to keep it embedded in docs/linux_dependencies.md or in a separate file under docs/?

Lets keep it embedded in linux_depedencies.md (with instructions on how/where to create the file)

Looks good to me!


Document that noSuchMethod only half works with Dart2?

An internal customer just hit this when moving tests from Dart1 to Dart2.
(They were on the DDC runtime whitelist, hence why it only came up now)
Here is a minimal reproducible sample of the user's code:
import 'dart:async';

void main() {
  MiRpc rpc = new FakeMiRpc();
  var request = new Request<String>();

  // 'Future<dynamic>' is not a subtype of type 'Future<String>'
  print(rpc.handle(request).runtimeType);
}

class Request<T> {}

// Real RPC class they use in production.
abstract class MiRpc {
  Future<T> handle<T>(Request<T> r);
}

// Fake implementation (uses Mockito) in tests.
class FakeMiRpc implements MiRpc {
  @override
  noSuchMethod(_) => new Future.value();
}

I had the following 3 ideas for them:

Stop using noSuchMethod. This would be a giant change for them.
Add another method, like handleUntyped:

class MiRpc {
  Future<T> handle<T>(Request<T> request);
  Future<dynamic> handleUntyped(Request<T> request);
}

class FakeMiRpc implements MiRpc {
  @override
  Future<T> handle<T>(Request<T> request) {
    return handleUntyped(request).then((r) => r as T);
  }

  @override
  noSuchMethod(_) => new Future.value();
}

... and then change the 100+ usages from
when(mockRpc.handle)
to
when(mockRpc.handleUntyped)

The hackiest, but another option. Encode an "escape hatch" into "class Request":

class Request<T> {
  Future<T> reifyAsT(Future<dynamic> future) => future.then((r) => r as T);
}

Then FakeMiRpc could use it:

noSuchMethod(Invocation i) {
  var request = ...
  var future = ...
  return request.reifyAsT(future);
}

I hope this example shows that we should clearly document and state that using noSuchMethod to proxy for methods with generic type arguments will likely not work in many cases (unlike Dart 1).
/cc @srawlins

How about noSuchMethod(_) => Future<Null>.value();?
This does not seem to be a noSuchMethod problem as much as it's a general strong-mode issue. The code is not type-safe, it is returning a Future<dynamic> where a Future<T> (with T being String) is required. That's a predictable strong mode type error, since dynamic is no longer a wild-card type. You need to return a Future which is actually assignable to Future<T>, but you would need that whether you use noSuchMethod or not.
It is true that noSuchMethod has a hard time with type parameters because it gets them as Type objects, and there is no way to reify those as real types again (deliberately). For traditional mocking, I don't see that as a big issue. You typically record a request-response sequence, and you can return a response that matches the actual request.

I believe the new behavior is documented in https://www.dartlang.org/guides/language/language-tour#nosuchmethod, but there's no mention of the difference in https://www.dartlang.org/dart-2. I don't think that /dart-2 is the best place to list every single difference, but it should point to a page that is much more thorough. That could be the SDK changelog, if the changelog were much more thorough than it is right now.

@lrhn:

How about noSuchMethod(_) => Future<Null>.value();?

Sure, that works until you need a valid value.

This does not seem to be a noSuchMethod problem as much as it's a general strong-mode issue.

It's a noSuchMethod problem in-that strong mode makes it less useful but users don't know that.

@matanlurey In this case, something like this should work:
class FakeMiRpc implements MiRpc {
  @override
  noSuchMethod(invocation) {
    var types = invocation.typeArguments;
    if (types.length == 1 && types.first == String) return Future<String>.value();
    return Future.value();
  }
}

I don't think there is anyway to thread the type param generally though (@leafpetersen @lrhn ?).

@matanlurey @lrhn - retitled the bug to (I think) match Matan's request better.

My original request was just around documenting that noSuchMethod doesn't support generic type arguments, not to augment it @vsmenon.

In a sense, it does (Invocation.typeArguments).

Oh sure. I just don't need there feature (or want the backend teams to have to implement it). I just wanted it documented so we had something to point to that NSM in Dart 2 isn't exhaustive


chore(deps): update ci dependencies

This PR contains the following updates:

Package
Type
Update
Change

common-dev-assets

digest
fc94bf1 -> 6eb04f4

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
patch
v1.26.3 -> v1.26.5

terraform-ibm-modules/common-pipeline-assets
action
minor
v1.19.0 -> v1.20.0

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper (github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper)
v1.26.5
Compare Source
Bug Fixes

deps: update gomod (#743) (1ab2e92)

v1.26.4
Compare Source
Bug Fixes

deps: update gomod (#738) (a9275e6)

terraform-ibm-modules/common-pipeline-assets (terraform-ibm-modules/common-pipeline-assets)
v1.20.0
Compare Source
Features

exposed the ability to pass TOOLCHAIN_ID in cra-config.yaml (#635) (d128eca)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

/run pipeline

/run pipeline

:tada: This PR is included in version 1.0.5 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


fastq for paired-end sequencing

I am trying to create fastq file(s) for each respective read group within the cram file.  There should be a total of 4 different files instead of the 1 large fastq file that has been made.  Are there any special parameters that need to be added if this is possible through cramtools?
java -jar cramtools-3.0.jar
fastq
-I cram_file.cram
-R ref_genome \

in the current implementation cramtools does not split data by read group. I've opened an issue:
https://github.com/enasequence/cramtools/issues/50

Thanks!


Add Price Endpoint

gatsby-theme-bch-wallet has a need for looking up the price for BCH in USD. It currently calls bch-api on api.fullstack.cash. To make that app even more local, ipfs-bch-wallet-consumer should have a price endpoint.
To complete this issue, this code from bch-api should be ported over to a new REST endpoint on ipfs-bch-wallet-consumer.

Fulfilled by #19


Bump jest version in @vue/cli-plugin-unit-jest

What problem does this feature solve?
It's been said that Jest 24 comes with the option to transpile globalSetup files using babel (to use ES6 modules), but current jest version in cli-plugin-unit-jest is "^23.6.0"
What does the proposed API look like?
"^24.1.0"

Beside the new options, the current jest ^23.6.0 transitively depends on braces ^1.8.2 through micromatch ^2.3.11.
This causes the vue-projects that use the current @vue/cli-plugin-unit-jest to have 63 low severity vulnerabilities because of https://www.npmjs.com/advisories/786.
Similar to #3257 but it is nightwatch there.

is there any workaround yet?

According to: facebook/jest#6743

Unless you're using micromatch to generate passwords or API tokens, I think it's safe to say that this is unlikely to actually be a concern

You should be able to use v23.6.0 in the meantime - of course, that's only if you aren't taking advantage of v24 already.

Any news? Are there any blockers to do so? It's been a while..

I've been successful with yarn resolutions adding the following to package.json
  "resolutions": {
    "@vue/cli-plugin-unit-jest/jest": "^24.6.0",
    "@vue/cli-plugin-unit-jest/babel-jest": "^24.6.0"
  },

It seems it is fixed in 4-alpha, but not in 3.8.2 yet. Could it be bumped in v3 branch as well? Getting one vulnerability message repeated 63 times for just a dev dependency, even if it is just low, is pretty disturbing..

We can't. Because it's a breaking change.


Kube API Error classification - classify known errors and do not log them

What would you like to be added: I'd like that when we receive an error from the API, that we classify those errors. In the case that Multus cannot talk to the API, that's a sort of fatal error. However, plenty of these kind of errors are actually known conditions that should be logged only during when there's a debug error logging level set.
Why is this needed: For example, when there's a "pod not found" type of error, this can happen when a pod is force deleted. This is a non-fatal type of error and only useful at a debug logging level.

I went ahead and implemented Dan's suggestion (which is more pragmatic than my own initial suggestion) in #311 -- I'm going to leave this comment open for now, as there may be other error conditions that we do want to classify from the API server. This one is well known, we might know more via future experience or the happenstance finding of a "crystal ball" (hoping for the crystal ball).


Describes regional locations too for 'location'

Description & motivation
location for BigQuery profiles can accept not only multi-regional locations, but also regional locations, because the location information is just passed to google.cloud.bigquery.Client.
Pre-release docs
Is this change related to an unreleased version of dbt?

[ ] Yes: please

update the base branch to next
add Changelog components: <Changelog>[New/Changed] in v0.x.0</Changelog>
add links to the "New and changed documentation" section of the latest Migration Guide

[ ] No: please ensure the base branch is current
[ ] Unsure: we'll let you know!

Checklist
If you added new pages (delete if not applicable):

[ ] The page has been added to website/sidebars.js
[ ] The new page has a unique filename

If you removed existing pages (delete if not applicable):

[ ] The page has been removed from website/sidebars.js
[ ] An entry has been added to _redirects

Thank you for the suggestions. I have taken them.


More Details.

Can I get more details about these parameters please.
options.IdleTimeout = TimeSpan.FromSeconds(10);
options.Cookie.HttpOnly = true;
options.Cookie.IsEssential = true;

Hi @SerkanTopkan
What kind of more details for the above paramaters? like the usage in the Responsive of the session service right?

Please revisit this issue. Because Responsive 3.0 has now been remove from Detection 5.0


clickhouse-client 19.9 is incompatible with 19.8

19.9 connecting to 19.8:
ClickHouse client version <IP_ADDRESS> (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 19.8.3 revision 54420.

:) select version()

SELECT version()

┌─version()─┐
│ <IP_ADDRESS>  │
└───────────┘
Code: 100. DB::Exception: Unknown packet 0 from server localhost:9000, ::1

19.8 connecting to 19.9:
ClickHouse client version <IP_ADDRESS> (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 19.9.3 revision 54421.

:) select version()

SELECT version()

┌─version()─┐
│ <IP_ADDRESS> │
└───────────┘
Code: 100. DB::Exception: Unknown packet 227213735 from server localhost:9000, ::1

Yesterday I was upgrading a 5-node cluster from 19.8 to 19.9.
After the first node was upgraded, distributed queries stopped working with the same message about unknown packet, and two of the four nodes with 19.8 received signal Segmentation fault

Fix https://github.com/yandex/ClickHouse/pull/6018.


fix typo in README.md, will to with

Codecov Report

Merging #588 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #588   +/-   ##
=======================================
  Coverage   41.71%   41.71%
=======================================
  Files          74       74
  Lines       16708    16708
  Branches     4369     4369
=======================================
  Hits         6969     6969
  Misses       7757     7757
  Partials     1982     1982

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4558dbf...0a68b7a. Read the comment docs.

Thanks!


How do I update the git-endpoint?

I registered a bower package to my fork prior to the original package owner accepting my pull request and now want to update the git-endpoint to the original package. I was forbidden to unregister so that I could register it anew. Does anyone know how I can update the git-endpoint for a given bower package?
Thanks in advance.

If you tell me the package name I can unregister it, so you can re-register it with the correct URL.

angular-highlight-partial-words
Thanks!

@moses-seeq Done. All yours! Thanks for using Bower 🍬

Thanks!  When will I be able to register the new end point?  It's telling me duplicate package now and I still see it pointing to my fork.

Hmm sorry I think something went wrong during unregistering.. I'm getting a
FORBIDDEN Must have push access to view repository collaborators. error atm.
Maybe cause I switched my GH name to all lowercase recently? @sheerun
Actually tried bower login with the old spelling as well, with which didn't work either.

Can you reopen this until it's resolved?  Is there anything I can do to help move toward a solution?

Nah, just needs some time now :)

Ok, thanks!

@moses-seeq I've unregistered it, you can register it again under different url

Thank you!


please correct sentence "...the apps from Finance and Operations will be synchronized"

The sentence is wrong:
With the dual-write functionality enabled, the apps from Finance and Operations will be synchronized in other Dynamics 365 apps in Draft state.
...Not the Apps from F&O will be synchronized, but the Products
Thanks for checking and fixing,
Bernhard

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a19d287f-1a09-4abb-1265-19d0be0bad1a
Version Independent ID: 9b7eec43-2a44-fe69-7c9a-e6153fc115ee
Content: Unified product experience - Finance & Operations | Dynamics 365
Content Source: articles/fin-ops-core/dev-itpro/data-entities/dual-write/product-mapping.md
Service: dynamics-ax-applications
Product: ****
Technology: ****
GitHub Login: @t-benebo
Microsoft Alias: ramasri

@t-benebo Can you take a look? Thanks.

We updated this topic.


Allow specifying UNIQUE (KEY | INDEX) in table_constraint for MySQL

For some reason getting this error locally
    UniqueKey.s line 4:15 - '(' or index_name expected, got 'unq_name'

I assume it's something dumb like id can't have underscores or something, because index_name is currently defined like
index_name ::= id

I tried locally for half an hour, then figured it's best to ask.

anything thats a rule from the parent grammar needs to have {} around it - so in your case something like {identifier} does

and then keywords have to imported up at the top, like CONSTRAINT

unfortunately none of these are obvious at all, I should really make it that way so that its easier for ppl to contribute

Nice, that seemed to do the trick.
Regarding it being non-obvious, I'm happy to put together a README explaining a bit about how it works, and list those out as gotchas so others don't make the same mistake. Aside from that, it was pretty easy to add! Especially following the bnf here https://dev.mysql.com/doc/refman/8.0/en/create-table.html

A contributing doc where you catalogue stuff you’re seeing would be AMAZING


pointer tag tracking: on creation, log the offsets it is created for

Hopefully this makes things like https://github.com/Manishearth/triomphe/issues/38 easier to diagnose.

@bors r+

:pushpin: Commit 6cc407e41965dd7efc19f798379397847e645c7a has been approved by RalfJung

:hourglass: Testing commit 6cc407e41965dd7efc19f798379397847e645c7a with merge c4b8c67ed9a136b15e768557722a251757555754...

:broken_heart: Test failed - checks-actions

@bors r+

:pushpin: Commit 07384d21f630ea1933b80b1bce96a2922631b8f4 has been approved by RalfJung

:hourglass: Testing commit 07384d21f630ea1933b80b1bce96a2922631b8f4 with merge f7f1e912bd56d5600d5823874acf184c33cc7b25...

:broken_heart: Test failed - checks-actions

@bors r+

:pushpin: Commit 33e5e1984cbe7535ef55b21c7eda04fffe3f4dd2 has been approved by RalfJung

:hourglass: Testing commit 33e5e1984cbe7535ef55b21c7eda04fffe3f4dd2 with merge 7bd15aa265e35514f44c4f7119ff5faad6f83cb7...

:broken_heart: Test failed - checks-actions

@bors r+

:pushpin: Commit 98254f67af20a67747835f03c314be9c9a42d769 has been approved by RalfJung

:hourglass: Testing commit 98254f67af20a67747835f03c314be9c9a42d769 with merge c9d5c3f88c2cdb96702ffb4d6183864315bc8ca1...

:sunny: Test successful - checks-actions
Approved by: RalfJung
Pushing c9d5c3f88c2cdb96702ffb4d6183864315bc8ca1 to master...


[C001071] Sen. Bob Corker

Sen. Bob Corker:
http://www.corker.senate.gov
Likely contact form URL to start with:
http://www.corker.senate.gov/public/index.cfm?p=ContactMe

YAML file lost. I am working on it.


Add Kadane's Algorithm to get largest subarray sum

Hello Sir, Please Assign me this issue as I want to add kadane's Algorithm to your repository

Hello, @ianshulx I want to add this program code in JAVA.
Please assign me this issue.
thank you

Hey @ianshulx, wanted to add the CPP implementation for this Algorithm.
Can you please assign me this issue.


python312Packages.channels: 4.1.0 -> 4.2.0

Diff: https://github.com/django/channels/compare/refs/tags/4.1.0...4.2.0
Changelog: https://github.com/django/channels/blob/4.2.0/CHANGELOG.txt

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 356426

x86_64-linux

:white_check_mark: 31 packages built:

authentik
netbox
paperless-ngx
python311Packages.channels
python311Packages.channels-redis
python311Packages.channels-redis.dist
python311Packages.channels.dist
python311Packages.netbox-bgp
python311Packages.netbox-bgp.dist
python311Packages.netbox-documents
python311Packages.netbox-documents.dist
python311Packages.netbox-reorder-rack
python311Packages.netbox-reorder-rack.dist
python311Packages.strawberry-django
python311Packages.strawberry-django.dist
python311Packages.strawberry-graphql
python311Packages.strawberry-graphql.dist
python312Packages.channels
python312Packages.channels-redis
python312Packages.channels-redis.dist
python312Packages.channels.dist
python312Packages.netbox-bgp
python312Packages.netbox-bgp.dist
python312Packages.netbox-documents
python312Packages.netbox-documents.dist
python312Packages.netbox-reorder-rack
python312Packages.netbox-reorder-rack.dist
python312Packages.strawberry-django
python312Packages.strawberry-django.dist
python312Packages.strawberry-graphql
python312Packages.strawberry-graphql.dist


Fix bug that would report offenses being auto-corrected when they were not

In fixing the some bugs in the Sample cop, I noticed that some offenses were reporting that they had been corrected when they were not. I found out that I should be using fail CorrectionNotPossible instead of return in autocorrect when the error cannot be corrected.

I have extracted a method for the if statement


Fix configuration for rtd yaml file.

There have been some changes to the yaml format since our docs last built, and there is now a 'build' section required. This updates our yaml to match the new requirement. See:
https://docs.readthedocs.io/en/stable/config-file/v2.html

I have tested this against rtd with my fork, and it built successfully, results can her see here:
https://readthedocs.org/projects/test-flask-container-scaffold/builds/23089094/


Is timeLimitInSeconds truly seconds?

I was struggling getting the CPP_Example working for a while.  It would start and instantly exit because the client requested it to be done.
Just for kicks I changed:
my_mission.timeLimitInSeconds(10);
to
my_mission.timeLimitInSeconds(20000);
And suddenly it works -- and runs for 20 seconds.  The field is called "InSeconds", the documentation in the header says it is in seconds, but the behavior is in milliseconds.  What is the expected behavior?
Sauce

Yes it's a bug, the mission XML is in ms but the C method is in seconds.

Pull request #754 if you fancy doing a quick review.

I haven't built from source yet, it'll take a little while.  It looks reasonable to me.
I found the same issue on the python side, but reviewing python_module.cpp it looks like python is just a wrapper to the C++ implementation anyway, right?  So this fix would apply there as well.
Sauce

Don't worry about building / testing unless you are doing that anyway as I'll test. Unfortunately our existing tests don't check how long the actually runs.


Use a slider for frame selection instead of input

A slider is more mobile-friendly and it will allow the user to quickly compare the difference between frames. It will also help clean up user input checking.

The slider works well until the user has a very long video, and the minimum distance results in very large number of frames generated. Could we add a text entry if the user desires as well?

That's a very good point. We can keep the text input and have a slider underneath that changes the text input. Best of both worlds.

That's what I was thinking, glad it sounds good to you too!

#133 adds the sliders


Formatting a png image file takes too long

What are you trying to achieve?

Could you help me understand why a png image of 6MB takes 4x time longer compared to an 20MB image?
Also could you help me with the questions below:

can we skip compression if already compressed at higher level?
can we detect compression through the library?

Have you searched for similar questions?

Yes i have already checked:

https://github.com/lovell/sharp/issues/526
https://github.com/lovell/sharp/issues/2213
Also most of the issues from https://github.com/lovell/sharp/issues?q=png+compress

Are you able to provide a minimal, standalone code sample that demonstrates this question?

// Environment:
//   MacBook Pro (13-inch, 2017, Two Thunderbolt 3 ports)
//   2,5 GHz Dual-Core Intel Core i7
//   MacOS Catalina 10.15.7 (19H15)
// node version v12.16.1
// sharpjs version 0.27.1
const sharp = require('sharp');
const { pipeline, PassThrough } = require('stream');
const { createReadStream, createWriteStream } = require('fs');
sharp.cache(false);
sharp.concurrency(1);

function resizer(readStream, { format, quality }) {
  const metaDataStream = sharp();
  const outputStream = new PassThrough();

  readStream.pipe(metaDataStream);

  metaDataStream.metadata().then(info => {
    console.log({
      ...info,
      xmp: Buffer.from(info.xmp, 'binary').toString()
    });
    let transformer = metaDataStream.clone();
    transformer = transformer.toFormat(format, { quality });

    pipeline([transformer, outputStream], (err) => {
      err && outputStream.emit('error', err);
    });
  }).catch(err => outputStream.emit('error', err));

  return outputStream;
}

function resizeImage(inputImagePath, outputImagePath) {
  console.time(`resize-${inputImagePath}`);
  const readStream = createReadStream(inputImagePath)
  const outputStream = createWriteStream(outputImagePath);

  readStream.on('error', console.error);
  outputStream.on('error', console.error);

  const resizeStream = resizer(readStream, { "format": "png" });
  outputStream.on('close', () => console.timeEnd(`resize-${inputImagePath}`));

  resizeStream.pipe(outputStream);
}

resizeImage('6mb.png', 'test-6mb.png'); //  resize-6mb.png: 12394.095ms
resizeImage('20mb.png', 'test-20mb.png'); //   resize-20mb.png: 3478.747ms

Are you able to provide a sample image that helps explain the question?

I cannot provide the 6mb picture due to copyrights and i cannot attach the 20mb since it's too big. Those are some informations based on the code snippet above:
6mb metadata info:
 {
    format: 'png',
    size: 6709910,
    width: 3840,
    height: 3840,
    space: 'srgb',
    channels: 3,
    depth: 'uchar',
    density: 72,
    isProgressive: false,
    paletteBitDepth: 8,
    hasProfile: false,
    hasAlpha: false,
    xmp: '<?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?> <x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="Adobe XMP Core 6.0-c002 79.164488, 2020/07/10-22:06:53        "> <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> <rdf:Description rdf:about="" xmlns:xmp="http://ns.adobe.com/xap/1.0/" xmlns:xmpMM="http://ns.adobe.com/xap/1.0/mm/" xmlns:stRef="http://ns.adobe.com/xap/1.0/sType/ResourceRef#" xmp:CreatorTool="Adobe Photoshop 22.0 (Windows)" xmpMM:InstanceID="xmp.iid:997FF380648611EB97D5B947DBE4422A" xmpMM:DocumentID="xmp.did:997FF381648611EB97D5B947DBE4422A"> <xmpMM:DerivedFrom stRef:instanceID="xmp.iid:997FF37E648611EB97D5B947DBE4422A" stRef:documentID="xmp.did:997FF37F648611EB97D5B947DBE4422A"/> </rdf:Description> </rdf:RDF> </x:xmpmeta> <?xpacket end="r"?>'
  }

20mb.png metadata info:
   {
    format: 'png',
    size: 21141605,
    width: 5891,
    height: 2271,
    space: 'srgb',
    channels: 4,
    depth: 'uchar',
    density: 72,
    isProgressive: false,
    hasProfile: false,
    hasAlpha: true,
    xmp: '<x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="XMP Core 5.4.0">\n   <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">\n      <rdf:Description rdf:about=""\n            xmlns:exif="http://ns.adobe.com/exif/1.0/"\n            xmlns:tiff="http://ns.adobe.com/tiff/1.0/"\n            xmlns:xmp="http://ns.adobe.com/xap/1.0/"\n            xmlns:exifEX="http://cipa.jp/exif/1.0/"\n            xmlns:photoshop="http://ns.adobe.com/photoshop/1.0/">\n         <exif:ShutterSpeedValue>10815/929</exif:ShutterSpeedValue>\n         <exif:ColorSpace>1</exif:ColorSpace>\n         <exif:FocalLength>103/25</exif:FocalLength>\n         <exif:PixelXDimension>6048</exif:PixelXDimension>\n         <exif:SensingMethod>2</exif:SensingMethod>\n         <exif:SubsecTimeDigitized>581</exif:SubsecTimeDigitized>\n         <exif:FocalLenIn35mmFilm>33</exif:FocalLenIn35mmFilm>\n         <exif:ExposureTime>1/3195</exif:ExposureTime>\n         <exif:WhiteBalance>0</exif:WhiteBalance>\n         <exif:ExifVersion>0221</exif:ExifVersion>\n         <exif:BrightnessValue>8727/794</exif:BrightnessValue>\n         <exif:ComponentsConfiguration>\n            <rdf:Seq>\n               <rdf:li>1</rdf:li>\n               <rdf:li>2</rdf:li>\n               <rdf:li>3</rdf:li>\n               <rdf:li>0</rdf:li>\n            </rdf:Seq>\n         </exif:ComponentsConfiguration>\n         <exif:SubsecTimeOriginal>581</exif:SubsecTimeOriginal>\n         <exif:Flash rdf:parseType="Resource">\n            <exif:Function>False</exif:Function>\n            <exif:Fired>False</exif:Fired>\n            <exif:Return>0</exif:Return>\n            <exif:Mode>0</exif:Mode>\n            <exif:RedEyeMode>False</exif:RedEyeMode>\n         </exif:Flash>\n         <exif:ApertureValue>4845/1918</exif:ApertureValue>\n         <exif:ExposureBiasValue>0</exif:ExposureBiasValue>\n         <exif:FNumber>12/5</exif:FNumber>\n         <exif:PixelYDimension>2332</exif:PixelYDimension>\n         <exif:CustomRendered>6</exif:CustomRendered>\n         <exif:MeteringMode>5</exif:MeteringMode>\n         <exif:ISOSpeedRatings>\n            <rdf:Seq/>\n         </exif:ISOSpeedRatings>\n         <exif:SceneType>1</exif:SceneType>\n         <exif:FlashPixVersion>0100</exif:FlashPixVersion>\n         <exif:SceneCaptureType>0</exif:SceneCaptureType>\n         <tiff:Orientation>1</tiff:Orientation>\n         <tiff:Make>Apple</tiff:Make>\n         <tiff:ResolutionUnit>2</tiff:ResolutionUnit>\n         <tiff:XResolution>72</tiff:XResolution>\n         <tiff:YResolution>72</tiff:YResolution>\n         <tiff:Model>iPhone 5</tiff:Model>\n         <xmp:CreatorTool>8.2</xmp:CreatorTool>\n         <xmp:CreateDate>2015-03-31T14:24:45</xmp:CreateDate>\n         <xmp:ModifyDate>2015-03-31T14:24:45</xmp:ModifyDate>\n         <exifEX:PhotographicSensitivity>50</exifEX:PhotographicSensitivity>\n         <exifEX:LensModel>iPhone 5 back camera 4.12mm f/2.4</exifEX:LensModel>\n         <exifEX:LensSpecification>\n            <rdf:Seq>\n               <rdf:li>103/25</rdf:li>\n               <rdf:li>103/25</rdf:li>\n               <rdf:li>12/5</rdf:li>\n               <rdf:li>12/5</rdf:li>\n            </rdf:Seq>\n         </exifEX:LensSpecification>\n         <exifEX:LensMake>Apple</exifEX:LensMake>\n         <photoshop:DateCreated>2015-03-31T14:24:45</photoshop:DateCreated>\n      </rdf:Description>\n   </rdf:RDF>\n</x:xmpmeta>\n'
  }

Also is there a way to retrieve the compression level of the input image?

Hello, without the specific images it's hard to tell exactly what's different. The "6mb.png" file is palette-based, so that might take longer to decode, plus it is 3 channel, so might miss out on some auto-vectorised optimisations that only benefit 4 channel images.
The first thing to do for any PNG performance problem is reduce the compressionLevel from its default value of 9 - perhaps try 6:
- transformer = transformer.toFormat(format, { quality });
+ transformer = transformer.toFormat(format, { quality, compressionLevel: 6 });

https://sharp.pixelplumbing.com/api-output#png
If you're able to create a standalone repo with complete code and sample images, we can try to run it via callgrind to discover where CPU time is being spent.

Also is there a way to retrieve the compression level of the input image?

Please see https://github.com/lovell/sharp/issues/2300#issuecomment-661127805 and subscribe to https://github.com/randy408/libspng/issues/114

This code is setting quality, which only applies for PNG output if you are using a globally-installed libvips compiled with support for libimagequant. Is this the case?
We are using sharpjs prebuilt with libvips binaries (as far as i know libimagequant is not included). I've seen that quality is ignored where there libimagequant is not included. I've used compressionLevel: 6 and it's faster but again it takes a lot time.
I will try to find a similar image with copyrights that will allow me to upload it in a public repo and let you know.
I subscribed to the libspng issue and read the comment. I will use try to run it with callgrind and discover CPU usage.

Thank you very much for your help.

We're planning to switch the prebuilt binaries to the SIMD-accelerated zlib-ng when there's a stable release, which should bring a big improvement to PNG de/compression times - see https://github.com/lovell/sharp-libvips/issues/25

SIMD-accelerated zlib-ng

The next release will include zlib-ng in the prebuilt binaries - please subscribe to #2604 for updates.

Have you searched for similar questions?

Yes i have already checked:
* #526

* #526

* #2213
  Also most of the issues from https://github.com/lovell/sharp/issues?q=png+compress

You mention issue 526 twice. Do you still remember the third issue that you checked?

Have you searched for similar questions?

Yes i have already checked:
* #526

* #526

* #2213
  Also most of the issues from https://github.com/lovell/sharp/issues?q=png+compress

@klouvas You mention issue 526 twice. Do you still remember the third issue that you checked?

Since i don't remember the 3rd issue i removed the 1 reference from 526. Thank you for pointing it out.


API documentation is missing

the link is spelled wrong, it should be Lux-Design-S3, not Lux-Desig-S3, where did you find the wrong link?

the link is spelled wrong, it should be Lux-Design-S3, not Lux-Desig-S3, where did you find the wrong link?

It's the hyperlink labeled "this document" under the heading Lux AI Season 3 Specifications on Kaggle points to the wrong address as shared above.

thanks, fixed now.


Issue: Missing Generation of pytorch_model.bin File During Model Tuning

Thank you for sharing your interesting project!
Recently, when I ran bash ./script/llama_prune.sh, the pruning step worked perfectly fine. However, during the tuning step, although there were no error information, the generated structure only included the following:

checkpoints-200

model.safetensors
optimizer.pt
rng_state.pth
scheduler.pt
trainer_state.json
training_args.bin

I noticed that the pytorch_model.bin file was not saved. I haven't modified the code, and I am using PyTorch version 2.1.2+cu121. Could you suggest what the possible reason for this might be?

Issue resolved. The reason lies in the newer versions of the transformers library, where safetensors has become the default format, replacing pytorch_model.bin, starting from transformers>=4.33.0. This issue can be addressed by either downgrading to transformers==4.33.0 using pip install transformers==4.33.0, or by setting use_self_serialization=False in model.save_pretrained().
Tracking here: https://github.com/huggingface/transformers/issues/28183

Two updates:

pip install transformers==4.33.0 will lead to the following issue:

AttributeError: 'LlamaTokenizer' object has no attribute 'added_tokens_decoder'. Did you mean: '_added_tokens_decoder'?

If using the latest transformers and setting self_serialization=False, there is still no pytorch_model.bin saved.

This issue still exists.

Issue resolved. The problem is that when constructing the trainer, save_safetensors=False should be set. Otherwise, the above safe_serialization=False will not work.
https://huggingface.co/docs/transformers/v4.36.1/en/main_classes/trainer#transformers.TrainingArguments.save_safetensors

@WilliamYi96 Can we recover pytorch.bin from the safe tensor representation ? I have already run the finetuning on a bigger dataset for some time and want to avoid triggering the learning. or can we resume from the checkpoint and save after running for some steps?

Issue resolved. The problem is that when constructing the trainer, save_safetensors=False should be set. Otherwise, the above safe_serialization=False will not work.
https://huggingface.co/docs/transformers/v4.36.1/en/main_classes/trainer#transformers.TrainingArguments.save_safetensors

Hi, I just set save_safetensors=False, but it still not works.


Mirror local video

Overview
Flip the local video horizontally so that it behaves like a mirror.  This is the standard behavior on most video chat applications.
Screenshot

Test Warnings
As an added bonus, this change clears up a prop type warning the test suite is currently throwing:
Before

After

Cool, thank you for your contribution 🙂