Standardize formatting of manifests/q/Qv2ray/Qv2ray/2.6.3

Microsoft Reviewers: Open in CodeFlow

URL: https://github.com/Qv2ray/Qv2ray/releases/download/v2.6.3/qv2ray-2.6.3-win64.exe
Status Code: 200
(Automated message - build 931)


prepend deltacould path to prevent errors on require

This is the reason for the patch:
deltacloudd -l

/Users/bulat/.rvm/rubies/ruby-1.8.7-p352/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require': no such file to load -- server.rb (LoadError)
	from /Users/bulat/.rvm/rubies/ruby-1.8.7-p352/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
	from /Users/bulat/.rvm/gems/ruby-1.8.7-p352/gems/deltacloud-core-0.4.1/bin/deltacloudd:115
	from /Users/bulat/.rvm/gems/ruby-1.8.7-p352/bin/deltacloudd:19:in `load'
	from /Users/bulat/.rvm/gems/ruby-1.8.7-p352/bin/deltacloudd:19

Hi - sorry for the delay - looks like we haven't checked the github page for a few days - for future its best to also send an email to<EMAIL_ADDRESS>when you issue a pull request (see http://incubator.apache.org/deltacloud/contact.html)
Does your patch fix the issue for you (looks like you're using Windows right?). It doesn't seem to fix it for me under Fedoara - I'll look into this and let you know - thanks for flagging the problem,
marios

no idea why i thought that was windows - the paths above look more like Mac OS? (doesn't make a difference, just out of curiosity)

Hi Bulat, thanks for taking the time to let us know about this issue - the issue was fixed with a minor addition to your patch and is now committed into the apache repo - if you make a pull request you should get the fix.
Closing the pull request.

Thanks

this is still not fixed in version 0.4.1:
bulat@Bulat-Shakirzyanovs-MacBook-Pro:~$ gem search deltacloud

*** LOCAL GEMS ***

deltacloud-core (0.4.1)

bulat@Bulat-Shakirzyanovs-MacBook-Pro:~$ deltacloudd -l
/Users/bulat/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:55:in `require': cannot load such file -- server.rb (LoadError)
	from /Users/bulat/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:55:in `require'
	from /Users/bulat/.rvm/gems/ruby-1.9.3-p0/gems/deltacloud-core-0.4.1/bin/deltacloudd:115:in `<top (required)>'
	from /Users/bulat/.rvm/gems/ruby-1.9.3-p0/bin/deltacloudd:19:in `load'
	from /Users/bulat/.rvm/gems/ruby-1.9.3-p0/bin/deltacloudd:19:in `<main>'

offending lines (114, 115):
if options[:drivers]
  require 'server.rb'

to fix, I changed line 115 to
  require File.dirname(__FILE__)+'/../server.rb'

still more fixes are necessary as for example lib/deltacloud/base_driver/base_driver.rb contains
require 'lib/deltacloud/base_driver/exceptions'

which should be
require 'deltacloud/base_driver/exceptions'

nevermind, I think this is because the gem needs to be updated as the code there is different from sources here


add start docker silently, down and stop docker command shortcut to m…

…waa-local-env
*Issue #, if available: None
*Description of changes:
Adding shortcut to start Airflow docker silently as a daemon, docker down and docker stop. The shortcut docker down allow the docker to be suspended and can be start anytime without having to reload docker from scratch. User can continue where they left off. Stop allow the user to shutdown the docker completely.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thanks for fixing the doc/command name mismatch.
I'll approve for now. But I'm still interested to hear @subashcanapathy's latest thoughts on having stop and down . He seemed keen on only having one. I'm agnostic here, I see the benefits for both approaches and will accept either.

Thank you @o-nikolas !

@o-nikolas this seems like a useful and practical addition. Can it be merged?

@o-nikolas this seems like a useful and practical addition. Can it be merged?

@ashley-mallia, See previous comment threads. Waiting on a second pass from @subashcanapathy since he had some concerns about the PR.


GH-2516: Fix code example in the documentation for for Non-Blocking Retries - DLT Failure Behavior

Fixes #2516

Thanks, once again, - would you mind also fixing these javadocs?
https://github.com/spring-projects/spring-kafka/blob/d34394eeb555103f9c449f088dc8d92671de103d/spring-kafka/src/main/java/org/springframework/kafka/retrytopic/RetryTopicConfigurer.java#L176-L183

Thanks; merged as 1dcf11dcd with updated copyright for changed file.

...and cherry-picked to 2.9.x as 4b45d2b19


Merge webapp-master into master

Summary
Merge webapp-master into master.

@enahum ready for review


Removing the downsampling layer leads to a high WER

Hi,
Using the default conformer_ctc/train.py and conformer_ctc/decode.py scripts of the Aishell example, I can get a training total loss=0.2 and a validation total loss=0.16 at epoch=7, and a decode WER around 8.x.
But since the default model with 4 times downsampling, gets a resolution of 0.04s for the timestamps. I hope to remove the downsampling layer to get more accurate timestampes. After changing the downsample layer to a linear layer, or modifying the stride and padding of the convolutional network and also, adjust the subsampling_factor to 1.
At epoch=1, the training total loss can be reduced to 0.14, and the validation total loss is 0.15, but the WER of decoding is 88.xx.
The decoding result is poor after the model seems to have converged, and only the first few words of each sentence can be decoded.

default model: max-duration = 200
log:
...
2022-11-25 04:28:40,033 INFO [train.py:512] (3/4) Epoch 7, batch 1850, loss[ctc_loss=0.1082, att_loss=0.2322, loss=0.195, over 4796.00 frames. ], tot_loss[ctc_loss=0.1164, att_loss=0.239, loss=0.2022, over 956037.74 frames. ], batch size: 29
2022-11-25 04:29:14,264 INFO [train.py:512] (3/4) Epoch 7, batch 1900, loss[ctc_loss=0.118, att_loss=0.249, loss=0.2097, over 4843.00 frames. ], tot_loss[ctc_loss=0.1163, att_loss=0.2389, loss=0.2021, over 954733.93 frames. ], batch size: 37
2022-11-25 04:29:47,567 INFO [train.py:512] (3/4) Epoch 7, batch 1950, loss[ctc_loss=0.09974, att_loss=0.2296, loss=0.1907, over 4856.00 frames. ], tot_loss[ctc_loss=0.1157, att_loss=0.2388, loss=0.2019, over 957035.01 frames. ], batch size: 37
2022-11-25 04:30:19,513 INFO [train.py:512] (3/4) Epoch 7, batch 2000, loss[ctc_loss=0.1163, att_loss=0.2486, loss=0.2089, over 4832.00 frames. ], tot_loss[ctc_loss=0.1159, att_loss=0.2383, loss=0.2016, over 954000.42 frames. ], batch size: 51
2022-11-25 04:30:19,514 INFO [train.py:529] (3/4) Computing validation loss
2022-11-25 04:30:44,486 INFO [train.py:538] (3/4) Epoch 7, validation: ctc_loss=0.05068, att_loss=0.211, loss=0.1629, over 1622729.00 frames.
2022-11-25 04:31:15,483 INFO [train.py:512] (3/4) Epoch 7, batch 2050, loss[ctc_loss=0.0982, att_loss=0.2067, loss=0.1741, over 4744.00 frames. ], tot_loss[ctc_loss=0.1151, att_loss=0.2374, loss=0.2007, over 954119.93 frames. ], batch size: 26
2022-11-25 04:31:47,114 INFO [train.py:512] (3/4) Epoch 7, batch 2100, loss[ctc_loss=0.1601, att_loss=0.2742, loss=0.24, over 4878.00 frames. ], tot_loss[ctc_loss=0.1149, att_loss=0.2379, loss=0.201, over 954653.91 frames. ], batch size: 43
2022-11-25 04:32:44,022 INFO [train.py:512] (3/4) Epoch 8, batch 0, loss[ctc_loss=0.1279, att_loss=0.2415, loss=0.2074, over 4861.00 frames. ], tot_loss[ctc_loss=0.1279, att_loss=0.2415, loss=0.2074, over 4861.00 frames. ], batch size: 57
...

linear input layer:  max-duration = 100
self.encoder_embed = nn.Linear(num_features, d_model)

log:
...
2022-11-25 08:04:00,335 INFO [train.py:512] (3/4) Epoch 1, batch 1850, loss[ctc_loss=0.1954, att_loss=0.1571, loss=0.1686, over 9806.00 frames. ], tot_loss[ctc_loss=0.196, att_loss=0.1606, loss=0.1712, over 1894948.66 frames. ], batch size: 21
2022-11-25 08:04:42,968 INFO [train.py:512] (3/4) Epoch 1, batch 1900, loss[ctc_loss=0.1863, att_loss=0.1531, loss=0.1631, over 9688.00 frames. ], tot_loss[ctc_loss=0.1955, att_loss=0.1601, loss=0.1707, over 1896855.38 frames. ], batch size: 19
2022-11-25 08:05:27,511 INFO [train.py:512] (3/4) Epoch 1, batch 1950, loss[ctc_loss=0.2106, att_loss=0.165, loss=0.1787, over 9540.00 frames. ], tot_loss[ctc_loss=0.1948, att_loss=0.1595, loss=0.1701, over 1895997.71 frames. ], batch size: 26
2022-11-25 08:06:11,834 INFO [train.py:512] (3/4) Epoch 1, batch 2000, loss[ctc_loss=0.2044, att_loss=0.171, loss=0.181, over 9832.00 frames. ], tot_loss[ctc_loss=0.1945, att_loss=0.1591, loss=0.1697, over 1897898.38 frames. ], batch size: 21
2022-11-25 08:06:11,834 INFO [train.py:529] (3/4) Computing validation loss
2022-11-25 08:07:06,694 INFO [train.py:538] (3/4) Epoch 1, validation: ctc_loss=0.1893, att_loss=0.1604, loss=0.1691, over 6512315.00 frames.
...
2022-11-25 08:39:03,480 INFO [train.py:512] (3/4) Epoch 1, batch 4150, loss[ctc_loss=0.2197, att_loss=0.1684, loss=0.1838, over 9434.00 frames. ], tot_loss[ctc_loss=0.1898, att_loss=0.15, loss=0.1619, over 1898860.84 frames. ], batch size: 19
2022-11-25 08:39:45,326 INFO [train.py:512] (3/4) Epoch 1, batch 4200, loss[ctc_loss=0.1869, att_loss=0.1434, loss=0.1564, over 9858.00 frames. ], tot_loss[ctc_loss=0.1903, att_loss=0.1503, loss=0.1623, over 1901405.51 frames. ], batch size: 23
2022-11-25 08:40:27,532 INFO [train.py:512] (3/4) Epoch 1, batch 4250, loss[ctc_loss=0.2009, att_loss=0.1585, loss=0.1712, over 9709.00 frames. ], tot_loss[ctc_loss=0.1898, att_loss=0.1497, loss=0.1617, over 1901266.65 frames. ], batch size: 24
2022-11-25 08:41:11,178 INFO [train.py:512] (3/4) Epoch 1, batch 4300, loss[ctc_loss=0.1829, att_loss=0.143, loss=0.155, over 9773.00 frames. ], tot_loss[ctc_loss=0.1884, att_loss=0.1487, loss=0.1606, over 1887813.74 frames. ], batch size: 18
2022-11-25 08:41:56,916 INFO [train.py:512] (3/4) Epoch 2, batch 0, loss[ctc_loss=0.1938, att_loss=0.1513, loss=0.164, over 9707.00 frames. ], tot_loss[ctc_loss=0.1938, att_loss=0.1513, loss=0.164, over 9707.00 frames. ], batch size: 31
...

conv input layer:  max-duration = 80
        self.conv = nn.Sequential(
            nn.Conv2d(
                in_channels=1, out_channels=odim, kernel_size=3, stride=1, padding=1
            ),
            nn.ReLU(),
            nn.Conv2d(
                in_channels=odim, out_channels=odim, kernel_size=3, stride=1, padding=1
            ),
            nn.ReLU(),
        )

        self.out = nn.Linear(odim * idim, odim)

2022-11-27 10:33:11,172 INFO [train.py:512] (3/4) Epoch 2, batch 3850, loss[ctc_loss=0.1931, att_loss=0.147, loss=0.1608, over 7543.00 frames. ], tot_loss[ctc_loss=0.1802, att_loss=0.1353, loss=0.1487, over 1504628.66 frames. ], batch size: 23
2022-11-27 10:34:17,783 INFO [train.py:512] (3/4) Epoch 2, batch 3900, loss[ctc_loss=0.1929, att_loss=0.1412, loss=0.1568, over 7913.00 frames. ], tot_loss[ctc_loss=0.1799, att_loss=0.1347, loss=0.1482, over 1503819.94 frames. ], batch size: 16
2022-11-27 10:35:23,413 INFO [train.py:512] (3/4) Epoch 2, batch 3950, loss[ctc_loss=0.193, att_loss=0.1423, loss=0.1575, over 7530.00 frames. ], tot_loss[ctc_loss=0.1803, att_loss=0.135, loss=0.1486, over 1507065.92 frames. ], batch size: 23
2022-11-27 10:36:29,280 INFO [train.py:512] (3/4) Epoch 2, batch 4000, loss[ctc_loss=0.1497, att_loss=0.1091, loss=0.1213, over 7659.00 frames. ], tot_loss[ctc_loss=0.1805, att_loss=0.1352, loss=0.1488, over 1505438.38 frames. ], batch size: 11
2022-11-27 10:36:29,281 INFO [train.py:529] (3/4) Computing validation loss
2022-11-27 10:38:27,992 INFO [train.py:538] (3/4) Epoch 2, validation: ctc_loss=0.1758, att_loss=0.145, loss=0.1543, over 6512315.00 frames.
...
2022-11-27 11:07:12,591 INFO [train.py:512] (3/4) Epoch 2, batch 5300, loss[ctc_loss=0.1973, att_loss=0.1465, loss=0.1617, over 7707.00 frames. ], tot_loss[ctc_loss=0.1805, att_loss=0.1338, loss=0.1478, over 1503153.06 frames. ], batch size: 19
2022-11-27 11:08:17,623 INFO [train.py:512] (3/4) Epoch 2, batch 5350, loss[ctc_loss=0.167, att_loss=0.1244, loss=0.1372, over 7403.00 frames. ], tot_loss[ctc_loss=0.1789, att_loss=0.1325, loss=0.1464, over 1499715.62 frames. ], batch size: 12
2022-11-27 11:09:21,871 INFO [train.py:512] (3/4) Epoch 2, batch 5400, loss[ctc_loss=0.126, att_loss=0.09863, loss=0.1068, over 7099.00 frames. ], tot_loss[ctc_loss=0.1772, att_loss=0.1314, loss=0.1451, over 1490781.35 frames. ], batch size: 8
2022-11-27 11:10:54,249 INFO [train.py:512] (3/4) Epoch 3, batch 0, loss[ctc_loss=0.192, att_loss=0.1493, loss=0.1621, over 7537.00 frames. ], tot_loss[ctc_loss=0.192, att_loss=0.1493, loss=0.1621, over 7537.00 frames. ], batch size: 22
...

At epoch=1, the training total loss can be reduced to 0.14, and the validation total loss is 0.15, but the WER of decoding is 88.xx.

Do you also change the subsampling factor for decode.py?

Yes, I do. The decode params are as follows:
params = AttributeDict(
        {
            # parameters for conformer
            "subsampling_factor": 1,
            "feature_dim": 80,
            "nhead": 4,
            "attention_dim": 512,
            "num_encoder_layers": 12,
            "num_decoder_layers": 6,
            "vgg_frontend": False,
            "use_feat_batchnorm": True,
            # parameters for decoder
            "search_beam": 20,
            "output_beam": 10,
            "min_active_states": 30,
            "max_active_states": 10000,
            "use_double_scores": True,
            "env_info": get_env_info(),
        }
    )

and the decode command is :
python conformer_ctc/decode.py --nbest-scale 0.5 \
                               --epoch 2 \
                               --avg 1 \
                               --method attention-decoder \
                               --max-duration 20 \
                               --num-paths 1 \
                               --exp-dir conformer_ctc/exp_ss1_conv

At epoch=1, the training total loss can be reduced to 0.14, and the validation total loss is 0.15, but the WER of decoding is 88.xx.

Do you also change the subsampling factor for decode.py?

Could you please show the output of
git diff

so that we can see what exactly has been changed

diff --git a/egs/aishell/ASR/conformer_ctc/decode.py b/egs/aishell/ASR/conformer_ctc/decode.py
index 751b7d5..f561d69 100755
--- a/egs/aishell/ASR/conformer_ctc/decode.py
+++ b/egs/aishell/ASR/conformer_ctc/decode.py
@@ -57,14 +57,14 @@ def get_parser():
     parser.add_argument(
         "--epoch",
         type=int,
-        default=49,
+        default=2,
         help="It specifies the checkpoint to use for decoding."
         "Note: Epoch counts from 0.",
     )
     parser.add_argument(
         "--avg",
         type=int,
-        default=20,
+        default=1,
         help="Number of checkpoints to average. Automatically select "
         "consecutive checkpoints before the checkpoint specified by "
         "'--epoch'. ",
@@ -115,7 +115,7 @@ def get_parser():
     parser.add_argument(
         "--exp-dir",
         type=str,
-        default="conformer_ctc/exp",
+        default="conformer_ctc/exp_ss",
         help="The experiment dir",
     )

@@ -142,7 +142,7 @@ def get_params() -> AttributeDict:
     params = AttributeDict(
         {
             # parameters for conformer
-            "subsampling_factor": 4,
+            "subsampling_factor": 1,
             "feature_dim": 80,
             "nhead": 4,
             "attention_dim": 512,
@@ -151,8 +151,8 @@ def get_params() -> AttributeDict:
             "vgg_frontend": False,
             "use_feat_batchnorm": True,
             # parameters for decoder
-            "search_beam": 20,
-            "output_beam": 7,
+            "search_beam": 10,
+            "output_beam": 10,
             "min_active_states": 30,
             "max_active_states": 10000,
             "use_double_scores": True,
@@ -373,9 +373,10 @@ def decode_dataset(

     results = defaultdict(list)
     for batch_idx, batch in enumerate(dl):
+
         texts = batch["supervisions"]["text"]
         cut_ids = [cut.id for cut in batch["supervisions"]["cut"]]
-
+        #print(len(texts))
         hyps_dict = decode_one_batch(
             params=params,
             model=model,
@@ -545,8 +546,9 @@ def main():
     args.return_cuts = True
     aishell = AishellAsrDataModule(args)
     test_cuts = aishell.test_cuts()
+    print('create test dataloaders')
     test_dl = aishell.test_dataloaders(test_cuts)
-
+    print('create test dataloaders successfully')
     test_sets = ["test"]
     test_dls = [test_dl]

diff --git a/egs/aishell/ASR/conformer_ctc/export.py b/egs/aishell/ASR/conformer_ctc/export.py
index 42b8c29..4a07ee3 100644
--- a/egs/aishell/ASR/conformer_ctc/export.py
+++ b/egs/aishell/ASR/conformer_ctc/export.py
@@ -85,7 +85,7 @@ def get_params() -> AttributeDict:
     params = AttributeDict(
         {
             "feature_dim": 80,
-            "subsampling_factor": 4,
+            "subsampling_factor": 1,
             "use_feat_batchnorm": True,
             "attention_dim": 512,
             "nhead": 4,
diff --git a/egs/aishell/ASR/conformer_ctc/subsampling.py b/egs/aishell/ASR/conformer_ctc/subsampling.py
index 542fb03..b7d2d41 100644
--- a/egs/aishell/ASR/conformer_ctc/subsampling.py
+++ b/egs/aishell/ASR/conformer_ctc/subsampling.py
@@ -43,15 +43,16 @@ class Conv2dSubsampling(nn.Module):
         super().__init__()
         self.conv = nn.Sequential(
             nn.Conv2d(
-                in_channels=1, out_channels=odim, kernel_size=3, stride=2
+                in_channels=1, out_channels=odim, kernel_size=3, stride=1, padding=1
             ),
             nn.ReLU(),
             nn.Conv2d(
-                in_channels=odim, out_channels=odim, kernel_size=3, stride=2
+                in_channels=odim, out_channels=odim, kernel_size=3, stride=1, padding=1
             ),
             nn.ReLU(),
         )
-        self.out = nn.Linear(odim * (((idim - 1) // 2 - 1) // 2), odim)
+        #self.out = nn.Linear(odim * (((idim - 1) // 2 - 1) // 2), odim)
+        self.out = nn.Linear(odim * idim, odim)

     def forward(self, x: torch.Tensor) -> torch.Tensor:
         """Subsample x.
diff --git a/egs/aishell/ASR/conformer_ctc/train.py b/egs/aishell/ASR/conformer_ctc/train.py
index a228cc1..52c14a3 100755
--- a/egs/aishell/ASR/conformer_ctc/train.py
+++ b/egs/aishell/ASR/conformer_ctc/train.py
@@ -57,7 +57,7 @@ def get_parser():
     parser.add_argument(
         "--world-size",
         type=int,
-        default=1,
+        default=4,
         help="Number of GPUs for DDP training.",
     )

@@ -95,7 +95,7 @@ def get_parser():
     parser.add_argument(
         "--exp-dir",
         type=str,
-        default="conformer_ctc/exp",
+        default="conformer_ctc/exp_ss1_conv",
         help="""The experiment dir.
         It specifies the directory where all training related
         files, e.g., checkpoints, log, etc, are saved
@@ -203,7 +203,7 @@ def get_params() -> AttributeDict:
             "reduction": "sum",
             "use_double_scores": True,
             # parameters for conformer
-            "subsampling_factor": 4,
+            "subsampling_factor": 1,
             "feature_dim": 80,
             "attention_dim": 512,
             "nhead": 4,
@@ -485,9 +485,10 @@ def train_one_epoch(
     tot_loss = MetricsTracker()

     for batch_idx, batch in enumerate(train_dl):
+
         params.batch_idx_train += 1
         batch_size = len(batch["supervisions"]["text"])
-
+        #print(batch_size)
         loss, loss_info = compute_loss(
             params=params,
             model=model,
@@ -602,7 +603,7 @@ def run(rank, world_size, args):
         vgg_frontend=False,
         use_feat_batchnorm=params.use_feat_batchnorm,
     )
-
+    print(model)
     checkpoints = load_checkpoint_if_available(params=params, model=model)

     model.to(device)
diff --git a/egs/aishell/ASR/conformer_ctc/transformer.py b/egs/aishell/ASR/conformer_ctc/transformer.py
index f93914a..90f575a 100644
--- a/egs/aishell/ASR/conformer_ctc/transformer.py
+++ b/egs/aishell/ASR/conformer_ctc/transformer.py
@@ -81,8 +81,8 @@ class Transformer(nn.Module):
         self.num_features = num_features
         self.num_classes = num_classes
         self.subsampling_factor = subsampling_factor
-        if subsampling_factor != 4:
-            raise NotImplementedError("Support only 'subsampling_factor=4'.")
+        #if subsampling_factor != 4:
+        #    raise NotImplementedError("Support only 'subsampling_factor=4'.")

         # self.encoder_embed converts the input of shape (N, T, num_classes)
         # to the shape (N, T//subsampling_factor, d_model).
@@ -93,7 +93,7 @@ class Transformer(nn.Module):
             self.encoder_embed = VggSubsampling(num_features, d_model)
         else:
             self.encoder_embed = Conv2dSubsampling(num_features, d_model)
-
+            #self.encoder_embed = nn.Linear(num_features, d_model)
         self.encoder_pos = PositionalEncoding(d_model, dropout)

         encoder_layer = TransformerEncoderLayer(
diff --git a/egs/aishell/ASR/tdnn_lstm_ctc/asr_datamodule.py b/egs/aishell/ASR/tdnn_lstm_ctc/asr_datamodule.py
index d24ba6b..8fe179c 100644
--- a/egs/aishell/ASR/tdnn_lstm_ctc/asr_datamodule.py
+++ b/egs/aishell/ASR/tdnn_lstm_ctc/asr_datamodule.py
@@ -78,7 +78,7 @@ class AishellAsrDataModule:
         group.add_argument(
             "--max-duration",
             type=int,
-            default=200.0,
+            default=80.0,
             help="Maximum pooled recordings duration (seconds) in a "
             "single batch. You can reduce it if it causes CUDA OOM.",
         )

Could you please show the output of
git diff

so that we can see what exactly has been changed

and the CER for ctc-decoding is
ctc-decoding	87.34

At epoch=1, the training total loss can be reduced to 0.14, and the validation total loss is 0.15, but the WER of decoding is 88.xx.

The loss is averaged over all frames. Since you are using subsampling factor==1, you effectively increase the number of feature frames when computing the average.
I suggest that you train the model for more epochs.

I suggest that you train the model for more epochs.

I also trained a model with a linear input layer over 63 epochs, the training loss is down to 0.1018, but the validation loss won't down since epoch 9(val loss=0.15) and the val loss at epoch 63 is up to 0.1668. Is this seem like an overfitting problem?
And still, only the first few words of each sentence can be decoded.
training log:
...
2022-11-25 00:10:59,013 INFO [train.py:529] (3/4) Computing validation loss
2022-11-25 00:12:00,241 INFO [train.py:538] (3/4) Epoch 63, validation: ctc_loss=0.169, att_loss=0.1659, loss=0.1668, over 6512315.00 frames.
...
2022-11-25 00:23:03,693 INFO [train.py:512] (3/4) Epoch 63, batch 7100, loss[ctc_loss=0.1711, att_loss=0.07278, loss=0.1023, over 5779.00 frames. ], tot_loss[ctc_loss=0.1755, att_loss=0.07464, loss=0.1049, over 1120268.00 frames. ], batch size: 10
2022-11-25 00:23:32,981 INFO [train.py:512] (3/4) Epoch 63, batch 7150, loss[ctc_loss=0.1624, att_loss=0.06555, loss=0.09462, over 5505.00 frames. ], tot_loss[ctc_loss=0.176, att_loss=0.07467, loss=0.1051, over 1117408.24 frames. ], batch size: 8
2022-11-25 00:24:02,118 INFO [train.py:512] (3/4) Epoch 63, batch 7200, loss[ctc_loss=0.1812, att_loss=0.07223, loss=0.1049, over 5441.00 frames. ], tot_loss[ctc_loss=0.1752, att_loss=0.0741, loss=0.1044, over 1110711.99 frames. ], batch size: 10
2022-11-25 00:24:31,838 INFO [train.py:512] (3/4) Epoch 63, batch 7250, loss[ctc_loss=0.185, att_loss=0.09373, loss=0.1211, over 5256.00 frames. ], tot_loss[ctc_loss=0.1743, att_loss=0.07371, loss=0.1039, over 1103355.27 frames. ], batch size: 21
2022-11-25 00:25:01,943 INFO [train.py:512] (3/4) Epoch 63, batch 7300, loss[ctc_loss=0.1681, att_loss=0.07336, loss=0.1018, over 5640.00 frames. ], tot_loss[ctc_loss=0.1738, att_loss=0.07414, loss=0.104, over 1094086.30 frames. ], batch size: 12
2022-11-25 00:25:34,936 INFO [train.py:512] (3/4) Epoch 64, batch 0, loss[ctc_loss=0.167, att_loss=0.07096, loss=0.09978, over 5454.00 frames. ], tot_loss[ctc_loss=0.167, att_loss=0.07096, loss=0.09978, over 5454.00 frames. ], batch size: 10
2022-11-25 00:26:05,136 INFO [train.py:512] (3/4) Epoch 64, batch 50, loss[ctc_loss=0.1665, att_loss=0.06461, loss=0.09519, over 5745.00 frames. ], tot_loss[ctc_loss=0.1684, att_loss=0.07287, loss=0.1015, over 251815.98 frames. ], batch size: 9

For overfitting, I raise the dropout rate of the model to 0.2. Here is the log of a linear input layer, and the attention loss of the validation set increases rather than decreases after the learning rate begins to decline.

I had a similar problem, have you solved it yet?

Each time the test set in aishell1 is decoded it outputs some uniform character, for example:'zan si a da sa san', 'zan si a da sa san', 'zan si a da sa san', 'zan si a da sa san', 'zan si a da sa san', 'zan si a da sa san', 'zan si a da sa san', 'zan si a da sa san', 'zan si a da sa san', 'zan si a da sa san', 'zan si  a da sa san', 'zan i  a da sa san', 'zan i  a da sa san', 'zan i  a da sa san', 'zan si  a da san san', 'zan si  a da san san'

@GabrielHaoHao
If you are using the conformer_cte recipe you shouldn't be, as there are much better recipes now.
Are you sure you didn't make any changes to the scripts?

i think it works best with the downsample layers.  you can show convergence plots if you want- maybe your model did not converge.  but that is not a good recipe.

@oshindow I successfully recognized aishell's test audio by trying to use a linear layer instead of a downsampling layer in the wenet framework.

2023-07-24 23:41:34,159 INFO [train.py:554] (1/2) Epoch 0, batch 8400, loss[ctc_loss=0.302, att_loss=0.2322, loss=0.2531, over 9505.00 frames. utt_duration=559.1 frames, utt_pad_proportion=0.0139, over 17.00 utterances.], tot_loss[ctc_loss=0.2931, att_loss=0.2315, loss=0.25, over 1900551.80 frames. utt_duration=460.4 frames, utt_pad_proportion=0.02389, over 4128.24 utterances.], batch size: 17
2023-07-24 23:41:47,338 INFO [train.py:554] (0/2) Epoch 0, batch 8450, loss[ctc_loss=0.292, att_loss=0.2407, loss=0.2561, over 9406.00 frames. utt_duration=495.1 frames, utt_pad_proportion=0.01384, over 19.00 utterances.], tot_loss[ctc_loss=0.2971, att_loss=0.2339, loss=0.2529, over 1904452.82 frames. utt_duration=450.2 frames, utt_pad_proportion=0.02336, over 4230.63 utterances.], batch size: 19
2023-07-24 23:41:47,349 INFO [train.py:554] (1/2) Epoch 0, batch 8450, loss[ctc_loss=0.2969, att_loss=0.2499, loss=0.264, over 9815.00 frames. utt_duration=467.4 frames, utt_pad_proportion=0.01396, over 21.00 utterances.], tot_loss[ctc_loss=0.2965, att_loss=0.2326, loss=0.2518, over 1901661.73 frames. utt_duration=453.1 frames, utt_pad_proportion=0.0249, over 4197.36 utterances.], batch size: 21
2023-07-24 23:42:00,848 INFO [train.py:554] (1/2) Epoch 0, batch 8500, loss[ctc_loss=0.2854, att_loss=0.2394, loss=0.2532, over 9547.00 frames. utt_duration=454.6 frames, utt_pad_proportion=0.01384, over 21.00 utterances.], tot_loss[ctc_loss=0.2959, att_loss=0.2325, loss=0.2515, over 1899738.04 frames. utt_duration=456.2 frames, utt_pad_proportion=0.02624, over 4164.40 utterances.], batch size: 21
2023-07-24 23:42:00,863 INFO [train.py:554] (0/2) Epoch 0, batch 8500, loss[ctc_loss=0.2862, att_loss=0.2374, loss=0.252, over 9610.00 frames. utt_duration=417.8 frames, utt_pad_proportion=0.01223, over 23.00 utterances.], tot_loss[ctc_loss=0.2952, att_loss=0.2333, loss=0.2519, over 1898918.12 frames. utt_duration=455.8 frames, utt_pad_proportion=0.02384, over 4165.73 utterances.], batch size: 23
2023-07-24 23:42:14,960 INFO [train.py:554] (0/2) Epoch 0, batch 8550, loss[ctc_loss=0.2797, att_loss=0.2419, loss=0.2532, over 9791.00 frames. utt_duration=543.9 frames, utt_pad_proportion=0.01101, over 18.00 utterances.], tot_loss[ctc_loss=0.2907, att_loss=0.2307, loss=0.2487, over 1892816.13 frames. utt_duration=468.8 frames, utt_pad_proportion=0.02474, over 4037.20 utterances.], batch size: 18
2023-07-24 23:42:14,962 INFO [train.py:554] (1/2) Epoch 0, batch 8550, loss[ctc_loss=0.2662, att_loss=0.2232, loss=0.2361, over 9569.00 frames. utt_duration=637.9 frames, utt_pad_proportion=0.01401, over 15.00 utterances.], tot_loss[ctc_loss=0.2952, att_loss=0.2316, loss=0.2507, over 1901428.71 frames. utt_duration=464.4 frames, utt_pad_proportion=0.02547, over 4094.69 utterances.], batch size: 15
————————————————————————————————————————————————————————
2023-07-25 10:01:24,117 INFO [train.py:554] (0/2) Epoch 15, batch 8200, loss[ctc_loss=0.2702, att_loss=0.2243, loss=0.238, over 9418.00 frames. utt_duration=495.7 frames, utt_pad_proportion=0.01258, over 19.00 utterances.], tot_loss[ctc_loss=0.2519, att_loss=0.2082, loss=0.2213, over 1894727.04 frames. utt_duration=464 frames, utt_pad_proportion=0.02453, over 4083.40 utterances.], batch size: 19
2023-07-25 10:01:38,358 INFO [train.py:554] (0/2) Epoch 15, batch 8250, loss[ctc_loss=0.2306, att_loss=0.1997, loss=0.209, over 9728.00 frames. utt_duration=694.9 frames, utt_pad_proportion=0.01856, over 14.00 utterances.], tot_loss[ctc_loss=0.2534, att_loss=0.2094, loss=0.2226, over 1893286.78 frames. utt_duration=453.7 frames, utt_pad_proportion=0.0258, over 4173.27 utterances.], batch size: 14
2023-07-25 10:01:38,366 INFO [train.py:554] (1/2) Epoch 15, batch 8250, loss[ctc_loss=0.2608, att_loss=0.212, loss=0.2266, over 9575.00 frames. utt_duration=456 frames, utt_pad_proportion=0.008799, over 21.00 utterances.], tot_loss[ctc_loss=0.2564, att_loss=0.2114, loss=0.2249, over 1905211.70 frames. utt_duration=444.6 frames, utt_pad_proportion=0.02233, over 4285.64 utterances.], batch size: 21
2023-07-25 10:01:51,360 INFO [train.py:554] (1/2) Epoch 15, batch 8300, loss[ctc_loss=0.2547, att_loss=0.2134, loss=0.2258, over 9773.00 frames. utt_duration=574.9 frames, utt_pad_proportion=0.01897, over 17.00 utterances.], tot_loss[ctc_loss=0.2575, att_loss=0.2123, loss=0.2259, over 1906473.26 frames. utt_duration=439.9 frames, utt_pad_proportion=0.02228, over 4333.47 utterances.], batch size: 17
2023-07-25 10:01:51,360 INFO [train.py:554] (0/2) Epoch 15, batch 8300, loss[ctc_loss=0.2475, att_loss=0.2067, loss=0.2189, over 9541.00 frames. utt_duration=596.3 frames, utt_pad_proportion=0.009448, over 16.00 utterances.], tot_loss[ctc_loss=0.2528, att_loss=0.2089, loss=0.2221, over 1893144.00 frames. utt_duration=461 frames, utt_pad_proportion=0.02574, over 4106.92 utterances.], batch size: 16
2023-07-25 10:02:04,495 INFO [train.py:554] (1/2) Epoch 15, batch 8350, loss[ctc_loss=0.2687, att_loss=0.2222, loss=0.2362, over 9697.00 frames. utt_duration=510.4 frames, utt_pad_proportion=0.01283, over 19.00 utterances.], tot_loss[ctc_loss=0.258, att_loss=0.2128, loss=0.2264, over 1905677.21 frames. utt_duration=445 frames, utt_pad_proportion=0.02254, over 4282.07 utterances.], batch size: 19
2023-07-25 10:02:04,513 INFO [train.py:554] (0/2) Epoch 15, batch 8350, loss[ctc_loss=0.2806, att_loss=0.2226, loss=0.24, over 9643.00 frames. utt_duration=507.5 frames, utt_pad_proportion=0.01642, over 19.00 utterances.], tot_loss[ctc_loss=0.2544, att_loss=0.2099, loss=0.2233, over 1895872.28 frames. utt_duration=458.6 frames, utt_pad_proportion=0.02538, over 4134.12 utterances.], batch size: 19
2023-07-25 10:02:17,841 INFO [train.py:554] (1/2) Epoch 15, batch 8400, loss[ctc_loss=0.2683, att_loss=0.2176, loss=0.2328, over 9719.00 frames. utt_duration=405 frames, utt_pad_proportion=0.0147, over 24.00 utterances.], tot_loss[ctc_loss=0.258, att_loss=0.2127, loss=0.2263, over 1907905.56 frames. utt_duration=447.4 frames, utt_pad_proportion=0.02173, over 4264.28 utterances.], batch size: 24
2023-07-25 10:02:17,845 INFO [train.py:554] (0/2) Epoch 15, batch 8400, loss[ctc_loss=0.2597, att_loss=0.2094, loss=0.2245, over 9821.00 frames. utt_duration=467.7 frames, utt_pad_proportion=0.01336, over 21.00 utterances.], tot_loss[ctc_loss=0.2552, att_loss=0.2104, loss=0.2239, over 1896564.32 frames. utt_duration=451.6 frames, utt_pad_proportion=0.02504, over 4199.72 utterances.], batch size: 21
2023-07-25 10:02:31,309 INFO [train.py:554] (0/2) Epoch 15, batch 8450, loss[ctc_loss=0.2676, att_loss=0.228, loss=0.2399, over 9849.00 frames. utt_duration=579.4 frames, utt_pad_proportion=0.01134, over 17.00 utterances.], tot_loss[ctc_loss=0.2555, att_loss=0.2111, loss=0.2244, over 1899608.11 frames. utt_duration=451.6 frames, utt_pad_proportion=0.02494, over 4206.08 utterances.], batch size: 17
2023-07-25 10:02:31,315 INFO [train.py:554] (1/2) Epoch 15, batch 8450, loss[ctc_loss=0.2395, att_loss=0.2017, loss=0.213, over 9233.00 frames. utt_duration=615.5 frames, utt_pad_proportion=0.01672, over 15.00 utterances.], tot_loss[ctc_loss=0.2566, att_loss=0.2124, loss=0.2257, over 1902985.92 frames. utt_duration=451.1 frames, utt_pad_proportion=0.02234, over 4218.98 utterances.], batch size: 15
2023-07-25 10:02:44,844 INFO [train.py:554] (0/2) Epoch 15, batch 8500, loss[ctc_loss=0.2764, att_loss=0.221, loss=0.2376, over 9457.00 frames. utt_duration=295.5 frames, utt_pad_proportion=0.02786, over 32.00 utterances.], tot_loss[ctc_loss=0.2542, att_loss=0.2102, loss=0.2234, over 1894295.07 frames. utt_duration=459.9 frames, utt_pad_proportion=0.02655, over 4118.75 utterances.], batch size: 32
2023-07-25 10:02:44,851 INFO [train.py:554] (1/2) Epoch 15, batch 8500, loss[ctc_loss=0.2671, att_loss=0.2237, loss=0.2367, over 9528.00 frames. utt_duration=381.1 frames, utt_pad_proportion=0.01264, over 25.00 utterances.], tot_loss[ctc_loss=0.2567, att_loss=0.2126, loss=0.2258, over 1899652.45 frames. utt_duration=447.5 frames, utt_pad_proportion=0.025, over 4244.59 utterances.], batch size: 25
2023-07-25 10:02:59,319 INFO [train.py:554] (0/2) Epoch 15, batch 8550, loss[ctc_loss=0.2712, att_loss=0.2247, loss=0.2386, over 8979.00 frames. utt_duration=256.5 frames, utt_pad_proportion=0.07718, over 35.00 utterances.], tot_loss[ctc_loss=0.2519, att_loss=0.2087, loss=0.2217, over 1878621.38 frames. utt_duration=460.6 frames, utt_pad_proportion=0.03093, over 4078.71 utterances.], batch size: 35
2023-07-25 10:02:59,326 INFO [train.py:554] (1/2) Epoch 15, batch 8550, loss[ctc_loss=0.1677, att_loss=0.1511, loss=0.1561, over 7831.00 frames. utt_duration=978.9 frames, utt_pad_proportion=0.2099, over 8.00 utterances.], tot_loss[ctc_loss=0.2537, att_loss=0.2105, loss=0.2234, over 1886784.67 frames. utt_duration=454.6 frames, utt_pad_proportion=0.03054, over 4150.05 utterances.], batch size: 8
2023-07-25 10:03:13,667 INFO [train.py:554] (0/2) Epoch 15, batch 8600, loss[ctc_loss=0.2772, att_loss=0.2206, loss=0.2376, over 8894.00 frames. utt_duration=254.1 frames, utt_pad_proportion=0.0892, over 35.00 utterances.], tot_loss[ctc_loss=0.2513, att_loss=0.208, loss=0.221, over 1864720.14 frames. utt_duration=455.3 frames, utt_pad_proportion=0.03887, over 4095.60 utterances.], batch size: 35
2023-07-25 10:03:13,677 INFO [train.py:554] (1/2) Epoch 15, batch 8600, loss[ctc_loss=0.269, att_loss=0.226, loss=0.2389, over 8736.00 frames. utt_duration=249.6 frames, utt_pad_proportion=0.1054, over 35.00 utterances.], tot_loss[ctc_loss=0.2513, att_loss=0.2088, loss=0.2216, over 1869457.40 frames. utt_duration=453.7 frames, utt_pad_proportion=0.03818, over 4120.70 utterances.], batch size: 35

@GabrielHaoHao If you are using the conformer_cte recipe you shouldn't be, as there are much better recipes now. Are you sure you didn't make any changes to the scripts?

Yes, I'm using the conformer_ctc recipe. I just used a linear layer instead of the original two convolutional layers (4x downsampling), the rest of the model is the structure of the conformer


Enhancement - Charges

Is your feature request related to a problem? Please describe.
Enhance Features, fix issues, and improve performance
Todo's

[x] #454
[x] #455
[x] #456
[x] #457
[x] #458
[x] #459
[x] #460
[x] #461
[x] #629
[x] #630
[x] #631
[x] #888
[x] #889
[x] #1028
[x] #1029

Closing this issue since all tasks has been completed


Twitch removed http://tmi.twitch.tv/group/user/channel/chatters endpoint

migrated to GQL in #91f7a


Possible bug in simple index test suite

While working on some related tests for my proof of concept solution to #726, I came across this test
It appears that we're trying to test different combinations of file types and version formats, but not all version formats are being actually used.
https://github.com/pypa/warehouse/blob/fea98d9c054ebae63bd5d1fc478428d2efde6670/tests/unit/legacy/api/test_simple.py#L144-L158
If this is not intended, I'd be happy to submit a small fix :)

I think that test could probably use some improvement, it's a bit old at this point and could benefit from parametrization, at the very lease. 🙂


Revert "MediaStream added"

Reverts webrtc-uwp/webrtc-windows#64

For context, it seems that the previous change broke the branch (added references to wrappers without generating them)


add configuration before PeerSession activation

Codecov Report

Merging #95 (72d7c34) into prep-release (8cc9d01) will decrease coverage by 0.06%.
Report is 5 commits behind head on prep-release.
The diff coverage is 5.88%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
@@               Coverage Diff                @@
##           prep-release      #95      +/-   ##
================================================
- Coverage         23.64%   23.59%   -0.06%
================================================
  Files                84       84
  Lines              4791     4807      +16
================================================
+ Hits               1133     1134       +1
- Misses             3658     3673      +15

Files
Coverage Δ

src/django_peerctl/models/peerctl.py
42.67% <100.00%> (+0.04%)
:arrow_up:

src/django_peerctl/rest/serializers/peerctl.py
0.00% <0.00%> (ø)

src/django_peerctl/rest/views/peerctl.py
0.00% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Added new tutorial to Image Classification zoo

Thank you for adding this tutorial to monk libraries. Your contribution is greatly appreciated


Consider canary testing for popular projects

Consider some sort of automated canary testing for a set of projects that make use of rewrite-clj.
Examples that come to mind: cljfmt, zprint, carve, clojure-lsp, antq.
The idea would be to run the target project's test suit (or a reasonable portion thereof if the suite is very long running) against rewrite-clj master.

Babashka tests that libraries continue to run under it, see tests for libraries.
We might take inspiration from its techniques.


Generic interface for associating package releases with vulnerabilities

What's the problem this feature will solve?
PyPI currently does not associate any vulnerability information for projects hosted on it.
It would be a huge security benefit for users of PyPI if there was a way project releases can be associated with vulnerabilities.
Describe the solution you'd like
Implement an interface that can be called by vulnerability sources to POST vulnerability information for a package to PyPI. Requests can be signed in a similar way to how the existing GitHub token scanning integration does it.
The details provided would would be fairly minimal and just include:

Package name.
Package versions affected.
Advisory ID.
Link to advisory details.

This information can then be displayed in the PyPI UI and returned in the PyPI API. The pip install command can also make use of this information to notify users of vulnerable packages they’re installing.
This will be built in a generic way that can support other vulnerability sources as well.
Additional context
Google (where I work) is working on an open source vulnerability database for open source packages. In particular we’re working on building a community owned vulnerability database for Python packages. This will be ready soon and we’d like to make this information more readily accessible.
We’d be able to contribute the necessary patches here for this web hook as well.

Thanks for bringing up this issue Oliver. I strongly believe that a more visible security posture will significantly contribute to security awareness for both developers and maintainers.
I'm a developer advocate at Snyk, which works in the open source security space, and have my own history and experience working within the Node.js Foundation's ecosystem security working group so I'm well excited about integrating security into PyPI and pip install.
As a TL;DR, and with respect to Google's new vulnerability database, I'd like to propose and volunteer and make available Snyk's vulnerability database to the Python foundation, to power the security advisories of both PyPI and pip, and I'm confident we can sponsor any work involved for doing that.
As for the "Why" -

The vulnerability database is curated and triaged by the team, which also actively employes security research and works within communities, and academic research groups to uncover, sometimes hundreds of vulnerabilities in bulk. Every report, even an established one from NVD with a CVE assigned to it, still gets triaged by the team for potential false positives. By far, I'd say one of Snyk's unique edges is the security intelligence of the database feed itself. I can give dozens of examples from the Node.js ecosystem of vulnerabilities that Snyk found first and even until today aren't being reported on by npm audit for example.
More than just the database - As an extension to the above, making a database available is one thing, but there's a long term relationship that would benefit the Python community in terms of discussing false positives, duplication and responsible disclosing of security issues that Snyk has a dedicated team for and have been actively doing that within other communities. We're happy to strengthen that relationship and help in triaging issues, and consulting maintainers about security fixes where we can be of help.

We greatly appreciate working within communities and projects to create that security awareness and my DevRel team at Snyk have done some great projects on this, to name a few:

WebPageTest now shows a security score when you scan a website to track the JS libs and vulnerabilities. We worked fully in the open by contributing a pull request to the open source project repository and contributed Snyk's frontend database snapshot and keeping it up to date.
Google Chrome's Lighthouse project has a best practice for using libraries without any security vulnerabilities - that too is a database that s contributed and powered by Snyk.
We've worked with several JS ecosystem projects like cdnjs, yarnpkg, jsdelivr and others to build an integration in which they display a security badge on each package page in the UI (similar to what we'd see interesting with PyPI)
Lastly, but perhaps one of the most fun projects we're building now is the Advisor which provides developers with a package health score about popularity, maintenance, security, and community, to help them assess an open source package. And of course we support Python! Here's an example: https://snyk.io/advisor/python/django

I'm excited to further discuss this and see a built-in security integration in Python related tooling as discussed here.

@lirantal Thanks for responding!

As a TL;DR, and with respect to Google's new vulnerability database, I'd like to propose and volunteer and make available Snyk's vulnerability database to the Python foundation, to power the security advisories of both PyPI and pip, and I'm confident we can sponsor any work involved for doing that.

This is great to hear. To add some additional clarity, there's three separate things here that I want to tease apart:

a Python-specific community-maintained package vulnerability database (proposal here)
the broader open-source vulnerability database that is working towards aggregating multiple public and community vulnerability databases, like the database in #1 (this is https://osv.dev/)
This feature request, which is for providing the generic interface for a service like #2 to integrate with PyPI

It sounds like Snyk's database could easily be a part of any one of these:

Snyk could contribute it's database to the community-maintained database
Snyk's database could be one of the databases that OSV aggregates
Snyk could directly integrate with PyPI's generic interface

Would love to hear about which point in that pipeline you think would be best for Snyk to be involved!

You've outlined that well Dustin, all are indeed viable options.
Snyk would be happy to volunteer its database to the Python community but I think the wording and semantics matter, as in, I don't think there's an actual one-time "contribution", but rather the Snyk team is actively committed to surfacing, traiging and collaborating with maintainers about security vulnerabilities, and so volunteering in terms of creating some sort of integration for the database feed, is more appropriate. That's semantics, and also details, but I thought it's worth mentioning.
To add to that, I think that while the database's quality, quantity and first-to-know properties are super important, the tooling also matters in order to leverage it. So, whatever database integration decided here, making sure that it is integrated natively in Python's developer tooling is going to make the real change we'd like to see in the world, in terms of awareness. I'll reiterate my JS example here with npm install to install dependencies, resulting with a vulnerability report, which is a good example of that.
I'd be happy to drill down into the details of working with the Snyk vulnerabilities database and we to start poc-ing that as needed.

Would be great to get the conversation going :-)

As part of Red Hat's project Thoth we use PyPA's advisory-db in a server side/cloud Python resolver. A demonstration of this feature is available in this recording. The resolution engine has a pluggable interface (see also adviser repo) so we can eventually plug what @lirantal and Snyk have - let's see how we can combine efforts in this area. We are releasing the resolver/recommendation engine publicly.
@di could this be considerred as a topic for the planned SIG efforts (based on the call we had earlier)?

@fridex that looks awesome!
I'd also like to follow up on how we might be able to move forward with getting Snyk (and others) to contribute its database and how that might look like. Out of the options that @di outlined in https://github.com/pypa/warehouse/issues/9407#issuecomment-844496709, I think these two options would be ideal:

Contributing vulnerabilities directly to https://github.com/pypa/advisory-db
A separate export of Snyk's python DB in the same format so that the existing OSV integration can help report Snyk vulnerabilities without much additional changes.

My team at Google is trying to get vulnerability databases to agree on an export/interchange format, so that sharing and aggregating across databases is easy and we reduce potential duplicate triage work. This is the same format that the current python advisory-db uses. See our blog post released today for details!
@lirantal WDYT? Would it make sense to have a call or start an email thread with interested folks on your side to discuss this? I think the collaboration details here could span beyond Python :)

This seems closely related to #798; is it a duplicate?


Usage of the 'filterlabels' option.

Hello,
I would like to "filter by label". Sadly the usage or syntax is unclear to me. Can you perhaps provide a few examples?
I can't filter on standard EKS node labels like e.g. failure-domain.beta.kubernetes.io/zone=eu-central-1b or similar.
For example, this does not work:
KubeNodeUsage -filterlabels="failure-domain.beta.kubernetes.io/zone=eu-central-1b"

Working with: Version: 3.0.1
Thanks,
Phil.

OK, just noticed this: https://github.com/AKSarav/Kube-Node-Usage/pull/15

I am working on the filter labels right now - Please check out the new Label as a Column feature added on v3.0.2

This is released with v3.0.2 - closing the issue


[Error] NotFoundError: Failed to execute 'removeChild' on 'Node': The node to be removed is not a child of this node.

Page
https://registry.platformio.org/libraries/fatbeard/vbus-arduino/installation
Stack
at h (https://cdn.registry.platformio.org/static/js/main.2eef75c9.js:2:436069)
    at Suspense
    at div
    at div
    at Q (https://cdn.registry.platformio.org/static/js/main.2eef75c9.js:2:150382)
    at div
    at main
    at Y (https://cdn.registry.platformio.org/static/js/main.2eef75c9.js:2:153156)
    at u (https://cdn.registry.platformio.org/static/js/main.2eef75c9.js:2:435443)
    at P (https://cdn.registry.platformio.org/static/js/main.2eef75c9.js:2:594134)
    at h (https://cdn.registry.platformio.org/static/js/main.2eef75c9.js:2:436069)
    at d (https://cdn.registry.platformio.org/static/js/main.2eef75c9.js:2:435499)
    at u (https://cdn.registry.platformio.org/static/js/main.2eef75c9.js:2:433393)
    at https://cdn.registry.platformio.org/static/js/main.2eef75c9.js:2:485124
    at L (https://cdn.registry.platformio.org/static/js/main.2eef75c9.js:2:593000)
    at I (https://cdn.registry.platformio.org/static/js/main.2eef75c9.js:2:594328)

What is your browser name and version?

Please reopen if you still need help


Loosen all safety restrictions of VertexAI so that response won't be blocked

Loosen all safety restrictions of VertexAI so that response won't be blocked
When an LLM API error happens, use the default value ('' for Gemini). This helps prevent non-terminating experiments.
Rename class name Gemini1_5 to Gemini1D5 to avoid linter complaints.

Only 7 benchmark trials failed to terminate in https://llm-exp.oss-fuzz.com/Result-reports/scheduled/2024-05-22-manual-all/, all likely due to this.

/gcbrun exp -n dg -b blocked

The blocked benchmarks work now.

https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2024-05-27-293-dg-blocked/index.html

/gcbrun skip


Feature request: Add preview URL for the package that is built using pika CI/CD

I tried Pika CI/CD in one of my repo and the experience was good. But the URL for package preview is not displayed on the CLI checks. It would be so great if the Preview URL is displayed. Currently, it says a text saying Package preview is ready
It took a couple of minutes to compare with the docs and frame the URL, for the package that is built which is https://github.pika.dev/JayaKrishnaNamburu/ngx-avatar/pr/8. Which comes by changing a few in the domain and the route param.
I am not sure about the Pika CI/CD repo, so posted it here 😅
Thank You ❤

Did you try clicking the "details" link next to Package preview is ready!? It should have a summary of all Preview Links along with other useful debugging info. Example: https://github.com/JayaKrishnaNamburu/ngx-avatar/pull/8/checks?check_run_id=187210558

Yes, ah my bad. I missed it or it took some time to render the preview result. But I think you can close this. It's good now 😄

np!

Hey @FredKSchott this will help you --> https://drive.google.com/file/d/1_TeBHYVFvqjRJHnqsRdafZTsNXgmqo7P/view?usp=sharing
I think this is why I felt the preview ur is missing

Oh wow, that's really strange. I click that same button on that same PR, and get redirected here:
https://github.com/JayaKrishnaNamburu/ngx-avatar/pull/8/checks?check_run_id=187210558

Do you have any odd settings on your account that could be causing this? Can you see this page? https://github.com/JayaKrishnaNamburu/ngx-avatar/pull/8/checks

Yeah, I think you got it right. I am using this in chrome --> https://github.com/sindresorhus/refined-github
When the extension is enabled it is redirecting to pika.dev

When it is disabled, it works fine

That is so strange. Can you confirm that you're on the latest version of refined-github? And if so, do you mind filing this exact issue with them? Happy to make any reasonable changes needed on their end, but I can't tell if this is a bug or expected behavior on their end.


Modify Azure gem to use new json product types.

this is done in the jellyfish-fog gem. closing.


Allow undefined where null is allowed

Describe the feature request
Is there a reason why undefined isn't supported? My component has optional titleStyle, so it can be undefined.
Now I have to convert undefined to false titleStyle != null
<Text
  style={[
    styles.title,
    styles[variant],
    titleStyle != null && titleStyle,
    hovered && styles.titleHover,
    pressed && styles.titlePressed,
  ]}
>
  {title}
</Text>

undefined should be supported when used there. I'll look into it.

@nmn Maybe it's because of exactOptionalPropertyTypes
https://github.com/Effect-TS/effect/tree/main/packages/schema#requirements

Thanks for catching this. Put up a PR with the fix.


"Internal server error" error="[plugin.downstreamError]

Hello,
I have:
zabbix 6.4.1
grafana 9.4.7 (commit: 4add91f03d, branch: HEAD)
I try to add the dashboard:

When I try it I have the next error logs in /var/log/grafana/grafana.log:
logger=live t=2023-04-20T04:31:29.609969219Z level=info msg="Initialized channel handler" channel=grafana/dashboard/uid/dND7S1PVz address=grafana/dashboard/uid/dND7S1PVz
logger=context userId=1 orgId=1 uname=admin t=2023-04-20T04:33:48.721072132Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Failed to query data: rpc error: code = Canceled desc = context canceled" remote_addr=<IP_ADDRESS> traceID=
logger=context userId=1 orgId=1 uname=admin t=2023-04-20T04:33:48.721734454Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=<IP_ADDRESS> time_ms=66 duration=66.41905ms size=116 referer="http://<IP_ADDRESS>:3000/d/dND7S1PVz/home?editPanel=2&orgId=1" handler=/api/ds/query

@alexanderzobnin Im facing same issue. When I have host with 300 items, im getting this error as well.
I tried upgrading zabbix plugin, upgraded zabbix to 6.4.6, upgrading grafana to latest version.  Also I tried to increase memory in PHP, I upgraded php to 8.2, nothing helped.

Hello anyone could solve this? Im getting the same issue trying to get hosts with a big amount of items (around 500) and time ranges over 3h after updating grafana to 9.3.2, zabbix to 6.0.10 and the plugin version 4.2.10.

Hello anyone could solve this? Im getting the same issue trying to get hosts with a big amount of items (around 500) and time ranges over 3h after updating grafana to 9.3.2, zabbix to 6.0.10 and the plugin version 4.2.10.

Hi Im using 9.3.2, zabbix to 6.0.10 and the plugin version 4.2.10 in Kubernetes.
After some troubleshooting I saw that the error is beceause the process of the plugin is getting killed and restarted. It is happening with hosts with 4k items which are split by 3 diferent tags.
The plugin holds correctly the 1 hour interval but more than that the plugin process starts to restart. We suspect it can be the amount of datapoints since when we tried in local environment for debugging and the datapoints for longer interval were about 200k.
I,ve tried the last version of the plugin as well but faced the same problem. There is in the roadmap something to fix this?

I am having the same issue. Zabbix version 6.4.12, Grafana v10.4.1 and plugin 4.4.6. On panels with host groups with more than 300 hosts, I get 500 Internal Server Error. Could this be Zabbix's fault? It looks like the API is giving the error


Package request: python-jwt (some confusing/incorrect package naming in nixpkgs)

PLEASE NOTE: The package in nixpkgs called python-jwt is not actually the python-jwt module on pypi. The aforementioned package is called jwt on pypi. To confuse things even further, there is also a pyjwt on pypi.
Project description
Module for generating and verifying JSON Web Tokens.
Metadata

homepage URL: https://pypi.org/project/python-jwt/
source URL: https://github.com/davedoesdev/python-jwt
license: mit
platforms: unix, linux, darwin

notifying maintainer: @thornycrackers


Update cert-manager-presubmits.yaml

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: abhay-krishna
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [abhay-krishna]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@abhay-krishna: Updated the job-config configmap in namespace default at cluster default using the following files:

key cert-manager-presubmits.yaml using file jobs/aws/eks-anywhere-build-tooling/cert-manager-presubmits.yaml
key presets.yaml using file jobs/presets.yaml

In response to this:

Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.


Bump candle version 0.3.2

Candle 0.3.2 just got out, and they seem to have stabilized Metal support, along with implementing slice_assign.

@louisfd can you rebase this PR against latest main ?


Shift-uppercase won't work if switch language on terminal page

Windows Terminal version (or Windows build number)
1.6.10571.0
Other Software
No response
Steps to reproduce
1 -> Open Terminal in different  language ( mine was Persian )
2 -> Switch language to English
3 -> Hold " Shift " to input upper case letter
Nothing happens
Expected Behavior
No response
Actual Behavior
Shift -upper case should be working in every situations

No I was on my local . I am using powershell

I have the same problem in PowerShell-type tabs. And this problem is persistent between windows installs. My locales are Russian and English.

Okay, so this is bizarre. We've seen this intermittently before, over in #7087 and #8450. From that thread: https://github.com/microsoft/terminal/issues/7087#issuecomment-676782357

Found a fix. If you have the same problem as me, then updating PSReadLine to the latest preview version can help you. Check out this PowerShell/PowerShell#10794 and this PowerShell/PSReadLine for more information.

@hamid7621 Could you try that, and see if that resolves the issue?

Thanks @zadjii-msft , it was helpful and issue resolved by  upgrading the module .


python pptx get letter size

I have a presentaion. It contains a slide with an Text Frame.
I import the presentaion and walk through the shapes of the slide.
I want to get the lettersize of the text.
How is that possible?

That would be the size attribute of the font object for each run (of each paragraph). From the way I've written this you'll see there might be multiple font sizes. That doesn't negate what you're trying to do.
One shortcut might be - for the single font size case:
font_size = text_frame.paragraphs[0].font.size

However, that's the font size, rather than the width of any piece of text. You can't get the latter.

I want to replace a text in a text frame.
This is a part of my code:
for string_to_replace in values:
if string_to_replace in shape.text:
text = shape.text
text_neu = text.replace(string_to_replace, values[string_to_replace])
text_frame = shape.text_frame
text_frame.clear()
p = text_frame.paragraphs[0]
p.text = text_neu
It replaces the text correctly, but the size of the letters is different after the replacement.
I tried to get the size of the textframe and put it on the textframe after replacement.
If I try to read with font.size it gives me None back...

My GUESS is it's the text_frame.clear() that's resetting to the default for this slide master. And that that is different from what the original paragraphs' runs were using.

Do you have a suggestion for my problem?
I have a presentation that contains a slide with a textframe. The text is: "Tested variable_variant".
The target is to get the text of the shape.
And then put in a string for the variable "variable_variant".
The position of the shape and the size of the text should be the same.

Is the term you are replacing always contained within a single run? If not you're going to break whatever (perhaps invisible) formatting variations.
(The general "search and replace" question comes up quite often, especially using the python-pptx tag on Stack Overflow. In fact this question might get better discussion there.)

Try it.
https://github.com/scanny/python-pptx/issues/884#issuecomment-1578676563
Or
def set_font(run: _Run, font: Font) -> _Run:
    run._r.insert(0, font._rPr)
    return run


[pornhub] fixed parsing categories

Please follow the guide below
Before submitting a pull request make sure you have:

[x] At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
[x] Searched the bugtracker for similar pull requests
[x] Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

[ ] I am the original author of this code and I am willing to release it under Unlicense
[x] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[ ] Bug fix
[x] Improvement
[ ] New extractor
[ ] New feature

fixed how extractor parses categories
partly fixes #13720

As already said: do not mix unrelated changes in single PR.


Proposal for EPC representation for the text-mined association KG

For each text-mined Biolink association, we would like to provide relevant EPC data including:

The sentence from which the assertion was mined
An identifier for the document that contains the sentence
The character offsets (relative to the sentence) for the text mentions of the subject and object of the assertion
A confidence score for this specific text-mined assertion (right now this is the score reported by the classifier that identified the relation)

This goal of this issue is to discuss how to represent the EPC data using the Attribute object that is defined in the TRAPI specification.
An initial proposal for Attribute representation is available in this document.
The proposal in this issue builds off of the original, and specifically addresses a need to group EPC into individual packets that contain the sentence and other relevant information so that multiple EPC packets can be associated with a single assertion.
Data for a text-mined assertion
graph": {
          "nodes": [
            {
              "id": "n0",
              "type": "biolink:ChemicalSubstance",
              "curie": "CHEBI:3215"      # bupivacaine
            },
            {
              "id": "n1",
              "type": "biolink:GeneOrGeneProduct",
              "curie": "PR:000031567"    # LRRC3B
            }
          ],
          "edges": [
            {
              "id": "e0",
              "source_id": "n0",
              "target_id": "n1",
              "type": "biolink:negatively_regulates_entity_to_entity"
            }
          ]
     }
# This assertion is supported by two sentences in the literature
      {
        'publication': 'PMID:29085514',
        'score': '0.99956816',
        'sentence': 'The administration of 50 µg/ml bupivacaine promoted maximum breast cancer cell invasion, and suppressed LRRC3B mRNA expression in cells.',
        'subject_spans': 'start: 31, end: 42',
        'object_spans': 'start: 104, end: 110',
        'provided_by': 'TMProvider'
      }

      {
        'publication': 'PMID:12345678',
        'score': '0.876',
        'sentence': 'This is a second sentence indicating that bupivacaine negatively regulates LRRC3B.',
        'subject_spans': 'start: 42, end: 53',
        'object_spans': 'start: 75, end: 81',
        'provided_by': 'TMProvider'
      }

Proposed Attribute representation
The proposed Attribute representation models this assertion as a single edge between bupivacaine and LRRC3B with two accompanying Attributes representing the EPC data. Nested Attributes are used to allow each packet of sentence information to be self-contained. Also demonstrated are attributes representing a confidence score for the concept recognition of each node (concept), and an aggregate confidence score computed for each edge.
nodes:
  - id: CHEBI:3215
     category: biolink:ChemicalSubstance
     name: "bupivacaine"
     attributes:
        - attribute_type_id: SEPIO:0000168  # confidence_score
           attribute_from_source: "has confidence score"
           value: 0.7578
           value_type_id: biolink:ConfidenceLevel
           value_type_from_source: "confidence score"
           value_source: TMProvider

  - id: PR:000031567
     category: biolink:GeneOrGeneProduct
     name: "LRRC3B"
     attributes:
        - attribute_type_id: SEPIO:0000168  # confidence_score
           attribute_from_source: "has confidence score"
           value: 0.5467
           value_type_id: biolink:ConfidenceLevel
           value_type_from_source: "confidence score"
           value_source: TMProvider

edges:
  - id: tmkp.Association001
    category: biolink:ChemicalToGeneAssociation
    subject: CHEBI:3215          # bupivacaine
    predicate: biolink:negatively_regulates_entity_to_entity
    object: PR:000031567       # LRRC3B
    attributes:

    - attribute_type_id: SEPIO:0000438  # has_supporting_evidence_from_source
       attribute_from_source:  "source publication"    # what the source might have called the relationship
       value: PMID:29085514
       value_type_id: biolink:Publication          # here a biolink term is used to type the value.
       value_type_from_source: "PMID"
       value_source: TMProvider
       attributes:
          - attribute_type_id: SIO:000028  # has part
             value: "The administration of 50 µg/ml bupivacaine promoted maximum breast cancer cell invasion, and suppressed LRRC3B mRNA expression in cells."
             value_type_id: EDAM:data_3671     # text, or SIO:000113 'sentence'
             value_type_from_source:  sentence text
             attributes:
                 - attribute_type_id: SIO:000028  # has part
                    value: '31|42'
                    value_type_id: SIO:001056 # character position
                    value_type_from_source:  subject span
                 - attribute_type_id: SIO:000028  # has part
                    value: '104|110'
                    value_type_id: SIO:001056 # character position
                    value_type_from_source:  object span
                 - attribute_type_id: SEPIO:0000440  # has_supporting_evidence
                    value: 0.99956816
                    value_type_id: EDAM:data_1772     # score
                    value_type_from_source:  sentence confidence score
                    value_source: TMProvider BERT model v0.1

    - attribute_type_id: SEPIO:0000438  # has_supporting_evidence_from_source
       attribute_from_source:  "source publication"    # what the source might have called the relationship
       value: PMID:12345678
       value_type_id: biolink:Publication          # here a biolink term is used to type the value.
       value_type_from_source: "PMID"
       value_source: TMProvider
       attributes:
          - attribute_type_id: SIO:000028  # has part
             value: "This is a second sentence indicating that bupivacaine negatively regulates LRRC3B.'"
             value_type_id: EDAM:data_3671     # text, or SIO:000113 'sentence'
             value_type_from_source:  sentence text
             attributes:
                 - attribute_type_id: SIO:000028  # has part
                    value: '42|53'
                    value_type_id: SIO:001056 # character position
                    value_type_from_source:  subject span
                 - attribute_type_id: SIO:000028  # has part
                    value: '75|81'
                    value_type_id: SIO:001056 # character position
                    value_type_from_source:  object span
                 - attribute_type_id: SEPIO:0000440  # has_supporting_evidence
                    value: 0.876
                    value_type_id: EDAM:data_1772     # score
                    value_type_from_source:  sentence confidence score
                    value_source: TMProvider BERT model v0.1

    - attribute_type_id: SEPIO:0000168  # confidence_score
       attribute_from_source: "has aggregate confidence score"
       value: 0.64711234
       value_type_id: biolink:ConfidenceLevel
       value_type_from_source: "aggregate confidence score"
       value_source: TMProvider

For comparison purposes, shown below is an alternative approach that uses no nesting of Attributes, and instead makes use of arrays to specify attribute values. For a given EPC packet, the sentence, score, subject & object spans, and PMID are inherently connected based on the array index used to store their values.
Note: This is the current output format used by the Service Provider to serve up the Text Mining Provider text-mined Biolink association KG.
edges:
  - id: 9445e98f72ada21aa572559e303e4d5ac414650f
    predicate: biolink:negatively_regulates,
    subject: CHEBI:3215          # bupivacaine
    object: PR:000031567       # LRRC3B
    attributes:
      - type: biolink:provided_by
        name: provided_by
        value: Text Mining KP
      - type: bts:api
        name: api
        value: Text Mining Targeted Association API
      - type: bts:score
        name: score
        value:
          - 0.99956816
          - 0.876
      - type: bts:sentence
        name: sentence
        value:
          - "The administration of 50 µg/ml bupivacaine promoted maximum breast cancer cell invasion, and suppressed LRRC3B mRNA expression in cells."
          - "This is a second sentence indicating that bupivacaine negatively regulates LRRC3B."
      - type: bts:subject_spans
        name: subject_spans
        value:
          - "31|42"
          - "42|53"
      - type: bts:object_spans
        name: object_spans
        value:
          - "104|110"
          - "75|81"
      - type: bts:publications
        name: publications
        value:
          - PMID:29085514
          - PMID:12345678

For comparison purposes, shown below is an alternative approach that uses no nesting of Attributes, and instead makes use of arrays to specify attribute values. For a given EPC packet, the sentence, score, subject & object spans, and PMID are inherently connected based on the array index used to store their values.
Note: This is the current output format used by the Service Provider to serve up the Text Mining Provider text-mined Biolink association KG.
edges:
  - id: 9445e98f72ada21aa572559e303e4d5ac414650f
    predicate: biolink:negatively_regulates,
    subject: CHEBI:3215          # bupivacaine
    object: PR:000031567       # LRRC3B
    attributes:
      - type: biolink:provided_by
        name: provided_by
        value: Text Mining KP
      - type: bts:api
        name: api
        value: Text Mining Targeted Association API
      - type: bts:score
        name: score
        value:
          - 0.99956816
          - 0.876
      - type: bts:sentence
        name: sentence
        value:
          - "The administration of 50 µg/ml bupivacaine promoted maximum breast cancer cell invasion, and suppressed LRRC3B mRNA expression in cells."
          - "This is a second sentence indicating that bupivacaine negatively regulates LRRC3B."
      - type: bts:subject_spans
        name: subject_spans
        value:
          - "31|42"
          - "42|53"
      - type: bts:object_spans
        name: object_spans
        value:
          - "104|110"
          - "75|81"
      - type: bts:publications
        name: publications
        value:
          - PMID:29085514
          - PMID:12345678


PHP 7.2 + QA Tools (follow up #1, #7, #8, #12)

Resolves #18
Closes #1
Closes #7
Closes #8
Closes #12

Thanks, @webimpress.


Small updates to the source documentation.

Hi everyone,
This PR contains a few minor updates to the in-source documentation.

@swift-ci smoke test


Broker Result adapter changes to centralize generating result bundle based on SDK

Added new IBrokerResultAdapter and corresponding AdalBrokerResultAdapter and MsalBrokerResultAdapter to consolidate result bundle based on the SDK
Added IntuneAppProtectionPolicyRequiredException to support intune's PROTECTION_POLICY_REQUIRED sub error.
Removed context from TokenCommand as it's being staticly held which can potentially lead to garbage collection issues.
Replaced throwIfNetworkNotAvailable to more comprehensive HttpWebRequest.throwIfNetworkNotAvailable.
Added logic to add iss claim as identity provider for V1 token if idp is not available

Thanks @iambmelt ,Agree with unit tests I will prioritize adding unit tests for these later this week. I would like the PR to get in for our next build.
@heidijinxujia , I have a question regarding the naming change you requested, let me know what you think

@kreedula  Sure, I will ping u after finishing the reviewing.

@heidijinxujia , replied to your comments, let me know what you think


[kube-up] [scale-out] daemonset controller should not create pod for TP master

What happened:
in kube-up scale-out 1 TP x 1 RP x 1 worker env, when a daemonset is created, cluster has pod created for the TP master, besides the expected pods for RP master and worker, as illustrated:
$ ./cluster/kubectl.sh get ds fluentd-gcp-v3.2.0 --kubeconfig=./cluster/kubeconfig.tp-1 -n kube-system
NAME                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
fluentd-gcp-v3.2.0   2         2         1       2            1           beta.kubernetes.io/os=linux   4h42m

$ ./cluster/kubectl.sh get pod --kubeconfig=./cluster/kubeconfig.tp-1 -n kube-system -o wide | grep fluentd-gcp-v3.2.0
fluentd-gcp-v3.2.0-bjfrj   ...    sonyadaemonset-011122-rp-1-master
fluentd-gcp-v3.2.0-fp6rx    ...   sonyadaemonset-011122-rp-1-minion-group-w9ds
fluentd-gcp-v3.2.0-tmq6m    ...   sonyadaemonset-011122-tp-1-master

What you expected to happen:
only pods for RP master and RP worker are created
How to reproduce it (as minimally and precisely as possible):
run kube-up script (of poc-2022-01-30) to start 1x1x1 scale-out cluster (besides the regular env vars for successful kube-up run)
export SCALEOUT_CLUSTER=true SCALEOUT_TP_COUNT=1 SCALEOUT_RP_COUNT=1
./cluster/kube-up.sh

run kubectl command to display the named pod
Anything else we need to know?:
at TP master, in kube-controller-mnager.log, there were records of these pods' creations, including the extraneous one for TP master
I0111 17:44:39.227415       1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"fluentd-gcp-v3.2.0", UID:"8dacca57-2f6f-4e1e-ac33-4236d9dc578d", APIVersion:"apps/v1", ResourceVersion:"340", FieldPath:"", Tenant:"system"}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: fluentd-gcp-v3.2.0-tmq6m
I0111 17:51:49.106898       1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"fluentd-gcp-v3.2.0", UID:"8dacca57-2f6f-4e1e-ac33-4236d9dc578d", APIVersion:"apps/v1", ResourceVersion:"477", FieldPath:"", Tenant:"system"}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: fluentd-gcp-v3.2.0-bjfrj
I0111 17:51:49.111157       1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"fluentd-gcp-v3.2.0", UID:"8dacca57-2f6f-4e1e-ac33-4236d9dc578d", APIVersion:"apps/v1", ResourceVersion:"477", FieldPath:"", Tenant:"system"}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: fluentd-gcp-v3.2.0-fp6rx

Environment:

Arktos version (use kubectl version):
Cloud provider or hardware configuration: GCP/kube-up
OS (e.g: cat /etc/os-release): Container-Optimized OS from Google (TP master)
Kernel (e.g. uname -a): 5.10.68+
Install tools: kube-up scale-out, COS os targeted
Network plugin and version (if this is a network-related bug):
Others:

Assign to Hongwei as owner of Daemonset.

Can this be resolved by marking TP master as not schedulable?

this is caused by the way TP KCM started twice during kube-up.

discussed in team meeting. this bug is not a blocker. fix will be deferred.

this is the code how the TP master ended up in the scheduler nodes cache, in file cmd/kube-scheduler/app/options/options.go:
// if the resource provider kubeconfig is not set, default to the local cluster
	if c.ComponentConfig.ResourceProviderKubeConfig == "" {
		klog.V(2).Infof("ResourceProvider kubeConfig is not set. default to local cluster client")
		c.NodeInformers = make(map[string]coreinformers.NodeInformer, 1)
		c.NodeInformers["tp"] = c.InformerFactory.Core().V1().Nodes()
	} else {
		kubeConfigFiles, existed := genutils.ParseKubeConfigFiles(c.ComponentConfig.ResourceProviderKubeConfig)
		// TODO: once the perf test env setup is improved so the order of TP, RP cluster is not required
		//       rewrite the IF block
		if !existed {
			klog.Warningf("ResourceProvider kubeConfig is not valid, default to local cluster kubeconfig file")
			c.NodeInformers = make(map[string]coreinformers.NodeInformer, 1)
			c.NodeInformers["rp0"] = c.InformerFactory.Core().V1().Nodes()
		} else {
			c.ResourceProviderClients = make(map[string]clientset.Interface, len(kubeConfigFiles))
			c.NodeInformers = make(map[string]coreinformers.NodeInformer, len(kubeConfigFiles))
			for i, kubeConfigFile := range kubeConfigFiles {
				rpId := "rp" + strconv.Itoa(i)
				c.ResourceProviderClients[rpId], err = clientutil.CreateClientFromKubeconfigFile(kubeConfigFile, "kube-scheduler")
				if err != nil {
					klog.Errorf("failed to create resource provider rest client, error: %v", err)
					return nil, err
				}

				resourceInformerFactory := informers.NewSharedInformerFactory(c.ResourceProviderClients[rpId], 0)
				c.NodeInformers[rpId] = resourceInformerFactory.Core().V1().Nodes()
				klog.V(2).Infof("Created the node informer %p from resourceProvider kubeConfig %d %s",
					c.NodeInformers[rpId].Informer(), i, kubeConfigFile)
			}
		}
	}


BlockType.getProperty() not working in Pre-Init

I am currently running

SpongeVanilla version: spongevanilla-1.12.2-7.1.0-BETA-93
Java version: OpenJDK Version "1.8.0_171"
Operating System: Linux Mint 18.3 Sylvia (Kernel: x86_64 Linux 4.13.0-45-generic)
Plugins/Mods: Empty (Only one my experimental plugin)

Issue Description
<anyBlockType>.getProperty(<AnyPropertyClass>.class) => Empty Optional
<anyBlockType>.getDefaultState().getProperty(<AnyPropertyClass>.class) => Empty Optional
<anyBlockType>.getApplicableProperties() => Empty Collection
The problem exists at least on all builds on version 1.12.2. Perhaps the subsystem of properties does not work at all on any classes.

Just tested this in development:
Using the following listener:
    @Listener
    public void onRightClick(InteractBlockEvent.Secondary.MainHand event, @Root Player player) {
        final Collection<Property<?, ?>> applicableProperties = event.getTargetBlock().getState().getApplicableProperties();
        System.err.println(applicableProperties);
    }

And I get this in the log:
[09:49:07] [Server thread/INFO] [STDERR]: [org.spongepowered.test.DataPropertiesTest:onRightClick:51]: [InstrumentProperty(SpongeInstrumentType{id=minecraft:harp, name=Harp}), StatisticsTrackedProperty(true), BlastResistanceProperty(0.6000000238418579), SurrogateBlockProperty(false), FullBlockSelectionBoxProperty(true), MatterProperty(SOLID), PassableProperty(false), HeldItemProperty(ItemColored{Name=null}), GravityAffectedProperty(false), LightEmissionProperty(0), ReplaceableProperty(false), SolidCubeProperty(true)]
[09:49:18] [Server thread/INFO] [STDERR]: [org.spongepowered.test.DataPropertiesTest:onRightClick:51]: [InstrumentProperty(SpongeInstrumentType{id=minecraft:harp, name=Harp}), StatisticsTrackedProperty(true), BlastResistanceProperty(0.6000000238418579), SurrogateBlockProperty(false), FullBlockSelectionBoxProperty(true), MatterProperty(SOLID), PassableProperty(false), HeldItemProperty(ItemColored{Name=null}), GravityAffectedProperty(false), LightEmissionProperty(0), ReplaceableProperty(false), SolidCubeProperty(true)]

And Likewise, using the same but with getType() after getState() will present me with this:
[09:53:09] [Server thread/INFO] [STDERR]: [org.spongepowered.test.DataPropertiesTest:onRightClick:51]: [MatterProperty(SOLID), SurrogateBlockProperty(false), PassableProperty(false), BlastResistanceProperty(0.6000000238418579), ReplaceableProperty(false), InstrumentProperty(SpongeInstrumentType{id=minecraft:harp, name=Harp}), HeldItemProperty(ItemColored{Name=null}), GravityAffectedProperty(false), SolidCubeProperty(true), FullBlockSelectionBoxProperty(true), LightEmissionProperty(0), StatisticsTrackedProperty(true)]

Is there a reproducible case with just sponge forge and a very basic plugin to reproduce this issue?

Incredible! It turns out these methods do not work in the pre-initialization phase. Methods become available only in the post-initialization phase. I load messages and configuration in the pre-initialization phase. The configuration checks the values at load time (it is checked whether the blocks are solid). So it should work?

It turns out these methods do not work in the pre-initialization phase.

That's quite the detail you forgot to mention.
I don't think preInit is the right place for what you're trying to do. Look at the doc in the API for more information.

Pre-init is very not game ready, lots of things still being registered or have yet to be registered, hence your issue.
I'd advise to at earliest use postinit to do config or message loading that depends on concrete types like Items, etc. or interacting with most of the Data API.
I can provide a temporary workaround during these early phases so that the default supplied providers will function, as such, I'll leave this issue open until I do.

@gabizou How can we better communicate the game life cycles to people? This seems more a UX / docs / Read The Manual type issue.
Our current docs: https://docs.spongepowered.org/stable/en/plugin/lifecycle.html?highlight=lifecycle
Our current JD: https://github.com/SpongePowered/SpongeAPI/tree/stable-7/src/main/java/org/spongepowered/api/event/game/state

Fixed by https://github.com/SpongePowered/SpongeCommon/commit/0c2d7dc166b8ecab837cd13228a196f4076801c4


Release/v1.0.0 prerelease.33

Prerelease 33 ( 2019-12-18 )
Tag: v1.0.0-prerelease.33

5ad398 Disclosure accordion variant
3ccec6 Update pfe-cta to include broadcast variables for all variants #659

@kylebuch8 heads up, most components version bumped, idk why
https://www.npmjs.com/search?q=%40patternfly elements

@starryeyez024 Looks like it was because pfe-sass was bumped from 32 to 33 and most other packages depend on it.


Add margin above tabpanels

Adds a 1em margin above tabpanels to avoid visual collision between text/content and tabs-line.

Quick example of intended change:
From this

To this


Add reference docs for deno outdated subcommand

Wrote a quick first draft, this will also need to be updated once 2.1 is released and we update the CLI args metadata.

@nathanwhit you should now be able to merge it


Serialize Object

Hi,
I am trying to create a json string using Morpheus to pass on to my post request in retrofit.
This is the acceptable format
{   "data": {     "attributes": {       "body": "Test Note",       "author": "Harver"     },     "type": "notes"   } }
But the serializable object I create using Morpheus actually is generated like below.
{   "data": {     "attributes": {       "serialVersionUID":<PHONE_NUMBER>577512005,       "body": "Notes",       "author": "Harver"     },     "id": null,     "type": "notes"   } }
I am unable to remove the serialVersionUID & id from the string result. Can you help me ?

Hey!
I think I have not implemented a way to remove fields when serializing. I can check back on the weekend for a solution.
If you want to help: do you have an idea how it should be added to the current api?
Or do you want to try to implement it yourself?

Hi,
I am trying to get a solution for this manually. But if this is sorted I will update this thread. If you can find solution Please update me too.Thanks


Change pinned timestamp file format stored in remote store

Description

As part of https://github.com/opensearch-project/OpenSearch/pull/15180, we introduced a way to pin/unpin timestamp and stored the end state in the configured remote store.
But storing all the pinned timestamps in a file can have concurrency issues even if we create immutable files to store the info.
While debugging, reading these pinned timestamp files in remote is also not always possible if the content is encrypted.
To tackle these problems, in this PR, we change the way to store pinned timestamp data. Instead of storing all the timestamps in one file, we create separate, empty file per <timestamp, pinning entity> tuple.
We name the file as <pinning entity>__<timestamp> which makes debugging easier just by listing the files.

Check List

[x] Functionality includes testing.
[ ] API changes companion pull request created, if applicable.
[ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Changes are mostly covered in the integ tests. CodeCov is failing as it does not consider coverage from integ tests.


Custom filter sorting

Hello,
Awesome library!
I'm using this library as my default filtering, sorting and paging library.
Wouldn't be nice to have Sorting on a property that is marked as a Custom filter?
Can you help?
Thanks in advance.

Thank you for your feedback!
Custom filters are properties that are not necessarily mapped to the database, that's why they are excluded from filtering and sorting. For example I can add a custom filter called OnlyHighPriority and according to its value apply the corresponding filtering, I think I did not reflect that correctly in the examples.
Do you have an example to understand your scenario a little more?

Hi, I think there could be a new method called TryGetSortBy which will receive the same parameter as TryGetFilter and will return a SortValue object as out parameter when success. I think that covers your requirement, doesn't it? I'll take a look at it these days

Yes. That seems to be perfect!
Thanks a lot ;)


Add command and entrypoint labels to containers

Description of the Change
FIXME: please add at least one sentence explaining why changes here are being made.

Pull Request Test Coverage Report for Build 3413

44 of 44 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 26.131%

Totals

Change from base Build 3383:
0.2%

Covered Lines:
3039

Relevant Lines:
11630

💛  - Coveralls


filtering with method "contains" or "equals" Uncaught TypeError

material-react-table version
1.4.2
react & react-dom versions
17.0.1
Describe the bug and the steps to reproduce it
When i try and filter a column, and change the filter type to "contains" or "equals" then i get the following error. I think it could be because some of the data in this column is blank?
Minimal, Reproducible Example - (Optional, but Recommended)
Uncaught TypeError: Cannot read properties of undefined (reading 'toString')
    at Object.Oo [as filterFn] (filterFns.ts:25:32)
    at o (getFilteredRowModel.ts:101:59)
    at g (getFilteredRowModel.ts:96:68)
    at getFilteredRowModel.ts:90:60
    at Object._getFilteredRowModel (utils.ts?6c1d:158:16)
    at Object.getFilteredRowModel (Filters.ts:424:22)
    at Object.getPreGroupedRowModel (Grouping.ts:217:42)
    at getGroupedRowModel.ts:10:47
    at Object._getGroupedRowModel (utils.ts?6c1d:143:21)
    at Object.getGroupedRowModel (Grouping.ts:227:22)

Screenshots or Videos (Optional)
No response
Do you intend to try to help solve this bug with your own PR?
No, because I don't know how
Terms

[X] I understand that if my bug cannot be reliably reproduced in a debuggable environment, it will probably not be fixed and this issue may even be closed.

I have made sure that specific column is output on my datasource now, and i do not get the error - so it's definitely when the field is not available.

This is pretty much a problem with TanStack Table itself where it does not like null values in data. This can be solved by having an accessorFn that defaults to an empty string or other default value instead of null.


Build status updates: 3.10 stable

Build sequence now tested on Docker & Travis, with latest HHVM builds. Works with 3.10.x

I can verify this works with 3.11.0 as well @chregu any thoughts on merging?

@paullryan hi! do you see transactions with 3.11 and this patch ? All my transactions are displayed as "Root path" (in APM).

@gorbunovav New Relic may not support setting of dynamic / individual transaction names, according to what I can make of this https://discuss.newrelic.com/t/how-does-the-nr-agent-determine-a-transaction/32867/2

@absalomedia hmm... I had 3.5.1 before update and some version of this extension from the same date, and I've seen transactions as normal urls.
Right after update I've started to see everything as "Root path". There is no any additional info about transactions also, despite I was enabling function level & external services profiling.
I run standard php application (Magento store actually) and I have seo-friendly urls, with everything rewritten to index.php, as usual.

@absalomedia I've added
newrelic_name_transaction($_SERVER['REQUEST_URI']);
and now I have transactions named after uris, but there is still no profiling information at all (everything is shown as WebTransaction)

@absalomedia I've played with the old version a little. I think I was not correct - trace/profiling data are in place. What I am missing are segments in breakdown table, but it looks, that old version was not filling them too.
So it is not issue of this patch.
The only difference with the old version, which I see, is automatic transaction naming after URIs.
Otherwise I can confirm, that it works ok with HHVM 3.11.1

Does this works for 3.12?

@dieend it should. I confirmed builds on 3.10 & 3.11 without issue.


Update drupal-org extension

Description

Added command to search for Drupal projects which include themes, modules, and distributions.
Also did minor Typescript refactor as agreed upon with the creator of the extension.

Screencast

Checklist

[x] I read the extension guidelines
[x] I read the documentation about publishing
[x] I ran npm run build and tested this distribution build in Raycast
[x] I checked that files in the assets folder are used by the extension itself
[x] I checked that assets used by the README are placed outside of the metadata folder

I don't know what is wrong with the GIT history, but for some reason it includes last PR's commit messages in this PR. What was actually changed in this PR was a bit of typescript refactor and added command to search projects.

I would have preferred a separate change for the refactor but that may not be a great idea here. I like how you have generalized the command state but wonder how to extend it if future commands need more data. That's not a showstopper here though.

I see what you mean there. I decided to do it this way in order to get it done quicker, however it would probably be best-practice to have something separate. I'd suggest we just keep it as-is for now, but note it for next time. Do you agree on that @hussainweb?

Generally, I am wondering if the practice of having these multiple util.ts and types.ts is typical in TypeScript projects. I don't have a lot of experience with TypeScript. I am only raising a question out of curiosity and am open to either direction.

I mean, it's not necessarily a TypeScript related practice, but from my experience, it has always been easier to maintain projects that implemented it this way, especially with the util.ts files. If some day we want to expand the capacity of one of the commands, it will be easier to review and contribute. That being said, the types.ts files don't necessarily have to be in different files, I just think it's a matter of personal preference.

Yes, I understand and agree. I see you have merged my other suggestion and this LGTM.


Got an unexpected keyword argument error when I call tf.contrib.layers.convolution2d on Tensorflow for Windows

When I invoke tf.contrib.layers.convolution2d the tensorflow execution terminates with an error about one of the parameters used
the call is
layer_one = tf.contrib.layers.convolution2d(
    float_image_batch,
    num_output_channels=32,
    kernel_size=(5,5),
    activation_fn=tf.nn.relu,
    weight_init=tf.random_normal,
    stride=(2, 2),
    trainable=True)

I have tried to fix according new documentation using tf.random_normal_init but get always the same problem
OS: Windows 10 x64
Tensorflow 0.12.0
Related Stackoverflow question: http://stackoverflow.com/questions/41539658/tensorflow-error-when-i-try-to-use-tf-contrib-layers-convolution2d

Looking at the documentation here:
https://www.tensorflow.org/api_docs/python/contrib.layers/higher_level_ops_for_building_neural_network_layers_#separable_convolution2d

weight_init is not a keyword, it has to be weight_initializer
num_output_channels is not a keyword, maybe you mean num_outputs

Please make sure you read the documentation carefully. Python wont accept any abbreviations or any similar keywords.

same problem got an unexpected keyword argument 'weight_initializer'


[pulsar-broker] execute zk-cache result-callback in external executor

Motivation
As per discussion at #3630, executing zk-cache callback into external executor rather executing into default fork-join executor used by cache.

I think running zk-cache callback into OrderedExecutor might not be the good idea because orderedExecutor-thread calls zk-cache and if callback gets executed into the same executor then it will again cause the deadlock. So, have to figure out better solution here.
ZookeeperCacheTest-OrderedScheduler-2-0" #14 prio=5 os_prio=31 tid=0x00007ffa04274000 nid=0x5903 waiting on condition [0x0000700006c46000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000799175880> (a java.util.concurrent.CompletableFuture$Signaller)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695)
        at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
        at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
        at org.apache.pulsar.zookeeper.ZooKeeperCache.getData(ZooKeeperCache.java:288)
        at org.apache.pulsar.zookeeper.ZooKeeperDataCache.reloadCache(ZooKeeperDataCache.java:119)
        at org.apache.pulsar.zookeeper.ZooKeeperCache$1.safeRun(ZooKeeperCache.java:132)
        at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:745)