torch.aten.native_layer_norm to linalg

We are having following problems with onnx lowering into tensor.expand_shape in these models (beit-base-patch16-224-pt22k-ft22k, deit-small-distilled-patch16-224, vit-base-patch16-224). Repro instructions can be found here::
beit-base-patch16-224-pt22k-ft22k.default.pytorch.torch.mlir:336:36: error: 'tensor.expand_shape' op invalid to have a single dimension (0) expanded into multiple dynamic dims (0,1)
    %result0, %result1, %result2 = torch.aten.native_layer_norm %293, %294, %10, %11, %float9.999990e-13 : !torch.vtensor<[?,?,768],f32>, !torch.list<int>, !torch.vtensor<[768],f32>, !torch.vtensor<[768],f32>, !torch.float -> !torch.vtensor<[?,?,768],f32>, !torch.vtensor<[?,?,1],f32>, !torch.vtensor<[?,?,1],f32>
                                   ^
beit-base-patch16-224-pt22k-ft22k.default.pytorch.torch.mlir:336:36: note: see current operation: %295 = "tensor.expand_shape"(%294) <{reassociation = [[0, 1]]}> : (tensor<?xf32>) -> tensor<?x?xf32>

deit-small-distilled-patch16-224.default.pytorch.torch.mlir:352:36: error: 'tensor.expand_shape' op expected dimension 0 of collapsed type to be dynamic since one or more of the corresponding dimensions in the expanded type is dynamic
    %result0, %result1, %result2 = torch.aten.native_layer_norm %297, %298, %11, %12, %float9.999990e-13 : !torch.vtensor<[?,198,384],f32>, !torch.list<int>, !torch.vtensor<[384],f32>, !torch.vtensor<[384],f32>, !torch.float -> !torch.vtensor<[?,198,384],f32>, !torch.vtensor<[?,198,1],f32>, !torch.vtensor<[?,198,1],f32>
                                   ^
deit-small-distilled-patch16-224.default.pytorch.torch.mlir:352:36: note: see current operation: %263 = "tensor.expand_shape"(%262) <{reassociation = [[0, 1]]}> : (tensor<198xf32>) -> tensor<?x198xf32>

vit-base-patch16-224.default.pytorch.torch.mlir:316:36: error: 'tensor.expand_shape' op expected dimension 0 of collapsed type to be dynamic since one or more of the corresponding dimensions in the expanded type is dynamic
    %result0, %result1, %result2 = torch.aten.native_layer_norm %272, %273, %10, %11, %float9.999990e-13 : !torch.vtensor<[?,197,768],f32>, !torch.list<int>, !torch.vtensor<[768],f32>, !torch.vtensor<[768],f32>, !torch.float -> !torch.vtensor<[?,197,768],f32>, !torch.vtensor<[?,197,1],f32>, !torch.vtensor<[?,197,1],f32>
                                   ^
vit-base-patch16-224.default.pytorch.torch.mlir:316:36: note: see current operation: %259 = "tensor.expand_shape"(%258) <{reassociation = [[0, 1]]}> : (tensor<197xf32>) -> tensor<?x197xf32>

Reproing with
HF_TOKEN=... python run.py --torchmlirbuild /path/to/torch-mlir/build --ireebuild /path/to/iree-build --cachedir ~/.cache/huggingface --tests pytorch/models/deit-small-distilled-patch16-224 -r test-onnx --tolerance .001 .001 --mode onnx --report

Minimal repro example:
func.func @native_layer_norm(%input: !torch.vtensor<[?,198,384],f32>, %weight: !torch.vtensor<[384],f32>, %bias: !torch.vtensor<[384],f32>, %eps: !torch.float) -> (!torch.vtensor<[?,198,384],f32>, !torch.vtensor<[?,198,1],f32>, !torch.vtensor<[?,198,1],f32>) {
  %int384 = torch.constant.int 384
  %normalized_shape = torch.prim.ListConstruct %int384 : (!torch.int) -> !torch.list<int>
  %result0, %result1, %result2 = torch.aten.native_layer_norm %input, %normalized_shape, %weight, %bias, %eps : !torch.vtensor<[?,198,384],f32>, !torch.list<int>, !torch.vtensor<[384],f32>, !torch.vtensor<[384],f32>, !torch.float -> !torch.vtensor<[?,198,384],f32>, !torch.vtensor<[?,198,1],f32>, !torch.vtensor<[?,198,1],f32>
  return %result0, %result1, %result2 : !torch.vtensor<[?,198,384],f32>, !torch.vtensor<[?,198,1],f32>, !torch.vtensor<[?,198,1],f32>
}

For some reason no compile error on this.
I tried the following:
Succeedes without an issue:
/home/azureuser/iree-build/tools/iree-compile --iree-input-demote-i64-to-i32 --iree-hal-target-backends=llvm-cpu  minimal.mlir > minimal.vmfb

Repro log with:

original
stripped
minimal example

+ echo --- reproing with original model mlir ---
--- reproing with original model mlir ---
+ /home/azureuser/iree-build/tools/iree-compile --iree-input-demote-i64-to-i32 --iree-hal-target-backends=llvm-cpu deit-small-distilled-patch16-224.default.pytorch.torch.mlir -o deit-small-distilled-patch16-224.default.vmfb
/home/azureuser/iree-build/tools/iree-compile: /home/azureuser/miniconda/lib/libtinfo.so.6: no version information available (required by /home/azureuser/iree-build/lib/libIREECompiler.so)
deit-small-distilled-patch16-224.default.pytorch.torch.mlir:364:36: error: 'tensor.expand_shape' op expected dimension 0 of collapsed type to be dynamic since one or more of the corresponding dimensions in the expanded type is dynamic
    %result0, %result1, %result2 = torch.aten.native_layer_norm %309, %310, %11, %12, %float9.999990e-13 : !torch.vtensor<[?,198,384],f32>, !torch.list<int>, !torch.vtensor<[384],f32>, !torch.vtensor<[384],f32>, !torch.float -> !torch.vtensor<[?,198,384],f32>, !torch.vtensor<[?,198,1],f32>, !torch.vtensor<[?,198,1],f32>
                                   ^
deit-small-distilled-patch16-224.default.pytorch.torch.mlir:364:36: note: see current operation: %263 = "tensor.expand_shape"(%262) <{reassociation = [[0, 1]]}> : (tensor<198xf32>) -> tensor<?x198xf32>
+ echo Return code: 1
Return code: 1
+ echo --- reproing with stripped model mlir ---
--- reproing with stripped model mlir ---
+ /home/azureuser/iree-build/tools/iree-compile --iree-input-demote-i64-to-i32 --iree-hal-target-backends=llvm-cpu stripped/deit-small-distilled-patch16-224.default.pytorch.torch.stripped.mlir -o deit-small-distilled-patch16-224.default.stripped.vmfb
/home/azureuser/iree-build/tools/iree-compile: /home/azureuser/miniconda/lib/libtinfo.so.6: no version information available (required by /home/azureuser/iree-build/lib/libIREECompiler.so)
stripped/deit-small-distilled-patch16-224.default.pytorch.torch.stripped.mlir:364:36: error: 'tensor.expand_shape' op expected dimension 0 of collapsed type to be dynamic since one or more of the corresponding dimensions in the expanded type is dynamic
    %result0, %result1, %result2 = torch.aten.native_layer_norm %309, %310, %11, %12, %float9.999990e-13 : !torch.vtensor<[?,198,384],f32>, !torch.list<int>, !torch.vtensor<[384],f32>, !torch.vtensor<[384],f32>, !torch.float -> !torch.vtensor<[?,198,384],f32>, !torch.vtensor<[?,198,1],f32>, !torch.vtensor<[?,198,1],f32>
                                   ^
stripped/deit-small-distilled-patch16-224.default.pytorch.torch.stripped.mlir:364:36: note: see current operation: %73 = "tensor.expand_shape"(%72) <{reassociation = [[0, 1]]}> : (tensor<198xf32>) -> tensor<?x198xf32>
+ echo Return code: 1
Return code: 1
+ echo --- reproing with minimal example ---
--- reproing with minimal example ---
+ /home/azureuser/iree-build/tools/iree-compile --iree-input-demote-i64-to-i32 --iree-hal-target-backends=llvm-cpu minimal.mlir -o minimal.vmfb
/home/azureuser/iree-build/tools/iree-compile: /home/azureuser/miniconda/lib/libtinfo.so.6: no version information available (required by /home/azureuser/iree-build/lib/libIREECompiler.so)
error opening input file: cannot open input file 'minimal.mlir': No such file or directory
+ echo Return code: 1
Return code: 1

IR dump after failure (with failing line highlighted)
https://gist.github.com/renxida/fdcec365e89ae317fdd8545fbd4ddaf8#file-gistfile0-txt-L117
Relevant section:
  %72 = "linalg.generic"(%69, %71) <{indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>, affine_map<(d0, d1) -> (d0)>], iterator_types = [#linalg.iterator_type<parallel>, #linalg.iterator_type<reduction>], operandSegmentSizes = array<i32: 1, 1>}> ({
  ^bb0(%arg3: f32, %arg4: f32):
    %1595 = "arith.addf"(%arg3, %arg4) <{fastmath = #arith.fastmath<none>}> : (f32, f32) -> f32
    "linalg.yield"(%1595) : (f32) -> ()
  }) : (tensor<198x384xf32>, tensor<198xf32>) -> tensor<198xf32>
  %73 = "tensor.expand_shape"(%72) <{reassociation = [[0, 1]]}> : (tensor<198xf32>) -> tensor<?x198xf32>
  %74 = "tensor.empty"(%53) : (index) -> tensor<?x198xf32>
  %75 = "linalg.generic"(%73, %74) <{indexing_maps = [affine_map<(d0, d1) -> (d0, d1)>, affine_map<(d0, d1) -> (d0, d1)>], iterator_types = [#linalg.iterator_type<parallel>, #linalg.iterator_type<parallel>], operandSegmentSizes = array<i32: 1, 1>}> ({

TLDR: Looks like we generate the problematic tensor.expand_shape during a FoldUnitExtentDims pass
Print-ir-after-all produces a 15MB text file.
/home/azureuser/iree-build/tools/iree-compile --iree-input-demote-i64-to-i32 --iree-hal-target-backends=llvm-cpu  stripped/deit-small-distilled-patch16-224.default.pytorch.torch.stripped.mlir -o deit-small-distilled-patch16-224.default.stripped.vmfb   --mlir-pretty-debuginfo         --mlir-print-ir-after-all &> printirafterall.mlir

But we can grep
┌─[0]─[azureuser@xida-cpu-0]─[~/SHARK-TestSuite/e2eshark/test-onnx/pytorch/models/deit-small-distilled-patch16-224]
└──╼ $cat printirafterall.mlir | grep -E '\(tensor<198xf32>\)|IR Dump'

<... a bunch of ir dump after messages with no matches to the tensor<198xf32> pattern ...>

// -----// IR Dump After ConvertElementwiseToLinalgPass (convert-elementwise-to-linalg) //----- //
// -----// IR Dump After RaiseSpecialOps (iree-global-opt-raise-special-ops) //----- //
// -----// IR Dump After DecomposeConcat (iree-global-opt-decompose-concat) //----- //
// -----// IR Dump After GeneralizeLinalgNamedOps (iree-global-opt-generalize-linalg-named-ops) //----- //
// -----// IR Dump After FoldUnitExtentDims Failed (iree-flow-fold-unit-extent-dims) //----- //
  %73 = "tensor.expand_shape"(%72) <{reassociation = [[0, 1]]}> : (tensor<198xf32>) -> tensor<?x198xf32>
  %86 = "tensor.expand_shape"(%85) <{reassociation = [[0, 1]]}> : (tensor<198xf32>) -> tensor<?x198xf32>
stripped/deit-small-distilled-patch16-224.default.pytorch.torch.stripped.mlir:364:36: note: see current operation: %73 = "tensor.expand_shape"(%72) <{reassociation = [[0, 1]]}> : (tensor<198xf32>) -> tensor<?x198xf32>

With the minimal ir, it correctly preserves the <?x198xf32>. See output at: https://gist.github.com/renxida/e4347a1ef027e9bf7ed3487e7d87d577
Command:
/home/azureuser/iree-build/tools/iree-compile --iree-input-demote-i64-to-i32 --iree-hal-target-backends=llvm-cpu  stripped/minimal.mlir -o minimal.vmfb   --mlir-pretty-debuginfo --mlir-print-ir-before=iree-flow-fold-unit-extent-dims --mlir-print-ir-after=iree-flow-fold-unit-extent-dims |& gh gist create - -d "minimal ir before and after FoldUnitExtentDims"

A search of the error message indicates that the error is generated by
https://github.com/shark-infra/llvm-project/blob/b3291793f11924a3b62601aabebebdcfbb12a9a1/mlir/lib/Dialect/Utils/ReshapeOpsUtils.cpp#L250-L257
This function LogicalResult mlir::reshapeLikeShapesAreCompatible is used for both the tensor.expand_shape and tensor.collapse_shape ops.
But this particular check
    if (dynamicShape) {
      if (!ShapedType::isDynamic(collapsedShape[map.index()])) {
        return emitError(
            "expected dimension " + Twine(map.index()) +
            " of collapsed type to be dynamic since one or more of the "
            "corresponding dimensions in the expanded type is dynamic");
      }
    }

may not make sense - - i can see a world where we allow collapsing unknown-shape dims.

https://gist.github.com/renxida/d39d9dd77991836d620737e57161d4a3
--compile-to=input

previously i thought the problem is introduced by iree-flow-fold-unit-extent-dims because the expand_shape wasn't there before the pass and after the pass it was there.
but then after slogging through it a bunch with stanley i noticed a couple of weird things in the ir from right before expand_shape:

a linalg generic that performed a reduce-sum over the last dim of a tensor, yet converted dim 0 from 1 to ?: tensor<1x198x384xf32>) outs(%27 : tensor<?x198x1xf32> (at: https://gist.github.com/renxida/cff78af3cc3dfde1e7755a8a6cdf24b9#file-before-expand-shape-mlir-L113) [^1]

a cast that turned a tensor with known shapes to unknown shapes %cast = tensor.cast %8 : tensor<1x196x384xf32> to tensor<?x?x?xf32> (see: https://gist.github.com/renxida/cff78af3cc3dfde1e7755a8a6cdf24b9#file-before-expand-shape-mlir-L64)

[^1]: full linalg generic here
  %28 = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d1, d2)>, affine_map<(d0, d1, d2) -> (d0, d1, 0)>], iterator_types = ["parallel", "parallel", "reduction"]} ins(%25 : tensor<1x198x384xf32>) outs(%27 : tensor<?x198x1xf32>) {
  ^bb0(%in: f32, %out: f32):
      %744 = arith.addf %in, %out : f32 // doing some sort of accumulative sum. adding output of previous loop. reduction.
      linalg.yield %744 : f32
  } -> tensor<?x198x1xf32>

looks like (1) from the last comment is introduced after iree-global-opt-decompose-concat
https://gist.github.com/renxida/5eeeaa252692438c560b000f6e58421d

Linalg generics that turn known shapes into unknown shapes:
cat printirafterall.mlir | grep -E 'tensor<1x198x?x.*outs.*\?x198|IR Dump'

Result:
// -----// IR Dump After ConvertTorchToLinalg (convert-torch-to-linalg) //----- //
  %533 = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2) -> (0, d1, d2)>, affine_map<(d0, d1, d2) -> (d0, d1, d2)>, affine_map<(d0, d1, d2) -> (d0, d1, d2)>], iterator_types = ["parallel", "parallel", "parallel"]} ins(%cast_641, %cast_153 : tensor<1x198x384xf32>, tensor<?x198x384xf32>) outs(%532 : tensor<?x198x384xf32>) {
  %855 = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2) -> (0, d1, d2)>, affine_map<(d0, d1, d2) -> (d0, d1, d2)>, affine_map<(d0, d1, d2) -> (d0, d1, d2)>], iterator_types = ["parallel", "parallel", "parallel"]} ins(%cast_1398, %cast_906 : tensor<1x198x384xf32>, tensor<?x198x384xf32>) outs(%854 : tensor<?x198x384xf32>) {

there are things like this that come relatively early in the pipeline (right after convert-torch-to-linalg).
These make sense because they're broadcasting an elementwise op between a 1x198x384 and a ?x194x384 to produce a ?x198x384
What doesn't make sense is when we have a 1x198x384 converted to a ?x198x1 with a single-operand reduce sum.
Need to filter specifically for those.
cat printirafterall.mlir | grep -E 'ins(%\d+tensor<1x198x?x.outs.?x198|IR Dump'

Let C2 denote the canonicalization pass after ExpandOps above.
C2 produces something likes this that eventually causes the error:
  %100 = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d1, d2)>, affine_map<(d0, d1, d2) -> (d0, d1, 0)>], iterator_types = ["parallel", "parallel", "reduction"]} ins(%97 : tensor<1x198x384xf32>) outs(%99 : tensor<?x198x1xf32>) {
  ^bb0(%in: f32 loc("after-memref-expand.mlir":587:8), %out: f32 loc("after-memref-expand.mlir":587:18)):
    %1984 = arith.addf %in, %out : f32 loc("after-memref-expand.mlir":588:13)
    linalg.yield %1984 : f32 loc("after-memref-expand.mlir":589:5)
  } -> tensor<?x198x1xf32> loc("after-memref-expand.mlir":586:10)

which, before canonicalization, looked like this:
  ^bb0(%in: f32, %out: f32):
    %4260 = arith.addf %in, %out : f32
    linalg.yield %4260 : f32
  } -> tensor<?x?x1xf32>

which means this is an issue caused by an incomplete shape inference that only assigned 1 to dim 0 of the input but not the output.

Summary so far:

original error: lowering native_layer_norm generated a tensor.expand_shape expanding a tensor<198xf32> to tensor<?x198xf32> after the pass that eliminates unit-extent dims.
this tensor.expand_shape causes an error because expand_shape expects only ? to correspond to ? dims, and not expanding a dim out to a ?
the tensor.expand_shape is materialized in between when the following pair of linalg ops are lowered:

// reduce sum over the last dim
// iterator types = parallel, parallel, reduction
%28 = linalg.generic ... ins(%25 : tensor<1x198x384xf32>) outs(%27 : tensor<?x198x1xf32>) ...

// elementwise division by a scalar
// iterator types = all parallel
%29 = linalg.generic ... ins(%28 : tensor<?x198x1xf32>) outs(%26 : tensor<?x198x1xf32>) ...

The root cause seems to be a linalg.generic that performs a reduce-sum with: ins(%25 : tensor<1x198x384xf32>) outs(%27 : tensor<?x198x1xf32>). Note that the reduce sum is performed on the 384 extent dim, but the first dim has input 1 but output ?
This doesn't make sense because we don't really touch the 1st dim and if the input is a 1, the output would also have to be a 1.
The reduce sum originally looked like this:
%338 = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2) -> (d0, d1, d2)>, affine_map<(d0, d1, d2) -> (d0, d1, 0)>], iterator_types = ["parallel", "parallel", "reduction"]} ins(%cast_173 : tensor<?x198x384xf32>) outs(%337 : tensor<?x?x1xf32>)
and the ? for the input is filled in by shape inference, but the ? for the out remained a ?.

Latest status: after updating to the latest torch-mlir and iree, encountering an Unsqueeze error. Either something fixed this, or an Unsqueeze error masked this.

https://github.com/nod-ai/e2eshark-reports/blob/main/2024-06-12/onnx_reports/statusreport.md
model pass based on this report


Twitter::Error::TooManyRequests: Rate limit exceeded

I've been using this gem for a while.
But suddenly, since last Friday 20 Nov 2015 afternoon (GMT), the program stopped getting any search results.  I'm getting
Twitter::Error::TooManyRequests: Rate limit exceeded
This problem has persisted for the past 5 days.
Clearly the error message is misleading, as I'm NOT exceeding the rate limit.
So either my credentials are expired or there're some issues with Twitter REST API.
Anyone else experienced this problem?
Thank you

We ran into this same issue. If you're calling to_a or each on your results, you will end up fetching the next page of search results.
https://github.com/sferik/twitter/blob/master/lib/twitter/enumerable.rb#L13
We fixed this in our app by:
query_results.attrs[:search_metadata][:next_results] = nil


命令"rm-js-comment"导致错误

([BABEL] unknown: Preset /* your preset */ requires a filename to be set when babel is called directly,
babel.transformSync(code, { filename: 'file.ts', presets: [/* your preset */] });

See https://babeljs.io/docs/en/options#filename for more information.)

I have fixed this issue, please update your version


export mk_msg and mk_msgs

This PR exports mk_msg and mk_msgs, which was overlooked when integrated msglm in #7.
This PR also makes some minor copy changes.

@comhar feel free to merge yourself once you've fixed the conflicts.


[BEAM-7574] fix Combine performance for SparkRunner

Fixes [BEAM-7574]

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

R: @dmvk @iemejia @robertwb

Performance testing of this fix is still ongoing.

First run of test job results in significant performance boost compared to previous version (minutes opposed to hours). My test job is a little specific, though. It has many little time windows. @dmvk can you test this on some job you have?

SUCCESS
--none--

Run Spark ValidatesRunner

Run Spark ValidatesRunner

Run Nexmark on the current PR and saw no regressions so far and clear improvements in Combine heavy queries. So positive about this, even if not checked the code yet.
Command:
./gradlew :sdks:java:testing:nexmark:run \
    -Pnexmark.runner=":beam-runners-spark" \
    -Pnexmark.args="
        --runner=SparkRunner
        --streaming=false
        --suite=SMOKE
        --manageResources=false
        --monitorJobs=true
        --enforceEncodability=true
        --enforceImmutability=true"

Results:
Master
  Conf  Runtime(sec)    (Baseline)  Events(/sec)    (Baseline)       Results    (Baseline)
  0000           1.5                     68870.5                      100000
  0001           0.6                    172711.6                       92000
  0002           0.4                    284900.3                         351
  0003           1.7                     58685.4                         580
  0004           0.9                     10893.2                          40
  0005           2.1                     46992.5                          12
  0006           0.6                     17006.8                         103
  0007           2.7                     37243.9                           1
  0008           1.2                     85324.2                        6000
  0009           0.5                     20080.3                         298
  0010           0.6                    161550.9                           1
  0011           1.0                     98328.4                        1919
  0012           0.8                    118063.8                        1919
  0013           0.4                    267379.7                       92000
  0014           2.9                     69710.7                      184000
==========================================================================================

This PR
  Conf  Runtime(sec)    (Baseline)  Events(/sec)    (Baseline)       Results    (Baseline)
  0000           1.3                     74906.4                      100000
  0001           0.6                    166666.7                       92000
  0002           0.3                    392156.9                         351
  0003           1.6                     61050.1                         580
  0004           0.8                     12610.3                          40
  0005           2.1                     48709.2                          12
  0006           0.6                     17574.7                         103
  0007           2.4                     42247.6                           1
  0008           1.0                     98814.2                        6000
  0009           0.4                     22421.5                         298
  0010           0.6                    171821.3                           1
  0011           1.4                     70028.0                        1919
  0012           0.8                    130378.1                        1919
  0013           0.4                    253164.6                       92000
==========================================================================================

Note that the execution is breaking on query 14 but that seems unrelated to the PR.

There seems to be performance regression on tests 11 and 12, can you post a link to there tests?

Of course, not sure if these differences are significant, does the test suite run multiple times?

I run it multiple times and this is an example of an average run. Of course in this case it uses a really small synthetic dataset but Nexmark has helped us multiple times to catch regressions or confirm improvements.
There is always a skew between runs of +/- 3%, it used to be more stable between runs in the pre gradle and pre meltdown days. I really would like to find a way to execute these tests in a more reliable way so any hint on that would be greatly apprecaited. In any case the average of runs is positive.

@iemejia cool, that sounds promising, thanks!

Commit 4a7531b had roughly 25% performance gain on my test job, which uses massive counts of windows per key.

Run Spark ValidatesRunner

Run Java Spark PortableValidatesRunner Batch

R: @RyanSkraba Can you help me with this review. Thanks!

Run Spark ValidatesRunner

@je-ik can you please check if there is some issue with the implementation?. From the jenkins logs it looks like at some point the timestamps are becoming null.

@iemejia sure, thanks. I  didn't check these changes against streaming validates runner, because I thought that these changes were related to batch only, but I might had been wrong.
@RyanSkraba thanks for review and for some comments. I will walk through it as soon as possible (during this week).

Run Spark ValidatesRunner

Run Spark ValidatesRunner
Run JavaPortabilityApi PreCommit

Run Spark ValidatesRunner

Run Spark ValidatesRunner

Run JavaPortabilityApi PreCommit

@iemejia @RyanSkraba all tests green, there was bug in Combine.globally, which is fixed now.

Run Java PreCommit

Run JavaPortabilityApi PreCommit

:+1: For the extra @SuppressWarnings("unchecked") -- I wouldn't let that block this PR, since they aren't a checkstyle error.  Thanks for this opportunity to dive quite a bit deeper into the SparkRunner!

Run Spark ValidatesRunner

I also plan to remove the AbstractSparkCombineFn as you suggested, just didn't have time to do it yet.

Should we wait for this fix or maybe we can do it in a subsequent PR @je-ik @RyanSkraba?
I would like to get this issue in 2.15.0 if possible (branch cut is in 2 days).

I'd like to do it in this PR. Will push ASAP. This PR then needs rebase and squash commits.

Ok, Thanks a lot @je-ik Just ping me when done for the final check + merge.

Retest this please

Run Spark ValidatesRunner

Run Java Spark PortableValidatesRunner Batch

Run Java PreCommit

Thanks for the clean-up!  In my opinion, this looks good to merge for 2.15.0!

@iemejia Tests pass, LGTM

@dmvk you can be interested in testing this (probably via the SNAPSHOTS after tomorrow). Any info on regressions or improvements for your use cases will be definitely appreciated.


Added Satori client support

@britzl Should the satori folder be added to include_dirs so that it can be fetched from Defold via git using the dependencies section?

Can you add and use a session object just like in Nakama?

@britzl Should the satori folder be added to include_dirs so that it can be fetched from Defold via git using the dependencies section?

Ah, yes, it should. I've added the folder.

Can you add and use a session object just like in Nakama?

Yes, satori.authenticate() returns an api session. You can use satori.set_bearer_token() to set the session token for future requests. What that what you meant?

@britzl could you add an example to the README?

Could you also leave instructions for running the Satori SDK tests in the README?

@britzl could you add an example to the README?

I've updated the readme with Satori info.

Could you also leave instructions for running the Satori SDK tests in the README?

Added!

Docs PR: https://github.com/heroiclabs/docs/pull/680

@DannyIsYog thanks! Will you create a new release?

Hey @britzl, no worries :)
Feel free to make a new release, I belive you have permissions for that.


OTIO Editing API Design

This is the relevant PR for Issue 771
This lays out the initial design specs for the edit platform within OTIO.

Codecov Report

Merging #719 into master will decrease coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #719      +/-   ##
==========================================
- Coverage   81.88%   81.86%   -0.02%
==========================================
  Files          72       72
  Lines        2755     2752       -3
==========================================
- Hits         2256     2253       -3
  Misses        499      499

Flag
Coverage Δ

#py27
81.86% <ø> (ø)

#py36
?

#py37
?

Impacted Files
Coverage Δ

...ntimelineio/opentimelineio-bindings/otio_utils.cpp
87.05% <0.00%> (-0.16%)
:arrow_down:

...elineio/opentimelineio-bindings/otio_anyVector.cpp
100.00% <0.00%> (ø)

...eio/opentimelineio-bindings/otio_anyDictionary.cpp
100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c61d309...eaefe8a. Read the comment docs.


⚠ PrivateHD has degraded performance

In 33feebe, PrivateHD ($PHD) experienced degraded performance:

HTTP code: 200
Response time: 6187 ms

Resolved: PrivateHD performance has improved in 674f97c.


BAEL-3022 - Converting Iterator to List

Included options to convert Iterator to a List using Plain Java, Java 8, Java 8 Streams, Guava and Apache libraries.

Abandoning this Pull request. I need to exclude code related to Hexagonal architecture


"export 'extras' was not found in 'mobx'

I am using yarn and trying to upgrade mobx to v4.1.1 and mobx-react to v4.4.3.
Yarn is giving me this warning:
warning " ><EMAIL_ADDRESS>has incorrect peer dependency<EMAIL_ADDRESS>|| ^3.0.0".
I ran yarn remove mobx mobx-react and when that was completed, I deleted the yarn.lock file and confirmed mobx and mobx-react were no longer in ./node_module.
Then I added the new versions to my package.json ...
    "mobx": "^4.1.1",
    "mobx-react": "^4.4.3",

But when I run yarn again,  I have the same problem, and when I look at  ./node_module/mobx-react/package.json I find this ...
  "peerDependencies": {
    "mobx": "^2.6.3 || ^3.0.0",
    "react": "^0.13.0 || ^0.14.0 || ^15.0.0 || ^16.0.0"
  },

Any suggestions how I can solve this problem, so I can get to the latest mobx and mobx-react versions?
Thanks!
Chris

I didn't realize there was a version 5. When I go to mobx-react on Github, the last version I see is 4.4.3 in the change log.
I'll change my package.json and give that a try.
Is version 5 a beta or final release?

final, will update the changelog, I think that was an oversight

mobx-react 5 is not support some brower,what can i fixed #449

@moonlitusun please open new issues for new issues. Is your problem same as #608 ?


Update CI

While here, drop python 3.8, and enable python 3.12.
And setup mergify

Hum, no, cmeel is not ready. Let's keep 3.8 <= x <= 3.11


Did the module align implement the alignment function？

In the script file align_dataset_mtcnn.py, i can only find the expanding margin but not alignment. Furthermore, the face landmarks returned by function align.detect_face.detect_face() is not used in the file. Did I miss some details?  Can anybody help me,? Thanks a lot.

No you are right there is no real alignment the face is just cropped out of the picture. But as quoted from the Google FaceNet Paper.

Another strength of our model is that it only requires
minimal alignment (tight crop around the face area). [17],
for example, performs a complex 3D alignment. We also
experimented with a similarity transform alignment and notice
that this can actually improve performance slightly. It
is not clear if it is worth the extra complexity.

Thanks very much.


Lock information on Keystore Tab (Watch Only Wallet)

It is possible to edit information when Keystore is of type 'Watch Only Wallet'.
Master fingerprint, Derivation, tpub / Vpub Fields are editable.
Like with other keystore types, the fields should be in locked state once the wallet is saved. Even though there is a fail-save (popup "Change Wallet Addresses?"), users should use the 'Replace' Button instead.

It is possible to edit information when Keystore is of type 'Watch Only Wallet'.

Yes. But the edited information is only used when the wallet is saved with the Apply button. If the user wants to go back to the original before saving, the Revert button is available.

Like with other keystore types, the fields should be in locked state once the wallet is saved. Even though there is a fail-save (popup "Change Wallet Addresses?"), users should use the 'Replace' Button instead.

There is a significant difference here from the other keystore types - the fields for a watch only wallet are only editable by changing these fields. So requiring the user to use the Replace button not only seems less intuitive to me, but would require the user to start from scratch rather than just editing one of the existing fields.


Unexpected compile error: "Cannot reference type: Microsoft.FSharp.Core.Option"

I'm having some compilation problems after upgrading to the latest version of Fable (0.4.4). I have been getting this error during compilation in a few places where I'm pattern matching on options and returning float values:

ERROR: Cannot reference type: Microsoft.FSharp.Core.Option

Here's some F# code that triggers the error:
let test (x: float option) =
   match x with
   | Some y -> y + 3.
   | None -> 0.

If I replace y + 3 with just y it compiles fine. If I use other float operations (+,/,-) I still get the error.

Thanks for the report, unfortunately this is a problem that surfaced in the latest version because other changes. I've released<EMAIL_ADDRESS>with the fix :+1:

Fantastic -- thanks!


🚧  Add new header for No 10 organisation page

This PR:

makes No 10 organisation page full width
adds a No 10 organisation header with an image with gradient overlay to improve colour contrast with heading text, and a three card image card grid without images
a fix for the image card margins and width on mobile - we should investigate the use cases without image for the component and potentially move this fix to the gem

Still to do:

add a couple of tests for the new header, fix failing tests

Fixes https://trello.com/c/pdyIMeno/38-implement-the-redesigned-no10-page-header-l
⚠ This repo is Continuously Deployed: make sure you follow the guidance ⚠
Follow these steps if you are doing a Rails upgrade.

Looks good, just a couple of places where the heading component from govuk_publishing_components could be used.


Test-DbaMaxDop - Fixed link to calculator

Type of Change

[X] Bug fix (non-breaking change, fixes #)
[ ] New feature (non-breaking change, adds functionality)
[ ] Breaking change (effects multiple commands or functionality)
[ ] Ran manual Pester test and has passed (`.\tests\manual.pester.ps1)
[ ] Adding code coverage to existing functionality
[ ] Pester test is included
[ ] If new file reference added for test, has is been added to github.com/sqlcollaborative/appveyor-lab ?
[ ] Nunit test is included
[X] Documentation
[ ] Build system

Purpose
The link to the calculator was broken, it had a -SqlInstance pasted on the end.
Approach
Removed -SqlInstance and now the link works 😍

💯


Event for toggling GF to sing with BF

Describe your problem here.
So, basically, I want to figure out a way to make GF sing with BF only when this specific event is toggled.
here is the script so far:
local isOn = false

function onEvent(name, value1, value2)
    if name == "GF+BF Sing" then
	isOn = not isOn
    end
end

function goodNoteHit(id, noteData, noteType, isSustainNote)
	if isOn then
		--code here
	end
end

For context, when the event is called, it will set isOn to true, making GF able to sing with BF. If the event gets called again, it sets isOn back to false.
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
No.

characterPlayAnim is what you're looking for

characterPlayAnim is what you're looking for

hey so uhhh, what do I do with this now? characterPlayAnim('GF', [what can I insert here?], false)

You can use the code from this #5471


Getting 404 for /socket-io/?EIO=3&transport=polling&t=NG2FwLh when netty-socketio server and tomcat run on same port

I'm trying to configure a Sprint Boot REST app that uses port 8080 to also use SocketIOServer on port 8080.
if i use another port for SocketIOServer, it works fine.
if i use port 8080, i get a port in use error unless i set the following:
@Bean
public SocketIOServer socketIOServer() {
    com.corundumstudio.socketio.Configuration config =
            new com.corundumstudio.socketio.Configuration();
    config.setHostname(nwtNasaWebsocketServer);
    config.setPort(Integer.parseInt(nwtNasaWebsocketPortString));
    SocketConfig sockConfig = new SocketConfig();
    sockConfig.setReuseAddress(true);
    config.setSocketConfig(sockConfig);
    return new SocketIOServer(config);
}

but then when i try to connect from the socket io client, i get a 404 for below call:
http://localhost:8080/socket-io/?EIO=3&transport=polling&t=NG2FwLh
I see similar issue opened previously https://github.com/mrniko/netty-socketio/issues/545 but no solution hint has been provided. Could someone please help here?

Hello, has this issue been resolved


photon-controller driver no longer working

Rancher Version: master 6/13
Steps to Reproduce:

try to add the photon-controller driver

Results: Goes directly to error
Expected: Should setup

Closing due to inactivity, please reopen if you think this is still relevant to current releases & important.


How to [insert question]?

Hey, is there a way to optimise for a target volatility (say 10%), based solely on covariance? i.e without specifying return estimates... If not, I'm happy to have a go at adding, let me know if you have a preferred approach. Cheers

Thanks for reaching out! I'm closing this as it is discussed in great detail in #116
Best,
Robert


Super Admin permissions are not listed in the Permission Tree of the Cluster setup

Environment
IS 5.4.0 Cluster setup (2 nodes) fronted by nginx
JDK - 1.8.0_45
DB - Mysql 5.6
Email as user enabled
Steps to recreate the issue

Create a role
Assign permission to the created role

Observations
'Super admin permission' is not listed on the permission tree

Only 'admin' permission is available in the "/_system/governance/permission" path

Expected Behaviour
Super admin permissions should be visible with the permission tree
Note - This behaviour only observed in the cluster setup. (Registry mounted and the carbon DB configured in H2)

Thank you for your contribution!
We are closing this issue since it has not been prioritized for a long time. Chances are that it has already been solved in more recent versions. If not, we will be re-evaluating this when it becomes a priority.


I still see connection settings after deleting the connection.

I expected the settings file to be cleaned up after deleting my connection but all the filters and library lists for the various profiles are still in the settings.json file.
I don't know if this is desired but it's not for me.
I wanted to "clean up" by deleting my connection but that did not work.  I still have to scroll through the settings.json file to perform the additional clean-up.
Is this a bug or a feature request?

A feature request.
I guess it's deleting it from the connections but not connectionSettings?
Liam

Yes. I spent a small amount of time cleaning up my connections settings so that I could restart with the newest release.


Some typos

Draft May 21. It's an amazing book and I can't wait to read further!

A.3.3 The softmax function is called S(x) but defined over \alpha.
The equation (7.231) holds for differentiable f, but not in general. In particular, it is not enough that the limes in the definition of D_v f(x) is defined at x (what a reader unfamiliar with the material might assume).
Equation 7.28 should say E[v^T A^T A v] I think
Under equation 7.204 you write Hence we can first solve L_{nn} y_n = b_n to find y_n. But shouldn't we start with L_{11} y_1 = b_1?

thanks, fixed


Rotating iCub GIFs  affects menu readability

On my laptop, the rotating iCub GIFs in the main menu affects the readability of the menu:

I would personally remove it, but perhaps there is a specific reason for which the GIF was placed there.
@marcoaccame

I had the same impression. We can remove it for improving readability.

Fixed in https://github.com/icub-tech-iit/documentation/commit/f6f43c88de75a5706525106ebfe6e344f3337ec9.


WinINet.InternetSetOption seems breaking change in 4.0.0

Describe the bug and how to reproduce
3.4.17

4.0.0

What code is involved
WinINet.InternetSetOption(WinINet.HINTERNET.NULL, WinINet.InternetOptionFlags.INTERNET_OPTION_PROXY_SETTINGS_CHANGED)

exception:
Error: System.ArgumentException: INTERNET_OPTION_PROXY_SETTINGS_CHANGED cannot be used to set options that do not require a value.
at Vanara.PInvoke.WinINet.InternetSetOption(HINTERNET hInternet, InternetOptionFlags option)
at Submission#5.<<Initialize>>d__0.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)

Expected behavior
No exception as 3.4.17.

Thank you. Fixed for 4.0.1.


Unable to get messageId and publishTime of PubSubMessage in Google Cloud Function

I am trying to follow the Spring Cloud Function GCP adapter example to receive Pub/Sub message  in Google Cloud Function.  While getting the message payload from the data field and the message attributes from 'attributes' field, the values of messageId and publishTime are coming as null.
Version of spring-cloud-function-adapter-gcp used is 3.1.2

@meltsufin can you please take a look at this?

@kmmanu I'm able to repro this issue. I think the reason that this is happening is because the event is somehow being processed through HttpFunction codepath rather than through RawBackgroundFunction. Our GcfJarLauncher implements both interfaces, and we would expect accept(String json, Context context) to be called rather than service(HttpRequest httpRequest, HttpResponse httpResponse), but that doesn't seem to be happening.
@dzou Can you please look into this more?

Sure I will take a look.

So it looks like the GCF function receives this as a payload from Pub/Sub: {"@type":"type.googleapis.com/google.pubsub.v1.PubsubMessage","data":"SGVsbG8gd29ybGQgZWVlZQ\u003d\u003d"}. There doesn't seem to be any info about timestamp/id.
On the other hand, I found that the Context object contains this info. You'll have to change your function to accept Message and access the Context that we set here.
Something like this:
	@Bean
	public Consumer<Message<PubSubMessage>> pubSubFunction() {
		return message -> {
			// The PubSubMessage data field arrives as a base-64 encoded string and must be decoded.
			// See: https://cloud.google.com/functions/docs/calling/pubsub#event_structure

			PubSubMessage payload = message.getPayload();
			String decodedMessage = new String(
					Base64.getDecoder().decode(message.getPayload().getData()), StandardCharsets.UTF_8);
			System.out.println("Hello!!! Received Pub/Sub message with data: " + decodedMessage);

                        // Print out timestamp and event id
			Context ctx = message.getHeaders().get("gcf_context", Context.class);
			System.out.println(ctx.eventId());
			System.out.println(ctx.timestamp());
		};
	}

What is puzzling is that the GCF framework seems to indicate it should be possible to accept timestamp, attributes, etc. in your Pub/Sub pojo. I wonder if something broke/changed underneath.

Thanks @dzou
The above code is working and I will use the Message header to get the message-Id and even-time as you suggested.


How to get the Plugin ID for the DbNotRespondingPlugin

How do you get the FEATURE_ID for the DbNotRespondingPlugin that the CURl command is expecting?
curl -X PUT "http://{IP_ADDRESS}/feature-flag-service/v1/flags/{FEATURE_ID}/"
-H  "accept: application/json"
-d "{"enabled": {VALUE}}"

Hi,
you can get a list of currently available problem pattern flags by sending a GET request to http://{IP_ADDRESS}/feature-flag-service/v1/flags?tag=problem_pattern
The specific id for DbNotRespondingPlugin is db_not_responding


exports support

Now that this is implemented in Node.js --experimental-modules we should be able to support package.json "exports". Once we have "aliases", we should be able to deprecate "map" for "exports" and "aliases".
One hard thing will be supporting the "browser" field properly under this model. It's still debateable whether the "browser" field object form should support mapping modules if we have differential exports and aliases. Perhaps "map" can remain the greater abstraction, or perhaps we can directly unify on a "browser" desugar into aliasing and exports.

Actually I figured, most of my monorepo related requirements https://github.com/jspm/jspm-cli/issues/1534 align with this but got a better deal in terms of https://github.com/jspm/jspm-cli/pull/2491 💯
Thanks for adding it in roadmap. This is super handy. I am looking forward for it.

This was included in https://github.com/jspm/jspm-resolve/pull/13.


strtolower() warning in the query builder

Laravel Version: 9.1.0
PHP Version: 8.1.1
Database Driver & Version: MySQL 8

Description:
A little reluctant to call this a bug but we're seeing warnings in production (hooray for the log deprecations channel!).
strtolower(): Passing null to parameter #1 ($string) of type string is deprecated in
./vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php on line 841 (and 840)

Steps To Reproduce:

I didn't see a trace but I'm guessing this is triggered in the where method, as we short-cut it often, but I have not been able to pin-point the exact source of the error unfortunately.

    /**      * Determine if the given operator is supported.      *      * @param  string  $operator      * @return bool      */     protected function invalidOperator($operator)     {         return ! in_array(strtolower($operator), $this->operators, true) &&                ! in_array(strtolower($operator), $this->grammar->getOperators(), true);     }
This part of the code threw that error. I guess you are giving wrong parameter when you are using where method, can you add your code that generating this error?

I found what's causing this and it is in fact an error from our end, however, I still think this is a bug.
For simplicity, our query looks like this (with expanded variables):
$query->where('some_field', null);

The order for the where method is:

Check for invalidOperator(), calling what you've mentioned @gguney. If it's null, strtolower() warns about it.
Later on there is an is_null check that catches our null value and passes it onto whereNull.

This seems to be fixed in https://github.com/laravel/framework/pull/41091 though. I haven't tested it yet but the check for ! is_string() should catch it.


adding my contribution at index.md

ampliation of introductory concepts

[x] I have read freeCodeCamp's contribution guidelines.
[ ] My pull request has a descriptive title (not a vague title like Update index.md)
[ ] My pull request targets the master branch of freeCodeCamp.
[ ] None of my changes are plagiarized from another source without proper attribution.
[ ] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

Thank you for  creating this PR.  However, since it was created, the underlying document has changed.  Can you please review and resolve the current merge conflicts?  It could be the changes to the underlying document are so significant that your changes are no longer relevant.  If the latter is the case, it might be easier to just close this PR and create a new PR with any changes you still feel are important to make.
If you need any assistance or have questions about resolving the merge conflicts, please reply back or contact someone in the Contributors Room on Gitter.
Thank you.

Hi @MichelPra
As there has been no response from your end for a while, we are closing this PR.
Feel free to request this PR to be reopened or create a new PR, if you still wish to make the changes you originally added plus any additional change requests asked of you.  Also, remember to update your fork, so you have the latest version of master in case it has changed significantly.
Happy Coding!


Items appear constricted or squeezed within itself when in itemsRepeater using UniformGridLayout

Describe the bug
I have setup a minimal reproduction of this using the latest version of avalonia here:
https://github.com/Eugenenoble2005/UniformGridLayoutBugRepo
When using the ItemsRepeater control and setting it's layout to UniformGridLayout, the contents will appear squeezed hiding much of it when the content is a number. No amount of internal padding set on the control fixes it , it's as if no content is rendered after the first digit. This only happens when UniformGridLayout is used. StackLayout, WrapLayout and DockLayout afaik do not cause this bug. Experienced this bug on Windows and Linux(arch and ubuntu).
Images:
Bug when using UniformGridLayout

No bug(expected output) when using WrapLayout:

To Reproduce
See cited reproduction:
https://github.com/Eugenenoble2005/UniformGridLayoutBugRepo
For example an itemsRepeater that prints 1 to 100 in buttons. The bug seems to only happen when numerical digits are being printed. Strings are fine and do not cause the bug, But strings that contain a number in them e.g "This is a long string 44" will cause the bug and the end of the sentence will be cut out. I have also tried converting the number to a string before binding them like so : number.toString(), the bug still occurs.
Expected behavior
No response
Avalonia version
11.1.0
OS
Windows, Linux
Additional context
No response

It's not squeezed, it's uniform. IIRC, the first item is measured and that is what is used for the layout. You can set MinItemWidth otherwise. Remember that these controls are designed to support many thousands of child controls (or more) so measuring each one individually to find a maximum is expensive.

@stevemonaco works great now. Thank you.


Added in the LoginCookieName directive to allow for EZproxy session cookie customisation

Added in the LoginCookieName directive to allow for EZproxy session cookie customisation

Thanks!


Run tests in different folder

Less noise in git diff :)

Thanks this is very handy and a simple solution :+1:


[Event Request] Table 79 "Company Information" - OnBeforeGetVATRegistrationNumberLbl()

Please add a new publisher event to function GetVATRegistrationNumberLbl() like below.
procedure GetVATRegistrationNumberLbl(): Text
Var
  ReturnValue : Text; //New local var
  IsHandled: Boolean; //New local var
begin
if Name = '' then // Is the record loaded?
  Get;

//>>Start Event Publisher
OnBeforeGetVATRegistrationNumberLbl(Rec,ReturnValue,IsHandled);
if IsHandled then
  exit(ReturnValue);
//>>End Event Publisher

if "VAT Registration No." = '' then
  exit('');
exit(FieldCaption("VAT Registration No."));
end;

New Publisher Event
local procedure OnBeforeGetVATRegistrationNumberLbl(CompanyInformation: Record "Company Information"; var ReturnValue: Text; var IsHandled: Boolean);

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


chore(release): 29.1.4

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 96.03%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
4045

Relevant Lines:
4144

💛  - Coveralls


data sample demand？

Hi, thanks for your great work!
Can u share some data sample, perhaps 100 item for quick test

Thank you for your attention to our work. I have released 3 groups of images here for a quick evaluation. The gradio demo with more test images and the held-out evaluation set will be available in several days.


unbound-control error if unbound is started with custom config filename

When the unbound is started with the custom config file name:
./unbound -c custom.conf
custom.cof:
remote-control:
control-enable: yes
control-interface: <IP_ADDRESS>
control-port: 8953
Running the following unbound-control command will result in the error:
linux:/usr/local/etc/unbound# unbound-control dump_cache -s <IP_ADDRESS>
[1561341472] unbound-control[19082:0] error: Could not open /usr/local/etc/unbound/unbound.conf: No such file or directory
[1561341472] unbound-control[19082:0] fatal error: could not read config file
I don't know whether it is a bug or my misunderstanding of the documentation.

I think the solution is
unbound-control -c custom.conf status
But you also give a -s <IP_ADDRESS> option.  This wants to read the /usr/local/etc config file and then override the IP address to <IP_ADDRESS> (that was already <IP_ADDRESS>).
What documentation or option could be improved to make it easier to find out (or add an option to make it easier to use somehow?).

I think -s option becomes obsolete when the config file name is not the default. Or if the control-interface in the config file is different from IP address given in -s (Obviously!). Probably -s should be removed after all since I cannot find any unique advantages of using it independently?

Yes the -s looks not useful for you, here.  It is useful in (test) cases elsewhere.  The config file is needed to also signal the names of the files with the TLS keys for the connection.
The -s option can be useful if you copy the files and config (snippet with remote-control) to another host, then you can contact the server with the -c and -s options both to specify the files with TLS keys and the -s with the IP-address that is different.  Althought in the file on the other host, you could of course also adjust the ip-address to work.
In newer unbound versions, you can use a local-socket, by specifying a filename for the socket (/tmp/unbound-control.sock or somesuch), and then it uses a local pipe.  The socket in its own directory is good, /run/unbound/unbound-control.sock for example, because then the permissions on that directory and on the file can be used to control access to the service.

Thank you very much. This was very useful.

I have a similar issue on 1.9.2:
/ # unbound-control -c /etc/unbound/unbound.conf reload
ok
/ # unbound-control -c /etc/unbound/unbound.conf reload
[1562830044] unbound-control[33:0] error: connect: Connection refused for /run/unbound/control.socket

While this can be found in the unbound logs :
11/07/2019 09:14:24[1562829264] unbound[15:0] notice: Restart of unbound 1.9.2.
11/07/2019 09:14:24[1562829264] unbound[15:0] error: Could not open /etc/unbound/unbound.conf: No such file or directory
11/07/2019 09:14:24[1562829264] unbound[15:0] warning: Continuing with default config settings

The behaviour on 1.8.3 is normal

So, the first reload command reloads unbound but because the configuration file could not be read unbound starts with the default options. By default the remote control is turned off that's why you can't connect anymore.
This seems like a chroot issue. When unbound initially starts it reads the configuration file and then sets up the chroot environment. When the reload command is issued unbound can't find the configuration file in the chroot environment.
If after checking your configuration you still have issues you can share the following output:
unbound-control -c /etc/unbound/unbound.conf get_option directory
and
unbound-control -c /etc/unbound/unbound.conf get_option chroot

@JasonGantner Where you able to figure out your issue?

Closing this one.

@gthess Sorry for the delay, I've unfortunately been away for a few month :/
Here's the output :
/ # unbound-control -c /etc/unbound/unbound.conf get_option directory
/usr/local/etc/unbound
/ # unbound-control -c /etc/unbound/unbound.conf get_option chroot
/usr/local/etc/unbound

Following your suggestions, I added chroot: "" to the config file and this fixed the issue.
(The Unbound server is dockerized so chrooting isn't needed)
Out of curiosity, is this a change of default behavior between 1.8.x and 1.9.x ?
(The issue did not occur with 1.8.x)
Thanks a lot for your help and sorry again for the delay

No problem.
There was no change in chroots's default value (default on). Is this a packaged unbound? I suspect that the maintainers may have forgotten to turn off the chroot for the new version.

I switched from packaged to self-built to enable GOST support and update to latest stable as soon as available instead of waiting for weeks or more to update. That explains the change in default value.
My bad.


Make lots of small miners

If millions of people around the world can mine 1 or 2 DOGE coin on their laptops per day, I think many will do so, prevent greedy mining in the algorithm, this improves decentralisation and hoarding of mining capabilities. Gives poor people in africa and other places opportunity to make money and learn about crypto.

good idea!

#2059 fixes selfish mining.

The trouble is asic resistance. If you can do it on your laptop, an ASIC can do it much faster. So far, only GPU mining ala ETC/ETH seems to have avoided this pitfall.

Love it! Dogecoin needs to remain accecible - to free people from poverty.
All though tech has moved on, staying as close to Satochi's vision of 'one CPU per vote' would be ideal.

Using this also we are just making doge again costlier and for producing 1 or 2 coins we still use alot of energy. And here the complexity arises!!

Sure, but the energy debate is a red hering.
Electricty grids are barely 100 old, theoretically, energy production is becoming more efficient every day. (Some bad actors and private intrests may be working against this objective.)
All modeling is done using a snapshot of todays energy generation costs, but this will decrease overtime and is on trend to become net surplus - free electricity.

If we wants to use optimum electricity usage as per today's
need/requirements then we can't give access to small miners as they consume
a lot for a small amount and we can not consider that they use any
recommended source of power!
On Tue, May 25, 2021, 1:43 PM freethe3rdworld @.***>
wrote:

Using this also we are just making doge again costlier and for producing 1
or 2 coins we still use alot of energy. And here the complexity arises!!
Sure, but the energy debate is a red hering.
Electricty grids are barely 100 old, theoretically, energy production is
becoming more efficient every day. (Some bad actors and private intrests
may be working against this objective.)
All modeling is done using a snapshot of todays energy generation costs,
but this will decrease overtime and is on trend to become net surplus -
free electricity.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dogecoin/dogecoin/issues/2060#issuecomment-847654828,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AUGXUR6SJFEBLIMDJ7YCV6LTPNLZRANCNFSM45ONG5ZA
.

If we wants to use optimum electricity usage as per today's need/requirements then we can't give access to small miners as they consume a lot for a small amount and we can not consider that they use any recommended source of power!
…
On Tue, May 25, 2021, 1:43 PM freethe3rdworld @.***> wrote: Using this also we are just making doge again costlier and for producing 1 or 2 coins we still use alot of energy. And here the complexity arises!! Sure, but the energy debate is a red hering. Electricty grids are barely 100 old, theoretically, energy production is becoming more efficient every day. (Some bad actors and private intrests may be working against this objective.) All modeling is done using a snapshot of todays energy generation costs, but this will decrease overtime and is on trend to become net surplus - free electricity. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2060 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUGXUR6SJFEBLIMDJ7YCV6LTPNLZRANCNFSM45ONG5ZA .

Even if they're pooling? And even if they running on idle/background?
So you're saying there's no way to include small miners, poorer people?

We can include small miners but only in case if they are using any optimum
mode of electricity.
See there is nothing we can say a No!
We cant say no one can beat Elon.
We cant say intergalactic travel and residence is not possible.
We cant use that impossible today!
On Tue, May 25, 2021, 2:06 PM freethe3rdworld @.***>
wrote:

If we wants to use optimum electricity usage as per today's
need/requirements then we can't give access to small miners as they consume
a lot for a small amount and we can not consider that they use any
recommended source of power!
… <#m_-843914008521000488_>
On Tue, May 25, 2021, 1:43 PM freethe3rdworld @.***> wrote: Using this
also we are just making doge again costlier and for producing 1 or 2 coins
we still use alot of energy. And here the complexity arises!! Sure, but the
energy debate is a red hering. Electricty grids are barely 100 old,
theoretically, energy production is becoming more efficient every day.
(Some bad actors and private intrests may be working against this
objective.) All modeling is done using a snapshot of todays energy
generation costs, but this will decrease overtime and is on trend to become
net surplus - free electricity. — You are receiving this because you
commented. Reply to this email directly, view it on GitHub <#2060
(comment)
https://github.com/dogecoin/dogecoin/issues/2060#issuecomment-847654828>,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AUGXUR6SJFEBLIMDJ7YCV6LTPNLZRANCNFSM45ONG5ZA
.
Even if they're pooling? And even if they running on idle/background?
So you're saying there's no way to include small miners, poorer people?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dogecoin/dogecoin/issues/2060#issuecomment-847672254,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AUGXURY2EV4NIKLZVOEKIH3TPNOSRANCNFSM45ONG5ZA
.

That's good.
Satoshi's vision was 1 cpu = 1 Vote. PoS = only the rich have a voice.
Energy effitience is less important than decentrilasation, freedom, in the grand scheme.
The energy debate is short sighted, and a non-issue compared to energy use in legacy finance, manfucturing, the mining of finite resources (metals, oil). (For what you get in return.)

You can create a small miner today!


"Create my GitHub Issue" goes to blank page or error page for only this challenge

https://www.freecodecamp.com/challenges/make-images-mobile-responsive
Issue Description
When attempting to report a possible bug for this challenge, I get only a blank page in Edge (Windows 10), or a Unicorn (This page is taking way too long to load) error on Chrome (Windows 10).  This has happened at various times today, in both browsers. I have tried this with other challenges and have no issue reaching the bug report page.
Windows 10
Edge and Chrome
Laptop
https://github.com/freecodecamp/freecodecamp/issues/new?&body=Challenge [Make Images Mobile Responsive](https%3A%2F%2Fwww.freecodecamp.com%2Fchallenges%2Fmake-images-mobile-responsive) has an issue.
User Agent is%3A <code>Mozilla%2F5.0 (Windows NT 10.0%3B Win64%3B x64) AppleWebKit%2F537.36 (KHTML%2C like Gecko) Chrome%2F55.0.2883.87 Safari%2F537.36<%2Fcode>.
Please describe how to reproduce this issue%2C and include links to screenshots if possible.

My code%3A
```html

<link href%3D"https%3A%2F%2Ffonts.googleapis.com%2Fcss%3Ffamily%3DLobster" rel%3D"stylesheet" type%3D"text%2Fcss">
<style>
  .red-text {
    color%3A red%3B
  }

  h2 {
    font-family%3A Lobster%2C Monospace%3B
  }

  p {
    font-size%3A 16px%3B
    font-family%3A Monospace%3B
  }

  .thick-green-border {
    border-color%3A green%3B
    border-width%3A 10px%3B
    border-style%3A solid%3B
    border-radius%3A 50%25%3B
  }

  .smaller-image {
    width%3A 100px%3B
  }
<%2Fstyle>

<div class%3D"container-fluid">
  <h2 class%3D"red-text">CatPhotoApp<%2Fh2>

  <p>Click here for <a href%3D"%23">cat photos<%2Fa>.<%2Fp>

  <a href%3D"%23"><img class%3D"smaller-image thick-green-border" src%3D"https%3A%2F%2Fbit.ly%2Ffcc-relaxing-cat" alt%3D"A cute orange cat lying on its back. "><%2Fa>

  <p>Things cats love%3A<%2Fp>
  <ul>
    <li>cat nip<%2Fli>
    <li>laser pointers<%2Fli>
    <li>lasagna<%2Fli>
  <%2Ful>
  <p>Top 3 things cats hate%3A<%2Fp>
  <ol>
    <li>flea treatment<%2Fli>
    <li>thunder<%2Fli>
    <li>other cats<%2Fli>
  <%2Fol>
  <form action%3D"%2Fsubmit-cat-photo">
    <label><input type%3D"radio" name%3D"indoor-outdoor"> Indoor<%2Flabel>
    <label><input type%3D"radio" name%3D"indoor-outdoor"> Outdoor<%2Flabel>
    <label><input type%3D"checkbox" name%3D"personality"> Loving<%2Flabel>
    <label><input type%3D"checkbox" name%3D"personality"> Lazy<%2Flabel>
    <label><input type%3D"checkbox" name%3D"personality"> Crazy<%2Flabel>
    <input type%3D"text" placeholder%3D"cat photo URL" required>
    <button type%3D"submit">Submit<%2Fbutton>
  <%2Fform>
<%2Fdiv>

```

@ElisabetK Hmmm, I'm unable to reproduce this issue, but then, I'm not on a Windows machine. You are talking about the "Create my Github Issue" button within the "Bug" modal, correct?
The link you posted seems to work, and takes you to a blank issue page.

@ElisabetK please provide some screenshots. Also please check that some browser add ons could be causing issue with page, for example we have had LastPass, AdBlocker etc. conflicting the challenge pages, in the past.

Closing, please comment with more information.


API endpoints response

Bug Report

I found a bug in the API
The following does not work: API endpoint /projects does not set status "OK" correctly
We need to set the status to fix it

Since this is a non-crucial problem and Spring does not offer a solution this won't get fixed.


Voice commands are no longer working

I installed version v9.63.06 on a Linux dedicated server and the voice commands are no longer working. The button still show the voice command menu correctly, but the characters don't speak and the chat won't show anything after the command is selected.
It happens in both the prebuilt version on our Steam group and the version I built myself from the source code with all default INI files.

My config is bad, it's not the mod fault. I'm sorry.


Error: 14 UNAVAILABLE: 403:Forbidden

Describe your environment

Operating System version: Ubuntu 20.04.1 LTS
Firebase SDK version: 11.4.1
Firebase Product: Firebase Admin
Node.js version: 18.13.0
NPM version: 8.19.3
(Used as a subdependency)
@google-cloud/firestore version: 6.4.2

Describe the problem
Steps to reproduce:
Our NodeJS backend, running ExpressJS and using Firebase Admin SDK has been running smoothly for 2 years. All of a sudden, on the morning of the 17th of Jan, our service was interupted by a seemignly routine Firebase call made by our baseModel. This causes all requests to fail until we reboot the backend. Since then, the error has happened randomly every 2-6+ hours with thousands of OK requests in between. The error seems to stem from the Firebase Admin SDK package/its depencies.
Relevant Code:
Error Log:
Jan 17 10:02:08 Error: 14 UNAVAILABLE: 403:Forbidden
Jan 17 10:02:08 at Object.callErrorFromStatus (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
Jan 17 10:02:08 at Object.onReceiveStatus (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:352:49)
Jan 17 10:02:08 at Object.onReceiveStatus (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)
Jan 17 10:02:08 at /app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:188:78
Jan 17 10:02:08 at processTicksAndRejections (internal/process/task_queues.js:82:9)
Jan 17 10:02:08 for call at
Jan 17 10:02:08 at ServiceClientImpl.makeServerStreamRequest (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:336:30)
Jan 17 10:02:08 at ServiceClientImpl.<anonymous> (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
Jan 17 10:02:08 at /app/node_modules/@google-cloud/firestore/build/src/v1/firestore_client.js:205:29
Jan 17 10:02:08 at /app/node_modules/google-gax/build/src/streamingCalls/streamingApiCaller.js:38:28
Jan 17 10:02:08 at /app/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
Jan 17 10:02:08 at Object.request (/app/node_modules/google-gax/build/src/streamingCalls/streaming.js:126:40)
Jan 17 10:02:08 at Timeout.makeRequest [as _onTimeout] (/app/node_modules/retry-request/index.js:139:28)
Jan 17 10:02:08 at listOnTimeout (internal/timers.js:531:17)
Jan 17 10:02:08 at processTimers (internal/timers.js:475:7)
Jan 17 10:02:08 Caused by: Error
Jan 17 10:02:08 at Firestore.getAll (/app/node_modules/@google-cloud/firestore/build/src/index.js:902:23)
Jan 17 10:02:08 at DocumentReference.get (/app/node_modules/@google-cloud/firestore/build/src/reference.js:211:32)
Jan 17 10:02:08 at Function.<anonymous> (/app/dist/models/baseModel.js:278:62)
Jan 17 10:02:08 at Generator.next (<anonymous>)
Jan 17 10:02:08 at /app/dist/models/baseModel.js:31:71
Jan 17 10:02:08 at new Promise (<anonymous>)
Jan 17 10:02:08 at __awaiter (/app/dist/models/baseModel.js:27:12)
Jan 17 10:02:08 at Function.findById (/app/dist/models/baseModel.js:274:16)
Jan 17 10:02:08 at /app/dist/routes/routeUtils.js:62:42
Jan 17 10:02:08 at Generator.next (<anonymous>) {
Jan 17 10:02:08 code: 14,
Jan 17 10:02:08 details: '403:Forbidden',
Jan 17 10:02:08 metadata: Metadata {
Jan 17 10:02:08 internalRepr: Map {
Jan 17 10:02:08 'content-type' => [Array],
Jan 17 10:02:08 'content-length' => [Array],
Jan 17 10:02:08 'date' => [Array],
Jan 17 10:02:08 'alt-svc' => [Array]
Jan 17 10:02:08 },
Jan 17 10:02:08 options: {}
Jan 17 10:02:08 }
Jan 17 10:02:08 }

Trickled down baseModel line from our code that does the Firebase fetch:
   static findById(id, showDeleted = false) {
        return __awaiter(this, void 0, void 0, function* () {
            try {
                if (!id)
                    throw new Error(`Invalid id provided for model ${this.name}`);
                const result = yield this.collection.doc(id).get();
                if (!result.exists)
                    return null;
                if (!showDeleted && result.data().isDeleted)
                    return null;
                return new this(Object.assign({}, result.data()));
            }
            catch (e) {
                console.error(e);
            }
        });
    }

Our package.json:
"dependencies": {
        "@sendgrid/mail": "^7.7.0",
        "axios": "^1.2.3",
        "cors": "^2.8.5",
        "crypto-js": "^4.1.1",
        "express": "^4.17.1",
        "firebase-admin": "^11.4.1",
        "lodash": "^4.17.21",
        "moment": "^2.29.4",
        "morgan": "^1.10.0",
        "nanoid": "^3.3.4",
        "stripe": "^11.6.0",
        "yup": "^0.32.11"
    },
    "devDependencies": {
        "@types/cors": "^2.8.13",
        "@types/crypto-js": "^4.1.1",
        "@types/express": "^4.17.15",
        "@types/lodash": "^4.14.191",
        "@types/morgan": "^1.9.4",
        "@types/node": "^18.11.18",
        "@types/yup": "^0.32.0",
        "@typescript-eslint/eslint-plugin": "^4.31.2",
        "@typescript-eslint/parser": "^4.31.2",
        "eslint": "^7.32.0",
        "nodemon": "^2.0.4",
        "ts-node": "^10.9.1",
        "typescript": "^4.9.4"
    }

Thanks for reporting. Looks like this issue may have also been reported through a support request. Can you confirm if that's true so we can coordinate? Thanks

Thank you for the prompt reply!
@MarkDuckworth Correct.
I am sorry if that was the wrong thing to do.
We've been getting desperate as the error has now been happpening almost hourly. Service is therefore interrupted quite regularly and has escalated on our side to the highest priority.
Cheers

I just wanted to confirm it is the same report, so that we can coordinate internally. Thank you.
I spoke with an engineer from the backend about the error code, but we're not yet sure what's causing it.
Can you double check the project ID you provided on the other support channel and send an update there if needed? Also, can you help us understand how often this call to Firestore is made successfully?

I have sent the project ID along through the support email thread.
This call to firestore is successfully made thousands of times inbetween getting the error.
I've also looked at the USAGE and it looks like we've average 21 Million+ of these READ requests per month and have been running essentially this configuration for about 2 years without ever seeing this error prior.

First encounter of this error was on Jan 17th at 5:02am [EST] while running a version of our backend that had not received a deploy in 10 days. (meaning nothing changed) On the 18th of Jan the error happened 3-4 times. Today, on the 19th of Jan, we've gotten the error a little over once per hour.

Important update:
This is now happening on another backend of ours that is running firebase-admin
The last deploy of this backend was from 6 weeks ago, with a non significant change. This has also been running for years without an issue prior.

Save New Duplicate & Edit Just Text Twitter
Jan 20 10:08:51 (node:1) UnhandledPromiseRejectionWarning: Error: 14 UNAVAILABLE: 403:Forbidden
Jan 20 10:08:51 at Object.callErrorFromStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call.js:30:26)
Jan 20 10:08:51 at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client.js:328:49)
Jan 20 10:08:51 at Object.onReceiveStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:304:181)
Jan 20 10:08:51 at Http2CallStream.outputStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:116:74)
Jan 20 10:08:51 at Http2CallStream.maybeOutputStatus (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:155:22)
Jan 20 10:08:51 at Http2CallStream.endCall (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:141:18)
Jan 20 10:08:51 at Http2CallStream.handleTrailers (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:266:14)
Jan 20 10:08:51 at ClientHttp2Stream.<anonymous> (/usr/src/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:315:26)
Jan 20 10:08:51 at ClientHttp2Stream.emit (events.js:200:13)
Jan 20 10:08:51 at emit (internal/http2/core.js:247:8)
Jan 20 10:08:51 Caused by: Error
Jan 20 10:08:51 at Query._get (/usr/src/app/node_modules/@google-cloud/firestore/build/src/reference.js:1485:23)
Jan 20 10:08:51 at Query.get (/usr/src/app/node_modules/@google-cloud/firestore/build/src/reference.js:1474:21)
Jan 20 10:08:51 at Function.find (/usr/src/app/dist/models/BaseModel.js:362:48)
Jan 20 10:08:51 at /usr/src/app/dist/app.js:43:27
Jan 20 10:08:51 at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
Jan 20 10:08:51 at next (/usr/src/app/node_modules/express/lib/router/route.js:137:13)
Jan 20 10:08:51 at Route.dispatch (/usr/src/app/node_modules/express/lib/router/route.js:112:3)
Jan 20 10:08:51 at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
Jan 20 10:08:51 at /usr/src/app/node_modules/express/lib/router/index.js:281:22
Jan 20 10:08:51 at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)
Jan 20 10:08:51 (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 6058)

Are you using App Check, or is that enabled for your Firebase app?
Also, can you tell me more about what happens after this first error? Are subsequent requests failing with the same error?

Thanks for the reply @MarkDuckworth .

We are not using App Check on any of the two projects that use firebase-admin that are currently crashing.

As soon as we get the error, any following requests will receive this error (and therefore, the backend logs become a spam wall of this error).

We have to restart the backend, which sometimes the error will happen at the first request, sometimes we'll be fine for 45 minutes.
(Maybe we there is a timeout or cooldown on your side that isn't reached when we reboot too quickly? or maybe it's random)

@iAMkVIN-S, I'm recommending that you try to resolve this through the support ticket you have. That will likely be preferred moving forward because it will allow you to share more information privately. There seem to be more eyes on that conversation as well. For us, it reduces duplicate work. If you disagree, please feel free to re-open this conversation.

@MarkDuckworth Thank you so much.
I've responded via the support request and will continue to do so there since the fix is unlikely to be done by our side.
I hope this thread is useful for others experiencing the same error to know that the team at Firebase/Google are taking the issue seriously.
Cheers

Describe your environment

Operating System version: Ubuntu 20.04.1 LTS
Firebase SDK version: 11.4.1
Firebase Product: Firebase Admin
Node.js version: 18.13.0
NPM version: 8.19.3
(Used as a subdependency)
@google-cloud/firestore version: 6.4.2

Describe the problem
Steps to reproduce:
Our NodeJS backend, running ExpressJS and using Firebase Admin SDK has been running smoothly for 2 years. All of a sudden, on the morning of the 17th of Jan, our service was interupted by a seemignly routine Firebase call made by our baseModel. This causes all requests to fail until we reboot the backend. Since then, the error has happened randomly every 2-6+ hours with thousands of OK requests in between. The error seems to stem from the Firebase Admin SDK package/its depencies.
Relevant Code:
Error Log:
Jan 17 10:02:08 Error: 14 UNAVAILABLE: 403:Forbidden
Jan 17 10:02:08 at Object.callErrorFromStatus (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
Jan 17 10:02:08 at Object.onReceiveStatus (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:352:49)
Jan 17 10:02:08 at Object.onReceiveStatus (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:328:181)
Jan 17 10:02:08 at /app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:188:78
Jan 17 10:02:08 at processTicksAndRejections (internal/process/task_queues.js:82:9)
Jan 17 10:02:08 for call at
Jan 17 10:02:08 at ServiceClientImpl.makeServerStreamRequest (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:336:30)
Jan 17 10:02:08 at ServiceClientImpl.<anonymous> (/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
Jan 17 10:02:08 at /app/node_modules/@google-cloud/firestore/build/src/v1/firestore_client.js:205:29
Jan 17 10:02:08 at /app/node_modules/google-gax/build/src/streamingCalls/streamingApiCaller.js:38:28
Jan 17 10:02:08 at /app/node_modules/google-gax/build/src/normalCalls/timeout.js:44:16
Jan 17 10:02:08 at Object.request (/app/node_modules/google-gax/build/src/streamingCalls/streaming.js:126:40)
Jan 17 10:02:08 at Timeout.makeRequest [as _onTimeout] (/app/node_modules/retry-request/index.js:139:28)
Jan 17 10:02:08 at listOnTimeout (internal/timers.js:531:17)
Jan 17 10:02:08 at processTimers (internal/timers.js:475:7)
Jan 17 10:02:08 Caused by: Error
Jan 17 10:02:08 at Firestore.getAll (/app/node_modules/@google-cloud/firestore/build/src/index.js:902:23)
Jan 17 10:02:08 at DocumentReference.get (/app/node_modules/@google-cloud/firestore/build/src/reference.js:211:32)
Jan 17 10:02:08 at Function.<anonymous> (/app/dist/models/baseModel.js:278:62)
Jan 17 10:02:08 at Generator.next (<anonymous>)
Jan 17 10:02:08 at /app/dist/models/baseModel.js:31:71
Jan 17 10:02:08 at new Promise (<anonymous>)
Jan 17 10:02:08 at __awaiter (/app/dist/models/baseModel.js:27:12)
Jan 17 10:02:08 at Function.findById (/app/dist/models/baseModel.js:274:16)
Jan 17 10:02:08 at /app/dist/routes/routeUtils.js:62:42
Jan 17 10:02:08 at Generator.next (<anonymous>) {
Jan 17 10:02:08 code: 14,
Jan 17 10:02:08 details: '403:Forbidden',
Jan 17 10:02:08 metadata: Metadata {
Jan 17 10:02:08 internalRepr: Map {
Jan 17 10:02:08 'content-type' => [Array],
Jan 17 10:02:08 'content-length' => [Array],
Jan 17 10:02:08 'date' => [Array],
Jan 17 10:02:08 'alt-svc' => [Array]
Jan 17 10:02:08 },
Jan 17 10:02:08 options: {}
Jan 17 10:02:08 }
Jan 17 10:02:08 }

Trickled down baseModel line from our code that does the Firebase fetch:
   static findById(id, showDeleted = false) {
        return __awaiter(this, void 0, void 0, function* () {
            try {
                if (!id)
                    throw new Error(`Invalid id provided for model ${this.name}`);
                const result = yield this.collection.doc(id).get();
                if (!result.exists)
                    return null;
                if (!showDeleted && result.data().isDeleted)
                    return null;
                return new this(Object.assign({}, result.data()));
            }
            catch (e) {
                console.error(e);
            }
        });
    }

Our package.json:
"dependencies": {
        "@sendgrid/mail": "^7.7.0",
        "axios": "^1.2.3",
        "cors": "^2.8.5",
        "crypto-js": "^4.1.1",
        "express": "^4.17.1",
        "firebase-admin": "^11.4.1",
        "lodash": "^4.17.21",
        "moment": "^2.29.4",
        "morgan": "^1.10.0",
        "nanoid": "^3.3.4",
        "stripe": "^11.6.0",
        "yup": "^0.32.11"
    },
    "devDependencies": {
        "@types/cors": "^2.8.13",
        "@types/crypto-js": "^4.1.1",
        "@types/express": "^4.17.15",
        "@types/lodash": "^4.14.191",
        "@types/morgan": "^1.9.4",
        "@types/node": "^18.11.18",
        "@types/yup": "^0.32.0",
        "@typescript-eslint/eslint-plugin": "^4.31.2",
        "@typescript-eslint/parser": "^4.31.2",
        "eslint": "^7.32.0",
        "nodemon": "^2.0.4",
        "ts-node": "^10.9.1",
        "typescript": "^4.9.4"
    }

I have the same issue. Please help us fix this.

Thanks for brining that to our attention @manoslive. I'll ping the internal thread and let them know it is affecting other users.
@iAMkVIN-S, are you still experiencing this?

@manoslive, can you tell us when this error started occurring for you?

I first noticed it on 01/17/23 at 1:57pm.
Also, I have issue starting it back up. Everytime I reboot i get the same error. Currently, we can't use our backend.
Here's some logs from my node.js backend:
(node:1) UnhandledPromiseRejectionWarning: Error: 14 UNAVAILABLE: 403:Forbidden
at Object.callErrorFromStatus (/usr/src/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
at Object.onReceiveStatus (/usr/src/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client.js:330:49)
at Object.onReceiveStatus (/usr/src/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
at /usr/src/app/node_modules/google-gax/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
at processTicksAndRejections (internal/process/task_queues.js:82:9)
Caused by: Error
at Query._get (/usr/src/app/node_modules/@google-cloud/firestore/build/src/reference.js:1485:23)
at Query.get (/usr/src/app/node_modules/@google-cloud/firestore/build/src/reference.js:1474:21)
at Function.find (/usr/src/app/dist/api/requests.js:420:35)
at /usr/src/app/dist/app.js:43:27
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
at next (/usr/src/app/node_modules/express/lib/router/route.js:137:13)
at Route.dispatch (/usr/src/app/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (/usr/src/app/node_modules/express/lib/router/layer.js:95:5)
at /usr/src/app/node_modules/express/lib/router/index.js:281:22
at Function.process_params (/usr/src/app/node_modules/express/lib/router/index.js:335:12)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 26)

@MarkDuckworth We are very much still getting the issue. This has become mission critical as its impacting our clients businesses with so much downtime.
Thankfully our backend does reboot when we get the error and sometimes get away with a couple hundreds thousand OK requests before we randomly get the error again, though not always on the first go.
Since Saturday Jan 21st in the early AM, we've logged everytime a reboot was necessary due to the error:
266 times in 96 hours.  How did we process 250 Million + requests over the last 2 years from this one backend alone, and without making any changes, woke up on Jan 17th to non-stop crashing.
 I communicated this to the backend team. @manoslive, if you can provide a project ID, that will help them diagnose. You may want to create a support ticket where you can provide your project ID. In that ticket reference internal issue b/266097991, so we can link your info to the right place.

@MarkDuckworth / @dconeybe - We've added another 24hrs without a crash on any of the projects from Idea 1.
It sounds like we have our temporary fix until a permanent solution on the Firebase side!

That's great news! Thanks for the update and for trying out the ideas. I'm glad we found something to get you unblocked.
One thing, though, is that since we cannot reproduce this issue ourselves, we may need you to reproduce it for us in order to investigate to find the root cause. Maybe consider how you might go about reproducing this problem in a way that doesn't grind your business to a halt.
We will get back to you with suggested next steps, either via this GitHub issue or via your support ticket.

@iAMkVIN-S Count me as one of the 1.1M weekly installers.  My cloud functions have been running smoothly for months, then yesterday I started receiving 403 Forbidden error messages with all of my functions that make calls to an outside API.  This is a critical issue as the cloud functions are core to my application.
Below is one of my functions that began failing yesterday.  I first have to retrieve a token from my firestore and then make the request to the outside API (Finicity).  I have confirmed the url and parameters are all generated and passed correctly in the logs of each failed attempt. Once the url is received, it is passed back to the end user.
It's important to note that when running my functions from a firebase shell environment, everything runs smoothly and the calls succeed .  However, when I invoke the functions from my application as an end user, I receive the 403 Forbidden error message.  To that end, I also have a function that runs every 80 minutes making a call to an outside api that, again runs fine when I invoke it locally, but when it is called automatically, it just started failing yesterday nearly every time.
Sample Function
`    exports.refreshPromiseCustomUrl = functions.https.onCall(async(data,context) => {
   const customerId = data.customerId;

    const refreshedUrl = await admin.firestore().collection('SampleCollection').doc('sampleDoc').get().then(doc => {
      const promises = [];
      promises.push(doc.data().accessToken);
      return Promise.all(promises)
    }).then(async token => {
      const secondPromise = []

      const dataForUrl = JSON.stringify({"partnerId":"MyPartnerId","customerId":customerId});
      const configForUrl = {method: 'post',url: 'https://api.finicity.com/connect/v2/generate',
      headers: {'Accept': 'application/json',
           'Content-Type': 'application/json',
          'Finicity-App-Key':'MyAppKey',
          'Finicity-App-Token': token[0],
          },
        data : dataForUrl
      };

      const theRequestUrl = await axios(configForUrl).then(res => {
        return res.data.link
      }).catch(e => {
        functions.logger.error('Error reaching out to finicity, the token: ', token[0], 'customer Id: ', customerId, 'error message: ', e)
        return ''
      })

      secondPromise.push(theRequestUrl)
      return Promise.all(secondPromise);

    }).catch(err => {
      functions.logger.error('error  ', err)
      return '';
    })

  return{theUrl: refreshedUrl[0]}

  })`

Error Message abbreviated
[Request failed with status code 403 ](response: { status: 403, statusText: 'Forbidden',)
package.json
{   "name": "functions",   "description": "Cloud Functions for Firebase",   "scripts": {     "lint": "eslint .",     "serve": "firebase serve --only functions",     "shell": "firebase functions:shell",     "start": "npm run shell",     "deploy": "firebase deploy --only functions",     "logs": "firebase functions:log"   },   "engines": {     "node": "16"   },   "dependencies": {     "@sendgrid/mail": "^6.5.5",     "axios": "^1.3.2",     "cors": "^2.8.5",     "firebase-admin": "^11.5.0",     "firebase-functions": "^4.2.1",     "googleapis": "^107.0.0",     "moment": "2.26.0",     "nodemailer": "^2.5.0",     "stripe": "^6.36.0",     "uuid": "^3.3.2"   },   "devDependencies": {     "eslint": "^5.12.0",     "eslint-plugin-promise": "^4.0.1"   },   "private": true }
I feel I have exhausted my options at this point.  I have updated all of my dependencies, tried variations of my functions, checked all of my parameters for the api call are set correctly, confirmed with Finicity that nothing has changed on their end, confirmed user access to my functions, tried using different function names, tried increasing the memory and instances allocated to each function...nothing has worked.
I will note, however, that randomly every once in a while a call will go through successfully.  No idea why.  I am absolutely stuck at this point.
Just hoping this issue gets resolved as soon as possible.

Hey @PL47productions
I'm not sure if your error is exactly the same as we were getting (Error: 14 UNAVAILABLE: 403:Forbidden)
However, did you try Idea 1 from @dconeybe above?
Where you Initialize Firebase Admin, you can pass in a param that tells the SDK to use the REST API when the request is not a snapshot.
What we had before the change:
admin.initializeApp({
    credential: <Your Credentials JSON>,
});

What we had after the change:
import { initializeApp } from "firebase-admin/app";
import { initializeFirestore } from "firebase-admin/firestore";

const app = initializeApp({
    credential: <Your Credentials JSON>,
});
initializeFirestore(app, { preferRest: true });

If your error is the same as ours, this will fix it temporarily, since I do not see a snapshot in your above request.

@PL47productions, thanks for letting us know that this is also impacting you. I updated the internal conversation with this information.
As @iAMkVIN-S mentioned, using preferRest may provide a workaround. You can consider using preferRest if you don't use onSnapshot(...) in your application.

@iAMkVIN-S Thanks and yes, not exactly the same issue...but in the same ball park.    Unfortunately your suggestion didn't do the trick for me.
@MarkDuckworth Thanks, this is a frustrating one for sure.  I'm testing the unirest library out instead of axios to see if that helps at all.

@PL47productions Could you reproduce with debug logging enabled and provide the logs via a https://gist.github.com/ link? If you're not comfortable sharing the logs publicly, please open a support ticket at https://firebase.google.com/support where the logs can be submitted privately.
First, enable debug logging by registering a logging function (you can probably just use console.log):
https://cloud.google.com/nodejs/docs/reference/firestore/latest/overview#functions
Second, enable verbose GRPC logging by setting GRPC_VERBOSITY set to debug:
https://github.com/grpc/grpc-node/blob/master/TROUBLESHOOTING.md

https://gist.github.com/PL47productions/87422a54807d6280462920f7e0cb2d01
The above link is the error output I receive for the function below that was running fine for months.
`  exports.refreshFinToken = functions.pubsub.schedule('every 80 minutes').onRun(async (context) => {
  const data = JSON.stringify({"partnerId":"MYID", "partnerSecret":"MYSECRET"});

   let config = {
     method: 'post',
     url: 'https://api.finicity.com/aggregation/v2/partners/authentication',
     headers: {
       'Accept': 'application/json',
       'Content-Type': 'application/json',
       'Finicity-App-Key':'MYKEY'
     },
     data : data
   };

 const updatedToken = await axios(config).then(response =>  {
     const promises = [];
     promises.push(response.data.token);

       return Promise.all(promises);

     }).catch(e => {
      functions.logger.error('Error retrieving token from finicity, error message: ', e)

      return '';
     }).then(async token => {
       const secondPromise = [];
       secondPromise.push(token[0])
       let tokenString = String(token[0])

       if(typeof token[0] === 'undefined'){
        sendTokenError();
        //handleRetryGetAccessToken()
        return Promise.all(secondPromise);
       }
       else {

        let updateResult =  await admin.firestore().collection('COLLECTION').doc('DOC').update({accessToken: tokenString}).then((res) => {

          return 'Successfully updated';
           }).catch((e) => {
            functions.logger.error('Error adding token to data base, token: ',token[0], 'error: ', e)

            return 'Error adding token to database';
           })

           return Promise.all(updateResult)
       }

     }).catch((e) => {

       return 'Encountered error'
     })

   return{success: updatedToken}

 })`

I'll try enabling verbose logging and send those along.

@PL47productions The error message from your logs look like the problem is connecting to https://api.finicity.com. This doesn't appear to be related to Firestore (this GitHub repository is for Firestore). Are you perhaps suggesting that this is an issue beyond Firestore and also affects other, unrelated products? That would make sense to me because otherwise we would be experiencing a massive Firestore outage rather than one or two customers experiencing this "403: Forbidden" error. Maybe there is some common infrastructure that coincidentally both your traffic and OP's traffic go through which cause this error and the traffic never even makes it to Firestore servers. That would also make sense because the Firestore backend engineers could not find any trace of OP's "403: Forbidden" errors in our server logs.

@dconeybe I don't think our error and @PL47productions are the same.
First, the error log is different.
Second, our failing requests were for a simple .get() from the Firebase Admin SDK. Without any external API in play.
Third, we've had 0 errors since adding the preferRest param in the Initialize.
We still are building up a sandbox backend to reproduce the error but with the additional logs requested in the support ticket. If time is on our side, we'll hopefully have that to you today.

Apologies, our issues are different indeed...mine is related to errors within firebase functions, not with firestore.  I opened a support ticket with firebase and in the meantime, I added a region code to all of my calls which seems to have fixed the issue...or it just randomly started working again just as randomly as it stopped.


nullable api schema fields

https://github.com/prisma/prisma/issues/3621

I believe it's fixable with piotrwitek/utility-types.


Use subimage versions during texture update

This PR updates the texture upload logic similar to #5758.
Updating the data of an existing gl-side texture is relatively slow using gl.texImage2D because a new gl-side texture is created every time. It's faster to use gl.texSubImage2D instead, which will update the existing texture data.
This PR requires thorough testing, especially with editor projects, to ensure edge cases are correctly covered.
Testing Notes:

each creation and update path needs testing: 2d and cubemap, pixel data and browser interface sources

This is great @slimbuck, also I did not know this, so I've learned something new 👏


new

@AnasXbouali Why do you close and reopen the PR so many times?
If you are working on the PR and you want to block merging you can click on "convert to draft" on top of the page.

@ocots I had a problem during the build, the issue was to add the NonlinearSolve package.

When you are ready to merge, you can click on "Ready for review" at the bottom of the page.


SSL Error. Failed to validate the certificate chain.

TAG: X509Util - Failed to validate the certificate chain, error: java.security.cert.CertPathValidatorException: Trust anchor for certification path not found.
Any ideas?

Thanks :) This worked for me too.

I know this is closed, but for posterity... this answer is not a good idea. It basically disables SSL and makes it almost pointless to be using it. This error points to an issue with your server SSL configuration, normally the lack of an intermediate certificate which helps the browser (webView, etc.) verify a certificate chain, from your personal certificate, to the certificate authorities that your browser will recognize as valid.
Here is a short article that describes the issue: http://steveliles.github.io/android_ssl_certificate_not_trusted.html
In terms of solving it, that will be up to you and what server/cert you are using. For instance, this is what we had to do, using nginx and startssl:
http://www.westphahl.net/blog/2012/01/03/setting-up-https-with-nginx-and-startssl/

yeah i think it's obvious that you wouldn't wanna use this solution in a production environment but it's perfectly fine to disable ssl on a development environment.

Oh yeah, fair enough! Just wanted to make sure no one used this in production thinking they were still secure!

@allthetime though related to other project (working on a corodva app) your comment really helped me understand what is going on and why my application no connecting after I made a release version (see here). And thanks @Gerst20051 for opening this issue ;)

A better solution that won't get you man in the middled.
`  @Override
public void onReceivedSslError(WebView view, SslErrorHandler handler, SslError error) {
SslCertificate sslCertificateServer = error.getCertificate();
Certificate pinnedCert = getCertificateForRawResource(R.raw.your_cert, mContext);
Certificate serverCert = convertSSLCertificateToCertificate(sslCertificateServer);
    if(pinnedCert.equals(serverCert)) {
        handler.proceed();
    } else {
        super.onReceivedSslError(view, handler, error);
    }
}

public static Certificate getCertificateForRawResource(int resourceId, Context context) {
    CertificateFactory cf = null;
    Certificate ca = null;
    Resources resources = context.getResources();
    InputStream caInput = resources.openRawResource(resourceId);

    try {
        cf = CertificateFactory.getInstance("X.509");
        ca = cf.generateCertificate(caInput);
    } catch (CertificateException e) {
        Log.e(TAG, "exception", e);
    } finally {
        try {
            caInput.close();
        } catch (IOException e) {
            Log.e(TAG, "exception", e);
        }
    }

    return ca;
}

public static Certificate convertSSLCertificateToCertificate(SslCertificate sslCertificate) {
    CertificateFactory cf = null;
    Certificate certificate = null;
    Bundle bundle = sslCertificate.saveState(sslCertificate);
    byte[] bytes = bundle.getByteArray("x509-certificate");

    if (bytes != null) {
        try {
            CertificateFactory certFactory = CertificateFactory.getInstance("X.509");
            Certificate cert = certFactory.generateCertificate(new ByteArrayInputStream(bytes));
            certificate = cert;
        } catch (CertificateException e) {
            Log.e(TAG, "exception", e);
        }
    }

    return certificate;
}

`