WIP

Summary of the changes / Why this improves CrateDB
Checklist

[ ] Added an entry in the latest docs/appendices/release-notes/<x.y.0>.rst for user facing changes
[ ] Updated documentation & sql_features table for user facing changes
[ ] Touched code is covered by tests
[ ] CLA is signed
[ ] This does not contain breaking changes, or if it does:

It is released within a major release
It is recorded in the latest docs/appendices/release-notes/<x.y.0>.rst
It was marked as deprecated in an earlier release if possible
You've thought about the consequences and other components are adapted
(E.g. AdminUI)

Measure the impact of 1a3144e79e1213f78afadcd324e38a5d003ae7b3 with array_length spec:
# Results (server side duration in ms)
V1: 5.9.0-57493750b5c4b9f3b5c4c831184a9fdaad49da8d
V2: 5.9.0-1a3144e79e1213f78afadcd324e38a5d003ae7b3

Q: select count(*) from t_arrays_biased_empty where xs = []
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      236.474 ±  516.642 |    174.894 |    179.742 |    183.047 |   7218.785 |
|   V2    |      253.184 ±  455.518 |    192.417 |    199.306 |    205.958 |   6389.766 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +   6.83%                           +  10.32%
There is a 26.83% probability that the observed difference is not random, and the best estimate of that difference is 6.83%
The test has no statistical significance

Q: select count(*) from t_arrays_biased_5_elements where xs = []
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |       39.225 ±  257.959 |     14.188 |     15.209 |     17.493 |   3641.926 |
|   V2    |       43.026 ±  247.901 |     18.294 |     19.770 |     26.988 |   3510.294 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +   9.24%                           +  26.08%
There is a 11.94% probability that the observed difference is not random, and the best estimate of that difference is 9.24%
The test has no statistical significance

Q: select count(*) from t_arrays_biased_empty where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        1.062 ±    1.167 |      0.699 |      0.929 |      1.036 |     17.157 |
|   V2    |        1.321 ±    1.624 |      0.754 |      1.007 |      1.350 |     23.108 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  21.78%                           +   8.07%
There is a 93.28% probability that the observed difference is not random, and the best estimate of that difference is 21.78%
The test has no statistical significance

Q: select count(*) from t_arrays_biased_empty where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        7.787 ±    7.178 |      4.005 |      7.415 |      7.743 |     67.818 |
|   V2    |        0.892 ±    0.199 |      0.693 |      0.843 |      0.937 |      2.604 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 158.88%                           - 159.15%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 158.88%
The test has statistical significance

Q: select count(*) from t_arrays_biased_empty where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      227.518 ±  169.992 |    193.759 |    198.936 |    203.215 |   1309.401 |
|   V2    |        0.748 ±    0.085 |      0.619 |      0.726 |      0.793 |      1.011 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 198.69%                           - 198.55%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 198.69%
The test has statistical significance

Q: select count(*) from t_arrays_biased_one_element where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        0.588 ±    0.108 |      0.428 |      0.572 |      0.629 |      1.065 |
|   V2    |        0.692 ±    0.106 |      0.548 |      0.662 |      0.732 |      1.219 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  16.26%                           +  14.61%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 16.26%
The test has statistical significance

Q: select count(*) from t_arrays_biased_one_element where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |       12.054 ±    2.250 |      7.502 |     13.315 |     13.583 |     14.634 |
|   V2    |        0.640 ±    0.078 |      0.509 |      0.623 |      0.674 |      1.233 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 179.82%                           - 182.13%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 179.82%
The test has statistical significance

Q: select count(*) from t_arrays_biased_one_element where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      170.430 ±  233.176 |    126.072 |    130.162 |    135.343 |   1687.026 |
|   V2    |        0.594 ±    0.082 |      0.430 |      0.581 |      0.629 |      0.901 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 198.61%                           - 198.22%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 198.61%
The test has statistical significance

Q: select count(*) from t_arrays_biased_5_elements where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        0.417 ±    0.056 |      0.322 |      0.406 |      0.436 |      0.695 |
|   V2    |        0.540 ±    0.096 |      0.390 |      0.519 |      0.570 |      1.084 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  25.65%                           +  24.46%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 25.65%
The test has statistical significance

Q: select count(*) from t_arrays_biased_5_elements where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |       11.783 ±    2.327 |      7.401 |     13.220 |     13.374 |     15.459 |
|   V2    |        0.490 ±    0.054 |      0.370 |      0.479 |      0.504 |      0.732 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 184.04%                           - 186.03%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 184.04%
The test has statistical significance

Q: select count(*) from t_arrays_biased_5_elements where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      331.265 ±  548.015 |    231.593 |    238.825 |    249.162 |   3814.315 |
|   V2    |        0.519 ±    0.074 |      0.406 |      0.510 |      0.552 |      0.841 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 199.37%                           - 199.15%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 199.37%
The test has statistical significance

Q: select count(*) from t_arrays_even where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        0.338 ±    0.053 |      0.262 |      0.326 |      0.354 |      0.668 |
|   V2    |        0.527 ±    0.289 |      0.396 |      0.489 |      0.534 |      4.456 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  43.70%                           +  39.86%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 43.70%
The test has statistical significance

Q: select count(*) from t_arrays_even where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |       11.276 ±    1.842 |      7.002 |     12.103 |     12.202 |     15.494 |
|   V2    |        0.596 ±    0.073 |      0.494 |      0.579 |      0.619 |      1.141 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 179.92%                           - 181.73%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 179.92%
The test has statistical significance

Q: select count(*) from t_arrays_even where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      145.338 ±  284.231 |     92.245 |     94.615 |    103.531 |   1982.475 |
|   V2    |        0.481 ±    0.063 |      0.369 |      0.470 |      0.506 |      0.713 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 198.68%                           - 198.02%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 198.68%
The test has statistical significance

System/JVM Metrics (durations in ms, byte-values in MB)
    |    YOUNG GC            |       OLD GC           |      HEAP         |     ALLOC
    |  cnt      avg      max |  cnt      avg      max |  initial     used |     rate      total
 V1 |  104     2.62     2.52 |    1    39.16    39.16 |     4295      266 |  1121.90     269723
 V2 |   37     3.52     3.37 |    1    47.43    47.43 |     4295      224 |  1544.70      94679

Perf impact of 20d4b5b53262c01061d8c810f3cc177baa8178fe measured with array_length spec:
# Results (server side duration in ms)
V1: 5.9.0-57493750b5c4b9f3b5c4c831184a9fdaad49da8d
V2: 5.9.0-20d4b5b53262c01061d8c810f3cc177baa8178fe

Q: select count(*) from t_arrays_biased_empty where xs = []
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      248.538 ±  444.596 |    192.682 |    198.165 |    201.543 |   6211.352 |
|   V2    |      105.019 ±  521.125 |     43.194 |     47.064 |     56.153 |   7156.495 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               -  81.19%                           - 123.23%
There is a 99.68% probability that the observed difference is not random, and the best estimate of that difference is 81.19%
The test has statistical significance

Q: select count(*) from t_arrays_biased_5_elements where xs = []
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |       42.787 ±  244.342 |     17.359 |     19.609 |     23.837 |   3455.423 |
|   V2    |       55.179 ±  248.202 |     29.700 |     30.786 |     36.160 |   3527.541 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  25.30%                           +  44.36%
There is a 38.49% probability that the observed difference is not random, and the best estimate of that difference is 25.30%
The test has no statistical significance

Q: select count(*) from t_arrays_biased_empty where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        1.050 ±    1.152 |      0.748 |      0.941 |      1.029 |     17.120 |
|   V2    |        1.261 ±    1.955 |      0.778 |      1.035 |      1.180 |     28.287 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  18.23%                           +   9.52%
There is a 80.99% probability that the observed difference is not random, and the best estimate of that difference is 18.23%
The test has no statistical significance

Q: select count(*) from t_arrays_biased_empty where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        7.434 ±    3.297 |      4.350 |      7.798 |      8.041 |     50.398 |
|   V2    |        0.943 ±    0.147 |      0.714 |      0.920 |      0.993 |      2.258 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 154.96%                           - 157.77%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 154.96%
The test has statistical significance

Q: select count(*) from t_arrays_biased_empty where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      244.194 ±  159.817 |    213.150 |    217.842 |    220.155 |   1274.177 |
|   V2    |        0.765 ±    0.095 |      0.622 |      0.734 |      0.806 |      1.113 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 198.75%                           - 198.66%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 198.75%
The test has statistical significance

Q: select count(*) from t_arrays_biased_one_element where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        0.548 ±    0.109 |      0.411 |      0.518 |      0.568 |      1.047 |
|   V2    |        0.727 ±    0.127 |      0.566 |      0.694 |      0.762 |      1.271 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  28.09%                           +  29.02%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 28.09%
The test has statistical significance

Q: select count(*) from t_arrays_biased_one_element where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |       12.993 ±    2.630 |      7.756 |     14.605 |     14.758 |     18.806 |
|   V2    |        0.716 ±    0.201 |      0.523 |      0.691 |      0.738 |      3.241 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 179.11%                           - 181.93%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 179.11%
The test has statistical significance

Q: select count(*) from t_arrays_biased_one_element where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      272.661 ±  210.638 |    230.029 |    235.571 |    242.777 |   1628.626 |
|   V2    |        0.621 ±    0.086 |      0.490 |      0.599 |      0.660 |      1.017 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 199.09%                           - 198.99%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 199.09%
The test has statistical significance

Q: select count(*) from t_arrays_biased_5_elements where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        0.413 ±    0.051 |      0.320 |      0.402 |      0.437 |      0.658 |
|   V2    |        0.560 ±    0.082 |      0.432 |      0.546 |      0.592 |      1.078 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  30.13%                           +  30.34%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 30.13%
The test has statistical significance

Q: select count(*) from t_arrays_biased_5_elements where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |       12.335 ±    2.878 |      7.623 |     14.445 |     14.627 |     15.017 |
|   V2    |        0.490 ±    0.054 |      0.402 |      0.475 |      0.510 |      0.728 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 184.73%                           - 187.27%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 184.73%
The test has statistical significance

Q: select count(*) from t_arrays_biased_5_elements where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      389.928 ±  524.957 |    294.262 |    301.259 |    312.801 |   3719.954 |
|   V2    |        0.497 ±    0.070 |      0.392 |      0.480 |      0.516 |      0.788 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 199.49%                           - 199.36%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 199.49%
The test has statistical significance

Q: select count(*) from t_arrays_even where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        0.368 ±    0.164 |      0.261 |      0.346 |      0.384 |      2.528 |
|   V2    |        0.521 ±    0.185 |      0.376 |      0.492 |      0.531 |      2.876 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  34.31%                           +  34.97%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 34.31%
The test has statistical significance

Q: select count(*) from t_arrays_even where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |       11.749 ±    2.180 |      7.413 |     13.089 |     13.209 |     15.282 |
|   V2    |        0.519 ±    0.068 |      0.400 |      0.507 |      0.564 |      0.825 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 183.07%                           - 185.09%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 183.07%
The test has statistical significance

Q: select count(*) from t_arrays_even where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      203.714 ±  267.321 |    150.563 |    155.665 |    168.250 |   1884.747 |
|   V2    |        0.476 ±    0.053 |      0.379 |      0.465 |      0.498 |      0.692 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 199.07%                           - 198.81%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 199.07%
The test has statistical significance

System/JVM Metrics (durations in ms, byte-values in MB)
    |    YOUNG GC            |       OLD GC           |      HEAP         |     ALLOC
    |  cnt      avg      max |  cnt      avg      max |  initial     used |     rate      total
 V1 |  109     2.37     2.52 |    0     0.00     0.00 |     4295      269 |  1300.39     279256
 V2 |   13     4.25     3.96 |    1    42.91    42.91 |     4295      268 |  1083.70      37801

TODO:

An array length field is indexed only not visible via TableInfo or as a Reference.
Should it be mapped to oid?

An array length field is indexed only not visible via TableInfo or as a Reference.

we should be able to drop the array length field when the array is dropped and added when array is added via create or alter or added dynamically. I think we need ArrayReference extending SimpleReference.
I will focus on fixing array_length without degrading performance first.

Did not split up the the commit since it seemed small enough, please let me know. I could split into 1) array_length fix, 2) introducing array length index 3) array_length utilizing array length index.

Please review this PR, @BaurzhanSakhariev , @matriv

I don't quite understand why some cases still have performance regression:
select count(*) from t_arrays_even where array_length(xs, 1) > 0 ( +34%)
select count(*) from t_arrays_biased_one_element where array_length(xs, 1) > 0 (+28)
and some others.
Even if buggy-but-fast version on master uses some fast Lucene query, we also use Lucene queries here.
We already know length, it's indexed  - we are supposed to get either performance improvement or at least get result with  "fast but buggy". Why it's slower?

I don't quite understand why some cases still have performance regression:
select count(*) from t_arrays_even where array_length(xs, 1) > 0 ( +34%) select count(*) from t_arrays_biased_one_element where array_length(xs, 1) > 0 (+28)
and some others.
Even if buggy-but-fast version on master uses some fast Lucene query, we also use Lucene queries here.
We already know length, it's indexed - we are supposed to get either performance improvement or at least get result aligned with "fast but buggy". Why it's slower?

I think from the measurement you are referring to are all > 0 (= [] is just flaky there was no change to impact on it) and it uses FieldExistsQuery. I just assumed that we cannot beat that query with something that actually counts.

Do I get it right, that all changes in data types and Indexers are boilerplate
to pass Version.CREATED to  the ArrayIndexer?
I wonder if we can avoid so many unrelated changes by doing smth similar to https://github.com/crate/crate/commit/a06b67d80353cdc281010456ebdc201e60406ffb
ie introduce another ValueIndexer.indexValue  which by default falls back to existing behavior for all value indexers and override it for ArrayIndexer. Then we will have to change ony "top level" callers -  Indexer/TranslogIndexer/ObjectIndexer (and maybe DynamicIndexer)

Do I get it right, that all changes in data types and Indexers are boilerplate to pass Version.CREATED to the ArrayIndexer?
I wonder if we can avoid so many unrelated changes by doing smth similar to a06b67d
ie introduce another ValueIndexer.indexValue  which by default falls back to existing behavior for all value indexers and override it for ArrayIndexer. Then we will have to change ony "top level" callers - Indexer/TranslogIndexer/ObjectIndexer (and maybe DynamicIndexer)

Sorry another quick question, wouldn't it be better to fix ValueIndexer instances to a specific table version? I think it is also better to reduce the number of args passed to indexValue method which could potentially called large number of times.

I think https://github.com/crate/crate/pull/16329/commits/07f5431d09c890c2c7c5ad188ae281f348fba1f6 still has too many changes to propagate version created to ArrayIndexer.
I actually meant to add version to the existing ValueInxeder.indexValue and add extra logic  (adding new filed) only for  ArrayIndexer
default void indexValue(
        @Nullable T value,
        @Nullable String storageIdentLeafName,
        XContentBuilder xcontentBuilder,
        Consumer<? super IndexableField> addField,
        Synthetics synthetics,
        Map<ColumnIdent, ColumnConstraint> toValidate,
        Version versionCreated //  ----> New parameter to the existing function
    ) throws IOException {
        if (storageIdentLeafName != null) {
            xcontentBuilder.field(storageIdentLeafName);
        }
        // common logic for all indexers,
        indexValue(value, xcontentBuilder, addField, synthetics, toValidate);
        if (this instanceof ArrayIndexer<?>) {
            // ugly instance of can be replaced by boolean ValueIndexer.isArrayIndexer()
            // with default impl-n returning false and only ArrayIndexer overwriting it to true
           // but you decide, can keep insatnceof

            // extra version dependant logic, attention, don;'t forget to move such logic out from ArrayIndexer.indexValue.
            // Instead, we keep it as is to be able to first call "common indexing" and than add extra logic.
            if (versionCreated.onOrAfter(Version.V_5_9_0)) {
                if (value == null) {
                    // map 'null' to '_array_length_ = -1'
                    addField.accept(new IntField(arrayLengthFieldName, -1, Field.Store.NO));
                } else {
                    // map '[]' to '_array_length_ = 0'
                    // map '[null]' to '_array_length_ = 1'
                    addField.accept(new IntField(arrayLengthFieldName, values.size(), Field.Store.NO));
                }
            }
        }
    }

// common logic for all indexers,
indexValue(value, xcontentBuilder, addField, synthetics, toValidate);
if (this instanceof ArrayIndexer<?>) {
  ...
}

It seems like it does not allow us(or makes it difficult) from removing or altering logic but only allow adding extras.

It seems like it does not allow us(or makes it difficult) from removing or altering logic but only allow adding extras.

Sorry, I didn't get, it doesn't work at all or it works but can have some bad consequences for future changes?
Either way, it's fine to keep your current version.
My only goal was to reduce boilerplate required to pass Version to a concrete ValueIndexer implementation but if it brings more problems than it solves then no need to do it.

I am working on a change to the API here that will mean valueindexer takes an IndexDocumentBuilder object, rather than an ever-telescoping list of parameters, and we can add the Version to that instead.  Can this wait for another day?

Initially tried to index _array_length_ of null as -1 but simple unindexing would be better since we can utilize FieldExistQuery on _array_length_ field for is null filtering. I will re-measure the perf once the refactoring on top of Alan's recent commit is done.

# Results (server side duration in ms)
V1: 5.9.0-a88951a72dcd148777a9346a0cc899d220c7814f
V2: 5.9.0-5f88374d9ee94ee62547dec9f032a42c4e0211ce

Q: select count(*) from t_arrays_biased_empty where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        1.052 ±    1.128 |      0.696 |      0.908 |      1.069 |     16.679 |
|   V2    |        1.198 ±    1.447 |      0.811 |      1.046 |      1.172 |     21.204 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  12.97%                           +  14.13%
There is a 73.87% probability that the observed difference is not random, and the best estimate of that difference is 12.97%
The test has no statistical significance

Q: select count(*) from t_arrays_biased_empty where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        7.496 ±    2.946 |      4.575 |      8.022 |      8.236 |     43.820 |
|   V2    |        0.860 ±    0.138 |      0.684 |      0.832 |      0.899 |      2.142 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 158.83%                           - 162.41%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 158.83%
The test has statistical significance

Q: select count(*) from t_arrays_biased_empty where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      195.955 ±  156.168 |    161.240 |    168.463 |    174.368 |   1240.158 |
|   V2    |        0.771 ±    0.296 |      0.594 |      0.725 |      0.781 |      4.330 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 198.43%                           - 198.29%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 198.43%
The test has statistical significance

Q: select count(*) from t_arrays_biased_one_element where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        0.622 ±    0.152 |      0.417 |      0.579 |      0.658 |      1.631 |
|   V2    |        0.686 ±    0.099 |      0.545 |      0.658 |      0.721 |      1.045 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +   9.83%                           +  12.72%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 9.83%
The test has statistical significance

Q: select count(*) from t_arrays_biased_one_element where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |       13.105 ±    2.616 |      8.007 |     14.695 |     14.874 |     18.103 |
|   V2    |        0.609 ±    0.063 |      0.510 |      0.596 |      0.630 |      0.935 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 182.23%                           - 184.41%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 182.23%
The test has statistical significance

Q: select count(*) from t_arrays_biased_one_element where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      213.422 ±  217.097 |    166.743 |    174.440 |    184.894 |   1590.989 |
|   V2    |        0.587 ±    0.088 |      0.441 |      0.576 |      0.614 |      0.939 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 198.90%                           - 198.68%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 198.90%
The test has statistical significance

Q: select count(*) from t_arrays_biased_5_elements where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        0.426 ±    0.076 |      0.317 |      0.400 |      0.455 |      0.754 |
|   V2    |        0.557 ±    0.100 |      0.421 |      0.539 |      0.599 |      1.086 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  26.81%                           +  29.51%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 26.81%
The test has statistical significance

Q: select count(*) from t_arrays_biased_5_elements where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |       12.627 ±    2.617 |      7.877 |     14.334 |     14.534 |     15.485 |
|   V2    |        0.485 ±    0.050 |      0.405 |      0.475 |      0.508 |      0.708 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 185.21%                           - 187.16%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 185.21%
The test has statistical significance

Q: select count(*) from t_arrays_biased_5_elements where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      606.326 ±  537.023 |    494.242 |    515.847 |    532.394 |   4623.967 |
|   V2    |        0.481 ±    0.049 |      0.387 |      0.470 |      0.500 |      0.708 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 199.68%                           - 199.64%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 199.68%
The test has statistical significance

Q: select count(*) from t_arrays_even where array_length(xs, 1) > 0
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |        0.354 ±    0.047 |      0.276 |      0.349 |      0.364 |      0.581 |
|   V2    |        0.468 ±    0.058 |      0.376 |      0.454 |      0.495 |      0.737 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               +  27.58%                           +  26.26%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 27.58%
The test has statistical significance

Q: select count(*) from t_arrays_even where array_length(xs, 1) >= 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |       11.772 ±    2.213 |      7.325 |     13.187 |     13.315 |     14.958 |
|   V2    |        0.563 ±    0.060 |      0.474 |      0.549 |      0.592 |      0.835 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 181.75%                           - 184.02%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 181.75%
The test has statistical significance

Q: select count(*) from t_arrays_even where array_length(xs, 1) > 1
C: 1
| Version |         Mean ±    Stdev |        Min |     Median |         Q3 |        Max |
|   V1    |      295.024 ±  280.527 |    233.114 |    242.598 |    258.602 |   2227.243 |
|   V2    |        0.428 ±    0.047 |      0.358 |      0.417 |      0.446 |      0.615 |
├---------┴-------------------------┴------------┴------------┴------------┴------------┘
|               - 199.42%                           - 199.31%
There is a 100.00% probability that the observed difference is not random, and the best estimate of that difference is 199.42%
The test has statistical significance

System/JVM Metrics (durations in ms, byte-values in MB)
    |    YOUNG GC            |       OLD GC           |      HEAP         |     ALLOC
    |  cnt      avg      max |  cnt      avg      max |  initial     used |     rate      total
 V1 |  120     2.46     2.35 |    0     0.00     0.00 |     4295      208 |  1330.27     309235
 V2 |   41     4.08     2.58 |    1    48.40    48.40 |     4295     2843 |  1713.66     104993


Proof of concept for using docker to build openjdk* for distros

See docker/README.md for details. I think it could be a way to support setting up say one jenkins environment to build for multiple distributions without building an environment for each.

Interesting, I'm using Docker already in my build factory.
Let me review this a bit more before merging.
Thanks

Interesting I'm actually using oki-docki to build my generic open jdk 8 distribution


tenant logout handler

There is no way to log out of Observatorium.
I propose a login/logout handler /oidc/{tenant}/logout to be added near https://github.com/observatorium/api/blob/main/authentication/oidc.go#L160 .  A similar handler will also be needed for /openshift.
This handler would set the tenant OIDC cookie to "" with expiration at 1-1-1970, and then redirect to some Observatorium path, triggering the OIDC login flow to let the user log in as someone else.
(Update: Lucas suggests reusing the login endpoint /oidc/{tenant}/login, and it seems that the functionality I need is implemented there, as long as the authentication is OIDC.)
I tried Lucas's suggestion locally.  If my tenant is using OIDC, I can implement "logout" by having the trace UI visit /v1/traces/{tenant}/login, which I've redirected to /oidc/{tenant}/login.  (This is oidc.go's handlerPrefix+loginRoute.)
Unfortunately this is the wrong URL if the tenant is using OpenShift -- for those I need to redirect to /openshift/tenant/login.  So perhaps Observatorium could have an /{tenant}/login endpoint that redirects based on the tenant's auth provider?
If this is desired I can probably implement it.

Notes from April 6 2021 Observatorium Community Meeting

Separation of UI and tenancy metadata (UI does not need to compute path)
/login /logout

Each tenant has a different cookie
Putting tenants in the path will be then easier

Prem proposal: Easier for Jaeger UI - relative UI /logout/{tenant} that proxies to proper endpoint in UI.

Better namespaces to not have top-level matchers
/login/{tenant}

Every tenant has a specific URL

Notes from June 1 Observatorium Community Meeting
I am experiencing this problem again.  I have installed Observatorium and am using Keycloak OIDC, but misconfigured tenants.yaml or rbac.yaml.
The symptom is that Keycloak believes I am logged in, but the login user has no permissions to see Observatorium resources.  / returns valid JSON, the real pages correctly return HTTP Unauthorized.
The problem is that /oidc//login redirects to Keycloak, which believes I am logged in, and redirects back without an opportunity to log in as a different user.  The only way to change users is by deleting Keycloak cookies on the browser.


Support for Rails 3.1 with asset pipeline disabled

I find it amusingly coincidental that you worked on the rails31 upgrade on the exact same day that we had decided to upgrade to rails 3.1.  Thanks for themes_for_rails as it has come in quite handy.
Since disabling the asset pipeline is an option in rails 3.1 and we're doing that as an interim step to our upgrade, I thought you might be interested in these changes.  I know you probably wanted to get rid of some of the old stuff, but I pull some of it back in to allow themes_for_rails to support 3.1 with and without asset pipeline.
You could also relax the gemspec dependency to allow for version below 3.1, but I'd imagine it be just as easy for people not to upgrade beyond 0.4.2 on Rails 3.0.
If you like this, please let me know and I can modify the test suite to test in both conditions (asset pipeline enabled and disabled).  Once again thanks and its good to see you're actively maintaining.

Ethan, sorry I did not apply this patch. I was traveling so I missed it.
 First attempt on Integrating ThemesForRails with this was a total fail, so I decided to put that on hold for now.
The current version (0.4.3) works properly with rails 3.1 and rails 3.2 (with some deprecation warnings). Master branch doesn't throw those messages.
Feel free to contact me about this any time you want.
Thanks


AKS module - change for_each conditions

Issue-id
PR Checklist

[ ] I have added example(s) inside the [./examples/] folder
[ ] I have added the example(s) to the integration test list for normal (~30 minutes) or long runner >30 minutes
[x] I have checked the coding conventions as per the wiki
[x] I have checked to ensure there aren't other open Pull Requests for the same update/change?

Description

Some attributes of the AKS Cluster module are not being correctly evaluated.
Originally I have noticed this behavior only for the upgrade_settings setting, but there are other settings that might fail if not fixed.
I'm basically replacing this:
for_each = try(var.settings.SETTING_NAME, null) == null ? [] : [1]
with this:
for_each = try(var.settings.SETTING_NAME[*], {})
across several dynamic blocks.
Does this introduce a breaking change

[ ] YES
[x] NO

Testing

Thanks @a-ferri for the PR. At least you are providing a consistent way to address the optional dynamic blocks. The problem we have not been able to address in a nice way is to differentiate the optional blocks that can only be set once vs the multiple.
we used to use the following code for One Optional block
for_each = try(var.settings.default_node_pool.upgrade_settings, null) == null ? [] : [1]

by reading the code it was clear it was an optional One Block. This is why in the content {} section we were referring by var.settings.default_node_pool.upgrade_settings.xxxx as it could not be iterated more than once.
With your proposition I agree it is clearer an the iterator is part of the content {} making the code cleaner but we are loosing the only One Block.
Have you found a nice solution for that? In the meantime I pull review it with the core maintainer team.

Hey @LaurentLesle ! Thanks for reviewing!
I'm not sure I'm following you! :) Assuming this code block as input:
default_node_pool = {
  ...

  upgrade_settings = {
    max_surge = string
  }
}

What are we try to accomplish by iterating only once? Avoiding users to provide bad inputs?

Hey @LaurentLesle
Any comments on that?
I've noticed that we have some bugs that would be fixed by merging it.
example:
  dynamic "upgrade_settings" {
    for_each = try(each.value.upgrade_settings, null) == null ? [] : [1]
    content {
      max_surge = upgrade_settings.value.max_surge ### <-- this won't work
    }
  }


Mounted local.cfg into docker/cli.yml` because it was ignored

Phases
MP
MM
MB
MR
JM
Total

ETA
0
0
0
0
0
0

Developing
0
0
0
0
0
0

Review
0
0
0
0
0
0

Total
-
-
-
-
-
0

ETA est.

0

ETA cust.
-
-
-
-
-
0

Problem description

Duplicity of this: https://github.com/dataquest-dev/dspace-angular/pull/555


Is this Flutter plugin still supported or can we take over the maintenance of it?

Yes please 🙏

Yes please 🙏


Modularizing Material and FortAwesome under Core

What:

Modularizing Material and FortAwesome

Modularizing Material and FortAwesome

Why would we want to import more material from start ?

Why would we want to import more material from start ?

When modularizing you can import Material in just one place. It is exported to the root.
I have imported all material components into one module to make it easier to develop. You can delete the compontents not in use.
I did the same thing for FortAwesome but for only Icons and features used.
Check it out.  It makes coding easier.

@dimitrileite yeah but that has a bad impact on tree-shaking / startup time right ?

 Congrats for the great work!


Asset Transfer issue . [Help]

BigchainDB version: Latest
Operating System: OSX
Deployment Type: [Docker|Host|IPDB|Other] google Cloud

If you are using IPDB, please specify your network type [test, prod]
and the bdb_url(BigchainDB URL) you are using.
For every other type of deployment, please specify the documentation/instructions
you are following. Have deployed the big chain node on google cloud

BigchainDB driver: [yes|no] Python Driver

If using a driver please specify, driver type [python|js|java]
and version.

Description
I'm trying to transfer the bicycle asset from  bob to another person
Steps to Reproduce
If you have the precise steps to reproduce, please specify. If you can reproduce
ocassionally, please provide additional information; e.g. screenshots, commands, logs, etc.
What I Did
from bigchaindb_driver import BigchainDB  # This imports the BigchainDB which has been running on the system
from bigchaindb_driver.crypto import generate_keypair  # This imports the generate_keypair module which is required for creating private and public kets
import time
bdb_root_url= 'http://<IP_ADDRESS>:9984'
#bdb_root_url = 'http://localhost:9984'

bdb = BigchainDB(bdb_root_url)  # This denotes the fact that we are not using authentication tokens
alice = generate_keypair()
bob = generate_keypair()
matt = generate_keypair()

def poll(transaction_id):
        for attempt in range(10):
            if bdb.transactions.status(transaction_id).get('status') == 'valid':
                break
            time.sleep(0.5)

'''
Here the asset being transferred is a bicycle which belongs to Alice
The variable below contains a data property which contains the information about the bicycle
'''

bicycle = {
    'data': {
        'bicycle': {
            'serial_number': 'XX2334C',
            'manufacturer': 'bkfab',
        },

    },
}
metadata = {'Country': 'France'}  # Any dictionary can be used as Metadata
prepared_creation_tx = bdb.transactions.prepare(operation='CREATE', signers=alice.public_key, asset=bicycle,
                                                metadata=metadata)  # This creates a digital asset
fulfilled_creation_tx = bdb.transactions.fulfill(prepared_creation_tx,
                                                 private_keys=alice.private_key)
sent_creation_tx = bdb.transactions.send(
    fulfilled_creation_tx)  # the authorized transaction is now sent over to the BigchainDB
txid = fulfilled_creation_tx['id']  # contains the transaction id
poll(txid)
creation_tx = fulfilled_creation_tx  # Alice retrieves the transaction
print("Alice retrieves", creation_tx)

'''
Preparing a transaction
To get inforamtion about the id of the asset we are transferrring
'''
asset_id = creation_tx['id']
print(asset_id)
transfer_asset = {
    'id': asset_id,
}

'''
Transfer transaction
'''
output_index = 0
output = creation_tx['outputs'][output_index]
transfer_input = {
    'fulfillment': output['condition']['details'],
    'fulfills': {
        'output_index': output_index,
        'transaction_id': creation_tx['id'],
    },
    'owners_before': output['public_keys'],
}
prepared_transfer_tx = bdb.transactions.prepare(
    operation='TRANSFER',
    asset=transfer_asset,
    inputs=transfer_input,
    recipients=bob.public_key,
)
# Fulfillment of transfer
fulfilled_transfer_tx = bdb.transactions.fulfill(
    prepared_transfer_tx,
    private_keys=alice.private_key,
)
# Send it across the Node
sent_transfer_tx = bdb.transactions.send(fulfilled_transfer_tx)
poll(sent_transfer_tx['id'])
print("Fulfilled transfer looks like", fulfilled_transfer_tx)
print("Is the bob the owner?", sent_transfer_tx['outputs'][0]['public_keys'][0] == bob.public_key)
print("Was Alice the previous owner? ", fulfilled_transfer_tx['inputs'][0]['owners_before'][0] == alice.public_key)
transfer_tx = fulfilled_transfer_tx

'''
Transfer transaction again
'''
asset_id = transfer_tx['asset']['id']
transfer_asset = {
    'id': asset_id,
}
print("Helllooo")
print(transfer_tx['asset']['id'])

output = creation_tx['outputs'][output_index]
transfer_input = {
    'fulfillment': output['condition']['details'],
    'fulfills': {
        'output_index': output_index,
        'transaction_id': transfer_tx['id'],
    },
    'owners_before': [bob.public_key]
}
prepared_transfer_tx_again = bdb.transactions.prepare(
    operation='TRANSFER',
    asset=transfer_asset,
    inputs=transfer_input,
    recipients=matt.public_key,
)
# Fulfillment of transfer
fulfilled_transfer_tx_again = bdb.transactions.fulfill(
    prepared_transfer_tx_again,
    private_keys=bob.private_key,
)
# Send it across the Node
sent_transfer_tx_again = bdb.transactions.send(fulfilled_transfer_tx_again)
poll(sent_transfer_tx_again['id'])
print("Fulfilled transfer looks like")
print(fulfilled_transfer_tx_again)

print("Is Matt the owner?", sent_transfer_tx_again['outputs'][0]['public_keys'][0] == matt.public_key)
print("Was Bob the previous owner? ", fulfilled_transfer_tx_again['inputs'][0]['owners_before'][0] == bob.public_key)

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback/error message here.

Alice retrieves {'inputs': [{'owners_before': ['FHPauTx5fppmZPnGQbisNBhD4khNP1Qfh8KT4xfXSqQG'], 'fulfills': None, 'fulfillment': 'pGSAINQ2ckSX
8IFr8c87scBi39hcGwQvf-dROCsD0O_k1cjNgUBwuKyMVHLRcQEKdsx6g4iHJkrhhWh8ymDhk6TQJ83yWLk3Q-rSPtj8IctlAEiwswMhiXT1IuVmvmujJeof3NwN'}], 'outputs': [{
'public_keys': ['FHPauTx5fppmZPnGQbisNBhD4khNP1Qfh8KT4xfXSqQG'], 'condition': {'details': {'type': 'ed25519-sha-256', 'public_key': 'FHPauTx5f
ppmZPnGQbisNBhD4khNP1Qfh8KT4xfXSqQG'}, 'uri': 'ni:///sha-256;nlz9fzQPP4b88gUIigL0vhTj1g1XCzo6MBU67b0w12E?fpt=ed25519-sha-256&cost=131072'}, 'a
mount': '1'}], 'operation': 'CREATE', 'metadata': {'Country': 'France'}, 'asset': {'data': {'bicycle': {'serial_number': 'XX2334C', 'manufactu
rer': 'bkfab'}}}, 'version': '1.0', 'id': '24137a4d9dbeca48448fa5e200f783eb84c55318afe9f7033c0691890004ce63'}
24137a4d9dbeca48448fa5e200f783eb84c55318afe9f7033c0691890004ce63
Fulfilled transfer looks like {'inputs': [{'owners_before': ['FHPauTx5fppmZPnGQbisNBhD4khNP1Qfh8KT4xfXSqQG'], 'fulfills': {'transaction_id': '
24137a4d9dbeca48448fa5e200f783eb84c55318afe9f7033c0691890004ce63', 'output_index': 0}, 'fulfillment': 'pGSAINQ2ckSX8IFr8c87scBi39hcGwQvf-dROCs
D0O_k1cjNgUAns7E-UNMlw-MRZ_zkZF0nzZMZi7Axr51Ptm614ASwxpOkrZhKMU2v5qHamBfwkxfOFCV2X_ESeeM6nHQzW8sF'}], 'outputs': [{'public_keys': ['4SLbEyyk4u
Xnr6JDiYLXuPzqysbEqV85G6JLsyF6Ymrs'], 'condition': {'details': {'type': 'ed25519-sha-256', 'public_key': '4SLbEyyk4uXnr6JDiYLXuPzqysbEqV85G6JL
syF6Ymrs'}, 'uri': 'ni:///sha-256;pNQKjRR-agmRXOkUf6WX_sRX_tmKEFIpCia-YCmYl-c?fpt=ed25519-sha-256&cost=131072'}, 'amount': '1'}], 'operation':
 'TRANSFER', 'metadata': None, 'asset': {'id': '24137a4d9dbeca48448fa5e200f783eb84c55318afe9f7033c0691890004ce63'}, 'version': '1.0', 'id': '1
8b3e74f2f38bc5beb3abde4df3b8a486261eed729f834bd6b2613bd36a90700'}
Is the bob the owner? True

Was Alice the previous owner?  True
Helllooo
24137a4d9dbeca48448fa5e200f783eb84c55318afe9f7033c0691890004ce63
Traceback (most recent call last):
  File "Bgdb.py", line 120, in <module>
    poll(sent_transfer_tx_again['id'])
  File "Bgdb.py", line 15, in poll
    if bdb.transactions.status(transaction_id).get('status') == 'valid':
  File "/anaconda/lib/python3.6/site-packages/bigchaindb_driver/driver.py", line 353, in status
    headers=headers)
  File "/anaconda/lib/python3.6/site-packages/bigchaindb_driver/transport.py", line 58, in forward_request
    headers=headers,
  File "/anaconda/lib/python3.6/site-packages/bigchaindb_driver/connection.py", line 57, in request
    raise exc_cls(response.status_code, text, json)
bigchaindb_driver.exceptions.NotFoundError: (404, '{\n  "message": "Not found", \n  "status": 404\n}\n', {'message': 'Not found', 'status': 40
4})

The first problem I noticed was in the section titled "Transfer transaction again". I'm guessing you want to create a second TRANSFER transaction which transfers the output on the first TRANSFER transaction.
CREATE --> TRANSFER 1 ---> TRANSFER 2
creation_tx --> transfer_tx --> ???
As you build the new transfer transaction (TRANSFER 2), you want to be transferring the output on TRANSFER 1 (transfer_tx). Why then, did you do:
output = creation_tx['outputs'][output_index]

Suggestion:  Change creation_tx to transfer_tx and see what happens.

@ttmc
I opened a similar issue a while back , which was solved by @diminator. I am running the code provided by him.
https://github.com/bigchaindb/bigchaindb/issues/1786

I checked your suggestion, it yields the same error
Raaghavvs-MacBook-Air:Blockchain bulletcoder$ python Bgdb.py
Alice retrieves {'inputs': [{'owners_before': ['8YGvPt81XvY3mABSgU61md81YveawY9igTPzwYcPoVLZ'], 'fulfills': None, 'fulfillment': 'pGSAIHAE0KZ0
OXPMPNeLco40H09tDESUkKv4QxKjNqu86JmOgUD6MtmIHhGH5IAfgxwdEgFgq3XYxYs3TVRGiZraAhsBNw56t4y-NyxnR3QpQXrygK9Blpr_Gr2wU4YXXppGdUgH'}], 'outputs': [{
'public_keys': ['8YGvPt81XvY3mABSgU61md81YveawY9igTPzwYcPoVLZ'], 'condition': {'details': {'type': 'ed25519-sha-256', 'public_key': '8YGvPt81X
vY3mABSgU61md81YveawY9igTPzwYcPoVLZ'}, 'uri': 'ni:///sha-256;u-GwvDBfg3sVA_x4XWzcarG1z5jXPwLT6L6bJdGtCPs?fpt=ed25519-sha-256&cost=131072'}, 'a
mount': '1'}], 'operation': 'CREATE', 'metadata': {'Country': 'France'}, 'asset': {'data': {'bicycle': {'serial_number': 'XX2334C', 'manufactu
rer': 'bkfab'}}}, 'version': '1.0', 'id': 'e4bcca7a7c9cb8a0af8ecf80ef73bddd89ca18259801f5854d564a0c12f0c350'}
e4bcca7a7c9cb8a0af8ecf80ef73bddd89ca18259801f5854d564a0c12f0c350
Traceback (most recent call last):
File "Bgdb.py", line 81, in
sent_transfer_tx = bdb.transactions.send(fulfilled_transfer_tx)
File "/anaconda/lib/python3.6/site-packages/bigchaindb_driver/driver.py", line 319, in send
method='POST', path=self.path, json=transaction, headers=headers)
File "/anaconda/lib/python3.6/site-packages/bigchaindb_driver/transport.py", line 58, in forward_request
headers=headers,
File "/anaconda/lib/python3.6/site-packages/bigchaindb_driver/connection.py", line 57, in request
raise exc_cls(response.status_code, text, json)
bigchaindb_driver.exceptions.BadRequest: (400, '{\n  "message": "Invalid transaction (TransactionNotInValidBlock): input e4bcca7a7c9cb8a0af8e cf80ef73bddd89ca18259801f5854d564a0c12f0c350 does not exist in a valid block", \n  "status": 400\n}\n', {'message': 'Invalid transaction (Tra
nsactionNotInValidBlock): input e4bcca7a7c9cb8a0af8ecf80ef73bddd89ca18259801f5854d564a0c12f0c350 does not exist in a valid block', 'status':
400})
Raaghavvs-MacBook-Air:Blockchain bulletcoder$

Please drag and drop your file Bgdb.py onto this page (as a file). GitHub should embed it as a file that I can then download to my computer. I'll take a look through it and see what I can see.

Thanks for your help, Troy :)
Bgdb.py.zip

To test your script (Bgdb.py), I set up a clean MongoDB + BigchainDB running on my local machine (on localhost). I edited the opening of the script to make bdb_root_url = 'http://localhost:9984'
Then I ran it (in a virtualenv where "python" is Python 3.6.3). Here is what happened:
$ python Bgdb.py
Alice retrieves {'inputs': [{'owners_before': ['D2MPxkSJNKCQmex1kQ3B7Utp4eKFw6GgDYB7nAEPynn1'], 'fulfills': None, 'fulfillment': 'pGSAILKk4rg0JVZqQvExe_VOEG36tToCdvYEcDbjMnF3PlGGgUB_vuiZ5nacXO51iQDAu3vRcqX1Z0TVxxZCA49MjaxsvrlmlZzwWlmhr4pX7Q6onW7GCfVGF5_5DXNohzvn478M'}], 'outputs': [{'public_keys': ['D2MPxkSJNKCQmex1kQ3B7Utp4eKFw6GgDYB7nAEPynn1'], 'condition': {'details': {'type': 'ed25519-sha-256', 'public_key': 'D2MPxkSJNKCQmex1kQ3B7Utp4eKFw6GgDYB7nAEPynn1'}, 'uri': 'ni:///sha-256;thX3MuR9Dd10Hk-LHQgm6j9aZZdMmgJl2Y2oz8Myp7E?fpt=ed25519-sha-256&cost=131072'}, 'amount': '1'}], 'operation': 'CREATE', 'metadata': {'Country': 'France'}, 'asset': {'data': {'bicycle': {'serial_number': 'XX2334C', 'manufacturer': 'bkfab'}}}, 'version': '1.0', 'id': '610e9a0ce248009de53ad1718048cce2566835a4f909b6f71955fc8b2c7d1a0b'}
610e9a0ce248009de53ad1718048cce2566835a4f909b6f71955fc8b2c7d1a0b
Fulfilled transfer looks like {'inputs': [{'owners_before': ['D2MPxkSJNKCQmex1kQ3B7Utp4eKFw6GgDYB7nAEPynn1'], 'fulfills': {'transaction_id': '610e9a0ce248009de53ad1718048cce2566835a4f909b6f71955fc8b2c7d1a0b', 'output_index': 0}, 'fulfillment': 'pGSAILKk4rg0JVZqQvExe_VOEG36tToCdvYEcDbjMnF3PlGGgUCGjwimKfPNB7I2Src_JNtgbcHhFSc1YGTAwELVZS7JkMwlJt1JSbKBqivNxcrtUlu99j73DYEWkb78O9mNFj0P'}], 'outputs': [{'public_keys': ['7u54hh2wtM2LJHo6xba5Lf8LP3HeeXNszCDqz5nncWak'], 'condition': {'details': {'type': 'ed25519-sha-256', 'public_key': '7u54hh2wtM2LJHo6xba5Lf8LP3HeeXNszCDqz5nncWak'}, 'uri': 'ni:///sha-256;GYztpE7zKWx2HM0tZiLwGtLW8S2JCbAgLBxb95ywL1A?fpt=ed25519-sha-256&cost=131072'}, 'amount': '1'}], 'operation': 'TRANSFER', 'metadata': None, 'asset': {'id': '610e9a0ce248009de53ad1718048cce2566835a4f909b6f71955fc8b2c7d1a0b'}, 'version': '1.0', 'id': '332c80172cbfc4198e39523e637a557cb0b85a4fb0d239a2f28d7856565c8a6f'}
Is the bob the owner? True
Was Alice the previous owner?  True
Helllooo
610e9a0ce248009de53ad1718048cce2566835a4f909b6f71955fc8b2c7d1a0b
Fulfilled transfer looks like
{'inputs': [{'owners_before': ['7u54hh2wtM2LJHo6xba5Lf8LP3HeeXNszCDqz5nncWak'], 'fulfills': {'transaction_id': '332c80172cbfc4198e39523e637a557cb0b85a4fb0d239a2f28d7856565c8a6f', 'output_index': 0}, 'fulfillment': 'pGSAIGZ86ynA4Con9PGWFmzeVmPnBJq2zjLwZ6fHT-OrmzyhgUB2NZ1RPz9EUis7nEQjJD-4H85rpZ6QQcsM18nluacJc6ZP2KH4ou5BO9OjwHfKZ2NHC6HzOK8BRHzRU2dZdI0L'}], 'outputs': [{'public_keys': ['HALZt5dyFkb5JMHfR6Aq6VyRMoMWezN5M5WTcAC9DqZT'], 'condition': {'details': {'type': 'ed25519-sha-256', 'public_key': 'HALZt5dyFkb5JMHfR6Aq6VyRMoMWezN5M5WTcAC9DqZT'}, 'uri': 'ni:///sha-256;l8R_-QQw5C9TrVZqWiOlkqZtzi9PT2cIxzIeqFms2GI?fpt=ed25519-sha-256&cost=131072'}, 'amount': '1'}], 'operation': 'TRANSFER', 'metadata': None, 'asset': {'id': '610e9a0ce248009de53ad1718048cce2566835a4f909b6f71955fc8b2c7d1a0b'}, 'version': '1.0', 'id': '93ac803a8acf11b5ef41f9b68fee041cac44a7c2905a5adfaec71088985b5cbd'}
Is Matt the owner? True
Was Bob the previous owner?  True

It all looks okay to me. Maybe you just need to try with a clean fresh backend database (MongoDB or RethinkDB)?

It works on my localhost as well, I recently installed a fresh version of both bigchain and Mongodb on the cloud. You want me to do it again?

You don't have to reinstall the MongoDB database. You can just "drop" it using bigchaindb drop (on the machine currently running MongoDB). It that's not enough, then maybe:
$ sudo mongod --shutdown
$ sudo rm -rf /data/db
$ sudo mkdir -p /data/db
$ sudo chmod -R 700 /data/db
$ sudo mongod --replSet=bigchain-rs

still the same issue Troy, I can't seem to transfer the asset to bob or Matt now :/

@raaghavvd This is an old issue and BigchainDB is very different from what it was when this issue was opened. Therefore I'm going to close this issue. If you have any issues with the current version of BigchainDB, please feel free to open a new issue.


manifest: Update TF-M with shared boot data fix

Update TF-M version with fix for build with shared boot data enabled, i.e. CONFIG_TFM_MCUBOOT_DATA_SHARING=y.
Regression from: 485fa940aacb2ae810dd722366f46386d93839ca
Fixes #61953

I'm not sure what the build failure is, as I don't see any of the build logs, just that it failed.

I'm not sure what the build failure is, as I don't see any of the build logs, just that it failed.

Was likely caused by spot instance termination, CI has passed after rerunning


Fix wording of question to make it more clear

It is a bit unclear what exactly is the goal output because the word "is" isn't in the goal.

If I'm misunderstanding what you mean from this challenge, feel free to close this pull request.

@ArkhamCookie Good catch - can you get rid of the extra is too?

@ArkhamCookie I got those. Thanks for the fixes


[Bug] typo in package name

I think
addons/addon-base-workflow/packages/base-worklfow-steps
should be named
addons/addon-base-workflow/packages/base-workflow-steps

Thank you for reaching out. Your request has been added to our backlog for research computing solutions. Our
team curates these requests for fit with our solutions vision on a regular basis. Please watch this space for new
updates. If you require immediate assistance, please reach out to your AWS account team. Please note that
security issues should be reported directly to AWS Security at<EMAIL_ADDRESS>


Update pos-payment-extension.md

:x: Organization membership check failed
Members of the MicrosoftDocs organization should use the private repo. Please move your changes to https://github.com/MicrosoftDocs/Dynamics-365-Operations, and close this pull request.


[ fix ] "go to definition" on Windows

Solving #44 , tested on my machine
I worry that I could break something on other OSes now, so please check if the change is ok! 🥺
The normalizedFileName issue prevented even stepping into the main function. But then, the tokens were wrongly offset because of surrogate pairs. If I had the 𝕊 character inside my agda file it would break - have had it worked on Mac?

Thank you! Works on macOS, too!


Add a few links at the top of README

How to get started with open source: https://www.youtube.com/watch?v=MkaIrwOlP6Y
defaang / dojo clan Discord (in case people want to ask questions or chat about open source): https://discord.com/invite/nNtVfKddDD

Please can you explain a little what this issue is all about?


Beginner tutorial on how to open and plot a single FFI.

Please review this tutorial.  There is no astroquery, just refers to a single FFI online.

I clarified the "bright spot" bit. And adding in a different link for the FFI, now it is the real URL.
Please check again.

I did a git amend to remove the executed cells, so I don't know how to remove this at this point. I'd just squash the history if you can.


[python] Enumerations-API refinement

Issue and/or context:
#1746
Changes:

Remove enumeration related arguments from DataFrame.create
Add enumeration values on first write
Depends on https://github.com/TileDB-Inc/TileDB-Py/pull/1847; passing locally when built against this branch

catch_clara.cc file failure in C++ build? Not related to changes in this PR. Will investigate tomorrow morning.

@nguyenv this is failing CI because it's not on top of #1838

There is actually two more corrections I need to do. I already solved one. If there are no new enumeration values, then do not evolve the schema (before, we were extending with an empty list). But there's one more issue having to do with Boolean enums.


add pre-commit

Codecov Report
Attention: 147 lines in your changes are missing coverage. Please review.

Comparison is base (e5d0034) 73.88% compared to head (0fc8315) 73.77%.
Report is 3 commits behind head on main.

Files
Patch %
Lines

stemflow/model/AdaSTEM.py
70.68%
51 Missing :warning:

stemflow/utils/quadtree.py
78.76%
24 Missing :warning:

stemflow/model/Hurdle.py
55.31%
21 Missing :warning:

stemflow/model_selection.py
40.62%
19 Missing :warning:

stemflow/utils/plot_gif.py
63.63%
16 Missing :warning:

tests/test_all.py
93.47%
6 Missing :warning:

stemflow/model/static_func_AdaSTEM.py
91.80%
5 Missing :warning:

stemflow/model/dummy_model.py
42.85%
4 Missing :warning:

stemflow/utils/validation.py
50.00%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #12      +/-   ##
==========================================
- Coverage   73.88%   73.77%   -0.11%
==========================================
  Files          14       14
  Lines        1164     1167       +3
==========================================
+ Hits          860      861       +1
- Misses        304      306       +2

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[dv] CSR Coverage Waiver

Coverpoints/bins affected

cp_csr_read_only/cp_csr_write

CSR_SCONTEXT
CSR_MCONTEXT
CSR_MSECCFGH
CSR_TDATA3
CSR_MCOUNTINHIBIT
CSR_MENVCFGH

cp_ignored_csrs / cp_ignored_csrs_w

CSR_SCONTEXT, CSR_MCONTEXT, CSR_TDATA3 are also the ignored CSRs, remove this coverpoint?

Justification
None of these is particularly complex but it's all small changes to spike that add up.
CSR_SCONTEXT / CSR_MCONTEXT are optional CSRs only accessible in debug mode, Ibex implements them as read as 0, write ignore. They do not need to implemented and aren't used by our debug setup. As they're not accessible via debug mode this is a low priority item to test
CSR_MSECCFGH/CSR_MENVCFGH - These CSRs have no implemented bits and are only accessible in M mode. They're in the M mode only CSR space so the generic check that accesses from U mode cause illegal instruction exceptions is sufficiently testing by other illegal CSR accesses from U mode.
CSR_TDATA3 - This is accessible in M mode and is an optional CSR for debug trigger points, Ibex implements it as read as 0 write ignore.
CSR_MCOUNTINHIBIT - The controls the performance counters and is not a high priority item.

estimate 4

We should consider stimulating these for V3

I think at least some of this has been addressed, need to revisit.

estimate range: 2 - 4


add boundary sdk, move system from core, update example

Overview

Create the boundary sdk feature

Adds the play area visualization script from #2538.

Move the boundary system implementation from Core -> SDK\Features\Boundary\System (per #2540)
Update the Boundary demo scene

Use the play area visualization script
Cleanup the sphere based boundary visualization script (remove rendering the quad)
Update the info panel to

list WMR immersive and OpenVR under "works on"
add scripts used and a description of the purpose of each

Added DNM while I work on the recommended boundary profile / inspector (based on #2533).

Looks good so far

Can we also add the Boundary Profile with a scene/prefab selection, and other options?

Can we also add the Boundary Profile with a scene/prefab selection, and other options?

That's why I have it marked DNM - i'm working on it :)

What exactly is the difference between PlayAreaVisualizer and FloorPlaneVisualizer?
They seem very identical. Perhaps we can lose one of them?

@davidkline-ms see https://github.com/davidkline-ms/MixedRealityToolkit-Unity/pull/1

Plz do not merge until changes resulting from @StephenHodgson's PR and an offline discussion are in


Fixed #19: Addd proguard config and api anotation

Added Proguard file with rules for the serialization.
Annotated all networking objects with GSON.

Test


Remove quotes from font face names enclosed by "local(" and ")"

For the moment, the module removes quotes only on font family names but according to the W3C spec.:

The syntax for a  is a unique font face name enclosed by "local(" and ")". The name can optionally be enclosed in quotes. If unquoted, the unquoted font family name processing conventions apply;

we are able to remove quotes also on font face names enclosed by "local(" and ")"  in a @font-face rule.
IMHO, it could be a new safe optimisation to add inside your removeQuotes option.
BUT when I worked on this feature, I found a f*** bug in Edge and IE :disappointed_relieved:
I submitted a feedback to the community there: https://connect.microsoft.com/IE/feedbackdetail/view/1957543/
We can wait and see if the Edge/IE team will take into consideration this feedback to introduce this optimisation into your plugin as soon as possible.

@7studio Thanks a lot! Keep me informed, please.


actually update the bot lol

there's actually some things i need to update here, so I'll probably do those updates sometime this week

wow that was a lie lol, jk updates should start soon

updates have been made


Integration for EtherFact

Many user wishing they can store the coins from our network on there trezor and have native support from you guys from the official site.
EIP155 enabled chain ID is 1.
https://www.etherfact.org is our website.
Many thanks to MyEtherWallet

up

@etherfact Can you fix the conflicts?

@gamalielhere Yes the conflicts I have fixed on commit will work for EtherFact network.

@etherfact I meant the github conflicts so we can merge your branch.

@gamalielhere
There was no conflict at all

wallet.etherfact.org
We are still using and no error occurred

Can you please confirm the chainid, it is empty and chainid 1 is main ETH, if you are using chainid 1 as well, then we cannot add this network since it'll enable replay attacks

i'm using ChainID 1

Closing this. Reason:

if you are using chainid 1 as well, then we cannot add this network since it'll enable replay attacks


Render stuff once

Fixes #2
Here's some output produced by this branch:
$ kubetop
                 POD               (CONTAINER)        %CPU         MEM      %MEM
s4-infrastructure-216976705-nkn7r                        1 114.688 MiB  117440.0
                                         (web)          1m 45.5469 MiB
                                       (flapp)           0 24.8477 MiB
                              (wormhole-relay)           0 22.8203 MiB
                       (foolscap-log-gatherer)           0 21.4727 MiB
image-building-3987116516-g6s93                          0 13.3359 MiB   13656.0
                              (image-building)           0 13.3359 MiB

Note the %MEM column is bull because the code doesn't load information about nodes yet.

I like it, anyhow.


:bug: remove the creationTimestamp from metadata when using SSA in work

Summary
Currently, if the required object has zero creationTime in metadata, it will cause kube-apiserver to increment generation even if nothing else changes when using SSA to apply work's manifests
Related issue(s)
https://github.com/kubernetes/kubernetes/issues/67610

/assign @qiujian16

/approve
/lgtm


tag new release on main repo

@vikulin for F-Droid to pickup the new release you have to push the tag here too ...
-> https://github.com/vikulin/yggdrasil-android/releases/tag/v1.6

Are you sure that the repo can be mine not the parent from yggdrasil-network?

Sorry I dont understand the question...
Do you suggest to use your fork as "official" git repo?

No, I mean I'm not sure F-Droid can pick up the fork repo.

I think the tag will be automatically seted up after new release is published. Please reopen the issue in that case


attempt to simplify some examples + fix stuff for recent mne

@hubertjb @robintibor @gemeinl
can you have a look?
if we could simplify further before the next release it would be great

@robintibor @hubertjb @gemeinl I think I am done here. See rendered doc at:
https://1786-232335424-gh.circle-artifacts.com/0/dev/index.html
let me know if you see anything weird.

Fixed small typo that I missed in PR before @agramfort
In general I don't have objections except I am wondering about the early stopping on the validation set. Normally, I would expect there to be a train/val/test split if one  uses early stopping, now it looks like we are doing early stopping on the set that we report metrics on... might be quite misunderstandable no? @agramfort

ahya this with the ==  I should really start to realize haha

Fixed small typo that I missed in PR before @agramfort
https://github.com/agramfort

thx

In general I don't have objections except I am wondering about the early
stopping on the validation set. Normally, I would expect there to be a
train/val/test split if one uses early stopping, now it looks like we are
doing early stopping on the set that we report metrics on... might be quite
misunderstandable no? @agramfort https://github.com/agramfort

that's a very good point. So maybe we should update the examples to use a
train, valid and test sets?

Well seems like a bigger thing to do that? why did you want to introduce early stopping?
In general, can we maybe merge this as is, make release now and do this separately?

Well seems like a bigger thing to do that? why did you want to introduce
early stopping?

well see these examples are educational materials. People get started with
deep learning
on EEG from such examples. What do we want them to do? What would you
recommend
them?

In general, can we maybe merge this as is, make release now and do this
separately?

I can revert the changes on EarlyStopping if you want as I agree with you
that without
a valid / test sets it's not great...

Well seems like a bigger thing to do that? why did you want to introduce early stopping?
well see these examples are educational materials. People get started with deep learning on EEG from such examples. What do we want them to do? What would you recommend them?
In general, can we maybe merge this as is, make release now and do this separately?
I can revert the changes on EarlyStopping if you want as I agree with you that without a valid / test sets it's not great...

Yeah I would revert. I would say general deep learning experience as well as our experience suggests that in many settings, with the right training pipeline (e.g. right regularizations and hyperparameters), there is no reason for early stopping: Longer training just keeps increasing validation scores or at least does not decrease them. So I think training without early stopping is also something beginners should learn as the default way.

so if you would revert I would make the release :)

done

Also keep in mind both popular learning rate schedules like cosine annealing and some state-of-the-art techniques to further improve accuracies like exponential moving average are not that compatible with  early stopping https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/

great, merged

Also keep in mind both popular learning rate schedules like cosine
annealing and some state-of-the-art techniques to further improve
accuracies like exponential moving average are not that compatible with
early stopping
https://pytorch.org/blog/how-to-train-state-of-the-art-models-using-torchvision-latest-primitives/

which part of this page are you point me to?

This as an example of a very state of the art training pipeline for deep learning computer vision that does not use early stopping. Also, parts that are at least not really in the spirit of using early stopping from my view:

cosine annealing learning rate schedule
exponential moving average of weights

Also in general wanted to bring this pipeline to your attention, several of the methods there are fairly generic and could also be tried for our deep learning pipelines on EEG

we should have a chat ;)


Extend Falco inputs by creating a Plugin: the basics

Signed-off-by: Issif<EMAIL_ADDRESS>
What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind user-interface

/kind content

/kind translation

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area blog

/area documentation

/area videos

What this PR does / why we need it:
Blog post: Extend Falco inputs by creating a Plugin: the basics
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: Issif
To complete the pull request process, please assign leogr after the PR has been reviewed.
You can assign the PR to them by writing /assign @leogr in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: 9fab7e09f2bee8381d2d3f6fc2120e649fef9210

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Issif, jasondellaluce
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [jasondellaluce]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Better way of representing binary data in .csv output

I just noticed this: https://latest.datasette.io/fixtures/binary_data.csv
rowid,data
1,b'\x15\x1c\x02\xc7\xad\x05\xfe'
2,b'\x15\x1c\x03\xc7\xad\x05\xfe'

There's no good way to represent binary data in a CSV file, but this seems like one of the more-bad options.

Most obvious option is base64. Any other potential solutions I'm missing?

The datasette-render-binary plugin does this, which I really like - but without the different coloured fonts I'm not sure how readable it would be as just plain text:

Really the goal here is to find the most human-friendly option, so that people looking at the output have a vague idea what's going on. That's why I'm not leaping at the chance to use base64.

Asked for ideas on Twitter: https://twitter.com/simonw/status/1318409558805467136?s=21

I could go with the same format as datasette-render-binary but using 0x00 as the format for the hex bytes.
0x15 0x1C 0x02 0xC7 JFIF 0x00 0x01

An even easier option: do what the Datasette UI does and output <Binary data: 7 bytes> for that CSV cell, as seen on https://latest.datasette.io/fixtures/binary_data

Or... default to <Binary data: 7 bytes> and support a ?_base64=1 option which outputs in base64 instead.

New idea: since binary in CSV doesn't make sense anyway, emulate Datasette's HTML UI default and output this:
id,title,data
1,Some title,<Binary data: 14 bytes>
2,Other title,<Binary data: 57 bytes>

Then allow users to add ?_base64=1 to the URL to get base64 instead
https://twitter.com/simonw/status/1318679950635888641

This feels good to me - it's consistent with how other features in Datasette work, and it means users who need the binary data in CSV (for whatever reason) can get it if they want to.

Enzo has a great solution here: https://twitter.com/enzo_mdd/status/1318685442976436226

Or maybe an option for a url. This keeps the CSV small but allows scripts to download binary data as needed.

In #1036 I'm planning on adding a way for users to access BLOB data. I can include that URL in the CSV output.

Blocked awaiting #1036

This is actually a bit tricky to implement, for a couple of reasons:

Need to generate a full URL, including the https://host/ bit. I've done this for next_url in the JSON output before, thankfully.
This only makes sense for CSV output for tables. If it's the CSV output of an arbitrary query there's no /db/table/-/blob/pk/column.blob page for me to link to.
Need to generate those /.../-/blob/... URLs for the data that is being output as CSV.

Here's how those absolute next_url values are generated: https://github.com/simonw/datasette/blob/5db7ae3ce165ded57c7fb1cfbdb3258b1cf06c10/datasette/views/table.py#L774-L776

I should turn datasette.absolute_url(...) into a documented internal API.

That documentation: https://docs.datasette.io/en/latest/internals.html#absolute-url-request-path

What should happen for CSV export of arbitrary SQL queries, where there's no obvious BLOB to link to?

For arbitrary CSV the only solution I can think of is to embed the base64 value.

SQLite actually has APIs that could help here: https://www.sqlite.org/c3ref/column_database_name.html - for any given SQL query they identify the origin/table/column that is the source of each resulting column.
Those aren't exposed in the Python sqlite3 module though, so using them could be extremely tricky.

I'm going to link to. the new .blob representation using the new ?_blob_hash=xxx argument to ensure that the content served is the expected binary blob.

Demos:
https://latest.datasette.io/fixtures/binary_data.csv?_size=max
rowid,data
1,http://latest.datasette.io/fixtures/binary_data/1.blob?_blob_column=data
2,http://latest.datasette.io/fixtures/binary_data/2.blob?_blob_column=data
3,

https://latest.datasette.io/fixtures.csv?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max
rowid,data
1,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=f3088978da8f9aea479ffc7f631370b968d2e855eeb172bea7f6c7a04262bb6d
2,http://latest.datasette.io/fixtures.blob?sql=select+rowid%2C+data+from+binary_data+order+by+rowid+limit+1001&_size=max&_blob_column=data&_blob_hash=b835b0483cedb86130b9a2c280880bf5fadc5318ddf8c18d0df5204d40df1724
3,


Remove ._old_format() from PermutationsExplainer

#2368

Thanks! And sorry for the slow review. (I was out after paternity leave for a while) This is not a recommended API anymore, so I added a depreciation noticed and updated the return value to match the old api (which is just the numpy array of values and not a full explanation object).

Thanks @slundberg, that does it!


ST3 help

Hello Mechgt - first - thanks for all your plugins! It was a good time for Sporttracks :(. One thing i would kindly ask you - by any chance, do you have the opensourcemap plugin and could share it with me? :D...Thanks a lot, Tudor (tsofron@hotmail.com).
PS: You can definitely close the 'issue' :D, but i thought this way you'll get a notification

No, sorry I don't have that.


jsoncrack JSON renderer

I recently came across a visualisation tool that renders JSON files as a collapsible interactive tree that might be interesting as a JSON renderer (I note there is already an experimental VSCode extension for it).

Thanks @psychemedia for the suggestion.
This would make a great third-party lab extension, which would also be compatible with Notebook 7.
Ideally jsoncrack should be distributed first to npm as it would make it much easier to use it in a JupyterLab extension. Unfortunately this doesn't seem to be the case at the moment according to https://github.com/AykutSarac/jsoncrack.com/issues/171 and https://github.com/AykutSarac/jsoncrack.com/issues/151.


Doesn't compile using MingW / GCC

Hey!
If I use stable-gnu then this crate doesn't compile.

I'm not too familar with the Rust windows GNU chain, but this library does contain a pre-built MSVC library from Superluminal that is included so not sure if that can be compatible with the GNU target.
But PRs welcome!

Ahh, okay. This is very interesting. There are multiple angles here:

Superluminal uses PDB, which are produced by Clang, but not by gcc (I think)
One can use superluminal on binaries produced by Mingw (gcc) but there are no source tracking.

Maybe one can conditionally link the libraries if the right target is made, so as to circumvent compile-error, while the whole using superluminal with gcc debug symbols is sorted out.
Unfortunately, I am need deep in trying to figure out how to include libs/dlls in a rust crate and
the frustration levels have peaked for me. I'll post and ask around, and definitely aspire to get back
to this.
Mingw and gcc is the only way we have to debug our binaries on windows. Furthermore, if one uses
mingw, then including 3rd party libraries is very easy. So there are clear advantages to the Mingw toolchain here.
Thanks for getting back to me with this

Right, feel free to close this.


typo propagation

in order to have to redeploy core only once, I can also add the correct field in the upsert
didn't do it directly since it may be more risky to deploy; tell me

As per discussion you also want to add "timestamp" along with "timetamp" at upsert right?

yes done


Issue with HTML email messages with tag %%MESSAGE%%.

Hi,
every HTML sended to the user if have the %%MESSAGE%% tag in the email,
the text ignores the text wrap.
For example in a note i can write 3 different row like:
1.Do this
2. Do that
3. Solved
In the HTML email message user see this:

Do this 2. Do that 3. Solved

This is not good. Also i had put the tag %%MESSAGE%% in note, PM, new staff reply all this HTML email not consider the row, all text is in the same row.
So for now I must use simple text until this won't be fixxed.
If i write message in Hesk in HTML email when i go on new line message should be printed in new line.

I hope version 2.1 will be relased soon so i can activate this fix and use HTML emails. Thanks for fix this error.


Different plotting reaults compared to matplotlib

What happened?
I got different results when I tried to plot 2D data test.npy.zip using matplotlib and xarray.
matplotlib

xarray

What did you expect to happen?
Same plot.
Minimal Complete Verifiable Example
import numpy as np
import xarray as xr
import matplotlib.pyplot as plt

test = np.load('test.npy')

plt.imshow(test, vmin=0, vmax=200)
plt.colorbar()

xr.DataArray(test).plot.imshow(vmin=0, vmax=200)

MVCE confirmation

[X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
[X] Complete example — the example is self-contained, including all data and the text of any traceback.
[X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
[X] New issue — a search of GitHub Issues suggests this is not a duplicate.
[X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output
No response
Anything else we need to know?

You can get the xarray produced plot with mpl by doing:
plt.imshow(test, vmin=0, vmax=200, interpolation="nearest", origin="lower")

And you can get the mpl plot in xarray by doing:
xr.DataArray(test).plot.imshow(vmin=0, vmax=200, origin="upper", interpolation="antialiased")

(note that for this plot, "antialiased effectively equates to 'hanning', also note that mpl's default behavior is affected by rcParameters, where xarray overrides the default regardless of the rcParam)
I think that the antialiasing is actually hiding the noise of the data as provided (there aren't enough pixels to show the actual data, so some resampling has to be done, but what is most representative is not easy to know without context, mpl's defaults will tend to be better for actual images, but xarrays are probably better for measured data)

Thanks for your explanation @ksunden. The data is produced from satellite retrieval, which is a kind of measured data.
BTW, what do you mean by actual images?

mpl's defaults will tend to be better for actual images, but xarrays are probably better for measured data

I also checked plt.pcolor(test, vmin=0, vmax=200) which should show the original data:

So, if I understand correctly, default mpl's plt.imshow() tends to show smooth images while xarray's plot.imshow() is more like plot() which is pcolormesh in mpl.

"actual" is not quite the best term, though "smooth" is also not quite what I mean, either.. not quite sure what word to use here...
But mpl's imshow falls into this awkward dual function of displaying full rgb(a) images as well as colormapped data. and thus the choices of default parameters are not able to suit all usecases.  Any measurement where an individual pixel should be considered separately falls into the category of "measured data" rather clearly. The inverted Y-axis is one of the things that I think actually more distinctly separates the groups, as the orientation of an image mattering is a key part of it to me.
Scientific cameras (including telescopes and e.g. laser lab cameras) fall into a bit of a mushy middle, but I would tend to prefer any smoothing to be done a bit more explicitly than relying on the downsampling of the image viewer, so I would tend to say that "nearest" (xarrays default) tends to be more representative, though context does matter (and if the purpose is "quick view of if the global trends of the data are 'interesting'" then perhaps using mpl's antialiased downsampling is sufficient. But my bias would be towards more explicit means of smoothing for final figures. They also fall into a strange zone where you have to know the particulars of the camera/etc to know which convention of origin and numbering pixels they use, as it is less standardized.
I've tended to use pcolor(and friends) for most of my data, though often because I find it easier to control the x/y axes values which for me are often not just 'pixels' (yes you can provide extents to imshow, but less control that way), but imshow does certainly have performance improvements over that for rectangular data.

Thank you a lot for the detailed comparisons. Because xarray is designed for scientific data (weather, satellite, observations, etc.), it sounds reasonable to use interpolation=nearest for DataArray.plot().


Update xiami to 3.0.9

After making all changes to the cask:

[ ] brew cask audit --download {{cask_file}} is error-free.
[ ] brew cask style --fix {{cask_file}} reports no offenses.
[ ] The commit message includes the cask’s name and version.

Please complete the steps in the PR template next time.


gencpp: 0.6.0-0 in 'kinetic/distribution.yaml' [bloom]

Increasing version of package(s) in repository gencpp to 0.6.0-0:

upstream repository: git@github.com:ros/gencpp.git
release repository: https://github.com/ros-gbp/gencpp-release.git
distro file: kinetic/distribution.yaml
bloom version: 0.6.2
previous version for package: 0.5.5-0

gencpp
* add plugin support for generated C++ message headers (#32 <https://github.com/ros/gencpp/pull/32>)
* put all the message integer constants into a common enum (#25 <https://github.com/ros/gencpp/issues/25>)

Waiting for other ROS core releases into Kinetic...


bug(navbar): Font is different in Explorer navbar when compared across the rest of the application

Describe the bug
The font on the navbar present in Explorer is different to the rest of the application
Slack context
To Reproduce
Steps to reproduce the behavior:

Go to https://cellxgene.cziscience.com/datasets
View the navbar
Open a new tab and go to https://cellxgene.cziscience.com/e/ecce9688-66ea-4600-98f7-84aed7421c01.cxg/
Notice the navbar font and possibly size is slightly different

Expected behavior
The font and size that is present in the rest of the application should be the same for Explorer.
Primary Navigation Tabs: fontBodyS/600 and gray300
Secondary Buttons (Help & Documentation, Log In/Account): fontBodyS/500 and gray300

Convo from Slack to preserve:
Huh I just checked the code and it seems like Explorer uses Roboto Condensed, instead of Inter . And thus Roboto Condensed is used in the navbar now
So I’m now unsure how it was using Inter before for the Navbar
The most recent change is probably DP changing from using Google hosted font to self hosted font.


Update test generation config to raise ValueError

What does this PR do?
Update tests that used to throw warnings and now are ValueError

Fixes # (issue)
Tests were out-of-date. Check transformers tests here
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you make sure to update the documentation with your changes?
[ ] Did you write any new necessary tests?

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Clang: 'std::shared_ptr<uint8_t []>' does not provide a subscript operator

/home/runner/work/codeGGenerator/codeGGenerator/src/C_variable.cpp:145:29: error: type 'std::shared_ptr<uint8_t []>' does not provide a subscript operator
data._code._data[valTarget + 1] = varAdd >> 8;//Address MSB
~~~~^~
/home/runner/work/codeGGenerator/codeGGenerator/src/C_variable.cpp:146:29: error: type 'std::shared_ptr<uint8_t []>' does not provide a subscript operator
data._code._data[valTarget + 3] = varAdd & 0x00FF;//Address LSB
~~~~^~
I really don't know how to fix it :(

i fix with reinterpreted_cast<unsigned char*>(data._code._data.get())[valTarget + 1]

i fix with reinterpreted_cast<unsigned char*>(data._code._data.get())[valTarget + 1]

removing the flag -stdlib=libc++ just fix everything !

removing the flag -stdlib=libc++ just fix everything !


[TEVA-1125] Ensure hint and label are inline

This addresses the last (I pray) of the css inconsistencies between design and the subject input introduced in the move to using the checkbox component
Jira ticket URL
https://dfedigital.atlassian.net/browse/TEVA-1125
Changes in this PR:

Ensure label and hint are inline for checkboxes

The only true solution to little design inconsistencies like this creeping in is to introduce snapshot testing. But in my experience is a complex thing to implement  and maintain on an app like TV which is predominantly a server side app, and hence the effort/value doesn't add up. This is my experience anyway - I have worked mostly with react or similar/SPAs for the past 4 years and snapshot testing is a little easier but still requires some pretty strict rules for coding markup/styles. Maybe a spike is warranted to see if snapshot testing is viable with recent tooling. On a side note I have had a word with myself to pay 100 rather than 95% attention to figma etc but stuff like this is going to happen every now and, unfortunately, as I/we aren't robots

just one question, why 1px instead of 0 - not that it really matters

The govuk frontend checkboxes component has 1px padding on the label, if the margin is 0 in the hint then they don't align


Throughput for autoregressive vs non-autoregressive

The paper states the following: "While the nonautoregressive model throughput is bounded to ∼ 2.8 samples/second for batch sizes bigger than 64, the autoregressive model throughput is linear in batch size, only limited by the GPU memory".
Could you explain why? Why would the throughput for the AR model be linear w.r.t. batch size, but the throughput for the non-AR model more or less constant?
I would expect more or less such a relation between throughput and sequence length, but I don't immediately see what causes this connection between throughput and batch size.

First of all, throughput in this case represents "how many audio samples can the model generate per second".
So, naively thinking, if we use 2 times larger batch size, the model can parallelly generate samples of twice the number, which is what happens in the AR case in the Figure 2.
The difference of inference speed (throughput) between AR and non-AR arises from the following reasons.

AR models can use key-value caching technique at inference time. Thanks to this, the model only have to compute key/values at the  current position (only one token), so the computaiton order is almost linear.
In contrast, non-AR models have to compute attention of "full sequence" at every inference step, which takes much much higher  computatianl cost. I think this is the reason why the throughput of non-AR case is bounded even when using larger batch size.


US 01.02.01

As a meal planner, I want to view an ingredient and its details.

Milestone: Halfway Checkpoint

Acceptance criteria

Users should be able to see the descriptions of an ingredient in the meal plan activity

Storyboard demonstrating the feature

Wiki entry describing the feature

UML for frontend components

Acceptance criteria added by Fan Zhu.

Finished by Fan Zhu

01.02.01 closed


feat(telepay): add telepay implementation with http support

Implement the telepay abstract package having the http package as a dependency

Is done


open file via file dialog doesn't work with security scoped bookmarks

Current Situation
File -> open doesn't work on mac properly. It doesn't create a valid bookmark, at least it can't be read in a subsequent run of LogoRRR.
The same feature works if a file is opened via drag'n drop.
Wanted behavior
File -> open should create a valid security scoped bookmark such that LogoRRR can open a file from a previous run without issues.

see also #145.


100.15.0 Release Branch

Description
Create a branch to merge 100.15 updates into main.
Assigning reviewers outside the SDK team since Tanner is currently on holiday.

Confirmed with James that the permission change was intentional since the pri files don't need the executable permission and it is okay to keep. All other feedback has been addressed. Please review again @JamesMBallard @ldanzinger


[GRO-1153] Updates onboarding data

FYI I also updated an existing event onboardingUserInputData, which I think should be okay because I haven't seen the event fire since we removed onboarding in April [sql]

Yeah hrm I thought we were going to trigger events each time the onboarding user clicks next, no?

yes we are! sorry the naming is confusing but I'd like to fire the onboardingUserInputData event whenever the user clicks next
added the events to this spreadsheet - https://docs.google.com/spreadsheets/d/12wj1rt-j5SguboPJMuaM1GqJD-5B5u5piipw8MQ5Zuk/edit#gid=275928850

LGTM!
Just summarising:

you created three context modules for OnboardingUserInputData
two events for startedOnboarding and completedOnboarding

Yes! Thanks for summarizing :) I definitely should have added that in my description haha, sorry about that!

:rocket: PR was released in v4.59.0 :rocket:


Add dualstack option to S3 storage driver

Allow the storage driver to optionally use AWS SDK's dualstack mode.
This allows the registry to communicate with S3 in IPv6 environments.
Fixes #3067
Signed-off-by: Adam Kaplan<EMAIL_ADDRESS>

fyi @dmage

Bump @justincormack @dmcgowan.
We have been running this code in production for OpenShift's internal registry for almost two years now. It enables support for S3 storage on dualstack/IPv6.

The AWS docs say the UseDualStack option is deprecated and you should use UseDualStackEndpoint https://docs.aws.amazon.com/sdk-for-go/api/aws/#Config.WithUseDualStack

Hey @adambkaplan can you please rebase so that GH Action CI gets triggered -- this PR has remnants of the previous CI setup.

@justincormack updated to use the new DualStack configuration option. Note that this required a dependency bump (which I put in a separate commit).
@milosgajdos rebased to trigger the new GH Actions.


Add "start" script?

Per discussion on the Getting Started guide PR, should these projects have a start script that does build + serve? Many npm packages do.

A unified script would be very helpful


How to hide dots?

How to hide dots in calendar component?

I would also be interested in knowing this. You may consider adding a section in the demo that shows how to style the dots themselves.