Change basis for selectivity estimation for operator scan predicate with ColumnVsValue

This PR improves the cardinality estimation for ColumnVsValue Operator Scan Predicates by fixing a faulty selectivity calculation.
To estimate the cardinality of a result table after a Scan Predicate Operator comparing a Column vs. a Value was applied, the following steps happen:

If the predicate_condition is not one of {IS (NOT) NULL, (NOT) LIKE, BETWEEN ... AND .. } (these are handled in a separate case), the previously calculated histogram of values in the Column is sliced according to the predicate. The sliced histogram then only holds the values of the column that fulfill the predicate.

737 const auto sliced_statistics_object =
            scan_statistics_object->sliced(predicate.predicate_condition, value_variant, value2_variant);

As a next step the selectivity of the predicate on the column is calculated as:

751  selectivity = sliced_histogram->total_count() / scan_statistics_object->total_count();

Here scan_statistics_object holds the previous, not-sliced histogram. This means that the selectivity is calculated based on the total_count of values in the histogram.

Finally the other columns AttributeStatistics are scaled according to the calculated selectivity, and the new row_count estimation for the result table is calculated as:

774 const auto row_count = Cardinality{input_table_statistics->row_count * selectivity};

So the selectivity is applied on the row_count of the input_table.
The estimation error is caused by the fact that histograms do not hold NULL values. Therefore the row_count of the input_table_statistics object can be different from the total_count of the histograms for the table's columns. (See next example picture of the cast_info table from the IMDb benchmark, which prints the value count of each column statistics object.)
As the selectivity is applied on the input_table, it should be calculated based on the row_count of the input_table and not on the differing values count in the histogram.

We argue that as NULL isn't a value in SQL and therefore cannot be compared to values, NULL values will have to be excluded from the estimates, and therefore should be included in the selectivity calculation. This is what happens if we calculate the selectivity based on input_table_statistics->row_count. As far as we can see, this should hold for all relevant cases (Equal, NotEqual, Less/GreaterThan(Equals), Between*), as a comparison against a value would always exclude NULL values in these cases.
In estimations for other Operator Types (e.g. inner_equi_join and semi_join), the selectivity is actually already being calculated like this.
912 const auto left_selectivity = Selectivity{
        left_input_table_statistics.row_count > 0 ? cardinality / left_input_table_statistics.row_count : 0.0f};

TODO:

[x] Compare cardinality benchmark results with improvement against previous cardinality estimations to verify impact.

Comparing the estimated cardinality of the result table of the JoinOrderBenchmark Queries, we see only improvements in estimation or equal estimations. (The only exception are queries with an estimated cardinality of lower 1, which are now sometimes estimated with even lower cardinalities. However, as the actual row count is always 1 in the PQPs, estimating a bit lower shouldn't be that negatively impactful. Also, these estimations are comparatively very much on target and were not the target of our improvement approaches.)
The differences in estimation are not always big, but the overall trend is definitely positive.
join_order_benchmark_cardinality_comparison.csv

Great, thanks for the detailed explanations!

The estimation error is caused by the fact that histograms do not hold NULL values. Therefore the row_count of the input_table_statistics object can be different from the total_count of the histograms for the table's columns.

That makes perfect sense. I was a bit confused by the sentence above at first because both input_table_statistics and scan_statistics_object should be histograms, shouldn't they? The difference results from using row_count vs total_count(), right? The first does not calculate anything but simply return the initial row_count value.

NULL values will have to be excluded from the estimates, and therefore should be included in the selectivity calculation

This confuses me a bit. Do you mean that NULL values will not be part of the result and, hence, this fact should be incorporated when calculating the selectivity?

as a comparison against a value would always exclude NULL values in these cases.

Agreed.

we see only improvements in estimation or equal estimations.

What do you mean?
Regarding the comparison of estimated cardinalities: Differences in the intermediate results (and not for the result table) will be much more interesting.

I benchmarked the impact of your change on one of our servers (maximum runs: 100). There are quite some differences (positive and negative) for some queries. I would recommend to investigate how the query plans were affected by the change and take a look at a query that is significantly slower now, e.g., 12b or 13c.
By the way: the new JOB dataset will be available once this PR is merged, which will probably happen in the next hour.

Configuration Overview - click to expand
 +Configuration Overview---------------------------------------------+------------------------------------------------+
 | Parameter        | job_old_cardinality_estimation.json            | job_new_cardinality_estimation.json            |
 +------------------+------------------------------------------------+------------------------------------------------+
 |  GIT-HASH        | 4fc0f3d139d4a522e2264b466617b6d765e940e1-dirty | 9a8e42220f15baee46259b62cfeaa88d3159b120-dirty |
 |  benchmark_mode  | Ordered                                        | Ordered                                        |
 |  build_type      | release                                        | release                                        |
 |  chunk_size      | 65535                                          | 65535                                          |
 |  clients         | 1                                              | 1                                              |
 |  compiler        | gcc 9.3                                        | gcc 9.3                                        |
 |  cores           | 0                                              | 0                                              |
 |  date            | 2021-07-06 12:44:43                            | 2021-07-06 11:16:53                            |
 |  encoding        | {'default': {'encoding': 'Dictionary'}}        | {'default': {'encoding': 'Dictionary'}}        |
 |  indexes         | False                                          | False                                          |
 |  max_duration    |<PHONE_NUMBER>0                                    |<PHONE_NUMBER>0                                    |
 |  max_runs        | 100                                            | 100                                            |
 |  time_unit       | ns                                             | ns                                             |
 |  using_scheduler | False                                          | False                                          |
 |  verify          | False                                          | False                                          |
 |  warmup_duration | 0                                              | 0                                              |
 +------------------+------------------------------------------------+------------------------------------------------+

 +---------++----------+----------+--------++----------+----------+--------+----------------------+
 | Item    || Latency (ms/iter)   | Change || Throughput (iter/s) | Change |              p-value |
 |         ||      old |      new |        ||      old |      new |        |                      |
 +---------++----------+----------+--------++----------+----------+--------+----------------------+
 | 10a     ||    194.4 |    190.1 |   -2%˄ ||     5.14 |     5.26 |   +2%˄ | (run time too short) |
 | 10b     ||    132.5 |    131.4 |   -1%˄ ||     7.54 |     7.61 |   +1%˄ | (run time too short) |
-| 10c     ||    313.6 |    359.6 |  +15%˄ ||     3.19 |     2.78 |  -13%˄ | (run time too short) |
+| 11a     ||     69.4 |     61.6 |  -11%˄ ||    14.40 |    16.24 |  +13%˄ | (run time too short) |
+| 11b     ||     69.6 |     61.9 |  -11%˄ ||    14.36 |    16.17 |  +13%˄ | (run time too short) |
+| 11c     ||     67.1 |     59.7 |  -11%˄ ||    14.89 |    16.75 |  +12%˄ | (run time too short) |
 | 11d     ||     28.8 |     29.2 |   +1%˄ ||    34.67 |    34.29 |   -1%˄ | (run time too short) |
 | 12a     ||     54.2 |     54.9 |   +1%˄ ||    18.44 |    18.23 |   -1%˄ | (run time too short) |
-| 12b     ||     81.2 |    185.8 | +129%˄ ||    12.32 |     5.38 |  -56%˄ | (run time too short) |
+| 12c     ||   2212.3 |   2085.9 |   -6%  ||     0.45 |     0.48 |   +6%  |               0.0000 |
 | 13a     ||    404.3 |    405.1 |   +0%˄ ||     2.47 |     2.47 |   -0%˄ | (run time too short) |
-| 13b     ||    110.6 |    230.6 | +109%˄ ||     9.04 |     4.34 |  -52%˄ | (run time too short) |
-| 13c     ||     48.9 |    126.9 | +159%˄ ||    20.43 |     7.88 |  -61%˄ | (run time too short) |
 | 13d     ||    782.7 |    781.2 |   -0%  ||     1.28 |     1.28 |   +0%  |               0.5551 |
+| 14a     ||   3014.8 |   2883.4 |   -4%  ||     0.33 |     0.35 |   +5%  |               0.0003 |
 | 14b     ||   2846.9 |   2913.9 |   +2%  ||     0.35 |     0.34 |   -2%  |               0.0251 |
 | 14c     ||   3106.1 |   2981.4 |   -4%  ||     0.32 |     0.34 |   +4%  |               0.0007 |
-| 15a     ||     85.4 |     95.3 |  +12%˄ ||    11.71 |    10.49 |  -10%˄ | (run time too short) |
-| 15b     ||     83.6 |     94.5 |  +13%˄ ||    11.96 |    10.58 |  -12%˄ | (run time too short) |
-| 15c     ||     55.4 |     69.4 |  +25%˄ ||    18.05 |    14.40 |  -20%˄ | (run time too short) |
-| 15d     ||     86.4 |     94.8 |  +10%˄ ||    11.57 |    10.54 |   -9%˄ | (run time too short) |
 | 16a     ||   2679.6 |   2751.5 |   +3%  ||     0.37 |     0.36 |   -3%  |               0.2270 |
 | 16b     ||   3881.6 |   3983.8 |   +3%  ||     0.26 |     0.25 |   -3%  |               0.0295 |
 | 16c     ||   2788.4 |   2844.2 |   +2%  ||     0.36 |     0.35 |   -2%  |               0.1804 |
 | 16d     ||   2785.5 |   2780.9 |   -0%  ||     0.36 |     0.36 |   +0%  |               0.8975 |
 | 17a     ||    817.8 |    831.9 |   +2%  ||     1.22 |     1.20 |   -2%  |               0.0057 |
 | 17b     ||    671.0 |    679.2 |   +1%  ||     1.49 |     1.47 |   -1%  |               0.0345 |
 | 17c     ||    642.7 |    650.7 |   +1%  ||     1.56 |     1.54 |   -1%  |               0.0326 |
 | 17d     ||    788.6 |    797.9 |   +1%  ||     1.27 |     1.25 |   -1%  |               0.0487 |
 | 17e     ||   2137.0 |   2147.4 |   +0%  ||     0.47 |     0.47 |   -0%  |               0.5385 |
 | 17f     ||   1294.7 |   1318.2 |   +2%  ||     0.77 |     0.76 |   -2%  |               0.0031 |
 | 18a     ||   1823.6 |   1816.9 |   -0%  ||     0.55 |     0.55 |   +0%  |               0.8088 |
+| 18b     ||    200.3 |    188.5 |   -6%˄ ||     4.99 |     5.31 |   +6%˄ | (run time too short) |
 | 18c     ||   2972.0 |   2917.9 |   -2%  ||     0.34 |     0.34 |   +2%  |               0.0033 |
-| 19a     ||   3347.3 |   4074.0 |  +22%  ||     0.30 |     0.25 |  -18%  |               0.0000 |
-| 19b     ||   1560.7 |   2247.5 |  +44%  ||     0.64 |     0.44 |  -31%  |               0.0000 |
-| 19c     ||   3232.1 |   3926.2 |  +21%  ||     0.31 |     0.25 |  -18%  |               0.0000 |
 | 19d     ||   2509.0 |   2520.8 |   +0%  ||     0.40 |     0.40 |   -0%  |               0.7412 |
-| 1a      ||     30.0 |     47.6 |  +59%˄ ||    33.34 |    21.01 |  -37%˄ | (run time too short) |
+| 1b      ||     13.1 |     10.4 |  -20%˄ ||    76.58 |    95.69 |  +25%˄ | (run time too short) |
+| 1c      ||     15.7 |     13.0 |  -17%˄ ||    63.69 |    76.85 |  +21%˄ | (run time too short) |
+| 1d      ||     13.0 |     10.5 |  -20%˄ ||    76.68 |    95.41 |  +24%˄ | (run time too short) |
+| 20a     ||    986.7 |    819.6 |  -17%  ||     1.01 |     1.22 |  +20%  |               0.0000 |
+| 20b     ||    817.3 |    703.1 |  -14%  ||     1.22 |     1.42 |  +16%  |               0.0000 |
 | 20c     ||    534.9 |    558.8 |   +4%˄ ||     1.87 |     1.79 |   -4%˄ | (run time too short) |
+| 21a     ||    110.1 |     79.0 |  -28%˄ ||     9.08 |    12.66 |  +39%˄ | (run time too short) |
+| 21b     ||     90.3 |     63.7 |  -30%˄ ||    11.07 |    15.70 |  +42%˄ | (run time too short) |
+| 21c     ||    110.0 |     81.1 |  -26%˄ ||     9.09 |    12.32 |  +36%˄ | (run time too short) |
+| 22a     ||   2396.2 |   1822.3 |  -24%  ||     0.42 |     0.55 |  +31%  |               0.0000 |
+| 22b     ||   1954.9 |   1718.2 |  -12%  ||     0.51 |     0.58 |  +14%  |               0.0000 |
+| 22c     ||   3341.6 |   3194.3 |   -4%  ||     0.30 |     0.31 |   +5%  |               0.0000 |
+| 22d     ||   3480.2 |   3328.3 |   -4%  ||     0.29 |     0.30 |   +5%  |               0.0000 |
-| 23a     ||     61.0 |     74.5 |  +22%˄ ||    16.40 |    13.43 |  -18%˄ | (run time too short) |
 | 23b     ||     67.8 |     67.7 |   -0%˄ ||    14.75 |    14.77 |   +0%˄ | (run time too short) |
-| 23c     ||     69.5 |     83.1 |  +20%˄ ||    14.39 |    12.03 |  -16%˄ | (run time too short) |
-| 24a     ||   1565.4 |   2231.6 |  +43%  ||     0.64 |     0.45 |  -30%  |               0.0000 |
-| 24b     ||   1558.0 |   2210.1 |  +42%  ||     0.64 |     0.45 |  -30%  |               0.0000 |
 | 25a     ||    419.9 |    437.2 |   +4%˄ ||     2.38 |     2.29 |   -4%˄ | (run time too short) |
 | 25b     ||    176.6 |    178.4 |   +1%˄ ||     5.66 |     5.61 |   -1%˄ | (run time too short) |
 | 25c     ||   3592.8 |   3569.6 |   -1%  ||     0.28 |     0.28 |   +1%  |               0.1955 |
-| 26a     ||    446.6 |    608.2 |  +36%˄ ||     2.24 |     1.64 |  -27%˄ | (run time too short) |
-| 26b     ||    399.6 |    559.8 |  +40%˄ ||     2.50 |     1.79 |  -29%˄ | (run time too short) |
-| 26c     ||    562.8 |    725.6 |  +29%˄ ||     1.78 |     1.38 |  -22%˄ | (run time too short) |
+| 27a     ||   2207.0 |   2042.9 |   -7%  ||     0.45 |     0.49 |   +8%  |               0.0000 |
+| 27b     ||   2200.9 |   2041.7 |   -7%  ||     0.45 |     0.49 |   +8%  |               0.0000 |
+| 27c     ||    110.8 |     99.9 |  -10%˄ ||     9.02 |    10.01 |  +11%˄ | (run time too short) |
+| 28a     ||   2954.2 |   2805.0 |   -5%  ||     0.34 |     0.36 |   +5%  |               0.0442 |
+| 28b     ||   2186.1 |   2040.8 |   -7%  ||     0.46 |     0.49 |   +7%  |               0.0013 |
 | 28c     ||   3015.2 |   2889.6 |   -4%  ||     0.33 |     0.35 |   +4%  |               0.0379 |
-| 29a     ||   1548.4 |   2225.8 |  +44%  ||     0.65 |     0.45 |  -30%  |               0.0000 |
 | 29b     ||    104.3 |    105.9 |   +1%˄ ||     9.59 |     9.45 |   -1%˄ | (run time too short) |
-| 29c     ||   1632.1 |   2308.5 |  +41%  ||     0.61 |     0.43 |  -29%  |               0.0000 |
 | 2a      ||     70.8 |     71.6 |   +1%˄ ||    14.13 |    13.96 |   -1%˄ | (run time too short) |
 | 2b      ||     57.2 |     57.9 |   +1%˄ ||    17.49 |    17.28 |   -1%˄ | (run time too short) |
 | 2c      ||     43.3 |     43.8 |   +1%˄ ||    23.08 |    22.85 |   -1%˄ | (run time too short) |
 | 2d      ||     98.0 |     99.0 |   +1%˄ ||    10.20 |    10.10 |   -1%˄ | (run time too short) |
 | 30a     ||    231.9 |    236.0 |   +2%˄ ||     4.31 |     4.24 |   -2%˄ | (run time too short) |
-| 30b     ||    217.3 |    430.4 |  +98%˄ ||     4.60 |     2.32 |  -50%˄ | (run time too short) |
+| 30c     ||   2760.1 |   2635.5 |   -5%  ||     0.36 |     0.38 |   +5%  |               0.0000 |
 | 31a     ||    226.8 |    229.3 |   +1%˄ ||     4.41 |     4.36 |   -1%˄ | (run time too short) |
-| 31b     ||    243.0 |    442.0 |  +82%˄ ||     4.12 |     2.26 |  -45%˄ | (run time too short) |
 | 31c     ||    260.3 |    261.7 |   +1%˄ ||     3.84 |     3.82 |   -1%˄ | (run time too short) |
 | 32a     ||     23.5 |     23.7 |   +1%˄ ||    42.53 |    42.23 |   -1%˄ | (run time too short) |
 | 32b     ||     53.3 |     53.1 |   -0%˄ ||    18.78 |    18.81 |   +0%˄ | (run time too short) |
 | 33a     ||     29.6 |     30.0 |   +1%˄ ||    33.80 |    33.37 |   -1%˄ | (run time too short) |
 | 33b     ||     29.4 |     29.6 |   +1%˄ ||    33.99 |    33.78 |   -1%˄ | (run time too short) |
 | 33c     ||     34.2 |     34.3 |   +0%˄ ||    29.21 |    29.11 |   -0%˄ | (run time too short) |
+| 3a      ||   2504.5 |   2339.8 |   -7%  ||     0.40 |     0.43 |   +7%  |               0.0000 |
 | 3b      ||     19.3 |     19.6 |   +2%˄ ||    51.70 |    50.88 |   -2%˄ | (run time too short) |
+| 3c      ||   3056.8 |   2896.4 |   -5%  ||     0.33 |     0.35 |   +6%  |               0.0000 |
 | 4a      ||    249.6 |    249.9 |   +0%˄ ||     4.01 |     4.00 |   -0%˄ | (run time too short) |
 | 4b      ||     17.4 |     17.3 |   -0%˄ ||    57.60 |    57.82 |   +0%˄ | (run time too short) |
 | 4c      ||    289.1 |    290.2 |   +0%˄ ||     3.46 |     3.45 |   -0%˄ | (run time too short) |
+| 5a      ||   2400.2 |   2255.6 |   -6%  ||     0.42 |     0.44 |   +6%  |               0.0000 |
-| 5b      ||    142.5 |    181.9 |  +28%˄ ||     7.02 |     5.50 |  -22%˄ | (run time too short) |
+| 5c      ||   2755.9 |   2636.4 |   -4%  ||     0.36 |     0.38 |   +5%  |               0.0000 |
 | 6a      ||    172.5 |    173.8 |   +1%˄ ||     5.80 |     5.75 |   -1%˄ | (run time too short) |
 | 6b      ||    194.7 |    196.1 |   +1%˄ ||     5.13 |     5.10 |   -1%˄ | (run time too short) |
 | 6c      ||    164.6 |    165.6 |   +1%˄ ||     6.08 |     6.04 |   -1%˄ | (run time too short) |
 | 6d      ||    769.8 |    776.2 |   +1%  ||     1.30 |     1.29 |   -1%  |               0.0000 |
 | 6e      ||    172.7 |    174.4 |   +1%˄ ||     5.79 |     5.73 |   -1%˄ | (run time too short) |
 | 6f      ||   1029.0 |   1042.5 |   +1%  ||     0.97 |     0.96 |   -1%  |               0.0000 |
 | 7a      ||     81.9 |     84.5 |   +3%˄ ||    12.22 |    11.83 |   -3%˄ | (run time too short) |
 | 7b      ||     57.1 |     58.7 |   +3%˄ ||    17.50 |    17.02 |   -3%˄ | (run time too short) |
 | 7c      ||    712.5 |    731.5 |   +3%  ||     1.40 |     1.37 |   -3%  |               0.0232 |
 | 8a      ||     55.2 |     57.4 |   +4%˄ ||    18.13 |    17.41 |   -4%˄ | (run time too short) |
-| 8b      ||    131.5 |    146.2 |  +11%˄ ||     7.61 |     6.84 |  -10%˄ | (run time too short) |
-| 8c      ||   1840.8 |   2154.1 |  +17%  ||     0.54 |     0.46 |  -15%  |               0.0000 |
+| 8d      ||    496.5 |    356.0 |  -28%˄ ||     2.01 |     2.81 |  +39%˄ | (run time too short) |
 | 9a      ||   1979.8 |   2041.7 |   +3%  ||     0.51 |     0.49 |   -3%  |               0.0000 |
-| 9b      ||    144.3 |    183.8 |  +27%˄ ||     6.93 |     5.44 |  -22%˄ | (run time too short) |
 | 9c      ||   1982.0 |   2037.9 |   +3%  ||     0.50 |     0.49 |   -3%  |               0.0000 |
 | 9d      ||   2233.5 |   2292.0 |   +3%  ||     0.45 |     0.44 |   -3%  |               0.0000 |
 +---------++----------+----------+--------++----------+----------+--------+----------------------+
 | Sum     || 114785.3 | 118367.7 |   +3%  ||          |          |        |                      |
-| Geomean ||          |          |        ||          |          |   -5%  |                      |
 +---------++----------+----------+--------++----------+----------+--------+----------------------+
 |   Notes || ˄ Execution stopped due to max runs reached                                         |
 +---------++----------+----------+--------++----------+----------+--------+----------------------+

Forget what I said earlier, I cannot explain above's benchmark results. Probably, somebody used (without asking for permission :) ) the server at the same time and ruined my benchmarks. Now, it looks much better and your change improves 13 queries and decreases only the performance of a single one:

Configuration Overview - click to expand
 +Configuration Overview---------------------------------------------+------------------------------------------------+
 | Parameter        | ../job_master_2021_07_07.json                  | job_new_cardinality_estimation_2.json          |
 +------------------+------------------------------------------------+------------------------------------------------+
 |  GIT-HASH        | 3bae40c078b6e1d87b2b1190da050f3b8b30a1c0-dirty | 3bae40c078b6e1d87b2b1190da050f3b8b30a1c0-dirty |
 |  benchmark_mode  | Ordered                                        | Ordered                                        |
 |  build_type      | release                                        | release                                        |
 |  chunk_size      | 65535                                          | 65535                                          |
 |  clients         | 1                                              | 1                                              |
 |  compiler        | gcc 9.3                                        | gcc 9.3                                        |
 |  cores           | 0                                              | 0                                              |
 |  date            | 2021-07-07 10:28:41                            | 2021-07-07 09:37:10                            |
 |  encoding        | {'default': {'encoding': 'Dictionary'}}        | {'default': {'encoding': 'Dictionary'}}        |
 |  indexes         | False                                          | False                                          |
 |  max_duration    |<PHONE_NUMBER>0                                    |<PHONE_NUMBER>0                                    |
 |  max_runs        | 100                                            | 100                                            |
 |  time_unit       | ns                                             | ns                                             |
 |  using_scheduler | False                                          | False                                          |
 |  verify          | False                                          | False                                          |
 |  warmup_duration | 0                                              | 0                                              |
 +------------------+------------------------------------------------+------------------------------------------------+

 +---------++----------+----------+--------++----------+----------+--------+----------------------+
 | Item    || Latency (ms/iter)   | Change || Throughput (iter/s) | Change |              p-value |
 |         ||      old |      new |        ||      old |      new |        |                      |
 +---------++----------+----------+--------++----------+----------+--------+----------------------+
 | 10a     ||    191.3 |    191.0 |   -0%˄ ||     5.23 |     5.24 |   +0%˄ | (run time too short) |
 | 10b     ||    133.0 |    132.3 |   -1%˄ ||     7.52 |     7.56 |   +1%˄ | (run time too short) |
 | 10c     ||    306.9 |    311.6 |   +2%˄ ||     3.26 |     3.21 |   -2%˄ | (run time too short) |
 | 11a     ||     70.7 |     69.7 |   -1%˄ ||    14.14 |    14.34 |   +1%˄ | (run time too short) |
 | 11b     ||     69.9 |     69.6 |   -0%˄ ||    14.31 |    14.38 |   +0%˄ | (run time too short) |
 | 11c     ||     67.9 |     67.1 |   -1%˄ ||    14.74 |    14.90 |   +1%˄ | (run time too short) |
 | 11d     ||     28.8 |     28.8 |   -0%˄ ||    34.73 |    34.76 |   +0%˄ | (run time too short) |
 | 12a     ||     54.2 |     53.9 |   -0%˄ ||    18.45 |    18.53 |   +0%˄ | (run time too short) |
 | 12b     ||     81.9 |     81.8 |   -0%˄ ||    12.21 |    12.22 |   +0%˄ | (run time too short) |
 | 12c     ||   2187.5 |   2201.6 |   +1%  ||     0.46 |     0.45 |   -1%  |               0.5518 |
 | 13a     ||    403.3 |    404.6 |   +0%˄ ||     2.48 |     2.47 |   -0%˄ | (run time too short) |
 | 13b     ||    109.9 |    111.3 |   +1%˄ ||     9.09 |     8.99 |   -1%˄ | (run time too short) |
 | 13c     ||     48.9 |     49.4 |   +1%˄ ||    20.47 |    20.24 |   -1%˄ | (run time too short) |
 | 13d     ||    782.6 |    772.4 |   -1%  ||     1.28 |     1.29 |   +1%  |               0.0000 |
 | 14a     ||   2990.2 |   2966.6 |   -1%  ||     0.33 |     0.34 |   +1%  |               0.4460 |
 | 14b     ||   2827.3 |   2817.4 |   -0%  ||     0.35 |     0.35 |   +0%  |               0.7227 |
 | 14c     ||   3094.5 |   3077.1 |   -1%  ||     0.32 |     0.32 |   +1%  |               0.6213 |
 | 15a     ||     85.1 |     85.7 |   +1%˄ ||    11.75 |    11.66 |   -1%˄ | (run time too short) |
 | 15b     ||     83.4 |     84.2 |   +1%˄ ||    11.99 |    11.88 |   -1%˄ | (run time too short) |
 | 15c     ||     55.1 |     55.5 |   +1%˄ ||    18.13 |    18.03 |   -1%˄ | (run time too short) |
 | 15d     ||     89.1 |     85.7 |   -4%˄ ||    11.22 |    11.67 |   +4%˄ | (run time too short) |
 | 16a     ||   2734.2 |   2681.9 |   -2%  ||     0.37 |     0.37 |   +2%  |               0.3601 |
 | 16b     ||   3888.3 |   3885.9 |   -0%  ||     0.26 |     0.26 |   +0%  |               0.9602 |
 | 16c     ||   2803.4 |   2821.2 |   +1%  ||     0.36 |     0.35 |   -1%  |               0.6386 |
 | 16d     ||   2770.6 |   2788.7 |   +1%  ||     0.36 |     0.36 |   -1%  |               0.6428 |
 | 17a     ||    817.9 |    821.2 |   +0%  ||     1.22 |     1.22 |   -0%  |               0.5210 |
 | 17b     ||    671.3 |    671.2 |   -0%  ||     1.49 |     1.49 |   +0%  |               0.9860 |
 | 17c     ||    641.5 |    641.6 |   +0%  ||     1.56 |     1.56 |   -0%  |               0.9810 |
 | 17d     ||    790.3 |    787.0 |   -0%  ||     1.27 |     1.27 |   +0%  |               0.4831 |
 | 17e     ||   2139.3 |   2127.9 |   -1%  ||     0.47 |     0.47 |   +1%  |               0.4827 |
 | 17f     ||   1297.1 |   1313.8 |   +1%  ||     0.77 |     0.76 |   -1%  |               0.2401 |
 | 18a     ||   1877.3 |   1875.8 |   -0%  ||     0.53 |     0.53 |   +0%  |               0.9589 |
+| 18b     ||    199.4 |    186.5 |   -6%˄ ||     5.01 |     5.36 |   +7%˄ | (run time too short) |
 | 18c     ||   2942.2 |   3009.1 |   +2%  ||     0.34 |     0.33 |   -2%  |               0.0012 |
 | 19a     ||   3292.4 |   3274.7 |   -1%  ||     0.30 |     0.31 |   +1%  |               0.1015 |
 | 19b     ||   1531.2 |   1528.4 |   -0%  ||     0.65 |     0.65 |   +0%  |               0.5436 |
 | 19c     ||   3181.1 |   3164.4 |   -1%  ||     0.31 |     0.32 |   +1%  |               0.1059 |
 | 19d     ||   2487.3 |   2461.6 |   -1%  ||     0.40 |     0.41 |   +1%  |               0.0010 |
 | 1a      ||     30.1 |     30.2 |   +1%˄ ||    33.23 |    33.06 |   -1%˄ | (run time too short) |
 | 1b      ||     13.0 |     13.0 |   -0%˄ ||    76.92 |    77.00 |   +0%˄ | (run time too short) |
 | 1c      ||     16.1 |     15.7 |   -3%˄ ||    62.16 |    63.77 |   +3%˄ | (run time too short) |
 | 1d      ||     13.1 |     13.0 |   -1%˄ ||    76.14 |    76.79 |   +1%˄ | (run time too short) |
 | 20a     ||    986.1 |    983.3 |   -0%  ||     1.01 |     1.02 |   +0%  |               0.2161 |
 | 20b     ||    818.5 |    813.5 |   -1%  ||     1.22 |     1.23 |   +1%  |               0.0016 |
 | 20c     ||    534.4 |    537.0 |   +0%˄ ||     1.87 |     1.86 |   -0%˄ | (run time too short) |
 | 21a     ||    110.8 |    109.9 |   -1%˄ ||     9.03 |     9.10 |   +1%˄ | (run time too short) |
+| 21b     ||     91.3 |     73.6 |  -19%˄ ||    10.96 |    13.58 |  +24%˄ | (run time too short) |
+| 21c     ||    110.5 |     89.0 |  -19%˄ ||     9.05 |    11.24 |  +24%˄ | (run time too short) |
+| 22a     ||   2369.6 |   1925.6 |  -19%  ||     0.42 |     0.52 |  +23%  |               0.0000 |
+| 22b     ||   2264.2 |   1824.4 |  -19%  ||     0.44 |     0.55 |  +24%  |               0.0000 |
 | 22c     ||   3340.1 |   3308.6 |   -1%  ||     0.30 |     0.30 |   +1%  |               0.2262 |
 | 22d     ||   3458.9 |   3462.3 |   +0%  ||     0.29 |     0.29 |   -0%  |               0.8671 |
 | 23a     ||     60.7 |     61.2 |   +1%˄ ||    16.48 |    16.34 |   -1%˄ | (run time too short) |
 | 23b     ||     67.5 |     68.1 |   +1%˄ ||    14.82 |    14.68 |   -1%˄ | (run time too short) |
 | 23c     ||     69.1 |     70.1 |   +2%˄ ||    14.48 |    14.26 |   -2%˄ | (run time too short) |
 | 24a     ||   1571.8 |   1576.5 |   +0%  ||     0.64 |     0.63 |   -0%  |               0.6724 |
 | 24b     ||   1545.5 |   1553.3 |   +1%  ||     0.65 |     0.64 |   -1%  |               0.6556 |
 | 25a     ||    423.7 |    421.9 |   -0%˄ ||     2.36 |     2.37 |   +0%˄ | (run time too short) |
 | 25b     ||    176.3 |    178.3 |   +1%˄ ||     5.67 |     5.61 |   -1%˄ | (run time too short) |
 | 25c     ||   3613.0 |   3611.0 |   -0%  ||     0.28 |     0.28 |   +0%  |               0.9012 |
 | 26a     ||    445.1 |    448.9 |   +1%˄ ||     2.25 |     2.23 |   -1%˄ | (run time too short) |
 | 26b     ||    397.9 |    400.4 |   +1%˄ ||     2.51 |     2.50 |   -1%˄ | (run time too short) |
 | 26c     ||    558.9 |    560.2 |   +0%˄ ||     1.79 |     1.79 |   -0%˄ | (run time too short) |
 | 27a     ||   2177.8 |   2187.9 |   +0%  ||     0.46 |     0.46 |   -0%  |               0.2812 |
 | 27b     ||   2190.5 |   2180.5 |   -0%  ||     0.46 |     0.46 |   +0%  |               0.2459 |
 | 27c     ||    111.0 |    111.4 |   +0%˄ ||     9.01 |     8.98 |   -0%˄ | (run time too short) |
 | 28a     ||   2930.5 |   2932.0 |   +0%  ||     0.34 |     0.34 |   -0%  |               0.9834 |
 | 28b     ||   2158.2 |   2161.1 |   +0%  ||     0.46 |     0.46 |   -0%  |               0.9480 |
 | 28c     ||   2995.0 |   2989.6 |   -0%  ||     0.33 |     0.33 |   +0%  |               0.9246 |
 | 29a     ||   1538.6 |   1531.7 |   -0%  ||     0.65 |     0.65 |   +0%  |               0.9434 |
 | 29b     ||    104.5 |    105.6 |   +1%˄ ||     9.57 |     9.47 |   -1%˄ | (run time too short) |
+| 29c     ||   1618.0 |   1468.4 |   -9%  ||     0.62 |     0.68 |  +10%  |               0.0494 |
+| 2a      ||     70.7 |     53.8 |  -24%˄ ||    14.13 |    18.57 |  +31%˄ | (run time too short) |
+| 2b      ||     57.1 |     43.8 |  -23%˄ ||    17.50 |    22.81 |  +30%˄ | (run time too short) |
+| 2c      ||     43.3 |     32.9 |  -24%˄ ||    23.07 |    30.41 |  +32%˄ | (run time too short) |
+| 2d      ||     97.7 |     77.3 |  -21%˄ ||    10.23 |    12.94 |  +26%˄ | (run time too short) |
+| 30a     ||    232.2 |    171.9 |  -26%˄ ||     4.31 |     5.82 |  +35%˄ | (run time too short) |
 | 30b     ||    216.1 |    220.6 |   +2%˄ ||     4.63 |     4.53 |   -2%˄ | (run time too short) |
 | 30c     ||   2740.5 |   2756.4 |   +1%  ||     0.36 |     0.36 |   -1%  |               0.4654 |
 | 31a     ||    225.7 |    228.8 |   +1%˄ ||     4.43 |     4.37 |   -1%˄ | (run time too short) |
 | 31b     ||    250.3 |    244.2 |   -2%˄ ||     4.00 |     4.09 |   +2%˄ | (run time too short) |
 | 31c     ||    260.5 |    262.1 |   +1%˄ ||     3.84 |     3.82 |   -1%˄ | (run time too short) |
 | 32a     ||     23.5 |     23.7 |   +1%˄ ||    42.49 |    42.24 |   -1%˄ | (run time too short) |
 | 32b     ||     52.6 |     53.1 |   +1%˄ ||    19.02 |    18.81 |   -1%˄ | (run time too short) |
 | 33a     ||     29.6 |     29.8 |   +1%˄ ||    33.82 |    33.59 |   -1%˄ | (run time too short) |
 | 33b     ||     29.3 |     29.8 |   +2%˄ ||    34.12 |    33.59 |   -2%˄ | (run time too short) |
 | 33c     ||     34.3 |     34.2 |   -0%˄ ||    29.19 |    29.22 |   +0%˄ | (run time too short) |
 | 3a      ||   2482.2 |   2464.8 |   -1%  ||     0.40 |     0.41 |   +1%  |               0.0000 |
 | 3b      ||     19.3 |     19.4 |   +0%˄ ||    51.74 |    51.62 |   -0%˄ | (run time too short) |
 | 3c      ||   3043.8 |   3048.5 |   +0%  ||     0.33 |     0.33 |   -0%  |               0.6342 |
 | 4a      ||    250.9 |    248.1 |   -1%˄ ||     3.99 |     4.03 |   +1%˄ | (run time too short) |
 | 4b      ||     17.3 |     17.1 |   -1%˄ ||    57.62 |    58.31 |   +1%˄ | (run time too short) |
 | 4c      ||    287.1 |    295.1 |   +3%˄ ||     3.48 |     3.39 |   -3%˄ | (run time too short) |
 | 5a      ||   2374.4 |   2369.7 |   -0%  ||     0.42 |     0.42 |   +0%  |               0.1491 |
 | 5b      ||    147.4 |    148.6 |   +1%˄ ||     6.78 |     6.73 |   -1%˄ | (run time too short) |
 | 5c      ||   2722.3 |   2729.1 |   +0%  ||     0.37 |     0.37 |   -0%  |               0.1302 |
 | 6a      ||    171.7 |    174.1 |   +1%˄ ||     5.82 |     5.74 |   -1%˄ | (run time too short) |
 | 6b      ||    194.4 |    195.5 |   +1%˄ ||     5.14 |     5.11 |   -1%˄ | (run time too short) |
 | 6c      ||    164.8 |    165.4 |   +0%˄ ||     6.07 |     6.05 |   -0%˄ | (run time too short) |
 | 6d      ||    769.8 |    772.0 |   +0%  ||     1.30 |     1.30 |   -0%  |               0.0588 |
 | 6e      ||    172.8 |    174.2 |   +1%˄ ||     5.79 |     5.74 |   -1%˄ | (run time too short) |
+| 6f      ||   1030.5 |    848.4 |  -18%  ||     0.97 |     1.18 |  +21%  |               0.0000 |
 | 7a      ||     82.0 |     80.5 |   -2%˄ ||    12.19 |    12.42 |   +2%˄ | (run time too short) |
 | 7b      ||     57.3 |     57.6 |   +1%˄ ||    17.44 |    17.35 |   -1%˄ | (run time too short) |
 | 7c      ||    716.6 |    712.9 |   -1%  ||     1.40 |     1.40 |   +1%  |               0.7033 |
 | 8a      ||     54.9 |     56.3 |   +2%˄ ||    18.21 |    17.77 |   -2%˄ | (run time too short) |
 | 8b      ||    130.8 |    128.2 |   -2%˄ ||     7.64 |     7.80 |   +2%˄ | (run time too short) |
-| 8c      ||   1877.8 |   2144.7 |  +14%  ||     0.53 |     0.47 |  -12%  |               0.0000 |
+| 8d      ||    492.7 |    351.0 |  -29%˄ ||     2.03 |     2.85 |  +40%˄ | (run time too short) |
 | 9a      ||   1971.1 |   1919.6 |   -3%  ||     0.51 |     0.52 |   +3%  |               0.0000 |
 | 9b      ||    144.4 |    142.1 |   -2%˄ ||     6.92 |     7.04 |   +2%˄ | (run time too short) |
 | 9c      ||   1962.3 |   1938.7 |   -1%  ||     0.51 |     0.52 |   +1%  |               0.0272 |
 | 9d      ||   2219.0 |   2193.5 |   -1%  ||     0.45 |     0.46 |   +1%  |               0.0293 |
 +---------++----------+----------+--------++----------+----------+--------+----------------------+
 | Sum     || 114658.0 | 113232.9 |   -1%  ||          |          |        |                      |
 | Geomean ||          |          |        ||          |          |   +2%  |                      |
 +---------++----------+----------+--------++----------+----------+--------+----------------------+
 |   Notes || ˄ Execution stopped due to max runs reached                                         |
 +---------++----------+----------+--------++----------+----------+--------+----------------------+

This is now included in #5 .


fused_adam.so: cannot open shared object file: No such file or directory

train code:
deepspeed training/trainer.py  --per-device-train-batch-size 2 --per-device-eval-batch-size 2 --input-model EleutherAI/pythia-6.9b --local-output-dir output --deepspeed config/ds_z3_bf16_config.json --warmup-steps 0
ImportError: /home/xxxxx/.cache/torch_extensions/py310_cu117/fused_adam/fused_adam.so: cannot open shared object file: No such file or directory

There is probably an earlier error about shared libraries not being available. Do you have all the additional CUDA libraries installed, like cublas? You can see some package install lines in the training script that show what you need to have installed for your CUDA version

I am also hitting this issue.
I have installed nvidia-cublas and nvidia-pyindex.
I think the issue is actually with my deepspeed installation, if I run:
ds_report

I get:
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
async_io ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/ubuntu/.local/lib/python3.10/site-packages/torch']
torch version .................... 2.0.0+cu117
deepspeed install path ........... ['/home/ubuntu/.local/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.9.1, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.5
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.7

Which seems to suggest that fused_adam is not installed:
fused_adam ............. [NO] ....... [OKAY]
Looking at deepspeeds documentation you should be able to install fused_adam by running:
DS_BUILD_FUSED_ADAM=1 pip install deepspeed
However, this doesn't seem to work at all after running:
pip uninstall deepspeed
DS_BUILD_FUSED_ADAM=1 pip install deepspeed
ds_report

I still see:
fused_adam ............. [NO] ....... [OKAY]
At a bit of a loss tbh, any help would be greatly appreciated.

Try torch 1.13.1 ? Not sure if 2.0 is causing issues

I checked out the deepspeed repository and ran:
DS_BUILD_FUSED_ADAM=1 pip3 install .
Then
fused_adam ............. [YES] ...... [OKAY]
deepspeed seems to be running OK for me now.

deepspeed wheel compiled w. ...... torch 1.13, cuda 11.7
I have installed those libraries, but still the same errors. Please help!
libcublas                 <IP_ADDRESS>                    0    nvidia/label/cuda-11.7.1
libcublas-dev             <IP_ADDRESS>                    0    nvidia/label/cuda-11.7.1
libcurand                 <IP_ADDRESS>                    0    nvidia/label/cuda-11.7.1
libcurand-dev             <IP_ADDRESS>                    0    nvidia/label/cuda-11.7.1
libcusolver               <IP_ADDRESS>                      0    nvidia/label/cuda-11.7.1
libcusolver-dev           <IP_ADDRESS>                      0    nvidia/label/cuda-11.7.1
libcusparse               <IP_ADDRESS>                     0    nvidia/label/cuda-11.7.1
libcusparse-dev           <IP_ADDRESS>                     0    nvidia/label/cuda-11.7.1

@weifj0212 What is the result of running 'ds_report' in a terminal? Are you able post the results?

DeepSpeed C++/CUDA extension op report
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
JIT compiled ops requires ninja
ninja .................. [OKAY]
op name ................ installed .. compatible
[WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
[WARNING]  async_io: please install the libaio-dev package with apt
[WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
[WARNING]  please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
DeepSpeed general environment info:
torch install path ............... ['/home/xxx/anaconda3/lib/python3.10/site-packages/torch']
torch version .................... 1.13.1+cu117
deepspeed install path ........... ['/home/xxx/anaconda3/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.9.0, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.7
deepspeed wheel compiled w. ...... torch 1.13, cuda 11.7

@weifj0212 it looks like fused_adam isn't installed on your machine:
fused_adam ............. [NO] ....... [OKAY]
I loosely followed:
https://www.deepspeed.ai/tutorials/advanced-install/#pre-install-deepspeed-ops
To install it, the instructions suggest that running the following should installed fused_adam:
pip uninstall deepspeed
DS_BUILD_FUSED_ADAM=1 pip install deepspeed

However, this didn't work for me I had to checkout their repo and install it manually (not sure why):
git clone https://github.com/microsoft/DeepSpeed.git
cd DeepSpeed
DS_BUILD_FUSED_ADAM=1 pip3 install .

Once it finished I ran ds_report which then looked like:
fused_adam ............. [YES] ...... [OKAY]

I am a complete novice at all of this so I really don't know if the same will work for you but it might be worth a try?

Thank you very much!
fused_adam ............. [YES] ...... [OKAY]
but have new error:
/home/xxx/.cache/torch_extensions/py310_cu117/utils/utils.so: cannot open shared object file: No such file or directory.

I think you don't have all the right CUDA libs installed somehow, hard to say.

@weifj0212 in the ds_report you posted there is also a "utils" component - I didn't have to install this so I can't say with any certainty but it might be worth installing deepspeed utils too?
DS_BUILD_UTILS=1 DS_BUILD_FUSED_ADAM=1 pip install deepspeed

Thanks very much!
I changed CUDA11.7 to CUDA11.6, and run "DS_BUILD_UTILS=1 DS_BUILD_FUSED_ADAM=1 pip install deepspeed", there are not errors.

Nice. CUDA 11.7 works too (that's what I use) but i suspect something else wasn't compatible in here in the shared libraries. It's tricky.

@srowen you mean 11.7 didnt work for multi node training?

No, I mean it does work.

No, I mean it does work.

@srowen okay! thanks

Tried this on 11.8 cuda with torch 2.0.0 but still get same error - DS_BUILD_UTILS=1 DS_BUILD_FUSED_ADAM=1 pip install deepspeed
DS_REPORT
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. #033[92m[OKAY]#033[0m
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
#033[93m [WARNING] #033[0m async_io requires the dev libaio .so object and headers but these were not found.
#033[93m [WARNING] #033[0m async_io: please install the libaio-dev package with apt
#033[93m [WARNING] #033[0m If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [YES] ...... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
#033[93m [WARNING] #033[0m sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
#033[93m [WARNING] #033[0m using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO]....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO]....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference ..[NO] ....... [OKAY]
utils .................. [YES] ...... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/opt/conda/lib/python3.10/site-packages/torch']
torch version .................... 2.0.0
deepspeed install path ........... ['/opt/conda/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.9.2, unknown, unknown
torch cuda version ............... 11.8
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.8

It tells you the problem:
#033[93m [WARNING] #033[0m sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0

Is that not just a warning?

I think it actually doesn't work here. You're also using deepspeed 0.9.2, and I know we had problems with >= 0.9.0. It could be that or any other differences in your env from what we show in this repo.

@anupam-dewan I have the same problems .Have you solved it ?

DS_REPORT

I think there are lots of answers here. You haven't said what you are doing or what you tried

@Samuel1989 @srowen
I do “cd DeepSpeed DS_BUILD_FUSED_ADAM=1 pip3 install .” And I have the error like this:
File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['which', 'x86_64-linux-gnu-g++']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for deepspeed
Running setup.py clean for deepspeed
Failed to build deepspeed
ERROR: Could not build wheels for deepspeed, which is required to install pyproject.toml-based projects
However, I do 'DS_BUILD_UTILS=1 DS_BUILD_FUSED_ADAM=1 pip install deepspeed' . it doesn't have the error above.
But when  I run my program  . it  also has  errors like "utils/utils.so: cannot open shared object file: No such file or directory" .
And I do "ds_report"  The output is
[2023-06-20 11:35:42,863] [INFO] [real_accelerator.py:110:get_accelerator]
Setting ds_accelerator to cuda (auto detect)
DeepSpeed C++/CUDA extension op report
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
JIT compiled ops requires ninja
ninja .................. [OKAY]
op name ................ installed .. compatible
[WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
[WARNING]  async_io: please install the libaio-dev package with apt
[WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
[WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
DeepSpeed general environment info:
torch install path ............... ['/home/algo/mzh/ChatGLM-6B-main-0615/myenv/lib/python3.10/site-packages/torch']
torch version .................... 1.13.1+cu117
deepspeed install path ........... ['/home/algo/mzh/ChatGLM-6B-main-0615/myenv/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.9.4, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.7
This problem has troubled me for a long time, can you help me? Thank you

@weifj0212 it looks like fused_adam isn't installed on your machine: fused_adam ............. [NO] ....... [OKAY]
I loosely followed: https://www.deepspeed.ai/tutorials/advanced-install/#pre-install-deepspeed-ops
To install it, the instructions suggest that running the following should installed fused_adam:
pip uninstall deepspeed
DS_BUILD_FUSED_ADAM=1 pip install deepspeed

However, this didn't work for me; I had to checkout their repo and install it manually (not sure why):
git clone https://github.com/microsoft/DeepSpeed.git
cd DeepSpeed
DS_BUILD_FUSED_ADAM=1 pip3 install .

Once it finished I ran ds_report which then looked like:
fused_adam ............. [YES] ...... [OKAY]

I am a complete novice at all of this so I really don't know if the same will work for you but it might be worth a try?

@Samuel1989 @srowen I do “cd DeepSpeed DS_BUILD_FUSED_ADAM=1 pip3 install .” And I have the error like this:
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/usr/lib/python3.10/subprocess.py", line 524, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['which', 'x86_64-linux-gnu-g++']' returned non-zero exit status 1.
  [end of output]
   note: This error originates from a subprocess, and is likely not a problem with pip.
   ERROR: Failed building wheel for deepspeed
   Running setup.py clean for deepspeed
   Failed to build deepspeed
   ERROR: Could not build wheels for deepspeed, which is required to install pyproject.toml-based projects

However, I do 'DS_BUILD_UTILS=1 DS_BUILD_FUSED_ADAM=1 pip install deepspeed' . it doesn't have the error above. But when I run my program . it also has errors like "utils/utils.so: cannot open shared object file: No such file or directory" .
And I do "ds_report" The output is
[2023-06-20 11:35:42,863] [INFO] [real_accelerator.py:110:get_accelerator]
Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/algo/mzh/ChatGLM-6B-main-0615/myenv/lib/python3.10/site-packages/torch']
torch version .................... 1.13.1+cu117
deepspeed install path ........... ['/home/algo/mzh/ChatGLM-6B-main-0615/myenv/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.9.4, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.7

This problem has troubled me for a long time, can you help me? Thank you

@weifj0212 it looks like fused_adam isn't installed on your machine: fused_adam ............. [NO] ....... [OKAY]
I loosely followed: https://www.deepspeed.ai/tutorials/advanced-install/#pre-install-deepspeed-ops
To install it, the instructions suggest that running the following should installed fused_adam:
pip uninstall deepspeed
DS_BUILD_FUSED_ADAM=1 pip install deepspeed

However, this didn't work for me; I had to checkout their repo and install it manually (not sure why):
git clone https://github.com/microsoft/DeepSpeed.git
cd DeepSpeed
DS_BUILD_FUSED_ADAM=1 pip3 install .

Once it finished I ran ds_report which then looked like:
fused_adam ............. [YES] ...... [OKAY]

I am a complete novice at all of this so I really don't know if the same will work for you but it might be worth a try?

I have the same problem with you, so have you solved it yet?

@Samuel1989 @srowen I do “cd DeepSpeed DS_BUILD_FUSED_ADAM=1 pip3 install .” And I have the error like this:
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/usr/lib/python3.10/subprocess.py", line 524, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['which', 'x86_64-linux-gnu-g++']' returned non-zero exit status 1.
  [end of output]
   note: This error originates from a subprocess, and is likely not a problem with pip.
   ERROR: Failed building wheel for deepspeed
   Running setup.py clean for deepspeed
   Failed to build deepspeed
   ERROR: Could not build wheels for deepspeed, which is required to install pyproject.toml-based projects

However, I do 'DS_BUILD_UTILS=1 DS_BUILD_FUSED_ADAM=1 pip install deepspeed' . it doesn't have the error above. But when I run my program . it also has errors like "utils/utils.so: cannot open shared object file: No such file or directory" .
And I do "ds_report" The output is
[2023-06-20 11:35:42,863] [INFO] [real_accelerator.py:110:get_accelerator]
Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/algo/mzh/ChatGLM-6B-main-0615/myenv/lib/python3.10/site-packages/torch']
torch version .................... 1.13.1+cu117
deepspeed install path ........... ['/home/algo/mzh/ChatGLM-6B-main-0615/myenv/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.9.4, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.7

This problem has troubled me for a long time, can you help me? Thank you

@weifj0212 it looks like fused_adam isn't installed on your machine: fused_adam ............. [NO] ....... [OKAY]
I loosely followed: https://www.deepspeed.ai/tutorials/advanced-install/#pre-install-deepspeed-ops
To install it, the instructions suggest that running the following should installed fused_adam:
pip uninstall deepspeed
DS_BUILD_FUSED_ADAM=1 pip install deepspeed

However, this didn't work for me; I had to checkout their repo and install it manually (not sure why):
git clone https://github.com/microsoft/DeepSpeed.git
cd DeepSpeed
DS_BUILD_FUSED_ADAM=1 pip3 install .

Once it finished I ran ds_report which then looked like:
fused_adam ............. [YES] ...... [OKAY]

I am a complete novice at all of this so I really don't know if the same will work for you but it might be worth a try?

I have the same problem with you, so have you solved it yet?

@Samuel1989 @srowen I do “cd DeepSpeed DS_BUILD_FUSED_ADAM=1 pip3 install .” And I have the error like this:
  File "/usr/lib/python3.10/subprocess.py", line 420, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/usr/lib/python3.10/subprocess.py", line 524, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['which', 'x86_64-linux-gnu-g++']' returned non-zero exit status 1.
  [end of output]
   note: This error originates from a subprocess, and is likely not a problem with pip.
   ERROR: Failed building wheel for deepspeed
   Running setup.py clean for deepspeed
   Failed to build deepspeed
   ERROR: Could not build wheels for deepspeed, which is required to install pyproject.toml-based projects

However, I do 'DS_BUILD_UTILS=1 DS_BUILD_FUSED_ADAM=1 pip install deepspeed' . it doesn't have the error above. But when I run my program . it also has errors like "utils/utils.so: cannot open shared object file: No such file or directory" .
And I do "ds_report" The output is
[2023-06-20 11:35:42,863] [INFO] [real_accelerator.py:110:get_accelerator]
Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
utils .................. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/home/algo/mzh/ChatGLM-6B-main-0615/myenv/lib/python3.10/site-packages/torch']
torch version .................... 1.13.1+cu117
deepspeed install path ........... ['/home/algo/mzh/ChatGLM-6B-main-0615/myenv/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.9.4, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.7

This problem has troubled me for a long time, can you help me? Thank you

@weifj0212 it looks like fused_adam isn't installed on your machine: fused_adam ............. [NO] ....... [OKAY]
I loosely followed: https://www.deepspeed.ai/tutorials/advanced-install/#pre-install-deepspeed-ops
To install it, the instructions suggest that running the following should installed fused_adam:
pip uninstall deepspeed
DS_BUILD_FUSED_ADAM=1 pip install deepspeed

However, this didn't work for me; I had to checkout their repo and install it manually (not sure why):
git clone https://github.com/microsoft/DeepSpeed.git
cd DeepSpeed
DS_BUILD_FUSED_ADAM=1 pip3 install .

Once it finished I ran ds_report which then looked like:
fused_adam ............. [YES] ...... [OKAY]

I am a complete novice at all of this so I really don't know if the same will work for you but it might be worth a try?

I have the same problem with you, so have you solved it yet?
Me too, have you solved it yet?

Please consider trying this:
git clone https://github.com/microsoft/DeepSpeed.git
cd DeepSpeed
DS_BUILD_UTILS=1 DS_BUILD_FUSED_ADAM=1 pip install .

If you encounter a mismatch between cuda and gcc, consider lowering the gcc version and running it again. I hope this helps.


Solution needed for Leetcode Problem

Add a simple solution with brief explanation for this problem Jump Game

please assign to me

Hey, I would like to add a java explanation for this code. Please assign me this issue.

@sumitmohite0501 @SugatoBagchi This issue is open for all. You can contribute to this. Make sure to follow the contribution guidelines.

#46 Opened a PR.


policy-rc.d denied execution of reload

There is a cron.daily job to rotate logs inside container and the syslog-ng log rotation tries to reload syslog-ng. When doing to, the reload is denied, though I don't know exactly why. I'm getting cron.daily reports that:
/etc/cron.daily/logrotate:
invoke-rc.d: policy-rc.d denied execution of reload.

syslog-ng seems to be running and logging OK though. Is this related to runit running the syslog deamon and logrorate script should be modified? Should the postrotate be changed from
invoke-rc.d syslog-ng reload > /dev/null

to
sv reload syslog-ng

?

This issue has been fixed. Thanks for reporting it and sorry for the delay.


Generate event when tags are added/removed

and complete the TODO of changing to findAndModifyMongo command

Coverage increased (+0.04%) to 91.343% when pulling b91cc757eb0455977d6c4dec11c7d9871d133ca6 on zyoung51:feature/event-on-tagging into ff5cdb15c0b11910948d4561ff7f0150ae382d57 on RackHD:master.

👍  after semicolons.

+1

Coverage increased (+0.04%) to 91.355% when pulling 0e3c4e0b729e58774d77328f1af41e17a3fbbbf7 on zyoung51:feature/event-on-tagging into 3a66f6eb8d71d4d4b032bed16f167c902d72733d on RackHD:master.


Run SonarCloud only on pull requests to master

Run SonarCloud less often to reduce build times for normal CI builds.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: nathangethub sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


please publish version 0.3 to Hex

the title says it all.
Thank you.

Sorry for the delay, I've released the current main branch to hex as 0.3.0.


Can I archive downloaded videos without skipping downloading videos already in the archive?

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm asking a question and not reporting a bug or requesting a feature
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2023.07.06 (update instructions) or later (specify commit)
[X] I've searched known issues and the bugtracker for similar questions including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Please make sure the question is worded well enough to be understood
I want to archive(to log) all videos that yt-dlp downloads. I can do it using --download-archive FILE option, but in this case yt-dlp skips videos that have been downloaded - I don't want that.
There's an option --no-download-archive, but if I use it, the archive does not get updated.
Provide verbose output that clearly demonstrates the problem

[ ] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[ ] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
No response

If you don't want to take advantage of the feature you can manually print to a "download archive" text file instead:
--print-to-file "%(extractor)s %(id)s" "download_archive.txt"


make Block cheaply cloneable

Since Block appears in APIs over multiple crates (mostly as &Block) and it doesn’t offer mutation methods, it would be reasonable to assume that an implementation of a consumer should want to hold on to the bytes it got — which today requires making a copy. The only adversely affected method would be Block::into_inner(), which probably would become less useful by such a change, as moving between Arc and Vec always requires an allocation.
What do you think?

To be honest, I haven't spend much time thinking about the Block interface, even when it was created. I'm still busy with the lower level things. I've no idea when I'll find to think this fully through. Hence I'd leave it to your judgement if it makes sense.

What's the use case you're looking to support? Is it that a consumer gets a &Block and wants their own Block without having to deal with lifetimes?

Yes, @MarcoPolo, that was what triggered my question. If we concede that cloning is expensive and that consumers may want to own them, then some traits should be changed to take Block instead of &Block — my estimate was that making Block cheaply cloneable would avoid the associated source code churn.


Iframe inside a local html is only loading child page in android

We are trying to develop an offline feature in which we will be downloading bunch of HTML files and load in local. We are facing a strange issue in webview  local HTML if it has a iframe.
When its loaded only iframe content is getting loaded in webview.
Can you please let me know what could be the issue here ?

WebView
    <WebView         source={{           uri: "file:///data/user/0/com.example/files/parent.html"         }}         onError={error => {           console.log('WebView onError - ', error);         }}         allowsFullscreenVideo         bounces={false}         renderError={() => {           return null;         }}         allowFileAccess={true}         allowFileAccessFromFileURLs={true}         allowingReadAccessToURL={true}         allowUniversalAccessFromFileURLs={true}         onMessage={event => {           console.log('WebView onMessage- ', event);         }}         mediaPlaybackRequiresUserAction={false}         allowsInlineMediaPlayback={true}         javaScriptEnabled={true}         domStorageEnabled={true}         startInLoadingState={true}         originWhitelist={['*']}         mixedContentMode={'always'}         style={{ flex: 1 }}       />
Parent.html

Child.html

Also having this issue. Hoping for a solution

stil searching for solution

same problem

Did you solve it now? I had the same problem; I loaded the html via a local bundled resource, which worked fine on IOS devices, but on Android devices some content didn't render properly, but when I started a http server, and all content can be rendered normally,


Feature Implementations

The website mentions the use of smart contracts as well as other important features like ZK-proofs. Where are the implementations in the repository?

We will probably implement the smart contract around April next year.
we will be Real time open source.
Thank you for your attention

Did mainnet launch?

@songnob the mainnet lauched  ,but only forked  , many functions are coming.


Panic when calling Tree.FindEntry

If you have a tree containing a directory called, say, foo then calling Tree.FindEntry("foo/bar") followed by Tree.FindEntry("foo/.") causes a panic. This is a fairly  obscure set of conditions, but it is a panic, so I'm filing an issue.
Here's a diff to the test suite demonstrating the problem:
$ git diff plumbing/object/tree_test.go
diff --git a/plumbing/object/tree_test.go b/plumbing/object/tree_test.go
index 889c63a..041811a 100644
--- a/plumbing/object/tree_test.go
+++ b/plumbing/object/tree_test.go
@@ -127,6 +127,15 @@ func (s *TreeSuite) TestFindEntryNotFound(c *C) {
        c.Assert(err, Equals, ErrEntryNotFound)
 }

+func (s *TreeSuite) TestFindEntryPanic(c *C) {
+       e, err := s.Tree.FindEntry("vendor/not-found")
+       c.Assert(e, IsNil)
+       c.Assert(err, Equals, ErrEntryNotFound)
+       e, err = s.Tree.FindEntry("vendor/.")
+       c.Assert(e, IsNil)
+       c.Assert(err, Equals, ErrEntryNotFound)
+}
+
 // Overrides returned plumbing.EncodedObject for given hash.
 // Otherwise, delegates to actual storer to get real object
 type fakeStorer struct {

I think this happens because this call to filepath.Join() returns a path with less parts than expected, which means pathParts becomes an empty slice, so looking up pathParts[0] creates an index out of range error.
I'm avoiding this in my code by calling filepath.Clean() on the path, but I'm not sure if this is a general solution that will work for everyone.

Also getting this problem. There's a problem with the implementation of the tree path cache.


⚠ Bofrost.be has degraded performance

In 0e77be6, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1280 ms

Resolved: Bofrost.be performance has improved in a20ea24 after 7 minutes.


Updating Card.io library doubled the size of my app.

I was about to release a new update to my app, noticed my app size was 20mb compared to the 10mb it was in the previous release.
I used git bisect to figure out where the problem came from and it narrowed it down to a commit where I updated all the libraries that I use. So I reverted all the library updates one by one to see what library was causing it, and it was Card.io.
Built the app with version 5.0.1 of this library, 11.1mb. updated the library to 5.1.1 (and eventually 5.1.2) and my app size jumped to 20.8mb.
Are you guys aware of this? I removed the library before upgrading as you mentioned in your readme so that's not the problem. There seems to be something wrong with your library with the recent two updates.

Thanks for letting me know. :+1:

@MiralDesai Can u share how did u achieve the different split abi's

@goonerDroid I just haven't updated the library. It's not the right thing to do so I don't recommend it but it works I guess

I want to echo @MiralDesai's comments.  I can't justify doubling my app size and the overhead of managing multiple apks is too high so I will be downgrading to a previous release.

Wait, what's the gain for the new version that supports 64 bit architectures that warrants the APK size explosion?

My apk file exploded after integrating the new paypal api which i think comes with card.io. so card.io is the culprit? how do i exclude card.io from the compile?

I also tried same to prevent increse size of apk,but app is crashing
http://stackoverflow.com/questions/38887998/paypal-is-not-working-after-removing-io-card


Fix Fermi level regex in different versions of FHIaims

@ndaelman-hu @ladinesa I implemented the regex for the two fermi_level quantities (depending on the version).
Please, feel free to test it (mainly @ndaelman-hu) and to improve the regexes (mainly @ladinesa).

@ladinesa Do the "missing" metadata in the parsers mean that the copy of the nomad develop branch has to be updated here?

Okay, so firstly, let me link this to the corresponding Gitlab issues, namely https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/939 and https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1131
@JosePizarro3 I didn't find any update to the FHI tester. One of us should update them.
Note that the current test is very subtle, but didn't pick up on the issue.
In test_fhiaimsparser.py there are 3 places where the Fermi energy is called: twice in Si_band_dos (v071914_7) and once in Fe_band_spinpol (v151211).
It is not directly tested in the old version (which you corrected), only the band gap is.
So, please add tests for Si_band_dos, similar to Fe_band_spinpol and maybe use comments to specify that two different versions / formats are being tested.
Secondly -and this is my own pet peeve (@ladinesa feel free to comment)- the reference data is too unclear, as it is referenced in SI units.
I suggest to, henceforth, always put the reference data in the same units as reported and convert the extracted data.
You can look at test_dftu() for an example.
This will greatly improve casual lookups.

If you are refering to the no-member error due to the missing metainfo defs like HubbardModel, we simply have to wait until a new nomad pypi is released then this will be automatically resolved. We do not need to update the parsers, we just rerun the actions. For the value comparisons, I did the opposite before but I agree that it is easier to look at to compare with the raw value and convert the nomad value. Can you please do this to all tests?

I added some tests now in test_band_spinpol and test_band_silicon.

@JosePizarro3 Thx! If passes your pytest locally, then I'm fine.
@ladinesa I will progressively implement the change in the future.

@ladinesa Nathan and I talked and he will push his changes later on. If you give your final approval I will merge.


WIP: Introducing 0.0.2 version of ocs csv

This is to get the discussion started around how we are going to do the next release after 0.0.1 version.

Will we use alpha/beta channels going forward? Or dev-preview/tech-preview/stable, etc?
What channel would 0.0.2 be in? If it's in the same channel, there is no reason any deployment should use 0.0.1 anymore. So remove that directory altogether?

I'm pretty sure we need to keep 0.0.1 and all previous csv versions.
Right. We do need the previous versions in there but I don't think they have to be part of a channel in the package.yaml.

We could consider something like 0.0.1 to always represent the "latest" nightly build, and 0.0.2 to always represent the dev version being tested
I am good with that to start with.

I'm pretty sure we need to keep 0.0.1 and all previous csv versions.

Right. We do need the previous versions in there but I don't think they have to be part of a channel in the package.yaml.

We could consider something like 0.0.1 to always represent the "latest" nightly build, and 0.0.2 to always represent the dev version being tested

I am good with that to start with.

@jarrpa @kshlm do either of you want to weigh in here? The idea would be that 0.0.1 and 0.0.2 would be special versions only used during development.
0.0.1 would point to the nightly build for whatever branch that's in question, and 0.0.2 would point to the build generated from the PR (or dev environment's) source tree. Our automation would ensure that in each PR we can update from the latest nightly to the code in the PR. This would hopefully help catch issues where we might be introducing a change that breaks upgradability.

Right. We do need the previous versions in there but I don't think they have to be part of a channel in the package.yaml.

@sacharya the channel stuff is where this gets complicated. Here's an alternative we could consider.
We omit the 0.0.2 CSV in the nightly. So when we install a new catalog source with 0.0.2, if 0.0.1 is installed it automatically get's updated. this would avoid having multiple channels entirely.
For the functional test we'd install the nightly, then update to a new catalog source that has 0.0.2 included.

I'm pretty sure we need to keep 0.0.1 and all previous csv versions.

Right. We do need the previous versions in there but I don't think they have to be part of a channel in the package.yaml.

We could consider something like 0.0.1 to always represent the "latest" nightly build, and 0.0.2 to always represent the dev version being tested

I am good with that to start with.

@jarrpa @kshlm do either of you want to weigh in here? The idea would be that 0.0.1 and 0.0.2 would be special versions only used during development.
0.0.1 would point to the nightly build for whatever branch that's in question, and 0.0.2 would point to the build generated from the PR (or dev environment's) source tree. Our automation would ensure that in each PR we can update from the latest nightly to the code in the PR. This would hopefully help catch issues where we might be introducing a change that breaks upgradability.

I'm okay with this. The CSV versions in the upstream are currently meaningless, it's what'll come out of the downstream builds that we'll want to handle.

So it looks like one action item from this would be to create a new catalog source image with 0.0.2 that "replaces" 0.0.1.
The UPGRADE_FROM image could be nightly snapshots with 0.0.1 only.
What do you think @kshlm ?

I think we should switch to proper version numbers now. We have a release-4.2 branch, on which should have CSVs with version 4.2.0. On the master branch, we should be setting the version to the next release, 4.3.0.
You would then have 4.3.0 replacing 4.2.0 in the CSV.

@kshlm Updated the PR. Please review. This is targeting master, so the CSV is pointing to :latest tags. In the 4.x branches, we'd point them to their respective images.

I'd rather not see any changes to the CSV versions in this repo that don't also come with changes to the e2e tests for upgrade. I'm most in favor of David's latest suggestion.

I'd rather not see any changes to the CSV versions in this repo that don't also come with changes to the e2e tests for upgrade. I'm most in favor of David's latest suggestion.

That is exactly what this is. #189 is almost good to go which provides the e2e upgrade testing, but we need the new csv (and registry image) first. This is inline with @davidvossel's suggestion AFAIK.

/retest


Can't use User Assigned Managed Identity to obtain SSL certificate from Key Vault for APIM custom domain

Hi,
I'm experiencing an issue when I try to use an User Assigned Managed Identity to obtain a custom TLS/SSL certificate for my API Management instance from Azure Key Vault, as described here: https://docs.microsoft.com/en-us/azure/api-management/api-management-howto-use-managed-service-identity#use-ssl-tls-certificate-from-azure-key-vault-ua.
I want to use the certificate for configuring a custom domain for the gateway:

The error message I get:

It does work when I select APIM's system assigned managed identity. Both the user assigned and system assigned identities have the same access policies on given key vault.

Am i doing something wrong or are user assigned managed identities not supported?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1e6fa6cc-8307-4391-7cd9-2b6d5ccd9b7b
Version Independent ID: c78c84c8-99e6-9746-f3f2-f1f5098582a3
Content: Use managed identities in Azure API Management
Content Source: articles/api-management/api-management-howto-use-managed-service-identity.md
Service: api-management
GitHub Login: @dlepow
Microsoft Alias: danlep

Hello @tijmenamsing - Thank you for your feedback! We will review and update as appropriate.

Thanks @mike-urnun-msft. I just found out that it seems to be related to the firewall of the key vault. I can only fetch the certificate using the User Assigned Managed Identity if I enable "allow access from all networks" on the key vault. The weird thing is that for System Assigned Managed Identity, I can enable the key vault's firewall. When enabled, I have selected the option to allow trusted microsoft services and I've added a rule to allow access from the subnet that my APIM instance is integrated with (I connected APIM to a virtual network in internal mode).

Allow access from all networks is not an option for me. Do you have any clues on why I only have this problem with user assigned managed identities?

@tijmenamsing - Apologies for slow response to the issue you point out. You are correct that the issue here is likely related to the firewall of the key vault. We've updated the article to note that, in the case of a key vault with a firewall enabled, you may only use the API Management instance's system-assigned managed identity. Thanks for letting us know.
#please-close


Howdy to automatically switch between different cameras.

When I use laptop camera, howdy works great and when I dock it up.  I use an external HDMI monitor based camera and at this time I usually keep the lid of the laptop closed and so the earlier camera is not available. Howdy goes by the config and still tries with the laptop camera and fails.
If I do a sudo howdy config and change the device_path to HDMI based camera and howdy starts working again.
It will be great if howdy can switch between available cameras automatically, with out users having to change the config every time, may be it can take two device paths in the config and try both of them.

Please use the search function and take a look at #515


Component level tokenization - Code snippet

User Story

As a [user role below]:
Carbon adopter

I need to:
theme various tokens for the Code snippet component

so that I can:
apply an overall theme in my application

Additional information

This is component-level tokenization work identified in the epic https://github.com/carbon-design-system/ibm-dotcom-library/issues/2503
Latest code can be found at _code-snippet.scss and _copy-button.scss changes in: https://github.com/carbon-design-system/carbon/compare/master...asudoh:alternate-type-space

Acceptance criteria

[ ] Component-level tokens applied in Carbon

Carbon rejected the pull request as they do not have the resources to take on the work.  Closing the epic (#2503) and all of the linked issues.
Copy of Jeff's email summarizing the next steps:
We have worked extensively with Akira Sudoh (former Carbon team member) and have recently created a comprehensive code contribution to Carbon to introduce API tokens that would enable the expressive theme. After review from the Carbon team, they have unfortunately rejected our code proposal, stating that it would be too much for their team to support at this time. They have taken considerable resource reductions and their team is much smaller than it was previously which most likely contributed to this decision on their end.
With this news, our team is essentially back to supporting a proxy on our end. However, we have had very valuable learnings when working with Akira, and plan to execute on this new contingency plan:

Identify key Carbon components we would like to support in the expressive theme ◦We can begin with a set of essential components we would like to support, then expand over time

We will make full use of our automated visual regression tool that we recently stood up. This will be extended to our standalone storybook instance with the Carbon components that have the expressive theme enabled. The work that needs to be done here: ◦Activate visual regression testing for Carbon expressive components
-Create additional e2e tests to cover all variants

Take all of Akira's work, and apply the tokenization to the IBM.com Library instead ◦Rather than a hardcoded way of applying the proxy, we are thinking of having the API tokens as an offering from our team instead. This can be ultimately used by other IBM.com Library adopters if the need arises, plus adds more flexibility and scalability on our end

Updates to documentation/design kits for our adopters ◦We will need to own and create all supporting documentation for the expressive theme, so there is some upfront and continuing work around this.

We have almost all of the carbon components with a hardcoded proxy already in place so it serves as a very good starting point. We will be applying the action items above in order to future proof it as a manageable solution going forward.


<EMAIL_ADDRESS>

https://github.com/Musliadi23/Musliadi/compare/main...Musliadi23-patch-1?quick_pull=1


Charts guide

Description
Reorganized chart component content and added an overview guide.
Reviewer Notes
Please sanity check that I didn't break links by updating the hierarchy.
Related Issue(s) / Ticket(s)
https://github.com/newrelic/developer-website/issues/975

@alexronquillo Thanks for submitting this PR, I'll pull it down to tomorrow and give it a glance, unfortunately the Amplify AWS preview isn't work for this change. :(

Not sure why Amplify isn't posting back the result, but the build was successful. Here is a link to the preview: https://pr-1030.dlyi50rq9kt6c.amplifyapp.com/
FWIW, assuming the build succeeds, all PR previews will be in this structure. Replace the number in that link above with the PR number and you'll be able to access any open PR preview.

@jerelmiller works for me! That's an awesome feature!

@jerelmiller ah good to know! it be cool if those preview links appeared in the github actions section of a PR, like netlify does it. :)

ah good to know! it be cool if those preview links appeared in the github actions section of a PR

It sorta does that, but seems like Amplify doesn't do a great job sometimes of reporting back on the status. Not sure why. We'll have to see if Amplify has any other means of displaying the preview status or not.

Great guide!


Miss reporting of % saved if optimizing from MB to KB

If the image originally was in MB and then after optimization is in KB, the % saved column doesn't report well the difference:

Closign as it has been addressed in Release v1.2.0


core-components: add new LogViewer component + use in plugins

Hey, I just made a Pull Request!
This introduces a new LogViewer component. It's a replacement for the existing usage of LazyLog with something that's based on MUI and has support for improved filtering and searching.

Since it's now based on MUI it supports theming:

The filtering is improved, supporting search across colored sections:

Work towards #7956
:heavy_check_mark: Checklist

[x] A changeset describing the change and affected packages. (more info)
[ ] Added or updated documentation
[x] Tests for new functionality and regression tests for bug fixes
[x] Screenshots attached (for UI changes)
[x] All your commits have a Signed-off-by line in the message. (more info)

huge PR huge feature 🙌

Oh but now it has conflicts.


cunicu: 0.5.65 -> 0.5.68

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/project/cunicu/versions.
meta.description for cunicu is: Zeroconf peer-to-peer mesh VPN using Wireguard® and Interactive Connectivity Establishment (ICE)
meta.homepage for cunicu is: https://cunicu.li
Updates performed

Golang update

To inspect upstream changes

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 0.5.68 with grep in /nix/store/01a0687kb42nh3ry659jrc3x8wi1jd8z-cunicu-0.5.68
found 0.5.68 in filename of file in /nix/store/01a0687kb42nh3ry659jrc3x8wi1jd8z-cunicu-0.5.68

Rebuild report (if merged into master) (click to expand)

2 total rebuild path(s)

2 package rebuild(s)

First fifty rebuilds by attrpath

cunicu

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/01a0687kb42nh3ry659jrc3x8wi1jd8z-cunicu-0.5.68 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A cunicu https://github.com/r-ryantm/nixpkgs/archive/41822d70a5b288ce9e13a821ab65d33b7a32c861.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/41822d70a5b288ce9e13a821ab65d33b7a32c861#cunicu

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/01a0687kb42nh3ry659jrc3x8wi1jd8z-cunicu-0.5.68
ls -la /nix/store/01a0687kb42nh3ry659jrc3x8wi1jd8z-cunicu-0.5.68/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review

x86_64-linux

:white_check_mark: 1 package built:

cunicu

Maintainer pings
cc @stv0g for testing.

[!TIP]
As a maintainer, if your package is located under pkgs/by-name/*, you can comment @NixOS/nixpkgs-merge-bot merge to automatically merge this update using the nixpkgs-merge-bot.

Add a :+1: reaction to pull requests you find important.

@NixOS/nixpkgs-merge-bot merge


Difference between Dilate and Erode morphological operation

The screenshot below show that there is a difference between an Erode of radius 5 and a Dilate of radius 5 both obtained using ImageMath and given the same isotropic volume of spacing 1 as input.

Background
While looking why ImageMath-Test9-Compare was failing in the CircleCi build associated with #625 , I found that eroding and dilating using a value of 5 gives different result.
Version of TubeTK
The information reported below are based of master at fecfd05 without any of the change specific to #625.
Downloading Baseline

Baseline GDS0015_1.mha fetched by executing ctest -R ImageMath-Test9

$ ctest -R ImageMath-Test9
Test project /home/jcfr/Projects/TubeTK-Refactor-BuildInITK-Release/TubeTK-build
    Start 593: ImageMath-Test9_fetchData
1/4 Test #593: ImageMath-Test9_fetchData ...........   Passed    0.00 sec
    Start 594: ImageMath-Test9
2/4 Test #594: ImageMath-Test9 .....................   Passed    0.02 sec
    Start 595: ImageMath-Test9-Compare_fetchData
3/4 Test #595: ImageMath-Test9-Compare_fetchData ...   Passed    0.00 sec
    Start 596: ImageMath-Test9-Compare
4/4 Test #596: ImageMath-Test9-Compare .............   Passed    0.03 sec
[...]

$ md5sum /home/jcfr/Projects/TubeTK-Refactor-BuildInITK-Release/TubeTK-build/MIDAS_Data/GDS0015_1.mha
7c243e2237956c5325af480fa43e4b10  /home/jcfr/Projects/TubeTK-Refactor-BuildInITK-Release/TubeTK-build/MIDAS_Data/GDS0015_1.mha

Generating outputs with ImageMath

Dataset GDS0015_1_dilate_5.mha obtained using:

/home/jcfr/Projects/TubeTK-Refactor-BuildInITK-Release/TubeTK-build/bin/ImageMath "/home/jcfr/Projects/TubeTK-Refactor-BuildInITK-Release/TubeTK-build/MIDAS_Data/GDS0015_1.mha" "-M" "1" "5" "255" "0" "-W" "2" /tmp/GDS0015_1_dilate_5.mha

Dataset GDS0015_1_erode_5.mha obtained using:

/home/jcfr/Projects/TubeTK-Refactor-BuildInITK-Release/TubeTK-build/bin/ImageMath "/home/jcfr/Projects/TubeTK-Refactor-BuildInITK-Release/TubeTK-build/MIDAS_Data/GDS0015_1.mha" "-M" "0" "5" "255" "0" "-W" "2" /tmp/GDS0015_1_erode_5.mha

Volume info

Baseline GDS0015_1.mha

Output  GDS0015_1_dilate_5.mha

Output  GDS0015_1_erode_5.mha

@aylward These are the results I was discussing with you earlier this afternoon.

By directly setting the radius on the BinaryBallStructuringElement  and commenting the for loops. (see snipped below), I get the expected erosion (expect in the corner).

typedef itk::BinaryBallStructuringElement<PixelType, VDimension>
        BallType;
      BallType ball;
      ball.SetRadius( radius );
      ball.CreateStructuringElement();

      typedef itk::ErodeObjectMorphologyImageFilter
                   <ImageType, ImageType, BallType>       ErodeFilterType;
      typedef itk::DilateObjectMorphologyImageFilter
                   <ImageType, ImageType, BallType>       DilateFilterType;
      switch( mode )
        {
        case 0:
          {
          //for( int r=0; r<radius; r++ )
            {
            typename ErodeFilterType::Pointer filter =
              ErodeFilterType::New();
            filter->SetBackgroundValue( backgroundValue );
            filter->SetKernel( ball );
            filter->SetObjectValue( foregroundValue );
            filter->SetInput( imIn );
            filter->Update();
            imIn = filter->GetOutput();
            }
          break;
          }
        case 1:
          {
          //for( int r=0; r<radius; r++ )
            {
            typename DilateFilterType::Pointer filter =
              DilateFilterType::New();
            filter->SetKernel( ball );
            filter->SetObjectValue( foregroundValue );
            filter->SetInput( imIn );
            filter->Update();
            imIn = filter->GetOutput();
            }
          break;
          }
        }

I think your results are right even for the corners. This is what I would expect using a ball as kernel. You could try using a cube to retrieve the original results with straight corners.

Given the implementation, the results are indeed expected.

JC, are you talking about performing an opening (erosion followed by a dilation) and then compare to the original image?

Looking at the structuring element, the morphological operator is correct,
but the use of looping for erosion equivalence does not hold.    Converting
the code to use a single ball structuring element of radius r instead.
Great catch JC!
On Thu, Mar 3, 2016 at 5:32 PM, Francois Budin<EMAIL_ADDRESS>wrote:

JC, are you talking about performing an opening
https://en.wikipedia.org/wiki/Opening_(morphology) (erosion
followed by a dilation) and then compare to the original image?
—
Reply to this email directly or view it on GitHub
https://github.com/KitwareMedical/TubeTK/issues/627#issuecomment-191996544
.

--
Stephen R. Aylward, Ph.D.
Senior Director of Operations, North Carolina, Kitware, Inc.
http://www.kitware.com and http://www.aylward.org
(919) 969-6990 x300

Fixed in https://github.com/KitwareMedical/TubeTK/commit/0413e309cdedde5114ba8087fcc1cdb08136d073


Custom View for front-page?

is it possible to set a custom view for the front-page/home?
I've created a custom view from the training (https://training.plone.org/5/volto/custom-views.html) and tried to set this as default view for the front-page. I can assign this to any sub-page in my volto installation but not for the front-page. I also tried to set the view via ZMI (/Plone/front-page/manage_propertiesForm) but nothing happens. Next step I tried was to add a new page, assign the new view and set it with "Select a content item as default" (/select_default_view).. but I still get the "Welcome to Volto!"...

@TBCcrash2064 there is a PLIP to make the Plone Site Root a Dexterity object with a custom view. This will come with Plone 6. Though, this does not help you today.
What we do as a workaround so far is that we create a "Homepage" Dexterity content type that we put in the portal root. Then we register a Volto view on that type that shows up by default on the  portal root.
The code for this is in our kitconcept.voltodemo package. Here is how we set up the homepage content object in the portal root:
https://github.com/kitconcept/kitconcept.voltodemo/blob/75111a19dd874d95b6f985cdad1f088fbeff001c/src/kitconcept/voltodemo/setuphandlers.py

The PLIP has been merged some time ago, and available in Plone 6. For Plone 5 there has been a workaround since some years ago.


Support thread pool for restore multi segments for one table in tiflash restart

What problem does this PR solve?
Issue Number: close #7188
Problem Summary:
What is changed and how it works?
Use threadpool to optimize initStores, to parallelize each segment in each table to restore files
For test the typical workload with 600G data
restart time for initStores --> 13s --> 2s
Check List
Tests

[ ] Unit test
[ ] Integration test
[ ] Manual test (add detailed scripts or steps below)
[ ] No code

Side effects

[ ] Performance regression: Consumes more CPU
[ ] Performance regression: Consumes more Memory
[ ] Breaking backward compatibility

Documentation

[ ] Affects user behaviors
[ ] Contains syntax changes
[ ] Contains variable changes
[ ] Contains experimental features
[ ] Changes MySQL compatibility

Release note

shorten TiFlash startup time

/run-all-tests

/run-all-tests

/cc @breezewish @JaySon-Huang

/merge