refactor: make dfschema wrap schemaref

Which issue does this PR close?
Closes #4680
follow on #8905
Rationale for this change
see related issue #4680, #7944 and #8905
See also mailing list discussion: https://lists.apache.org/thread/79rpwd22bncdxbkwlq842rwr485kwwyh
What changes are included in this PR?
change the DFSchema and remove DFField struct
from
pub struct DFSchema {
    /// Fields
    fields: Vec<DFField>,
    /// Additional metadata in form of key value pairs
    metadata: HashMap<String, String>,
    /// Stores functional dependencies in the schema.
    functional_dependencies: FunctionalDependencies,
}

to
pub struct DFSchema {
    /// Inner Arrow schema reference.
    inner: SchemaRef,
    /// Optional qualifiers for each column in this schema. In the same order as
    /// the `self.inner.fields()`
    field_qualifiers: Vec<Option<OwnedTableReference>>,
    /// Stores functional dependencies in the schema.
    functional_dependencies: FunctionalDependencies,
}

and update the related API
Are these changes tested?
test with existing tests
Are there any user-facing changes?
New API for create DFSchema

pub fn new_with_metadata(fields: Vec<Field>, metadata: HashMap<String, String>)->Self>
pub fn try_from_qualified_schema<'a>(qualifier: impl Into<TableReference<'a>>,schema: &Schema,)->Result<Self>
pub fn from_qualified_fields(qualified_fields: Vec<(Option<OwnedTableReference>, Arc<Field>)>,metadata: HashMap<String, String>) -> Result<Self>
pub fn from_field_specific_qualified_schema<'a>(qualifiers: Vec<Option<impl Into<TableReference<'a>>>>, schema: &SchemaRef) -> Result<Self>

New API to find qualified Field, the old API only return Field

pub fn qualified_field(&self, i: usize) -> (Option<&OwnedTableReference>, &Field): returns an immutable reference of a specific Field and its qualifier by an offset
pub fn qualified_field_with_name(&self, qualifier: Option<&TableReference>, name: &str) -> Result<(Option<&OwnedTableReference>, &Field)>: find the qualified field with the given name
pub fn qualified_fields_with_unqualified_name(&self, name: &str) -> Vec<(Option<&OwnedTableReference>, &Field)>: Find all fields that match the given name and return them with their qualifier
pub fn qualified_field_with_unqualified_name(&self,name: &str) -> Result<(Option<&OwnedTableReference>, &Field)>: Find the qualified field with the given unqualified name
pub fn qualified_field_from_column(&self,column: &Column) -> Result<(Option<&OwnedTableReference>, &Field)>: Find the field with the given qualified column
pub fn iter(&self) -> impl Iterator<Item = (Option<&OwnedTableReference>, &FieldRef)>: Iterate over the qualifiers and fields in the DFSchema

Other new API

pub fn columns(&self) -> Vec<Column>: return all Columns for the schema
pub fn columns_with_unqualified_name(&self, name: &str) -> Vec<Column>: find all fields that match the given name and convert to column

I think the change is too large to review at once and easily conflicts with new changes. Is there a way to rewrite them partially?

I think the change is too large to review at once and easily conflicts with new changes. Is there a way to rewrite them partially?

Hi, @jayzhan211. Thanks for your feedback. I understand your concern. The number of changes is indeed large and may cause some conflicts. However, I want to emphasize that although there are many changes, most focus on dfschema.rs. These changes are related to each other, so splitting them into multiple small PRs may be more complicated and may even introduce new problems. I will try my best to keep the code updated to reduce possible conflicts. Thank you again for your understanding and patience.

this is a benchmark result; the baseline is cf0f8eececd37f593b811320f89c0edd00fd3945 in the main.
logical_select_one_from_700
                        time:   [636.48 µs 637.79 µs 639.20 µs]
                        change: [+17.618% +18.087% +18.592%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low severe
  2 (2.00%) low mild
  5 (5.00%) high mild
  1 (1.00%) high severe

physical_select_one_from_700
                        time:   [3.7188 ms 3.7238 ms 3.7295 ms]
                        change: [+24.347% +24.897% +25.417%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

logical_select_all_from_1000
                        time:   [23.270 ms 23.296 ms 23.324 ms]
                        change: [-72.737% -72.690% -72.643%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  6 (6.00%) high mild
  2 (2.00%) high severe

Benchmarking physical_select_all_from_1000: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 16.1s, or reduce sample count to 30.
physical_select_all_from_1000
                        time:   [161.09 ms 161.35 ms 161.63 ms]
                        change: [-74.231% -74.190% -74.138%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 16 outliers among 100 measurements (16.00%)
  6 (6.00%) high mild
  10 (10.00%) high severe

logical_trivial_join_low_numbered_columns
                        time:   [602.80 µs 603.92 µs 605.05 µs]
                        change: [+6.0809% +6.6720% +7.2644%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) low mild
  2 (2.00%) high mild
  2 (2.00%) high severe

logical_trivial_join_high_numbered_columns
                        time:   [623.12 µs 624.41 µs 625.75 µs]
                        change: [+2.5065% +2.9476% +3.3981%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

Benchmarking logical_aggregate_with_join: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.7s, enable flat sampling, or reduce sample count to 60.
logical_aggregate_with_join
                        time:   [1.1193 ms 1.1213 ms 1.1235 ms]
                        change: [+15.657% +16.111% +16.610%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) low mild
  2 (2.00%) high mild
  7 (7.00%) high severe

physical_plan_tpch_q1   time:   [6.7090 ms 6.7185 ms 6.7288 ms]
                        change: [-4.6107% -4.3279% -4.0398%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

physical_plan_tpch_q2   time:   [10.510 ms 10.526 ms 10.544 ms]
                        change: [-0.3679% -0.1707% +0.0234%] (p = 0.10 > 0.05)
                        No change in performance detected.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe

physical_plan_tpch_q3   time:   [3.5859 ms 3.5923 ms 3.5996 ms]
                        change: [+4.4629% +4.7337% +5.0124%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe

physical_plan_tpch_q4   time:   [2.4776 ms 2.4814 ms 2.4856 ms]
                        change: [-6.8969% -6.6831% -6.4669%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe

physical_plan_tpch_q5   time:   [5.4360 ms 5.4433 ms 5.4511 ms]
                        change: [+7.1262% +7.4714% +7.7929%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe

Benchmarking physical_plan_tpch_q6: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 8.5s, enable flat sampling, or reduce sample count to 50.
physical_plan_tpch_q6   time:   [1.6593 ms 1.6620 ms 1.6651 ms]
                        change: [+3.6049% +4.0420% +4.4263%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 15 outliers among 100 measurements (15.00%)
  10 (10.00%) high mild
  5 (5.00%) high severe

physical_plan_tpch_q7   time:   [7.1784 ms 7.1881 ms 7.1990 ms]
                        change: [-0.9565% -0.7731% -0.5913%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe

physical_plan_tpch_q8   time:   [9.3676 ms 9.3803 ms 9.3946 ms]
                        change: [-11.944% -11.733% -11.519%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe

physical_plan_tpch_q9   time:   [7.0398 ms 7.0497 ms 7.0603 ms]
                        change: [-12.724% -12.537% -12.338%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

physical_plan_tpch_q10  time:   [5.3599 ms 5.3822 ms 5.4061 ms]
                        change: [+0.7193% +1.2464% +1.7604%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high mild

physical_plan_tpch_q11  time:   [4.3129 ms 4.3200 ms 4.3277 ms]
                        change: [+6.8734% +7.1393% +7.4163%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe

physical_plan_tpch_q12  time:   [3.5347 ms 3.5404 ms 3.5463 ms]
                        change: [+3.9650% +4.1949% +4.4171%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

physical_plan_tpch_q13  time:   [2.2836 ms 2.2871 ms 2.2909 ms]
                        change: [+7.2793% +7.5424% +7.7873%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

physical_plan_tpch_q14  time:   [2.9526 ms 2.9590 ms 2.9660 ms]
                        change: [-0.6401% +0.3243% +1.2630%] (p = 0.51 > 0.05)
                        No change in performance detected.
Found 15 outliers among 100 measurements (15.00%)
  9 (9.00%) high mild
  6 (6.00%) high severe

physical_plan_tpch_q16  time:   [4.4232 ms 4.4304 ms 4.4386 ms]
                        change: [+3.9770% +4.2441% +4.5300%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  3 (3.00%) high mild
  4 (4.00%) high severe

physical_plan_tpch_q17  time:   [4.1021 ms 4.1085 ms 4.1160 ms]
                        change: [+7.2418% +7.4904% +7.7389%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe

physical_plan_tpch_q18  time:   [4.8702 ms 4.9019 ms 4.9339 ms]
                        change: [+8.7271% +9.5424% +10.331%] (p = 0.00 < 0.05)
                        Performance has regressed.

physical_plan_tpch_q19  time:   [9.1227 ms 9.1335 ms 9.1454 ms]
                        change: [+2.0674% +2.2535% +2.4382%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe

physical_plan_tpch_q20  time:   [5.5458 ms 5.5523 ms 5.5593 ms]
                        change: [+7.5830% +7.8068% +8.0257%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

physical_plan_tpch_q21  time:   [8.2274 ms 8.2406 ms 8.2548 ms]
                        change: [+3.6119% +3.8229% +4.0440%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high mild

physical_plan_tpch_q22  time:   [3.7159 ms 3.7217 ms 3.7282 ms]
                        change: [+1.3475% +1.5751% +1.8234%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe

Benchmarking physical_plan_tpch_all: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 11.4s, or reduce sample count to 40.
physical_plan_tpch_all  time:   [112.98 ms 113.14 ms 113.32 ms]
                        change: [-0.6799% -0.1096% +0.3016%] (p = 0.72 > 0.05)
                        No change in performance detected.
Found 11 outliers among 100 measurements (11.00%)
  8 (8.00%) high mild
  3 (3.00%) high severe

logical_plan_tpch_all   time:   [12.974 ms 13.015 ms 13.056 ms]
                        change: [+1.8106% +2.1746% +2.5372%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

@alamb, @matthewmturner and @jayzhan211, I believe this PR is ready for review now. The main modifications are in the dfschema.rs. I'm eager to hear your thoughts on the API design, as well as any suggestions for improving the speed of the planner. Your feedback would be greatly appreciated to make the changes more effective

@haohuaijin I see that Option and Field are tuples in many places, is there any reason not to have DFField wrapping these two values?

@haohuaijin I see that Option<OwnedTableRefence> and Field are tuples in many places, is there any reason not to have DFField wrapping these two values?

There is no reason not to do this, but we should change the DFField type, like below, because we return &, not the actual value.
struct DFField<'a> {
    qualifier: Option<&'a OwnedTableReference>,
    field: &'a Field,
}

or
type DFField = (Option<&'a OwnedTableReference>, &'a Field);

@haohuaijin I see that Option<OwnedTableRefence> and Field are tuples in many places, is there any reason not to have DFField wrapping these two values?

There is no reason not to do this, but we should change the DFField type, like below, because we return &, not the actual value.
struct DFField<'a> {
    qualifier: Option<&'a OwnedTableReference>,
    field: &'a Field,
}

or
type DFField = (Option<&'a OwnedTableReference>, &'a Field);

what do you think about it?

what is the reason to return with reference?

what is the reason to return with reference? I think it is fine if unavoidable.

We primarily use Option<OwnedTableReference> and Field in functions whose names start with qualified_field_*. These functions are designed to locate a qualified field within a DFSchema. Consequently, we return a (Option<&OwnedTableReference>, &Field), which are references to the original DFSchema. If we were to return DFField, it would result in data cloning.
pub struct DFField {
    /// Optional qualifier (usually a table or relation name)
    qualifier: Option<OwnedTableReference>,
    /// Arrow field definition
    field: FieldRef,
}

If you don't mind, I would like to change the code and play around it.

My benchmark of sql_planner

If you don't mind, I would like to change the code and play around it.

Of course, feel free to experiment with the code. I'm looking forward to seeing your modifications

Some places got owned DFField, some got referenced. Should we have two struct one for owned, another for referenced? 🤔

I ran these commands. I am rerunning them now to regather the numbers
git checkout feat/make-dfschema-wrap-schemaref
cargo bench --bench sql_planner -- all --save-baseline before

MERGE_BASE=$(git merge-base HEAD apache/main)
echo "** Comparing to ${MERGE_BASE}"

git checkout ${MERGE_BASE}
cargo bench --bench sql_planner -- all --save-baseline change

critcmp before change

I ran these commands. I am rerunning them now to regather the numbers
git checkout feat/make-dfschema-wrap-schemaref
cargo bench --bench sql_planner -- all --save-baseline before

MERGE_BASE=$(git merge-base HEAD apache/main)
echo "** Comparing to ${MERGE_BASE}"

git checkout ${MERGE_BASE}
cargo bench --bench sql_planner -- all --save-baseline change

critcmp before change

Isn't your "before" and "change" reversed?
the before should be at the merge-base, right?

Ok, I re-ran the benchmark and indeed it shows this PT is actually 4x faster for queries that select large numbers of columns ❤
group                            dfschema                               main
-----                            --------                               ----
logical_plan_tpch_all            1.01     16.4±0.15ms        ? ?/sec    1.00     16.2±0.20ms        ? ?/sec
logical_select_all_from_1000     1.00     25.6±0.26ms        ? ?/sec    3.86     98.8±0.44ms        ? ?/sec
physical_plan_tpch_all           1.00    127.3±0.82ms        ? ?/sec    1.01    128.2±0.60ms        ? ?/sec
physical_select_all_from_1000    1.00    175.1±0.53ms        ? ?/sec    4.14    724.3±1.56ms        ? ?/sec

Benchmark scripts

git checkout feat/make-dfschema-wrap-schemaref
cargo bench --bench sql_planner -- all --save-baseline dfschema

MERGE_BASE=$(git merge-base HEAD apache/main)
echo "** Comparing to ${MERGE_BASE}"

git checkout ${MERGE_BASE}
cargo bench --bench sql_planner -- all --save-baseline main

critcmp main dfschema

Here is the result of running the entire benchmark:
group                                         dfschema                               main
-----                                         --------                               ----
logical_aggregate_with_join                   1.13  1366.0±39.42µs        ? ?/sec    1.00  1210.0±10.10µs        ? ?/sec
logical_plan_tpch_all                         1.00     16.4±0.16ms        ? ?/sec    1.00     16.3±0.13ms        ? ?/sec
logical_select_all_from_1000                  1.00     25.7±0.18ms        ? ?/sec    3.86     98.9±0.42ms        ? ?/sec
logical_select_one_from_700                   1.20   837.5±10.22µs        ? ?/sec    1.00   696.2±17.76µs        ? ?/sec
logical_trivial_join_high_numbered_columns    1.04    787.7±7.54µs        ? ?/sec    1.00    754.7±9.10µs        ? ?/sec
logical_trivial_join_low_numbered_columns     1.07   768.8±10.41µs        ? ?/sec    1.00    719.7±8.67µs        ? ?/sec
physical_plan_tpch_all                        1.00    127.1±0.50ms        ? ?/sec    1.01    128.8±0.74ms        ? ?/sec
physical_plan_tpch_q1                         1.00      7.5±0.04ms        ? ?/sec    1.03      7.7±0.05ms        ? ?/sec
physical_plan_tpch_q10                        1.00      6.1±0.05ms        ? ?/sec    1.02      6.2±0.03ms        ? ?/sec
physical_plan_tpch_q11                        1.04      5.0±0.02ms        ? ?/sec    1.00      4.8±0.03ms        ? ?/sec
physical_plan_tpch_q12                        1.02      3.9±0.02ms        ? ?/sec    1.00      3.9±0.02ms        ? ?/sec
physical_plan_tpch_q13                        1.02      2.6±0.01ms        ? ?/sec    1.00      2.6±0.02ms        ? ?/sec
physical_plan_tpch_q14                        1.03      3.4±0.03ms        ? ?/sec    1.00      3.3±0.02ms        ? ?/sec
physical_plan_tpch_q16                        1.01      5.0±0.02ms        ? ?/sec    1.00      5.0±0.02ms        ? ?/sec
physical_plan_tpch_q17                        1.05      4.9±0.02ms        ? ?/sec    1.00      4.6±0.03ms        ? ?/sec
physical_plan_tpch_q18                        1.03      5.4±0.03ms        ? ?/sec    1.00      5.3±0.03ms        ? ?/sec
physical_plan_tpch_q19                        1.01      9.6±0.04ms        ? ?/sec    1.00      9.5±0.05ms        ? ?/sec
physical_plan_tpch_q2                         1.00     11.5±0.04ms        ? ?/sec    1.03     11.9±0.08ms        ? ?/sec
physical_plan_tpch_q20                        1.04      6.4±0.05ms        ? ?/sec    1.00      6.1±0.04ms        ? ?/sec
physical_plan_tpch_q21                        1.01      9.2±0.05ms        ? ?/sec    1.00      9.1±0.05ms        ? ?/sec
physical_plan_tpch_q22                        1.01      4.4±0.02ms        ? ?/sec    1.00      4.4±0.02ms        ? ?/sec
physical_plan_tpch_q3                         1.02      4.1±0.02ms        ? ?/sec    1.00      4.0±0.02ms        ? ?/sec
physical_plan_tpch_q4                         1.00      3.0±0.02ms        ? ?/sec    1.08      3.2±0.01ms        ? ?/sec
physical_plan_tpch_q5                         1.06      6.2±0.20ms        ? ?/sec    1.00      5.9±0.06ms        ? ?/sec
physical_plan_tpch_q6                         1.02  1977.9±12.03µs        ? ?/sec    1.00  1930.7±10.86µs        ? ?/sec
physical_plan_tpch_q7                         1.00      8.1±0.03ms        ? ?/sec    1.02      8.3±0.04ms        ? ?/sec
physical_plan_tpch_q8                         1.00     10.6±0.06ms        ? ?/sec    1.11     11.8±0.06ms        ? ?/sec
physical_plan_tpch_q9                         1.00      8.0±0.04ms        ? ?/sec    1.13      9.0±0.05ms        ? ?/sec
physical_select_all_from_1000                 1.00    175.0±0.52ms        ? ?/sec    4.14    724.3±1.39ms        ? ?/sec
physical_select_one_from_700                  1.23      4.9±0.02ms        ? ?/sec    1.00      4.0±0.02ms        ? ?/sec
alamb@aal-dev:~/arrow-datafusion4$

The queries that are reported slower are all very low latency to begin with and I think may be more measurement noise

I think the "all" bechmarks are more meaningful as they do more work / more queries

I'm working on introducing DFFieldRef and it seems a LARGE change too (including relocating some code to fight for borrow checker). I don't plan to work on this PR, but follow up PRs.
pub struct DFFieldRef<'a> {
    /// Optional qualifier (usually a table or relation name)
    qualifier: Option<&'a OwnedTableReference>,
    /// Arrow field definition
    field: &'a Field,
}

I will apply some suggestions today to make the code better.

Great job @haohuaijin, glad to see this near the finish line.

I plan to merge this tomorrow unless anyone else would like time to review. Thanks again everyone

I found it difficult to cleanup the tuple into DFField struct because there 4 kinds of combination I need to construct.
Normal case
pub struct DFFieldRef<'a> {
    /// Optional qualifier (usually a table or relation name)
    qualifier: Option<&'a OwnedTableReference>,
    /// Arrow field definition
    field: &'a Field,
}

Sometimes we need FieldRef to avoid the cost of clone for field.
pub struct DFFieldRefWithArc<'a> {
    /// Optional qualifier (usually a table or relation name)
    qualifier: Option<&'a OwnedTableReference>,
    /// Arrow field definition
    field: &'a FieldRef,
}

It is usually easier to deal with owned dffield sometimes without additional cost, because we can't avoid clone for Schema::new_with_metadata
pub struct DFField {
    /// Optional qualifier (usually a table or relation name)
    pub qualifier: Option<OwnedTableReference>,
    /// Arrow field definition
    pub field: FieldRef,
}

And of course, we have field not fieldRef, it seems better not to wrap it with Arc if possible.
pub struct DFField {
    /// Optional qualifier (usually a table or relation name)
    pub qualifier: Option<OwnedTableReference>,
    /// Arrow field definition
    pub field: FieldRef,
}

I suggest we want until the imminent 37.0.0 release -- https://github.com/apache/arrow-datafusion/pull/9697 -- so we have additional time to test this change before releasing it to the larger community

Since Arrow 37 is now basically released I think this PR is ready to merge.
I took the liberty of merging up from main to resolve some conflicts, and once CI is complete I intend to merge this PR
Thanks again @haohuaijin

Great job getting this over the line @haohuaijin

🚀


fix doc about parquet under incubation

Thank you for your interest in contributing to Impala! We do code review via gerrit rather than GitHub PRs; you can see the process described at https://cwiki.apache.org/confluence/display/IMPALA/Contributing+to+Impala

Posted to gerrit; doing pre-commit Jenkins job now:
https://gerrit.cloudera.org/#/c/8378/


Improve landing page

Update the landing page to show a hero section and some of the latest course
Add Noto Font family for latin charaters to keep consistent with Amharic and Arabic

Looks really nice ya Mozafar !


Change does_entity_exists create error message if entity does not exist

The does_entity_exists function is directly calling the get_entity function. If entity does not exists this function will fail and create an error message. The expected behavior would not be an error message when probing for existing element.

@windrad6 I checked on this. The function's response is correct, but the log is misleading if the entity does not exist. Probably, it is easier and cleaner to write an extra request for this, right?


change filetree formatting for readability

The filetree format in the tutorial sample code makes it look like the hellodojo.html file should be inside the demo/ folder:

this change makes the example code match the written guideline and makes it clear the hellodojo.html file should be outside the demo/ folder:

(the formatting hasn't changed otherwise, I just used Mou to render the markdown without styling for the above image)

Thanks. This has been fixed in 67c4e30abbe1231c1ccef4fc02273437df53a0c4


Weights for CUDNN GRU are in the wrong order

I've been delving in CuDNNGRU (making a conversion to non-CuDNN for after training)
I think the weights in the kernel and recurrent matrices of the CuDNNGRU are in the wrong order. For example the kernel matrix is returned as a numpy array sized as [channels,weights].
But actually if you delve into it, the weights are actually in [weights,channels] order.
The gist has a demo of this, along with what I have just worked on to find this out - a GRU cell which is compatible with the CuDNNGRU but doesn't need Cuda (so I can train on tensorflow, but deploy models without tensorflow/cuda). https://gist.github.com/joemarshall/338b0f0c0741408d044f3104b0d3b91d There is an equivalent cell in Tensorflow also, so I don't know how useful it is.
One could fix this by flipping the arrays in the CuDNNGRU cell before they go into the tensorflow code, but it would break all saved networks, so I'm not sure what should be done about it or whether it matters.

Hello, sorry to wake up a long forgotten thread, but think I need your help @bzamecnik.
Here is my usecase: I am trying to load weights from a trained keras.layers.CuDNNGRU into a torch.nn.GRU (I know, risky business from the get go). Problem: even after loading the weights, seemingly correctly, both implementations return widely different results.
You will find in this gist  some code to reproduce the error, together with a Dockerfile to reproduce my environment exactly:

Ubuntu 16.04
CUDA 8.0
CuDNN 5.x
torch 0.4.1
keras 2.2.4
tensorflow-gpu 1.2.0

You will find in the gist two script two scripts which should show you that I cannot make it work in any direction:

creating torch GRU, load its weights into CudNNGRU and check for equality
creating CudNNGRU, load its weights into torch GRU and check for equality

Note that the structure of the keras.layers.CudNNGRU weights when using get_weights is as follows:

input matrix of shape (input_dimension, 3 * hidden_dimension)
recurrent matrix of shape (hidden_dimension, 3 * hidden_dimension)
bias of shape (2 * 3 * hidden_dimension) where the first half is the input bias, second half is the recurrent bias (that part I am unsure of, cannot understand it from Keras' source code)

The structure of the torch.nn.GRU weights is as follows:

input matrix torch.nn.GRU.weight_ih_l0 has shape (3 * hidden_dimension, input_dimension)
recurrent matrix torch.nn.GRU.weight_hh_l0 has shape (3 * hidden_dimension, hidden_dimension)
input bias torch.nn.GRU.bias_ih_l0 has shape (3 * hidden_dimension,)
recurrent bias torch.nn.GRU.bias_hh_l0 has shape (3 * hidden_dimension,)

Hi @naifrec,
As discussed above, the weight matrices of the CuDNNGRU layer are in the wrong order and need to be transformed before being used in another library.
Also, the PyTorch weight matrix represents [kernel_r, kernel_z, kernel_h] while Keras weight matrix represents [kernel_z, kernel_r, kernel_h].
For converting from Keras to PyTorch, you can use:
def convert_input_kernel(kernel):
    kernel_z, kernel_r, kernel_h = np.hsplit(kernel, 3)
    kernels = [kernel_r, kernel_z, kernel_h]
    return np.vstack([k.reshape(k.T.shape) for k in kernels])

def convert_recurrent_kernel(kernel):
    kernels = np.hsplit(kernel, 3)
    return np.vstack([k.T for k in kernels])

# layer = CuDNNGRU(...)
weights = layer.get_weights()
weight_ih = torch.from_numpy(convert_input_kernel(weights[0]))
weight_hh = torch.from_numpy(convert_recurrent_kernel(weights[1]))

Similarly, for converting from PyTorch to Keras,
def convert_input_kernel(kernel):
    kernel_r, kernel_z, kernel_h = np.vsplit(kernel, 3)
    kernels = [kernel_z, kernel_r, kernel_h]
    return np.hstack([k.reshape(k.T.shape) for k in kernels])

def convert_recurrent_kernel(kernel):
    kernels = np.vsplit(kernel, 3)
    return np.hstack([k.T for k in kernels])

# gru = torch.nn.GRU(...)
weights[0] = convert_input_kernel(gru.weight_ih_l0.detach().numpy())
weights[1] = convert_recurrent_kernel(gru.weight_hh_l0.detach().numpy())

Hello @yuyang-huang, thank you so much for your answer!
I indeed realized over night that whatever @joemarshall originally proposed had to be applied block-wise, I applying his trick over the entire recurrent and input matrices before and it failed of course.
Thank you again for the code snippet!

Hello @yuyang-huang, it seems that the biases also suffer from an order issue.
I updated my keras_torch_gru_minima_example.py script in my [gist],() as well as the Dockerfile to add a few dependencies likefire, to make it easier for you to see the problem.
I realized that the equality tests between torch and keras GRU pass when you use the randomly initialized weights by Keras CuDRNNGRU. However, the biases are initialized to zero. As soon as you change them to be non-zero, then the output activation differ between torch and keras.
You can try this out yourself by running the following commands:
python3.6 keras_torch_gru_minimal_example.py --input_dimension 7 --gru_size 13 --atol 1e-6 --non_zero_bias False # should return WITHOUT Not equal to tolerance rtol=0, atol=1e-06 message
python3.6 keras_torch_gru_minimal_example.py --input_dimension 7 --gru_size 13 --atol 1e-6 --non_zero_bias True # should return WITH Not equal to tolerance rtol=0, atol=1e-06 message

So my guess is that there is the same type of thing happening to the bias than to the kernels. Biases are most likely stored in an matrix of shape (2, 3 * gru_size) in the CuDNNGRU underlying C class, then flattened to this long vector of shape (2 * 3 * gru_size,). I have tried a combination of split, reshape and flatten but got nothing to work yet. Are you able to modify convert_bias to get the output to match?
It is also possible that torch.nn.GRU and keras.layers.CuDNNGRU just use biases differently at a higher level, but I sincerely doubt it.
Thanks for the help.

hey,
when I initially found out the ordering thing for the weights, I did a load of tests with manually set weights, sticking in 1 to a single weight in the matrix and leaving the rest zero, and seeing which cell I had to change to get the same results from a model in the non cudnn matrix. Doing that meant I could isolate each one and work out the wrongness. Better than using random initialisers. I guess you need to do the same with the bias (I'm not sure that I didn't already do this in my experimentation, but I can't remember and I don't have the code to hand right now).
On naif<EMAIL_ADDRESS>28 Feb 2019 3:49 p.m. wrote:
Hello @yuyang-huanghttps://github.com/yuyang-huang, it seems that the biases also suffer from an order issue.
I updated my keras_torch_gru_minima_example.py script in my [gist],() as well as the Dockerfile to add a few dependencies likefire, to make it easier for you to see the problem.
I realized that the equality tests between torch and keras GRU pass when you use the randomly initialized weights by Keras CuDRNNGRU. However, the biases are initialized to zero. As soon as you change them to be non-zero, then the output activation differ between torch and keras.
You can try this out yourself by running the following commands:
python3.6 keras_torch_gru_minimal_example.py --input_dimension 7 --gru_size 13 --atol 1e-6 --non_zero_bias False # should return WITHOUT Not equal to tolerance rtol=0, atol=1e-06 message
python3.6 keras_torch_gru_minimal_example.py --input_dimension 7 --gru_size 13 --atol 1e-6 --non_zero_bias True # should return WITH Not equal to tolerance rtol=0, atol=1e-06 message
So my guess is that there is the same type of thing happening to the bias than to the kernels. Biases are most likely stored in an matrix of shape (2, 3 * gru_size) in the CuDNNGRU underlying C class, then flattened to this long vector of shape (2 * 3 * gru_size,). I have tried a combination of split, reshape and flatten but got nothing to work yet. Are you able to modify convert_bias to get the output to match?
It is also possible that torch.nn.GRU and keras.layers.CuDNNGRU just use biases differently at a higher level, but I sincerely doubt it.
Thanks for the help.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/keras-team/keras/issues/8720#issuecomment-468323097, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABXse9gJHLYy3tCLe3ybh52oWfzuCzhAks5vR_pogaJpZM4Q5XRB.
This message and any attachment are intended solely for the addressee
and may contain confidential information. If you have received this
message in error, please contact the sender and delete the email and
attachment.
Any views or opinions expressed by the author of this email do not
necessarily reflect the views of the University of Nottingham. Email
communications with the University of Nottingham may be monitored
where permitted by law.

Oh hang on, code is in the gist above, you need reshape and permute dimensions to fix keras weights and biases

Hi @naifrec,
Yes, indeed, the biases also need to be reordered. The convert_recurrent_kernel() above was also wrong.
The following worked for me, when the weights are filled with random values:
Keras -> torch:
def convert_input_kernel(kernel):
    kernel_z, kernel_r, kernel_h = np.hsplit(kernel, 3)
    kernels = [kernel_r, kernel_z, kernel_h]
    return np.vstack([k.reshape(k.T.shape) for k in kernels])

def convert_recurrent_kernel(kernel):
    kernel_z, kernel_r, kernel_h = np.hsplit(kernel, 3)
    kernels = [kernel_r, kernel_z, kernel_h]
    return np.vstack(kernels)

def convert_bias(bias):
    bias = bias.reshape(2, 3, -1)
    return bias[:, [1, 0, 2], :].reshape(-1)

torch -> Keras (just the reverse transformation):
def convert_input_kernel(kernel):
    kernel_r, kernel_z, kernel_h = np.vsplit(kernel, 3)
    kernels = [kernel_z, kernel_r, kernel_h]
    return np.hstack([k.reshape(k.T.shape) for k in kernels])

def convert_recurrent_kernel(kernel):
    kernel_r, kernel_z, kernel_h = np.vsplit(kernel, 3)
    kernels = [kernel_z, kernel_r, kernel_h]
    return np.hstack(kernels)

def convert_bias(bias):
    bias = bias.reshape(2, 3, -1)
    return bias[:, [1, 0, 2], :].reshape(-1)

@yuyang-huang when converting pytorch to keras, there are 2 biases in pytorch, shoud i concat them first, and then call the convert_bias function?

@attitudechunfeng Yes, the bias_ih goes first, then bias_hh.

The Answer
Hello, I was observing the same issue for a while, and I think the answer stems in the GRU implementation and not a bug.
Obviously, gru_with_backend_selection unit only uses recurrent biases. Hence the weight bias's tensor shape is [1, 3* gru_units] and not in [2, 3*gru_units] in which 2 is associated with kernel and recurrent weights.
This is more clearly discussed in [https://github.com/tensorflow/tensorflow/blob/v2.3.1/tensorflow/python/keras/layers/recurrent_v2.py#L187-L504](GRU Git source code).

The Answer
Hello, I was observing the same issue for a while, and I think the answer stems in the GRU implementation and not a bug.
Obviously, gru_with_backend_selection unit only uses recurrent biases. Hence the weight bias's tensor shape is [1, 3* gru_units] and not in [2, 3*gru_units] in which 2 is associated with kernel and recurrent weights.
This is more clearly discussed in GRU Git source code.

Hi @naifrec,
Yes, indeed, the biases also need to be reordered. The convert_recurrent_kernel() above was also wrong.
The following worked for me, when the weights are filled with random values:
Keras -> torch:
def convert_input_kernel(kernel):
    kernel_z, kernel_r, kernel_h = np.hsplit(kernel, 3)
    kernels = [kernel_r, kernel_z, kernel_h]
    return np.vstack([k.reshape(k.T.shape) for k in kernels])

def convert_recurrent_kernel(kernel):
    kernel_z, kernel_r, kernel_h = np.hsplit(kernel, 3)
    kernels = [kernel_r, kernel_z, kernel_h]
    return np.vstack(kernels)

def convert_bias(bias):
    bias = bias.reshape(2, 3, -1)
    return bias[:, [1, 0, 2], :].reshape(-1)

torch -> Keras (just the reverse transformation):
def convert_input_kernel(kernel):
    kernel_r, kernel_z, kernel_h = np.vsplit(kernel, 3)
    kernels = [kernel_z, kernel_r, kernel_h]
    return np.hstack([k.reshape(k.T.shape) for k in kernels])

def convert_recurrent_kernel(kernel):
    kernel_r, kernel_z, kernel_h = np.vsplit(kernel, 3)
    kernels = [kernel_z, kernel_r, kernel_h]
    return np.hstack(kernels)

def convert_bias(bias):
    bias = bias.reshape(2, 3, -1)
    return bias[:, [1, 0, 2], :].reshape(-1)

Hi, @yuyang-huang !
I am trying convert torch1.9 GRU weights to tensorflow2.6 in this way. However, it can't work. Is there a problem here? Below is the code. Hope for your help!!
import tensorflow as tf
import numpy as np
from tensorflow.keras import layers
import torch
import torch.nn as nn

input = torch.ones(1,1,5, dtype=torch.float32)
tf_input = tf.convert_to_tensor(input.detach().numpy())
gru = nn.GRU(input_size=5, hidden_size=3, num_layers=1, batch_first=True,bidirectional=False)
h1_o = torch.zeros([1, 1, 3], dtype=torch.float32)
tornn, (h1_o) = gru(input,(h1_o))

tfgru = layers.GRU(3, return_sequences=True, return_state=True, use_bias=True,)
h0 = tf.zeros([1, 3], tf.float32)
tfrnn, h0 = tfgru(tf_input, initial_state=[h0])

def convert_input_kernel(kernel):
    kernel_r, kernel_z, kernel_h = np.vsplit(kernel, 3)
    kernels = [kernel_z, kernel_r, kernel_h]
    return np.hstack([k.reshape(k.T.shape) for k in kernels])

def convert_recurrent_kernel(kernel):
    kernel_r, kernel_z, kernel_h = np.vsplit(kernel, 3)
    kernels = [kernel_z, kernel_r, kernel_h]
    return np.hstack(kernels)

def convert_bias(bias):
    bias = bias.reshape(3, 3, -1)
    return bias[:, [1, 0, 2], :].reshape(-1)

a = convert_input_kernel(gru.weight_ih_l0.detach().numpy())
b = convert_input_kernel(gru.weight_hh_l0.detach().numpy())
c = convert_bias(gru.bias_ih_l0.detach().numpy())
d = convert_bias(gru.bias_hh_l0.detach().numpy())
tfgru.set_weights([a, b, np.vstack((c,d))])

h0 = tf.zeros([1, 3], tf.float32)
tfrnn, h0 = tfgru(tf_input, initial_state=[h0])

print(np.sum((tf_input.numpy() - input.detach().numpy())**2))
print('MSE',np.sum((tfrnn.numpy() - tornn.detach().numpy())**2))
print('MSE',np.sum((h0.numpy() - h1_o.squeeze(0).detach().numpy())**2))
'''


Fix OSError: [WinError 193]  on Windows debuggers

Description:
If you are trying to debug the home-assistant code on Windows i.e. with PyCharm it will fail with this error:
Traceback (most recent call last):
  File "C:/Users/___User___/Downloads/home-assistant-dev/homeassistant/__main__.py", line 374, in <module>
    sys.exit(main())
  File "C:/Users/___User___/Downloads/home-assistant-dev/homeassistant/__main__.py", line 366, in main
    exit_code = setup_and_run_hass(config_dir, args)
  File "C:/Users/___User___/Downloads/home-assistant-dev/homeassistant/__main__.py", line 252, in setup_and_run_hass
    subprocess.check_call(nt_args)
  File "C:\Program Files\Python36\lib\subprocess.py", line 286, in check_call
    retcode = call(*popenargs, **kwargs)
  File "C:\Program Files\Python36\lib\subprocess.py", line 267, in call
    with Popen(*popenargs, **kwargs) as p:
  File "C:\Program Files\Python36\lib\subprocess.py", line 707, in __init__
    restore_signals, start_new_session)
  File "C:\Program Files\Python36\lib\subprocess.py", line 992, in _execute_child
    startupinfo)
OSError: [WinError 193] %1 ist keine zulässige Win32-Anwendung (not a valid win32-application)

Related issue (if applicable): fixes #
Pull request in home-assistant.github.io with documentation (if applicable): home-assistant/home-assistant.github.io#<home-assistant.github.io PR number goes here>
This small change fixes that behaviour.
Example entry for configuration.yaml (if applicable):

Checklist:
If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.github.io

If the code communicates with devices, web services, or third-party tools:

[ ] Local tests with tox run successfully. Your PR cannot be merged unless tests pass
[ ] New dependencies have been added to the REQUIREMENTS variable (example).
[ ] New dependencies are only imported inside functions that use them (example).
[ ] New dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[ ] New files were added to .coveragerc.

If the code does not interact with devices:

[ ] Local tests with tox run successfully. Your PR cannot be merged unless tests pass
[ ] Tests have been added to verify that the new code works.

Hi @tringler,
It seems you haven't yet signed a CLA. Please do so here.
Once you do that we will be able to review and accept this pull request.
Thanks!

See also: https://github.com/home-assistant/home-assistant/pull/6601/files#r106408047


Duplicate and Environment Conflict when Creating Role Assignment in PROD for FOM TEST for Client

Describe the bug
When a role assignment record exists in an application's environment (DEV), the same role assignment cannot be created in the another environment(TEST). The system returns a duplicate record notification.
When the record is deleted in one environment(DEV) and re-created in a different environment(TEST), the record is displayed in the previous environment (DEV)
To Reproduce
Steps to reproduce the behavior:

Go to 'FAM Dev'
Click on 'FOM DEV' application on the Select Application screen
Create a role assignment with a client # for FOM
Save the record
Navigate back to Select application on FAM
Click on 'FOM TEST' application
Create the same Role assignment with the same client #

The screen returns an error message: Duplicate record cannot be created

Delete the same role assignment from FOM Dev application on FAM
Recreate the Record for FOM TEST application, FAM notifies that record has been successfully created

Record is not displayed on the Manage Access screen

Navigate back to the FOM Dev application on FAM Dev

Record can be seen on the Manage Access Screen.
Expected behavior
user assignment should be unique across multiple environments
**Recreated records should be created for the environment in which the record is being created **
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Nice find. We just need to fix logic (either db constraint or business rule) to check environment as well. But I'm confused why that wasn't already done - the application should be part of what's checked for duplicate, and the application is scoped to environment. Unless we are missing application from the check?

Backend code fix and new tests are added into PR #520.

Thanks @ianliuwk1019 is this deployed to Dev yet?

Yes it is currently deployed to dev as the pr is merged and closed, but I moved this ticket back for your review @ArogeG . Also FOM's thickline is fixed as well.
(logout - need to do it in FAM) is still in progress. So FOM test still has problem for logout.

Sounds good, thanks! It seems to be working as expected now. I will do some more test tomorrow.


Having the same issue as in #685

Crazy colors in vscode 1.69.1 , exactly as shown in #685
Non typescript Quasar project.
Was behaving fine. been away from it for about a week.
Disabled prettier, no change. Disable Volar, problem gone

i had same issue in 0.38.8, temporarily fixed by roll back to 0.38.3

0.38.9 was just released. It seems to have helped me.

0.38.9 was just released. It seems to have helped me.

I confirm that I have had this problem for about a week or maybe a little longer, and I believe it has been resolved with the latest update.


Fix incorrect initial scaling

Also, I replaced inverse scaling (like 1 / scaling) with natural scaling (like 1 ... 2, what returned by window.scale_factor())
I think easier to work with natural scaling factor and multiplication, instead of inverse scaling factor and division.
It looks fine, and works as expected, but please, review carefully parts where i changed scaling.
Also, instead of passing pair of width/height to handle_new_grid_size i pass LogicalSize. So it's impossible to pass incorrect value.
(Probably, this could be splitted into "just fix" and "refactoring", but i hope it's small enough and can be merged as is)
What kind of change does this PR introduce?

[x] Fix
[ ] Feature
[ ] Codestyle
[x] Refactor
[ ] Other

Did this PR introduce a breaking change?
A breaking change includes anything that breaks backwards compatibility either at compile or run time.

[ ] Yes, please list breaking changes
[x] No

Refs #771

Also, i changed this file, for commit to pass pre-commit hooks, but that's completely unrelated to this issue. Should i create another PR with that fix?
diff --git a/src/editor/grid.rs b/src/editor/grid.rs
index 8990da2..3a835cb 100644
--- a/src/editor/grid.rs
+++ b/src/editor/grid.rs
@@ -243,7 +243,7 @@ mod tests {
         character_grid.characters = vec![grid_cell.clone(); context.area];

         // RUN FUNCTION
-        character_grid.resize(width, height);
+        character_grid.resize((width, height));

         assert_eq!(character_grid.width, width);
         assert_eq!(character_grid.height, height);

Beauty. Thanks for looking into this!

Also, i changed this file, for commit to pass pre-commit hooks, but that's completely unrelated to this issue. Should i create another PR with that fix?
diff --git a/src/editor/grid.rs b/src/editor/grid.rs
index 8990da2..3a835cb 100644
--- a/src/editor/grid.rs
+++ b/src/editor/grid.rs
@@ -243,7 +243,7 @@ mod tests {
         character_grid.characters = vec![grid_cell.clone(); context.area];

         // RUN FUNCTION
-        character_grid.resize(width, height);
+        character_grid.resize((width, height));

         assert_eq!(character_grid.width, width);
         assert_eq!(character_grid.height, height);

I think this is a good change regardless. I don't mind batching them up.

Thank you for fixing this! ❤


ReferenceError: WebSocket is not defined

Happened few times with your latest version I checked out yesterday.
Points to this line
module.exports = WebSocket || MozWebSocket || window.WebSocket || window.MozWebSocket

User Agents were
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
and
Opera/9.80 (J2ME/MIDP; Opera Mini/4.2.23449/37.8474; U; en) Presto/2.12.423 Version/12.16

Fixed and released as 3.2.1.

thanks guys for the super-fast patch


add alt text to all images and figures in .rst pages

Current alt text coverage in doc pages is 154 out of 358 images and figures, or 43%:
~/cocalc-doc/source$ git grep --extended-regexp "(image|figure)::" | wc -l
358
~/cocalc-doc/source$ git grep "alt:" | wc -l
154
~/cocalc-doc/source$ echo $((154*100/358))%
43%

Current alt text coverage in doc pages is 203 out of 362 images and figures, or 56%:

Current alt text coverage in doc pages is 247 out of 393 images and figures, or 62%:

Current alt text coverage in doc pages is 346 out of 464 images and figures, or 74%.

Current alt text coverage in doc pages is 377 out of 491 images and figures, or 76%.

The following 21 files in cocalc-doc may have figure or image objects with missing alt text. In addition to adding alt text, these articles should be reviewed because they are probably outdated:

[ ] account/support.rst
[ ] howto/activeloop-hub.rst
[ ] howto/dask.rst
[ ] howto/export-timetravel.rst
[ ] howto/fortran.rst
[ ] howto/google-api.rst
[ ] howto/gsl.rst
[ ] howto/mongodb.rst
[ ] howto/neuron.rst
[ ] howto/overleaf.rst
[ ] howto/plotly-dash.rst
[ ] howto/project-invitation-tokens.rst
[ ] howto/ruby.rst
[ ] howto/shiny-server.rst
[ ] howto/upload.rst
[ ] latex-faq.rst
[ ] latex-features.rst
[ ] platform-news.rst
[ ] snippets.rst
[ ] software.rst
[ ] teaching-students.rst

Script to screen for missing alt tags
#!/bin/bash

# usage: cmd [alt|fig]

cd ~/cocalc-doc/source

case $1 in
  alt)
git ls-files |
  grep ".rst$" |
  xargs -d '\n' grep -l ":alt:" |
  sort |
  less
  ;;

  fig)
git ls-files |
  grep ".rst$" |
  xargs -d '\n' egrep -l "(image|figure)::" |
  sort |
  less
  ;;

  *)
  echo ERROR
  exit 1
esac

Done.


OpenAI Adapter Chunk response missing handling for tool_calls

Checked other resources

[X] I added a very descriptive title to this issue.
[X] I searched the LangChain documentation with the integrated search.
[X] I used the GitHub search to find a similar question and didn't find it.
[X] I am sure that this is a bug in LangChain rather than my code.
[X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code
In the following code, there is no support for tool_calls args in langchain_community.adapters ->  openai
def _convert_message_chunk(chunk: BaseMessageChunk, i: int) -> dict:
    _dict: Dict[str, Any] = {}
    if isinstance(chunk, AIMessageChunk):
        if i == 0:
            # Only shows up in the first chunk
            _dict["role"] = "assistant"
        if "function_call" in chunk.additional_kwargs:
            _dict["function_call"] = chunk.additional_kwargs["function_call"]
            # If the first chunk is a function call, the content is not empty string,
            # not missing, but None.
            if i == 0:
                _dict["content"] = None
        else:
            _dict["content"] = chunk.content
    else:
        raise ValueError(f"Got unexpected streaming chunk type: {type(chunk)}")
    # This only happens at the end of streams, and OpenAI returns as empty dict
    if _dict == {"content": ""}:
        _dict = {}
    return _dict
    ```

### Error Message and Stack Trace (if applicable)

_No response_

### Description

I am trying to create OpenAI compatible endpoints using Langchain. Langchain community project provides openai adapter but it is missing handling for tool_calls in the streaming workflow.

### System Info

langchain==0.2.6
langchain-anthropic==0.1.19
langchain-aws==0.1.10
langchain-community==0.2.6
langchain-core==0.2.15
langchain-experimental==0.0.63
langchain-openai==0.1.15
langchain-text-splitters==0.2.2
openinference-instrumentation-langchain==0.1.20

Hi there! We are a group of 3 students from the University of Toronto and we are very interested in contributing to this issue, and implementing the tool_calls handler! Do you have any additional implementation details or specific cases we should consider for this feature? Thanks!


Relative subninja paths don't resolve

I would attempt a PR for this, but I'm not entirely sure this is a bug or if it's intended.
When including a subninja, I get

ninja: error: 'mod.c', needed by 'mod.o', missing and no known rule to make it

although running ninja directly in that directory works just fine. Under the suspicion that relative paths may be to blame, I copied the project structure to a new directory, prepended each input/output with the absolute path, and ran ninja from the parent directory (containing the build.ninja that subninja'd the module). It built just fine.
Is this suggesting we should be generating our ninja configurations with absolute paths? This has a lot of problems, along with the fact not every generator I've seen does this. Further, the documentation doesn't make a distinction, and Ninja runs just fine otherwise.
I want to err on the side of it's a bug, but I feel like a PR that does runtime path canonicalization might introduce a performance hit (though I'm waving my hands here without any benchmarks to back up that suspicion).

subninja paths are relative to the build directory, not to the file containing the subninja line.

That makes no sense, though. That means generators are going to have to all implement some way to prepend a prefix to the files, which means changed the CWD commands run in (which may change the original intent of the build configuration depending on how the generate/command runs).
What, then, are contrasting use cases for subninja vs include?
A subninja with paths relative to the build file would be useful in that depdencies that are configured to run within their own directory can do so without having to modify their paths, but can still contribute to the dependency graph of the parent ninja configuration.
include functionality would be unmodified, and would act exactly how subninjas act other than the fact rule names would now be in a combined namespace (as per #921).

The idea is that the generator generates all .ninja files, so they can write paths relative to the build directory. It's an interesting idea to combine ninja files generated by different generators (it sounds like that's what you want to do?), but that's not something that's supported at the moment.
Correct, the difference between subninja and include is that the former adds a scope and the latter doesn't. The use case for this is that the toplevel ninja can define common build rules such as cc that reference variables such as cflags, and each subninja can set cflags to what's appropriate for that target, and there can be one-off actions in there. To see an example, you can run python misc/write_fake_manifests.py /tmp/foo` to write a bunch of .ninja files to /tmp/foo that use this pattern.

That makes sense, though I still don't see much of a benefit (other than scope).

It's an interesting idea to combine ninja files generated by different generators (it sounds like that's what you want to do?)

Exactly. Being able to include Ninja itself as a submodule for my generator, and then another CMake project (which is configured to output Ninja build files), and then generating the Ninja config files for both and including them as subninjas in my project's build.ninja file in order to be able to build them as if they were on their own, but still allow my project to use their outputs (and thus their dependency graphs) in order to build the whole project at once.
If that makes sense. Immediate use-case that I see with my generator in particular is that it's borrowing a few concepts from Tup (which IIRC influenced some design decisions within Ninja itself) in that I can include N subprojects, all with their own build.ninja files, and then tap into their graphs to allow me to automatically construct a much larger dependency graph.
I personally think that'd make subninja a whole lot more useful, though I could see it being a breaking change. However, I don't see a way around this unless 1) I modify the dependencies' build.ninja files with a patch or 2) sacrifice the ability to harness the dependencies' dependency graphs.
Thoughts?

How about:
subninja path/to/build.nina relative path/to

and an absent relative defaults to ./. That would make it non-breaking but still give the functionality if a generator so desires.
Or, following in the steps of other Ninja constructs, maybe
subninja path/to/build.ninja
  relative = path/to

Suppose you have a project at .../foo and it has a subdirectory bar, and that Ninja had the relative-path logic you suggest.
If your build system wants to write all build outputs to /foo/obj, a subninja in /foo/bar that used directory-relative paths would need to know to write its output into ../obj/bar, as that's the path to the file from that subdirectory.  So whatever is generating your build.ninja files must already be aware of the global path hierarchy, in which case making all paths relative is effectively the same problem as prepending a bar/ to the paths in the bar/ directory.
Maybe there are enough people who write build output in their source directories that the above doesn't matter, though.  I mostly hear from people who want even stronger separation -- like the people who sent patches to Ninja to make it so they can build Ninja with the build output in a totally unrelated directory.

I'm not sure I understand. Subninjas, in the use-case I've described in particular, don't normally know about the parent ninja's structure (at least, don't need to know). I'm sure someone out there could find a case where they do, though.
The problem I'm facing right now is dependencies (third party libraries, etc.) that don't use my generator (which is 100% of them) currently need to be built using a bootstrapper, and have to be bootstrapped every time they're updated or changed, etc. Most generators I work with have the options to output to a ninja configuration, but they're all configured for that directory alone (i.e. relative to that directory).
Being able to perform selective rebuilds using their configurations and graphs would be huge. Currently, I cannot do this due to the fact subninja assumes paths relative to the build directory.

Evan: The way I understood this is that you'd create a tree like this:
builddir
  subbuild1
  subbuild2

and since generators usually support putting the build dir in arbitrary places, building project 1 in subbuild1 and project 2 in subbuild 2 should work. Then there's a toplevel ninja file in builddir that Qix- wants to drive building the subprojects.
Unrelated: this relative feature also has to change the cwd to its argument if any rules depend on the current directory being equal to their build dir (say, if a rule strips built artifacts or something).

I'm hitting this problem now with Ninja and Premake.  All of my libraries exist as their own projects in entirely unrelated directories which I pull into which ever application is being built at the time.
So an example directory structure for my libraries looks something like this:
Core
  Gwee
    Build
      gwee.lua
    <gwee sources>
  Carbon
    Build
      carbon.lua
    <carbon sources>
  etc...

And a project which uses some of those libraries would have something like this:
Project
  Build
    premake5.lua
  Engine
    Build
      engine.lua
    <engine sources>
  App
    Build
      app.lua
    <app sources>

So premake5.lua includes engine.lua, app.lua, and whichever libraries it needs, such as gwee.lua.
The Premake Ninja generator correctly produces .ninja files next to each lua project file in their respective directories, and the build.ninja file in Project/Build correctly references them with relative paths.
But none of them actually build because subninja doesn't change to the directory containing the actual ninja file.
So how is this supposed to work?

All relative paths in Ninja are interpreted as relative to the directory
you run Ninja from (which you can set with the -C flag).
On Sun, Jul 9, 2023 at 7:46 AM James @.***> wrote:

I'm hitting this problem now with Ninja and Premake. All of my libraries
exist as their own projects in entirely unrelated directories which I pull
into which ever application is being built at the time.
So an example directory structure for my libraries looks something like
this:
Core
Gwee
Build
gwee.lua

Carbon
Build
carbon.lua

etc...
And a project which uses some of those libraries would have something like
this:
Project
Build
premake5.lua
Engine
Build
engine.lua

App
Build
app.lua

So premake5.lua includes engine.lua, app.lua, and whichever libraries it
needs, such as gwee.lua.
The Premake Ninja generator correctly produces .ninja files next to each
lua project file in their respective directories, and the build.ninja
file in Project/Build correctly references them with relative paths.
But none of them actually build because subninja doesn't change to the
directory containing the actual ninja file.
So how is this supposed to work?
—
Reply to this email directly, view it on GitHub
https://github.com/ninja-build/ninja/issues/977#issuecomment-1627736055,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAA6B3IGUTWHILWTEXMN3DXPK75PANCNFSM4BJAOQYA
.
You are receiving this because you commented.Message ID:
@.***>

I'm running Ninja from the Build directory of the main project.    The individual ninja files for each library are built with their internal paths relative to themselves.  Hence the above compile error of not being able to find the source files being compiled.
Each project that I have uses Premake and pulls in the libraries it needs as other Premake lua files so that the generated files for any given project are self contained in a single build script.  (Or in the case of Xcode/Visual Studio a single workspace/solution that has all libraries needed for that project.)  But crucially, the project files (and ninja files) for libraries are located within each library's Build directory.
Basically, I end up with:
Core
  Gwee
    Build
      gwee.lua
      build_Gwee_Debug_x64.ninja
      build_Gwee_Release_x64.ninja
  Carbon
    Build
      carbon.lua
      build_Carbon_Debug_x64.ninja
      build_Carbon_Release_x64.ninja
  etc...

Project
  Build
    premake5.lua
    build.ninja
    build.py
  Engine
    Build
      engine.lua
      build_Engine_Debug_x64.ninja
      build_Engine_Release_x64.ninja
  App
    Build
      app.lua
      build_App_Debug_x64.ninja
      build_App_Release_x64.ninja

Where build.ninja just consists of:
# solution build file
# generated with premake ninja

# build projects
subninja ../../Core/Carbon/Build/build_Carbon_Debug_x64.ninja
subninja ../../Core/Carbon/Build/build_Carbon_Release_x64.ninja
subninja ../../Core/Gwee/Build/build_Gwee_Debug_x64.ninja
subninja ../../Core/Gwee/Build/build_Gwee_Release_x64.ninja
subninja ../Engine/Build/build_Engine_Debug_x64.ninja
subninja ../Engine/Build/build_Engine_Release_x64.ninja
subninja ../App/Build/build_App_Debug_x64.ninja
subninja ../App/Build/build_App_Release_x64.ninja

# targets
build Debug: phony Carbon_Debug_x64 Gwee_Debug_x64 Engine_Debug_x64 App_Debug_x64
build Release: phony Carbon_Release_x64 Gwee_Release_x64 Engine_Release_x64 App_Release_x64

# default target
default App_Debug_x64

Is running subninja from the directory that contains the referenced ninja file all that is needed?

On Sun, Jul 9, 2023 at 10:50 AM James @.***> wrote:

I'm running Ninja from the Build directory of the main project.
The individual ninja files for each library are built with their internal
paths relative to themselves. Hence the above compile error of not being
able to find the source files being compiled.

Ninja doesn't support the idea of composing multiple different projects
with different notions of the source root.  It assumes all your build.ninja
files are generated under the same global view of paths.
Message ID: @.***>

On Sun, Jul 9, 2023 at 11:12 AM Evan Martin @.***> wrote:

On Sun, Jul 9, 2023 at 10:50 AM James @.***> wrote:

I'm running Ninja from the Build directory of the main project.
The individual ninja files for each library are built with their internal
paths relative to themselves. Hence the above compile error of not being
able to find the source files being compiled.

Ninja doesn't support the idea of composing multiple different projects
with different notions of the source root.  It assumes all your build.ninja
files are generated under the same global view of paths.

...er sorry, I misunderstood.  It looks like your project does this
already.  Yes, I think you just need to invoke Ninja from the right initial
directory (or use -C).

Message ID: @.***>

I'm running Ninja from the Build directory that contains the build.ninja file.  Looking at the individual ninja files, it does look like their paths are correct, but things just aren't working.  (And having looked inside the individual ninja files, them having paths relative to the current project isn't what I want or need anyway because that won't work for other projects.)
So rewriting my entire build system from scratch so it works on Linux isn't something I'm prepared to do, so maybe Ninja isn't going to work for me.

In general, the Ninja behavior here is fairly straightforward ("all paths
in all files must be relative to the same directory"), which means any
complexity/configuration here is managed by premake.  You might have more
success asking a premake list.
On Sun, Jul 9, 2023 at 12:03 PM James @.***> wrote:

I'm running Ninja from the Build directory that contains the build.ninja
file. Looking at the individual ninja files, it does look like their paths
are correct, but things just aren't working. (And having looked inside the
individual ninja files, them having paths relative to the current project
isn't what I want or need anyway because that won't work for other
projects.)
So rewriting my entire build system from scratch so it works on Linux
isn't something I'm prepared to do, so maybe Ninja isn't going to work for
me.
—
Reply to this email directly, view it on GitHub
https://github.com/ninja-build/ninja/issues/977#issuecomment-1627798086,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAA6BYALRC7AQE3EOGWFFDXPL57VANCNFSM4BJAOQYA
.
You are receiving this because you commented.Message ID:
@.***>

Yeah, it's always tricky to know exactly where the responsibility falls when you're dealing with issues like these.  :)


High levels of extreme frustration 🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨

So, the app is burried in WinApps folder. Way way way out of hand to make it ... what? Put it simply in a clear path people!!!
wt or wt.exe does not work, and yes, the alias in enabled. Why is this so hard to accomplish? What is the end purpose is beyond my simpleton imagination, but, apparently you have too much of it when it comes to overly complicate things.
colortool does not set the color theme and the color theme is different in terminal than in the cmd or powershell. Why bother in the first place.
I tried eating this cake but it just won't get down easily. And it's not me! I had very high expectations. I was mistaken. You are incapable of simple order and clarity.
Back to using a desktop for multiple cmds, alt-tab-ing and sanity.

If you cannot engage in discourse following the code of conduct in this repository, I’m afraid I’ll just have to close this issue.


[fix][test] Fix flaky test: PrometheusMetricsTest.testDuplicateMetricTypeDefinitions

Motivation
Fix #17921
The test may fails if a previous test in the same fork started the transactions and therefore the transactions metrics are published. In the PulsarService shutdown hooks the metrics are never unregistered.
Modifications

Unregister transactions metrics in the PulsarService shutdown

Verifying this change

[x] Make sure that the change passes the CI checks.

Documentation

[ ] doc
[ ] doc-required
[x] doc-not-needed
[ ] doc-complete

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@92b4708). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #18077   +/-   ##
=========================================
  Coverage          ?   70.74%
  Complexity        ?      437
=========================================
  Files             ?       26
  Lines             ?     2246
  Branches          ?      245
=========================================
  Hits              ?     1589
  Misses            ?      484
  Partials          ?      173

Flag
Coverage Δ

unittests
70.74% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.


Ensure pymodbus is installed with serial support

Fixes an issue with the docker container when trying to use serial devices.
Prior to this change running
modpoll --rtu /dev/ttyUSB0 --config https://raw.githubusercontent.com/gavinying/modpoll/master/examples/modsim.csv

within the docker container would cause the following error:
Traceback (most recent call last):
  File "/home/tobias/.local/lib/python3.10/site-packages/pymodbus/client/serial.py", line 206, in connect
    self.socket = serial.serial_for_url(
NameError: name 'serial' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tobias/.local/bin/modpoll", line 8, in <module>
    sys.exit(app())
  File "/home/tobias/.local/lib/python3.10/site-packages/modpoll/main.py", line 84, in app
    modbus_poll()
  File "/home/tobias/.local/lib/python3.10/site-packages/modpoll/modbus_task.py", line 429, in modbus_poll
    master.connect()
  File "/home/tobias/.local/lib/python3.10/site-packages/pymodbus/client/serial.py", line 217, in connect
    except serial.SerialException as msg:
NameError: name 'serial' is not defined

This error is fixed by installing pymodbus with serial support.

Hi @schroepf , thank you very much for raising this issue, your fix absolutely works, but I think it is better to directly integrate pyserial library into our package as an optional dependency (version 0.6.3+), so that users can easily get serial support using,
pip install -U modpoll[serial]

Feel free to try the latest version.
I will close the PR for now, welcome to open another one if needed. Thanks.


Request: mylar3

Can you add Mylar3
https://docs.linuxserver.io/images/docker-mylar3
Thanks a lot!!

Hi, I've pushed a first version with smb and local disk mount support. Could you please test it? thanks!

It works, thanks for adding it.


Illegal invocation error with navigator.geolocation.getCurrentLocation

What version of bluebird is the issue happening on?
3.4.1

What platform and version? (For example Node.js 0.12 or Google Chrome 32)
Opera 38

Did this issue happen with earlier version of bluebird?
Don't know, but some clues I've found by googling do indicate that people managed to run the same code I am trying to run.

Here is the simplest code example to test the issue: https://jsfiddle.net/akdog4r0/
Or the code here:
var getLocation = Promise.promisify(navigator.geolocation.getCurrentPosition);

getLocation().then(function(pos) {
	console.log(pos);
}).catch(function(err) {
	console.log(err);
});

Running this code throws TypeError: Illegal invocation and points to the getLocation() call. This happened also on some local app, js fiddle is just to illustrate the problem.

Have a look at https://stackoverflow.com/questions/31639639/cant-apply-lodash-partial-to-function-created-with-bluebird-promisifyall, https://stackoverflow.com/questions/38577330/cannot-read-property-of-undefined-after-promise-promisify, https://stackoverflow.com/questions/32509904/how-to-ensure-correct-this-with-promise-promisify and many others :-)

Maybe I don't understand the problem clearly, but nothing that I've gathered from those articles have helped me to solve the problem...

getCurrentPosition is a method of the navigator.geolocation object, and like any other callback function you have to bind it to its context.

@vincaslt you have a couple of problems here, firstly that error is happening because you are treating a method as if it were a function. The difference between the two is that a method cares about the value of this, which is being lost in this case.
Taking Bluebird completely out of the picture, the following will fail with the exact same error:
var getLocation = navigator.geolocation.getCurrentPosition;
getLocation(result => console.log(result)); // Illegal Invocation

You need to preserve the context, by adding a second argument to your promisify() call (Note: This won't actually work in your case, see below.)
var getLocation = Promise.promisify(navigator.geolocation.getCurrentPosition, navigator.geolocation);

getLocation().then(result => console.log(result)); // no illegal invocation error, but weird behaviour, WTF?

Your second issue is that getLocation() is a browser API and does not take a node style callback, so promisify() isn't going to work at all in this case! You need to fall back to the promise constructor:
var getLocation = new Promise(function (resolve, reject) {
  navigator.geolocation.getCurrentPosition(resolve, reject);
});


[SYSTEMML-1193] In naive-bayes-predict.dml restore ifdef for probabilities parameter

Restore conditional for checking $probabilities before attempting to use in write() to resolve error when running performance tests.

Below was the observed error mentioned in JIRA:
ERROR api.DMLScript: ../algorithms/naive-bayes-predict.dml line 58:13 The parameter $probabilities either needs to be passed through commandline or initialized to default value.

The following parse issue was encountered:
../algorithms/naive-bayes-predict.dml [line 58:13] [Validation error] -> write(probs, $probabilities, format=cmdLine_fmt)
The parameter $probabilities either needs to be passed through commandline or initialized to default value.
Successfully ran the performance test with updated naive-bayes-predict.dml.

LGTM

@gweidner @bertholdreinwald Is this PR needed for the currently implementation of the performance tests?

Yes, the error was also observed in latest round of performance testing (but can be worked-around).
Instead added probabilities argument in runNaiveBayes.sh for executing naive-bayes-predict.dml to prevent error when running parent runAllMultinomial.sh of performance test suite.  Reverted naive-bayes-predict.dml to previous version without ifdef that was removed in [SYSTEMML-234][SYSTEMML-208].  Also removed "Note this is an optional argument" from naive bayes predict description in algorithms reference.
Verified error no longer occurred, naive-bayes-predict ran successfully from runNaiveBayes.sh using local small private cluster, and probabilities files created in hdfs.

LGTM. Thanks for the additional updates @gweidner!


[SPARK-36166][TESTS] Support Scala 2.13 test in dev/run-tests.py

What changes were proposed in this pull request?
For Apache Spark 3.2, this PR aims to support Scala 2.13 test in dev/run-tests.py by adding SCALA_PROFILE.
In addition, test-dependencies.sh is skipped for Scala 2.13 because we currently don't maintain the dependency manifests yet. This will be handled after Apache Spark 3.2.0 release.
Why are the changes needed?
To test Scala 2.13 with dev/run-tests.py.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Manual. The following is the result. Note that this PR aims to run Scala 2.13 tests instead of passing them. We will have daily GitHub Action job via #33358 and will fix UT failures if exists.
$ dev/change-scala-version.sh 2.13

$ SCALA_PROFILE=scala2.13 dev/run-tests.py
...
========================================================================
Running Scala style checks
========================================================================
[info] Checking Scala style using SBT with these profiles:  -Phadoop-3.2 -Phive-2.3 -Pscala-2.13 -Pkubernetes -Phadoop-cloud -Phive -Phive-thriftserver -Pyarn -Pmesos -Pdocker-integration-tests -Pkinesis-asl -Pspark-ganglia-lgpl
...
========================================================================
Building Spark
========================================================================
[info] Building Spark using SBT with these arguments:  -Phadoop-3.2 -Phive-2.3 -Pscala-2.13 -Pspark-ganglia-lgpl -Pmesos -Pyarn -Phive-thriftserver -Pkinesis-asl -Pkubernetes -Pdocker-integration-tests -Phive -Phadoop-cloud test:package streaming-kinesis-asl-assembly/assembly
...

[info] Building Spark assembly using SBT with these arguments:  -Phadoop-3.2 -Phive-2.3 -Pscala-2.13 -Pspark-ganglia-lgpl -Pmesos -Pyarn -Phive-thriftserver -Pkinesis-asl -Pkubernetes -Pdocker-integration-tests -Phive -Phadoop-cloud assembly/package
...

========================================================================
Running Java style checks
========================================================================
[info] Checking Java style using SBT with these profiles:  -Phadoop-3.2 -Phive-2.3 -Pscala-2.13 -Pspark-ganglia-lgpl -Pmesos -Pyarn -Phive-thriftserver -Pkinesis-asl -Pkubernetes -Pdocker-integration-tests -Phive -Phadoop-cloud
...

========================================================================
Building Unidoc API Documentation
========================================================================
[info] Building Spark unidoc using SBT with these arguments:  -Phadoop-3.2 -Phive-2.3 -Pscala-2.13 -Pspark-ganglia-lgpl -Pmesos -Pyarn -Phive-thriftserver -Pkinesis-asl -Pkubernetes -Pdocker-integration-tests -Phive -Phadoop-cloud unidoc
...

========================================================================
Running Spark unit tests
========================================================================
[info] Running Spark tests using SBT with these arguments:  -Phadoop-3.2 -Phive-2.3 -Pscala-2.13 -Pspark-ganglia-lgpl -Pmesos -Pyarn -Phive-thriftserver -Pkinesis-asl -Pkubernetes -Pdocker-integration-tests -Phive -Phadoop-cloud test
...

Test build #141102 has started for PR 33376 at commit 522125c.

Test build #141102 has finished for PR 33376 at commit 522125c.

This patch fails executing the dev/run-tests script.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141102/

Test build #141105 has started for PR 33376 at commit 1db44bf.

Test build #141105 has finished for PR 33376 at commit 1db44bf.

This patch fails executing the dev/run-tests script.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141105/

Test build #141106 has started for PR 33376 at commit a97a616.

cc @gengliangwang , @HyukjinKwon , @sarutak

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45616/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45618/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45616/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45618/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45620/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45616/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45618/

Test build #141106 has finished for PR 33376 at commit a97a616.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141106/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45620/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45620/

Retest this please

GitHub Action is tested already and the last commit affects Jenkins only. After checking Jenkins trigger, I'll merge this PR. Thank you, @HyukjinKwon and @sarutak .

Hmm. Jenkins PR Builder seems to be stuck.The last run was 30 minutes ago.

https://amplab.cs.berkeley.edu/jenkins/view/Pull Request Builders/job/SparkPullRequestBuilder/

I'll merge this and test later when it comes back. If there is a missing part for Jenkins, I'll make a follow-up.
For me, Jenkins is not a thing we can access or invest more.

I think that's fine to go ahead and merge.

Thanks @dongjoon-hyun ! Late LGTM.


tbarn:coc-enforcement-doc

tbarn:coc-enforcement-doc

Header
Header

Cell
Cell

Cell
Cell

https://github.com/OAI/OpenAPI-Specification/issues/3288#issuecomment-1567222667

It's not clear what this issue is for. If you still need something from us, please re-open and add a description.


Way to find the measurements of NPCs in the scene

I stored the measurements provided by the server to the python client as part of the sensor data. But I found that each measurement provides the measurements related to all the NPCs in the map.
Is there a way to just get the measurements of the NPCs in the current view of the sensor?
I want to try using CARLA to simulate a kind of cruise control and would need the speed and position of the car in front of me in the scene. (If possible all the components in the scene).

Hi @YashBansod, please check issue #63.


Defined content: Area of Interest/aoi

A generic suggestion, based on discussions with REP and others. There may be a need for exporting polygons denoting various different areas for various different purposes. A collective term for these could be e.g. area of interest.
Examples could be:

Shallow gas
Warnings
Data quality

A suggestion then, is to allow aoi as a valid content. The data type will usually (always?) be polygons. The use case is for REP to enable visualization of all aois found within an FMU case, along with their name and their associated description.

Need also mentioned by Snorre in relations to 4D seismic analysis.


error in readme

Following code from readme:
table.insert(require('gen').prompts, {
    Elaborate_Text = {
        prompt = "Elaborate the following text:\n$text",
        replace = true
    },
    Fix_Code = {
        prompt = "Fix the following code. Only ouput the result in format ```$filetype\n...\n```:\n```$filetype\n$text\n```",
        replace = true,
        extract = "```$filetype\n(.-)```"
    }
})

Will give wrong structure in prompts table like this:
prompts = { old_prompt={}, old_prompt2={},   {inserted_promp1={}, inserted_prompt2={}}
Which then gives error when running :Gen.
what worked was
require('gen').prompts['my custom prompt'] = { ... }

Thank you again, @JoseConseco , it's fixed now!


spm support

package.swift file added for supporting spm. You can configure package file.
It is tested in my local environment.

@shima11 would love to see this merged, as it is a great feature that would enable me to utilize this much needed package!

@shima11 Sent you an email

hi @j-dogcoder,
Until spm support will be merged, you can use forked repo https://github.com/sahbazali/FlexiblePageControl as swift package.
Note: I am using this repo in project and because it is not merged, I added extra change for my own purposes. If you want to use exact version of master you can add this repo in spm with this options;
repo: https://github.com/sahbazali/FlexiblePageControl
dependency rule: commit -> 145a62193018004af28d01b636d6db5b14891250

hi @j-dogcoder,
Until spm support will be merged, you can use forked repo https://github.com/sahbazali/FlexiblePageControl as swift package.
Note: I am using this repo in my project and because it is not merged, I added extra changes for my own purposes. If you want to use exact version of master you can add this repo in spm with this options;
repo: https://github.com/sahbazali/FlexiblePageControl dependency rule: commit -> 145a621

Hey @sahbazali! Thank you so much. I will take a look later today.

moved to #45, because there is additional changes.
@j-dogcoder, you can use now directly with the branch master-spm-support of forked repo -> https://github.com/sahbazali/FlexiblePageControl
I'm closing this mr because there is additional change than spm support.


mpg321: add pkg option to set libao default audio driver

Tested manually with "alsa" and "pulse" drivers.

Built on platform(s)

[x] NixOS

[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)

I guess it works because toString [] == "", I think this list concatenation is more explicit and nicer :-)
Rebased to master in 948934864dbf063e742456aa079aa65c0cd7321b. Many thanks!


Its possible to try to start a PyFR simulation without selecting a backend

So think the way to clear this up is to only present the user with clusters that have  a PyFr backend defined.

There is the added complication of EC2 clusters which currently don't define a backend. However, the solution here is probably to add code to the TaskFlow todo this.

Fixed by #447


Use actual native 8 and 16 bit write functions

My custom driver needs actual native 8 and 16 bit writes, could not find a way to overrule this include file and get these changes out of tree.

My apologies


Nested create-route not recognized

hello,
I'm using admin-on-rest version 1.1.1 with a setup that looks like this:
httpClient.js:
import { fetchUtils } from 'admin-on-rest'

const HttpClient = (url, options = {}) => {
    if (!options.headers) {
        options.headers = new Headers({ Accept: 'application/json' })
    }

    // ...handle authorization and set some header

    return fetchUtils.fetchJson(url, options)
}

export default HttpClient

restClient.js
import { jsonServerRestClient } from 'admin-on-rest'
import HttpClient from './httpClient.js'

export default jsonServerRestClient('', HttpClient)

...so I'm pretty basic in that department.
What I now have is a couple of routes looking like this:
<Route exact path="/stream" render={(routeProps) => <StreamList hasCreate resource="stream" {...routeProps} />} />,
<Route exact path="/stream/create" render={(routeProps) => <StreamCreate resource="stream" {...routeProps} />} />,
<Route exact path="/stream/:id" render={(routeProps) => <StreamEdit hasDelete resource="stream" {...routeProps} />} />,
<Route exact path="/stream/:id/delete" render={(routeProps) => <Delete resource="stream" {...routeProps} />} />,
<Route exact path="/stream/query/:id" component={StreamQueryEdit} />,

...this is all pretty standard as well and works fine.
My problem now is that I have another route /config/what, where I want to have nested routes for one of the resources. The routing looks like this:
<Route exact path="/config/how/antonym" render={(routeProps) => <AntonymList hasCreate resource="antonym" {...routeProps} />} />
<Route exact path="/config/how/antonym/create" render={(routeProps) => <AntonymCreate resource="antonym" {...routeProps} />} />
<Route exact path="/config/how/antonym/:id" render={(routeProps) => <AntonymEdit resource="antonym" {...routeProps} />} />
<Route exact path="/config/how/antonym/:id/delete" render={(routeProps) => <Delete resource="antonym" {...routeProps} />} />

Inside the /config/how/ these nested routes are rendered like this:
if (this.props.children) {
    return this.props.children
}

The List-, Edit- and Delete-Route all work like expected here. The only problem is that once I navigate to the Create-Route I get a 500-error from the server.
Turns out that after some investigating, the framework does not recognize this route as a create Route but instead thinks of it as a GET request for the id of "create".
Here is what I tracked the issue down to:

This suggests that for some reason admin-on-rest thinks of this request as a GET request. This also causes the code to go through Edit.js (instead of "Create"?) And I'm really not sure why.
I tried to find the place where the type parameters are being set to see what might cause this behaviour ...but no luck so far.
It's especially weird that both the list and the edit route work fine in these cases. It's just the create-route...

Hi, and thanks for your question. As explained in the admin-on-rest contributing guide, the right place to ask a "How To" question, get usage advice, or troubleshoot your own code, is StackOverFlow.
This makes your question easy to find by the core team, and the developer community. Unlike Github, StackOverFlow has great SEO, gamification, voting, and reputation. That's why we chose it, and decided to keep GitHub issues only for bugs and feature requests.
So I'm closing this issue, and inviting you to ask your question at:
http://stackoverflow.com/questions/tagged/admin-on-rest
And once you get a response, please continue to hang out on the admin-on-rest channel in StackOverflow. That way, you can help newcomers and share your expertise!

Hi @fzaninotto ,
actually I started yesterday by posting a queestion on stackoverflow because I was not sure if this behaviour is a bug or I'm just doing something wrong:
https://stackoverflow.com/questions/45015394/admin-on-rest-nested-create-route-not-working
...and the first response that I got was basically that I should open an issue here because the poster had a similiar issue with nested routes before and suggested it being a bug.
So can you maybe confirm this behaviour is as intended and the mistake is on my part?

From my point of view, this is a request for support. If this was a bug description, you should remove all the code that is specific to your use case, and write it so we could easily understand it in the general way. It's not the case of this issue.
Besides, I simply don't understand this sentence:

Turns out that after some investigating, the framework does not recognize this route as a create Route but instead thinks of it as a GET request for the id of "create".

You're mixing up react-router routes and rest client in the same sentence, where they have nothing to see with each other.


[BUG] Cannot read properties of undefined (reading 'syncing')

What happened?
[BUG] Cannot read properties of undefined (reading 'syncing')
Version
1.3.5114
What platform are you using?
Windows
What ui mode are you using?
GUI
Relevant log output or stacktrace
Cannot read properties of undefined (reading 'syncing')

URL
#/dashboard/wallets/1

Stacktrace
at webpack://@chia/wallets/dist/esm.js:395:53 syncing
at webpack://@chia/gui/node_modules/react-dom/cjs/react-dom.production.min.js:157:136 Ch
at webpack://@chia/gui/node_modules/react-dom/cjs/react-dom.production.min.js:180:153 oi
at webpack://@chia/gui/node_modules/react-dom/cjs/react-dom.production.min.js:269:342 a=b.pendingProps;f=e._init;e=f(e._payload);b.type=e;f=b.tag=hk(e);a=lg(e,a);switch
at webpack://@chia/gui/node_modules/react-dom/cjs/react-dom.production.min.js:250:346 Pj
at webpack://@chia/gui/node_modules/react-dom/cjs/react-dom.production.min.js:250:277 Pj
at webpack://@chia/gui/node_modules/react-dom/cjs/react-dom.production.min.js:250:137 Pj
at webpack://@chia/gui/node_modules/react-dom/cjs/react-dom.production.min.js:243:162 Lj
at webpack://@chia/gui/node_modules/react-dom/cjs/react-dom.production.min.js:123:114 jg
at webpack://@chia/gui/node_modules/scheduler/cjs/scheduler.production.min.js:18:342 unstable_next

Fixed via https://github.com/Chia-Network/chia-blockchain-gui/pull/749/files


Bug : Display and Scrolling Issue on Desktop View at User Profile Page

Describe the bug
When accessing the user profile page at https://app.opensauced.pizza/user/kubeskills/card, the full page content is not visible, nor is scrolling possible in the desktop view. However, when switching to mobile view or resizing the browser window to a mobile size, the content becomes fully visible and scrollable. This indicates a potential responsiveness issue where the desktop layout does not accommodate all content or provide scroll functionality, while the mobile layout does
https://github.com/open-sauced/app/assets/66130223/a350d97c-eeac-466b-9e4c-4c005f6fabe5
Steps to reproduce
1 .) Go to https://app.opensauced.pizza/user/kubeskills/card  ( When logged in )
2.) Try to scroll in a desktop view
Browsers
Chrome, Brave Browser
Additional context (Is this in dev or production?)
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Contributing Docs

[X] I agree to follow this project's Contribution Docs

I just saw someone already reported this issue, My bad. I'm going to close this issue.


Add basic Prometheus metrics for binlog-collector

The following metrics have been added:

pxc_binlog_collector_success_total
pxc_binlog_collector_failure_total
pxc_binlog_collector_last_processing_timestamp
pxc_binlog_collector_last_upload_timestamp
pxc_binlog_collector_gap_detected_total

Additionally, a simple /health endpoint has been added
CHANGE DESCRIPTION
Problem:
We don't have any prometheus metrics from pitr deployment, so the observability of binlog backups is quite limited.
Cause:
No prometheus metrics in binlog-collector application.
Solution:
Add prometheus metrics to binlog-collector application.
CHECKLIST
Jira

[ ] Is the Jira ticket created and referenced properly?
[ ] Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
[ ] Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

[ ] Is an E2E test/test case added for the new feature/change?
[ ] Are unit tests added where appropriate?
[ ] Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

[ ] Are all needed new/changed options added to default YAML files?
[ ] Are all needed new/changed options added to the Helm Chart?
[ ] Did we add proper logging messages for operator actions?
[ ] Did we ensure compatibility with the previous version or cluster upgrade process?
[ ] Does the change support oldest and newest supported PXC version?
[ ] Does the change support oldest and newest supported Kubernetes version?

Test name
Status

affinity-8-0
passed

auto-tuning-8-0
passed

cross-site-8-0
passed

custom-users-8-0
passed

demand-backup-cloud-8-0
passed

demand-backup-encrypted-with-tls-8-0
passed

demand-backup-8-0
passed

haproxy-5-7
passed

haproxy-8-0
passed

init-deploy-5-7
passed

init-deploy-8-0
passed

limits-8-0
passed

monitoring-2-0-8-0
passed

one-pod-5-7
passed

one-pod-8-0
passed

pitr-8-0
passed

pitr-gap-errors-8-0
passed

proxy-protocol-8-0
passed

proxysql-sidecar-res-limits-8-0
passed

pvc-resize-5-7
passed

pvc-resize-8-0
passed

recreate-8-0
passed

restore-to-encrypted-cluster-8-0
passed

scaling-proxysql-8-0
passed

scaling-8-0
passed

scheduled-backup-5-7
passed

scheduled-backup-8-0
passed

security-context-8-0
passed

smart-update1-8-0
passed

smart-update2-8-0
passed

storage-8-0
passed

tls-issue-cert-manager-ref-8-0
passed

tls-issue-cert-manager-8-0
passed

tls-issue-self-8-0
passed

upgrade-consistency-8-0
passed

upgrade-haproxy-5-7
passed

upgrade-haproxy-8-0
passed

upgrade-proxysql-5-7
passed

upgrade-proxysql-8-0
failure

users-5-7
passed

users-8-0
passed

validation-hook-8-0
passed

We run 42 out of 42

commit: https://github.com/percona/percona-xtradb-cluster-operator/pull/1916/commits/4dff4fd5d6fee4e773fc41c6461c203cf6abae45
image: perconalab/percona-xtradb-cluster-operator:PR-1916-4dff4fd5


api email Unknown route

It doesn't see email API?
Or did I miss something?
GET /email/domain/{domain}
./ovh-eu email help
Unknown route /email

The trick is: there is no top level "email" API. There are 3 sub-level APIs though:
./ovh-eu --help | grep email
    email-imap-copy                 Copy messages between two mailboxes
    email-exchange                  Operations about the EXCHANGE service
    email-domain                    Operations about the MX service

OK, thanks.
What is the "algorithm" to convert from REST url to command line?
When (why) such "trick" appears?

Each top level API is mapped to a top level command. Some top level commands have a "/" in it, it is replaces by a '-'


Update oidc.md

added some key notes that customers struggle with when using dev portal and oidc and kong manager as they test and then think about how to deploy in production.
Review

@reviewer
Summary

Many customers run into issues with using oidc with dev portal. this doc edit aims to clear up that confusion up front.
Reason

https://konghq.atlassian.net/browse/FTI-955
Many customers using Dev portal have been tripped up by this issue.
Testing

added a couple major clarifications to the dev portal that have tripped up customers recently.

@mannykhadilkar thank you for the PR. The docs team discussed this yesterday and we decided to add this to a troubleshooting section as a workaround until the FTI ticket is resolved. I've added @HenriPro as a SME eng reviewer of your proposed content.
I'll be out on PTO next week so will address this when I return, and wrap up 2.4 deliverables.
cc: @GayleNeumann

Support article is here: https://support.konghq.com/support/s/article/Seeing-an-developer-already-exists-when-registering-a-new-developer-in-the-Developer-Portal-using-OpenID and Manny is sadly no longer with us, so closing. We'll fix the docs when the underlying issue is fixed.


SCOM: MetricsTest does not provide a way to test the SCOM_minK value

(This may or may not be relevant if the proposal for #169 is approved.)
Problem:
As per #142, I am implementing calculation of SCOM_minK value for SCOM. I have the calculation ready. It will be stored in a new var in the metric.xml, but MetricsTest currently only tests the //class/@value of metrics.
Proposal
Ideally, enable tests of arbitrary xpaths in MetricsTest. Otherwise, enable tests of <var> using an xpath similar to : //var[@id = 'SCOM_minK'].

@yegor256/z please, pay attention to this issue

@llorllale but only @value matters for us.  I didn't understand, what is broken?

@yegor256 For SCOM_minK I need the tests to be more detail oriented.
SCOM_minK is another number calculated for SCOM separate from its @value. The paper says that if SCOM < SCOM_minK then you should consider splitting the class into more cohesive classes.

@llorllale we don't need to test those supplementary variables. Each metric has them. We just have them in the XML report for visibility. That's enough.