hubverse-transform: supply a parquet schema

Currently, hubverse-transform infers the parquet schema to apply when converted incoming model-output data to parquet. Because each file arrives and is transformed as a single unit, pyarrow has a limited amount of information to use when determining the parquet schema.
This resulted in some annoying behavior when working with the CDC's FluSight data. For example:
# when using pyarrow to read data
ArrowInvalid: Failed to parse string: 'large_increase' as a scalar of type double

# when using pandas.read_parquet method
ArrowInvalid: Failed to parse string: 'US' as a scalar of type int64

The current state is kind of workable but not ideal:

DuckDB has a "combine schema" option that handles different schemas on different underlying files
hubData can handle it
The pandas workaround is to read a single parquet file into a pyarrow table, and then that schema into the pandas read_parquet method 👎

I thought hubData was handling the differing schemas ok, but @elray1 reported this error message, which I suspect is related
> flusion_table <- hub_con %>%
+     filter(output_type == "quantile", location == "US") %>%
+     hubData::collect_hub()
Error in `dplyr::collect()`:
! NotImplemented: Function 'equal' has no kernel matching input types (string, int64)
Run `rlang::last_trace()` to see where the error occurred.

> rlang::last_trace()
<error/rlang_error>
Error in `dplyr::collect()`:
! NotImplemented: Function 'equal' has no kernel matching input types (string, int64)
---
Backtrace:
    ▆
 1. ├─... %>% hubData::collect_hub()
 2. └─hubData::collect_hub(.)
 3.   └─base::tryCatch(...)
 4.     └─base (local) tryCatchList(expr, classes, parentenv, handlers)
 5.       └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 6.         └─value[[3L]](cond)
Run rlang::last_trace(drop = FALSE) to see 2 hidden frames.

@annakrystalli I know you've wrestled with the mis-matched schema implication of location and output_type_id.
Curious to hear your thoughts: do you think it's reasonable to force those to column types to str when we're transforming the model-output files as part of syncing to the cloud?
Location seems like a good candidate for that type of brute force operation--it doesn't really make sense to have FIPS codes coming in as integers. Am less sure about output_type_id (FWIW, when creating the DuckDB file, I used the union_by_name option to merge schemas, which resulted in a VARCHAR data type for output_type_id

can we generate a schema based on contents of tasks.json?  i.e., if the data type of entries in tasks.json specifying options for a location is character, then we could infer that the schema should specify character type, but if it contains integer values then we could infer that the schema should specify integer type.

hubData::connect_hub() does use tasks.json to create a schema. There's a few things that may be going on but I'll need a more reproducible example to try and figure it out, i.e. the code to create the hub_con object in @elray's example.
The location column is coming out as character when you do collect so seems a bit strange that the filter wasn't working.

As I am curious about the solution, I encounter the same issue in the past and the only solution I found was the fix the input file.
I created a small example using on of our example hub for the hub_con object:
I first created 2 parquet file in a model-output folder that I created, with one file with a location column in numeric:
hub_path <- "./example-complex-forecast-hub/"
hub_con <- hubData::connect_hub(hub_path)
test <- hubData::create_model_out_submit_tmpl(hub_con, round_id = "2023-05-27")
a <- dplyr::filter(test, output_type == "quantile", location != "US") %>%
  dplyr::mutate(value = sample(seq(0,100, by = 0.1), nrow(.), replace = T),
                location = as.numeric(location))
b <- test %>%
  dplyr::mutate(value = sample(seq(0,100, by = 0.1), nrow(.), replace = T))
arrow::write_parquet(a, "model-output/teama-model/2023-10-14-teama-model.parquet")
arrow::write_parquet(b, "model-output/teamb-model/2023-10-14-teamb-model.parquet")

And when I try to read the files with one of the connect function:
hubData::connect_model_output("model-output/", file_format = "parquet") %>%
  dplyr::filter(output_type == "quantile", location == "US") %>%
  hubData::collect_hub()

I got the error:
Error in `arrow::open_dataset()`:
  ! Type error: Unable to merge: Field location has incompatible types: double vs string

even if I remove the filter:
hubData::connect_model_output("model-output/", file_format = "parquet") %>%
   #dplyr::filter(output_type == "quantile", location != "US") %>%
   hubData::collect_hub()
Error in `arrow::open_dataset()`:
! Type error: Unable to merge: Field location has incompatible types: double vs string
Run `rlang::last_trace()` to see where the error occurred.

And when I add the schema:
I got an error too (I different one but still an error):
config_json <- hubUtils::read_config(hub_path, "tasks")
or_schema <- hubData::create_hub_schema(config_json)
hubData::connect_model_output("model-output/", file_format = "parquet",
                              schema = or_schema) %>%
  dplyr::filter(output_type == "quantile", location == "US") %>%
  hubData::collect_hub()

Error in `dplyr::collect()`:
  ! NotImplemented: Function 'equal' has no kernel matching input types (string, double)

However if I comment the filter, it works:
> hubData::connect_model_output("model-output/", file_format = "parquet",
                               schema = or_schema) %>%
  # dplyr::filter(output_type == "quantile", location == "US") %>%
   hubData::collect_hub()
# A tibble: 1,028,224 × 9
   model_id  reference_date target horizon location target_end_date output_type output_type_id value
 * <chr>     <date>         <chr>    <int> <chr>    <date>          <chr>       <chr>          <dbl>
 1 teama-mo… 2023-05-27     wk in…       0 1        2022-10-22      quantile    0.01            88.6
 2 teama-mo… 2023-05-27     wk in…       1 1        2022-10-22      quantile    0.01            74.6
 3 teama-mo… 2023-05-27     wk in…       2 1        2022-10-22      quantile    0.01            44.9
 4 teama-mo… 2023-05-27     wk in…       3 1        2022-10-22      quantile    0.01            17.6
 5 teama-mo… 2023-05-27     wk in…       0 2        2022-10-22      quantile    0.01            57.1
 6 teama-mo… 2023-05-27     wk in…       1 2        2022-10-22      quantile    0.01            69.8
 7 teama-mo… 2023-05-27     wk in…       2 2        2022-10-22      quantile    0.01            69.4
 8 teama-mo… 2023-05-27     wk in…       3 2        2022-10-22      quantile    0.01            96.6
 9 teama-mo… 2023-05-27     wk in…       0 4        2022-10-22      quantile    0.01            17.4
10 teama-mo… 2023-05-27     wk in…       1 4        2022-10-22      quantile    0.01            63.8
# ℹ 1,028,214 more rows
# ℹ Use `print(n = ...)` to see more rows

When you add the schema option to the connect_model_output() you are overriding the unify_schemas = TRUE which is currently being passed to arrow::open_dataset().  This is why the error message is different.
In the first case (when you are not overriding unify_schemas=TRUE, unify_schemas itself fails (this failure happens even before a call to collect()), because it can't unify the different types (double and string) of location field in the two separate parquet files
In the second case, when you override unify_schemas=TRUE, by passing the or_schema object, you are indicating what the schema should be.. However, when you add a dplyr::filter(), the object that gets passed to collect() is of class arrow_dplyr_query Such an object first gets computed in arrow:::compute.arrow_dplyr_query() which fails, because it can't compute the query on the teama dataset.
In the absence of the dplyr::filter(), the object getting passed to collect() is of class Dataset, which goes to the Scan->Build->Finish pipeline without any Filter(Expression)) step.
You can even force the error that you see in the second scenario (with or_schema and with dplyr::filter() included), via arrow expressions... something like this:
# Create object of class `Dataset`
x = connect_model_output("model-output/"), file_format = "parquet", schema = or_schema)

# No error
ScannerBuilder$create(x)$Finish()$ToTable())

# Error if you add Filter Expression
ScannerBuilder$create(x)$Filter(Expression$field_ref("location") == "US")$Finish()$ToTable()

Just to follow on from above, you can cast to character. i.e. filter(x, as.character(location)=="US"); you can see that this type of dplyr::filter() call gets translated as:
ExecPlan with 4 nodes:
3:SinkNode{}
  2:ProjectNode{projection=[reference_date, target, horizon, location, target_end_date, output_type, output_type_id, value, model_id]}
    1:FilterNode{filter=(cast(location, {to_type=string, allow_int_overflow=false, allow_time_truncate=false, allow_time_overflow=false, allow_decimal_truncate=false, allow_float_truncate=false, allow_invalid_utf8=false}) == "US")}
      0:SourceNode{}

Or, equivalently
ScannerBuilder$create(x)$Filter(arrow:::cast(Expression$field_ref("location"), string()) == "US")$Finish()$ToTable()

library(dplyr)

There’s a lot going on here. I’ve done some experiments which can be
found in this fork and branch of
example-complex-scenario-hub.
I made a few changes to ensure files conformed to hub standards adn
could be validated.
All the code can be found in report.Rmd.
Differences in reading csvs between arrow and readr
The first issue appears to be a result of how arrow reads csvs,
especially compared to readr.
Below I create two csvs, both with US removed from the location
column. I then write them out using arrow and readr.
dir.create("test", showWarnings = FALSE)

arrow::read_csv_arrow(
  "model-output/HUBuni-simexamp/2021-03-07-HUBuni-simexamp.csv"
) %>%
  filter(location != "US") %>%
  slice(1:1000) %>%
  arrow::write_csv_arrow(file.path("test", "2021-03-07-HUBuni-simexamp-arrow.csv"))

readr::read_csv("model-output/HUBuni-simexamp/2021-03-07-HUBuni-simexamp.csv") %>%
  filter(location != "US") %>%
  slice(1:1000) %>%
  readr::write_csv(file.path("test", "2021-03-07-HUBuni-simexamp-readr.csv"))
#> Rows: 838656 Columns: 8
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr  (4): scenario_id, location, target, output_type
#> dbl  (3): horizon, output_type_id, value
#> date (1): origin_date
#>
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

CSVs can often be difficult to work with because they don’t explicitly
encapsulate a schema like e.g. parquet files do.
Looking at the structure of the two files created:
readr csv
origin_date,scenario_id,location,target,horizon,output_type,output_type_id,value
2021-03-07,A-2021-03-05,02,inc death,1,quantile,0.01,0
2021-03-07,A-2021-03-05,02,inc death,1,quantile,0.025,0
2021-03-07,A-2021-03-05,02,inc death,1,quantile,0.05,1
2021-03-07,A-2021-03-05,02,inc death,1,quantile,0.1,1
2021-03-07,A-2021-03-05,02,inc death,1,quantile,0.15,2

arrow csv
"origin_date","scenario_id","location","target","horizon","output_type","output_type_id","value"
2021-03-07,"A-2021-03-05","02","inc death",1,"quantile",0.01,0
2021-03-07,"A-2021-03-05","02","inc death",1,"quantile",0.025,0
2021-03-07,"A-2021-03-05","02","inc death",1,"quantile",0.05,1
2021-03-07,"A-2021-03-05","02","inc death",1,"quantile",0.1,1
2021-03-07,"A-2021-03-05","02","inc death",1,"quantile",0.15,2

you’d think arrow had done a better job of encoding the character
nature of the location column.
However, it seems that arrow does a worse job of reading in the
location column, regardless of what function was used to write the
file.
arrow does NOT correctly read in location, regardless of what function
was used to write the file,
arrow::read_csv_arrow(
  file.path("test", "2021-03-07-HUBuni-simexamp-arrow.csv")
)
#> # A tibble: 1,000 × 8
#>    origin_date scenario_id  location target   horizon output_type output_type_id
#>    <date>      <chr>           <int> <chr>      <int> <chr>                <dbl>
#>  1 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.01
#>  2 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.025
#>  3 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.05
#>  4 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.1
#>  5 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.15
#>  6 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.2
#>  7 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.25
#>  8 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.3
#>  9 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.35
#> 10 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.4
#> # ℹ 990 more rows
#> # ℹ 1 more variable: value <dbl>

arrow::read_csv_arrow(
  file.path("test", "2021-03-07-HUBuni-simexamp-readr.csv")
)
#> # A tibble: 1,000 × 8
#>    origin_date scenario_id  location target   horizon output_type output_type_id
#>    <date>      <chr>           <int> <chr>      <int> <chr>                <dbl>
#>  1 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.01
#>  2 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.025
#>  3 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.05
#>  4 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.1
#>  5 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.15
#>  6 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.2
#>  7 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.25
#>  8 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.3
#>  9 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.35
#> 10 2021-03-07  A-2021-03-05        2 inc dea…       1 quantile             0.4
#> # ℹ 990 more rows
#> # ℹ 1 more variable: value <dbl>

while readr correctly reads in location regardless of what function
was used to write the file.
readr::read_csv(file.path("test", "2021-03-07-HUBuni-simexamp-readr.csv"))
#> Rows: 1000 Columns: 8
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr  (4): scenario_id, location, target, output_type
#> dbl  (3): horizon, output_type_id, value
#> date (1): origin_date
#>
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
#> # A tibble: 1,000 × 8
#>    origin_date scenario_id  location target   horizon output_type output_type_id
#>    <date>      <chr>        <chr>    <chr>      <dbl> <chr>                <dbl>
#>  1 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.01
#>  2 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.025
#>  3 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.05
#>  4 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.1
#>  5 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.15
#>  6 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.2
#>  7 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.25
#>  8 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.3
#>  9 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.35
#> 10 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.4
#> # ℹ 990 more rows
#> # ℹ 1 more variable: value <dbl>

readr::read_csv(file.path("test", "2021-03-07-HUBuni-simexamp-arrow.csv"))
#> Rows: 1000 Columns: 8
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr  (4): scenario_id, location, target, output_type
#> dbl  (3): horizon, output_type_id, value
#> date (1): origin_date
#>
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.
#> # A tibble: 1,000 × 8
#>    origin_date scenario_id  location target   horizon output_type output_type_id
#>    <date>      <chr>        <chr>    <chr>      <dbl> <chr>                <dbl>
#>  1 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.01
#>  2 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.025
#>  3 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.05
#>  4 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.1
#>  5 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.15
#>  6 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.2
#>  7 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.25
#>  8 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.3
#>  9 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.35
#> 10 2021-03-07  A-2021-03-05 02       inc dea…       1 quantile             0.4
#> # ℹ 990 more rows
#> # ℹ 1 more variable: value <dbl>

Why is this relevant?
I suspect what is going on in the case of hub transformations and the
errors @bsweger is experiencing is that arrow might be converting
location to integers on read (in at least one file where there is no
“US” value?) and then writing them out as integers as well.
I have worried from the beginning about issues with the transformations
if they are not schema aware and I think this may be an example of that.
I think the best way to handle this is to make sure that the schema is
preserved when writing out the parquet files and test for it as well.
Differences in how schemas are applied to datasets between csv and parquet
Having said all that, there seem to be additional issues with arrow
datasets. All of this reminds of edge cases I’ve come across
before
and even reached out to the arrow team about but we were never able to
reproduce the issue when building up from smaller examples… I think we
may have just figured it out!
The problem seems to be parquet files and that applying the schema does
not appear to work in the same way as for csv files.
Create a csv file with integer location column
readr::read_csv("model-output/hubcomp_examp/2021-03-07.csv") %>%
  filter(location != "US") %>%
  mutate(location = as.integer(location)) %>%
  readr::write_csv(
    file.path(
      "model-output/hubcomp-examp",
      "2021-03-07-hubcomp-examp.csv"
    )
  )
#> Rows: 838656 Columns: 8
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr  (4): scenario_id, location, target, output_type
#> dbl  (3): horizon, output_type_id, value
#> date (1): origin_date
#>
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

Create a parquet file with integer location column
readr::read_csv("model-output/hubcomp_examp/2021-03-07.csv") %>%
  filter(location != "US") %>%
  mutate(location = as.integer(location)) %>%
  arrow::write_parquet(
    file.path(
      "model-output/hubcomp-examp",
      "2021-03-07-hubcomp-examp.parquet"
    )
  )
#> Rows: 838656 Columns: 8
#> ── Column specification ────────────────────────────────────────────────────────
#> Delimiter: ","
#> chr  (4): scenario_id, location, target, output_type
#> dbl  (3): horizon, output_type_id, value
#> date (1): origin_date
#>
#> ℹ Use `spec()` to retrieve the full column specification for this data.
#> ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

Connect to whole hub
Dataset opens but we get the familiar error when filtering
hubData::connect_hub(".")
#>
#> ── <hub_connection/UnionDataset> ──
#>
#> • hub_name: "Complex Scenario Hub"
#> • hub_path: '.'
#> • file_format: "csv(3/3)" and "parquet(5/5)"
#> • file_system: "LocalFileSystem"
#> • model_output_dir: "./model-output"
#> • config_admin: 'hub-config/admin.json'
#> • config_tasks: 'hub-config/tasks.json'
#>
#> ── Connection schema
#> hub_connection
#> origin_date: date32[day]
#> scenario_id: string
#> location: string
#> target: string
#> horizon: int32
#> output_type: string
#> output_type_id: double
#> value: double
#> model_id: string
#> age_group: string
#> target_date: date32[day]

hubData::connect_hub(".") %>%
  filter(location == "US") %>%
  hubData::collect_hub()
#> Error in `dplyr::collect()`:
#> ! NotImplemented: Function 'equal' has no kernel matching input types
#>   (string, int32)

Connect to parquet only
Same error as above
hubData::connect_hub(".", file_format = "parquet")
#>
#> ── <hub_connection/FileSystemDataset> ──
#>
#> • hub_name: "Complex Scenario Hub"
#> • hub_path: '.'
#> • file_format: "parquet(5/5)"
#> • file_system: "LocalFileSystem"
#> • model_output_dir: "./model-output"
#> • config_admin: 'hub-config/admin.json'
#> • config_tasks: 'hub-config/tasks.json'
#>
#> ── Connection schema
#> hub_connection with 5 Parquet files
#> origin_date: date32[day]
#> scenario_id: string
#> location: string
#> target: string
#> horizon: int32
#> age_group: string
#> target_date: date32[day]
#> output_type: string
#> output_type_id: double
#> value: double
#> model_id: string

hubData::connect_hub(".", file_format = "parquet") %>%
  filter(location == "US") %>%
  hubData::collect_hub()
#> Error in `dplyr::collect()`:
#> ! NotImplemented: Function 'equal' has no kernel matching input types
#>   (string, int32)

Connect to csv only
Works!
hubData::connect_hub(".", file_format = "csv")
#>
#> ── <hub_connection/FileSystemDataset> ──
#>
#> • hub_name: "Complex Scenario Hub"
#> • hub_path: '.'
#> • file_format: "csv(3/3)"
#> • file_system: "LocalFileSystem"
#> • model_output_dir: "./model-output"
#> • config_admin: 'hub-config/admin.json'
#> • config_tasks: 'hub-config/tasks.json'
#>
#> ── Connection schema
#> hub_connection with 3 csv files
#> origin_date: date32[day]
#> scenario_id: string
#> location: string
#> target: string
#> horizon: int32
#> output_type: string
#> output_type_id: double
#> value: double
#> model_id: string

hubData::connect_hub(".", file_format = "csv") %>%
  filter(location == "US") %>%
  hubData::collect_hub()
#> # A tibble: 29,952 × 9
#>    model_id      origin_date scenario_id  location target    horizon output_type
#>  * <chr>         <date>      <chr>        <chr>    <chr>       <int> <chr>
#>  1 hubcomp_examp 2021-03-07  A-2021-03-05 US       inc death       1 quantile
#>  2 hubcomp_examp 2021-03-07  A-2021-03-05 US       inc death       1 quantile
#>  3 hubcomp_examp 2021-03-07  A-2021-03-05 US       inc death       1 quantile
#>  4 hubcomp_examp 2021-03-07  A-2021-03-05 US       inc death       1 quantile
#>  5 hubcomp_examp 2021-03-07  A-2021-03-05 US       inc death       1 quantile
#>  6 hubcomp_examp 2021-03-07  A-2021-03-05 US       inc death       1 quantile
#>  7 hubcomp_examp 2021-03-07  A-2021-03-05 US       inc death       1 quantile
#>  8 hubcomp_examp 2021-03-07  A-2021-03-05 US       inc death       1 quantile
#>  9 hubcomp_examp 2021-03-07  A-2021-03-05 US       inc death       1 quantile
#> 10 hubcomp_examp 2021-03-07  A-2021-03-05 US       inc death       1 quantile
#> # ℹ 29,942 more rows
#> # ℹ 2 more variables: output_type_id <dbl>, value <dbl>

I suspect this is most likely a result of the fact that the schema is
supplied differently when opening csv vs parquet datasets (including
under the hood in connect_hub()).
config_tasks <- hubUtils::read_config(".", "tasks")
schema <- hubData::create_hub_schema(
  config_tasks,
  partitions = list(model_id = arrow::utf8())
)

In csv datasets we use argument col_types to specify the schema:
arrow::open_dataset(
  "model-output",
  format = "csv",
  partitioning = "model_id",
  col_types = schema,
  unify_schemas = TRUE,
  strings_can_be_null = TRUE,
  factory_options = list(exclude_invalid_files = TRUE)
) %>%
  filter(location == "US") %>%
  hubData::collect_hub()
#> # A tibble: 29,952 × 9
#>    model_id        origin_date scenario_id  location target  horizon output_type
#>  * <chr>           <date>      <chr>        <chr>    <chr>     <int> <chr>
#>  1 HUBuni-simexamp 2021-03-07  A-2021-03-05 US       inc de…       1 quantile
#>  2 HUBuni-simexamp 2021-03-07  A-2021-03-05 US       inc de…       1 quantile
#>  3 HUBuni-simexamp 2021-03-07  A-2021-03-05 US       inc de…       1 quantile
#>  4 HUBuni-simexamp 2021-03-07  A-2021-03-05 US       inc de…       1 quantile
#>  5 HUBuni-simexamp 2021-03-07  A-2021-03-05 US       inc de…       1 quantile
#>  6 HUBuni-simexamp 2021-03-07  A-2021-03-05 US       inc de…       1 quantile
#>  7 HUBuni-simexamp 2021-03-07  A-2021-03-05 US       inc de…       1 quantile
#>  8 HUBuni-simexamp 2021-03-07  A-2021-03-05 US       inc de…       1 quantile
#>  9 HUBuni-simexamp 2021-03-07  A-2021-03-05 US       inc de…       1 quantile
#> 10 HUBuni-simexamp 2021-03-07  A-2021-03-05 US       inc de…       1 quantile
#> # ℹ 29,942 more rows
#> # ℹ 2 more variables: output_type_id <dbl>, value <dbl>

while in parquet datasets we use argument schema to specify the
schema:
arrow::open_dataset(
  "model-output",
  format = "parquet",
  partitioning = "model_id",
  schema = schema,
  unify_schemas = TRUE,
  factory_options = list(exclude_invalid_files = TRUE)
) %>%
  filter(location == "US") %>%
  hubData::collect_hub()
#> Error in `dplyr::collect()`:
#> ! NotImplemented: Function 'equal' has no kernel matching input types
#>   (string, int32)

This does sort of beg the question of what is the purpose of schemas for
parquet files if they are not applied in the same way as for csv files.
I think this is a question for the arrow team.
The importance of validation
Note also that none of these files would pass validation. Given the
brittleness of arrow, validation of data on the way in is very
important. It seems like a lot of this issues are coming from hubs that
have not been fully validated or have been transformed in a non schema
aware way. There is a limit to what can be done to fix issues with the
data at access time. It seems like this is especially important for
parquet files, which effectively include a schema as part of file
metadata.
hubValidations::validate_submission(
  hub_path = ".",
  file_path = file.path(
    "hubcomp-examp",
    "2021-03-07-hubcomp-examp.parquet"
  ),
  skip_submit_window_check = TRUE
)
#> ✔ example-complex-scenario-hub: All hub config files are valid.
#> ✔ 2021-03-07-hubcomp-examp.parquet: File exists at path
#>   'model-output/hubcomp-examp/2021-03-07-hubcomp-examp.parquet'.
#> ✔ 2021-03-07-hubcomp-examp.parquet: File name
#>   "2021-03-07-hubcomp-examp.parquet" is valid.
#> ✔ 2021-03-07-hubcomp-examp.parquet: File directory name matches `model_id`
#>   metadata in file name.
#> ✔ 2021-03-07-hubcomp-examp.parquet: `round_id` is valid.
#> ✔ 2021-03-07-hubcomp-examp.parquet: File is accepted hub format.
#> ✔ 2021-03-07-hubcomp-examp.parquet: Metadata file exists at path
#>   'model-metadata/hubcomp-examp.yaml'.
#> ✔ 2021-03-07-hubcomp-examp.parquet: File could be read successfully.
#> ✔ 2021-03-07-hubcomp-examp.parquet: `round_id_col` name is valid.
#> ✔ 2021-03-07-hubcomp-examp.parquet: `round_id` column "origin_date" contains a
#>   single, unique round ID value.
#> ✔ 2021-03-07-hubcomp-examp.parquet: All `round_id_col` "origin_date" values
#>   match submission `round_id` from file name.
#> ✔ 2021-03-07-hubcomp-examp.parquet: Column names are consistent with expected
#>   round task IDs and std column names.
#> ! 2021-03-07-hubcomp-examp.parquet: Column data types do not match hub schema.
#>   `location ` should be "character " not "integer ", `horizon ` should be
#>   "integer " not "double "
#> ✖ 2021-03-07-hubcomp-examp.parquet: `tbl` contains invalid values/value
#>   combinations.  Column `location` contains invalid values "2", "1", "5", "4",
#>   "6", "8", and "9".

hubValidations::validate_submission(
  hub_path = ".",
  file_path = file.path(
    "hubcomp-examp",
    "2021-03-07-hubcomp-examp.csv"
  ),
  skip_submit_window_check = TRUE
)
#> ✔ example-complex-scenario-hub: All hub config files are valid.
#> ✔ 2021-03-07-hubcomp-examp.csv: File exists at path
#>   'model-output/hubcomp-examp/2021-03-07-hubcomp-examp.csv'.
#> ✔ 2021-03-07-hubcomp-examp.csv: File name "2021-03-07-hubcomp-examp.csv" is
#>   valid.
#> ✔ 2021-03-07-hubcomp-examp.csv: File directory name matches `model_id` metadata
#>   in file name.
#> ✔ 2021-03-07-hubcomp-examp.csv: `round_id` is valid.
#> ✔ 2021-03-07-hubcomp-examp.csv: File is accepted hub format.
#> ✔ 2021-03-07-hubcomp-examp.csv: Metadata file exists at path
#>   'model-metadata/hubcomp-examp.yaml'.
#> ✔ 2021-03-07-hubcomp-examp.csv: File could be read successfully.
#> ✔ 2021-03-07-hubcomp-examp.csv: `round_id_col` name is valid.
#> ✔ 2021-03-07-hubcomp-examp.csv: `round_id` column "origin_date" contains a
#>   single, unique round ID value.
#> ✔ 2021-03-07-hubcomp-examp.csv: All `round_id_col` "origin_date" values match
#>   submission `round_id` from file name.
#> ✔ 2021-03-07-hubcomp-examp.csv: Column names are consistent with expected round
#>   task IDs and std column names.
#> ✔ 2021-03-07-hubcomp-examp.csv: Column data types match hub schema.
#> ✖ 2021-03-07-hubcomp-examp.csv: `tbl` contains invalid values/value
#>   combinations.  Column `location` contains invalid values "2", "1", "5", "4",
#>   "6", "8", and "9".

Summary

Any transformations should be schema aware and we should have
integrity tests to ensure this.
when setting up hubs from historic data we should indeed try to run
validations on files (which may bump up the priority of
https://github.com/Infectious-Disease-Modeling-Hubs/hubValidations/issues/83
and
https://github.com/Infectious-Disease-Modeling-Hubs/hubverse-actions/issues/12).
This could help in identify potential issues earlier on. Having said
that, that might not always be possible. In such cases however, I
don’t think we can expect downstream data access functions to work
seamlessly on data that has not been validated.
Maybe we should just plug everything into a duckdb database!

Thank you all for the detailed answer!
I agree with Anna, that " best way to handle this is to make sure that the schema is preserved when writing out the parquet files and test for it as well." . That what I ended up doing too.
I am also curious about the " [..] purpose of schemas for .parquet files if they are not applied in the same way as for csv files. I think this is a question for the arrow team.".

Thanks for all of the great digging and background info--this is great!
It's worth noting that this class of error is occurring on validated files. So I agree that we need to get right with schemas to the extent possible, since different downstream tools are able to infer mixed schemas with varying results.
In the meantime, I updated the R snippet on the gist to this incantation that works with the problematic filter. This seems related to the prior note from @lmullany?
[not saying this is the ideal permanent solution, just wanted to give people a way to work with the files in their current state]
 hub_path_cloud <- s3_bucket('cdcepi-flusight-forecast-hub/')
data_cloud <- connect_hub(hub_path_cloud, file_format="parquet") %>% collect()

filtered <- data_cloud %>%
filter(output_type == "quantile", location == "US")

Doing the collect() before filtering solved the ! NotImplemented: Function 'equal' has no kernel matching input types (string, int64) issue. I'll let the R experts weigh in on other ways this approach can break things.

Yes we will have the same issue. I encounter the same issue in SMH.

It's worth noting that this class of error is occurring on validated files. So I agree that we need to get right with schemas to the extent possible, since different downstream tools are able to infer mixed schemas with varying results.

Are they validated files as submitted or are they files that have been converted to parquet via arrow? If they are files that actually passed validation on the way in then we need to flag those and understand what's going on a bit better.

Doing the collect() before filtering solved the ! NotImplemented: Function 'equal' has no kernel matching input types (string, int64) issue. I'll let the R experts weigh in on other ways this approach can break things.

This will work but it means loading the full dataset into memory before filtering.

Are they validated files as submitted or are they files that have been converted to parquet via arrow? If they are files that actually passed validation on the way in then we need to flag those and understand what's going on a bit better.

They are files that were validated when submitted as .csvs and then then converted to parquet via arrow (hence the question about other hubs that receive parquet submissions)

This will work but it means loading the full dataset into memory before filtering. That's really the opposite of what the workflow is designed to do so I would fix the data instead and change the snippet back.

For sure--but while we're still sorting through how to fix the data, it's worth presenting people with a short-term option for interacting with the FluSight data (which aren't large) using R.
I'm still looking to verify whether or not we're going to have the mis-matched schema issue on parquet files submitted by modelers (i.e., no cloud transforms). @LucieContamin has weighed in; my next step is to refer to Nick's list of hubverse hubs and see if there's anything on the list that uses parquet.

They are files that were validated when submitted as .csvs and then then converted to parquet via arrow (hence the question about other hubs that receive parquet submissions)

I think this is related to the first issue I highlighted at the beginning of my comment regarding arrow changing the data types of csv columns when it reads them in and then encoding that in the parquet file schema when writing out. So I think it's to do with the transformation to parquet when pushing to cloud. Non cloud hubs do not re-write any data post validation so shouldn't be affected.
I think if we make the cloud transformations schema aware, we won't get these issues.

I should mention that if a parquet file schema does not conform to the hub schema on the way in it does not pass validation.

I should also mention, to clarify, that the SMH file issue was on a file that was NOT validated with HubValidation, and will probably fail the validation.

@LucieContamin Thanks for the clarification--that's helpful!
I hope it's clear that I'm not pushing back on the idea of making the cloud transforms schema aware or advocating for less-optimal methods for data access!
My goal is to fully understand the problem space and apply the fix at the correct point in the process, so appreciate y'alls patience as I poke around.

I just wanted to note that, along with the data type issue,  the transformation has also changed the location values. In the files where the column was converted to integer, the leading zeros were dropped so some values no longer correspond to valid FIPS values:
library(dplyr)
library(hubData)
hub_path_cloud <- s3_bucket('cdcepi-flusight-forecast-hub/')
data_cloud <- connect_hub(hub_path_cloud, file_format="parquet") %>% collect()
data_cloud$location |> unique()
#> [1] "6"  "01" "02" "04" "05" "06" "08" "09" "10" "11" "12" "13" "15" "16" "17"
#> [16] "18" "19" "20" "21" "22" "23" "24" "25" "26" "27" "28" "29" "30" "31" "32"
#> [31] "33" "34" "35" "36" "37" "38" "39" "40" "41" "42" "44" "45" "46" "47" "48"
#> [46] "49" "50" "51" "53" "54" "55" "56" "72" "US" "1"  "2"  "5"  "9"  "4"  "8"

Yep, that makes sense.
I was looking at the FIPS codes also and have a somewhat related question about the validation process.
How do we want/expect modelers to respond to validation warnings vs errors? When running the validation process against a parquet file without all 50 states and no string value (like "US"), the validation will fail because the 01-09 FIPS codes get their leading zero dropped and don't validate against the list of valid locations in tasks.json.
! 2024-05-12-parquet-test_data.parquet: Column data types do not match hub schema.  `origin_date ` should be "Date " not "character ",
  `location ` should be "character " not "integer ", `output_type_id ` should be "character " not "double "
✖ 2024-05-12-parquet-test_data.parquet: `tbl` contains invalid values/value combinations.  Column `location` contains invalid values "9",
  "8", and "6".

But if you then remove all model-output entries for locations 01 - 09,  the validation returns the "data types do not match hub schema" warning but not the error (because two-digit FIPS code do match the list of valid locations).
It's a contrived edge case that probably isn't worth spending a lot of time on...more just an operational question. We don't expect downstream data access tools to work smoothly if hub users ignore validation errors. Do we have the same stance for warnings?

TL;DR: My example is a very contrived edge case, and I agree that our time is best spent making the cloud transform schema-aware. If we want to dive into this further, probably worth switching to a synchronous convo.
To reproduce:

Clone this fork of the rsv-forecast-hub repo (I forked this one because it's small and accepts parquet submissions). The fork contains a model-output file with location FIPS codes > 10 (and no "US")
Verify that the above model-output file validates with a warning but no error:

hubValidations::validate_submission(
    hub_path="~/code/rsv-forecast-hub/",
    file_path="parquet-test_data/2024-05-05-parquet-test_data.parquet")

Reproduce the data issue:

> rsv_path <- "~/code/rsv-forecast-hub"
> rsv_con <- connect_hub(rsv_path)
> rsv_con %>% filter(location=="US") %>% collect()
Error in `compute.arrow_dplyr_query()`:
! NotImplemented: Function 'equal' has no kernel matching input types (string, int64)

Again, this is a very contrived example, and I don't think it's worth spending a ton of time on.
Essentially, in this scenario, location values are interpreted as integers when the modeler writes out the parquet file. However, two digit integers match the list of accepted locations, so the validation process throws the warning but no error.
If the modeler had included a location value of 09, for example, parquet writes it as 9, which does not match the accepted value of 09 and fails validations.

During the recent Hubverse retreat, @annakrystalli @LucieContamin and I determined that the best way forward here is to port the R-based create-hub-schema function (from the hubData package) to Python.
Then the transform package can enforce a schema when reading incoming model-output files.
As a first step, this PR refactors the transform function to ensure that it will have a more straightforward way to access a hub's hub-config directory: https://github.com/hubverse-org/hubverse-transform/pull/20

Am moving this out of "in progress" state in favor of the temporary, forced schema fix outlined in #24
Inferring a hub's correct schema from the config files is still the plan, but that work has temporarily taken a back seat to other cloud-based concerns.


log block info in deploy script

Goals of PR
needed for magi sequencing

Re: my comments on main vs. develop this is good to go into main, ignore what I said before.


Add ability to get skipped tests in embedding-utils.js

What is your Test Scenario?

I am trying to add some personal static analysis to my testcafe tests according to different features I am testing. I found there is getTypeScriptTestList in the following API, https://github.com/DevExpress/testcafe/blob/fcd93e8dbee76ea70786ee3aee4fb35990260e2a/src/embedding-utils.js#L54 (I am using Typescript), and I can get a list of the tests with very basic info without running all the tests, however, there seems no information getting the skipped tests, and it would be great if it's also included.
What are you suggesting?

The current returned properties include: name, loc, start, end, and meta.
It would be great to add one like skipped?
What alternatives have you considered?

Maybe some more info can be added as well that I am currently not aware of?
Additional context

I found this feature request https://github.com/DevExpress/testcafe/issues/4340 and mine is more like adding one more property based on it.
Sorry, just found the feature request was not similar, mine is adding one more ability in the current existing embedding-utils.
Not sure if it's possible to add, many thanks to testcafe team and the community!

Hello.
Currently, we do not plan to modify these utils. If you want, you can research the possibility of adding a new property and make a PR.


Implement 'cancel' in batch edit

When cancelling in batch edit, the state should be reset to what it was initially. This might be easily achievable by leveraging the hashchange logic.

Assigning to @Coenego for review

Coverage remained the same at 100.0% when pulling 535fb778bbeca654bb5dc6a5fe56a8bc7bf5ee07 on bp323:cancelEdit into 086c1753472a2405d91569171847491def078eea on CUL-DigitalServices:master.


Is the environment of xmem the same as that of stcn?

Is the environment of xmem the same as that of stcn? Why can't the environment of stcn run xmem on the same machine?

The environment is listed here: https://github.com/hkchengrex/XMem/blob/main/docs/GETTING_STARTED.md
Most of them are the same except PyTorch. I usually see the latest version of PyTorch for development. Some of the newer features are not supported by the old PyTorch version used in STCN.

The environment is listed here: https://github.com/hkchengrex/XMem/blob/main/docs/GETTING_STARTED.md
Most of them are the same except PyTorch. I usually use the latest version of PyTorch for development. Some of the newer features are not supported by the old PyTorch version used in STCN.

但是我使用同样的机器也要更新pytorch版本吗？


Needs Realtime metrics showing its activity of merge and push

We observed that realtime stopped merge and push when its ZK connection was interrupted. Every metric including "processed events" was fine and it made us really hard to debug why druid was losing messages.

Fixed by #1021


Restart is failing on Windows when the service is not started

[2018-02-18 23:27:29] [NOTICE] Source [https://download.dnscrypt.info/resolvers-list/v2/public-resolvers.md] loaded
[2018-02-18 23:27:29] [NOTICE] dnscrypt-proxy 2.0.0
[2018-02-18 23:27:29] [FATAL] Failed to restart DNSCrypt client proxy: Le service n’a pas été démarré.

I guess you expect the service to be started if it's not (that's what systemctl does).
I that's probably more an upstream (https://github.com/kardianos/service/blob/master/service_windows.go#L322) consistency problem, but I'm posting here anyway.
(I migrated to v2 yesterday from 2 v1 instances, but I'm experiencing issues: sometime resolution fails, and it even completely stopped working a some point, I enabled logs & hope to have something to post)

That's... expected. At least, it makes sense. "restart" means "re-start" :)

It doesn't feel natural .. but ok, I'll keep my old sc stop & sc start then :)
Thanks for the clarification.

Probably it should emit a log entry or even a warning of some sort.
Like "Service is not running. Do you want to start it? Please consider using a Start command."


Add {channel}_sum_rel column for overall_df_with_rel dataframe

Is your feature request related to a problem? Please describe.
We do autofluorescence subtractions on individual "cells", so we can't use the existing combined mother-bud value from cca_functions.calculate_relatives_data(). We also use a flatfield background correction, so we would like to use the raw sum values. However, we don't have a sum_rel data, so we can't add mother and bud values.
Describe the solution you'd like
Add {channel}_sum_rel column for overall_df_with_rel dataframe
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

Hi Francesco,
Thank you for your quick response
Is there a new version I can pull with this update?
Jacob

Hi Jacob, I just released v1.2.4rc.15  update with pip install --upgrade cellacdc
When you run the notebook you will now get a new column called {channel}_raw_sum_rel plus the sum of mother and bud in the column called {channel}_combined_raw_sum_mother_bud
Let me know if it works, thanks!


cannot edit in read only mood

"After compiling a code in v

s code i can't write anything in the terminal,it just read the code.In the vs code settings code runner:Run in the Terminal option is enabled.Last three or four days i faces this problem."
please give me any solution

An extension is launching that terminal as it's called "Compile Run"


Safari 8.0 OAuth Login and Cookie Settings

Whenever I try to log in to drone using Safari, I get redirected to GitHub to log in, type in my name and password, and then get an error message from GitHub. It doesn't appear to redirect back to drone. However, I'm already logged in to GitHub and using it.
My only guess is an issue with redirects or cookies.

I'm going to close this as an upstream issue since there isn't much we can do on our end. I appreciate the effort that went into debugging and documenting the issue. It has helped a number of other projects that were seeing similar issues.

@foresmac Did you ever hear back from GitHub or find an adequate solve? I'm clueless what to do for all my Safari 8 users with issues

I just basically had to change the settings back to "Allow from websites I visit". If there's any other workaround, I haven't been privy to it.

I opened a radar about this a while back. And, it sounds like it is a bug that is supposedly fixed in the public beta of the next OS release: https://bugs.webkit.org/show_bug.cgi?id=139683


Database Restore

Issue Type: Bug
Hi,
I'm trying to restore a SQL Server database from a .bak file. When I'm on the server dashboard from localhost, I go to click the Restore button as that is what all tutorials are showing. I then get the error:
Restore command is not supported in Server Context. Please select a Database and try again.
I can find virtually no info on this warning so I am at a standstill.
I have tried to restore the database via a few other methods but have had no luck. This seems to be the simplest way from all the tutorials I have found, but unfortunately I am having no luck.
Would love any kind of help!
Peter
Azure Data Studio version: Azure Data Studio 1.9.0 (78a42e1d112ae3231777722b51eaf44f83ddbe55, 2019-07-10T04:16:23.244Z)
OS version: Darwin x64 18.6.0

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz (12 x 2600)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: enabledrasterization: enabledsurface_synchronization: enabled_onvideo_decode: enabledwebgl: enabledwebgl2: enabled

Load (avg)
3, 3, 3

Memory (System)
16.00GB (1.65GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions: none

@PGrantcharov thanks for reporting the issue. I understand that the presence of restore button on server tab is contradicting with the warning message. We will work to fix that.
As a workaround - Please navigate to a database dashboard by double clicking a db name in server dashboard OR right clicking a database in Connections view and select Manage. Once the database dashboard opens - select backup/restore functionality from this dashboard and proceed with restore. You can choose any database for target (it does not matter which database you started the restore from.

Please let us know if this unblocks you.

The issue here is when a developer is attempting to create a database from a .bak file via the Restore method, which is commonly done in SSMS. The use case here is when folks are moving a db from one development machine to another, or when initially creating a database for a project from a provided .bak file. In my case today, I was installing some tutorial files from a GitHub repo that contained a .bak file for database creation. This is a pretty common use case.

Restore from Server Dashboard is re-enabled in latest insiders build so closing this bug. Please reactivate/let us know, if you still see this issue.


Fix typo in the exercise "testing-code" workflow implementation

What was changed

There is a typo in the testing-code exercise that caused a mismatch with the test assertion
Why?

Test cases will be always failed in the exercise
Checklist

Closes  (None)

How was this tested:

Run through the steps in the exercise readme.

Any docs updates needed?

 None

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Closed this PR. I just realize the bug is by intention.


Fetch job's logs since specific timestamp

Right now it's only possible to read job's logs from the end with --tail flag. This makes it hardly possible to look up the logs from a few days ago (--tail=1000000 in combination of a lot of patience). There should be --since parameter (and maybe --until) allowing to do that like it is in docker:
       --since=""       Show  logs since timestamp (e.g. 2013-01-02T13:23:37Z) or rela‐
       tive (e.g. 42m for 42 minutes)
       -n, --tail="all"      Number of lines to show from the end of the logs
       --until=""      Show logs before a timestamp (e.g. 2013-01-02T13:23:37Z) or rel‐
       ative (e.g. 42m for 42 minutes)

It refers to reading logs by CLI Racetrack client, but maybe this mechanism could be implemented in Dashboard ("Runtime logs" button) too, if possible.

I don't disagree with the feature, but is there someone specific asking for this right now?

No one asked about it now. Maybe some day someone complained about too much logs to browse, but it was 11 months ago when the original issue was created. I've transferred it from an old issue tracker.


boards: mimxrt*: Temporarily disable Ethernet [REVERT ME]

Many of the networking PRs are failing in these boards because
of this build error:
ERROR: section nocache is 128 bytes in the virtual/runtime address
space, but only 0 in the loaded/XIP section!
See #16778 for details.
So temporarily disable Ethernet support so that unrelated PRs can
be merged. This should be reverted after the root cause is found
and #16778 is fixed.
Signed-off-by: Jukka Rissanen<EMAIL_ADDRESS>

We have been agreeing already that the whitelisting in various networking tests should be removed. So if some board is no longer working properly, then we should avoid trying to compile that board. Easiest way is to disable sanitychecker to use that board for networking tests that want ethernet interface.

So if some board is no longer working properly, then we should avoid trying to compile that board.

 If you're too, then I wonder why my proposal to disable misbehaving sam_e70_xplained boards wasn't supported for ~half a year?

Easiest way is to disable sanitychecker to use that board for networking tests that want ethernet interface.

That's the problem with this change - the config for this board says that it has Ethernet, and that's true, it has, on hardware level. You change says that it doesn't have Ethernet, which is well, not true. If you want to disable that board by blacklisting it from particular tests, I'm +1 right away.

You change says that it doesn't have Ethernet, which is well, not true.

And besides, the build failure doesn't even happen because of the Ethernet, it happens with some link-time helper script.

I just want to the PRs to progress and get merged. If there is proper fix, that is just fine. I do not want to start to add the whitelists again to multiple PRs if there is a simple and temporary workaround.

No need for this as #16923 solves the issue.


problem

I encountered this error using youtube-dl version 2015.02.04
/usr/bin/youtube-dl https://www.youtube.com/watch?v=Bb4TPuXCd8U
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/bin/youtube-dl/main.py", line 19, in
File "/usr/bin/youtube-dl/youtube_dl/init.py", line 382, in main
File "/usr/bin/youtube-dl/youtube_dl/init.py", line 372, in _real_main
File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1352, in download
File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 619, in extract_info
File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 268, in extract
File "/usr/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 802, in _real_extract
File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 396, in _download_webpage
File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 320, in _download_webpage_handle
File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 294, in _request_webpage
File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 460, in report_download_webpage
File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 452, in to_screen
File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 388, in to_screen
File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 402, in to_stdout
File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 391, in _write_string
File "/usr/bin/youtube-dl/youtube_dl/utils.py", line 943, in write_string
IOError: [Errno 32] Broken pipe
It has done it several times with any youtube video
I tried to look at the files mentioned in the error message but I can't find them.
Sorry I'm a bit of a n00b at this.

I might mention that stdin & stdout are being redirected and this seems to be the root of the problem.

After reading http://stackoverflow.com/questions/15793886/how-to-avoid-a-broken-pipe-error-when-printing-a-large-amount-of-formatted-data
I added a -q to the command line and it provided a work-around.


Add BHive Comparison script

This patch adds a script for comparing BHive format CSVs (the first column being the hex bb and the second column being the throughput value). The script just outputs the average deviation between the two values.

Opening this as a draft for now as it needs some additional work and isn't ready for review. Need something to remind me that it exists though when I go back to find it.


BullMQ is not compatible

import * as trpcNext from "@trpc/server/adapters/next";
import { appRouter } from "@queuedash/api";
import Bull from "bull";
import { Queue } from "bullmq";

export default trpcNext.createNextApiHandler({
  router: appRouter,
  onError({ error }) {
    if (error.code === "INTERNAL_SERVER_ERROR") {
      // send to bug reporting
      console.error("Something went wrong", error);
    }
  },
  batching: {
    enabled: true,
  },
  createContext: () => ({
    queues: [
      {
        // queue: new Bull("sbt_airdrop", {
        //   redis: {
        //     host: "localhost",
        //     port: 6379,
        //   }
        // }),
        queue: new Queue("sbt_airdrop", {
          connection: {
            host: "localhost",
            port: 6379,
          },
        }),
        displayName: "SBT Airdrop",
        type: "bull" as const,
      },
    ],
  }),
});

Something went wrong xe [TRPCError]: n.queue.client.info is not a function
    at /Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:41:4620
    at d (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1757)
    at Object.next (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1040)
    at c (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:619)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'INTERNAL_SERVER_ERROR',
  [cause]: undefined
}
Something went wrong xe [TRPCError]: n.queue.client.info is not a function
    at /Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:41:4620
    at d (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1757)
    at Object.next (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1040)
    at c (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:619)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'INTERNAL_SERVER_ERROR',
  [cause]: undefined
}
Something went wrong xe [TRPCError]: n.queue.client.info is not a function
    at /Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:41:4620
    at d (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1757)
    at Object.next (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1040)
    at c (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:619)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'INTERNAL_SERVER_ERROR',
  [cause]: undefined
}
Something went wrong xe [TRPCError]: n.queue.client.info is not a function
    at /Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:41:4620
    at d (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1757)
    at Object.next (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1040)
    at c (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:619)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'INTERNAL_SERVER_ERROR',
  [cause]: undefined
}
Something went wrong xe [TRPCError]: n.queue.client.info is not a function
    at /Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:41:4620
    at d (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1757)
    at Object.next (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1040)
    at c (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:619)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'INTERNAL_SERVER_ERROR',
  [cause]: undefined
}
Something went wrong xe [TRPCError]: n.queue.client.info is not a function
    at /Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:41:4620
    at d (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1757)
    at Object.next (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1040)
    at c (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:619)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'INTERNAL_SERVER_ERROR',
  [cause]: undefined
}
Something went wrong xe [TRPCError]: n.queue.client.info is not a function
    at /Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:41:4620
    at d (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1757)
    at Object.next (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:1040)
    at c (/Users/liho/Desktop/queuedash/examples/with-next/node_modules/@queuedash/api/dist/main.js:1:619)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 'INTERNAL_SERVER_ERROR',
  [cause]: undefined
}

@lihoaigc try with bullmq as the type instead of bull


CI: Run tests in Cygwin for Cygwin CI

The current version installs the Cygwin base packages (but not any python versions), then runs the rest of the steps using the previously-installed Windows-native python.  This should run the tests in a Cygwin python.

I think it's gonna be a while, so I am gonna rebase and merge this PR.

I think it's gonna be a while, so I am gonna rebase and merge this PR.

Should we install GitPython from the repository rather than from PyPI so the fix is included?  I think the tests still fail with the released version.

We are now installing from the repo but there are still errors, I don't know what's going on. @DWesl can you try running the tests with nox on a Cygwin environment?

@DWesl The current way to generate wheel tags from extension filenames is doomed. I think the sane approach is the one in #202. It should already work with Cygwin but IIUC our CI setup is incomplete. Do you mind if I cherry pick the CI extension commits from this PR into that one?

We are now installing from the repo but there are still errors, I don't know what's going on. @DWesl can you try running the tests with nox on a Cygwin environment?

The most recent commit passes locally, except for what I suspect are local network errors.

@DWesl The current way to generate wheel tags from extension filenames is doomed. I think the sane approach is the one in #202. It should already work with Cygwin but IIUC our CI setup is incomplete. Do you mind if I cherry pick the CI extension commits from this PR into that one?

You are welcome to try it.

CI is not very happy. There seems to be something wrong with git in Cygwin.

Superseded by #202


Potential security vulnerability

A potential security vulnerability has been disclosed for this repo https://huntr.dev/bounties/d6418ac2-fce8-4963-b6e5-9b0025238451/ .
But i see it still not addressed .
Kindly validate the bug

this report has been submitted 3 month ago

Thank you @ranjit-git, could you please disclose this to<EMAIL_ADDRESS>I've also issued a PR adding a SECURITY.md for future reports.


Fixes #93 by passing full URI to redis.createClient when specified

My initial approach was to use url.parse() from the Node.js URL module to get the hostname and port. This did work, but I noticed that redis also lets you specify the database index and other options via the URL, so it seemed better to pass it wholesale to the createClient call if specified.

Hi @objectiveSee, thanks for taking a look at this. I've made the suggested review markup and pushed to the PR branch. Is there anything else I can do to help move this PR along?

Hi @objectiveSee, @BridgeAR. Having reviewed this, would you be happy to merge it? - It's been open for a while now...

Thank you !

Thanks @rauchg. Do you know when you'll next be publishing this package?


TypeError in /clan/bam-bam-ops-2j0j9c8r

Error in ClashLeaders
TypeError in /clan/bam-bam-ops-2j0j9c8r
Cannot read property '__symbol@@0.3433117047347418' of undefined
View on Bugsnag
Stacktrace
https://www.clashleaders.com/static/js/vendors.8d68595a3daede5c357b.js:13 - s.Et.Ht.get [as __symbol@@0.3433117047347418]

View full stacktrace
Created automatically via Bugsnag

An error linked to this issue has been marked as fixed in Bugsnag
TypeError in /clan/bam-bam-ops-2j0j9c8r


SAL.FS: Removing all symlinks under diectory fails if this directory contains a symlink for a directory.

Steps:
1- Create a directory test_dir.
2- Create a file test_file inside test_dir
3- Create a sub directory subdir under test_dir.
4- Create links for test_dir and the destination sym_test
j.sal.fs.symlinkFilesInDir("/tmp/test_dir", "/tmp/sym_test", includeDirs=True)

5- Remove all symlinks in the destination directory.
j.sal.fs.removeLinks("/tmp/sym_test")

Result:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/sandbox/lib/jumpscale/Jumpscale/sal/fs/SystemFSDecorators.py", line 213, in wrapper
    return func(*args, **kwargs)
  File "/sandbox/lib/jumpscale/Jumpscale/sal/fs/SystemFS.py", line 674, in removeLinks
    self.unlink(item)
  File "/sandbox/lib/jumpscale/Jumpscale/sal/fs/SystemFSDecorators.py", line 182, in wrapper
    % (parameter.name, jslocation(), func.__name__, value)
Jumpscale.core.InstallTools.JSExceptions.__init__.<locals>.Value1: ?[1;31mEXCEPTION:
    Argument filename in j.sal.fs.unlink expects a file path! /tmp/sym_test/subdir is not a file.?[0;0m

?[1;31mEXCEPTION:
    Argument filename in j.sal.fs.unlink expects a file path! /tmp/sym_test/subdir is not a file.?[0;0m

Fix : https://github.com/threefoldtech/jumpscaleX_core/commit/5188c5ffc78e27d07ce2d4f6ffdbf1ce8c6f7c2f

verified on development 5188c5f
JSX> j.sal.fs.symlinkFilesInDir("/tmp/test_dir", "/tmp/sym_test", includeDirs=True)
JSX> j.sal.fs.removeLinks("/tmp/sym_test")


Cannot Pass Literals are argument that aren't hashable

Reporting a bug

[X] I have tried using the latest released version of Numba (most recent is
visible in the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG).
[X] I have included a self contained code sample to reproduce the problem.
i.e. it's possible to run as 'python bug.py'.

Passing literals that aren't hashable fails at lowering in Numba. Here is a reproducer with a slice.
import numba

@numba.njit
def f(x):
    slc = numba.literally(x)
    return slc

print(f(slice(1, 2)))

Which produces the following stack trace
Traceback (most recent call last):
  File "test_unhashable_literal.py", line 8, in <module>
    print(f(slice(1, 2)))
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/dispatcher.py", line 443, in _compile_for_args
    return_val = self._compile_for_args(*args)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/dispatcher.py", line 487, in _compile_for_args
    raise e
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/dispatcher.py", line 420, in _compile_for_args
    return_val = self.compile(tuple(argtypes))
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/dispatcher.py", line 965, in compile
    cres = self._compiler.compile(args, return_type)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/dispatcher.py", line 125, in compile
    status, retval = self._compile_cached(args, return_type)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/dispatcher.py", line 139, in _compile_cached
    retval = self._compile_core(args, return_type)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/dispatcher.py", line 152, in _compile_core
    cres = compiler.compile_extra(self.targetdescr.typing_context,
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/compiler.py", line 693, in compile_extra
    return pipeline.compile_extra(func)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/compiler.py", line 429, in compile_extra
    return self._compile_bytecode()
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/compiler.py", line 497, in _compile_bytecode
    return self._compile_core()
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/compiler.py", line 472, in _compile_core
    raise e
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/compiler.py", line 463, in _compile_core
    pm.run(self.state)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/compiler_machinery.py", line 349, in run
    raise e
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/compiler_machinery.py", line 341, in run
    self._runPass(idx, pass_inst, state)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/compiler_machinery.py", line 296, in _runPass
    mutated |= check(pss.run_pass, internal_state)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/compiler_machinery.py", line 269, in check
    mangled = func(compiler_state)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/typed_passes.py", line 398, in run_pass
    lower.create_cpython_wrapper(flags.release_gil)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/lowering.py", line 292, in create_cpython_wrapper
    self.context.create_cpython_wrapper(self.library, self.fndesc,
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/cpu.py", line 184, in create_cpython_wrapper
    library.add_ir_module(wrapper_module)
  File "/Users/nicholasriasanovsky/Documents/bodo/nick-numba/numba/core/codegen.py", line 727, in add_ir_module
    ir = cgutils.normalize_ir_text(str(ir_module))
  File "/Users/nicholasriasanovsky/miniconda3/envs/NUMBA55/lib/python3.8/site-packages/llvmlite/ir/module.py", line 242, in __repr__
    lines += self._get_body_lines()
  File "/Users/nicholasriasanovsky/miniconda3/envs/NUMBA55/lib/python3.8/site-packages/llvmlite/ir/module.py", line 212, in _get_body_lines
    lines += [str(v) for v in self.globals.values()]
  File "/Users/nicholasriasanovsky/miniconda3/envs/NUMBA55/lib/python3.8/site-packages/llvmlite/ir/module.py", line 212, in <listcomp>
    lines += [str(v) for v in self.globals.values()]
  File "/Users/nicholasriasanovsky/miniconda3/envs/NUMBA55/lib/python3.8/site-packages/llvmlite/ir/values.py", line 1002, in __str__
    self.descr(buf)
  File "/Users/nicholasriasanovsky/miniconda3/envs/NUMBA55/lib/python3.8/site-packages/llvmlite/ir/values.py", line 997, in descr
    self.descr_body(buf)
  File "/Users/nicholasriasanovsky/miniconda3/envs/NUMBA55/lib/python3.8/site-packages/llvmlite/ir/values.py", line 991, in descr_body
    blk.descr(buf)
  File "/Users/nicholasriasanovsky/miniconda3/envs/NUMBA55/lib/python3.8/site-packages/llvmlite/ir/values.py", line 1131, in descr
    buf += ["  {0}\n".format(instr) for instr in self.instructions]
  File "/Users/nicholasriasanovsky/miniconda3/envs/NUMBA55/lib/python3.8/site-packages/llvmlite/ir/values.py", line 1131, in <listcomp>
    buf += ["  {0}\n".format(instr) for instr in self.instructions]
  File "/Users/nicholasriasanovsky/miniconda3/envs/NUMBA55/lib/python3.8/site-packages/llvmlite/ir/_utils.py", line 48, in __str__
    s = self.__cached_str = self._to_string()
  File "/Users/nicholasriasanovsky/miniconda3/envs/NUMBA55/lib/python3.8/site-packages/llvmlite/ir/values.py", line 548, in _to_string
    self.descr(buf)
  File "/Users/nicholasriasanovsky/miniconda3/envs/NUMBA55/lib/python3.8/site-packages/llvmlite/ir/instructions.py", line 218, in descr
    .format(self.opname, return_value.type,
AttributeError: 'tuple' object has no attribute 'type'

The source of the problem appears to be serialize_object because the line gv = self.module.__serialized[obj] always fails when a literal isn't hashable.
I think a reasonable solution would be to support passing a key to serialize_object. box_literal could then construct a unique object to use as a key.
A possible implementation could build a wrapper type around the literal type that reuse the literal type's hash (but uses a different object to avoid collisions with actually serializing the type).

You can assign this issue to me. I opened a first PR, but if this needs a discussion I'm happy to make changes.


#420

This is good, you should render the attachment at size one-third rather than two-thirds for performance.
I would suggest one-sixth but that's actually too small even for your grid.

closes #420


357 explainability finalize output

replaced by https://github.com/np-guard/vpc-network-config-analyzer/pull/378


#5854 – Add "Copy to Clipboard" to the export window

How the feature works? / How did you fix the issue?
This PR adds a 'Copy to clipboard' button to the export window in both micromolecule and macromolecule modes
Check list

[ ] unit-tests written
[ ] e2e-tests written
[ ] documentation updated
[x] PR name follows the pattern #1234 – issue name
[x] branch name doesn't contain '#'
[x] PR is linked with the issue
[x] base branch (master or release/xx) is correct
[x] task status changed to "Code review"
[x] reviewers are notified about the pull request

@NataliaLoginova @mnmsvlw Please add a test-id to this button during the development phase, if possible. This will make automation easier for us.

The test id "copy-to-clipboard" has been added to this button


Update hello-world.md

Added an ES6 syntax example. which helps to learners.
Thank you for the PR! Contributors like you keep React awesome!
Please see the Contribution Guide for guidelines:
https://github.com/reactjs/reactjs.org/blob/master/CONTRIBUTING.md
If your PR references an existing issue, please add the issue number below:
#

Closing this. We have plenty of ES6 examples in our docs now.


How to enable vllm

Feature request
How to enable vllm
Motivation
How to enable vllm
Your contribution
How to enable vllm

Use 0.9 and a supported model

@OlivierDehaene Hi, don't if you are stuff in huggingface,
but PLEASE indicates more details not just type use 0.9

Does it enabled by default?
if not, how to enable?
any documentations indicates how to use?

can be so impatient to github community member.

Please read your initial "Feature request" and tell me you did the effort to actually express intelligibly a feature request ?
Is this a bug ?
There are many ways you could have phrased that, starting with a request to improve the docs because you couldn't find what you want.
Our effort in replies scale with your effort.

Yes
It's always on ,there's no opt-out, it's just better overall that what was there previously.

There's no doc for it because well, it's not necessary, if you use flash models, you just get it.
It's also NOT vllm. It's a custom variant of PagedAttention, which is what makes vllm faster. We do reuse a slightly modified version of their low level kernel.

@Narsil Hi, where does PagedAttention custom kernel included in?


[pylint] use git ls-files for pylint

Use git ls-files to get list of python files
instead of using find command.
Fix pylint issues in exceptions.py
Closes: #217

See https://github.com/canonical/hotsos/issues/217#issuecomment-975374982

See #217 (comment)

Used options to git ls-files to retreive untracked files: see commit 3c6cc5f66367c5707820a4d41ae813a732b4216b

See #217 (comment)

Done

Squash the commits?

Squash the commits?

Done


uninstalling soundcast?

can you please give me instructions on how to fully uninstall soundcast 1.7?

@meri434 just delete it from the Applications folder. If you also want to uninstal Soundflower, follow this instructions (use the same version you installed)


Update covid19israel.source-spec.yaml

removing irrelevant dependency.

this causes more pipelines to run in parallel, is there a reason you want to do that?

I assumed that it is good practice. The thing is that this code is not dependant on other codes but the idf_ neighborhoods keeps failing (our fault) and I want this code to continue running regardless.

given the RAM problems, I suggest to run less things in parallel
if you want to ignore an error for a pipeline step, wrap it in a try catch

I'll move things around

given the instability of the pipelines, maybe it's best to write a success / failure file at the end of each step,
then add a final pipeline step which depends on all other steps and checks success/failure files and raises an exception if any step failed - to get a visual feedback in the pipelines dashboard if there were any failures

I'll figure something out and then fix this PR

This hopefully should work. I am leaving some things to run simultaneously since they should not take up too much RAM resources.
Also I moved the code that keeps on failing to the end. Apparently (according to other team members) none of the other sections is dependent upon it so this would make work better.
Thanks!

I just noticed that everything seems to run OK.
I say don't merge this PR for the time being.
Sorry for all the trouble


Footer restructure

It seems that our footer on the main website has a bit too much info, and it's not even organized properly imo.
Let's discuss here how we can make it more useful and less cluttered for visitors.

Just a note: The "Getting Started" link curremtly does not work, it should point to this URL: https://docs.btcpayserver.org/getting-started/


truncate method, tests, docs

Ref: #183

Basically a minimized version of Lodash truncate. Contains the core functionality with a bit less checking for edge cases and conversion of regex to strings. Not quite in the spirit of a reduce only library, but let me know.

Not quite in the spirit of a reduce only library, but let me know.

A small deviation from the 'spirit' of the project isn't a big deal. TBH, I'm surprised this hasn't happened more often.

@jkvyff FYI, I had to add some junk conditionals to force typescript to accept the type split d8f964c3123e97067cc5a1710f3aee39db8f1501


[enhancement]: Implement sound transcriptions

Initially on YouTube

Fixed by https://github.com/david-allison/manx-corpus-search/pull/126


Filter fields in embedded document

I think this is a feature request but is there a way to filter fields from embedded documents?  Think of a base document (in mongodb) that looks like this in JSON:
[{
  "_id":  12,
  "name": "test 1",
  "embdded_doc": {
     "_id": 1237652,
     "other_info": "this and that",
     "color": "red"
  }
}]

and you only want to output: _id, name, embedded_doc.color
We are trying to avoid having to serialize the entire embedded doc since it could a be large memory footprint, especially when we are returning a list of the parent objects.
Thanks

This is not currently supported. Why don't you simply create a serializer that only serializes some of the fields?

We were hoping not to have to create one for every API's different use cases.  We have a lot of listing api's that only need a subset of the model and their child documents fields so we feel the right approach is to have a general purpose interface that can filter on the request thus no need extra development and testing on each model.

I see. This library is intentionally pretty simple, it just filters the fields of the top-level serializer.
If you want to apply filters to nested serializers, that'll become a bit more complicated. If someone would provide a clean pull request with this feature and good test coverage, I wouldn't say no, but I don't have the capacity to add that feature right now.
Maybe https://github.com/rsinger86/drf-flex-fields already does what you are looking for?

I had looked at that one, but they seem to acknowledge the existence of the child documents and allow for expansion but I didn't notice a good way to restrict the output, and it would be more complex if we have multiple child document in a parent document.
Thanks though, I appreciate you taking the time to review the request.

Maybe you can create a feature request against drf-flex-fields. They seem to aim for more flexibilit than I do :)
Otherwise, writing custom code for the specific filtering needs might be quite simple to implement.


fix(typescript): change config

add inlineSources
#137

Codecov Report

Merging #138 (7e5417a) into master (e198ad0) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##            master      #138   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            4         4
  Lines           31        31
  Branches        13        13
=========================================
  Hits            31        31

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e198ad0...7e5417a. Read the comment docs.


Missing Jared Golden (ME-02) for 116th

The election was challenged and the lawsuit only dropped at the end of December, so maybe things are weird.

Still can't find a bioguide for him, but his info is up finally on the clerk site:
http://clerk.house.gov/member_info/mem_contact_info.aspx?statdis=ME02
Maine-2nd, Democrat
1223 Longworth HOB
Washington, DC 20515-1902
Phone:<PHONE_NUMBER>
https://golden.house.gov/


Results analysis

Hi, Thank you for your effors.
In the calculation using the files provided in this example, the results in the

none of the resulting values match in the files affinity_prediction.csv and complete_affinity_prediction.csv. What is the difference between the two files?

is the ranking done by LDDT, does this mean that rank1 is better than rank2 or rank3 ...?

affinity_prediction give you a weighted average. ranking is done by LDDT plus a bit of predicted affinity.
you can check the code for details.

Hello - should these affinity values be negative? When running DynamicBind with your test data as well as mine (with known ligand binders), I get all positive affinity values.


Uncaught TypeError: Cannot read property 'map' of undefined

how to solve this problem in html?
contents = [{"data":{"text":"내용
"},"type":"paragraph"},{"data":{"text":"헤더1","level":2},"type":"header"},{"data":{"style":"ordered","items":["리스트1","리스트2"]},"type":"list"}]
//contents is string
const edjsParser = edjsHTML();
let html = edjsParser.parse(contents);

The reason you have encountered this error, is because the parser is not able to initialise with data you provided.  The possible solutions are
Solution 1: Pass all of the editorjs data as it comes. Do not modify it.
Solution 2: Pass it as blocks variable inside an object.
blocks = [{"data":{"text":"내용
"},"type":"paragraph"},{"data":{"text":"헤더1","level":2},"type":"header"},{"data":{"style":"ordered","items":["리스트1","리스트2"]},"type":"list"}]
//contents is string
const edjsParser = edjsHTML();
let html = edjsParser.parse({blocks});

Solution 3: Pass it as blocks key of the object.
contents = [{"data":{"text":"내용
"},"type":"paragraph"},{"data":{"text":"헤더1","level":2},"type":"header"},{"data":{"style":"ordered","items":["리스트1","리스트2"]},"type":"list"}]
//contents is string
const edjsParser = edjsHTML();
let html = edjsParser.parse({blocks: contents});

Solution 4: Use the edjsParser.parseBlock() to parse one block at a time.
const block = {"data":{"text":"내용
"},"type":"paragraph"};
const html = edjsParser.parseBlock(block);

editorjs keeps all its data under the blocks property. So, you must pass it under the same name. You can use any of the provided approaches above. If you encounter any other issues, let me know.

Thanks to your answer I solved it.


(Lesson 1) Why use 0x3F000000 as base address?

It would be great if you could add this kind of explanation to the README.md, as the manual-page link that you referred us to was showing a different base address (0x7E000000), and it didn't really click for me what was going on for a while... Here was a great/helpful analysis/explanation of what's going on, in simple: https://www.raspberrypi.org/forums/viewtopic.php?t=142439 .

@riyadshauk in the  BCM2837 ARM Peripherals manual , section 1.2.3 ARM physical addresses it says  the following:
Physical addresses range from 0x3F000000 to 0x3FFFFFFF for peripherals. The
bus addresses for peripherals are set up to map onto the peripheral bus address range
starting at 0x7E000000. Thus a peripheral advertised here at bus address 0x7Ennnnnn is
available at physical address 0x3Fnnnnnn.

It looks correct to me. Maybe I should just emphasize the fact that we are using physical addresses rather than bus addresses in the tutorial. i'll try to add a little bit more explanations.

Yeah, sorry... I missed that detail in the manual. Yeah, exactly that suffices (that exact excerpt). It would be helpful if you spoon-fed us to look at that section, maybe? I'm a noob with electronics and am not accustomed to seeing such a big manual for just hardware. But maybe I'm expected to just be better at looking through the linked manual?
Maybe after this sentence in the Lesson 1 README,
"The meaning of each bit in each device register is described in the BCM2837 ARM Peripherals manual.",
you could add something like:
"Take a look at section 1.2.3 ARM physical addresses in BCM2837 ARM Peripherals manual and the surrounding documentation for more context on why we use 0x3F000000 as a base address (even though 0x7E000000 is used throughout the manual)."
pointing it out in such a way might just save confusion/time for others... Though, I understand that a lot of things are important to look at, and highlighting one specific part of a document might make other parts seem less relevant/important, so idk...
In conclusion, it seems to me (personally) that revising that paragraph to include a specific reference to section 1.2.3 would be more helpful than not.

Thanks @riyadshauk  for your feedback. I updated the lesson accordingly to your suggestion. https://github.com/s-matyukevich/raspberry-pi-os/commit/d4dd37b4fd736f87e57a9f1d3b66d3b460a4bb33


iOS — Open Drawer Animation Bug (v2.0.0) 🐛

When opening a drawer with toggleLeftWindow() or via pan gesture cancel (line 1059), the animation does not begin (or complete) until the drawer has finished opening (1.0/100%). See attached screencast) 📽 (using ANIMATION_SLIDE).
Note(s):
· Tested with left drawer only
· Animation methods which do not require the centerWindow to 'pull' the drawer window (ex. ANIMATION_FADE or ANIMATION_NONE) seem to work fine
· Seems to affect opening of the drawer only (closing works as expected)
· Maybe CATransform3DMakeTranslation?
Side Note:
· The thin black vertical line shown in the screencast can be fixed by removing this line or possibly by setting [self.view setBackgroundColor:[UIColor blackColor]]; to [self.view setBackgroundColor:[UIColor clearColor]];
— Most prominent when using ANIMATION_SLIDE & a light theme (background color) on leftDrawer and centerWindow
Environment:
· dk.napp.drawer: 2.0.0
· TiSDK: 7.1.0.v20180108165029 (app relies on: https://jira.appcelerator.org/browse/TIMOB-24404)
· Alloy: 1.10.7
· Hyperloop: 3.0.1
· iOS: 10.3 & 11.2
· Device(s): iPhone 6s (10.3.3) & iPhone X (11.2.2)
· Simulator: iPhone X (11.2)

Just upgraded from V2.10.0, which worked fine for me using ANIMATION_PARALLAX_FACTOR_7, to V3.0.0 and have the same issue with the black vertical line.

What's up, @MarkBoyer? — you can fix the vertical black line issue as suggested in the "Side Note" of the comment above (preferred method) or try setting Ti's default backgroundColor to something like the below (in alloy.js):
Ti.UI.setBackgroundColor('rgba(255,255,255,1)');

Note: the above line works well when using a light (white) background in your drawer (left/center/right) windows

Hopefully this helps until a better solution (or fix) is made to the sub module MMDrawerController 🙂

Thanks for the quick reply @designbymind.  I made the change to the module that you suggest and rebuilt it and while the block vertical stripe is gone, it's still not visually pleasing IMHO.  There seems to be a bigger issue with the slide.  Unfortunately, the native stuff is not my strong suit.  I just changed to using the "ANIMATION_FADE" animation mode and is looks much better to me.  Perhaps someone with more native experience can fix this.

@MarkBoyer — Right on. Something was definitely broken in v.2.0.0 in regards to any animation that 'pulls' the drawer. The only two animation modes that I've noted to work as expected are: ANIMATION_NONE  & ANIMATION_FADE as they don't require the drawer (left or right) to 'slide' (move along an x-axis). Moving forward: there were quite a few additions/source formatting in v.2.0.0, However, I cannot currently allocate the time to look into a fix for the animation issues. For now, I'm sticking with v.1.2.4 which is stable and I myself am currently using it in both development & production iOS apps 📱

Same problem with "ANIMATION_SLIDE_SCALE" (V 2.0.0). Opening the drawer results in a blank view while animating  and - after animation has been ended - the view is shown.

Any update on this? Can anyone offer a starting point on how to fix this?


Bug with bindings

Found a bug with binding multiple things when writing the self-hosted compiler.
See commit for a snapshot of the issue.
The problem looks something like this:
fn Parser.parse_type(*Parser: self) -> [Result<Opt<Token> HayError>] {
    // ...
    unary_match Result.unwrap as [op_tok]
        &op_tok Token.unary_operator as [result] {
            &result Result.is_err if {
                result Result.unwrap_err Result.Err::<Opt<Token> HayError> return
            } else {
                result Result.unwrap
            }
        }
        self Parser.parse_type as [result] {
            &result Result.is_err if {
                drop result Result.unwrap_err Result.Err::<Opt<Token> HayError> return
            } else {
                result Result.unwrap
            }
        }
        as [op maybe_type]

gets resolved when turned into:
fn Parser.parse_type(*Parser: self) -> [Result<Opt<Token> HayError>] {
    // ...
    unary_match Result.unwrap as [op_tok]
        &op_tok Token.unary_operator as [result] {
            &result Result.is_err if {
                result Result.unwrap_err Result.Err::<Opt<Token> HayError> return
            } else {
                result Result.unwrap
            }
        }
        as [op] // This is the change
        self Parser.parse_type as [result] {
            &result Result.is_err if {
                result Result.unwrap_err Result.Err::<Opt<Token> HayError> return
            } else {
                result Result.unwrap
            }
        }
        as [maybe_type]

So much of the compiler has changed since this snapshot. It's certainly possible for this to still exist, but it will be near impossible to recreate. i'm going to close this for now


SOLR-14401: Add back in missing metric and fix graphana dashboard

https://issues.apache.org/jira/browse/SOLR-14401
This adds back a metric that was removed in the previous PR (solr_metrics_core_query_local_count) but renames it to follow the common pattern used elsewhere (solr_metrics_core_query_requests_total).
I have also fixed the three uses of _local that persisted in the graphana dashboard.
This fixes the broken prometheus-exporter docker tests.

No worries, thanks for taking a look!


Errors downloading from youtube

Repetitive error using youtube-dl installed through MacPorts on mac OSX 10.9.5

youtube-dl --extract-audio --ignore-errors --audio-format mp3 --verbose https://www.youtube.com/watch?v=pbMgo0__ETg
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--extract-audio', u'--ignore-errors', u'--audio-format', u'mp3', u'--verbose', u'https://www.youtube.com/watch?v=pbMgo0__ETg']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.12.13
[debug] Python version 2.7.11 - Darwin-13.4.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 2.8.3, ffprobe 2.8.3
[debug] Proxy map: {}
[youtube] pbMgo0__ETg: Downloading webpage
[youtube] pbMgo0__ETg: Downloading video info webpage
[youtube] pbMgo0__ETg: Extracting video information
[youtube] {43} signature length 43.43, html5 player en_US-vflrSqbyh
[youtube] pbMgo0__ETg: Downloading player https://s.ytimg.com/yts/jsbin/player-en_US-vflrSqbyh/base.js
ERROR: Signature extraction failed: Traceback (most recent call last):
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 877, in _decrypt_signature
video_id, player_url, s
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 792, in _extract_signature_function
res = self._parse_sig_js(code)
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 855, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/opt/local/bin/youtube-dl/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by ExtractorError(u"Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 877, in _decrypt_signature
video_id, player_url, s
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 792, in _extract_signature_function
res = self._parse_sig_js(code)
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 855, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/opt/local/bin/youtube-dl/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "/opt/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 663, in extract_info
ie_result = ie.extract(url)
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 290, in extract
return self._real_extract(url)
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 1431, in _real_extract
encrypted_sig, video_id, player_url, age_gate)
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 887, in _decrypt_signature
'Signature extraction failed: ' + tb, cause=e)
ExtractorError: Signature extraction failed: Traceback (most recent call last):
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 877, in _decrypt_signature
video_id, player_url, s
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 792, in _extract_signature_function
res = self._parse_sig_js(code)
File "/opt/local/bin/youtube-dl/youtube_dl/extractor/youtube.py", line 855, in _parse_sig_js
initial_function = jsi.extract_function(funcname)
File "/opt/local/bin/youtube-dl/youtube_dl/jsinterp.py", line 241, in extract_function
raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
(caused by ExtractorError(u"Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

You are using an outdated version of youtube-dl. See our FAQ if you need help updating.