Table configuration settings should work in both metadata.yml and datasette.yml

The https://docs.datasette.io/en/latest/configuration.html page should gather together the various table settings, including:
https://docs.datasette.io/en/latest/facets.html#facets-in-metadata
databases:
  sf-trees:
    tables:
      Street_Tree_List:
        facets:
        - qLegalStatus
        facet_size: 10

https://docs.datasette.io/en/latest/full_text_search.html#configuring-full-text-search-for-a-table-or-view
databases:
  russian-ads:
    tables:
      display_ads:
        fts_table: ads_fts
        fts_pk: id
        searchmode: raw

And a bunch of stuff from metadata (has this moved to datasette.yaml yet?)
https://docs.datasette.io/en/latest/metadata.html#setting-a-default-sort-order
databases:
  mydatabase:
    tables:
      example_table:
        sort_desc: created
        size: 10
        sortable_columns:
        - height
        - weight
        label_column: title
        hidden: true

There's a full list here:
https://docs.datasette.io/en/latest/metadata.html#table-level-metadata

[ ] hidden
[ ] sort/sort_desc
[ ] size
[ ] sortable_columns
[ ] label_column
[ ] facets
[ ] fts_table
[ ] fts_pk
[ ] searchmode

it looks like most (all?) of those things are still part of table metadata, e.g.
https://github.com/simonw/datasette/blob/b466749e88b2ffbd925b6b3e777c8527ebc54e78/datasette/database.py#L420-L423
https://github.com/simonw/datasette/blob/b466749e88b2ffbd925b6b3e777c8527ebc54e78/datasette/database.py#L490-L497
etc.

Changing this issue to be about deciding what to do with these. I'll bump it out of the 1.0a8 milestone for the moment. @asg017 did we talk about this before?

Made a decision on this: as with plugin settings and allow blocks (#2249) these will work in both metadata AND configuration, but will only be documented in configuration.

Reminder to move this section of the docs to configuration.rst istead: https://docs.datasette.io/en/latest/metadata.html#table-level-metadata

To be sure I'm getting all of the table metadata stuff, I tried rg:
rg table_metadata -A 15 datasette

datasette/filters.py:        table_metadata = datasette.table_metadata(database, table)
datasette/filters.py-        db = datasette.get_database(database)
datasette/filters.py-        fts_table = request.args.get("_fts_table")
datasette/filters.py:        fts_table = fts_table or table_metadata.get("fts_table")
datasette/filters.py-        fts_table = fts_table or await db.fts_table(table)
datasette/filters.py:        fts_pk = request.args.get("_fts_pk", table_metadata.get("fts_pk", "rowid"))
datasette/filters.py-        search_args = {
datasette/filters.py-            key: request.args[key]
datasette/filters.py-            for key in request.args
datasette/filters.py-            if key.startswith("_search") and key != "_searchmode"
datasette/filters.py-        }
datasette/filters.py-        search = ""
datasette/filters.py:        search_mode_raw = table_metadata.get("searchmode") == "raw"
datasette/filters.py-        # Or set search mode from the querystring
datasette/filters.py-        qs_searchmode = request.args.get("_searchmode")
datasette/filters.py-        if qs_searchmode == "escaped":
datasette/filters.py-            search_mode_raw = False
datasette/filters.py-        if qs_searchmode == "raw":
datasette/filters.py-            search_mode_raw = True
datasette/filters.py-
datasette/filters.py-        extra_context["supports_search"] = bool(fts_table)
datasette/filters.py-
datasette/filters.py-        if fts_table and search_args:
datasette/filters.py-            if "_search" in search_args:
datasette/filters.py-                # Simple ?_search=xxx
datasette/filters.py-                search = search_args["_search"]
datasette/filters.py-                where_clauses.append(
datasette/filters.py-                    "{fts_pk} in (select rowid from {fts_table} where {fts_table} match {match_clause})".format(
--
datasette/inspect.py:        table_metadata = database_metadata.get("tables", {}).get(table, {})
datasette/inspect.py-
datasette/inspect.py-        try:
datasette/inspect.py-            count = conn.execute(
datasette/inspect.py-                f"select count(*) from {escape_sqlite(table)}"
datasette/inspect.py-            ).fetchone()[0]
datasette/inspect.py-        except sqlite3.OperationalError:
datasette/inspect.py-            # This can happen when running against a FTS virtual table
datasette/inspect.py-            # e.g. "select count(*) from some_fts;"
datasette/inspect.py-            count = 0
datasette/inspect.py-
datasette/inspect.py-        column_names = table_columns(conn, table)
datasette/inspect.py-
datasette/inspect.py-        tables[table] = {
datasette/inspect.py-            "name": table,
datasette/inspect.py-            "columns": column_names,
--
datasette/inspect.py:            "hidden": table_metadata.get("hidden") or False,
datasette/inspect.py-            "fts_table": detect_fts(conn, table),
datasette/inspect.py-        }
datasette/inspect.py-
datasette/inspect.py-    foreign_keys = get_all_foreign_keys(conn)
datasette/inspect.py-    for table, info in foreign_keys.items():
datasette/inspect.py-        tables[table]["foreign_keys"] = info
datasette/inspect.py-
datasette/inspect.py-    # Mark tables 'hidden' if they relate to FTS virtual tables
datasette/inspect.py-    hidden_tables = [
datasette/inspect.py-        r["name"]
datasette/inspect.py-        for r in conn.execute(
datasette/inspect.py-            """
datasette/inspect.py-                select name from sqlite_master
datasette/inspect.py-                where rootpage = 0
datasette/inspect.py-                and sql like '%VIRTUAL TABLE%USING FTS%'
--
datasette/facets.py:def load_facet_configs(request, table_metadata):
datasette/facets.py-    # Given a request and the metadata configuration for a table, return
datasette/facets.py-    # a dictionary of selected facets, their lists of configs and for each
datasette/facets.py-    # config whether it came from the request or the metadata.
datasette/facets.py-    #
datasette/facets.py-    #   return {type: [
datasette/facets.py-    #       {"source": "metadata", "config": config1},
datasette/facets.py-    #       {"source": "request", "config": config2}]}
datasette/facets.py-    facet_configs = {}
datasette/facets.py:    table_metadata = table_metadata or {}
datasette/facets.py:    metadata_facets = table_metadata.get("facets", [])
datasette/facets.py-    for metadata_config in metadata_facets:
datasette/facets.py-        if isinstance(metadata_config, str):
datasette/facets.py-            type = "column"
datasette/facets.py-            metadata_config = {"simple": metadata_config}
datasette/facets.py-        else:
datasette/facets.py-            assert (
datasette/facets.py-                len(metadata_config.values()) == 1
datasette/facets.py-            ), "Metadata config dicts should be {type: config}"
datasette/facets.py-            type, metadata_config = list(metadata_config.items())[0]
datasette/facets.py-            if isinstance(metadata_config, str):
datasette/facets.py-                metadata_config = {"simple": metadata_config}
datasette/facets.py-        facet_configs.setdefault(type, []).append(
datasette/facets.py-            {"source": "metadata", "config": metadata_config}
datasette/facets.py-        )
datasette/facets.py-    qs_pairs = urllib.parse.parse_qs(request.query_string, keep_blank_values=True)
--
datasette/facets.py:            table_metadata = tables_metadata.get(self.table) or {}
datasette/facets.py:            if table_metadata:
datasette/facets.py:                table_facet_size = table_metadata.get("facet_size")
datasette/facets.py-        custom_facet_size = self.request.args.get("_facet_size")
datasette/facets.py-        if custom_facet_size:
datasette/facets.py-            if custom_facet_size == "max":
datasette/facets.py-                facet_size = max_returned_rows
datasette/facets.py-            elif custom_facet_size.isdigit():
datasette/facets.py-                facet_size = int(custom_facet_size)
datasette/facets.py-            else:
datasette/facets.py-                # Invalid value, ignore it
datasette/facets.py-                custom_facet_size = None
datasette/facets.py-        if table_facet_size and not custom_facet_size:
datasette/facets.py-            if table_facet_size == "max":
datasette/facets.py-                facet_size = max_returned_rows
datasette/facets.py-            else:
datasette/facets.py-                facet_size = table_facet_size
datasette/facets.py-        return min(facet_size, max_returned_rows)
--
datasette/app.py:            table_metadata = ((databases.get(database) or {}).get("tables") or {}).get(
datasette/app.py-                table
datasette/app.py-            ) or {}
datasette/app.py:            search_list.insert(0, table_metadata)
datasette/app.py-
datasette/app.py-        search_list.append(metadata)
datasette/app.py-        if not fallback:
datasette/app.py-            # No fallback allowed, so just use the first one in the list
datasette/app.py-            search_list = search_list[:1]
datasette/app.py-        if key is not None:
datasette/app.py-            for item in search_list:
datasette/app.py-                if key in item:
datasette/app.py-                    return item[key]
datasette/app.py-            return None
datasette/app.py-        else:
datasette/app.py-            # Return the merged list
datasette/app.py-            m = {}
datasette/app.py-            for item in search_list:
datasette/app.py-                m.update(item)
--
datasette/app.py:    def table_metadata(self, database, table):
datasette/app.py-        """Fetch table-specific metadata."""
datasette/app.py-        return (
datasette/app.py-            (self.metadata("databases") or {})
datasette/app.py-            .get(database, {})
datasette/app.py-            .get("tables", {})
datasette/app.py-            .get(table, {})
datasette/app.py-        )
datasette/app.py-
datasette/app.py-    def _register_renderers(self):
datasette/app.py-        """Register output renderers which output data in custom formats."""
datasette/app.py-        # Built-in renderers
datasette/app.py-        self.renderers["json"] = (json_renderer, lambda: True)
datasette/app.py-
datasette/app.py-        # Hooks
datasette/app.py-        hook_renderers = []
--
datasette/database.py:        explicit_label_column = self.ds.table_metadata(self.name, table).get(
datasette/database.py-            "label_column"
datasette/database.py-        )
datasette/database.py-        if explicit_label_column:
datasette/database.py-            return explicit_label_column
datasette/database.py-        column_names = await self.execute_fn(lambda conn: table_columns(conn, table))
datasette/database.py-        # Is there a name or title column?
datasette/database.py-        name_or_title = [c for c in column_names if c.lower() in ("name", "title")]
datasette/database.py-        if name_or_title:
datasette/database.py-            return name_or_title[0]
datasette/database.py-        # If a table has two columns, one of which is ID, then label_column is the other one
datasette/database.py-        if (
datasette/database.py-            column_names
datasette/database.py-            and len(column_names) == 2
datasette/database.py-            and ("id" in column_names or "pk" in column_names)
datasette/database.py-        ):
--
datasette/views/row.py:            "units": self.ds.table_metadata(database, table).get("units", {}),
datasette/views/row.py-        }
datasette/views/row.py-
datasette/views/row.py-        if "foreign_key_tables" in (request.args.get("_extras") or "").split(","):
datasette/views/row.py-            data["foreign_key_tables"] = await self.foreign_key_tables(
datasette/views/row.py-                database, table, pk_values
datasette/views/row.py-            )
datasette/views/row.py-
datasette/views/row.py-        return (
datasette/views/row.py-            data,
datasette/views/row.py-            template_data,
datasette/views/row.py-            (
datasette/views/row.py-                f"row-{to_css_class(database)}-{to_css_class(table)}.html",
datasette/views/row.py-                "row.html",
datasette/views/row.py-            ),
datasette/views/row.py-        )
--
datasette/views/table.py:    table_metadata = datasette.table_metadata(database_name, table_name)
datasette/views/table.py:    column_descriptions = table_metadata.get("columns") or {}
datasette/views/table.py-    column_details = {
datasette/views/table.py-        col.name: col for col in await db.table_column_details(table_name)
datasette/views/table.py-    }
datasette/views/table.py-    pks = await db.primary_keys(table_name)
datasette/views/table.py-    pks_for_display = pks
datasette/views/table.py-    if not pks_for_display:
datasette/views/table.py-        pks_for_display = ["rowid"]
datasette/views/table.py-
datasette/views/table.py-    columns = []
datasette/views/table.py-    for r in description:
datasette/views/table.py-        if r[0] == "rowid" and "rowid" not in column_details:
datasette/views/table.py-            type_ = "integer"
datasette/views/table.py-            notnull = 0
datasette/views/table.py-        else:
datasette/views/table.py-            type_ = column_details[r[0]].type
--
datasette/views/table.py:            elif column in table_metadata.get("units", {}) and value != "":
datasette/views/table.py-                # Interpret units using pint
datasette/views/table.py:                value = value * ureg(table_metadata["units"][column])
datasette/views/table.py-                # Pint uses floating point which sometimes introduces errors in the compact
datasette/views/table.py-                # representation, which we have to round off to avoid ugliness. In the vast
datasette/views/table.py-                # majority of cases this rounding will be inconsequential. I hope.
datasette/views/table.py-                value = round(value.to_compact(), 6)
datasette/views/table.py-                display_value = markupsafe.Markup(f"{value:~P}".replace(" ", " "))
datasette/views/table.py-            else:
datasette/views/table.py-                display_value = str(value)
datasette/views/table.py-                if truncate_cells and len(display_value) > truncate_cells:
datasette/views/table.py-                    display_value = display_value[:truncate_cells] + "\u2026"
datasette/views/table.py-
datasette/views/table.py-            cells.append(
datasette/views/table.py-                {
datasette/views/table.py-                    "column": column,
datasette/views/table.py-                    "value": display_value,
datasette/views/table.py-                    "raw": value,
--
datasette/views/table.py:    table_metadata = datasette.table_metadata(database_name, table_name)
datasette/views/table.py:    if "sortable_columns" in table_metadata:
datasette/views/table.py:        sortable_columns = set(table_metadata["sortable_columns"])
datasette/views/table.py-    else:
datasette/views/table.py-        sortable_columns = set(await db.table_columns(table_name))
datasette/views/table.py-    if use_rowid:
datasette/views/table.py-        sortable_columns.add("rowid")
datasette/views/table.py-    return sortable_columns
datasette/views/table.py-
datasette/views/table.py-
datasette/views/table.py:async def _sort_order(table_metadata, sortable_columns, request, order_by):
datasette/views/table.py-    sort = request.args.get("_sort")
datasette/views/table.py-    sort_desc = request.args.get("_sort_desc")
datasette/views/table.py-
datasette/views/table.py-    if not sort and not sort_desc:
datasette/views/table.py:        sort = table_metadata.get("sort")
datasette/views/table.py:        sort_desc = table_metadata.get("sort_desc")
datasette/views/table.py-
datasette/views/table.py-    if sort and sort_desc:
datasette/views/table.py-        raise DatasetteError(
datasette/views/table.py-            "Cannot use _sort and _sort_desc at the same time", status=400
datasette/views/table.py-        )
datasette/views/table.py-
datasette/views/table.py-    if sort:
datasette/views/table.py-        if sort not in sortable_columns:
datasette/views/table.py-            raise DatasetteError(f"Cannot sort table by {sort}", status=400)
datasette/views/table.py-
datasette/views/table.py-        order_by = escape_sqlite(sort)
datasette/views/table.py-
datasette/views/table.py-    if sort_desc:
datasette/views/table.py-        if sort_desc not in sortable_columns:
datasette/views/table.py-            raise DatasetteError(f"Cannot sort table by {sort_desc}", status=400)
--
datasette/views/table.py:    table_metadata = datasette.table_metadata(database_name, table_name)
datasette/views/table.py:    units = table_metadata.get("units", {})
datasette/views/table.py-
datasette/views/table.py-    # Arguments that start with _ and don't contain a __ are
datasette/views/table.py-    # special - things like ?_search= - and should not be
datasette/views/table.py-    # treated as filters.
datasette/views/table.py-    filter_args = []
datasette/views/table.py-    for key in request.args:
datasette/views/table.py-        if not (key.startswith("_") and "__" not in key):
datasette/views/table.py-            for v in request.args.getlist(key):
datasette/views/table.py-                filter_args.append((key, v))
datasette/views/table.py-
datasette/views/table.py-    # Build where clauses from query string arguments
datasette/views/table.py-    filters = Filters(sorted(filter_args), units, ureg)
datasette/views/table.py-    where_clauses, params = filters.build_where_clauses(table_name)
datasette/views/table.py-
datasette/views/table.py-    # Execute filters_from_request plugin hooks - including the default
--
datasette/views/table.py:        table_metadata, sortable_columns, request, order_by
datasette/views/table.py-    )
datasette/views/table.py-
datasette/views/table.py-    from_sql = "from {table_name} {where}".format(
datasette/views/table.py-        table_name=escape_sqlite(table_name),
datasette/views/table.py-        where=(
datasette/views/table.py-            ("where {} ".format(" and ".join(where_clauses))) if where_clauses else ""
datasette/views/table.py-        ),
datasette/views/table.py-    )
datasette/views/table.py-    # Copy of params so we can mutate them later:
datasette/views/table.py-    from_sql_params = dict(**params)
datasette/views/table.py-
datasette/views/table.py-    count_sql = f"select count(*) {from_sql}"
datasette/views/table.py-
datasette/views/table.py-    # Handle pagination driven by ?_next=
datasette/views/table.py-    _next = _next or request.args.get("_next")
--
datasette/views/table.py:    # page_size = _size or request.args.get("_size") or table_metadata.get("size")
datasette/views/table.py:    page_size = request.args.get("_size") or table_metadata.get("size")
datasette/views/table.py-    if page_size:
datasette/views/table.py-        if page_size == "max":
datasette/views/table.py-            page_size = datasette.max_returned_rows
datasette/views/table.py-        try:
datasette/views/table.py-            page_size = int(page_size)
datasette/views/table.py-            if page_size < 0:
datasette/views/table.py-                raise ValueError
datasette/views/table.py-
datasette/views/table.py-        except ValueError:
datasette/views/table.py-            raise BadRequest("_size must be a positive integer")
datasette/views/table.py-
datasette/views/table.py-        if page_size > datasette.max_returned_rows:
datasette/views/table.py-            raise BadRequest(f"_size must be <= {datasette.max_returned_rows}")
datasette/views/table.py-
datasette/views/table.py-        extra_args["page_size"] = page_size
--
datasette/views/table.py:                    metadata=table_metadata,
datasette/views/table.py-                    row_count=extra_count,
datasette/views/table.py-                )
datasette/views/table.py-            )
datasette/views/table.py-        return facet_instances
datasette/views/table.py-
datasette/views/table.py-    async def extra_facet_results(facet_instances):
datasette/views/table.py-        "Results of facets calculated against this data"
datasette/views/table.py-        facet_results = {}
datasette/views/table.py-        facets_timed_out = []
datasette/views/table.py-
datasette/views/table.py-        if not nofacet:
datasette/views/table.py-            # Run them in parallel
datasette/views/table.py-            facet_awaitables = [facet.facet_results() for facet in facet_instances]
datasette/views/table.py-            facet_awaitable_results = await run_sequential(*facet_awaitables)
datasette/views/table.py-            for (

And piping that through llm:
rg table_metadata -A 15 datasette | llm -m gpt-4-turbo --system \
  'List all of the keys that are looked up in the table_metadata dictionary, e.g. sort and sort_desc - add a bit of a description for each one'

Got a good result from that:

The keys looked up in the table_metadata dictionary across different contexts in the Datasette source code and their brief descriptions are as follows:

fts_table: Specifies the Full-Text Search (FTS) virtual table associated with a regular table, if any. This is used to support full-text search on the table's contents.

fts_pk: Specifies the primary key column of the FTS table. This is often "rowid" by default, but can be customized.

searchmode: Defines the mode for full-text search, such as "raw", indicating how search queries should be interpreted.

hidden: A boolean value indicating whether the table should be hidden from the Datasette interface. This is commonly used for tables that are used internally or are not meant to be directly exposed to end users.

facets: Specifies configurations for facets, which are ways to categorize and filter data within a table based on column values.

facet_size: Defines the maximum number of facet values to return.

label_column: Specifies a column in a table to use as a label for identifying rows, particularly in links or dropdowns where referencing the table.

units: A dictionary mapping column names to units (e.g., meters, dollars) so that Datasette can display these values with their appropriate units.

columns: A nested dictionary providing additional metadata about specific columns in the table, such as descriptions or custom display configurations.

sortable_columns: A list of columns that are allowed to be sorted by users. This can be used to restrict sorting capabilities for performance reasons or because some columns are not meaningful to sort.

sort: Specifies a default column to sort by when viewing the table’s data in Datasette.

sort_desc: Similar to "sort", but indicates that the default sorting should be in descending order.

size: Specifies the default page size or the number of rows to display when viewing the table's data.

Each of these keys allows for detailed customization of how tables are presented and interacted with in Datasette, ranging from search capabilities and faceting to display preferences like sorting, hiding tables, and managing pagination.

And piping that through llm:
rg table_metadata -A 15 datasette | llm -m gpt-4-turbo --system \
  'List all of the keys that are looked up in the table_metadata dictionary, e.g. sort and sort_desc - add a bit of a description for each one'

Got a good result from that:

The keys looked up in the table_metadata dictionary across different contexts in the Datasette source code and their brief descriptions are as follows:

fts_table: Specifies the Full-Text Search (FTS) virtual table associated with a regular table, if any. This is used to support full-text search on the table's contents.

fts_pk: Specifies the primary key column of the FTS table. This is often "rowid" by default, but can be customized.

searchmode: Defines the mode for full-text search, such as "raw", indicating how search queries should be interpreted.

hidden: A boolean value indicating whether the table should be hidden from the Datasette interface. This is commonly used for tables that are used internally or are not meant to be directly exposed to end users.

facets: Specifies configurations for facets, which are ways to categorize and filter data within a table based on column values.

facet_size: Defines the maximum number of facet values to return.

label_column: Specifies a column in a table to use as a label for identifying rows, particularly in links or dropdowns where referencing the table.

units: A dictionary mapping column names to units (e.g., meters, dollars) so that Datasette can display these values with their appropriate units.

columns: A nested dictionary providing additional metadata about specific columns in the table, such as descriptions or custom display configurations.

sortable_columns: A list of columns that are allowed to be sorted by users. This can be used to restrict sorting capabilities for performance reasons or because some columns are not meaningful to sort.

sort: Specifies a default column to sort by when viewing the table’s data in Datasette.

sort_desc: Similar to "sort", but indicates that the default sorting should be in descending order.

size: Specifies the default page size or the number of rows to display when viewing the table's data.

Each of these keys allows for detailed customization of how tables are presented and interacted with in Datasette, ranging from search capabilities and faceting to display preferences like sorting, hiding tables, and managing pagination.

Next step: create a new Datasette method called table_config() that reads from both locations, then refactor everywhere to call that instead.
I could even see if that method could be an await datasette.table_config(database, table) awaitable - if it COULD then this is something which I could turn into a plugin hook later on, with the ability to look up configuration in databases or via remote HTTP calls.
Since this is a new method now might be a good time to do this - I have avoided making the existing datasette.metadata() method awaitable because there are already a bunch of plugins that use it.

It's going to be a bit weird to have await datasette.table_config() when it's just datasette.metadata() for other stuff. But I still think it's worth exploring, just to get an idea of if it would be feasible or not.

Next step to resolve the facets part of this is for this code here to reference table_config rather than table_metadata:
https://github.com/simonw/datasette/blob/5d21057cf1e4171bc2741d3d8d9da83aee1165b5/datasette/views/table.py#L1264-L1282
table_metadata came from here:
https://github.com/simonw/datasette/blob/5d21057cf1e4171bc2741d3d8d9da83aee1165b5/datasette/views/table.py#L965

Since datasette.table_metadata() was never documented I'm going to remove it, replaced with a new await datasette.table_config().

Turns out datasette-graphql DOES use table_metadata:
  File "/Users/simon/Dropbox/Development/datasette-graphql/datasette_graphql/utils.py", line 681, in introspect_tables
    datasette_table_metadata = datasette.table_metadata(
AttributeError: 'Datasette' object has no attribute 'table_metadata'

I'm going to finish this in a PR so I can iterate on the last remaining tests.


Remove arrow function

I've been having an issue when building with webpack my application. The issue is with uglifyjs trying to minify my vendor files. I've been trying to found a solution on my configurations files but nothing worked. So I'm just updating to and older version of the same code.

Thanks! This is shipped in 3.7.0


feat(azurearcdata): Welcome 2024-05-01-preview

Note: This API is already published into RPSaaSMaster

ARM (Control Plane) API Specification Update Pull Request

[!TIP]
Overwhelmed by all this guidance? See the Getting help section at the bottom of this PR description.

PR review workflow diagram
Please understand this diagram before proceeding. It explains how to get your PR approved & merged.

Purpose of this PR
What's the purpose of this PR? Check the specific option that applies. This is mandatory!

[ ] New resource provider.
[x] New API version for an existing resource provider. (If API spec is not defined in TypeSpec, the PR should have been created in adherence to OpenAPI specs PR creation guidance).
[ ] Update existing version for a new feature. (This is applicable only when you are revising a private preview API version.)
[ ] Update existing version to fix OpenAPI spec quality issues in S360.
[ ] Convert existing OpenAPI spec to TypeSpec spec (do not combine this with implementing changes for a new API version).
[ ] Other, please clarify:

edit this with your clarification

Due diligence checklist
To merge this PR, you must go through the following checklist and confirm you understood
and followed the instructions by checking all the boxes:

[x] I confirm this PR is modifying Azure Resource Manager (ARM) related specifications, and not data plane related specifications.
[x] I have reviewed following Resource Provider guidelines, including
ARM resource provider contract and
REST guidelines (estimated time: 4 hours).
I understand this is required before I can proceed to the diagram Step 2, "ARM API changes review", for this PR.

Additional information

 Viewing API changes
For convenient view of the API changes made by this PR, refer to the URLs provided in the table
in the Generated ApiView comment added to this PR. You can use ApiView to show API versions diff.

Suppressing failures
If one or multiple validation error/warning suppression(s) is detected in your PR, please follow the
suppressions guide to get approval.

Getting help

First, please carefully read through this PR description, from top to bottom. Please fill out the Purpose of this PR and Due diligence checklist.
If you don't have permissions to remove or add labels to the PR, request write access per aka.ms/azsdk/access#request-access-to-rest-api-or-sdk-repositories
To understand what you must do next to merge this PR, see the Next Steps to Merge comment. It will appear within few minutes of submitting this PR and will continue to be up-to-date with current PR state.
For guidance on fixing this PR CI check failures, see the hyperlinks provided in given failure
and https://aka.ms/ci-fix.
For help with ARM review (PR workflow diagram Step 2), see https://aka.ms/azsdk/pr-arm-review.
If the PR CI checks appear to be stuck in queued state, please add a comment with contents /azp run.
This should result in a new comment denoting a PR validation pipeline has started and the checks should be updated after few minutes.
If the help provided by the previous points is not enough, post to https://aka.ms/azsdk/support/specreview-channel and link to this PR.

/azp run

The first commit needs to be an exact copy of the previous API version. All new changes should only be added in the subsequent commits.
This allows the reviewer to get a clear understanding of the actual changes being introduced. With the way the PR is raised now, it is not possible for the reviewer to tell what the changes are. Please either abandon the PR and raise another one with the recommendation or create a new set of commits on this PR following the recommendation. If you are doing the pattern please indicate which commit is the exact copy of the previous version.

Thanks
We just want to take this existing preview version in PRSaaSMaster (same version 1:1 that has already been reviewed and in production) and bring it over to public preview without changes.
I am not sure how to do this given the previous version comes from a broken base over a year ago?

/mergeRequested

/azp run


Assign MATCH query to variable

Is it possible to assign a query to a variable ?
For instance
MATCH p=(n:Node)-[r:REL]->(x:Node)
RETURN p
I would like to be able to assign to and access 'p' in the following example
db.match([
node('u', 'User', {id: '333333'}),
relation('out', ['MEMBER_OF_GROUP','HAS_REPORT_ROLE'],'*'),
node('r', 'Role', {allowed_delete: true})
]).return('count(u) > 0 as authorised')
.run()
.then(function (results) {
l.info(Is Authorised:  + JSON.stringify(results[0]));
});

Unfortunately it's not possible to do that right now unless you fallback to writing the first match clause with .raw():
db.raw`p=(U:User {id: ${id}})-[:MEMBER_OF_GROUP|HAS_REPORT_ROLE *]->(r:Role {allowed_delete: ${true}})`
  .return(....)
  ...

It is currently on the roadmap but I don't have a ETA

Thanks for the quick feedback. Does that also mean apoc functions like shortest path cannot be used outside of .raw ?

Yeah there is no special syntax for those.

I've created a PR for this which simply adds another Pattern class, PathNamePattern, to allow this.

By the way neo4j allows naming paths not only in MATCH, but also in CREATE and MERGE. Though the cypher reference  mentions them only for MATCH.


Add tophermoa

Thanks for your contribution 😊.  Don't forget to ✨ the Repository.

This is my first open source contribution. Thanks for you help!

Most welcome brother to the Open source community, I hope you will contribute to the most renounced project of open source in the future.


People remain immune to knockdown if they get unbuckled while stunned

Description

People can't be knocked down if they get stunned while buckled. If they get unbuckled while in this state, they stay remain in this state.
Screenshots

https://user-images.githubusercontent.com/60792108/164998018-9e57db18-175b-47aa-b324-62954aafc49a.mp4
Additional context

this got fixed at some point


[Bug] insomnia-plugin-default-headers - Cannot copy as curl with headers

[Bug] – When I try to "Copy as curl" a query, I get my curl incomplete without the headers that I used on my environment vars

Insomnia Version: ___ Version 5.13.0 (5.13.0)
Operating System: ___ OS X High Sierra 10.13.1

Details
{
"DEFAULT_HEADERS": {
		"Authorization": "Bearer eyJhbGciOiJIUzUxMiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJVc2VyOjEiLCJleHAiOjE1MjA4MDQyMjYsImlhdCI6MTUxODIxMjIyNiwiaXNzIjoiQmFyayIsImp0aSI6IjVhYjgyMDJhLTRkYWItNGNiYS05ZjAwLThmNGMxMGRiYTllNSIsInBlbSI6e30sInN1YiI6IlVzZXI6MSIsInR5cCI6eyJzb21lIjoiY2xhaW0ifX0.eAi8i5fnWmI2_59yBC3Pch9OoCWBRbC3YLv3WpqbUZpqNlbj-Q-y92PlGGj_lsP2mtjFClbvHiSLKq3_9yhmRg",
		"Connection": "close",
		"Content-Type": "application/json"
	}
}

Curl returned:
curl --request POST \
  --url http://localhost/api \
  --data '{"query":"subscription AttentionRouteCreated($attention_id: Int!) {\n  attention_route_created(attention_id: $attention_id) {\n    id,\n    attention { id },\n    lat,\n    lng,\n    inserted_at,\n    updated_at\n  }\n}","variables":{"attention_id":17},"operationName":"AttentionRouteCreated"}'

Thanks for reporting this @Yamilquery! Should be an easy fix

Which version of the app is this fixed in? I'm seeing the same issue with <IP_ADDRESS>5

Will be released soon


Add Option for Dropping Leading H1 Heading

This functionality is designed to eliminate duplicate page titles in Confluence (once from the page title itself, and a second from the rendered H1), but preserve them for display in GitHub or other non-contextual environments.

Adds a --drop-h1 flag that removes leading ATX H1 headings, excluding them from the Confluence wiki markup output.
Adds .idea/ (GoLand) IDE preferences to .gitignore

Merged, thanks for the contribution!


Added CSS type to props

Only merge this if it helps, totally fine if the typings come from the polishing process.

The problem right now is I don't know what or how many props are going to change. It'll occur on a per-component basis. I recommend keeping this as a draft for reference, but 110% needs to happen during or after the refinement passes

Closing this as the new PR has replaced this.


[r] Support Bioconductor AKA SingleCellExperiment ingest

Sibling to #1276.
@mojaveazure this is lower-pri than other items this sprint; OK to leave until the next sprint if needed.

Duplicate of #1087


Reuse Huggingface model cache directory as standard

As the idea is it seems (or should be?) minimizing the compute resources needed for these models, one useful optimization is to avoid unnecessarily re-downloading models from Huggingface (when for instance they have already been downloaded with the huggingface_hub API to the standard cache directory such as ~/.cache/huggingface/hub/)
diff --git a/hqq/models/base.py b/hqq/models/base.py
index e305391..2ba7fa1 100755
--- a/hqq/models/base.py
+++ b/hqq/models/base.py
@@ -282,7 +283,10 @@ class BaseHQQModel:
         save_dir = pjoin(cache_dir, save_dir_or_hub)

         if not os.path.exists(save_dir):
-            save_dir = snapshot_download(repo_id=save_dir_or_hub, cache_dir=cache_dir)
+            if cache_dir == "":
+                save_dir = snapshot_download(repo_id=save_dir_or_hub)
+            else:
+                save_dir = snapshot_download(repo_id=save_dir_or_hub, cache_dir=cache_dir)
             save_dir = pjoin(save_dir)

         # Check

may need some adjustment according to how you were intending it should work; for instance, I think my code would not create a (new) copy in a directory more local to the cwd (though it should? handle older files downloaded to such locations), you may think you need to add that for the sake of matching the original functionality

Hi @MarkBenjamin , does it re-download the model if you pass the cache_dir as a parameter ?
model  = HQQModelForCausalLM.from_pretrained(model_id, cache_dir=cache_path, torch_dtype=compute_dtype, attn_implementation=attn_imp)

It seems to try to
HQQModelForCausalLM.from_quantized('mobiuslabsgmbh/Llama-2-7b-chat-hf_1bitgs8_hqq', cache_dir='/home/mark/.cache/huggingface/hub/') for instance

it looks as though cache_dir isn't being passed from HQQWrapper.from_quantized() to BaseHQQModel.try_snapshot_download()
diff --git a/hqq/engine/base.py b/hqq/engine/base.py
index 27b3050..e2f4acf 100755
--- a/hqq/engine/base.py
+++ b/hqq/engine/base.py
@@ -74,12 +74,12 @@ class HQQWrapper:
         cls,
         save_dir_or_hub,
         compute_dtype: torch.dtype = float16,
-        device="cuda",
-        cache_dir: str = "",
+        device=torch.device('cuda' if torch.cuda.is_available() else 'cpu'),
+        cache_dir: str | None = "",
         adapter: str = None,
     ):
         # Both local and hub-support
-        save_dir = BaseHQQModel.try_snapshot_download(save_dir_or_hub)
+        save_dir = BaseHQQModel.try_snapshot_download(save_dir_or_hub, cache_dir=cache_dir)
         arch_key = cls._get_arch_key_from_save_dir(save_dir)
         cls._check_arch_support(arch_key)

You need to put here as well: https://github.com/mobiusml/hqq/blob/master/hqq/models/base.py#L303
That part in /hqq/engine/base.py is not supposed to download the whole repo, only the config.json to figure out which architecture the model is, but somehow forgot to change it.
I actually don't see this issue. Just use a fixed cache_dir, like cache_dir='' and it should work just fine.

I seem to have it working now, whether I specify the huggingface cache path or specifically send None; the diff (plus some type hints allowing specifically passing None) cleaned of the torch cpu code would be
diff --git a/hqq/engine/base.py b/hqq/engine/base.py
index 27b3050..a767fad 100755
--- a/hqq/engine/base.py
+++ b/hqq/engine/base.py
@@ -75,11 +75,11 @@ class HQQWrapper:
         save_dir_or_hub,
         compute_dtype: torch.dtype = float16,
         device="cuda",
-        cache_dir: str = "",
+        cache_dir: str | None = "",
         adapter: str = None,
     ):
         # Both local and hub-support
-        save_dir = BaseHQQModel.try_snapshot_download(save_dir_or_hub)
+        save_dir = BaseHQQModel.try_snapshot_download(save_dir_or_hub, cache_dir=cache_dir)
         arch_key = cls._get_arch_key_from_save_dir(save_dir)
         cls._check_arch_support(arch_key)

diff --git a/hqq/models/base.py b/hqq/models/base.py
index e305391..32e0a88 100755
--- a/hqq/models/base.py
+++ b/hqq/models/base.py
@@ -278,8 +278,11 @@ class BaseHQQModel:
         cls.save_weights(weights, save_dir)

     @classmethod
-    def try_snapshot_download(cls, save_dir_or_hub: str, cache_dir: str = ""):
-        save_dir = pjoin(cache_dir, save_dir_or_hub)
+    def try_snapshot_download(cls, save_dir_or_hub: str, cache_dir: str | None = ""):
+        if cache_dir is None:
+            save_dir = pjoin("", save_dir_or_hub)
+        else:
+            save_dir = pjoin(cache_dir, save_dir_or_hub)

         if not os.path.exists(save_dir):
             save_dir = snapshot_download(repo_id=save_dir_or_hub, cache_dir=cache_dir)

I guess you wouldn't see it unless you have already (or subsequently?) directly called
from huggingface_hub import snapshot_download
snapshot_download('mobiuslabsgmbh/Llama-2-7b-chat-hf_1bitgs8_hqq')

(unless your envvars have altered the HF_HUB_CACHE or (legacy) HUGGINGFACE_HUB_CACHE or (climbing the tree) HF_HOME or XDG_CACHE_HOME values?)

Sorry @MarkBenjamin , I missed this.
In general, if you specify the cache directory when you load via from_pretrained this problem doesn't happen. At least, no one has reported this issue before.
I can try using the logic you mentioned in try_snapshot_download or you can do a pull request if you want, happy to review it !

Will do 🙂
Best regards
Mark

Closing since this was merged, thanks!


Cannot see folders and files names in denaro file manager

I can browse into folders and select a file, but the text is not displayed

This was fixed in the latest beta


test.config sounds like general config

Xenon specific system property should have the name 'xenon' in its variable name.
For example: xenontest.config

Fixed in 2.0.0.


Port to RxJava 2.

Since RxJava 1 entered in a feature freeze mode on June 1, 2017; this PR ports the app to the new and shining RxJava 2 xD
Notes:

This PR depends on #75 .
I didn't use Flowable, Single and Completable yet to make the transition smoother.

Any new abot this PR? I havent seen any movement in this repo

Can one of the admins verify this patch?

Can this PR be merged? It hasn't moved and no reply till date

Done :smile: Conflicts resolved and typos fixed.
Thank you guys for this amazing project, i'm always happy to help.
I'm hoping to be able to contribute more when the Architecture Components reach a stable release.


[Protection] Add Act/Control DRM detection

https://www.gamersgate.com/product/postal-3/
Found in Postal 3 according to PCGAMINGWIKI

At least link to the wiki that has the info 😛 https://www.pcgamingwiki.com/wiki/Postal_III
It links to a site that is down, but is archived. Seems to be an actual, honest-to-glob form of DRM that has it's roots in StarForce tech (https://web.archive.org/web/20110303104113/http://actcontrol.net/).


ent/entc: rename time mixin fields

Summary:

created_at -> creation_time
updated_at -> update_time

Differential Revision: D18035896

Out of curiosity, any reason why created_at/updated_at should be avoided?

Out of curiosity, any reason why created_at/updated_at should be avoided?

It's personal preference and there's no reason why created_at/updated_at should be avoided.
Note that the reason for the rename being that I plan to also rename time fields predicates to Before / After instead of LT / GT and CreatedAtBefore / UpdatedAtAfter just sound incorrect. I addition I though of calling those fields Created / Updated but those names imply boolean types.


Icon for file type python is not very clear

The icon st-primer used for python file:
https://github.com/karelvuong/st-primer/blob/master/icons%2Ffile_type_python%402x.png
looks a little bit dim compare to the one that cobalt2 used:
https://github.com/wesbos/cobalt2/blob/master/icons%2Ffile_type_python%402x.png
I replaced it with cobalt2's, comparison as below:
Original Primer:

With python icon replaced with cobalt2's:

In my opinion the latter is better, how do you think :)

Love it @reorx! Thanks for this, I'll add it in soon.

This has been committed and is staged for the next release. Closing this issue!

Nice job :)


COMPELSONLaboratories.MOBILedit!SIMClone version <IP_ADDRESS>

[ ] Have you signed the Contributor License Agreement?
[ ] Have you checked that there aren't other open pull requests for the same manifest update/change?
[ ] Have you validated your manifest locally with winget validate --manifest <path>?
[ ] Have you tested your manifest locally with winget install --manifest <path>?
[ ] Does your manifest conform to the 1.4 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

Microsoft Reviewers: Open in CodeFlow

@wingetbot waivers Add Validation-Domain

This software warns that a SIM card reader is necessary for the package to function, but it appears to run despite the VM not having one.


add initial version of W3C tracestate

This is the initial implementation of W3C tracestate in Go.
This is my first time writing program in Go, couple notes/questions:

I want to discuss if want to put thread safety guard on Tracestate.Fork(), or leave it as a responsibility for the API consumer.
Currently the Tracestate object will be copied every time when a new span got created, I plan to implement COW (copy on write) to improve perf.
My next pull request will implement the ochttp integration, which will propagate tracestate in HTTP headers.

@rakyll @ramonza, would you give me some hints? I'm getting the following AppVeyor build error which claims gcc compiler missing:
runtime/cgo
exec: "gcc": executable file not found in %PATH%
FAIL	go.opencensus.io/exporter/jaeger [build failed]
FAIL	go.opencensus.io/exporter/prometheus [build failed]
FAIL	go.opencensus.io/exporter/stackdriver/propagation [build failed]
FAIL	go.opencensus.io/exporter/zipkin [build failed]
FAIL	go.opencensus.io/internal [build failed]
FAIL	go.opencensus.io/plugin/ocgrpc [build failed]
FAIL	go.opencensus.io/plugin/ochttp [build failed]
FAIL	go.opencensus.io/plugin/ochttp/propagation/b3 [build failed]
FAIL	go.opencensus.io/plugin/ochttp/propagation/tracecontext [build failed]
FAIL	go.opencensus.io/stats [build failed]
FAIL	go.opencensus.io/stats/view [build failed]
FAIL	go.opencensus.io/tag [build failed]
FAIL	go.opencensus.io/trace [build failed]
FAIL	go.opencensus.io/trace/propagation [build failed]
FAIL	go.opencensus.io/zpages [build failed]
Command exited with code 2

@ramonza, agree, that's what I've mentioned in the PR description "My next pull request will implement the ochttp integration, which will propagate tracestate in HTTP headers.", I will add the actual propagation and the test cases in the next PR.

I would prefer to have it in one PR because I don't think just this alone makes sense on its own -- it doesn't add any functionality and I'm worried that if users see it they will be confused as to why its there.

I am going to look into the AppVeyor failure, it is unrelated.

Close this as it is a dup with #890 which @rghetia is working on.


[translations] Option to skip & rollback if error occurs

It would be benefit full if you can chose that the release script should update the ip-package.json or only the version.
Also, like currently, if an error occurs on API level the script should rollback any modifications and/or allow to continue ad ignore the error

Heh, I really need to finish #118

In Addition i still like the Option to skip Translation… like this Example 😂

Some comment based on my experience
Skipping translations which are already present would be a good feature if requested. Maybe the skipping should be limited to specified languages as most of us will not be able to update or check russion, chinese, ...
Personally I would be satisfied if I could verify the generated (german) text and adapt the english original with very little effort. Currently I must abort the release run, revert the git changes and then change the english text. Maybe a "retranslate" loop would be useful. Or the option to cancel the releasescript and revert the changes already done.

see slightly related issue #125

review of translations is implemented

I don't think this issue is done? This is about rollback in case of an error.

I don't think this issue is done? This is about rollback in case of an error.

Good point, would be even more additional value if indeed rollback would also be possible


Blocking "multiverse.help" DOES NOT prevent people from crashing your server.

I cant even reply to my ticket as you blocked replys.
https://github.com/Multiverse/Multiverse-Core/issues/2706
Your advice to block multiverse.help to prevent the hack to crash the server does not work, last night when I was sleeping I had a host of people from a youtube channel come and crash my server over and over and over, after you recommended denying the permission multiverse.help This did NOT work and caused absolute chaos.

https://www.spigotmc.org/resources/multiverse-patches.96390/

Please read my replies, I dont need to keep repeating myself.


Filter trace listener

Implementation of filters within the NLog Trace Listener

fixes #799

Thanks for this PR.
I'm a bit confused. Why can we do this with the regular filters? (https://github.com/NLog/NLog/wiki/Filters)

I need to configure everything within the trace listeners. Not all logging
goes to nlog so just to simplify configuration and know what is happening.
On 18 Jul 2015 3:48 am, "Julian Verdurmen"<EMAIL_ADDRESS>wrote:

Thanks for this PR.
I'm a bit confused. Why can we do this with the regular filters? (
https://github.com/NLog/NLog/wiki/Filters)
—
Reply to this email directly or view it on GitHub
https://github.com/NLog/NLog/pull/798#issuecomment-122354908.

OK clear.

Is the filter check don't needed at TraceTransfer(...) ?

Wasn't sure how the wiki updates will work. Hopefully adding this submodule solves it

We normally just edit the wiki with the edit button on github (yes it's then visible already) ;)

Hi @thomkinson,
Do you have time to check te comments? Otherwise we can't include this in 4.1.

Been heaps busy, so will get onto it tonight. apologies

Note: We will close 4.1 next week.

Should we move this to 4.2?
We really like to release 4.1 soon.

bump to 4.2. This one need some more docs before merge.

Please create a new PR when you have to fix this.

This would be awesome - any update on this @thomkinson?


MVP-470 name of the offender

This pull request removes the two name fields from the name of offender screen and replaces them with just one field. For MVP it's been agreed that if users needs to add more than one name they will have to put them all in the same field.

@gavinwye @johnkane1, should it be offender**'**s name

apostrophe added

Thanks for the apostrophe!


draw boundingRect on top of still image not showing

boundingRect is not showing on a captured image or from the gallery for the object detection.
Please see the code below for your reference.
` String text = 'Objects found: ${objects.length}\n\n';
for (final object in objects) {
text +=
'Object:  trackingId: ${object.trackingId} - ${object.labels.map((e) => e.text)}\n\n';
}
_text = text;
// TODO: set _customPaint to draw boundingRect on top of image
Size originalSize = MediaQuery.of(_mContext).size;
Size imageSize = Size(360.0 * originalSize.aspectRatio, 360.0);
// // if image source from gallery
// // image display size is scaled to 360x360 with retaining aspect ratio
if (originalSize.aspectRatio > 1) {
imageSize = Size(360.0, 360.0 / originalSize.aspectRatio);
} else {
imageSize = Size(360.0 * originalSize.aspectRatio, 360.0);
}
  final painter = ObjectDetectorPainter(
      objects,
      InputImageRotation.rotation0deg,
      imageSize);
  _customPaint = CustomPaint(painter: painter);`

Drawing the boundingRect on top of your input image is outside the scope of these plugin. Refer to README for the features of this plugin: https://github.com/bharat-biradar/Google-Ml-Kit-plugin/tree/master/packages/google_mlkit_object_detection.


Add MSAA mode for Quest

Due to performance issues we want to do a single pass render on the Quest. The Quest has a wonderful feature where we can directly render into the textures supplied by the Oculus SDK and have MSAA applied as long as we set our frame buffer up correctly.
There was a previous hacked version that attempted to do the same but broke the Rift plugin and potentially also the OpenXR drivers.
This PR introduces two new MSAA modes specific to the Quest and only work with the GLES2 driver (the GLES3 drivers effects setup make this approach impossible as far as I can tell so they just behave as the normal MSAA 2x and 4x modes).
When these modes are chosen Godot does not build its MSAA buffers instead setting up the FBO for the external texture correctly.
If these new modes are used on anything but Quest hardware MSAA remains turned off.
As we're already rendering to the external texture it now simply skips all the extra logic in post and the Quest does its thing.

@akien-mga you'll need to make a call on the fact we're adding a new feature after the feature freeze, I think this one is worth it :)

Thanks!


[DOC] Benchmark user guide improvements

What do you want to do?

[ ] Request a change to existing documentation
[x] Add new documentation
[ ] Report a technical problem with the documentation
[ ] Other

Tell us about your request. Provide a summary of the request and all versions that are affected.
The User Guide doesn't tell a complete story quite yet. Lets add additional content and restructure the section so it can go through everything:

Concepts
Installing Benchmark
Configuring OpenSearch Benchmark
Understanding workloads

Anatomy of a workload
Choosing a workload

Description of available workloads and what do they.
The most popular workloads:

NYC Taxis: A basic, well-rounded search workload.
HTTP_logs: A well-rounded time-series workload. If your cluster uses HTTP log data, this might be a good fit for you.

Creating a custom workload.

Running a workload

Connecting your current OpenSearch cluster

Docker

Provisioning an OpenSearch cluster with OSB
Running the test

Enabling telemetry devices

Understanding Benchmark results

Understanding the summary report
Going deeper with data stores

Telemetry device results are published to the data store.

Running distributed loads

Tasks

[ ] Spin Anatomy of a Workload into its own page.
[ ]

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.

@Naarcha-AWS - What is left to do on this issue? Can you please summarize what is left to do? Thanks!

Hi @Naarcha-AWS, We received feedback on this page: https://opensearch.org/docs/latest/benchmark/user-guide/telemetry/:
20231123 | It would be nice to have steps on how to do it.
Since the title of the topic is "Enabling telemetry devices", I would expect a process. Are we going to add something to this topic? If not, maybe we can add this text to the topic on 'Summary reports"?

Link to the current User Guide: https://opensearch.org/docs/latest/benchmark/user-guide/index/
The only remaining items are the following:

Going deeper with data stores
Enabling telemetry devices

Closing this as complete for now.


LeaderEvent should be published before candidate.onGranted

See https://github.com/spring-cloud/spring-cloud-cluster/issues/21

Rebased and merged as 6153eef


add domens swot/lib/ru/gov

Add domens ATU DGTU

@Dattebayu This domain does not belong to a particular school. It seems to belong to an umbrella institution (such as Ministry of Education or similar government body). We cannot add umbrella institutions domain to this repository. Pease apply for free educational license with an official document that proves your student status. You can do it using "OFFICIAL DOCUMENT" tab at https://www.jetbrains.com/shop/eform/students. Thank you for your understanding!


[SPARK-40453][SPARK-41715][CONNECT] Take super class into account when throwing an exception

What changes were proposed in this pull request?
This PR proposes to take the super classes into account when throwing an exception from the server to Python side by adding more metadata of classes, causes and traceback in JVM.
In addition, this PR matches the exceptions being thrown to the regular PySpark exceptions defined:
https://github.com/apache/spark/blob/04550edd49ee587656d215e59d6a072772d7d5ec/python/pyspark/errors/exceptions/captured.py#L108-L147
Why are the changes needed?
Right now, many exceptions cannot be handled (e.g., NoSuchDatabaseException that inherits AnalysisException) in Python side.
Does this PR introduce any user-facing change?
No to end users.
Yes, it matches the exceptions to the regular PySpark exceptions.
How was this patch tested?
Unittests fixed.

cc @xinrong-meng @grundprinzip @ueshin @zhengruifeng PTAL

cc @xinrong-meng @grundprinzip @ueshin @zhengruifeng PTAL

cc @xinrong-meng @grundprinzip @ueshin @zhengruifeng PTAL

cc @xinrong-meng @grundprinzip @ueshin @zhengruifeng PTAL

cc @xinrong-meng @grundprinzip @ueshin @zhengruifeng PTAL

Merged to master and branch-3.4.


We’re in the files could I add a new item/block

You mean you want to add a new element? Well, it's actually possible! First you have to download WASM and Rust. Then you can download the zip file and write some Rust code according to the documentation.


Variable depth FCN via model subclassing -- layers are not added to model

Hi,
my goal is to define a dynamically generated FCN model, where I define the amount of filters as a list, and the model is generated with len(filters) Encoder and len(filters) Decoder blocks. I defined my blocks and model according to this Tensorflow Guide.
Now, to achieve my dynamic model class, I have written all class variables into the model.__dict__:
class FullyConvolutionalNetworkDynamic(Model, ABC):
    def __init__(self,
                 num_filter: List[int],
                 kernel_size: List[int],
                 strides: List[int],
                 name: str = "FullyConvolutionalNetwork",
                 activation: str = "relu",
                 data_format: str = "channels_last",
                 *args, **kwargs):
        super(FullyConvolutionalNetworkDynamic, self).__init__(name=name, *args, **kwargs)

        # define symmetric encoder and decoder blocks
        for i, (f, k, s) in enumerate(zip(num_filter, kernel_size, strides)):
            self.__dict__[f"encoder{i}"] = EncoderBlock(num_conv=f, kernel=k, stride=s,
                                                        activation=activation,
                                                        name=f"enc{i}", data_format=data_format)
            self.__dict__[f"decoder{i}"] = DecoderBlock(num_conv=f, kernel=k, stride=s,
                                                        activation=activation,
                                                        name=f"dec{i}", data_format=data_format)

        # output section
        self.conv1x1 = Conv1D(1, 1, strides=1, padding='same', name="1x1", data_format=data_format)
        self.flatten = Flatten()

Now that my layers are written to model.__dict__, I tried to add them to my call function from here:
    @tf.function
    def call(self, x, training=False):
        # encoder path
        for layer in sorted([key for key in self.__dict__ if "encoder" in key]):
            x = self.__dict__[layer](x)

        # decoder path
        for layer in sorted([key for key in self.__dict__ if "decoder" in key], reverse=True):
            x = self.__dict__[layer](x)

        # output section
        x = self.conv1x1(x)
        return self.flatten(x)

However, when I now create an instance of my model, the model summary states that only the layers defined in the output section are added to the model:
>>> model = FullyConvolutionalNetwork(num_filters=[4, 8, 16], kernel_size=[3, 3, 3], strides=[1, 1, 1])
>>> model.build(input_shape=[None, 256, 10])
>>> model.summary()

Model: "FullyConvolutionalNetwork"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
1x1 (Conv1D)                 multiple                  5
_________________________________________________________________
flatten (Flatten)            multiple                  0
=================================================================
Total params: 5
Trainable params: 5
Non-trainable params: 0
_________________________________________________________________

I already use the model but "hard-coded" to three layers depth on each path. When I try to recreate my hard-coded model with the dynamic approach above, the model.__dict__ indeed contains my EncoderBlock and DecoderBlock layers, they are just not added to the model. The "hard-coded" version works flawlessly! Any help to solve my problem is greatly appreciated! Thank you in advance!

Hi @MathiesW, thanks for reaching out. The issue backlog is not the best place for support requests. Can you repost this to one of the forums below? That way your question will have more visibility.

TensorFlow forum
Keras mailing list.

Thanks!


Garbled text in Chrome/Edge (Works fine in Firefox)

In the latest versions of Chrome and Edge text in all PDF's are garbled. I couldn't find any issue mentioning it so submitted a new issue
To "fix" the garbled text you have to disable 2d canvas-something in the browser which is enabled by default.
PDFjs is not compatible with the new default settings in Chrome and Edge
https://github.com/wojtekmaj/react-pdf/issues/1025
Two weeks ago I got a new computer (windows 11) and by default text is antialiased in the system (not only in the browser). It has probably something to do with that. On my old computer PDFjs render text correctly, but on my old computer (also windows 11) the text is pixelated (like it usually is on windows)
I don't know what kind of setting makes windows render text by antialiasing at system level? But thats probaby the "new-thing" on windows 11
PDFjs v3.1.81

Duplicate of #14641

This is not a duplicate...

@Snuffleupagus  this is not the same error

All of these issues are related to hardware-acceleration being enabled, in combination with certain GPUs, in Chromium-based browsers; hence the duplicate is correct since it's a browser bug and not a PDF.js-library one!


Fix: Check for git authentication when GITHUB_TOKEN exists

TL;DR
When flytectl demo start is ran with GITHUB_TOKEN, it throws 401 error. Setting GITHUB_TOKEN empty works.
https://github.com/flyteorg/flyte/issues/4879
Type

[x] Bug Fix
[ ] Feature
[ ] Plugin

Are all requirements met?

[x] Code completed
[ ] Smoke tested
[ ] Unit tests added
[ ] Code documentation added
[ ] Any pending items have an associated Issue

Complete description

Validating GITHUB client before creating gh

Tracking Issue
https://github.com/flyteorg/flyte/issues/
Follow-up issue

@pingsutw could we bypass the test for this?


Only 2 different ads displayed

No matter what I set to setLimitOfAds(), only two different ads are displayed.
ie: list with 30 items,
mAdapterWrapper.setNoOfDataBetweenAds(8);
mAdapterWrapper.setLimitOfAds(3);
I do have 3 NativeExpressAd in the list, but first and third are the same object.
If i dismiss the first ad by clicking the little blue cross in the top right corner, the third ad is also dismissed.
I tried with setLimitOfAds(6) with bigger list, and it's the same result, only two different ads are displayed A - B - A - B - A ....

@loki666 Sounds like correct behavior :) See https://github.com/clockbyte/admobadapter/wiki/Cookbook#in-case-you-want-to-use-a-multiple-unit-ids to publish different ads.

Either I wasn't clear enough, or I didn't understood how the AdapterWrapper is working...
I only have one ad unit id, and I'm asking for a maximum of 3 ads in my list.
What I'm observing, is that 1st and 3rd AdView show the same ads (ok It might happen), but when I dismiss the 1st AdView (by clicking the little blue cross) the 3rd ad is also dissmissed.
Which leads me to believe that the AdapterWrapper was reusing the same AdView instance for the 1st and 3rd ad.
So even, if I ask for 3 (or 4, or 10) ads, I actually get only 2 different AdView, reused in the list.
I though (maybe wrongly), that the Adapter would create 3 different AdView with 3 different AdRequest for the same Ad unit Id.

ok a bit more debugging.
It seems that there is only 2 different ads in my list (even if I request 3) because my list only require 2 ViewHolder for the Ads.
I would have expected of the Adapter, to indeed create only 2 ViewHolder, but still request 3 different ads, recycle the ViewHolder but still bind the 3 ads.
Is that the intended behavior of the Adapter?

@loki666 You are quite right, Adapter would request N (almost) times given max ads count = N. But it's not a recycling issue or something. The content of ad block is filled by Admob. So if you want to publish N different ads and you want an expected behavior then please create N unit ids and pass its to admobadapter like it is shown  in wiki :)
Also...Do you use Express ads, right?

Yes I do use Express Ads,
following the code I don't think it's working like that...
the onBindViewHolder() does nothing if it's an AdExpress type. (where it should attach the NativeExpressAdView corresponding to the position in the ViewHolder's View)
only onCreateViewHolder() creates a Wrapper which hold the NativeExpressAdView.
In my case the RecyclerView needs only 2 ViewHolder for the Ads, but It depends on the setNoOfDataBetweenAds() and the screen size.
If I would have set a higher value (or on a smaller screen), the RecyclerView would only need 1 ViewHolder for the ad type,
resulting in only one NativeExpressAdView instance recycled in my list, no matter which value I set to setLimitOfAds().
setLimitOfAds affect the number of ads item inserted in the RecyclerView, not the number of NativeExpressAdView instances used in the list.
I also tried to use the constructor which use the ExpressAdPreset, but the result is exactly the same.
Only two ViewHolder needed, so only two NativeExpressAdView created recycled trough the list.
There should be as much NativeExpressAdView instances as setLimitOfAds() (less only if there is not enough items in the wrapped Adapter, but let's assume the list big enough), each NativeExpressAdView send its own AdRequest (via the Fetcher). The onBindViewHolder() should attach the correct NativeExpressAdView instance in the recycled ViewHolders.

@loki666 The onBindViewHolder does nothing because admobadapter currently doesn't support ViewHolder pattern for express ads, but only for source (your data) items. So it works in the following way:
At 1st adapter wrapper would fetch and cache 2 ad blocks for future, see AdmobExpressAdapterWrapper#init() the last string prefetchAds(AdmobFetcherExpress.PREFETCHED_ADS_SIZE);
When you scrolling down your RecyclerView adapter wrapper calculates each next displaying item is of an Ad type. Then it gets the ad from AdmobFetcherExpress#getAdForIndex(int adPos). If it was already stored for this position then Fetcher would return it back (express ads should refresh its' content themselves so we don't care about refetching it manually). Otherwise Fetcher would fetch one and cache it for this position. See AdmobExpressAdapterWrapper#onCreateViewHolder(...)
NativeExpressAdViewHolder item = adFetcher.getAdForIndex(mCntAdCreated++);
if (item == null)
    item = prefetchAds(1);

You'd pay attention to Admob wouldn't provide you N different ads for the single unit id. Also I don't know why Admob returns cycling ad views for the single unit id in the list but I've noticed it too. Please try to make a simple test app with an Activity which contains 2 ad views using the same unit id (without recyclerview, caching, viewholders and other stuff like that). What will be a result then?

could we continue this on gitter.im ? it will be more convenient

@loki666 sure

https://github.com/googleads/googleads-mobile-android-examples/blob/master/advanced/NativeExpressRecyclerViewExample/app/src/main/java/com/google/android/gms/example/nativeexpressrecyclerviewexample/RecyclerViewAdapter.java the correct implementation of a binding (ViewHolder pattern for express native ads).

@loki666 Hi, I've reimplemented the ads loading and presentation in admobadapter due to your request. Now admobadapter should handle the recycling of both the ListView and RecyclerView. You could update from the Experimental branch to get it. Soon it will be available in the Release branch with tag 1.4.0 and in the maven central too. Thanks a lot for your observations and investigations!
So closing it for now, but feel free to write if you'll find any issue.


MessagePack processor support

Hi! I am really excited about Benthos, thank you for this cool tool!
For my company's usecase it would be nice to have support for MessagePack format. This should be similar to Avro (even simpler, I think, because MsgPack has no schema).
I'll try to contribute it in my spare time.

Hey @isae :smile: this seems like a good addition, it'd probably make sense as a processor, looks like https://github.com/vmihailenco/msgpack is a good candidate.

Implemented as a processor: https://github.com/Jeffail/benthos/commit/c517a65345fdd62a3af3e8cda3087ea9616db1ba, and some complementary bloblang methods: https://github.com/Jeffail/benthos/commit/2e4a4565cba344ef9343b7850f5258ad9867f24c


Oneline option without utf symbols

Replace: sun - 'o', rain //, ", show - **, cloud - mm, fog - ==

It is a good idea, accepted.

Implemented in #583.
Please use %x for that


Form

What is your username and discriminator?
@xSlendiX [RaduKern]#5822
Why did you join Discord Linux?
I use Discord on 3 linux devices: My Android phone, My 2 laptops
Where did you hear about Discord Linux?
The Discord Developers server
What is the passphrase that was assigned to you by the bot?
I didn't get one for some reason.

From Ethan Weegee#9302 :
The passphrase you have entered is not correct.  Please check the message that mentions you in the welcome channel to find your passphrase.


UsedOTNames

UsedOTNames currently stores uncapitalized OT names.
The capitalization should be moved to before the name is stored.
This is not necessarily a bug. It is a matter of semantics and clarity.

Please take a look at PR #146 and let me know if everything looks okay.

@SrChip15 Wow, thank you so much for the pull request! It makes me so happy whenever we get one!
Unfortunately, I think this issue is a little more involved than expected. The changes you made were not sufficient as some tests (npm test) failed, but I took the liberty of adapting your commit to make them work again (I am actually quite surprised I had pushing rights...)!
Thank you so much again!

@Alan-Cha Your reply is incredibly quick. Thanks!
Last night, I did notice the failing tests from your CI tool, 6 tests from 3 suites were failing. I quickly looked into this and decided to work on it today, since it was getting late for me. Maybe the next time, I'll be more careful. Thanks for fixing this though.

No problem! The tests were failing even though your work looked good so I thought perhaps this "good first issue" was more involved than intended. I didn't want you to spend too much time trying to debug if that is the case so I decided to look into it myself.
In any case, don't feel discouraged from sending PRs! We love getting them! 😄

Again, thank you so much for the support and encouragement. Lessons learned - irrespective of the label, be more proactive and diligent while attempting to work on a feature/enhancement/bug.
Yes, for sure, you can expect to see more PRs from me. 😃


Replace FNV by rustc-hash

Both construction have similar goals but rustc-hash ist better suited for contemporary CPU as it works one word at a time instead of byte per byte.

Codecov Report

Merging #1650 (bbb058d) into main (0c2bd36) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1650      +/-   ##
==========================================
+ Coverage   94.02%   94.04%   +0.01%
==========================================
  Files         254      254
  Lines       48158    48158
==========================================
+ Hits        45282    45290       +8
+ Misses       2876     2868       -8

Impacted Files
Coverage Δ

src/aggregation/agg_result.rs
75.51% <ø> (ø)

src/indexer/json_term_writer.rs
99.73% <ø> (ø)

src/aggregation/bucket/range.rs
97.02% <100.00%> (ø)

src/aggregation/bucket/term_agg.rs
99.13% <100.00%> (ø)

src/aggregation/intermediate_agg_result.rs
98.30% <100.00%> (ø)

src/fastfield/multivalued/writer.rs
98.86% <100.00%> (ø)

src/fastfield/writer.rs
88.52% <100.00%> (ø)

src/postings/postings_writer.rs
98.24% <100.00%> (ø)

src/tokenizer/stop_word_filter.rs
77.94% <100.00%> (ø)

src/schema/schema.rs
98.79% <0.00%> (+0.15%)
:arrow_up:

... and 2 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Nice thanks. I'll update the benchmarks to see if there's an difference https://pseitz.github.io/tantivy_bench_graph/index.html


Shopware is returning 404 for all the operation on frontend ui

PHP Version
7.4.23
Shopware Version
5.7.2
Expected behaviour
User should be able to create the test accounts/register and sign up, dummy products and place order/checkout on the machine.
Actual behaviour
I have installed Shopware 57.2 on Ubuntu 18.04 machine. I am able to access the backend at http://<IP_ADDRESS>/shopware5/shopware.php/backend but when I visit the frontend at http://<IP_ADDRESS>/shopware5/shopware.php , I can see the DemoShop UI Page but when I click on any button/url like Sign Up or register it shows 404.
The Apache rewrite module is also enabled.
Here is my .conf file:
<VirtualHost *:80>
    <Directory "/var/www/html/shopware5/">
        Options FollowSymLinks
        AllowOverride All
        Require all granted
    </Directory>

    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html/shopware5
    ServerName examples.com

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

The project is placed under /var/www/html/shopware5.
I am suspecting some issue on the redirection side. Can someone please take a look at this.
How to reproduce
I am facing the issue by following the normal installation method mentioned here:
https://docs.shopware.com/en/shopware-5-en/first-steps/installing-shopware#shopware-one-file-installer
After following everything, I am facing this issue.

at first, this is the place for Shopware 6 issues 😉
second, make sure that you configured the right host and path in your shop config. this should match the path to your Shopware installation


Custom schemas supported?

Hey,
I just started using the usd-unity-sdk. I would like to setup my usd scene from different prims, one of them are prefabs. Meaning my goal would be to specify prefabs directly in the usd file and then spawn them from there, like I would spawn a cube for example. Is that supported somehow? My idea was to generate a custom schema called "Prefab" and create some "PrefabImporter" in the same way the "CubeImporter" etc works, to achieve this. But this seems kind of a big effort and would really create the need to jump deep into your source code and change it.
Is that even a good idea, or would you have a better suggestion to achieve this? Maybe by using some CustomData with an already existing schema? Is it even possible to create custom schemas an add them to your package?
Thanks for the good work and your support!

Hey @vickycl , thanks for your reply.
It's indeed the first case you describe, that I'm trying to achieve. I would like to include already existing prefabs made in Unity into my USD file and then load them at runtime.
I managed to just add custom data to the Xform prim in my USD file and then read the custom data in Unity, by adding my code into the SceneImporter.BuildScene() method, when reading Xforms. That custom data includes the information from where to load my prefab. But that unfortunately alters the original code quite a bit.
But I feel like a cleaner way would indeed be the way, of creating a custom schema and include it. Even if you don't have a general case for handling it in the package, could you maybe guide me in how to achieve it? I created my schema files for prefabs, but I am a bit confused on where I have to place them to use them correctly. I saw that the schema files are placed inside the "\Runtime\Plugins\x86_64\usd" folder, namely the plugInfo.json, generatedSchema.usda and schema.usda. But I think there is more generated content that needs to be placed and assigned somewhere, or?
Thanks for the support.

Hi! We've added a feature request for better custom schema support to our Roadmap for better internal visibility. If you head over to the Roadmap you can add more detail about your use case (on the 'USD Import ticket') for this feature to help us prioritize as we hope to be able to work on this in the future :)


tekton task to run unit tests

Changes
This adds a Task to run pipeline's unit tests

This will move to tektoncd/catalog later on as it's meant to be a reusable task :angel:
This will be hooked-up to the CI once we work on #922

Improvements possible:

[ ] Save the cover file somewhere (in a resource or as a resource :smiling_imp:)
[x] More parameters (to be able to filter tests, or add build tags)

Closes #532
cc @bobcatfish
Signed-off-by: Vincent Demeester<EMAIL_ADDRESS>Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them:

[ :no_good_man: ] Includes tests (if functionality changed/added)
[ :no_good_man: ] Includes docs (if user facing)
[x] Commit messages follow commit message best practices

See the contribution guide
for more details.

@bobcatfish removed the wip as I feel it could get in like that :angel:
I'll prepare a PR on tektoncd/catalog for some go related re-usable tasks (inspired by this one)

/test pull-tekton-pipeline-integration-tests

/lgtm


misc STO backlog items STO-7006, 7003, 6983, 6975, 7153

Jira tickets and doc preview links:

STO-7006 :: Checkmarx scanner reference :: need more info on 'exclude' CLI commands:

Checkmarx scanner reference > Additional CLI flags

STO-7003 :: Snyk Multi Project Scan Settings with Depth
STO-6983 :: Snyk tool args does not support Context Specific Options

Snyk scanner reference > Additional CLI flags

STO-6975 :: OWASP :: Add FAQ re running Yarn scans

STO Knowledge base > FAQs > OWASP scans
OWASP scanner reference > Troubleshoot Yarn Audit Analyzer exceptions

STO-7153 :: refactor STO knowledge base

Security Testing Orchestration (STO) FAQs

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65d28f7fb12513a59fbdb6c4--harness-developer.netlify.app

Please check the Execution Link of the Pipeline for the Website Draft URL. This is located in the Preview Step behind the Harness VPN and also is available in #hdh_alerts. E.g Website Draft URL: https://unique-id--harness-developer.netlify.app. Current Draft URL is: https://65de0f9d20b1dc0716cdb36e--harness-developer.netlify.app


disk: Fix nxp usdhc driver optional gpio properties

The power and chip detect gpios are optional device tree properties for
the nxp usdhc driver. Fixes a build error on the mm_swiftio board, which
does not set the power gpio property.
Signed-off-by: Maureen Helm<EMAIL_ADDRESS>Fixes #29048
cc: @lgl88911

@MaureenHelm Thanks for fix issue, Verify pass on mm_swiftio.


Compatibility with python 3.12

Support Python 3.12 #344
Updated the pyproject.toml and test matrix in .github workflow ci.yaml file to allow python 3.12
pyproject.toml
[tool.poetry.dependencies]
python = "^3.8, < 3.13"

.github/workflow/ci.yaml
test->strategy->matrix->python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]

Obsolete via #349


[Request] Add Malisis Doors support as airlocks

Malisis Doors support as alternative airlocks would be pretty nice. I can add the door block to the sealed block config, however this simply makes the door sealed when open or closed, and it causes the player to take suffocation damage if standing in the door block. It would be nice to have integrated support for Malisis Doors so we can use them as proper airlocks.

I second this. Not only Malisis Doors, but every door from every (or at least the most known ones) mod that looks like it could be sealed.


Poetry install is taking a long time while installing dependencies

Hi,
Poetry install command is taking too much time. Has anyone faced similar issue ? Please suggest some hack

Hi,
Poetry install command is taking too much time. Has anyone faced similar issue ? Please suggest some hack

me too.


add telegraf service and grafana dashboard for haproxy stats

Warning: https://github.com/appcelerator/docker-haproxy/pull/2 (stats listener updates for haproxy:1.0.1) should be merged before this one.

new telegraf-haproxy service, gathering stats from the haproxy service
new Grafana dashboard for Haproxy stats

How to test:
$ swarm pull
$ swarm start
$ swarm monitor
once all services are up, connect to grafana (localhost:6001), authenticate with the admin login, and check the Load Balancer dashboard, data should be displayed.

@ndegory Please rebase against master since we've made other swarm dependency changes (eg, zookeeper+kafka -> nats). Thanks!

was rebased, but tests are still failing. My tests are also failing on the master branch, so I can't confirm or disprove tests on this branch.

👍


Modified Python Dockerfiles to allow for the submission of Python app…

…lications without the --jars parameter. Fixes #409.
Not sure why the singular path provided to -cp does not work, but this fixes it. Also may provide extensibility in that a SPARK_CLASSPATH could be provided in a custom-built spark-base image without requiring modification of the individual driver and executor images.
cc @erikerlandson @ifilonenko @mccheah

Not sure why the singular path provided to -cp does not work, but this fixes it. Also may provide extensibility in that a SPARK_CLASSPATH could be provided in a custom-built spark-base image without requiring modification of the individual driver and executor images.

This should be handled by SPARK_EXTRA_CLASSPATH - SPARK_CLASSPATH isn't expected to be overridden by other Docker images. Although it looks like SPARK_EXTRA_CLASSPATH isn't supported in executors - this was supposed to have been done in https://github.com/apache-spark-on-k8s/spark/pull/383/files but I think we lost it in a bad merge somewhere.

I'd also like to trace down the specific problem the existing shell command is hitting.

@mccheah If it helps, I was able to replicate the error by using kubectl run on a driver pod and executing a stripped-down version of the existing shell command.
${JAVA_HOME}/bin/java -cp /opt/spark/jars/* com.google.gson.Gson gives me the error that we are currently seeing. SPARK_CLASSPATH resolves to /opt/spark/jars/* and the gson class is just a random class I picked out of the selected jars.
After playing around with this command, I got it to work by just adding a colon in front of the classpath — :/opt/spark/jars/*.

I suppose wildcards aren't being expanded out properly. I wonder if we can instead use ls to list the directory and then join the paths together with :, and pass that as SPARK_CLASSPATH. Alternatively, if we know exactly why prepending : works and have it documented then that will suffice.

Yeah, that's the problem. Just tried again with -cp /opt/spark/jars/"*" (just putting quotes around the asterisk) and it resolves properly. Should I forgo the colon for this solution?

I'd like to know the rules of thumb for how Java is treating these inputs and how Bash is expanding them, etc. Though we had a tough time understanding issues like this in e.g. https://github.com/apache-spark-on-k8s/spark/pull/444

I'm not too sure myself. It might be a difference in Bash vs. JRE handling of wildcards. Quoting the wildcard most likely delegates to the JRE for expansion, and adding a colon necessitates parsing of the provided classpath by the JRE and the subsequent expansion of parsed directories.
Maybe @ash211 or @erikerlandson could provide some insight on why this is happening and/or what the best way to patch this would be (colons vs. asterisk)?

FYI, with #462 we should be able to see exactly what command is ultimately being executed by bash from the pod console outputs

@erikerlandson I may be wrong, but I don't think set -x resolves environment variable values.

@erikerlandson @mccheah @sahilprasad any progress on this PR?

@ifilonenko @mccheah @erikerlandson I'm partial to just adding quotes. Cleaner and solves the submission errors.

@ssuchter @erikerlandson @ifilonenko thoughts on my latest commit?

Integration tests are broken - "Could not find or load main class"... I wonder what's going on there?

@mccheah @ssuchter just pushed a version that should fix the failing integration tests. I opted for pre-pending a :, since it seems assigning a wildcard to a variable in bash is a whole different mess than simply evaluating.

Hey all, I ran into this issue independently yesterday and tracked it down before I saw https://github.com/apache-spark-on-k8s/spark/issues/409
I think there is a better solution than prepending a ":" to the $SPARK_CLASSPATH value, and that is to quote $SPARK_CLASSPATH when it is used as the value for the -cp flag in the various Dockerfiles under  resource-managers/kubernetes/docker-minimal-bundle/src/main/docker (not just driver-py and executor-py, this is potentially a general problem).
This is the standard way of preventing the shell from expanding a variable before passing it to a command. So in the driver-py case as an example, the last line of the Dockerfile just becomes:
${JAVA_HOME}/bin/java "${SPARK_DRIVER_JAVA_OPTS[@]}" -cp "$SPARK_CLASSPATH" -Xms$SPARK_DRIVER_MEMORY -Xmx$SPARK_DRIVER_MEMORY -Dspark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS $SPARK_DRIVER_CLASS $PYSPARK_PRIMARY $PYSPARK_FILES $SPARK_DRIVER_ARGS

That's it, no other change to SPARK_CLASSPATH needed.
What's happening with the unmodified Dockerfile (to recap) is that the wildcard list is being expanded as a space seperated list after the -cp flag, and the second value is taken by java as the target to execute. The rest of the line is thrown away. (You can see this by rebuilding the image and inserting an "echo" of the java invocation before the actual command, like so:
...
if ! [ -z ${SPARK_MOUNTED_FILES_FROM_SECRET_DIR+x} ]; then cp -R "$SPARK_MOUNTED_FILES_FROM_SECRET_DIR/." .; fi && \
echo ${JAVA_HOME}/bin/java "${SPARK_DRIVER_JAVA_OPTS[@]}" -cp $SPARK_CLASSPATH-Xms$SPARK_DRIVER_MEMORY -Xmx$SPARK_DRIVER_MEMORY -Dspark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS $SPARK_DRIVER_CLASS $PYSPARK_PRIMARY $PYSPARK_FILES $SPARK_DRIVER_ARGS && \
...

You can also verify this behavior easily by downloading the spark distro and playing from a shell:
[machine spark-2.2.0-k8s-0.5.0-bin-2.7.3]$ SPARK_CONTEXT=jars/*
[machine spark-2.2.0-k8s-0.5.0-bin-2.7.3]$ echo $SPARK_CONTEXT
jars/activation-1.1.1.jar jars/antlr-2.7.7.jar jars/antlr4-runtime-4.5.3.jar jars/antlr-runtime-3.4.jar jars/aopalliance-1.0.jar jars/aopalliance-repackaged-2.4.0-b34.jar jars/apacheds-i18n-2.0.0-M15.jar jars/apacheds-kerberos-codec-2.0.0-M15.jar ...
[machine spark-2.2.0-k8s-0.5.0-bin-2.7.3]$ echo "$SPARK_CONTEXT"
jars/*

The trouble is I see with prepending a ":" is that yes, it happens to prevent the shell expansion because the shell doesn't recognize it as a valid path any longer, but it is also undefined behavior for java. It literally seems to imply an empty path on the classpath list. This might confuse noobs and I think clouds the actually issue (and could perpetuate confusion in other code :) )  Why not defeat the wildcard expansion with the standard shell mechanism?
I have a PR with changes to the set of docker files that use -cp SPARK_CONTEXT in this way, but I thought I would comment on the original issue here first. I can push that PR if you like.
Best, Trevor

I've been conferring with @tmckayus on this, and the evidence suggests we should just quote the classpath instead of prepending a colon.  Either way we ought to get this nit resolved and close out the issue.  I'm in favor of pushing the alternate PR for comparison.

Okay, for comparison the other PR is https://github.com/apache-spark-on-k8s/spark/pull/541

This can be closed right? I hit https://github.com/apache-spark-on-k8s/spark/issues/409 as well and just tried the fix in #541 which fixes it for me.


Very confused, need to authenticate via App, not environment

I would like to authenticate my server within the start of Node application which receives messages from Pub/Sub. None of these samples includes the authentication part for Pub/Sub client which receives messages.
In example. How can I add authentication in this file: samples/topics.js
Every single example assumes that everyone uses environment variables. I can get the script work fine if environment variables is set, but reading auth data from JSON is yet unknown. I found out some google-auth-library that creates a JWT client, but for turn of JWT it does not include a single example how to use any of Cloud features with authenticated JWT.
Can somebody help? The documents are kinda confusing when there's tons of redirects on different pages so I cannot find combination for app based auth and usage of Pub/Sub.

@MPNull would providing a credentials object in the client config work for you?
const {PubSub} = require('@google-cloud/pubsub');
const pubsub = new PubSub({
  credentials: {
    client_email<EMAIL_ADDRESS>    private_key: '...'
  }
});

I do believe it works. I'll test it as soon as I get back on PC.

👋 @MPNull just a gentle ping, wanted to check in and see if the posted solution worked out for you or not!

I do believe that's it. I have problems to authenticate on Android (Java) so I actually moved to HTTP API instead so I were too bored to rewrite this on server. Can be closed.

Cool, thanks!


Fails to read externs file on any project

I'm using psc <IP_ADDRESS>, psc-ide 6.0 (commit d86acd6b533a136bb28fabea29fb5955692a51a4), up-to-date psc-ide-vim, pulp 8.0.0.
I started a new project, and the server in debug mode:
mkdir failing
cd failing
pulp init
pulp build
psc-ide-server --debug

I open up src/Main.purs in vim, call :PSCIDEtype on anything, and it tells me this:
Answer was: Right (TextResult "/home/thimoteus/Desktop/test/failing")
{"params":{"dependencies":["Main"],"modules":[]},"command":"load"}
Answer was: Left (GeneralError "Parsing the extern at: /home/thimoteus/Desktop/test/failing/output/Main/externs.json failed")
{"command":"cwd"}
Answer was: Right (TextResult "/home/thimoteus/Desktop/test/failing")
{"params":{"search":"log","filters":[]},"command":"type"}
Answer was: Right (CompletionResult [])

Here's the contents of output/Main/externs.json.

Hmm I couldn't reproduce. My --debug output looks like this however:
~/Documents/test
❯ psc-ide-server --debug
[Debug] Accepted a connection @(main:Main server/Main.hs:119:5)
[Debug] {"command":"load","params":{"modules":[],"dependencies":["Main"]}} @(main:Main server/Main.hs:121:5)
[Debug] Inserting Module: Main @(pscid_FaLaR9ggoFzDXE6SxNHQhH:PureScript.Ide.State src/PureScript/Ide/State.hs:64:5)
[Debug] Loaded extern file at: /home/creek/Documents/test/output/Main/externs.json @(pscid_FaLaR9ggoFzDXE6SxNHQhH:PureScript.Ide src/PureScript/Ide.hs:144:5)
[Debug] Inserting Module: Prelude @(pscid_FaLaR9ggoFzDXE6SxNHQhH:PureScript.Ide.State src/PureScript/Ide/State.hs:64:5)
[Debug] Loaded extern file at: /home/creek/Documents/test/output/Prelude/externs.json @(pscid_FaLaR9ggoFzDXE6SxNHQhH:PureScript.Ide src/PureScript/Ide.hs:144:5)
[Debug] Inserting Module: Control.Monad.Eff @(pscid_FaLaR9ggoFzDXE6SxNHQhH:PureScript.Ide.State src/PureScript/Ide/State.hs:64:5)
[Debug] Loaded extern file at: /home/creek/Documents/test/output/Control.Monad.Eff/externs.json @(pscid_FaLaR9ggoFzDXE6SxNHQhH:PureScript.Ide src/PureScript/Ide.hs:144:5)
[Debug] Inserting Module: Control.Monad.Eff.Console @(pscid_FaLaR9ggoFzDXE6SxNHQhH:PureScript.Ide.State src/PureScript/Ide/State.hs:64:5)
[Debug] Loaded extern file at: /home/creek/Documents/test/output/Control.Monad.Eff.Console/externs.json @(pscid_FaLaR9ggoFzDXE6SxNHQhH:PureScript.Ide src/PureScript/Ide.hs:144:5)
[Debug] Answer was: Right (TextResult "Loaded extern file at: /home/creek/Documents/test/output/Main/externs.json, Dependencies for Main loaded.") @(main:Main server/Main.hs:104:13)
[Debug] Accepted a connection @(main:Main server/Main.hs:119:5)
[Debug] {"command":"type","params":{"filters":[{"filter":"modules","params":{"modules":["Control.Monad.Eff.Console","Control.Monad.Eff","Prelude","Main"]}}],"search":"log"}} @(main:Main server/Main.hs:121:5)
[Debug] Answer was: Right (CompletionResult [Completion ("Control.Monad.Eff.Console","log","forall eff. String -> Eff ( console :: CONSOLE | eff ) Unit")]) @(main:Main server/Main.hs:104:13)

I recently switched to a logging library for psc-ide so I think you might still have an old version around...

Ugh, I had forgot to put the new executable on my path :tired_face:
After that, all is fine.


[instagram] Error logging in, cannot download Instagram Highlights

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2022.10.04 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
Uzbekistan
Provide a description that is worded well enough to be understood
I can't get information about instagram highlits. If you specify --cookies-from-browser, everything is fine
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
yt-dlp -j -vU https://www.instagram.com/stories/highlights/17945585140590250/ --username "my gmail" --password "my password"
[debug] Command-line config: ['-j', '-vU', 'https://www.instagram.com/stories/highlights/17921449453583708/', '--username', 'PRIVATE', '--password', 'PRIVATE']
[debug] Encodings: locale cp1251, fs utf-8, pref cp1251, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.10.04 [4e0511f] (win32_exe)
[debug] Python 3.8.10 (CPython 64bit) - Windows-10-10.0.19044-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg n5.0.1-5-g240d82f26e-20220614 (setts), ffprobe n5.0.1-5-g240d82f26e-20220614
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Loaded 1690 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.10.04, Current version: 2022.10.04
yt-dlp is up to date (2022.10.04)
[instagram:story] Downloading login webpage
ERROR:<PHONE_NUMBER>3583708: An extractor error has occurred. (caused by KeyError('config')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 670, in extract
  File "yt_dlp\extractor\common.py", line 576, in initialize
  File "yt_dlp\extractor\instagram.py", line 67, in _perform_login
KeyError: 'config'

Highlights are not accessible when not logged in (via web browser or yt-dlp).
This is an issue with logging in. See also: #5154, although this error is new.

Is it possible to authorize automatically?

Is it possible to authorize automatically?

since login w/credentials seems to be broken for this extractor currently, the only way is to use --cookies-from-browser or --cookies

And the cookie is temporary, it turns out it will need to update manually the cookie after some time?

--cookies-from-browser will extract the instagram cookies from your browser's cookie storage each time you run yt-dlp with that option. If you have logged into instagram or browsed instagram while logged in with your browser recently, the cookies should be up-to-date.
I would caution you to use your login cookies only when absolutely necessary / sparingly, since usage of yt-dlp can be detected by instagram and it could lead to an account suspension or ban.

Thanks for the advice, but isn't there a way to download a lot of videos automatically without getting blocked?

Closing in favor of #5154


Filter improvement (DM and Log)

Filters should DM the person the message they removed, as well as log it in a bot-log channel.

#177 was the logging part of this, stull no DMing users though


Enhancement: centralize converting CSV to JSON in m365 tenant report commands

The commands m365 tenant report getOffice365ActivationsUserDetail, m365 tenant report getOffice365ActivationsUserCounts and m365 tenant report getOffice365ActivationCounts use the same conversion method. It's better to centralize this method in the Utils class.

@waldekmastykarz Can you assign to me please ?

Brilliant! Appreciate your help @Abderahman88 👏


Black screen on prepatcher launch

I'm not the greatest as deciphering Logs but it seems to be vanilla expanded framework is causing an issue. Attached is the log file if you would like to take a look. If you need me to do anything to troubleshoot let me know.
Player.log

It should be fixed now

do i need to redownload to resolve?

I uploaded the fix in https://github.com/Zetrith/Prepatcher/releases if you aren't using Steam


legend_title bug

geemap version:
Python version: 3.7
Operating System: ubuntu bionic

Description
When calling Map.add_legend(legend_title = ... the keyword is capitalized in the method call, however, the corresponding keywords in legends.py are not. This means some legend entries can not be found.
What I Did
Map.add_legend(legend_title = "OPERNICUS/Landcover/100m/Proba-V/Global")
fails because the entry is capitalized in the method but not in legends.py

Thanks for reporting this bug. It has been fixed. Please update the package using the following and restart the kernel.
geemap.update_package()


ChakraCore static library

Hello friends! I want to embed ChakraCore into my project.
But it's undesirable to use chakra's dll. Can I build chakra as static library(or libraries) without dll?
P.S. Chakra is better than v8, thank you Microsoft!

There is no (supported) way to build a static library on Windows, as far as I know.  Linux and macOS yes, but not Windows sadly. ☹

While it is not officially supported a windows static library of chakracore could likely be built with ~1 hour of fiddling see prior discussion here: https://github.com/microsoft/ChakraCore/issues/279