Access policies for selects over SQL adapter

adds sql conn setting apply_access_policies_sql
it is passed into qlcompiler as apply_user_access_policies when compiling DML or computeds,
when compiling table references and apply_access_policies_sql is enabled, instead of composing inheritance view, we now compile edgeql select ObjectType { id, __type__, ... all ptrs ... }, which will include select access policies.

If apply_access_policies_sql is not set, we still use inheritance views (or base table directly if ONLY is used), which yields faster compilation times.
Two important details:

When apply_access_policies_sql is set, system columns (tableoid, xmin, ...) are all NULL.

Link and pointer tables currently do not have access policies applied. We should probably implement a behavior where access polices are applied to the source of the link/pointer table.

* When `apply_access_policies_sql` is set, system columns (tableoid, xmin, ...) are all NULL.

Does this need to be true?

As a reference, this is current compiled SQL:
================================================================================
SQL Input
================================================================================
FROM "Document"
SELECT name

================================================================================
SQL Output
================================================================================

    WITH
    "t_default::Document~1" AS NOT MATERIALIZED
        ((
            FROM LATERAL (
                FROM edgedbpub."136e9c0e-8b26-11ef-b231-d7914aacd091" AS "Document~3"
                CROSS JOIN LATERAL (
                    FROM LATERAL (
                        SELECT/*<pg.SelectStmt at 0x7f23ad52e810>*/
                            (($1)::text IS NOT DISTINCT FROM "Document~3"."1374dbac-8b26-11ef-ade6-4d0e67fc01fe") AS "expr~2_value~1"
                    ) AS "expr-2~2"
                    CROSS JOIN (SELECT (NULL)::uuid AS "v~2") AS "v~1"
                    CROSS JOIN LATERAL (
                        SELECT/*<pg.SelectStmt at 0x7f23ad65d3d0>*/
                            ("Document~3".id IS NOT DISTINCT FROM "v~1"."v~2") AS "expr~3_value~1"
                    ) AS "expr-3~2"
                    SELECT/*<pg.SelectStmt at 0x7f23ad52e2d0>*/
                        ("expr-2~2"."expr~2_value~1" OR
                         "expr-3~2"."expr~3_value~1") AS "expr~4_value~1"
                ) AS "expr-4~2"
                SELECT/*<pg.SelectStmt at 0x7f23ad6b77d0>*/
                    "Document~3".id AS "Document_value~1",
                    ('136e9c0e-8b26-11ef-b231-d7914aacd091')::uuid AS "__type___identity~1",
                    "Document~3"."1374dbac-8b26-11ef-ade6-4d0e67fc01fe" AS "name_serialized~1",
                    "Document~3"."989c2c37-8b26-11ef-94b7-df58d575b083" AS "owner_identity~1"
                WHERE
                    "expr-4~2"."expr~4_value~1"
            ) AS "expr-5~2"
            SELECT/*<pg.SelectStmt at 0x7f23ad6b7470>*/
                "expr-5~2"."Document_value~1" AS "expr~5_value~1",
                "expr-5~2"."__type___identity~1" AS "__type___identity~2",
                "expr-5~2"."name_serialized~1" AS "name_serialized~2",
                "expr-5~2"."owner_identity~1" AS "owner_identity~2"
        )),
    "t_schema::ObjectType~1" AS NOT MATERIALIZED
        ((
            FROM LATERAL (
                FROM edgedbstd_v6_2f20952c31."2662a1b4-4f3f-5875-b6eb-ce52101a90a3" AS "ObjectType~2"
                CROSS JOIN LATERAL (
                    FROM LATERAL (
                        SELECT/*<pg.SelectStmt at 0x7f23ad52fc50>*/
                            (NOT "ObjectType~2"."101d5f8c-251a-5fee-b30d-c0a3ed3ded56") AS "expr~6_value~1"
                    ) AS "expr-6~2"
                    CROSS JOIN (SELECT (NULL)::uuid AS "v~4") AS "v~3"
                    CROSS JOIN LATERAL (
                        SELECT/*<pg.SelectStmt at 0x7f23ad52fbc0>*/
                            ("ObjectType~2".id IS NOT DISTINCT FROM "v~3"."v~4") AS "expr~7_value~1"
                    ) AS "expr-7~2"
                    SELECT/*<pg.SelectStmt at 0x7f23ad52f980>*/
                        ("expr-6~2"."expr~6_value~1" OR
                         "expr-7~2"."expr~7_value~1") AS "expr~8_value~1"
                ) AS "expr-8~2"
                SELECT/*<pg.SelectStmt at 0x7f23ad52e8a0>*/
                    "ObjectType~2".id AS "ObjectType_value~1"
                WHERE
                    "expr-8~2"."expr~8_value~1"
            ) AS "expr-9~2"
            SELECT/*<pg.SelectStmt at 0x7f23ad52e3f0>*/
                "expr-9~2"."ObjectType_value~1" AS "expr~9_identity~1"
        )),
    "tbl~1" AS
        ((
            FROM (
                FROM "t_default::Document~1" AS "t~1"
                SELECT/*<pg.SelectStmt at 0x7f23ad6b7590>*/
                    "t~1"."expr~5_value~1" AS "expr~5_value~2",
                    "t~1"."__type___identity~2" AS "__type___identity~3",
                    "t~1"."name_serialized~2" AS "name_serialized~3",
                    "t~1"."owner_identity~2" AS "owner_identity~3"
            ) AS "q~1"
            SELECT/*<pg.SelectStmt at 0x7f23ad6b5e20>*/
                (
                    FROM LATERAL (
                        FROM LATERAL (
                            SELECT/*<pg.SelectStmt at 0x7f23ad5095b0>*/
                                "q~1"."__type___identity~3" AS "__type___identity~4"
                        ) AS "q~2"
                        JOIN LATERAL (
                            FROM "t_schema::ObjectType~1" AS "t~2"
                            SELECT/*<pg.SelectStmt at 0x7f23ad52e450>*/
                                "t~2"."expr~9_identity~1" AS "expr~9_identity~2"
                        ) AS "q~3"
                            ON ("q~2"."__type___identity~4" = "q~3"."expr~9_identity~2")
                        SELECT/*<pg.SelectStmt at 0x7f23ad6b7ad0>*/
                            "q~3"."expr~9_identity~2" AS "__type___value~1",
                            "q~1"."__type___identity~3" AS "__type___serialized~1"
                    ) AS "ObjectType___type__~3"
                    SELECT/*<pg.SelectStmt at 0x7f23ad6b7680>*/
                        "ObjectType___type__~3"."__type___serialized~1" AS "__type___serialized~2"
                ) AS __type__,
                "q~1"."expr~5_value~2" AS id,
                (
                    SELECT/*<pg.SelectStmt at 0x7f23ad50a7e0>*/
                        "q~1"."name_serialized~3" AS "name_serialized~4"
                ) AS "1374dbac-8b26-11ef-ade6-4d0e67fc01fe",
                (
                    FROM LATERAL (
                        FROM LATERAL (
                            SELECT/*<pg.SelectStmt at 0x7f23ad6dd1c0>*/
                                "q~1"."owner_identity~3" AS "owner_identity~4"
                        ) AS "q~4"
                        JOIN edgedbpub."8d25852f-8b26-11ef-bc83-8f394cce6cde" AS "User~1"
                            ON ("q~4"."owner_identity~4" = "User~1".id)
                        SELECT/*<pg.SelectStmt at 0x7f23ad52f800>*/
                            "User~1".id AS "owner_value~1",
                            "q~1"."owner_identity~3" AS "owner_serialized~1"
                    ) AS "User_owner~3"
                    SELECT/*<pg.SelectStmt at 0x7f23ad52eb40>*/
                        "User_owner~3"."owner_serialized~1" AS "owner_serialized~2"
                ) AS "989c2c37-8b26-11ef-94b7-df58d575b083",
                NULL AS tableoid,
                NULL AS xmin,
                NULL AS cmin,
                NULL AS xmax,
                NULL AS cmax,
                NULL AS ctid
        ))
    FROM "tbl~1" AS "rel~1"
    SELECT/*<pg.SelectStmt at 0x7f23add0fa40>*/
        "rel~1"."1374dbac-8b26-11ef-ade6-4d0e67fc01fe" AS name

The newly compiled SQL:

================================================================================
SQL Input
================================================================================
FROM "Document"
SELECT *

================================================================================
SQL Output
================================================================================

    WITH
    "t_default::Document~1" AS NOT MATERIALIZED
        ((
            FROM LATERAL (
                FROM edgedbpub."df301555-8beb-11ef-8a5c-713229fba659" AS "Document~3"
                CROSS JOIN LATERAL (
                    FROM LATERAL (
                        SELECT/*<pg.SelectStmt at 0x7ff62cfee150>*/
                            (($1)::text IS NOT DISTINCT FROM "Document~3"."df360b51-8beb-11ef-b3e3-fb16e5854ad0") AS "expr~2_value~1"
                    ) AS "expr-2~2"
                    CROSS JOIN (SELECT (NULL)::uuid AS "v~2") AS "v~1"
                    CROSS JOIN LATERAL (
                        SELECT/*<pg.SelectStmt at 0x7ff62cfeef60>*/
                            ("Document~3".id IS NOT DISTINCT FROM "v~1"."v~2") AS "expr~3_value~1"
                    ) AS "expr-3~2"
                    SELECT/*<pg.SelectStmt at 0x7ff62cfee0c0>*/
                        ("expr-2~2"."expr~2_value~1" OR
                         "expr-3~2"."expr~3_value~1") AS "expr~4_value~1"
                ) AS "expr-4~2"
                SELECT/*<pg.SelectStmt at 0x7ff62cfa73b0>*/
                    "Document~3".id AS "Document_value~1",
                    ('df301555-8beb-11ef-8a5c-713229fba659')::uuid AS "__type___value~1",
                    "Document~3"."df360b51-8beb-11ef-b3e3-fb16e5854ad0" AS "name_value~1",
                    "Document~3"."df3659c6-8beb-11ef-9890-79f309dd9ffb" AS "owner_value~1",
                    "Document~3".tableoid AS "tableoid_value~1",
                    "Document~3".xmin AS "xmin_value~1",
                    "Document~3".cmin AS "cmin_value~1",
                    "Document~3".xmax AS "xmax_value~1",
                    "Document~3".cmax AS "cmax_value~1",
                    "Document~3".ctid AS "ctid_value~1"
                WHERE
                    "expr-4~2"."expr~4_value~1"
            ) AS "expr-5~2"
            SELECT/*<pg.SelectStmt at 0x7ff62cfa51f0>*/
                "expr-5~2"."Document_value~1" AS "expr~5_value~1",
                "expr-5~2"."__type___value~1" AS "__type___value~2",
                "expr-5~2"."name_value~1" AS "name_value~2",
                "expr-5~2"."owner_value~1" AS "owner_value~2",
                "expr-5~2"."tableoid_value~1" AS "tableoid_value~2",
                "expr-5~2"."xmin_value~1" AS "xmin_value~2",
                "expr-5~2"."cmin_value~1" AS "cmin_value~2",
                "expr-5~2"."xmax_value~1" AS "xmax_value~2",
                "expr-5~2"."cmax_value~1" AS "cmax_value~2",
                "expr-5~2"."ctid_value~1" AS "ctid_value~2"
        )),
    "tbl~1" AS
        ((
            FROM (
                FROM "t_default::Document~1" AS "t~1"
                SELECT/*<pg.SelectStmt at 0x7ff62cfefb30>*/
                    "t~1"."expr~5_value~1" AS "expr~5_value~2",
                    "t~1"."__type___value~2" AS "__type___value~3",
                    "t~1"."name_value~2" AS "name_value~3",
                    "t~1"."owner_value~2" AS "owner_value~3",
                    "t~1"."tableoid_value~2" AS "tableoid_value~3",
                    "t~1"."xmin_value~2" AS "xmin_value~3",
                    "t~1"."cmin_value~2" AS "cmin_value~3",
                    "t~1"."xmax_value~2" AS "xmax_value~3",
                    "t~1"."cmax_value~2" AS "cmax_value~3",
                    "t~1"."ctid_value~2" AS "ctid_value~3"
            ) AS "q~1"
            SELECT/*<pg.SelectStmt at 0x7ff62cfefe00>*/
                "q~1"."expr~5_value~2" AS "Document_serialized~1",
                "q~1"."__type___value~3" AS "__type___value~4",
                "q~1"."name_value~3" AS "name_value~4",
                "q~1"."owner_value~3" AS "owner_value~4",
                "q~1"."tableoid_value~3" AS "tableoid_value~4",
                "q~1"."xmin_value~3" AS "xmin_value~4",
                "q~1"."cmin_value~3" AS "cmin_value~4",
                "q~1"."xmax_value~3" AS "xmax_value~4",
                "q~1"."cmax_value~3" AS "cmax_value~4",
                "q~1"."ctid_value~3" AS "ctid_value~4"
        ))
    FROM "tbl~1" AS "rel~1"
    SELECT/*<pg.SelectStmt at 0x7ff62cfec7a0>*/
        "rel~1"."Document_serialized~1" AS id,
        "rel~1"."__type___value~4" AS __type__,
        "rel~1"."name_value~4" AS name,
        "rel~1"."owner_value~4" AS owner_id


Fix warnings from anndata v0.6.22

Fixes for deprecation warnings from anndatav0.6.22. This is mostly by replacing access to .X with access via {obs,var}_vector or sc.get.obs_df.
Supercedes #713, fixes #700 and #690.
Functions changed:

sc.pl.violin

Dataframe for plotting now constructed with obs_df, access to adata.X no longer used.

sc.pl.scatter

Changed default layer from "X" to "None". "X" is still supported, but should throw a deprecation warning if it's explicitly used.
Replace usage of ._get_obs_array with .obs_vector

sc.pl._tools.scatterplots.plot_scatter

Normalized access to layers, now sparse and dense should similarly.

sc.get.obs_df

Added support for use_raw

Great! Don't we want to add anndata v0.6.22 to the requirements and the version check at startup in scanpy/__init__.py? At least we should make sure anndata is 0.6.21 as otherwise .obs_vector doesn't exist (https://anndata.readthedocs.io/en/stable/#post-v0-6-june-6-2019).
Otherwise, happy to make Scanpy v1.4.4.

Sure! I wansn't sure if there were other bugs to fix or PRs to merge before we wanted to make a release. I'd also like to get @fidelram to give this a look over. I think I didn't break anything, but he'd be in a better position to tell if that were the case.

@ivirshup I checked and now I don't get  warnings :)
However, I could not plot the layer. In the following example, I make a new layer that is the negative of the default layer. As you see, bot the default and the negative ('test') layer are identical.

Ah, it looks like use_raw was being set to True, even if layers was passed.
https://github.com/theislab/scanpy/blob/c748b3558b38e908f00b16b0c18e2846d3599e5c/scanpy/plotting/_tools/scatterplots.py#L74-L76
Any idea why this didn't trigger this test?
https://github.com/theislab/scanpy/blob/c748b3558b38e908f00b16b0c18e2846d3599e5c/scanpy/tests/test_plotting.py#L294-L298

Also, heads up, a bunch of plots break going from matplotlib v3.1.0 -> v3.1.1.

Fixed and added a couple tests. I didn't change the test mentioned above though, which might be a good thing to do.

@ivirshup Indeed the problem is use_raw=True by default. In the test, I think what happens is that the raw data is being plotted and thus no error appears. The tolerance for the image difference may hide the problem if indeed the test image is correct.
To avoid this confusion when plotting a layer I think it is better to override use_raw. This is how it was supposed to be working before the changes according to the documentation:
layer : typing.Union[str, NoneType], optional (default: None)
    Name of the AnnData object layer that wants to be plotted. By default
    adata.raw.X is plotted. If `use_raw=False` is set, then `adata.X` is plotted.
    If `layer` is set to a valid layer name, then the layer is plotted. `layer`
    takes precedence over `use_raw`.

The current logic is around this lines https://github.com/theislab/scanpy/blob/master/scanpy/plotting/_tools/scatterplots.py#L744
PS: the use_raw has been a source of many confusions for me. Now I know when raw is used by default but for new users this may not be obvious. One solution is to add a warning message everytime that use_raw is set to True by the code.

@ivirshup I tested and now is working as expected. Thanks for adding the new tests. From my side is ready to go.

@fidelram I've made sure that only use_raw or layer has been specified, though the default of use_raw being True is still used if layer isn't set. I think it would be good if this was covered in the docs for use_raw as well.
I also get tripped up by use_raw pretty frequently. I think a warning for this behavior would be good, but I don't like the idea of the default setting issuing a warning. What if we changed the default to false? This would need a deprecation warning for a bit and waiting until the next big release though.

I've updated the docs a little and am going to go ahead and merge this. Thanks for the feedback @fidelram!

@falexwolf, realized I didn't change the AnnData dependency. I'm not totally sure what to do with that, since we've already got a requirement on 0.6.22 due to scipy and statsmodels.


Environment variables are not getting updated

I am using expo ~43.0.0
The package reads the old value even if the server is re-started.
Tried all the solutions described in the similar issue. Didn't help.
Workaround (not ideal)
If I change the App.js file(I've imported env. variables here), the variables do get updated.
But, why should we change the file?
I'm not sure if this is the issue with react-native-dotenv package or, expo.

I just had a similar issue. I ran the following cmd to fix it:
rm -rf $TMPDIR/metro-* && rm -rf $TMPDIR/haste-map-* yarn cache clean

thank you @Coci0AlexMeew ! I have also reopened #75 so I'm marking this as a duplicate


Documentation/Help should be multi-page with auto-generated Nav & TOC

Currently the help page ("documentation and support") is just one static page, which doesn't allow for a lot of flexibility or expansion. ScholarSphere's documentation should be multiple pages with auto-generated navigation & table of contents.
Currently, we are using MKdocs to prepare a number of content and topic-specific guides (repo, static site). MKdocs works really well, but I would prefer to integrate this material into ScholarSphere more seamlessly, not leave it on a separate site. What's the best way to do this?

one solution is to generate static documentation pages as part of deployment.

@whereismyjetpack , I believe we agreed on docs.scholarsphere.psu.edu for the documentation site. Is there anything you need from me?
I'm updating the main issue (above) with additional tasks I think are necessary for completing this.

I've put a request into central IT for a CNAME


Add Typescript ESLint plugin to the examples

Describe the feature you'd like to request
I'd like to request some sort of guide on how to setup Typescript ESlint plugin.
I am unable to set up Typescript ESlint following this guide with the monorepo structure detailed in the Turborepo examples because it's not clear how to use it when both eslint and tsconfig files are inside packages. My preferred setup would be to extend plugin:@typescript-eslint/recommended-requiring-type-checking to leverage the power of Typescript in linting.
Here's what I tried: https://github.com/la55u/turborepo-kitchen-sink

Start from the kitchen-sink example
Added  @typescript-eslint/parser and @typescript-eslint/eslint-plugin to the eslint-config-custom/package.json
Modified eslint-config-custom/index.js to:

index.js
```module.exports = {
  extends: [
    "next",
    "turbo",
    "prettier",
    "plugin:@typescript-eslint/recommended",
    "plugin:@typescript-eslint/recommended-requiring-type-checking",
  ],
  parser: "@typescript-eslint/parser",
  plugins: ["@typescript-eslint"],
  parserOptions: {
    tsconfigRootDir: __dirname,
    project: ["**/packages/**/tsconfig.json"],
  },
  settings: {
    react: {
      version: "detect",
    },
  },
};
```

Run yarn run lint

I get the following error which I am unable to solve (I already because I already added this option).
Error: Error while loading rule '@typescript-eslint/await-thenable': You have used a rule which requires parserServices to be generated. You must therefore provide a value for the "parserOptions.project" property for @typescript-eslint/parser.

It seems like a common usecase to incude this plugin yet there is no docs on it how to use it, I think it would be nice to add it.
Describe the solution you'd like
Add plugin:@typescript-eslint/recommended-requiring-type-checking, to all the examples or create a separate example for it or just make a guide in the docs for how to use them in a monorepo setup where the tscondig and eslint configs are both in a shared package.
Describe alternatives you've considered
Leave it to the user to figure out which is obviously not ideal in my opinion.

Please read this for information about how to setup typescript-eslint with a monorepo: https://typescript-eslint.io/docs/linting/typed-linting/monorepos/#one-tsconfigjson-per-package-and-an-optional-one-in-the-root

I've read it as you can see in my example I tried to add parserOptions to the lint config but it doesn't work. What am I doing wrong? Thank you

@la55u I am facing the same problem.
When using plugins in extensions, ESLint does not use the package context of the extension. It uses a package from where the lint is being executed. That means I must add all plugins to every package that uses my custom extension.
I'd be interested in a solution for that as well

This is blocking from adopting turborepo also. I wonder what other people are doing that this isn't a problem!

@la55u @raphaelluchini I think I figured it out, with the help of @itsUndefined
Don't define the parserOptions inside the custom package, do it inside the library using it!

Custom is provided as a plugin, so I try adding it with a relative path. it works.
/** @type {import('@typescript-eslint/utils').TSESLint.Linter.Config} */
module.exports = {
  parser: '@typescript-eslint/parser',
  parserOptions: {
    project: [
      './tsconfig.json'
    ],
  },
  // others..

I installed kitchen-sink from scratch in order to tackle this problem and found a solution. But you're not going to like it.
It assumes that each monorepo has only one Next app. Do this in the app's .eslintrc.js:
module.exports = {
  root: true,
  extends: ['custom'],
  settings: {
    next: {
      rootDir: '.',
    },
  },
  parserOptions: {
    project: ['./tsconfig.json'],
  },
};

And do this in the .eslintrc.js of each React library that the app uses:
module.exports = {
  root: true,
  extends: ['custom'],
  settings: {
    next: {
      rootDir: '../../apps/storefront',
    },
  },
  parserOptions: {
    project: ['./tsconfig.json'],
  },
};

Here's packages/eslint-config-custom/package.json:
{
  "name": "eslint-config-custom",
  "version": "0.0.0",
  "private": true,
  "license": "MIT",
  "main": "index.js",
  "dependencies": {
    "@typescript-eslint/eslint-plugin": "^5.53.0",
    "@typescript-eslint/parser": "^5.53.0",
    "eslint-config-next": "latest",
    "eslint-config-prettier": "^8.3.0",
    "eslint-plugin-react": "7.28.0",
    "eslint-config-turbo": "latest"
  },
  "publishConfig": {
    "access": "public"
  }
}

And here's packages/eslint-config-custom/index.js:
module.exports = {
  extends: [
    'next',
    'turbo',
    'prettier',
    'plugin:@typescript-eslint/recommended-requiring-type-checking',
  ],
  parser: '@typescript-eslint/parser',
  plugins: ['@typescript-eslint'],
  settings: {
    react: {
      version: 'detect',
    },
  },
  parserOptions: {
    babelOptions: {
      presets: [require.resolve('next/babel')],
    },
  },
};

Oh wait! @gracefullight's solution was sufficient. I was also trying to get rid of the following linting errors, WHICH ALSO OCCUR IN THE VANILLA EXAMPLE:
logger:lint: Pages directory cannot be found at /Users/joe/repos/kitchen-sink/packages/logger/pages or /Users/joe/repos/kitchen-sink/packages/logger/src/pages. If using a custom path, please configure with the `no-html-link-for-pages` rule in your eslint config file.
logger:lint: Warning: React version was set to "detect" in eslint-plugin-react settings, but the "react" package is not installed. Assuming latest React version for linting.
ui:lint: Pages directory cannot be found at /Users/joe/repos/kitchen-sink/packages/ui/pages or /Users/joe/repos/kitchen-sink/packages/ui/src/pages. If using a custom path, please configure with the `no-html-link-for-pages` rule in your eslint config file.

Hey, folks! If anyone has the configuration and wants to make a PR to add it to kitchen-sink, that would be great!

@anthonyshew, I can do this, but it requires modifying every .estlintrc.js in every example that uses TypeScript. I want to double check that it must be done on a per-workspace basis before proceeding.

Looking into this again, I wonder if we should document this solution rather than change all the configuration files. The change is only needed when using @typescript-eslint plugins. Why complicate configuration unnecessarily?

We're in the process of updating both our internal lint configs and our example lint configs to have TS lint configured by default. All of our internal packages and most of our example (soon to be all) now extend from the @vercel/style-guide which makes configuring TS super simple.
Check out the basic example (npx create-turbo@latest <dir> <package-manager>), or the Vercel style guide.


configure default length

Since the ji32k7au4a83 incident I tend o use passwords with an odd number of characters, because I assume this can not happen then. gopass has a default length value set to 24 though:
mcn@tx ~ gopass generate -c thismount/mysite/mylogin
How long should the password be? [24]: 25

It would be cool to be able to set the default value to something else. Since I never found a setting therefore in the docs, I herewith kindly request this feature :)

Bummer!


adding a note about prior gtest in system affecting build

Added a note in readme to warn users about existing libgtest.a in the system causing build failure

This happens when DEPENDENCIES_FORCE_DOWNLOAD is on, doesn't it?

that flag is not being explicitly set; if setting that flag alleviates the problem, I can modify the note accordingly

If that option is not on, cmake script should first try to find and use installed GTest library and it doesn't download GTest if it finds one.

Right, but gtest such a common lib with devs, this is commonly encountered issue,
and this is extra documentation to help developers
I am going to close this PR; I will issue a new PR with updates to contributing guidelines

Aaa, ok. I thought you've encountered that issue and that's why you're adding that note.

Yes, I did encounter that problem, I did not use the flag (which is the cleaner option) to get around the problem. I have updated the note.


Timezone issues

It seems that jobby runs at EST timezone on my machine.
The server is configured to show a different timezone, and I also tried using date_default_timezone_set in the jobby file but the cron expressions are still considered as if they are EST.
Is this by design, or a bug, or have I missed something?

I don't believe this has anything to do with jobby. Jobby doesn't do any timezone-related logic. I believe that this means that it simply uses the timezone of the server (i.e. the time that you see when you run date in the shell, not the timezone of PHP).
Usually, this is okay since the times created by your app runs on the same server and thus uses the same timezone. If your app user's enter local time in a different timezone, and then you have a jobby job operating on those times in EST, then it's up to the app to make that translation.

I am not sure what is the reason, but it doesnt run at the expected time now.
And, I do believe Jobby should obey PHP timezone, since it is a PHP script...

Commenting since this is still open. A good reason for this happening could be that timezone is not set in php.ini. Set it in php.ini restart sever if necessary, and see if you still get the same error.

Commenting since this is still open. A good reason for this happening could be that timezone is not set in php.ini. Set it in php.ini restart sever if necessary, and see if you still get the same error.

Does this work?

If i set timezone in php.ini , the timezone is right, but i set it in script , i still show the EST timezone


ERROR Error: Uncaught (in promise): [object Event]

Im using this plugin angular 5 and I encounter this error every time when I take screenshot. Please suggest how to handle in Typescript not in Jquery all.
Thanks

I run into this same issue. I have never been able to fix it.

I've implemented html2canvas on a Angular 6 application and it's working as expected, with the following code:
`import * as html2canvas from 'html2canvas';
const element = document.getElementById('filters');
html2canvas(element).then((canvas) => {
const imgData = canvas.toDataURL('image/png');
const link = document.createElement('a');
link.download = 'my-image-name.jpeg';
link.href = imgData; link.click();
});`
Since the element 'filters' exists, there's no problem. Could u try it or provide some example of your code?

let element = document.querySelector(".some-area");
        html2canvas(element, { logging: false, useCORS: true }).then(canvas => {
          var imgData = canvas.toDataURL("image/png");
          //Error comes after above statement I think so.
        });

But I have tried with getElementByClassName(), Its throws error. How its working for you?

     html2canvas(document.getElementById('image-area'), {
         useCORS: true,
         logging: false,
         scale: 2.5
        }).then(canvas => {

Is it working with document.getElementById ??

Getting by class name doesn't work at all. Getting by Id works but throws that error and can stall anywhere between 1 and 10 seconds.

Im using querySelector as per documentation. But showing error.

querySelector  still throws the same error for me. What version of angular are you running. My guess it is something to do with our version.

Im using AngularCLI 1.6.6

You find any way to fix this?s

I have this also sometimes, not always. I'm not sure but I think this is linked to the fact that it happens when for that session, a browser pop-up is used to (in my application) login to firebase (using firebase authentication)

I also have the same issue. I am using HTML2CANVAS and I see Error: Uncaught (in promise): [object Event]. I am using Ionic 3 and tested on Chrome, Mozilla Firefox and Internet Explorer.
Could you please help.

Try to run outside angular zone:
this.zone.runOutsideAngular(() => {
//your code
})

In our situation, the plugin works fine until we add an external script (google tag manager) which is async. Using this.ngZone.runOutsideAngular does not help.

Confirming this issue. For me, it was a Profitwell script that interfered. The Zendesk widget also interferes in the sense that it leads to multiple renderings but the end result is still OK. The solution for me was to add some logic to the Profitwell script tag that caused it not to be run on the route where html2canvas is being used. I wonder though whether a configurable timeout that gave these async scripts time to load might be a solution?

I've found that in my case the error was coming from iframeLoader defined in https://github.com/niklasvh/html2canvas/blob/master/src/Clone.js at around line 572.
I could solve the issue by changing the code to return Promise.resolve(cloneIframeContainer);
I'm pretty sure this fix breaks a lots of other uses cases, but for what I needed it solved the issue.

@adamszeibert Can you push your solution?

@hubaozhang I've found a real solution. Here's the PR: https://github.com/niklasvh/html2canvas/pull/1681

Hello, i am using html2canvcas in my react v16 project. but throws error
below is my code,
componentWillReceiveProps(nextProps,nextState){
if (this.props.imgFlag !== nextProps.imgFlag) {
if (nextProps.imgFlag === true) {
let myelement = $("#responsiveReactGridLayout");
html2canvas(myelement, {
background: '#ffffff',
onrendered: function(canvas){
        var imgData = canvas.toDataURL('image/jpeg');

      }
    })
  }
}

}
can anyone suggest what could be done?
thanks  !

html2canvas(document.getElementById('print-section')
, { logging: false, useCORS: true, scale: 2.5 }).then(canvas =>
{
var imgWidth = 208;
var imgHeight = canvas.height * imgWidth / canvas.width;
  const contentDataURL = canvas.toDataURL('image/png');
  let pdf = new jspdf('p', 'mm', 'a4'); // A4 size page of PDF
  var position = 0;
  pdf.addImage(contentDataURL, 'PNG', 0, position, imgWidth, imgHeight);
  pdf.save('receipt.pdf');
});

In my case, the error was being thrown because they were foreign iframes in my website (mainly for tracking purposes). html2canvas will try to clone these iframes and fail due to CORS restrictions.
This quick fix worked for me:
html2canvas(element, {
    ignoreElements: function (element) {
        return element.nodeName === 'IFRAME';
    }
}).then(function (canvas) {
    ...
});

@mumpo
yes, thank you very much! This solved it for me as well.

@mumpo That worked for me as well - THANK YOU!

Im using this plugin angular 5 and I encounter this error every time when I take screenshot. Please suggest how to handle in Typescript not in Jquery all.
Thanks

I have a same problem. Have you solved it？

I have a same problem. Who has a solution?

This should be fixed in v1.1.1. If you are still experiencing this, could you please make a repro of it


run GO jenkins checks on Canto

No great urgency, but some time
Original comment by: ValWood

This isn't urgent since I can just fix the errorrs via the jenkins checks. I think most violations are prevented. I can open tickets for specific checks if they become annoying ...

Putting this as future, because I don't think it will improve the speed of the checking or the fixing much. I do them ~ monthy


fix(deps): upgrade typescript to v4

This also bump jest and ts-jest dependencies

Hi.
When does this get merged?
Node 16 + Gatsby + gatsby-plugin-local-search = [DEP0148] DeprecationWarning:
Use of deprecated folder mapping "./" in the "exports" field module
resolution of the package at .../node_modules/gatsby-plugin-local-search/node_modules/tslib/package.json.
Update this package.json to use a subpath pattern like "./*".
Many thanks


Replace Boxes with Arc in the Expr enum.

Is your feature request related to a problem or challenge?
No response
Describe the solution you'd like
According to following stackoverflow discussion. Boxs can deep copy when called with .clone() method (according to .clone() implementation of the underlying type.).
For Box<Expr> this is the case. I think this usage might be the reason of some deep stack usages seen during the planning.
See related issues: #9375, #8837.
I think, replacing Box<Expr> usages with Arc<Expr> under the enum Expr would improve performance. I am not familiar with the implications of these two approaches in other places. I wonder what community thinks about this change. Would it be better, unnecessary, etc?
Describe alternatives you've considered
No response
Additional context
No response

Is it possible to remove Box to lower the cost?

~Is it possible to remove Box to lower the cost?~ It seems we need to deal with recursive types
Summary for my note:

whether Box<T> does shallow copy or deep copy depends on T, not always deep copy.
Rc<T> / Arc<T> has overhead so whether they are faster than Box<T> may need benchmarks to ensure that.

Exactly. I also think that there are pros and cons of these approaches. For recursive types, I think deep clone problem is more evident.

It would be an interesting experiment,
@jayzhan211 please elaborate how Box does shallow copy? I checked
https://github.com/rust-lang/rust/blob/3cbb93223f33024db464a4df27a13c7cce870173/library/alloc/src/boxed.rs#L1306
It has its own .clone for &str and [] but I didnt see how the same type can be switched between shallow and deep copy.

It would be an interesting experiment, @jayzhan211 please elaborate how Box does shallow copy? I checked https://github.com/rust-lang/rust/blob/3cbb93223f33024db464a4df27a13c7cce870173/library/alloc/src/boxed.rs#L1306
It has its own .clone for &str and [] but I didnt see how the same type can be switched between shallow and deep copy.

I don't know how to check if the types are shallow copy too.
I only know that Box::clone is doing T::clone, see write_clone_into_raw, it does self.clone() which is ``T.clone()`
Based on https://stackoverflow.com/questions/31012923/what-is-the-difference-between-copy-and-clone. and many random comments. Clone do either shallow copy or deep copy.
#[cfg(not(no_global_oom_handling))]
/// Specialize clones into pre-allocated, uninitialized memory.
/// Used by `Box::clone` and `Rc`/`Arc::make_mut`.
pub(crate) trait WriteCloneIntoRaw: Sized {
    unsafe fn write_clone_into_raw(&self, target: *mut Self);
}

#[cfg(not(no_global_oom_handling))]
impl<T: Clone> WriteCloneIntoRaw for T {
    #[inline]
    default unsafe fn write_clone_into_raw(&self, target: *mut Self) {
        // Having allocated *first* may allow the optimizer to create
        // the cloned value in-place, skipping the local and move.
        unsafe { target.write(self.clone()) };
    }
}

#[cfg(not(no_global_oom_handling))]
impl<T: Copy> WriteCloneIntoRaw for T {
    #[inline]
    unsafe fn write_clone_into_raw(&self, target: *mut Self) {
        // We can always copy in-place, without ever involving a local value.
        unsafe { target.copy_from_nonoverlapping(self, 1) };
    }
}

I think, replacing Box usages with Arc under the enum Expr would improve performance.

I basically agree with this assessment but I have an alternate proposal for how to improve performance

I agree that the current clone() of Exprs (and Box::clone does a deep copy)
I believe that the cloneing of Exprs takes a significant amount of planning time in DataFusion (I was looking at cargo bench --bench sql_planner the other day and substantial amounts of time are spent cloning
I think the "ideal" rust pattern for rewriting Exprs is what we have in our ExprRewriters -- take an owned Expr and then destructure / rewrite it as needed
There are many places in our code where Expr::clone() is called Unecessirly
If we changed Expr to use Arc instead of Box I think we would get a planning speedup, but it would be less performant than the ideal pattern (as rewritten nodes would likely rewrite copying, much like LogicalPlan), and it would be a large API change for downstream consumers

Here is an example of what I think is a good pattern (there are no copies except when needed)
https://github.com/apache/arrow-datafusion/blob/9d0c05b9c2e5f9e774ab1ea08599ac8d4b23c93f/datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs#L638-L648
Here is an example of where Expr cloning is being used unnecessarily
https://github.com/apache/arrow-datafusion/blob/9d0c05b9c2e5f9e774ab1ea08599ac8d4b23c93f/datafusion/optimizer/src/simplify_expressions/inlist_simplifier.rs#L45-L49
Thus my suggestion is to go through the planner and remove the calls to Expr::clone() as much as possible (likely letting cargo bench --bench sql_planner be our guide.

Thanks @alamb for your answer. I also think that removing existing unnecessary .clones, replacing them with owned variants is a better approach with keeping the type as is. If you have some preliminary findings such as "Rule x has a lot of .clone with lots of overhead." We can prioritize those sections.

I did some profiling locally by running the following
cargo bench --bench sql_planner -- physical_plan_tpch_all

My analysis is that almost 40% of the planning time is spent in SimplifyExprs and CommonSubexprEliminate
I suspect there are many ways we could reduce clones in those passes

I'm interested in optimizing these (avoid clones), btw What is this profiling tool?

I'm interested in optimizing these (avoid clones),

❤

btw What is this profiling tool?

I used Instruments (CPU profiler) that comes as part of XCode on Mac OSX
I have also used hotspot for Linux https://github.com/KDAB/hotspot which has similar capabilities
Maybe I should make a video about "how to profile / interpret stack traces to optimize DataFusion" 🤔

BTW I think https://github.com/apache/arrow-datafusion/issues/9140 would be a good first start as the inlist simplifier both uses clones as well as does (yet another) tree walk
Maybe we could port it over into the main ExprSimplifer loop in a few PRs 🤔

Maybe I should make a video about "how to profile / interpret stack traces to optimize DataFusion" 🤔

Would be great: video or screens, whatever works and we can attach it to DF docs

Would be great: video or screens, whatever works and we can attach it to DF docs

I will try and do this over the next week or so

I thought about this challenge last night and wrote up my thoughts here: https://github.com/apache/arrow-datafusion/issues/9637

Would be great: video or screens, whatever works and we can attach it to DF docs

I will try and do this over the next week or so

@jayzhan211  and @comphead  here is a video showing what I do to profile datafusion: https://youtu.be/P3dXH61Kr5U -- do you think it is worthwhile adding to the docs?

Would be great: video or screens, whatever works and we can attach it to DF docs

I will try and do this over the next week or so

@jayzhan211 and @comphead here is a video showing what I do to profile datafusion: https://youtu.be/P3dXH61Kr5U -- do you think it is worthwhile adding to the docs?

I'd say its great, thanks @alamb, the font not always clear though, but the video gives the understanding what should be happening.
Today/tomorrow I'm planning to add a profiling doc for MacOS only, how to do a profiling and build flamegraphs. Unix and Window related contributors can add their part later

https://github.com/apache/arrow-datafusion/pull/9711

Would be great: video or screens, whatever works and we can attach it to DF docs

I will try and do this over the next week or so

@jayzhan211 and @comphead here is a video showing what I do to profile datafusion: https://youtu.be/P3dXH61Kr5U -- do you think it is worthwhile adding to the docs?

@alamb Thanks for your video, I think it is really helpful.


There's no way to mark items as shoddy

We can add shoddy as a tag and build out automation from there.


Host not showing on siahub

Hi,
my Sia host is not showing up on siahub.info. However, it is connected to the network, receives contracts, etc.
I have changed my node's DNS name long ago (was jytou.synology.me, is now me.jytou.fr, note that both DNS entries point to the same IP address, and the first is no longer registered). I have reannounced my host on the network just in case but I was receiving contracts so it was working correctly. It was still showing on siahub not so long ago.
I'm running the latest version of Sia UI 1.3.7.
Anything that could help me understand what is going on or if it is a bug on your side?
Thanks,
JY.

It looks like, after reannouncing my host yesterday, it does show up in siahub now. Not sure what went wrong before.


NOOS-517/hofix-buildx-cmd

Description

Ensure local workflow is identical for commands buildx and build+push

JIRA Reference: NOOS-517

LGTM


api: Update, officially loosen, or make optional numpy dependency

Overview
The version of numpy that the opentrons package is pinned to via api/Pipfile is quite old: 1.15.1, released Aug. 21, 2018. Since that version, numpy has bumped to 1.21.2, and Python 3.8 and 3.9 have come into standard usage (with 3.10 on the horizon).
We have users of the opentrons package running on Python 3.9. With numpy pinned to 1.15.1, those users must have a working C compiler to install such an old version of numpy, because there are only prebuilt binaries for Python 3.7 in PyPI.
Acceptance criteria

[ ] Update numpy to a more modern version with precompiled wheels for newer versions of Python, and/or
[ ] Loosen the version spec of numpy to 1.x.x or similar, and/or
[ ] Do both, and/or
[ ] Make numpy optional

Not sure if this is technically feasible, but it might be helpful

Of those options, I think (1) or (3) is most acceptable
Proposal

Update numpy spec to latest (or a recent version that has builds available for Python 3.8, 3.9, and maybe 3.10)
Keep numpy spec loose but constrained by major in setup.py; 1.x.x

Closing due do canonization of setup.py as our official dependency source of truth via #8548


Consider uploading to pypi

There are at least 2 dependent projects now:

Intel's FastRAG which vendors the repo
llama_index: https://github.com/jerryjliu/llama_index/pull/6818

It would be nice if users could pip install colbert-ir instead of having to install from source.

Agreed. We already have a working setup.py so this sounds not too hard

Note that colbert name is already taken. https://pypi.org/project/Colbert/
Another requirement is to be able to do pip install colbert[cpu]

Hello! Is there a timeline for this work by any chance?

@krypticmouse @hmoazam @hichewss are on it, among other work they're doing, some roadblocks due to faiss but getting close

Agreed. We already have a working setup.py so this sounds not too hard

I believe that #268 indicates that the setup.py is not working fully. It is not exposing the .cpp source files, if I'm correct.


Improve InferenceService status

Is your feature request related to a problem? If so, please describe.
https://github.com/kserve/modelmesh-serving/blob/main/docs/predictors/inferenceservice-cr.md

conditions - Various condition entries. Pertinent entries are:
PredictorReady: predictor readiness condition. Status is true when the predictor's endpoints are ready to serve inferencing requests. Note that this does not necessarily mean requests will respond immediately, the corresponding model may or may not be loaded in memory. In the case that it isn't there may be some delay before the response comes back.
Ready: aggregated condition of all conditions.

The “Ready” status is confusing. The model can be callable when the PredictorReady is False. What are the conditions here? What does PredictorReady mean? Is Predictor CR still used or is it just a legacy way of calling things? It would be nice if there is a way distinguish if the model has some non-retriable error like the wrong file path, or if it’s just temporarily not loaded due to lack of usage and can be available when called.

Describe your proposed solution
Talking with @yuzisun, he suggests that

currently it is really reflecting the “model deployment status”  as your active model is still callable but the target model deployment may fail.

Rename the current Status to "model deployment status", and have something to indicate the current model's status. Right now I think it's just the activeModelState.

Describe alternatives you have considered
Since

Upon creation, the active model status of an InferenceService will always transition to Loaded state (unless the loading fails), but later if unused, it is possible that the active model status ends up in a Standby state which means the model is still available to serve requests but the first request could incur a loading delay.

Use activeModelState in combination with lastFailureInfo, if a model has been loaded before, it shouldn't have any lastFailureInfo.

Additional context

Moving to kserve repo as this is a general issue for both kserve and modelmesh

@tessapham Can you help attach your proposal here?


User-facing doc for strategic merge patch

Maybe we need a user-facing doc for strategic merge patch since they are considered as part of the API as well.
Right now we only have a developer-facing doc at https://github.com/kubernetes/community/blob/master/contributors/devel/strategic-merge-patch.md.
Maybe we can try to move it to the user-facing doc or write a separate doc.
We need to figure out a good place for it.
/assign @steveperry-53

I'm working on a new Task topic that will cover kubectl edit and kubectl patch:
PR 4519

Can this be closed @mengqiy or @steveperry-53

@erictune, @mengqiy, I'm fine with closing this issue.
https://kubernetes.io/docs/tasks/run-application/update-api-object-kubectl-patch/


SR700 firmware bug when current_temp == 250degF

The SR700 sends a malformed packet whenever current_temp is 250 degrees F (0xFA in hex).  This causes issues in this module's packet decoding software in the comm process.
With the device's standard software, this FW protocol bug would not manifest much, as the 250F temperature mark is quickly traversed as the machine heads to settle to the L, M or H target temperatures (all way above 250F).  However, with freshroastsr700's thermostat software PID feature, this defect tends to manifest more frequently.
This was discovered as part of rewriting the protocol parser in the main communications loop.
Here's a log showing the defective packets.  The software that produced this readout is a work in progress and a proposed workaround for the FW bug will follow this issue report.
Please connect your roaster...
--------------------------------------------
Setting next process step: 0
time:60, target: 300degF, fan: 9, state: roasting
--------------------------------------------
WARNING:root:read packet data len not 10, got: 22
WARNING:root:RD:<PHONE_NUMBER>09060000faaaaa617400040209060000fb

WARNING:root:RD raw: aaaa617400040209060000faaaaa617400040209060000fbaafa
WARNING:root:RD rpkt:<PHONE_NUMBER>09060000faaaaa617400040209060000fb

WARNING:root:read packet data len not 10, got: 22
WARNING:root:RD:<PHONE_NUMBER>09060300faaaaa617400040209060300f8

WARNING:root:RD raw: aaaa617400040209060300faaaaa617400040209060300f8aafa
WARNING:root:RD rpkt:<PHONE_NUMBER>09060300faaaaa617400040209060300f8

WARNING:root:read packet data len not 10, got: 22
WARNING:root:RD:<PHONE_NUMBER>09030000faaaaa617400040209030300f7

WARNING:root:RD raw: aaaa617400040209030000faaaaa617400040209030300f7aafa
WARNING:root:RD rpkt:<PHONE_NUMBER>09030000faaaaa617400040209030300f7
--------------------------------------------
Setting next process step: 1
time:60, target: 350degF, fan: 5, state: roasting
--------------------------------------------
--------------------------------------------
Setting next process step: 2
time:60, target: 400degF, fan: 5, state: roasting
--------------------------------------------
--------------------------------------------
Setting next process step: 3
time:60, target: 450degF, fan: 4, state: roasting
--------------------------------------------
--------------------------------------------
Setting next process step: 4
time:30, target: 150degF, fan: 9, state: cooling
--------------------------------------------
--------------------------------------------
Setting next process step: 5
time:1, target: 150degF, fan: 1, state: idle
--------------------------------------------

Addressed in pull request as of v0.2.0dev8.


Support material fraction in nrb format

It looks like there is a new xcat nrb format that the current version of gecatsim is unable to parse. From what I can tell, now xcat can simulate one surface being multiple different materials with different fractions whereas the current parsing only lets you parse 1 material per surface(assuming im reading the code correctly).
Relevant parsing code: https://github.com/xcist/main/blob/74309d1b216eff0073d9602c9ce6c1ac7ebd1a2c/gecatsim/clib_build/src/nCAT_main.c#L3332-L3338
Relevant data from nrb files generated from xcat.
musc212
1
1.0
20 :M
33 :N
U Knot Vector
0.000000
0.000000
0.000000
0.000000
...
lrib4
0,4
0.500000,0.500000
45 :M
23 :N
U Knot Vector
0.000000
0.000000
0.000000
0.000000
0.050000
0.100000
0.150000

I didn't want to try to incorporate this into the model since I'm not fully familiar with how to do so, and it would probably require a new projector as well(which I'm not certain the best way to implement it), so i figured an issue is best here.

Hi purepani,
The combination of multiple materials with different fractions can be defined as one material.
Then, you need to hack Phantom_NCAT.py --> set_material, and add that user-defined material.
Last, in the XCAT phantom definition, you set the material index.
Maybe we can add a cfg parameter to allow the user to add new materials.

Would it be possible to have the projector take into account the material fractions so that new materials wouldn't need to be added and the phantom definition wouldn't need to be edited?

Would it be possible to have the projector take into account the material fractions so that new materials wouldn't need to be added and the phantom definition wouldn't need to be edited?

It requires updating the projector C source code, which is a bit larger effort than changing the python phantom code.

That would be great!
The C code is nCAT_main.c.
You'll need to change how it reads the XCAT data file, and how it initializes the Mu table for the objects.

I found an "output old ct format" in the xcat parameter files which creates a file format that the program is expecting. How would you want to manage these 2 formats? Automatically detect which format it is? or just have 2 separate functions for each version? or something else?


fix #364

To fix #364 I decided to go two ways:

Add a message in the GUI so it's clear/visible that we are waiting for the list to be displayed.
Speed up the listing via an improved query, this means that I created a new file db.util where we should store all these improvements from now on. To better reproduce the issue I added a few pools (total 94). With these, the original method takes 7.57 s ± 44.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each), while the new added method takes 134 ms ± 1.6 ms per loop (mean ± std. dev. of 7 runs, 10 loops each). Note that there is still room for performance improvement but that will take multiple changes; explained as a comment in the method so based on the time it will take to make these changes and the current performance improvement, I decided to not solve and not add an issue. Once we get to 1000s of pools we can address.

Assigned 3 reviewers but 1-2 should be fine; just not sure about availability.

This seems like a strategy we might need to employ in general to get back more performance.

Will merge once Travis build completes successfully.


Refactor StdNames

Codecov Report

Merging #184 (1cdc16d) into master (8470f33) will increase coverage by 0.24%.
The diff coverage is 97.14%.

:exclamation: Current head 1cdc16d differs from pull request most recent head 729748c. Consider uploading reports for the commit 729748c to get more accurate results

@@            Coverage Diff             @@
##           master     #184      +/-   ##
==========================================
+ Coverage   91.61%   91.86%   +0.24%
==========================================
  Files          30       30
  Lines         990     1008      +18
==========================================
+ Hits          907      926      +19
+ Misses         83       82       -1

Files Changed
Coverage Δ

src/transforms/stdnames.jl
98.00% <97.14%> (-2.00%)
:arrow_down:

... and 1 file with indirect coverage changes


[NF] GUI exe and installer

Hello all,
This PR introduces a new script to generate an executable to start the GUI and a dedicated installer. The idea is to add a new section on the website to upload an installer after each release.
Best regards,
Pierre

Hello @BonneelP ,
another question: Are there any (recent) plans to extend the GUI beyond machine editing? Or what's the reason for releasing the exec. generator at all?
Best regards, Sebastian

Hello Sebastian,
We are looking to ease the access to pyleecan. With the installer and the new button on the last step (cf #415) now pyleecan can be used as a mesh generator for electrical machine without having to install python or git.
I take the opportunity to share our upcoming work on pyleecan:

we are still working on the equivalent circuit for asynchronous machine (cf dedicated issues) that will reshape the Electrical module
we are planning to add the skew in MagFEMM and in the GUI
we will add a GUI to SciDataTool (that will be available in pyleecan) to interact with the plot without knowing the syntax.

So no simulation GUI for now but if one day someone is interested to work of this topic, the executable will already be available.
Best regards,
Pierre

Hello Pierre,
so there are some things coming together... :-)
I recently thought about releasing a generalized gui (also for simulations) due to the features of #420. (I have also added a simple MeshSolution viewer internally.)

We will add a GUI to SciDataTool (that will be available in pyleecan) to interact with the plot without knowing the syntax.

That's great, so I don't have to care about this, since this would be some of my next things to do. One thing to discuss here may still be left, but I will open a new discussion right after this comment.

We are still working on the equivalent circuit for asynchronous machine (cf dedicated issues) that will reshape the Electrical module.

I thought I already introduced some EEC for IM, but I forgot :-) Do you mean #379 or is there another issue?
Best regards, Sebastian

... haven't tested yet, but is there also the ablility to export gmsh geo files, since meshes with general size assumptions may sometimes miss the intended purpose.

Hello,
Yes most of the pieces are slowly taking shape to provide one day a simulation GUI :) We can also extend #415 to parametrize the exports.
Yes the first EEC was a great help but we need to go further. In particular #372, #413, #418
We still have a lot to experiment so we haven't fully decided what the architecture should be. Other issues should follow.
Regarding gmsh geo, yes that would be interesting. We have added the gmsh buttons to introduce this feature to the users. We know that we can go much further but we lack contributors on the topic :/
Best regards,
Pierre


Carsoncook/gh777/allow url encoded chars by default

Your checklist for this pull request
🚨Please review the guidelines for contributing to this repository.

[x] If the changes in this PR is part of the next future release, make this pull request against the docs-staging branch which will be published at the next release boundary. If the changes in this PR are part of the current release, use the default base branch, master. For more information about branches, see https://github.com/zowe/docs-site/tree/master#understanding-the-doc-branches.

[ ] If this PR relates to GitHub issues in docs-site or other repositories, please list in Description, prefixed with close, fix or resolve keywords.

Description (including links to related git issues)
Adds documentation for new default configuration, allowing encoded characters and slashes by default. Code change is in https://github.com/zowe/api-layer/pull/804

@CarsonCook Hi Carson, just want to check which release this PR is targeted for. Is it for Zowe v1.15?

Hi @nannanli, yes, it is for 1.15

@CarsonCook You might want to just copy the changes from this PR and paste them right into a new branch that's based on master. If you merge this to staging now, it won't be live until 1.16.0


Refactor tests to run individually.

Background
Previously when running yarn test, the test results did not specify
which test actually failed. Now they will specify the exact tests that fail.
When running tests, there is a single test that runs 30+ sub-tests.
If there is a failure, there is no information about which of the 30 tests
that fails.
If we switch to more of a 'native' Jest testing structure, we can see the
individual test results. But due to limitations in Clasp, we can no longer
easily run the testsuite from the DEBUG menu.
Test Refactoring
This change includes modifications to allow both a more 'native' Jest
environment, as well as retaining the ability to run tests in the Sheet.
Example Passing Tests:

Example Failing Tests:

Example Failing Tests in Sheets:

Also note that I had to also use #59 (Fix utils.ts imports) in my own environment in order to get it running in Sheets.
This PR only has my changes though.

Hi @ranmocy ,
Please take a look at this PR. I feel it really adds to the testability of the program. I add a lot of new tests, and the tests continue to be able to be run by both using yarn test as well as from the Sheet Debug menu.

Thanks!


Idiomatic Rust refactoring

I was looking at recovery code trying to find a way to optimize it. In the end I went through a large chunk of the code, replacing manual indexing of vectors with idiomatic iterators and pre-allocating vectors with known size to avoid re-allocations.
End result isn't impressive, from my testing improvement is 1-5%, benchmark is very noisy, I blame it on heterogeneous cores of my 13900K processor. But it is faster in general.
One question I have is whether recover_poly_from_samples should be removed from recovery.rs and its changes moved into FsPoly::recover_poly_from_samples (there is some extra parallelization there for instance, but for the most part functions are just duplicates of each other).

One question I have is whether recover_poly_from_samples should be removed from recovery.rs and its changes moved into FsPoly::recover_poly_from_samples (there is some extra parallelization there for instance, but for the most part functions are just duplicates of each other).

Since recover_poly_from_samples is already implemented in the recovery.rs files of all the other implementations, I believe it would be best to leave it as it is.
Details

Since recover_poly_from_samples is already implemented in the recovery.rs files of all the other implementations, I believe it would be best to leave it as it is.

Maybe it needs to be removed from everywhere then? PolyRecover defines that every implementation must have it regardless, recover.rs seems redundant in that case.

BTW, I think this PR is complete, I will probably submit other changes in separate PRs


Broombridge schema for two-body integrals doesn't match spec?

Under the Two-Electron Integrals Object, the input is frequently said to come in the form [i, j, k, l], which is then used throughout formulas and symmetries.
However, this disagrees with the schema outlined in the spec sheet (see page 20, listing 9, code line 21), which says that the format should operate in [i, l, j, k]. Am I misreading something here? (E.g., are we already using the Mulliken convention here)

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 6747132f-77dc-58cc-b521-8838f9eaef3b
Version Independent ID: f9cb66f1-7677-f488-71cb-f9721d66067a
Content: Broombridge Schema Specification (ver 0.2) - Microsoft Quantum
Content Source: articles/user-guide/libraries/chemistry/schema/spec_v_0_2.md
Product: quantum
GitHub Login: @guanghaolow
Microsoft Alias<EMAIL_ADDRESS>

It seems that we still need to add a couple of lines clarifying the use of Dirac/Mulliken indexing and the fact that the deserializer does the conversion between them. I will take care of that.


#168 Fix layout for pool settings

Summary
Small fix for the settings layout on the /pools/{address}/manage page.
Testing Instructions

Connect your wallet.
Click the link node-runners in the header.
Click manage icon in a pool. PS: Create one if necessary.
Scroll down to the settings area.
You should be able to see the misaligned fields according to the screenshot.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>6
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 77.511%

Totals

Change from base Build<PHONE_NUMBER>7:
0.0%

Covered Lines:
338

Relevant Lines:
401

💛  - Coveralls


Component: Title

Describe the bug
p:menubar (p:menu, p:meneButton as well) is not accessable. That is, when tabbing through the menu using a screen reader like JAWS, it does not read any of the menu items, even all the generated  elements have titles or aria-lables.
Reproducer
See here: https://www.primefaces.org/showcase/ui/menu/menubar.xhtml
To Reproduce
Open a screen reader (JAWS for example), tab through "File" "Edit" menus and submenus, the screen reader will read nothing.
Expected behavior
It would be nice to have an unified focus and screen reader interaction behaviour, like p:tabMenu, which works perfectly.
Sample XHTML
It seems that the focus is always landed on the surrounding elements like , or <li … />, while the accessable values are inside the inner  elements. However, simply adding a title or aria-label to <li … /> does not work either.

@zhiying99 why did you open an issue at PrimeFaces Extensions and not at https://github.com/primefaces/primefaces/issues ?

sorry, an error

@zhiying99 yes, and please set a title on the new issue

ok, thank you!


Proposal of improving the navigation bar on the left

One problem I found was that every time I clicked on chapter 33 in the left navigation bar, the left navigation bar would refresh to chapter 1 at the top. If I wanted to go to chapter 34, I would have to scroll down again to find chapter 34.
Can we change it so that each time we click the corresponding section of the directory navigation bar, the navigation bar position remains unchanged on the next refresh.

Searched and found out the same issue at https://github.com/mkdocs/mkdocs/issues/803


Updating HotRockets to 0.90 and adding Community Configs

closes https://github.com/KSP-CKAN/CKAN-meta/issues/5
Community Configs are only 0.25 compliant in accordance with curseforge, they probably work with 0.90 but right now I don't have the motivation to confirm this by reading through the whole HotRockets thread on forums so they're stuck at 0.25 for now.
Do note! I remembered newlines at the end of BOTH files \o/

#rebuild

Well that is interesting... build fails because Jenkins is running 0.90 and the CommunityConfigs are for 0.25. I guess this will have to be manually reviewed

Mayby just use ksp_version_min and ksp_version_max? :)

I could but it won't make a difference since the configs are only available for HotRockets 7.25 according to http://kerbal.curseforge.com/ksp-mods/220207-hotrockets-particle-fx-replacement/files and as such Jenkins will constantly fail since he runs 0.90.
I could overrun the 0.25 thing if someone could confirm the configs do work in 0.90 with HotRockets 7.9.

Configs confirmed for 0.90 by author here


[Question]: [Question]: After asking a question once, an error was reported the second time. What is the reason?

What is your question?
An error will be reported the second time you ask, and you can only create a new chat.

More Details
LibreChat         | 2024-01-30 07:27:21 error: There was an issue generating title with BingAI CaptchaChallenge: User needs to solve CAPTCHA to continue.
chat-meilisearch  | [2024-01-30T07:27:21Z INFO  actix_web::middleware::logger] <IP_ADDRESS> "GET /indexes/convos/documents/b8d859e0-ed58-4267-ad65-78819def81ad HTTP/1.1" 200 76 "-" "node" 0.000451
chat-meilisearch  | [2024-01-30T07:27:35Z INFO  actix_web::middleware::logger] <IP_ADDRESS> "GET /indexes/convos/documents/b8d859e0-ed58-4267-ad65-78819def81ad HTTP/1.1" 200 76 "-" "node" 0.000270
chat-meilisearch  | [2024-01-30T07:27:35Z INFO  actix_web::middleware::logger] <IP_ADDRESS> "POST /indexes/messages/documents HTTP/1.1" 202 136 "-" "node" 0.007627
chat-meilisearch  | [2024-01-30T07:27:35Z INFO  index_scheduler::batch] document addition done: DocumentAdditionResult { indexed_documents: 1, number_of_documents: 19 }
chat-meilisearch  | [2024-01-30T07:27:35Z INFO  index_scheduler] A batch of tasks was successfully completed.
LibreChat         | 2024-01-30 07:27:37 error: [/ask/bingAI] Error handling BingAI response Unexpected server response: 200
LibreChat         | 2024-01-30 07:27:37 error: [/ask/bingAI] Error handling BingAI response Unexpected server response: 200
chat-meilisearch  | [2024-01-30T07:27:37Z INFO  actix_web::middleware::logger] <IP_ADDRESS> "POST /indexes/messages/documents HTTP/1.1" 202 137 "-" "node" 0.007244
chat-meilisearch  | [2024-01-30T07:27:37Z INFO  index_scheduler::batch] document addition done: DocumentAdditionResult { indexed_documents: 1, number_of_documents: 20 }
chat-meilisearch  | [2024-01-30T07:27:37Z INFO  index_scheduler] A batch of tasks was successfully completed.
What is the main subject of your question?
No response
Screenshots
No response
Code of Conduct

[ ] I agree to follow this project's Code of Conduct

BingAI issues can be due to fail safes Microsoft introduces or the way they treat specific network connections.
In your case, you need to solve a CAPTCHA as shown in your logs:

User needs to solve CAPTCHA to continue.

We have some common solutions listed on our discord you can try: https://discord.com/channels/1086345563026489514/1143941308684177429


[INLONG-7508][Sort] Carry right RowKind when cdc-base sends RowData to sink

Prepare a Pull Request
(Change the title refer to the following example)

Title Example: [INLONG-XYZ][Component] Title of the pull request

(The following XYZ should be replaced by the actual GitHub Issue number)

Fixes #7508

Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve?
cdc-base sends RowData to sink, should carry right RowKind

Modifications
Describe the modifications you've done.
AppendMetadataCollector make a new GenericRowData instance, should pass right RowKind.
Verifying this change
(Please pick either of the following options)

[ ] This change is a trivial rework/code cleanup without any test coverage.

[ ] This change is already covered by existing tests, such as:
(please describe tests)

[ ] This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

Now sink parse rowkind from data when all database migration. So rowkind use INSERT.

Source database has a update operation,  cdc send update type in json data.  Sink connector cannot known whether is Update_Before or Update_After.
The 3rd screenshot shows that sink connector always get update before node null from json data.

When source table has a update operation, cdc send update type in json data. Sink connector cannot known whether is Update_Before or Update_After.
The 3rd screenshot shows that sink connector always get updateBeforeNode null from json data.

@liaorui @EMsnap have a PR issue this problem. https://github.com/apache/inlong/issues/7397

The opType in allmigrate is represented as the parameter TYPE in canal json. maybe you can get the optype from there

Close this PR since PR #7397 has solved this problem.


[SPARK-20602] [ML]Adding LBFGS optimizer and Squared_hinge loss for LinearSVC

What changes were proposed in this pull request?
jira: https://issues.apache.org/jira/browse/SPARK-20602
Currently LinearSVC in Spark only supports OWLQN as the optimizer ( check https://issues.apache.org/jira/browse/SPARK-14709). I made comparison between LBFGS and OWLQN on several public dataset and found LBFGS converges much faster for LinearSVC in most cases.
The following table presents the number of training iterations and f1 score of both optimizers until convergence

data source: https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html
training code: new LinearSVC().setMaxIter(10000).setTol(1e-6) with master=local[1]
LBFGS requires far less iterations in most cases and probably is a better default optimizer.
How was this patch tested?
strengthen existing unit tests

Test build #76463 has started for PR 17862 at commit d46e5ed.

ping @jkbradley Sorry I know this is like the last minute for 2.2, but the change may be important for user experience. If we're not comfortable making API change right now, how about we just change the default optimizer to LBFGS before 2.2 releases?

Test build #76463 has finished for PR 17862 at commit d46e5ed.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76463/
Test FAILed.

Update: switch to HasSolver trait and use OWLQN as default optimizer

Test build #76508 has started for PR 17862 at commit f7d5559.

Test build #76508 has finished for PR 17862 at commit f7d5559.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76508/
Test PASSed.

@hhbyyh can we smooth the hinge-loss using soft-max (variant of ReLU) and then use LBFGS ?

@debasish83 There're several approaches trying to smooth the hinge loss. https://en.wikipedia.org/wiki/Hinge_loss. For the one you're proposing, do you know if it's used in other SVM implementations? Please help provide some reference or evaluation data if possible. Thanks.

Test build #76637 has started for PR 17862 at commit 8a7c10f.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76637/
Test FAILed.

Merged build finished. Test FAILed.

Test build #78002 has started for PR 17862 at commit 3707580.

Test build #78002 has finished for PR 17862 at commit 3707580.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78002/
Test FAILed.

Merged build finished. Test FAILed.

Test build #78019 has started for PR 17862 at commit 2ca5a74.

Test build #78019 has finished for PR 17862 at commit 2ca5a74.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78019/
Test PASSed.

Merge the change from https://github.com/apache/spark/pull/17645 into a single change.

Test build #78024 has started for PR 17862 at commit 5f7f456.

@hhbyyh Thanks for doing the extra work to use the new aggregator here. I do think it's better to separate those changes from this one, though. There is actually more that needs to be done for the conversion (need to use RDDLossFunction and also add a test suite for the aggregator). Would you mind submitting a PR for just the conversion changes?

Test build #78024 has finished for PR 17862 at commit 5f7f456.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78024/
Test PASSed.

Sure. That's reasonable. I'll move the hingeAggregator to a new PR.

Test build #78119 has started for PR 17862 at commit 0297057.

Test build #78119 has finished for PR 17862 at commit 0297057.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78119/
Test PASSed.

Merged build finished. Test PASSed.

Test build #78731 has started for PR 17862 at commit 7be6bac.

Test build #78731 has finished for PR 17862 at commit 7be6bac.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/78731/
Test PASSed.

Merged build finished. Test PASSed.

On many large dataset, LinearSVC cannot get the similar result with sklearn. e.g., SKLearn may get coefficients (5, 10, 15, 20), and spark LinearSVC will get (10, 20, 30, 40). It's different but in most cases it's proportional scaling. This may partly because the different handling of intercept scaling in sklearn and spark. But on larger dataset, I found LBFGS and OWLQN also presents the similar issue.
I wonder if it's normal to get different (but proportional)  coefficients from different optimization method for the same dataset and parameter settings.

@hhbyyh If different handling of intercept scaling is the major cause for result difference between sklearn and Spark, do you check whether fit model without intercept will produce same model? Theoretically, different optimizers should converged to the same result if the function is convex. Thanks.

@yanboliang Without intercept, sklearn and Spark LinearSVC will get the same coefficients on several dataset I tested.

Test build #79002 has started for PR 17862 at commit aaf35ec.

Test build #79002 has finished for PR 17862 at commit aaf35ec.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79002/
Test PASSed.

@hhbyyh Make sense, does it mean both LBFGS and OWLQN produce the same solution if fitting without intercept? If so, I'm prefer to change the solver to LBFGS rather than adding a new option. Since we treat LBFGS and OWLQN as the same solver in other algorithms like LinearRegression and LogisticRegression.
BTW, we can discuss whether we should follow the same way(compared with sklearn/R) to handle intercept in follow-up work. Thanks.

Yes, Both LBFGS and OWLQN generate similar model with sklearn if without intercept.
About replacing OWLQN with LBFGS, I noticed if using hinge loss, sometimes OWLQN uses fewer iterations than LBFGS as shown in the table in the PR description.  Shall we just discard OWLQN support? Thanks.

I'm in favor of discarding OWLQN. Take LiR or LoR as examples, if you replace LBFGS with OWLQN for regression with L2 regularization, we can saw OWLQN may converge faster than LBFGS in a certain case, but in most of the cases, LBFGS is faster. But I'm still open to hear others' thought. Thanks.

cc @WeichenXu123 What do you think about this?

+1 @WeichenXu123 IIRC softmax regression also include a non-derivable point, we can use LBFGS to solve it as well. We can support squared hinge loss which is smooth function in the future, so users can switch to squared hinge loss if they hit the condition with little probability.

Given the discussion above, I plan to replace OWLQN with LBFGS. I will send update soon.

Catching up here...  To make sure I caught the decisions made in the discussion above, is it correct that this PR will:

Add support for squared hinge loss, and use that as the default (which I fully support)
Switch from OWLQN to LBFGS (which is fine with me if reasonably large-scale tests support that choice)

On this note, has anyone tested on large-scale datasets (ideally with millions of rows and columns)?

If those are the decisions, can you please update the PR description and JIRA as such when you update the PR @hhbyyh ?  Thank you!

+1 @jkbradley for test on large-scale datasets. @hhbyyh Do you have time to test it? If not, I can help. Thanks.

+1 for adding test on large-scale datasets.
Another thing I want to know is that: you can compare the final loss value on the result coefficients, between LIBLINEAR(scikit-learn), LBFGS, OWLQN, so that we can know which optimizer can get more optimized result (testing and comparing on different kind of dataset).

Sure, I can find some larger dataset to test with.
But I guess, as showed in the PR description, LBFGS will generally outperform OWLQS, but not in all the cases. I assume single large scale dataset is not sufficient for the testing, so if you plan to test on some data set, please leave comment here. Thank you.

Test build #81367 has started for PR 17862 at commit cec628b.

Test build #81367 has finished for PR 17862 at commit cec628b.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81367/
Test FAILed.

Tested with several larger data set with Hinge Loss function, to compare l-bfgs and owlqn solvers.

For each iteration, owlqn takes longer time ( 2 or 3 times) than l-bfgs. Logistic Regression also exhibits the similar behavior.

@hhbyyh Test result looks good!
OWLQN takes longer time for each iteration, because each iteration's line search, it made more passes on dataset.

Test build #81836 has started for PR 17862 at commit 0f5cad5.

Test build #81836 has finished for PR 17862 at commit 0f5cad5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/81836/
Test PASSed.

Test build #82376 has started for PR 17862 at commit bf4d955.

Test build #82376 has finished for PR 17862 at commit bf4d955.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82376/
Test FAILed.

Test build #82377 has started for PR 17862 at commit 1f8e984.

Test build #82377 has finished for PR 17862 at commit 1f8e984.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82377/
Test PASSed.

Test build #82766 has started for PR 17862 at commit 0bb5afe.

Test build #82766 has finished for PR 17862 at commit 0bb5afe.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82766/
Test PASSed.

Please let me know if there's any unresolved comments. Thanks.

Thanks @WeichenXu123 for the comments.

Test build #105501 has started for PR 17862 at commit 0bb5afe.

Test build #105501 has finished for PR 17862 at commit 0bb5afe.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/105501/
Test FAILed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/15801/
Test PASSed.

Build finished. Test PASSed.

Test build #112418 has started for PR 17862 at commit 64bc339.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/17388/
Test PASSed.

Test build #112418 has finished for PR 17862 at commit 64bc339.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/112418/
Test FAILed.

@hhbyyh This PR is stale. If there's nobody interested in this and no further updates, would you mind to close it ? Thanks!


[test-triage] rom_e2e_smoke has started failing consistently

Hierarchy of regression failure
Chip Level
Failure Description
Looking at the grafana chart, it looks like this broke some time between 31st March and 2nd April. Running the head of master (ec711524c05e92a306b3e28c0343522909a1b9a7), it eventually times out on my machine. I suspect the problem comes reasonably early because I see the following message in the log:
UVM_INFO @  10.380001 us: (sw_logger_if.sv:161) [tb.u_sim_sram.u_sim_sram_if.u_sw_logger_if.parse_sw_log_file.unnamed$$_0.unnamed$$_1] Loading sw log db file empty_test_slot_a_sim_dv.logs.txt ...

Warning-[STASKW_CO] Cannot open file
../src/lowrisc_dv_sw_logger_if_0/sw_logger_if.sv, 156
  The file 'rom_with_fake_keys_sim_dv.logs.txt' could not be opened. No such
  file or directory.
  Please ensure that the file exists with proper permissions.

I immediately wondered the obvious thing: why doesn't this instantly kill the test? Looking at the surrounding lines in sw_logger_if.sv, I think that @timothytrippel intentionally didn't kill the test (see fd9a3c14c67). I wonder whether this is too permissive: maybe we should have a "no, really, this needs to work" flag.
Steps to Reproduce

GitHub Revision: ec711524c05e92a306b3e28c0343522909a1b9a7
dvsim invocation command to reproduce the failure, inclusive of build and run seeds:util/dvsim/dvsim.py hw/top_earlgrey/dv/chip_sim_cfg.hjson -i rom_e2e_smoke --fixed-seed=123 --tool=vcs

Tests with similar or related failures
No response

Hmm. I've just tried running this with a known-good SHA (4ee21f808f, which was used for the nightly on 28th March. The resulting log file has the same warning, so maybe that wasn't a smoking gun after all.

Looking more closely at a failing log, the failures appear after this message:
Job chip_earlgrey_asic-sim-vcs_run_default killed due to: Exit reason: User job exceeded runlimit: User job timed out has 14 failures:

I suspect this means that we're just running out of time in the tests and getting killed by the runner. This might have nothing to do with this particular test at all. :-(

Hmm, I can't remember why I made that change to continue if the log Db file was not present, perhaps it was because we were not generating one for each SW image at the time? but maybe we are now... that part probably requires more investigation but like you mention I don't think that is the reason `rom_e2e_smoke is failing.
I think the reason this failing is possibly due to a sigverify issue, I think it makes sense to just leave this open for a bit and fix it in a couple weeks. @moidx is currently breaking things as he:

converts sigverify to use ECDSA (instead of RSA), and
moves first stage secure boot keys from ROM to OTP.

I think he is tracking all of this in #22697.

Hi @rswarbrick, I can take this one for now since I have to clean-up all the rom e2e tests after switching to ECDSA. https://github.com/lowRISC/opentitan/issues/21204

@hayleynewton checked this today and found a 100% pass rate again


Reviewed tox.ini

Minor change, to verify everything works as expected after the transfer to codrsquad

Codecov Report

Merging #68 (0a25094) into master (b3a756a) will decrease coverage by 16.11%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##            master      #68       +/-   ##
============================================
- Coverage   100.00%   83.88%   -16.12%
============================================
  Files            7        7
  Lines         1580     1564       -16
============================================
- Hits          1580     1312      -268
- Misses           0      252      +252

Impacted Files
Coverage Δ

setupmeta/commands.py
73.26% <0.00%> (-26.74%)
:arrow_down:

setupmeta/content.py
80.88% <0.00%> (-19.12%)
:arrow_down:

setupmeta/model.py
84.68% <0.00%> (-15.32%)
:arrow_down:

setupmeta/versioning.py
87.43% <0.00%> (-12.57%)
:arrow_down:

setupmeta/hook.py
95.00% <0.00%> (-5.00%)
:arrow_down:

setupmeta/scm.py
100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b3a756a...0a25094. Read the comment docs.

Codecov Report

Merging #68 (0a25094) into master (b3a756a) will decrease coverage by 16.11%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##            master      #68       +/-   ##
============================================
- Coverage   100.00%   83.88%   -16.12%
============================================
  Files            7        7
  Lines         1580     1564       -16
============================================
- Hits          1580     1312      -268
- Misses           0      252      +252

Impacted Files
Coverage Δ

setupmeta/commands.py
73.26% <0.00%> (-26.74%)
:arrow_down:

setupmeta/content.py
80.88% <0.00%> (-19.12%)
:arrow_down:

setupmeta/model.py
84.68% <0.00%> (-15.32%)
:arrow_down:

setupmeta/versioning.py
87.43% <0.00%> (-12.57%)
:arrow_down:

setupmeta/hook.py
95.00% <0.00%> (-5.00%)
:arrow_down:

setupmeta/scm.py
100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b3a756a...0a25094. Read the comment docs.


Option noEmit doesn't work

I was getting this error while trying to build typescript files: error: Compiling of app/drawable.ts failed. Debug Failure. False expression: Output generation failed
Looks like if I put "noEmit": true, in my tsconfig.json, to prevent my error checker from generating any files, brunch-typescript fails to override that.
Even if I put this in my brunch-config.coffee:
plugins:
    brunchTypescript:
        noEmit: false
        removeComments: true
        watch: false

I found a dirty hack to fix this: put this line in transpile.js (after options.noResolve = true; line):
options.noEmit = false;

Hi @hhyyrylainen I've just published a new version (2.1.0) which references TypeScript 2.1.* and fixes this issue.
If you need a fix for this which supports 2.0.* then let me know.

I have now tested the new version and it works great.
Thanks for taking the time to fix this.

I have now tested the new version and it works great.
Thanks for taking the time to fix this.


mu: generate mu4e-autoloads.el

Description of changes

What is autoload?

Basically, it allows you to delay the loading of a package1.

What is a name-autoloads.el file?

When installing a package using package.el, Emacs generates1 a
name-autoloads.el file for that package, which is used to autoload the
principal user commands defined2 in the package.

What problem does this patch solve?

There are some autoload commands defined3 by mu4e.  Before, those
autoload commands cannot be autoloaded because of the missing
mu4e-autoloads.el file since mu4e installed from nixpkgs does not use
package.el.  This patch fixes that.
To sum up, with this patch, you can call (mu4e) without
calling (require 'mu4e) first.

Isn't it better to contribute to the build system of the upstream mu
project to generate that mu4e-autoloads.el file?

It seems impossible4 to do so.

Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.05 Release Notes (or backporting 22.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[X] Fits CONTRIBUTING.md.

Can you write something about why this is useful/necessary in the commit description?

@ofborg eval

@ofborg eval

I will not complain about writing style. The commit message is clear enoughd


ExtractorError: Could not find JS function u'gr'

Version 2015.12.13-gentoo_no_offensive_sites
$ youtube-dl --verbose --audio-format m4a 'https://www.youtube.com/watch?v=pvpIN5A4WRU'
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'--audio-format', u'm4a', u'https://www.youtube.com/watch?v=pvpIN5A4WRU']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.12.13-gentoo_no_offensive_sites
[debug] Python version 2.7.10 -<EMAIL_ADDRESS>[debug] exe versions: ffmpeg 2.8.3, ffprobe 2.8.3, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] pvpIN5A4WRU: Downloading webpage
[youtube] pvpIN5A4WRU: Downloading video info webpage
[youtube] pvpIN5A4WRU: Extracting video information
[youtube] {22} signature length 43.43, html5 player en_US-vflnrstgx
[youtube] pvpIN5A4WRU: Downloading player https://s.ytimg.com/yts/jsbin/player-en_US-vflnrstgx/base.js
ERROR: Signature extraction failed: Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 877, in _decrypt_signature
    video_id, player_url, s
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 792, in _extract_signature_function
    res = self._parse_sig_js(code)
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 855, in _parse_sig_js
    initial_function = jsi.extract_function(funcname)
  File "/usr/lib64/python2.7/site-packages/youtube_dl/jsinterp.py", line 241, in extract_function
    raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
 (caused by ExtractorError(u"Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 877, in _decrypt_signature
    video_id, player_url, s
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 792, in _extract_signature_function
    res = self._parse_sig_js(code)
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 855, in _parse_sig_js
    initial_function = jsi.extract_function(funcname)
  File "/usr/lib64/python2.7/site-packages/youtube_dl/jsinterp.py", line 241, in extract_function
    raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/youtube_dl/YoutubeDL.py", line 663, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/common.py", line 290, in extract
    return self._real_extract(url)
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 1431, in _real_extract
    encrypted_sig, video_id, player_url, age_gate)
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 887, in _decrypt_signature
    'Signature extraction failed: ' + tb, cause=e)
ExtractorError: Signature extraction failed: Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 877, in _decrypt_signature
    video_id, player_url, s
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 792, in _extract_signature_function
    res = self._parse_sig_js(code)
  File "/usr/lib64/python2.7/site-packages/youtube_dl/extractor/youtube.py", line 855, in _parse_sig_js
    initial_function = jsi.extract_function(funcname)
  File "/usr/lib64/python2.7/site-packages/youtube_dl/jsinterp.py", line 241, in extract_function
    raise ExtractorError('Could not find JS function %r' % funcname)
ExtractorError: Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
 (caused by ExtractorError(u"Could not find JS function u'gr'; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.",)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

You are using an outdated version of youtube-dl. Read our FAQ if you have troubles updating.


fix(*): Method.Random[Key] behavior

blocked by https://github.com/josh-development/utilities/pull/226

Looks good other than the issues above. Will have to make sure the tests pass.


Code fix download image

when the request method is "post",it will return the same key for different requests

Can you add tests that demonstrate the problem you’re fixing here?

Thanks for the PR! However, while there may be a caching issue here, appending the fully body data as the key value is not the correct solution. Feel free to submit another PR with a better solution though!