Support SingleStore/MemSQL

SingleStore/MemSQL is "The world's fastest, most capable cloud database for operational analytics, machine learning, & AI" that is compatible with MySQL and their Helios is "A Fully-Managed, On-Demand, & Elastic Cloud Database"

Currently this happens on prisma introspect or prisma migrate save --experimental:
C:\Users\Jan\Documents\throwaway\memsql>prisma introspect
Environment variables loaded from prisma\.env

Introspecting based on datasource defined in prisma\schema.prisma …
  IntrospectionEngine:rpc starting introspection engine with binary<EMAIL_ADDRESS>+0ms
  IntrospectionEngine:rpc SENDING RPC CALL {"id":1,"jsonrpc":"2.0","method":"introspect","params":[{"schema":"// This is your Prisma schema file,\n// learn more about it in the docs: https://pris.ly/d/prisma-schema\n\ndatasource db {\n  provider = \"mysql\"\n  url      = env(\"DATABASE_URL\")\n}\n\ngenerator client {\n  provider = \"prisma-client-js\"\n}\n\nmodel User {\n  id    String  @default(cuid()) @id\n  email String  @unique\n  name  String?\n}"}]} +7ms
  IntrospectionEngine:rpc {
  IntrospectionEngine:rpc   jsonrpc: '2.0',
  IntrospectionEngine:rpc   error: {
  IntrospectionEngine:rpc     code: 4466,
  IntrospectionEngine:rpc     message: 'An error happened. Check the data field for details.',
  IntrospectionEngine:rpc     data: {
  IntrospectionEngine:rpc       is_panic: false,
  IntrospectionEngine:rpc       message: "Error in connector: Error querying the database: Error querying the database: Error querying the database: Server error: `ERROR HY000 (1706): Feature 'Audit logging with binary protocol' is not supported by MemSQL.'",
  IntrospectionEngine:rpc       backtrace: null
  IntrospectionEngine:rpc     }
  IntrospectionEngine:rpc   },
  IntrospectionEngine:rpc   id: 1
  IntrospectionEngine:rpc } +950ms
Error: Error: Error in connector: Error querying the database: Error querying the database: Error querying the database: Server error: `ERROR HY000 (1706): Feature 'Audit logging with binary protocol' is not supported by MemSQL.'

    at IntrospectionEngine.<anonymous> (C:\ProgramData\nvm\v12.6.0\node_modules\@prisma\cli\build\index.js:2:1995969)
    at Generator.next (<anonymous>)
    at C:\ProgramData\nvm\v12.6.0\node_modules\@prisma\cli\build\index.js:2:1991014
    at new Promise (<anonymous>)
    at n (C:\ProgramData\nvm\v12.6.0\node_modules\@prisma\cli\build\index.js:2:1990762)
    at Object.<anonymous> (C:\ProgramData\nvm\v12.6.0\node_modules\@prisma\cli\build\index.js:2:1995175)
    at IntrospectionEngine.handleResponse (C:\ProgramData\nvm\v12.6.0\node_modules\@prisma\cli\build\index.js:2:1993052)
    at LineStream.<anonymous> (C:\ProgramData\nvm\v12.6.0\node_modules\@prisma\cli\build\index.js:2:1994813)
    at LineStream.emit (events.js:203:13)
    at addChunk (_stream_readable.js:294:12)

Seems it does not like prepared statements in the default configuration: https://docs.memsql.com/v7.1/concepts/prepared-statements/

Hey @janpio I was wondering if this has had any updates since last year?
I've recently joined SingleStore and was trying out the latest release of Prisma Migrate. I was able to get a bit further than you but still getting errors from the introspection engine.
braposo@braposo-1:~/Projects/hello-prisma$ npx prisma migrate dev --name init --create-only
Environment variables loaded from .env
Prisma schema loaded from prisma/schema.prisma
Datasource "db": MySQL database "test" at "<IP_ADDRESS>:30141"

Error: Database error: Error accessing result set, column not found: datetime_precision
   0: sql_migration_connector::flavour::mysql::sql_schema_from_migration_history
             at migration-engine/connectors/sql-migration-connector/src/flavour/mysql.rs:270
   1: sql_migration_connector::sql_database_migration_inferrer::calculate_drift
             at migration-engine/connectors/sql-migration-connector/src/sql_database_migration_inferrer.rs:57
   2: migration_core::api::DevDiagnostic
             at migration-engine/core/src/api.rs:97

The columns table in the information_schema database doesn't contain the required datetime_precision column so I was wondering if there's a way to disable that check somehow?
I found this issue from another project talking about a similar problem: https://github.com/go-gorm/mysql/issues/13
Please let me know if there's anything I can do to help getting Prisma to work with SingleStore!

I've also tested with @prisma/client and it's actually working for the reading (find, aggregate, count, etc) actions, but there's some syntax issue for the writing actions (create, update, delete).
 Error occurred during query execution:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(Server(ServerError { code: 1064, message: "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'? LIMIT ? OFFSET ?\' at line 1", state: "42000" })) })
    at cb (/home/braposo/Projects/hello-prisma/node_modules/@prisma/client/runtime/index.js:34778:17)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

Is there a way to see what's the actual SQL query that is being sent? I've tried enabling debugging mode but it doesn't print that.
Otherwise it's actually working really nicely, which is very exciting!

Do you want me to create a separte issue for this or is it fine to talk about this here as well?

The columns table in the information_schema database doesn't contain the required datetime_precision column so I was wondering if there's a way to disable that check somehow?

We would need to adapt our Migration and Introspection Implementation to skip that step for SingleStore - which might not be trivial.
In general 2 things would be helpful:

Is there a Docker image that we can easily integrate into our test suite?
Is there a reliable way to "detect" and identify a SingleStore instance vs. a default MySQL?

Is there a way to see what's the actual SQL query that is being sent? I've tried enabling debugging mode but it doesn't print that.

https://www.prisma.io/docs/concepts/components/prisma-client/debugging and https://www.prisma.io/docs/concepts/components/prisma-client/working-with-prismaclient/logging should give you what you are looking for I think to see the queries that are being executed under the hood and pinpoint the one giving you the error.
Generally SingleStore is not a priority for us, as our customers and users are asking for other things. But I am happy to help you out answering questions and possibly also seeing about getting SingelStore into our CI setup. If we find deeper problems (like the first point about datetime_precision) I might not be able to guarantee Engineering capacity to fix those any time soon - but let's see how far we get until that is the case!

hey @janpio! First of all thank you so much for the quick reply and I'm sorry for not getting back to you earlier but it was hackathon at SingleStore and I couldn't get all the answers for you in time.

Not out of the box (I think this is coming from prisma/prisma-engines@e106258/libs/sql-schema-describer/src/mysql.rs#L228-L252). We would need to adapt our implementation there to skip that step for SingleStore, adapt the query or similar - which might not be trivial.

No problem, I wasn't really expecting a workaround from your side, was more to understand what you were querying for and your answer is very helpful! I'll be working with some people from the engine team to see if I can unblock the remaining integration problems from our side.

In general 2 things would be helpful:

Is there a Docker image that we can easily integrate into our test suite? (a la https://github.com/prisma/prisma-engines/blob/master/docker-compose.yml)
Is there a reliable way to "detect" and identify a SingleStore instance vs. a default MySQL?

Yes there is! We publish a memsql-in-a-box docker image that you can use in your CI. For example:

singlestore:
    image: 'memsql/cluster-in-a-box:centos-7.3.6-ae5d1857b8-3.2.5-1.11.0'
    restart: always
    environment:
      ROOT_PASSWORD: "password"
      LICENSE_KEY: $LICENSE_KEY
      START_AFTER_INIT: true
    ports:
      - "3306:3306"

To get a free license key you can signup at http://portal.singlestore.com/ but please let me know if you have any trouble with that.

An easy way to check is by running select @@memsql_version query which will only work for SingleStore DB

prisma.io/docs/concepts/components/prisma-client/debugging and prisma.io/docs/concepts/components/prisma-client/working-with-prismaclient/logging should give you what you are looking for I think to see the queries that are being executed under the hood and pinpoint the one giving you the error.

Thanks! I was able to get some additional logging from your suggestions and was able to reproduce the issue using the mysql2 library so this isn't exclusive to your package. Do you know which mysql connector are you using in the background? Is it a custom one?

Generally SingleStore is not a priority for us, as our customers and users are asking for other things. But I am happy to help you out answering questions and possibly also seeing about getting SingelStore into our CI setup. If we find deeper problems (like the first point about datetime_precision) I might not be able to guarantee Engineering capacity to fix those any time soon - but let's see how far we get until that is the case!

Totally understand that and I believe we'll be able to ge the integration working from our side, your guidance would be great! This isn't something we're working on actively either, it's mostly a side/pet project of mine, but hopefully I can dedicate some time to it!

Do you know which mysql connector are you using in the background? Is it a custom one?

We are using the mysql_async crate to connect to MySQL.

Good that I have a day off today @braposo, so I could already play around a bit to get this into our CI.
First bump in getting the test for our Introspection Engine, which introspects the database schema and creates a Prisma Schema from it (and is used in most of our other tests to confirm results of the Migration Engine, and database setup for Query Engine tests), to run was that we rely on the mysql database being present to connect to the database server. Fixed via a small query after starting the server: - run: mysql -h <IP_ADDRESS> -P 3306 -u root -pprisma -e "CREATE DATABASE mysql;" --loose-default-auth=mysql_native_password (Also documented for your users at https://www.singlestore.com/forum/t/mysqldb-exceptions-operationalerror-2012-error-in-server-handshake/2242/5?u=janpio.prisma)
Second one was that the database just needed a bit to start - so the above only worked after waiting a bit.
Afterwards I hit multiple different problems:

Error querying the database: Error accessing result set, column not found: datetime_precision you mentioned above already.
Then I quickly get a space error for some reason:thread 'commenting_out::a_table_without_required_uniques::mysql_8' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: QueryError(Server(ServerError { code: 1772, message: "CREATE PARTITION DATABASE for operation `CREATE DATABASE` failed with error 1883:Leaf Error (<IP_ADDRESS>:3307): Available disk space is below the value of \'minimal_disk_space\' global variable (100 MB). This query cannot be executed.", state: "HY000" })), original_code: Some("1772"), original_message: Some("CREATE PARTITION DATABASE for operation `CREATE DATABASE` failed with error 1883:Leaf Error (<IP_ADDRESS>:3307): Available disk space is below the value of \'minimal_disk_space\' global variable (100 MB). This query cannot be executed.") }', /home/runner/work/prisma-engines/prisma-engines/introspection-engine/introspection-engine-tests/src/test_api.rs:39:18

orthread 'relations::relations_should_avoid_name_clashes_2::mysql_8' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: QueryError(Server(ServerError { code: 1883, message: "Available disk space is below the value of \'minimal_disk_space\' global variable (100 MB). This query cannot be executed.", state: "HY000" })), original_code: Some("1883"), original_message: Some("Available disk space is below the value of \'minimal_disk_space\' global variable (100 MB). This query cannot be executed.") }', /home/runner/work/prisma-engines/prisma-engines/introspection-engine/introspection-engine-tests/src/test_api.rs:39:18

Source: https://github.com/prisma/prisma-engines/pull/1802/checks?check_run_id=2245842332
Some googling told me this was a error message from the Singlestore server (e.g. https://www.singlestore.com/forum/t/cluster-in-a-box-minimal-disk-space-error/2503), but my tries to fix it by adding a volume to the Docker Container etc did not work - I could not find a copy pastable example.

Can you help with that maybe? Current code: https://github.com/prisma/prisma-engines/pull/1802/files#diff-e45e45baeda1c1e73482975a664062aa56f20c03dd9d64a827aba57775bed0d3R4

No fun in waiting: I quickly modified the query that uses datetime_precision to always return NULL instead (https://github.com/prisma/prisma-engines/pull/1802/files#diff-9996eb1f4322b87712082043032bbd3ca8a0e8ccde7ecf19d80fefd7968c03b9L244), and that unblocked these two tests to make them succeed 💯
So if we can fix this space error, we might get many more tests succeeding as well 👍

oh wow this is great @janpio! Let me reach out to the people that might be able to figure out what's going on with docker and hopefully we can get over that hurdle as well!
Thank you so much for your help, this is very exciting! (at least for me :sweat_smile:!)

Hello @janpio! Thank you for your effort on this one! :smile:
That error indicates you don't have enough disk space to create the database.
Can you please run df -h in the container to understand the amount of free disk space we're working on with?
Thank you!

This is a GH Actions run on an Ubuntu container - usually there are around ~15GB free space available there. This is the https://github.com/prisma/prisma-engines/blob/janpio-patch-7/docker-compose.yml file I am running docker-compose up -d on @FranciscoMSM. We probably just have to give some of that available space to the container.

I sprinkled a bunch of commands into that workflow:
      - run: df -h
      - run: docker-compose run mysql-8-0 df -h

Results will be at https://github.com/prisma/prisma-engines/pull/1802/checks?check_run_id=2247005960 in a bit.

Could it be related to this comment? https://www.singlestore.com/forum/t/cluster-in-a-box-minimal-disk-space-error/2503/11
would something like this work?
volumes:
      - /memsql:/var/lib/memsql:rw

I also noticed that in other examples it's using tmpfs, have you tried:
tmpfs: /var/lib/memsql

sorry if I can't be more helpful at this point but hopefully @FranciscoMSM can come with better suggestions if this doesn't work!

I do not know the difference between volumes, tmpfs, and even the syntax of volumes is not totally clear to me and my google is failing me.
I tried with tmpfs: /var/lib/memsql now and that makes this line appear in df -h output on the docker container:
tmpfs           3.4G     0  3.4G   0% /var/lib/memsql

But then I can not connect to the database:
> Run mysql -h <IP_ADDRESS> -P 3307 -u root -pprisma -e "SHOW DATABASES;" --loose-default-auth=mysql_native_password
ERROR 2003 (HY000): Can't connect to MySQL server on '<IP_ADDRESS>' (111)

So something I am doing wrong somewhere.

Ok, after a lot of fiddling after dinner I have many more tests running 💯
I am now using a GH Action to free up more space on the machine, and then make that space available to /var/lib/memsql inside Docker: 65G
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/buildvg-buildlv   65G   80M   65G   1% /var/lib/memsql

That means I can now actually run a lot more tests, and get lots of real, actual errors:
---- commenting_out::a_table_without_fully_required_compound_unique::mysql_8 stdout ----
thread 'commenting_out::a_table_without_fully_required_compound_unique::mysql_8' panicked at 'The test function returned an Err.: Error querying the database: Server error: `ERROR HY000 (1896): Cannot declare primary key or unique key named: 'sqlite_autoindex_Post_1' when using an empty shard key'

Location:
    /home/runner/work/prisma-engines/prisma-engines/introspection-engine/introspection-engine-tests/src/lib.rs:58:47', introspection-engine/introspection-engine-tests/tests/commenting_out/mod.rs:127:1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- commenting_out::a_table_without_required_uniques::mysql_8 stdout ----
thread 'commenting_out::a_table_without_required_uniques::mysql_8' panicked at 'The test function returned an Err.: Error querying the database: Server error: `ERROR HY000 (1896): Cannot declare primary key or unique key named: 'opt_unique' when using an empty shard key'

Location:
    /home/runner/work/prisma-engines/prisma-engines/introspection-engine/introspection-engine-tests/src/lib.rs:58:47', introspection-engine/introspection-engine-tests/tests/commenting_out/mod.rs:101:1

---- commenting_out::a_table_without_uniques_should_ignore::mysql_8 stdout ----
thread 'commenting_out::a_table_without_uniques_should_ignore::mysql_8' panicked at 'The test function returned an Err.: Error querying the database: Server error: `ERROR HY000 (1706): Feature 'FOREIGN (non-SHARD) key to a sharded table' is not supported by MemSQL.'

Location:
    /home/runner/work/prisma-engines/prisma-engines/introspection-engine/introspection-engine-tests/src/lib.rs:58:47', introspection-engine/introspection-engine-tests/tests/commenting_out/mod.rs:7:1

---- enums::a_table_enums_should_return_alphabetically_even_when_in_different_order::mysql_8 stdout ----
thread 'enums::a_table_enums_should_return_alphabetically_even_when_in_different_order::mysql_8' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: QueryError(Server(ServerError { code: 1778, message: "Distributed MemSQL requires the length of the database name to be at most 60 characters.", state: "HY000" })), original_code: Some("1778"), original_message: Some("Distributed MemSQL requires the length of the database name to be at most 60 characters.") }', /home/runner/work/prisma-engines/prisma-engines/introspection-engine/introspection-engine-tests/src/test_api.rs:39:18

---- commenting_out::remapping_field_names_to_empty::mysql_8 stdout ----
thread 'commenting_out::remapping_field_names_to_empty::mysql_8' panicked at 'assertion failed: `(left == right)`

Diff < left / right > :
<"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/remapping_field_names_to_empty?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel User {\n  last Int @id @default(autoincrement())\n}\n"
>"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/remapping_field_names_to_empty?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel User {\n  last BigInt @id @default(autoincrement())\n}\n"

', introspection-engine/introspection-engine-tests/src/test_api.rs:220:9

---- enums::a_table_with_enum_default_values::mysql_8 stdout ----
thread 'enums::a_table_with_enum_default_values::mysql_8' panicked at 'assertion failed: `(left == right)`

Diff < left / right > :
<"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/a_table_with_enum_default_values?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel Book {\n  id    Int     @id @default(autoincrement())\n  color Book_color @default(black)\n}\n\nenum Book_color {\n  black\n  white\n}\n"
>"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/a_table_with_enum_default_values?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel Book {\n  id    BigInt     @id @default(autoincrement())\n  color Book_color @default(black)\n}\n\nenum Book_color {\n  black\n  white\n}\n"

', introspection-engine/introspection-engine-tests/src/test_api.rs:220:9

---- enums::a_table_with_an_enum_default_value_that_is_an_empty_string::mysql_8 stdout ----
thread 'enums::a_table_with_an_enum_default_value_that_is_an_empty_string::mysql_8' panicked at 'assertion failed: `(left == right)`

Diff < left / right > :
<"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/a_table_with_an_enum_default_value_that_is_an_empty_string?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel Book {\n  id    Int     @id @default(autoincrement())\n  color Book_color @default(EMPTY_ENUM_VALUE)\n}\n\nenum Book_color {\n  black\n  EMPTY_ENUM_VALUE @map(\"\")\n}\n"
>"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/a_table_with_an_enum_default_value_that_is_an_empty_string?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel Book {\n  id    BigInt     @id @default(autoincrement())\n  color Book_color @default(EMPTY_ENUM_VALUE)\n}\n\nenum Book_color {\n  black\n  EMPTY_ENUM_VALUE @map(\"\")\n}\n"

', introspection-engine/introspection-engine-tests/src/test_api.rs:220:9

---- enums::a_table_with_enums::mysql_8 stdout ----
thread 'enums::a_table_with_enums::mysql_8' panicked at 'assertion failed: `(left == right)`

Diff < left / right > :
<"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/a_table_with_enums?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel Book {\n  id     Int      @id @default(autoincrement())\n  color  Book_color\n  color2 Book_color2\n}\n\nenum Book_color {\n  black\n  white\n}\n\nenum Book_color2 {\n  black2\n  white2\n}\n"
>"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/a_table_with_enums?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel Book {\n  id     BigInt      @id @default(autoincrement())\n  color  Book_color\n  color2 Book_color2\n}\n\nenum Book_color {\n  black\n  white\n}\n\nenum Book_color2 {\n  black2\n  white2\n}\n"

', introspection-engine/introspection-engine-tests/src/test_api.rs:220:9

---- enums::a_table_with_enum_default_values_that_look_like_booleans::mysql_8 stdout ----
thread 'enums::a_table_with_enum_default_values_that_look_like_booleans::mysql_8' panicked at 'assertion failed: `(left == right)`

Diff < left / right > :
<"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/a_table_with_enum_default_values_that_look_like_booleans?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel News {\n  id        Int         @id @default(autoincrement())\n  confirmed News_confirmed @default(true)\n}\n\nenum News_confirmed {\n  true\n  false\n  rumor\n}\n"
>"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/a_table_with_enum_default_values_that_look_like_booleans?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel News {\n  id        BigInt         @id @default(autoincrement())\n  confirmed News_confirmed @default(true)\n}\n\nenum News_confirmed {\n  true\n  false\n  rumor\n}\n"

', introspection-engine/introspection-engine-tests/src/test_api.rs:220:9

---- identify_version::introspect_mysql_non_prisma::mysql_8 stdout ----
thread 'identify_version::introspect_mysql_non_prisma::mysql_8' panicked at 'The test function returned an Err.: Error querying the database: Server error: `ERROR 42000 (1064): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'point)' at line 1'

Location:
    /home/runner/work/prisma-engines/prisma-engines/introspection-engine/introspection-engine-tests/src/lib.rs:58:47', introspection-engine/introspection-engine-tests/tests/identify_version/mod.rs:186:1

---- identify_version::introspect_mysql_prisma_1::mysql_8 stdout ----
thread 'identify_version::introspect_mysql_prisma_1::mysql_8' panicked at 'The test function returned an Err.: Error querying the database: Server error: `ERROR HY000 (1706): Feature 'DATETIME type with precision other than 0 or 6' is not supported by MemSQL.'

Location:
    /home/runner/work/prisma-engines/prisma-engines/introspection-engine/introspection-engine-tests/src/lib.rs:58:47', introspection-engine/introspection-engine-tests/tests/identify_version/mod.rs:202:1

---- identify_version::introspect_mysql_prisma_1_1::mysql_8 stdout ----
thread 'identify_version::introspect_mysql_prisma_1_1::mysql_8' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: QueryError(Server(ServerError { code: 1842, message: "Database name (introspect_mysql_prisma_1_1) conflicts with an existing partition database name (partition databases are formatted as database_#, where # is a number)", state: "HY000" })), original_code: Some("1842"), original_message: Some("Database name (introspect_mysql_prisma_1_1) conflicts with an existing partition database name (partition databases are formatted as database_#, where # is a number)") }', /home/runner/work/prisma-engines/prisma-engines/introspection-engine/introspection-engine-tests/src/test_api.rs:39:18

---- identify_version::introspect_mysql_prisma2::mysql_8 stdout ----
thread 'identify_version::introspect_mysql_prisma2::mysql_8' panicked at 'called `Result::unwrap()` on an `Err` value: Error { kind: QueryError(Server(ServerError { code: 1772, message: "CREATE PARTITION DATABASE for operation `CREATE DATABASE` failed with error 1883:Leaf Error (<IP_ADDRESS>:3307): Available disk space is below the value of \'minimal_disk_space\' global variable (100 MB). This query cannot be executed.", state: "HY000" })), original_code: Some("1772"), original_message: Some("CREATE PARTITION DATABASE for operation `CREATE DATABASE` failed with error 1883:Leaf Error (<IP_ADDRESS>:3307): Available disk space is below the value of \'minimal_disk_space\' global variable (100 MB). This query cannot be executed.") }', /home/runner/work/prisma-engines/prisma-engines/introspection-engine/introspection-engine-tests/src/test_api.rs:39:18

After that we are back to many, many of the same space errors unfortunately :(
And indeed:
/dev/mapper/buildvg-buildlv   65G   62G  3.0G  96% /var/lib/memsql

Not the 100MB the error message mentions, but 62G are now filled with something.
Can you help me make the Docker container use less space? Or at least understand what is taking up that space @braposo?

Back to the list of actual errors:
First three are about keys and empty shard key and sharded table:

thread 'commenting_out::a_table_without_fully_required_compound_unique::mysql_8' panicked at 'The test function returned an Err.: Error querying the database: Server error: `ERROR HY000 (1896): Cannot declare primary key or unique key named: 'sqlite_autoindex_Post_1' when using an empty shard key'
thread 'commenting_out::a_table_without_required_uniques::mysql_8' panicked at 'The test function returned an Err.: Error querying the database: Server error: `ERROR HY000 (1896): Cannot declare primary key or unique key named: 'opt_unique' when using an empty shard key'
thread 'commenting_out::a_table_without_uniques_should_ignore::mysql_8' panicked at 'The test function returned an Err.: Error querying the database: Server error: `ERROR HY000 (1706): Feature 'FOREIGN (non-SHARD) key to a sharded table' is not supported by MemSQL.'

No idea what any of this means.

One about using a to long table name:

thread 'enums::a_table_enums_should_return_alphabetically_even_when_in_different_order::mysql_8' panicked at 'called Result::unwrap() on an Err value: Error { kind: QueryError(Server(ServerError { code: 1778, message: "Distributed MemSQL requires the length of the database name to be at most 60 characters.", state: "HY000" })), original_code: Some("1778"), original_message: Some("Distributed MemSQL requires the length of the database name to be at most 60 characters.") }'

That one is just configuration so should be easy to fix.

Multiple about a Int/BigInt mismatch in the returned schema:

thread 'commenting_out::remapping_field_names_to_empty::mysql_8' panicked at 'assertion failed: (left == right)

Diff < left / right > :
<"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/remapping_field_names_to_empty?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel User {\n  last Int @id @default(autoincrement())\n}\n"
>"datasource db {\n  provider = \"mysql\"\n  url      = \"***<IP_ADDRESS>:3307/remapping_field_names_to_empty?connect_timeout=20&socket_timeout=60\"\n}\n\nmodel User {\n  last BigInt @id @default(autoincrement())\n}\n"

My guess would be that Signelstore somehow creates a different column as we tell it to because it is smart in some way? Or the information about that difference is represented differently maybe?

A MySQL syntax error:

thread 'identify_version::introspect_mysql_non_prisma::mysql_8' panicked at 'The test function returned an Err.: Error querying the database: Server error: `ERROR 42000 (1064): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'point)' at line 1'

Possibly something not supported by SingleStore.

Something about datetime precision:

thread 'identify_version::introspect_mysql_prisma_1::mysql_8' panicked at 'The test function returned an Err.: Error querying the database: Server error: `ERROR HY000 (1706): Feature 'DATETIME type with precision other than 0 or 6' is not supported by MemSQL.'

I bet this is a side effect of the hack that I added to unblock the missing  datetime_precision

And finally something about a name clash:

thread 'identify_version::introspect_mysql_prisma_1_1::mysql_8' panicked at 'called Result::unwrap() on an Err value: Error { kind: QueryError(Server(ServerError { code: 1842, message: "Database name (introspect_mysql_prisma_1_1) conflicts with an existing partition database name (partition databases are formatted as database_#, where # is a number)", state: "HY000" })), original_code: Some("1842"), original_message: Some("Database name (introspect_mysql_prisma_1_1) conflicts with an existing partition database name (partition databases are formatted as database_#, where # is a number)") }'

I guess Singlestore has rules about what you can call the tables, that conflicts with what we generate automatically for the test.

This is some incredible work @janpio! Thank you so much for doing this and for taking the time to get this a bit further!
I'm going to take this back to the team and try to find the answers to your questions. I'll let you know when I have something to share.
In the meantime, is there a way for us to test SingleStore against your test suite outside of your CI? It's probably faster if we can run the tests locally as we're making the changes to the engine.
Thanks again, this is really amazing!

The PR linked above shows pretty much that in its GH Actions configuration.
TLDR is: Check out https://github.com/prisma/prisma-engines, install the 1.48.0 Rust toolchain, make a DB run at localhost:3007 (plus create an empty mysql database) and then execute cargo test mysql_8 in introspection-engine/introspection-engine-tests. (As Singlestore is MySQL compatible (kinda), this "reuses" our MySQL (8.0) test suite (which usually runs against a MySQL DB at 3007). This avoids us having to add all the usual boilerplate to add a new database and the reason why I could do so as a small side project.)
I could continue with Migration and Query Engine tests here now, but I would probably as quickly hit the memory problem again. And as both these test suites are considerably bigger than the Introspection Engine one, the % of tests we can run will probably be a lot lower :/ So looking forward to the feedback on how to tweak the container configuration to use less space.

(I ran our Migration Engine and Query Engine tests on SingleStore as well - and as expected Leaf Error (<IP_ADDRESS>:3307): Available disk space is below the value of \'minimal_disk_space\' global variable (100 MB). This query cannot be executed. prevents most to all tests being run for now.)

Seems some logging is going crazy:
root@fv-az186-340:/home/runner/work/prisma-engines/prisma-engines/memsql/8b0b24b6-1868-4160-8baf-e6a07d34ee89/data/logs# ls -l
...
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_1_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_1_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_2_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_2_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_3_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_3_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_4_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_4_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_5_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_5_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_6_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_6_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_7_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 introspect_mysql_non_prisma_empty_7_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>4 Apr  4 14:54 introspect_mysql_non_prisma_empty_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>4 Apr  4 14:53 introspect_mysql_non_prisma_empty_log_v1_16384
-rw------- 1 systemd-coredump<PHONE_NUMBER>4 Apr  4 14:54 introspect_mysql_non_prisma_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>4 Apr  4 14:53 introspect_mysql_non_prisma_log_v1_16384
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_0_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_0_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_1_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_1_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_2_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_2_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_3_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_3_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_4_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_4_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_5_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_5_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_6_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_6_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_7_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:54 introspect_mysql_prisma2_7_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>4 Apr  4 14:54 introspect_mysql_prisma2_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>4 Apr  4 14:54 introspect_mysql_prisma2_log_v1_16384
-rw------- 1 systemd-coredump<PHONE_NUMBER>6 Apr  4 14:54 memsql_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>6 Apr  4 14:47 memsql_log_v1_4096
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_0_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_0_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_1_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_1_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_2_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_2_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_3_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_3_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_4_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_4_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_5_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_5_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_6_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_6_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_7_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:48 mysql_7_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>4 Apr  4 14:48 mysql_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>4 Apr  4 14:48 mysql_log_v1_16384
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_0_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_0_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_1_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_1_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_2_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_2_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_3_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_3_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_4_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_4_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_5_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_5_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_6_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_6_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_7_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>56 Apr  4 14:53 remapping_field_names_to_empty_7_log_v1_65536
-rw------- 1 systemd-coredump<PHONE_NUMBER>4 Apr  4 14:54 remapping_field_names_to_empty_log_v1_0
-rw------- 1 systemd-coredump<PHONE_NUMBER>4 Apr  4 14:53 remapping_field_names_to_empty_log_v1_16384
root@fv-az186-340:/home/runner/work/prisma-engines/prisma-engines/memsql/8b0b24b6-1868-4160-8baf-e6a07d34ee89/data/logs#

58G of logging that is super verbose, but not very helpful. Any way I can disable that in the container @braposo @FranciscoMSM? That might already be enough to get the tests running.

Ok, those seem to be snapshots of the databases. No idea why they are so huge. I could not find a way to disable snapshots either. Right now it seems as creating many databases is incompatible with this Docker container :/

Now I'm trying to get some help on how to set it as part of the container in a box image so I'll get back to you when I get that information.

I think this is of the same class as core_file which I played with already - so I think I might know already. (Link to really terrible code, where I messed with those thinking it would help: https://github.com/prisma/prisma-engines/pull/1802/files#diff-32349d539eeee14551827f0971cfb90f048fa8bceb05904631f4838c530fa865R51-R65)

Hey! I've run into the same error as you did initially (Error accessing result set, column not found: datetime_precision) when trying to do my very first prisma db pull on an existing DB we have. I wanted to check in and see if there are any updates in this area, or if I can help debug. I'm new to both singlestore and prisma, so you may have to be a little patient with me.

No, that error message indicates that one of the fundamental columns that we need for db pull and that usually exists in MySQL, does not in SingleStore. We would need to adapt our code (after finding an alternative) to make this work.

Hello @janpio
using Prisma with Singlestore is quite hard so far !
when it comes to the DDL, push changes, indices ...etc !
is there any plan sooner to support Singlestore

No, we are not working on improving support for SingleStore right now. We have too many other things to do first (see other issues, roadmap and releases) before we will get to add additional connectors/databases that do not work out of the box.

@janpio make sense thank you

@MohannadBahaa did you move to another solution or did you find a way to get SingleStore to work with Prisma?

@xpluscal unfortunately, I have moved with Sequelize check this start-with-singlestore-node-orm

@MohannadBahaa, what was the experience using Sequelize on Singlestore? Had any issue with Sequelize using Singlestore?

Experience: Consuming data from SQS then using Sequelize for copying to Singlestore.
Cons: Managing the connection pool was a challenge. This was particularly the case when dealing with a high volume of data - around 1 million records per hour.


Fix TestLookupEmptyNameFails

TestLookupEmptyNameFails was an identical copy of TestLookupInvalidSid
in the same file. This commit changes the account name queried by the
former to an empty string which is what the name of the test case
suggests.

@bitgestalt Please be sure to sign your commit with git commit -s

@bitgestalt Please be sure to sign your commit with git commit -s

Sorry for that, fixed.


{Misc} Update codeowners for pr review

Description

update codeonwers for pr review
Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change.
[Component Name 2] az command b: Add some customer-facing feature.

This checklist is used to make sure that common guidelines for a pull request are followed.

[ ] The PR title and description has followed the guideline in Submitting Pull Requests.

[ ] I adhere to the Command Guidelines.

add to S168

/azp run - Check the Format of Pull Request Title

/azp run

the CI passed in ADO, seems status not synced into github https://dev.azure.com/azure-sdk/public/_build/results?buildId=352157&view=logs&j=956eb394-0083-5a2b-15a5-c7a5af602a2b


Use OCI manifest from skopeo copy dir for analysis

If uncompressed layers are stored to the registry, skopeo will compress them on the fly when executing "skopeo copy ... oci:dir".  In this case there will be disparity between the image manifest that was previously fetched with "skopeo inspect", which had references to uncompressed layers. Analysis will look up for uncompressed layers from the OCI directory and raise an exception since they cannot be found - analysis will fail.
This change makes the analysis use the OCI image manifest and therefore resolve to the correct filenames.
Fixes #1080
Signed-off-by: Tero Saarni<EMAIL_ADDRESS>

Thanks for submitting this! I'd also like to copy the image you used into our testing repo so we can use it for regression tests in the future as well. I'll look into it, but if you used a specific process to upload it I would appreciate that info so we can use it was well.

Thanks for submitting this! I'd also like to copy the image you used into our testing repo so we can use it for regression tests in the future as well. I'll look into it, but if you used a specific process to upload it I would appreciate that info so we can use it was well.

I manually uncompressed a layer from alpine, modified its manifesta and then used the registry REST API directly from command line to upload, since skopeo always would compress the layers.  But this is so convoluted, I bet there must be some straightforward way to trigger this as well.  I'll look into it and will come back.
By the way, I had some doubts when removing the manifest and parent_manifest from the retrying_pull_image() parameters. It seemed to me like they are not necessary anymore and maybe they were only needed before moving to OCI directory layout for the work dir.  If that is true, then maybe some more cleanups can be done in skopeo_wrapper.py:download_image().

I've found out that the problematic images that lead to submitting  #1080 were uploaded by proprietary software which directly interacts with the registry REST API.  So far, I don't have any indication that there exists any publicly available tool that would result in similar images.
I've created a simple script that automates my manual upload steps: upload-as-uncompressed.py
It is very bare bones, no error handling etc but might be ok for demonstration purposes...

@tsaarni thanks for submitting this PR. It looks like Circle CI missed this PR for some reason. Can you rebase on master or merge master into your branch? Hopefully that will get a new CI job to run.

@Vijay-P Sure, I've now rebased the PR.

@Vijay-P Sure, I've now rebased the PR.

Thanks. Not sure what the issue is, but creating a temporary duplicate branch and PR triggered the CI workflow.

I think this will have an impact on images that use a foreign.diff layer type. The manifest is passed in for that computation, so that the layers can be fetched. But, since https://github.com/containers/image/pull/1063 was merged, we may not need that code at all anymore if an appropriate update to skopeo was done as well. It doesn't look like it was, but will confirm.

This PR seems to be pretty old, but problem seems to be still present. I've just tested this change on my custom build and it allowed me to scan images that were previously failing (we are not authors of these - they are actually from Microsoft).
Can we merge this?


[Task]: Document the Steps to setup Airtable locally

SubTasks

Document the steps to setup Airtable datasource locally
Log the issues faced and add Troubleshooting steps to resolve them.

https://www.notion.so/appsmith/Setting-up-Airtable-datasource-locally-a2d14005e6034d7fa71f153ede07d43e?pvs=4


Exception when invalid IP address

Description
Someone sent malformed requests to try to find vulnerabilities. Some of them raised exceptions in NewRelic's code.
Expected Behavior
Not raise an exception and let the app handle it.
Troubleshooting or NR Diag results
Here's the complete stack trace for the exception:
Traceback (most recent call last):
  File "/app/.heroku/python/lib/python3.12/site-packages/waitress/channel.py", line 428, in service
    task.service()
  File "/app/.heroku/python/lib/python3.12/site-packages/waitress/task.py", line 168, in service
    self.execute()
  File "/app/.heroku/python/lib/python3.12/site-packages/waitress/task.py", line 434, in execute
    app_iter = self.channel.server.application(environ, start_response)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/site-packages/newrelic/api/wsgi_application.py", line 599, in _nr_wsgi_application_wrapper_
    transaction = WSGIWebTransaction(target_application, environ, source=wrapped)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/site-packages/newrelic/api/web_transaction.py", line 692, in __init__
    self._request_uri = urlparse.urlparse(self._request_uri)[2]
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/urllib/parse.py", line 395, in urlparse
    splitresult = urlsplit(url, scheme, allow_fragments)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/urllib/parse.py", line 500, in urlsplit
    _check_bracketed_host(bracketed_host)
  File "/app/.heroku/python/lib/python3.12/urllib/parse.py", line 446, in _check_bracketed_host
    ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.heroku/python/lib/python3.12/ipaddress.py", line 54, in ip_address
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: "'xwork.MethodAccessor.denyMethodExecution'" does not appear to be an IPv4 or IPv6 address

It's probably irrelevant, but the HTTP request was
GET //${#context['xwork.MethodAccessor.denyMethodExecution']=!(#_memberAccess['allowStaticMethodAccess']=true),(@java.lang.Runtime@getRuntime()).exec('ipconfig').waitFor()}.action

The stack trace seems to point to this line:
https://github.com/newrelic/newrelic-python-agent/blob/56fbda161ec68c63c80755caf2ff5ed1ddd84819/newrelic/api/web_transaction.py#L692
As I mentioned above, the request was malformed intentionally by an attacker. It looks like they sent "'xwork.MethodAccessor.denyMethodExecution'" for the IP address and it crashed when NewRelic tried to parse it.
Steps to Reproduce
I'm not sure how to forge a request like this and/or where that fake IP address is taken from, but it would probably reproduce the issue.
Your Environment
It's a Python web app, with the Pyramid web framework, running with Waitress (HTTP server) on Heroku, behind Fastly. Python 3.12.1.
We run the app with newrelic-admin run-program python path/to/app.py.

I just wanted to mention that this is still an issue. Someone forging a request just triggered the error again. We're currently on Python 3.12.3, with NewRelic 9.10.0.


Support System Profiles

Right now, the cdap_profile resource requires a namespace, and omitting it uses the "default" namespace, so it is currently impossible to configure System Compute Profiles.

It's been a while since I looked into this but one of the caveats is data fusion by default sets some system profile properties, so if we send an API with new ones it will overwrite the existing ones. Trying to fetch and append existing ones is unidiomatic for Terraform. You also can't create/delete system namespace so it doesn't fit super well to the Terraform model. Thus, we chose to just support profiles at the namespace level.
Thus, I would encourage using namespace level profiles.
If this is a highly desired feature I would consider accepting a PR that added a 'type' field that defaults to 'NAMESPACE', but can also be set to 'SYSTEM'. In the case it is NAMESPACE then we check the namespace field, else it is ignored.
It would need to fit the Terraform model however, so would need answers to the existing fields and creation/deletion questions above.


Merge main -> google

f56dddef Update link to Using Colab doc. (#2938)
93f98331 Fix doc generation on Windows. (#2940)
bfbf93d5 Make DotGeneral op live in their own dispatch function. (#2936)
172ea202 Only rewrite 1x1_conv -> dot only for unit strides & dilations (#2916)
da8bba9a Add control to enable LICM from command line and fix a bug (#2927)
2eec61cb Add support for trivial dot_general for vulkan-spirv backend. (#2875)
7f66021d Merge google -> main (#2926)

Failing lint/submodules check is because I branched off just before the synchronize submodules action ran. Merging with that failing on the PR shouldn't be an issue.


Remove config from exception when Hikari data source can't be created

Problem
When creating a DataSource with DataSourceLayer.fromConfig with a wrong configuration (rejected by the server), the exception thrown contains all the configuration serialised and so could contain the password.
By default, if we just want to log what comes from Quill, the password could be leaked clear.

Solution
I removed the config serialised in the exception message to let the application log what it's relevant.
Notes
Additional notes.
Checklist

[x] Unit test all changes
[x] Update README.md if applicable
[x] Add [WIP] to the pull request title if it's work in progress
[x] Squash commits that aren't meaningful changes
[x] Run sbt scalariformFormat test:scalariformFormat to make sure that the source files are formatted

@getquill/maintainers

@mdulac Thanks for your contribution! ❤


Fixed it to work with the latest Swift version and iOS 12.

Notifications SDK amongst other things changed in iOS 12.

Thanks.


Add arm64 macOS build arch for future Apple Silicon mac support

Platforms Impacted

[ ] iOS
[x] macOS

Description of changes
Add platform-specific versions of our publish_overrides.xcconfig file that include publish_overrides.xcconfig but then independently specify the extra archs necessary for each platform. On macOS we want to add arm64 (but not arm64e yet). On iOS we want to continue to build arm64e slices.
Update our Azure Pipeline yaml to respect the new override xcconfigs, and separately update our local script that mirrors the logic in the azure pipeline to match as well.
Verification
Run scripts/nuget_publish.sh to simulate publishing the NuGet package, ensure it succeeds as expected. Ensure that the proper archs are built for each.

Before
After

Pull request checklist
This PR has considered:

[ ] Light and Dark appearances
[ ] VoiceOver and Keyboard Accessibility
[ ] Internationalization and Right to Left layouts
[ ] Different resolutions (1x, 2x, 3x)
[ ] Size classes and window sizes (iPhone vs iPad, notched devices, multitasking, different window sizes, etc)

Microsoft Reviewers: Open in CodeFlow

Note: Since we're not building our NuGet publish with Xcode 12 yet, this shouldn't have any effect. I'll be making a followup change adding a separate beta publishing pipeline that uses the latest Xcode 12 beta to allow us to temporarily have side-by-side support for different xcode versions.


[xxxx] Removed pgde course

Context
Pgde course model
Changes proposed in this pull request
Removed unneeded file
Removed pgde course as well
Guidance to review
:ship:
Checklist

[ ] Make sure all information from the Trello card is in here
[ ] Attach to Trello card
[ ] Rebased master
[ ] Cleaned commit history
[ ] Tested by running locally

 Remove pgde course from database

🚢 :shipit: 🚀

@timabell will dropping a table be an issue for https://github.com/DFE-Digital/manage-courses-api
Till answered hanging this to dry...

Holding off merging till all the rollover churn is out of the way and the dust has settled.


MarketOnCloseOrders not included in BacktestLoggingControl example

MarketOnCloseOrders not included in out of the box BacktestLoggingControl example (https://github.com/liampauling/flumine/blob/master/examples/controls/backtestloggingcontrol.py). Using in live framework. Only changed order.txt path.
`import csv
import logging
from flumine.controls.loggingcontrols import LoggingControl
logger = logging.getLogger(name)
FIELDNAMES = [
"bet_id",
"strategy_name",
"market_id",
"selection_id",
"trade_id",
"date_time_placed",
"price",
"price_matched",
"size",
"size_matched",
"status",
"side",
"elapsed_seconds_executable",
"order_status",
"market_note",
"trade_notes",
"order_notes",
]
class StrategyLoggingControl(LoggingControl):
NAME = "STRATEGY_LOGGING_CONTROL"
def __init__(self, *args, **kwargs):
    super(StrategyLoggingControl, self).__init__(*args, **kwargs)
    self._setup()

def _setup(self):
    with open("C:/Users/Administrator/AppData/Local/Programs/Python/Python39/Lib/site-packages/flumine/strategies/orders.txt", "w") as m:
        csv_writer = csv.DictWriter(m, delimiter=",", fieldnames=FIELDNAMES)
        csv_writer.writeheader()

def _process_cleared_orders_meta(self, event):
    orders = event.event
    with open("C:/Users/Administrator/AppData/Local/Programs/Python/Python39/Lib/site-packages/flumine/strategies/orders.txt", "a") as m:
        for order in orders:
            try:
                order_data = {
                    "bet_id": order.bet_id,
                    "strategy_name": order.trade.strategy,
                    "market_id": order.market_id,
                    "selection_id": order.selection_id,
                    "trade_id": order.trade.id,
                    "date_time_placed": order.responses.date_time_placed,
                    "price": order.order_type.price,
                    "price_matched": order.average_price_matched,
                    "size": order.order_type.size,
                    "size_matched": order.size_matched,
                    "status": order.runner_status,
                    "side": order.side,
                    "elapsed_seconds_executable": order.elapsed_seconds_executable,
                    "order_status": order.status.value,
                    "market_note": order.trade.market_notes,
                    "trade_notes": order.trade.notes_str,
                    "order_notes": order.notes_str,
                }
                csv_writer = csv.DictWriter(m, delimiter=",", fieldnames=FIELDNAMES)
                csv_writer.writerow(order_data)
            except Exception as e:
                logger.error(
                    "_process_cleared_orders_meta: %s" % e,
                    extra={"order": order, "error": e},
                )

    logger.info("Orders updated", extra={"order_count": len(orders)})

def _process_cleared_markets(self, event):
    cleared_markets = event.event
    for cleared_market in cleared_markets.orders:
        logger.info(
            "Cleared market",
            extra={
                "market_id": cleared_market.market_id,
                "bet_count": cleared_market.bet_count,
                "profit": cleared_market.profit,
                "commission": cleared_market.commission,
            },
        )

`

So it seems MarketOnCloseOrders aren't being cleared properly on flumine because no price
{"asctime": "2021-11-23 06:57:50,183", "levelname": "ERROR", "message": "_process_cleared_orders_meta: 'MarketOnCloseOrder' object has no attribute 'price'", "order": "Order<PHONE_NUMBER>38: Execution complete", "error": "'MarketOnCloseOrder' object has no attribute 'price'"}
Maybe this is why they aren't making it to the logging control?

Ah good spot, that would be why


[stable/burrow] Add @compleatang to owners of burrow chart

Now that I'm added to the organization; happy to help maintain this chart.

/assign @unguiculus

/lgtm


Fix/clamav updating

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.0 out of 4 committers have signed the CLA.:x: Rogier Goede:x: ppj-vandorst:x: boldpawn:x: DragnipurRogier Goede seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Support for pnpm

Currently, Hugo looks for the npx binary to execute PostCSS commands. In my case, I prefer using pnpm to save time and disk space. Would it be possible to add support for pnpx to Hugo, which is the pnpm counterpart for npx?

I haven't looked into it too much but I've recently been running into JsBuild errors while trying to use pnpm:
Cannot read file "node_modules/some-module": Incorrect function

But it seems to work as expected when using npm.


Remove cobertura link from comments

@sectioneight

Looks good, assuming I didn't add some test that looks for that comment text

Coverage remained the same at 92.152% when pulling 2e4a0a40562724321bc9a582d7c0f3c68874385a on jx_remove_cobertura into a1f01adf672b5d006ebb438dd6052c115577f7d9 on master.


GCP needs admin permissions before running

If I try to run Sonobuoy on a new GKE cluster, it fails with the following error:
Error from server (Forbidden): error when creating "/dev/fd/11": clusterroles.rbac.authorization.k8s.io "sonobuoy-serviceaccount" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["*"], APIGroups:["*"], Verbs:["*"]}]<EMAIL_ADDRESS>[system:authenticated] map[authenticator:[GKE]]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]
zsh: exit 1     kubectl apply -f

The solution is to create an admin role for the user:
kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin<EMAIL_ADDRESS>We should document this, or add a preflight check, or otherwise expose it to the user

@liz do you think adding this issue to the readme is sufficient?

For now I think that should be good!


Remove dependency on segmatch and segmatch_ros

This is for branch feature/laser_slam_worker. The solution would be to move the corresponding functions in common.h of segmatch and segmatch_ros to laser_slam and laser_slam_ros respectivelly.

Done in  #7.


(chore) generate joanie api client

Proposal
POC that add a simple script to generate joanie's api client with openapi-typescript-codegen

note: we may be able to add react-query logic directly into a custom request.ts file


Embedded RabbitMq conficts with already running server

If a machine already has a running instance of RabbitMq then an attempt to launch embedded RabbitMq would cause an error.
At least a random port should be selected by default to solve this issue somehow.

From your previously reported issue (#28), it seems like you ran into this issue when you were running the JUnit tests and you abruptly stop the test execution, then tried it again and this second time, the RabbitMQ broker wouldn't even start because the previous process was still running, and in this case, you'd prefer to have a random port be assigned so it won't clash with other instances, correct?
If so, i do agree that the code in the tests itself could figure out a random port and instruct the Embedded RabbitMQ library to use such port. However, I don't think it should be up to the library itself to figure out a random port and use that port automatically.
Would you like to give this a try yourself?
I did think about creating a JUnit rule class that can be used to automatically start the Embedded RabbitMQ server, have a single instance available for all unit/integration tests and provide clean/independent "virtual hosts" for each test class to avoid clashes. However, this isn't a priority at this point in time, mostly because writing/maintaining a test-library might not be worth the effort when there will be so many different ways to do achieve the same thing. I'll probably just add sample code to the wiki so that people can copy and customize it.

Actually, no. I've killed all instances of RabbitMQ, and started it through systemd. Embedded RabbitMq runs on the default port and this causes the issue.
A random port for embedded resources is a very common pattern, imho. There is no problem to correctly setup it. An example would be nice, but I suggest to use random port by default.

A random port for embedded resources is a very common pattern, imho.

I've checked the following projects and this are my findings:

embedded-redis: starts using default Redis port 6379 (proof)
embedded-mongo: does start with a random port. (proof)
embedded-mysql: starts with default mysql port 3310  (proof)
embedded-postgres: starts with random port (proof)

From this small sample, it seems like it's both common and not. Got other projects you know of?
In any case, this research is making me lean more towards supporting the random port logic inside the library code instead of the code example/test-support library, however, i still feel iffy about doing this as the default port.
Btw, I'm not sure by when i'll be able to work on this, but if you're interested and have the time, feel free to submit a PR :)

Ok, I admit I was wrong about random ports, but I think implementing this in the library is a great idea.

I think implementing this in the library is a great idea.

Yeah, thanks for bringing this up :)

See PR #30


Platform error, installed from HACS

                Platform error binary_sensor.rpi_gpio - Exception importing custom_components.rpi_gpio.binary_sensor

`binary_sensor:

platform: rpi_gpio
host: 192.168.1.???
sensors:

port: 17
name: "Alarm Set"
port: 27
name: "Alarm Activated"`

Any help woud be much appreciated

Please provide the full log after restart, it is impossible to understand from this what is wrong. Also add some information about your setup (OS, installation type)

Looks like it is my fault. I used the old setup that was removed from HA to monitor the GPIO of a Pi on my LAN. From the log below it looks like your code only runs on a Pi for checking it's own GPIO. Unfortunately.
Home Assistant 2022.12.7 Supervisor 2022.11.2 Operating System 9.4 Frontend 20221213.1 - latest
This error originated from a custom integration.
Logger: homeassistant.loader
Source: custom_components/rpi_gpio/init.py:3
Integration: rpi_gpio (documentation, issues)
First occurred: 20:01:15 (1 occurrences)
Last logged: 20:01:15
Unexpected exception importing platform custom_components.rpi_gpio.binary_sensor
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/loader.py", line 728, in get_platform
cache[full_name] = self._import_platform(platform_name)
File "/usr/src/homeassistant/homeassistant/loader.py", line 745, in _import_platform
return importlib.import_module(f"{self.pkg_path}.{platform_name}")
File "/usr/local/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 992, in _find_and_load_unlocked
File "", line 241, in _call_with_frames_removed
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/config/custom_components/rpi_gpio/init.py", line 3, in
from RPi import GPIO  # pylint: disable=import-error
File "/usr/local/lib/python3.10/site-packages/RPi/GPIO/init.py", line 23, in
from RPi._GPIO import *
RuntimeError: This module can only be run on a Raspberry Pi!

This is called "Home Assistant Raspberry Pi GPIO custom integration", it is used to help maintaining the one removed from core.
Thanks for reporting back, closing this issue 👍


Update test eop merge

Add test for merging eopatches

I wouldn't create too advanced examples here, since this is mostly tested in test_eodata_merge.py. A simple test or two that it is correctly linked to EOPatch should suffice

I wouldn't create too advanced examples here, since this is mostly tested in test_eodata_merge.py. A simple test or two that it is correctly linked to EOPatch should suffice

Test case is significantly simplified, i wanted to write L529 less convoluted but kinda give up after a while ....


WARNING - Failed to fetch room id from Webcast

Hello!
We have been seeing this warning message (WARNING - Failed to fetch room id from Webcast) with some of the livestreams that we are working with. Can someone help me understand why and when this message is dropped and what could be the probable way to solve this?
We are trying to catch the livestream data, video, comments and stream activities but this is causing the event data to be missing.
Would appreciate if a prompt response is received.
Thanks in advance.

me too

Due to the current approach which is being overhauled.
See https://github.com/isaackogan/TikTokLive/issues/137


ubuntumono-b-webfont not found

There are no ubuntumono-b-webfont fonts in fonts folder.
Seems you should change ubuntumono-b-webfont to ubuntumono-bold-webfont in styles.

Faced the same issue trying to use fonts with Webpack CSS loader.


:apple: Menus: replace three period with ellipsis

From:
[OS X Human Interface Guidelines -Terminology and Wording] (https://developer.apple.com/library/mac/documentation/UserExperience/Conceptual/OSXHIGuidelines/TerminologyWording.html#//apple_ref/doc/uid/20000957-CH15-SW3)

IMPORTANT
Be sure to create the ellipsis character using the key combination Option-; (Option-semicolon). This ensures that an assistive app can provide the correct interpretation of the character to a disabled user. If you use three period characters to simulate an ellipsis, many assistive apps will be unable to make sense of them. Also, three period characters and an ellipsis don't look the same because the periods are spaced differently than the points of an ellipsis.

It looks https://travis-ci.org/atom/atom/jobs/69127662 fails because of a bug in tree-view-spec.coffee introduced in https://github.com/atom/tree-view/commit/d47a1841116a4ebb9f3027a10e0a9c5ce360f5ce
and solved in pull request https://github.com/atom/tree-view/pull/500

Good catch! Thanks for this. I'm wondering though if we should also update the Windows and Linux menus too – I'm not sure what the conventions are for those platforms, though.

I'm wondering though if we should also update the Windows and Linux menus too

:+1:

@paulcbetts Is that the standard for Windows too then? I was just taking a look at the Windows HIG and their examples all use ... (3 dot characters) instead of a proper ellipsis (…).

@mnquintana Looking at VS, it appears to use ellipses, I could be wrong though, it's a little hard to tell

It seems that it is a standard for Windows too:

Indicate a command that needs additional information (including a confirmation) by adding an ellipsis at the end of the label.

Source: https://msdn.microsoft.com/en-us/library/dn742392.aspx#usingellipses

@kubetz Right, that's the doc I linked to. But they don't make it explicit like Apple does that "ellipsis" means "the ellipsis character" and not 3 dots that look like an ellipsis, especially confusing because none of their examples use the ellipsis character.

@mnquintana Sorry for not noticing that link :blush: .
Isn't Word autocorrecting three dots into ellipsis character? I found on web peopel complaining that it does, but don't have it installed to verify it myself. If that it the case then it can be used as an argument for switching to "…"

That is the reason I named this branch "osx_ellipsis"… to understand what to do on others platforms It's another job ;-)

@isghe We looked into it a bit further, and it would be great if you could update the menus for Windows and Linux too. :smile:

@mnquintana I will, but… I cannot test by myself, because actually I don't have a Windows or Linux development platform.

I did also for Windows and Linux… but take care, I cannot test those modifications by myself right now.

I build atom on Windows with these changes and here is the screenshot of file menu:

So it looks fine.

Thanks for this! :boat:


Improve speed of loading

Compare https://www.povray.org/documentation/view/3.6.1/603/ for further information about the /NORESTORE flag

looks like a good idea to me

Thanks, getting back from holidays, will review/merge this week.


fix: add permissions to docker security build

Overview

fixes https://github.com/celestiaorg/celestia-app/issues/1378
Checklist

[ ] New and updated code has appropriate documentation
[ ] New and updated code has new and/or updated testing
[ ] Required CI checks are passing
[ ] Visual proof for any user facing features like CLI or documentation updates
[ ] Linked issues closed with keywords

good one @SweeXordious ! 🙌
looks like it's working:
https://github.com/celestiaorg/celestia-app/actions/runs/4167493697

@evan-forbes
if you have some spare time, please approve this one, it fixes the permissions issues.
Already tested in celestia-node : here
Actions executed: here
thanks in advance!

Matt is on his 2/19 OOO day :D this PR will age quite well :laughing: :laughing:


⚠ Bofrost.es has degraded performance

In 9ae55b9, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1618 ms

Resolved: Bofrost.es performance has improved in d130d02 after 6 minutes.


question: why monkey patch instead of maintain restx

Are these monkey patches not merged into the restx because there is no maintainer who will merge them?
Can you license these patches under something that doesn't have a viral GPL license?

Hi, yes because fixes are not merged. So I put them into one monkey patch package. Can you suggest fine license for this. I am totally new in this. Thank you for help :)

I consider the BSD License works for you. Closing issue.

Works for me! Thank you for getting this handled, and I will be accepting your monkey patches into my project. Cheers!


Add missing templates and order them

Missed in https://github.com/alphagov/static/pull/2554 😬

Not related to this PR, but related to the previous one: does omit_footer_links here  do anything though? 🤔
Had a look at the gem base template, and that flag isn't mentioned anywhere currently.

Not related to this PR, but related to the previous one: does omit_footer_links here do anything though? 🤔
Had a look at the gem base template, and that flag isn't mentioned anywhere currently.

Busted. No, it doesn't. Will add it in this PR, test the page again and ask for a re-review. Thank you!


Misleading information in the readme about SQLite concurrency

Wanted to call out that SQLite's "lack of concurrency" as it is called out in the readme:

no amount of asynchronicity will save you from SQLite3's serialized nature.

is a bit misleading. The serial nature is really a problem of the pairing of SQLite w/ JavaScript and doesn't exist in languages able to spawn threads. In languages that support threads, you can spawn many concurrent read connections and not be constrained to serial reads when in WAL mode.
This is also true for writes for those using SQLite's HCTree -- https://sqlite.org/hctree/doc/hctree/doc/hctree/index.html

no amount of asynchronicity will save you...

You could work around this in NodeJS by spawning dedicated worker threads for long-running reads.

https://github.com/WiseLibs/better-sqlite3/blob/master/docs/threads.md

@tantaman, Everything you said is true, but it's debatable which explanation is more misleading.
99.9% of users aren't going to be using HCTree, given that it's just a prototype, is probably not production ready, and requires a custom build of SQLite off the main branch.
It's true that SQLite can handle concurrent reads, but the entire transaction has to be read-only. Therefore, the "truly optimal" way of using SQLite would be to have a dedicated writer thread and many read-only threads, while using WAL mode. That's not at all what node-sqlite3 does (which is the de-facto comparison for this library). They just queue up everything into the Node.js thread pool, and run every statement one-by-one (not actually utilizing any concurrency), because they don't know which statements/transactions are read-only or not (in fact, determining this in a generic way is impossible, since extensions can introduce custom functions which write to the database). And of course, if you don't have many read-only transactions, then spawning those read-only threads would be a big waste.
As @m4heshd pointed out, we do support worker threads for this exact reason. I suppose we could add more documentation on how to parallelize read-only transactions, but my guess is that if you need this, you already know how to do it.

Maybe you can rephrase it make clear what asynchronicity refers to context. A more elegant version of the following:
-no amount of asynchronicity will save you from SQLite3's serialized nature
+no amount of asynchronicity (in sense of giving control back to the Node.js event loop) will save you from SQLite3's serialized nature

Thanks for the thoughtful replies. I do like @Prinzhorn's re-wording.


Make serialization of fractions keyed.

This also adds more tests and splits tests into their own classes.

Why? Mojang serializes a few vanilla stuff, like block positions, as non-keyed arrays. Serializing as array is usually cheaper than using keys, and this is not that hard to understand.

The primary worry is that streams are unordered so serializing may swap the numerator and denominator.

Java streams are ordered when the backing collection is an array or a list

but we're just doing IntStream.of or LongStream.of, and I'm not sure I can guarantee that's an array or list

Ah in the case of IntStream.of, it is ordered

Should be fine, then. We might still switch it to keyed anyway but we're not sure yet. Player gave a few words on it in the channel.

this been the fluid api im surprised this hasn't been bike shead'ed

Being keyed will be "more clear"


Script not working for x86_64-apple-darwin

I'm getting the following error when trying to install the package using yarn.
error Couldn't find package "<EMAIL_ADDRESS>required by "<EMAIL_ADDRESS>on the "npm" registry
I'm using x86_64-apple-darwin. I tried using export TARGET=x86_64-apple-darwin but no luck. Thank you.

Hey there,
I encountered the same error while trying to deploy to Vercel using yarn. It looks like the package<EMAIL_ADDRESS>required by<EMAIL_ADDRESS>isn't available on the npm registry for the x86_64-apple-darwin architecture.
I initially faced this issue too but later realized it was because astro-min doesn't work well with SSR (Server-Side Rendering) or hybrid outputs in Astro, especially when deploying with Vercel. Since I'm using pnpm and a Turbo repo, I had to double-check my Astro configuration. Make sure you have prerender: true set in your Astro components or that your project's build output is set to static. Adjusting these settings helped resolve the deployment issue on Vercel in my case.

I have the same problem, I have configured my Astro project in "static" mode, I'm using bun as package manger and Windows as OS (in case it influences something).
here's the error message in case it's helpful
Command failed with exit code 1: bun add<EMAIL_ADDRESS>bun add v1.1.3 (2615dc74)
 Resolving dependencies

error: package "@minify-html/node-linux-arm64" not found registry.npmjs.org/@minify-html%2fnode-linux-arm64 404

error: package "@minify-html/node-darwin-arm64" not found registry.npmjs.org/@minify-html%2fnode-darwin-arm64 404
 Resolved, downloaded and extracted [19]
error<EMAIL_ADDRESS>failed to resolve
error<EMAIL_ADDRESS>failed to resolve

Astro could not update your astro.config.js file safely.
Reason: Unable to install dependencies

It's working again, with v1.3.1 😎
The arm64 versions are somehow not available on npm registry, but just bun seems to request all optional packages. So right now it only won't build, if you actually have an apple arm device. I couldn't get a macos runner to build them, at least not yet.. and there will be some more improvement soon 😃
@shawnvanderveer I also noticed that, strange pnpm behaves diffrent on local machine and on vercel..

There is no @minify-html/node-darwin-arm64 package on npm. The package was mistakenly published as @minify-html/node-darwin-aarch64.(https://github.com/wilsonzlin/minify-html/issues/172)
However, the corrected version has not yet been released.
As a temporary workaround for arm64 macOS, add the following to your package.json (I use pnpm):
"@minify-html/node-darwin-arm64": "npm<EMAIL_ADDRESS>


Improve the contributor card design

Please describe the feature you want this project to offer
Trigger a pop-up that allows the user to customize the details to be added in the card.
Describe the solution you'd like
The users should be able to customize the card according to themselves and add the card to the contributors list
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context (Optional)
Add any other context or screenshots about the feature request here.

Hi @C-Harshul can you please share your  Flutter SDK Version, so that I can work on the issue
Regards,
Udit Kumar Swaroopa

Hi @C-Harshul can you please share your Flutter SDK Version, so that I can work on the issue
Regards,
Udit Kumar Swaroopa

The Flutter SDK version that I'm working on is 1.20.4 of the stable channel. Good luck contributing to this project

I am adding a bit finishing touch that is why I am taking some time, I hope you are not in a hurry

I am adding a bit finishing touch that is why I am taking some time, I hope you are not in a hurry

Not in a hurry but try to finish asap.


Tools installed by the deployment script don't work with the generated files

deploy.sh does for example npm install -g truffle, which installs the latest truffle version. But the latest truffle version comes with a newer web3 version than the one used in the createRealAccount.js for example.
Or the contracts are for Solidity 0.4.24, but the installed solc is 0.5.something.
You should either run your tools with freshly generated contracts (generated by fondu.io), check what works and what doesn't and change everything so it works again.
Or make all scripts and configs so that they use a fixed version that will also work in a year, even if it's not the latest version in a year anymore (for example, with truffle-config.js containing the following, solc 0.4.24 will be used by default:
compilers: {
    solc: {
      version: "0.4.24"
    }
},

)

Another option would be to provider a Docker image with all the right dependencies and tools installed, so only the generated contract files would have to be volume mapped into the container and that's it. That would actually be the easiest solution.

I'll probably just update the dependencies, installing Docker for such a small script might be an overkill for a general user

Looks like I've fixed it. Thank you for the report!


[WIP] Retrieve VMware Storage Profile Inventory

Purpose or Intent
Retrieve a list of VMware Storage Profiles from MiqPbmInventory using the pbmProfilesByUid method.
Links

https://www.vmware.com/support/developer/vc-sdk/spbm-sdk/

Depends on: https://github.com/ManageIQ/manageiq/pull/9604

:+1: once #9604 is merged.

Tested and LGTM

@miq-bot remove-label wip

@agrare darga yes or no?

this is new feature and require migration, so no.

@chessbyte darga/no


Global property substitution

This PR adds a concept of global properties, which are user-defined properties in the scene file that can be substituted for values elsewhere in the file. This is useful for values that are referenced in multiple places in the scene file, and would benefit from a centralized definition for easier editing. Examples include common colors, language preferences, or visibility flags used to tweak styles.
Global properties are scoped as everything under the top-level global block of the scene file, including nested properties:
global:
   labels: true # label visibility flag
   language: name:en # preferred language for label display
   colors: # group common colors together
      color1: red
      color2: green
      color3: blue

These properties can then be substituted elsewhere in the scene file with global. syntax (included nested dot notation, global.group.property), and are also available in JS function filters and properties under the same syntax:
layers:
   road-labels:
      data: { ... }
      visible: global.labels
      draw:
         text:
            text_source: [global.language, name] # display labels in preferred language, fallback to default name
            ...

   point-labels:
      data: { ... }
      draw:
         text:
            text_source: |
               function() {
                  // Make a compound label with "Preferred Name (Local Name)"
                  var preferred = feature[global.language];
                  if (preferred && preferred !== feature.name) {
                     return preferred + '\n(' + feature.name + ')';
                  }
                  return feature.name;
               }

   landuse:
      data: { ... }
      draw:
         polygons:
            color: global.colors.color2 # nested property
            ...

Global properties are similar to YAML's native anchor/reference capability, with the most significant difference being that they are applied after YAML parsing but before other Tangram parsing. This means we can:

Alter them programmatically and rebuild geometry. For example, this enables a dynamic language preference setting. In the example above, the global.language property could be hook up to a language selector dropdown or app preference.
Import and optionally override scene properties once we have scene import functionality (e.g. a base scene A could define properties which are then referenced by a scene B that has an import: A). This is outside the scope of current Tangram functionality, but coming soon.

Yeah, regarding the issue of name collision between built-in and user-provided values: It makes some sense to me to create a separate top-level element just for user-provided values. This prevents collision with any of the scene properties that have built-in semantics (background, animated, and any future properties we add). And looking at the number of anchored values in eraser-map (which this mechanism would presumably replace) the collection of custom properties could be quite large, probably large enough to merit its own top-level element simply for organizational purposes.
If we were to do this, the challenge would be giving this element a name that is as succinct and intuitive as 'scene'. Possibilities: global, prefs, my, meta

I would probably lean towards global. We're using it for shader blocks, so the term is in use (in a different context, which could be confusing, though we have color for both draw params and shader blocks so that's... consistent anyway :)

vars: maybe some benefit in sounding like the new CSS Variables (but they use var keyword and can cause confusion with JS var inside functions anyway).
props: I don't generally like abbreviations, and we already have layer-level properties
my: perl flashbacks
meta: these don't actually seem like metadata
_: could put them in a YAML-safe character, like jquery or underscore, succinct (_.language) but probably arcane syntax wise (and we can't use $, which is probably easiest to understand, because it's already used for $zoom and $layer special properties)
settings: too long?

@blair1618 switching to global

(updated PR description to reflect global syntax)

@blair1618 @tallytalwar also moved ahead with deprecating layer-specific properties, as we concluded (we think anyway :) that anything their scope of behavior can be accomplished with the new global properties.
@meetar we will need to convert some styles that used layer properties, like the Explorer, to use this new syntax. Should be a straightforward change though (and less duplicated code!).


Add the puppet template to Khaleesi

Need to add the puppet template to Khaleesi with reduced timer on how often the agent checks in.  Should set it to 10 minutes as I have found thats what works best when using Khaleesi to rebuild nodes and wait for them to check in.

Completed this by adding template and also changes the setting in foreman.  These changes were merged into khaleesi branch -> opnfv


Bugs

This is a lighter implementation of previous, and takes an intermediate step of abstracting out functions to retrieve metadata. This would allow us to more easily play with metadata without affecting the functions, and to encourage dependent themes and functions to start not retrieving the data directly.

thanks!


TextSize support

Text size is not supported in RichText Editor

Support for AbsoluteSizeSpan and RelativeSizeSpan are in there as of 0.4.0.


keep-alive and router-view cause subpages to render repeatedly

Version
4.1.6
Reproduction link
codesandbox.io
Steps to reproduce
This is my demo： https://codesandbox.io/s/vue3-teleport-forked-61bxxv?file=/src/page1/index.vue
After entering the page, click '111' to jump to another page, and you can see that the 'onMounted' method of this page is triggered twice
What is expected?
Subpages are only rendered once
What is actually happening?
Subpages are only rendered multiple times

Duplicate of https://github.com/vuejs/router/issues/626

Please, search existing issues and pull requests before opening issues, it saves maintainers a lot of time 🙏


Updates to AgUnit based on StatLight breaking API changes.

I've made a number of changes to StatLight that broke how AgUnit was interacting with it. However one change I made that should help agunit is the InputOptions class. Now as statlight versions and adds options, I an put in sane defaults that should hopfully keep AgUnit from having to update it's interaction with StatLight much.

Don't apply any of this yet. I'm ripping StatLight apart and am not sure how it will affect AgUnit yet.

I tried upgrading AgUnit to use the latest StatLight, but the major refactorings makes it a less than trivial task.  Any plans of getting AgUnit running again for SL5 support?

Did you take a look at the changes I have in this pull request? If you apply those changes, it should get you going.


No data for most panels in Grafana dashboard

Hi team,
First of all I'd like to thank you for creating such a great project to make deploying ClickHouse in K8S a simple thing.
I followed the instructions to setup the Grafana dashboard, but I found that there's no data for most of the panels, like this:

while some panels do have data:

do you have some clues where I made some mistakes? thank you.

I can see there's some error log in the metric-exporter container like this:
W0702 10:24:40.820220       1 exporter.go:210] Error querying system.metrics for host chi-dmp-data-ch-ch01-2-1.clickhouse-operator.svc.cluster.local err: dial tcp <IP_ADDRESS>:8123: connect: connection refused
E0702 10:24:40.825501       1 connection.go:140] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-dmp-data-ch-ch01-2-1.clickhouse-operator.svc.cluster.local:8123/) dial tcp <IP_ADDRESS>:8123: connect: connection refused
...
operator.svc.cluster.local err: dial tcp: lookup chi-dmp-data-ch-ch01-2-0.clickhouse-operator.svc.cluster.local on <IP_ADDRESS>:53: no such host
E0702 10:23:40.831823       1 connection.go:140] QueryContext():FAILED Query(http://clickhouse_operator:***@chi-dmp-data-ch-ch01-2-0.clickhouse-operator.svc.cluster.local:8123/) dial tcp: lookup chi-dmp-data-ch-ch01-2-0.clickhouse-operator.svc.cluster.local on <IP_ADDRESS>:53: no such host

seems like there's some connection issue and dns lookup issue, but I can ping those host in another pod which is also in the cluster:
#nc -zv chi-dmp-data-ch-ch01-2-1.clickhouse-operator.svc.cluster.local 8123
Connection to chi-dmp-data-ch-ch01-2-1.clickhouse-operator.svc.cluster.local (<IP_ADDRESS>) 8123 port [tcp/*] succeeded!

# nc -zv chi-dmp-data-ch-ch01-2-0.clickhouse-operator.svc.cluster.local 8123
Connection to chi-dmp-data-ch-ch01-2-0.clickhouse-operator.svc.cluster.local (<IP_ADDRESS>) 8123 port [tcp/*] succeeded!

any thoughts on this?

Do you use service mesh like istio?

Hi @Slach , no I don’t use service mesh

could you share
kubectl exec -n clickhouse-operator clickhouse-operator-XXX -c metrics-exporter -- curl -vvv http:// chi-dmp-data-ch-ch01-2-1.clickhouse-operator.svc.cluster.local:8123/ping

-XXX it's unique name for pod where run clickhouse-operator

could you share
kubectl exec -n clickhouse-operator clickhouse-operator-XXX -c metrics-exporter -- curl -vvv http:// chi-dmp-data-ch-ch01-2-1.clickhouse-operator.svc.cluster.local:8123/ping

-XXX it's unique name for pod where run clickhouse-operator

$ kubectl exec -n clickhouse-operator clickhouse-operator-7c666f95cd-6bqmz -c metrics-exporter -- curl -vvv http:// chi-dmp-data-ch-ch01-2-1.clickhouse-operator.svc.cluster.local:8123/ping
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying <IP_ADDRESS>:8123...
* Connected to chi-dmp-data-ch-ch01-2-1.clickhouse-operator.svc.cluster.local (<IP_ADDRESS>) port 8123 (#0)
> GET /ping HTTP/1.1
> Host: chi-dmp-data-ch-ch01-2-1.clickhouse-operator.svc.cluster.local:8123
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon, 03 Jul 2023 12:26:02 GMT
< Connection: Keep-Alive
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Keep-Alive: timeout=3
< X-ClickHouse-Summary: {"read_rows":"0","read_bytes":"0","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"0","result_bytes":"0"}
<
{ [9 bytes data]
100     4    0     4    0     0    697      0 --:--:-- --:--:-- --:--:--   Ok.
800
* Connection #0 to host chi-dmp-data-ch-ch01-2-1.clickhouse-operator.svc.cluster.local left intact

ok, now try to fetch metrics inside metrics-exporter container
kubectl exec -n clickhouse-operator clickhouse-operator-7c666f95cd-6bqmz -c metrics-exporter -- curl -vvv http://localhost:8888/metrics | grep fetch

doesn't see non zero fetch errors, so all metrics shall be scraped from prometheus side
could you check grafana again?

still no data for most panels

ok. let's check from prometheus side
how do you install prometheus?
could you share?
kubectl get pods --all-namespaces | grep prometheus

we install the prometheus by using the prometheus operator:
$ kubectl get pods --all-namespaces | grep prometheus
prometheus-operator   alertmanager-monitoring-alertmanager-0            2/2     Running   1 (32h ago)   32h
prometheus-operator   alertmanager-monitoring-alertmanager-1            2/2     Running   1 (32h ago)   32h
prometheus-operator   monitoring-kube-state-metrics-76c994748b-vhpf6    1/1     Running   0             32h
prometheus-operator   monitoring-operator-644cc87cc6-q4cfn              1/1     Running   0             32h
prometheus-operator   node-exporter-5ssp5                               1/1     Running   0             32h
prometheus-operator   node-exporter-dgj7d                               1/1     Running   0             32h
prometheus-operator   node-exporter-n8bm5                               1/1     Running   0             32h
prometheus-operator   node-exporter-rqzmc                               1/1     Running   0             32h
prometheus-operator   prometheus-monitoring-prometheus-0                2/2     Running   0             32h
prometheus-operator   prometheus-monitoring-prometheus-1                2/2     Running   0             32h

ok. let's check your prometheus
could you run
kubectl port-forward -n prometheus-operator pod/prometheus-monitoring-prometheus-0 9090:9090

open http://localhost:9090/targets
Do you see clickhouse nodes as target? Are clickhouse targets UP scraping success?

Hi @Slach,
It seems the issues is caused by the prometheus operator, after I re-install the prometheus-operator, it works fine now.
thank you so much for your help!


GraphQL sitemap now parses personalize data from site queries

Adds support for personalize variants data into GraphQL sitemap service in nextjs project
Description / Motivation
Site routes returned by GraphQL can have variants data included. Variants will now be included in site paths, when provided.
Testing Details
One unit test added
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

In the PBI I see that we should use segmentId, I suppose that in the API it wasn't changed to the variantId, doesn't it?


Surface multi-currency expenses in the interface

Part of #5146
@opencollective/design to define a pattern / generic style for this one, but we can already start prototyping based on the proposal below.

[x] On the Expense object, we should expose two fields:

Expense.amount: amount in expense.currency
Expense.amountInAccountCurrency: amount in collective.currency

[x] In the expenses list, expense page, and on the host admin, always show the expense amount with its target currency along with an estimate of the fees

[x] Adapt the "pay expense" modal

[ ] Update the way we compute "Page total" for multi-currencies
[ ] (Bonus) Adapt the transaction details

@Betree : I can help with this one if there's no objections. 🙂

@SudharakaP I already started digging into this one, we can have a call to better dispatch the work if your plate is empty and you need more issues to work on.

This is done, I've extracted the last bit to https://github.com/opencollective/opencollective/issues/5384


[NTR/PRKAG2] PRKAG2-related cardiomyopathy

Preferred gene-related syndrome label
PRKAG2-related cardiomyopathy
Synonyms
PRKAG2 syndrome, PRKAG2 cardiac syndrome, PRKAG2 cardiomyopathy
Parent term (use OLS, or your favorite ontology browser)
MONDO:0005217
Definition
A metabolic heart condition characterized by variable cardiac hypertrophy, ventricular pre-excitation, and aberrant glycogen storage in the cardiac tissue due to a pathogenic variant in PRKAG2 that results in a net anabolic effect in cardiac cells.
**Definition source (Please give PubMed ID, if applicable, in format PMID:#######)
PMID:29298659
Children terms (if applicable) Should any existing terms be re-classified as children underneath this new proposed term?
MONDO:0010946
MONDO:0009867
MONDO:0008685
Your nano-attribution (ORCID) or URL for a working group
If you don't have an ORCID, you can sign up for one here
https://orcid.org/0009-0007-0138-2054
This request is being made on behalf of the ClinGen Hereditary Cardiovascular Disease GCEP.
@courtneythaxton
@nicolevasilevsky
@ahamosh

thanks @awaddell01 - we'll address this as soon as we can

I tried this with curategpt and discovered that the term already existed
http://purl.obolibrary.org/obo/MONDO_0010946
I think adding synonyms to this term as requested should close this

@cmungall The HCVD GCEP would prefer if the new name could replace the term from your message above, along with changing the definition rather than adding as a synonym if possible. I'll add the link for the ClinGen webpage for the group below:
https://www.clinicalgenome.org/affiliation/40104/

II think they want to move this up the hierarchy to cardiomyopathy as opposed to just hypertrophic cardiomyopathy.  Is that correct?