fix(mockGrpcUnary): ensure route registration is awaited to avoid race conditions where a route might be attempted before registration completes

BREAKING CHANGE:
mockGrpcUnary is now an async function, and so must be awaited to get the MockedGrpcCall

I still need to confirm this is actually a real issue, but it is definitely strange that we weren't waiting for registration to resolve before. I'll test the branch before this should go in


Unix socket connection not working

Tried all possibilities of connections strings to the DB but still unsuccessful deploying to Google Cloud (App Engine/Cloud Run). Connection strings are replaced with placeholders for security.
Step #1 - "builder": $ DEBUG=* RUST_BACKTRACE=full yarn lift:up && yarn build
Step #1 - "builder": $ prisma2 lift up
Step #1 - "builder": info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Step #1 - "builder": info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Step #1 - "builder":
Step #1 - "builder": INFO `yarn_run_gcp_build` had stderr output:
Step #1 - "builder": 2019-09-09T11:44:30.680Z LiftEngine:rpc SENDING RPC CALL { id: 1,
Step #1 - "builder": jsonrpc: '2.0',
Step #1 - "builder": method: 'listMigrations',
Step #1 - "builder": params:
Step #1 - "builder": { projectInfo: '',
Step #1 - "builder": sourceConfig:
Step #1 - "builder": 'datasource db {\n provider = "postgresql"\n url = "postgresql://<usr>:<pass>@<ip>/<db>?schema=public"\n}\n\ngenerator photon {\n provider = "photonjs"\n}\n\ngenerator nexus_prisma {\n provider = "nexus-prisma"\n}\n\nmodel File {\n id String @id @default(cuid())\n name String\n project Project\n url String\n type String\n}\n\nmodel User {\n id String @id @default(cuid())\n authId String @unique\n email String @unique\n projects Project[]\n name String\n profileImg String?\n postVersions PostVersion[]\n faqs Faq[]\n}\n\nmodel Company {\n id String @id @default(cuid())\n name String\n projects Project[]\n}\n\nmodel Project {\n id String @id @default(cuid())\n name String\n company Company\n webUrl String?\n logo String\n logoSmall String\n postCategories PostCategory[]\n faqCategories FaqCategory[]\n posts PostVersion[]\n faqs Faq[]\n admins User[]\n files File[]\n}\n\nmodel Post {\n id String @id @default(cuid())\n draftedVersion PostVersion @relation(name: "draft")\n publishedVersion PostVersion @relation(name: "published")\n}\n\nmodel PostVersion {\n id String @id @default(cuid())\n draftedPost Post? @relation(name: "draft")\n publishedPost Post? @relation(name: "published")\n title String\n text String\n language String\n categories PostCategory[]\n author User\n published Boolean\n project Project\n ogTitle String?\n ogDescription String?\n createdAt DateTime\n updatedAt DateTime @updatedAt\n}\n\nmodel Faq {\n id String @id @default(cuid())\n question String\n answer String\n category FaqCategory?\n published Boolean\n language String\n project Project\n author User\n createdAt DateTime @default(now())\n updatedAt DateTime @updatedAt\n}\n\nmodel PostCategory {\n id String @id @default(cuid())\n name String\n project Project\n posts PostVersion[]\n}\n\nmodel FaqCategory {\n id String @id @default(cuid())\n name String\n project Project\n faqs Faq[]\n}' } }
Step #1 - "builder": 2019-09-09T11:44:33.703Z LiftEngine:stderr thread 'main' panicked at 'Box<Any>', migration-engine/core/src/main.rs:55:23
Step #1 - "builder": 2019-09-09T11:44:33.703Z LiftEngine:stderr stack backtrace:
Step #1 - "builder": 2019-09-09T11:44:33.730Z LiftEngine:stderr 0: backtrace::backtrace::libunwind::trace
Step #1 - "builder": 2019-09-09T11:44:33.730Z LiftEngine:stderr at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/libunwind.rs:88
Step #1 - "builder": 2019-09-09T11:44:33.731Z LiftEngine:stderr 1: backtrace::backtrace::trace_unsynchronized
Step #1 - "builder": 2019-09-09T11:44:33.731Z LiftEngine:stderr at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/mod.rs:66
Step #1 - "builder": 2019-09-09T11:44:33.732Z LiftEngine:stderr 2: std::sys_common::backtrace::_print
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/sys_common/backtrace.rs:47
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 3: std::sys_common::backtrace::print
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/sys_common/backtrace.rs:36
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 4: std::panicking::default_hook::{{closure}}
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/panicking.rs:200
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 5: std::panicking::default_hook
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/panicking.rs:214
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 6: std::panicking::rust_panic_with_hook
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/panicking.rs:477
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 7: std::panicking::begin_panic
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 8: migration_engine::main
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 9: std::rt::lang_start::{{closure}}
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 10: std::rt::lang_start_internal::{{closure}}
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/rt.rs:49
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 11: std::panicking::try::do_call
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr at src/libstd/panicking.rs:296
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 12: __rust_maybe_catch_panic
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr at src/libpanic_unwind/lib.rs:82
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 13: std::panicking::try
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr at src/libstd/panicking.rs:275
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 14: std::panic::catch_unwind
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr at src/libstd/panic.rs:394
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 15: std::rt::lang_start_internal
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr at src/libstd/rt.rs:48
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 16: main
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 17: __libc_start_main
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 18: _start
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Step #1 - "builder": Error: Error: Error in migration engine: thread 'main' panicked at 'Box<Any>', migration-engine/core/src/main.rs:55:23
Step #1 - "builder": stack backtrace:
Step #1 - "builder": 0: backtrace::backtrace::libunwind::trace
Step #1 - "builder": at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/libunwind.rs:88
Step #1 - "builder": 1: backtrace::backtrace::trace_unsynchronized
Step #1 - "builder": at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/mod.rs:66
Step #1 - "builder": 2: std::sys_common::backtrace::_print
Step #1 - "builder": at src/libstd/sys_common/backtrace.rs:47
Step #1 - "builder": 3: std::sys_common::backtrace::print
Step #1 - "builder": at src/libstd/sys_common/backtrace.rs:36
Step #1 - "builder": 4: std::panicking::default_hook::{{closure}}
Step #1 - "builder": at src/libstd/panicking.rs:200
Step #1 - "builder": 5: std::panicking::default_hook
Step #1 - "builder": at src/libstd/panicking.rs:214
Step #1 - "builder": 6: std::panicking::rust_panic_with_hook
Step #1 - "builder": at src/libstd/panicking.rs:477
Step #1 - "builder": 7: std::panicking::begin_panic
Step #1 - "builder": 8: migration_engine::main
Step #1 - "builder": 9: std::rt::lang_start::{{closure}}
Step #1 - "builder": 10: std::rt::lang_start_internal::{{closure}}
Step #1 - "builder": at src/libstd/rt.rs:49
Step #1 - "builder": 11: std::panicking::try::do_call
Step #1 - "builder": at src/libstd/panicking.rs:296
Step #1 - "builder": 12: __rust_maybe_catch_panic
Step #1 - "builder": at src/libpanic_unwind/lib.rs:82
Step #1 - "builder": 13: std::panicking::try
Step #1 - "builder": at src/libstd/panicking.rs:275
Step #1 - "builder": 14: std::panic::catch_unwind
Step #1 - "builder": at src/libstd/panic.rs:394
Step #1 - "builder": 15: std::rt::lang_start_internal
Step #1 - "builder": at src/libstd/rt.rs:48
Step #1 - "builder": 16: main
Step #1 - "builder": 17: __libc_start_main
Step #1 - "builder": 18: _start
Step #1 - "builder": note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Step #1 - "builder": at ChildProcess.child.on (/workspace/node_modules/prisma2/build/index.js:107250:37)
Step #1 - "builder": at ChildProcess.emit (events.js:198:13)
Step #1 - "builder": at ChildProcess.EventEmitter.emit (domain.js:448:20)
Step #1 - "builder": at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
Step #1 - "builder": error Command failed with exit code 1.
Step #1 - "builder": error Command failed with exit code 1.
Step #1 - "builder":
Step #1 - "builder": ERROR error: `yarn_run_gcp_build` returned code: 1
Step #1 - "builder": INFO yarn_run_gcp_build took 5 seconds
Step #1 - "builder": INFO build process for FTL image took 44 seconds
Step #1 - "builder": INFO full build took 44 seconds
Step #1 - "builder": ERROR `yarn_run_gcp_build` had stderr output:
Step #1 - "builder": 2019-09-09T11:44:30.680Z LiftEngine:rpc SENDING RPC CALL { id: 1,
Step #1 - "builder": jsonrpc: '2.0',
Step #1 - "builder": method: 'listMigrations',
Step #1 - "builder": params:
Step #1 - "builder": { projectInfo: '',
Step #1 - "builder": sourceConfig:
Step #1 - "builder": 'datasource db {\n provider = "postgresql"\n url = "postgresql://<user>:<pass>@<ip>/<db>?schema=public"\n}\n\ngenerator photon {\n provider = "photonjs"\n}\n\ngenerator nexus_prisma {\n provider = "nexus-prisma"\n}\n\nmodel File {\n id String @id @default(cuid())\n name String\n project Project\n url String\n type String\n}\n\nmodel User {\n id String @id @default(cuid())\n authId String @unique\n email String @unique\n projects Project[]\n name String\n profileImg String?\n postVersions PostVersion[]\n faqs Faq[]\n}\n\nmodel Company {\n id String @id @default(cuid())\n name String\n projects Project[]\n}\n\nmodel Project {\n id String @id @default(cuid())\n name String\n company Company\n webUrl String?\n logo String\n logoSmall String\n postCategories PostCategory[]\n faqCategories FaqCategory[]\n posts PostVersion[]\n faqs Faq[]\n admins User[]\n files File[]\n}\n\nmodel Post {\n id String @id @default(cuid())\n draftedVersion PostVersion @relation(name: "draft")\n publishedVersion PostVersion @relation(name: "published")\n}\n\nmodel PostVersion {\n id String @id @default(cuid())\n draftedPost Post? @relation(name: "draft")\n publishedPost Post? @relation(name: "published")\n title String\n text String\n language String\n categories PostCategory[]\n author User\n published Boolean\n project Project\n ogTitle String?\n ogDescription String?\n createdAt DateTime\n updatedAt DateTime @updatedAt\n}\n\nmodel Faq {\n id String @id @default(cuid())\n question String\n answer String\n category FaqCategory?\n published Boolean\n language String\n project Project\n author User\n createdAt DateTime @default(now())\n updatedAt DateTime @updatedAt\n}\n\nmodel PostCategory {\n id String @id @default(cuid())\n name String\n project Project\n posts PostVersion[]\n}\n\nmodel FaqCategory {\n id String @id @default(cuid())\n name String\n project Project\n faqs Faq[]\n}' } }
Step #1 - "builder": 2019-09-09T11:44:33.703Z LiftEngine:stderr thread 'main' panicked at 'Box<Any>', migration-engine/core/src/main.rs:55:23
Step #1 - "builder": 2019-09-09T11:44:33.703Z LiftEngine:stderr stack backtrace:
Step #1 - "builder": 2019-09-09T11:44:33.730Z LiftEngine:stderr 0: backtrace::backtrace::libunwind::trace
Step #1 - "builder": 2019-09-09T11:44:33.730Z LiftEngine:stderr at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/libunwind.rs:88
Step #1 - "builder": 2019-09-09T11:44:33.731Z LiftEngine:stderr 1: backtrace::backtrace::trace_unsynchronized
Step #1 - "builder": 2019-09-09T11:44:33.731Z LiftEngine:stderr at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/mod.rs:66
Step #1 - "builder": 2019-09-09T11:44:33.732Z LiftEngine:stderr 2: std::sys_common::backtrace::_print
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/sys_common/backtrace.rs:47
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 3: std::sys_common::backtrace::print
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/sys_common/backtrace.rs:36
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 4: std::panicking::default_hook::{{closure}}
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/panicking.rs:200
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 5: std::panicking::default_hook
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/panicking.rs:214
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 6: std::panicking::rust_panic_with_hook
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/panicking.rs:477
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 7: std::panicking::begin_panic
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 8: migration_engine::main
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 9: std::rt::lang_start::{{closure}}
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 10: std::rt::lang_start_internal::{{closure}}
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr at src/libstd/rt.rs:49
Step #1 - "builder": 2019-09-09T11:44:33.733Z LiftEngine:stderr 11: std::panicking::try::do_call
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr at src/libstd/panicking.rs:296
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 12: __rust_maybe_catch_panic
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr at src/libpanic_unwind/lib.rs:82
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 13: std::panicking::try
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr at src/libstd/panicking.rs:275
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 14: std::panic::catch_unwind
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr at src/libstd/panic.rs:394
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 15: std::rt::lang_start_internal
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr at src/libstd/rt.rs:48
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 16: main
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 17: __libc_start_main
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr 18: _start
Step #1 - "builder": 2019-09-09T11:44:33.734Z LiftEngine:stderr note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Step #1 - "builder": Error: Error: Error in migration engine: thread 'main' panicked at 'Box<Any>', migration-engine/core/src/main.rs:55:23
Step #1 - "builder": stack backtrace:
Step #1 - "builder": 0: backtrace::backtrace::libunwind::trace
Step #1 - "builder": at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/libunwind.rs:88
Step #1 - "builder": 1: backtrace::backtrace::trace_unsynchronized
Step #1 - "builder": at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/mod.rs:66
Step #1 - "builder": 2: std::sys_common::backtrace::_print
Step #1 - "builder": at src/libstd/sys_common/backtrace.rs:47
Step #1 - "builder": 3: std::sys_common::backtrace::print
Step #1 - "builder": at src/libstd/sys_common/backtrace.rs:36
Step #1 - "builder": 4: std::panicking::default_hook::{{closure}}
Step #1 - "builder": at src/libstd/panicking.rs:200
Step #1 - "builder": 5: std::panicking::default_hook
Step #1 - "builder": at src/libstd/panicking.rs:214
Step #1 - "builder": 6: std::panicking::rust_panic_with_hook
Step #1 - "builder": at src/libstd/panicking.rs:477
Step #1 - "builder": 7: std::panicking::begin_panic
Step #1 - "builder": 8: migration_engine::main
Step #1 - "builder": 9: std::rt::lang_start::{{closure}}
Step #1 - "builder": 10: std::rt::lang_start_internal::{{closure}}
Step #1 - "builder": at src/libstd/rt.rs:49
Step #1 - "builder": 11: std::panicking::try::do_call
Step #1 - "builder": at src/libstd/panicking.rs:296
Step #1 - "builder": 12: __rust_maybe_catch_panic
Step #1 - "builder": at src/libpanic_unwind/lib.rs:82
Step #1 - "builder": 13: std::panicking::try
Step #1 - "builder": at src/libstd/panicking.rs:275
Step #1 - "builder": 14: std::panic::catch_unwind
Step #1 - "builder": at src/libstd/panic.rs:394
Step #1 - "builder": 15: std::rt::lang_start_internal
Step #1 - "builder": at src/libstd/rt.rs:48
Step #1 - "builder": 16: main
Step #1 - "builder": 17: __libc_start_main
Step #1 - "builder": 18: _start
Step #1 - "builder": note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Step #1 - "builder": at ChildProcess.child.on (/workspace/node_modules/prisma2/build/index.js:107250:37)
Step #1 - "builder": at ChildProcess.emit (events.js:198:13)
Step #1 - "builder": at ChildProcess.EventEmitter.emit (domain.js:448:20)
Step #1 - "builder": at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
Step #1 - "builder": error Command failed with exit code 1.
Step #1 - "builder": error Command failed with exit code 1.
Step #1 - "builder":
Step #1 - "builder": error: `yarn_run_gcp_build` returned code: 1
Step #1 - "builder": Traceback (most recent call last):
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
Step #1 - "builder": "__main__", fname, loader, pkg_name)
Step #1 - "builder": File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
Step #1 - "builder": exec code in run_globals
Step #1 - "builder": File "/usr/local/bin/ftl.par/__main__.py", line 65, in <module>
Step #1 - "builder": File "/usr/local/bin/ftl.par/__main__.py", line 57, in main
Step #1 - "builder": File "/usr/local/bin/ftl.par/__main__/ftl/common/ftl_error.py", line 58, in UserErrorHandler
Step #1 - "builder": IOError: [Errno 2] No such file or directory: '""/output'
Finished Step #1 - "builder"
ERROR
ERROR: build step 1 "gcr.io/gae-runtimes/nodejs10_app_builder:nodejs10_20190602_10_16_0_RC00" failed: exit status 1

The error message looks very similar to https://github.com/prisma/prisma2/issues/381 which got recently closed.
Can you please tell me where the database is hosted?

Also, please make sure you are on the latest version of prisma2

This database is hosted on Google Cloud (Cloud SQL). I managed to make it work by connecting to a public IP. Sockets did not work, now trying to use SSL, but when I add it to the db connection string (sslcert, sslkey) it does not accept it

Yes, right now we don't support those options.
Can you open an issue(ie feature request) for that 🙏

@pantharshit00  What options do you mean? Sockets or SSL?

We don't support sslcert and sslkey options right now.

But sockets should work, is that correct?

There is one more weird thing happening. Prisma init via sockets works, but migration (prisma2 lift up) fails

@pantharshit00 Do you have some updates on this?

I think sockets are not supported in the Rust implementation right now.
The CLI uses the nodejs driver to do introspection right now.

@pantharshit00 But in the prism2 init flow it looks flow like it is connecting through socket. Is this going to be solved somehow? Sockets are widely used in Google Cloud and other Cloud providers for connecting to databases.

In the docs of Prisma 2 it says to use any supported connection string and Rust's Postgresql implementation supports it https://docs.rs/postgres/0.15.2/postgres/struct.Connection.html#impl

I need to confirm this with the backend team. I don't work on the Rust part of the codebase, it might be a specific issue.
I will try to make a concrete reproduction for this.

error when start after init
➜  prisma prisma2 lift save Error: Error in migration engine: thread 'tokio-runtime-worker-1' panicked at 'get data_type', src/libcore/option.rs:1034:5 stack backtrace:    0: std::panicking::default_hook::{{closure}}    1: std::panicking::default_hook    2: std::panicking::rust_panic_with_hook    3: std::panicking::continue_panic_fmt    4: rust_begin_unwind    5: core::panicking::panic_fmt    6: core::option::expect_failed    7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once    8: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T,I>>::from_iter    9: sql_schema_describer::mysql::SqlSchemaDescriber::get_table   10: <core::iter::adapters::Map<I,F> as core::iter::traits::iterator::Iterator>::fold   11: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T,I>>::from_iter   12: <sql_schema_describer::mysql::SqlSchemaDescriber as sql_schema_describer::SqlSchemaDescriberBackend>::describe   13: <sql_migration_connector::sql_database_migration_inferrer::SqlDatabaseMigrationInferrer as migration_connector::database_migration_inferrer::DatabaseMigrationInferrer<sql_migration_connector::sql_migration::SqlMigration>>::infer   14: <migration_core::api::MigrationApi<C,D> as migration_core::api::GenericApi>::infer_migration_steps   15: migration_core::api::rpc::RpcApi::create_sync_handler   16: tokio_executor::enter::exit   17: tokio_threadpool::blocking::blocking   18: <futures::future::lazy::Lazy<F,R> as futures::future::Future>::poll   19: futures::future::chain::Chain<A,B,C>::poll   20: <futures::future::then::Then<A,B,F> as futures::future::Future>::poll   21: <futures::future::lazy::Lazy<F,R> as futures::future::Future>::poll   22: futures::future::chain::Chain<A,B,C>::poll   23: <futures::future::then::Then<A,B,F> as futures::future::Future>::poll   24: <futures::future::map::Map<A,F> as futures::future::Future>::poll   25: <futures::future::either::Either<A,B> as futures::future::Future>::poll   26: <futures::future::map::Map<A,F> as futures::future::Future>::poll   27: <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll   28: <futures::stream::and_then::AndThen<S,F,U> as futures::stream::Stream>::poll   29: <futures::stream::forward::Forward<T,U> as futures::future::Future>::poll   30: <futures::future::map::Map<A,F> as futures::future::Future>::poll   31: <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll   32: futures::task_impl::std::set   33: std::panicking::try::do_call   34: __rust_maybe_catch_panic   35: tokio_threadpool::task::Task::run   36: tokio_threadpool::worker::Worker::run_task   37: tokio_threadpool::worker::Worker::run   38: std::thread::local::LocalKey<T>::with   39: std::thread::local::LocalKey<T>::with   40: tokio_reactor::with_default   41: tokio::runtime::threadpool::builder::Builder::build::{{closure}}   42: std::thread::local::LocalKey<T>::with   43: std::thread::local::LocalKey<T>::with note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.

error when start after init
➜  prisma prisma2 lift save
Error: Error in migration engine: thread 'tokio-runtime-worker-1' panicked at 'get data_type', src/libcore/option.rs:1034:5
stack backtrace:
   0: std::panicking::default_hook::{{closure}}
   1: std::panicking::default_hook
   2: std::panicking::rust_panic_with_hook
   3: std::panicking::continue_panic_fmt
   4: rust_begin_unwind
   5: core::panicking::panic_fmt
   6: core::option::expect_failed
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &mut F>::call_once
   8: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T,I>>::from_iter
   9: sql_schema_describer::mysql::SqlSchemaDescriber::get_table
  10: <core::iter::adapters::Map<I,F> as core::iter::traits::iterator::Iterator>::fold
  11: <alloc::vec::Vec<T> as alloc::vec::SpecExtend<T,I>>::from_iter
  12: <sql_schema_describer::mysql::SqlSchemaDescriber as sql_schema_describer::SqlSchemaDescriberBackend>::describe
  13: <sql_migration_connector::sql_database_migration_inferrer::SqlDatabaseMigrationInferrer as migration_connector::database_migration_inferrer::DatabaseMigrationInferrer<sql_migration_connector::sql_migration::SqlMigration>>::infer
  14: <migration_core::api::MigrationApi<C,D> as migration_core::api::GenericApi>::infer_migration_steps
  15: migration_core::api::rpc::RpcApi::create_sync_handler
  16: tokio_executor::enter::exit
  17: tokio_threadpool::blocking::blocking
  18: <futures::future::lazy::Lazy<F,R> as futures::future::Future>::poll
  19: futures::future::chain::Chain<A,B,C>::poll
  20: <futures::future::then::Then<A,B,F> as futures::future::Future>::poll
  21: <futures::future::lazy::Lazy<F,R> as futures::future::Future>::poll
  22: futures::future::chain::Chain<A,B,C>::poll
  23: <futures::future::then::Then<A,B,F> as futures::future::Future>::poll
  24: <futures::future::map::Map<A,F> as futures::future::Future>::poll
  25: <futures::future::either::Either<A,B> as futures::future::Future>::poll
  26: <futures::future::map::Map<A,F> as futures::future::Future>::poll
  27: <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll
  28: <futures::stream::and_then::AndThen<S,F,U> as futures::stream::Stream>::poll
  29: <futures::stream::forward::Forward<T,U> as futures::future::Future>::poll
  30: <futures::future::map::Map<A,F> as futures::future::Future>::poll
  31: <futures::future::map_err::MapErr<A,F> as futures::future::Future>::poll
  32: futures::task_impl::std::set
  33: std::panicking::try::do_call
  34: __rust_maybe_catch_panic
  35: tokio_threadpool::task::Task::run
  36: tokio_threadpool::worker::Worker::run_task
  37: tokio_threadpool::worker::Worker::run
  38: std::thread::local::LocalKey<T>::with
  39: std::thread::local::LocalKey<T>::with
  40: tokio_reactor::with_default
  41: tokio::runtime::threadpool::builder::Builder::build::{{closure}}
  42: std::thread::local::LocalKey<T>::with
  43: std::thread::local::LocalKey<T>::with
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.`

@AhmedElywa are you running prisma prisma2 lift save or prisma2 lift save in prisma folder?

@AhmedElywa are you running prisma prisma2 lift save or prisma2 lift save in prisma folder?
the folder name is prisma command prisma2 lift save

@pantharshit00 Hi, do you have some updates regarding this issue?

We will write a spec for this so that we integrate this in the schema.
We are on it, might take some time as it is a schema change.

Great, thanks!

@pantharshit00 or @malekjaroslav Can you please summarize what this issue is about? Is it still reproducible
Is it just about SSL support or something broader?

@pantharshit00 or @malekjaroslav Can you please summarize what this issue is about? Is it still reproducible
Is it just about SSL support or something broader?

It is about Unix socket support for connection to the DB. For example when using Google Cloud, the easiest and safest way to connect is through a socket connection. But we we unable to connect with a unix socket.
As SSL is not supported yet, there is no secure way to connect to the DB.

@pantharshit00 please help me push this one forward :)

@malekjaroslav do you mind sharing an example of what gets populated in CLOUD_SQL_CONNECTION_NAME?
I'm looking at the following example from: https://cloud.google.com/sql/docs/mysql/connect-app-engine
let pool;
const createPool = async () => {
  pool = await mysql.createPool({
    user: process.env.DB_USER, // e.g. 'my-db-user'
    password: process.env.DB_PASS, // e.g. 'my-db-password'
    database: process.env.DB_NAME, // e.g. 'my-database'
    // If connecting via unix domain socket, specify the path
    socketPath: `/cloudsql/${process.env.CLOUD_SQL_CONNECTION_NAME}`,
    // If connecting via TCP, enter the IP and port instead
    // host: 'localhost',
    // port: 3306,

    //...
  });
};
createPool();

Background:
There seems to be a couple different ways to support this:

Postgres: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING
Mongo: https://docs.mongodb.com/manual/reference/connection-string/#unix-domain-socket
MySQL via JDBC: https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory#creating-the-jdbc-url

Proposed Solution:
Ideally we can unify our unix domain socket syntax. I'd prefer the following:
datasource pg {
  provider = "postgresql"
  url = "postgresql:///var/lib/postgres/database.sock"
}

datasource ms {
  provider = "mysql"
  url = "mysql:///var/lib/mysql/database.sock"
}

Open Questions
Will there be an issue with the ///?
The postgres documentation states:

The host component is interpreted as described for the parameter host. In particular, a Unix-domain socket connection is chosen if the host part is either empty or starts with a slash, otherwise a TCP/IP connection is initiated.
Note, however, that the slash is a reserved character in the hierarchical part of the URI. So, to specify a non-standard Unix-domain socket directory, either omit the host specification in the URI and specify the host as a parameter, or percent-encode the path in the host component of the URI:

postgresql:///dbname?host=/var/lib/postgresql
postgresql://%2Fvar%2Flib%2Fpostgresql/dbname

I'm not following here, can someone explain this to me?

@matthewmueller : If i understand your suggestion correctly that would mean a deviation from the strategy we chose for our datasource URLs. From the spec:

If a given database already defines a connection URL standard the datasource URL is based on this standard. Additional parameters required by the Prisma tool suite are added on top of the existing standard.

Your suggestion would not follow this approach for MySQL, right? (see https://dev.mysql.com/doc/refman/8.0/en/connecting-using-uri-or-key-value-pairs.html)
If we want to go down that route we should make have a framework for when we go with a custom format and when not. Users need to understand when they can consult the docs of the given database and when not.

Thanks for your feedback @mavilein.
If I'm understanding your suggestion properly, in our example case of Unix Domain Sockets for MySQL, our connection string should look like this:
datasource ms {
  provider = "mysql"
  url = "root@localhost?socket=(/tmp/mysql.sock)"
}

Please correct if I'm misunderstanding ☺

I'm operating under the assumption that the connection string format is a wild west, so we're taming it by providing a consistent syntax and managing the variability on our end.
Using the Unix Domain Sockets as an example and Google Cloud SQL as the provider, the URL wouldn't work as is. You would need to connect with this connection string:
mysql+pymysql://<db_user>:<db_pass>@/<db_name>?unix_socket=/cloudsql/<cloud_sql_instance_name>

Source: https://cloud.google.com/sql/docs/mysql/connect-app-engine

I definitely see your point about not being to use the SQL documentation to figure out the connection string, but we've already kind of invalidated that resource with PSL.
That being said, I'd be on board for a system that follows the language's connection string and also reformats for these popular cloud platforms, it just seems like more engineering work.

Please correct if I'm misunderstanding ☺
No need to. You understood it correctly.

I'm operating under the assumption that the connection string format is already a wild west, so we're taming it by providing a consistent syntax and managing the variability on our end.

That is an approach that we explicitly moved away from.

Using the Unix Domain Sockets as an example and Google Cloud SQL as the provider, the URL wouldn't work as is. You would need to connect with this connection string:

I don't think this is true. This example looks like as if it is applying to the Python driver only.

@malekjaroslav what is your current workaround?
I'm in the same configuration: Cloud SQL + Cloud Run.
I can only make it working through public IP, no SSL and accepting requests from <IP_ADDRESS>/0, that is... bad.

@lvauvillier, well this is my workaround as well, but SSL should be supported by now.
More details here: https://github.com/prisma/prisma2/issues/568#issuecomment-539043245

Update to the proposal above.
⚠ This isn't implemented yet. The first step is to define the syntax
After speaking with @mavilein, we'll use the connection string that the database provides, rather than try to come up with a universal URL string syntax.

Postgres: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING
Mongo: https://docs.mongodb.com/manual/reference/connection-string/#unix-domain-socket
MySQL: https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory#creating-the-jdbc-url

datasource pg {
  provider = "postgresql"
  url = "postgresql:///dbname?host=/var/lib/postgresql"
}

datasource ms {
  provider = "mysql"
  url = "root@localhost?socket=(/tmp/mysql.sock)"
}

datasource mongo {
  provider = "mongo"
  url = "mongodb://%2Ftmp%2Fmongodb-27017.sock"
}

So to get this working on GCE, it'd look something like this:
datasource ms {
  provider = "mysql"
  url = "${env("DB_USER")}:${env("DB_PASS")}@localhost/${env("DB_NAME")}?socket=(/cloudsql/${env("CLOUD_SQL_CONNECTION_NAME")})"
}

I was reading the source code and it could be quite easy to implement:
Here we get the host from the Uri. But in a case of a socket the uri would look like<EMAIL_ADDRESS>and host_str returns localhost which is wrong.
The postgres crate states:

host - The host to connect to. On Unix platforms, if the host starts with a / character it is treated as the path to the directory containing Unix domain sockets. Otherwise, it is treated as a hostname. Multiple hosts can be specified, separated by commas. Each host will be tried in turn when connecting. Required if connecting with the connect method.

Which means if we give it the path it should use a socket instead of TCP.
For mysql
https://docs.rs/mysql_async/0.21.0-alpha.4/mysql_async/struct.OptsBuilder.html#method.socket
We can do the following: If Url cannot see a host in the uri, we try to look for path which should give us the socket.
How this works now is that you do not need to define any hostname and it will be automatically set to localhost. I'd like to enforce the hostname and if you don't write it, we'll try to get the path and use a socket.

Implementation, please read the tests:
https://github.com/prisma/quaint/pull/27/files#diff-a34a59925e5160fbec39a56d4fa6b3c6R362-R368
https://github.com/prisma/quaint/pull/27/files#diff-0a8a3c3d21eb76727df625b9399a69c4R491-R504

Ok, this is now merged to the engines master, some notes:

MySQL connection strings look like this: mysql://user:password@localhost/database?socket=/var/run/mysqld/mysqld.sock, where you must have the host written but if socket parameter is set, it'll try to connect using a unix socket. This needs to point to the correct sock file.
PostgreSQL is a bit different: postgresql://user:password@/database?host=/var/run/postgresql/. Here if the host parameter is set, we'll try to connect using it instead of the host part of the URI. This way we can point it to a socket, but compared to MySQL we pass the directory containing the file .s.PGSQL.5432=. If host or the host param is not set, we connect to localhost.

Works only in the Rust core. I tried to run prisma2 lift save using the uris above and got an error Invalid URL: postgresql://root:prisma@/prisma?host=/var/run/postgresql/. Seems like our TypeScript code is doing some parsing and needs to be checked out by @timsuchanek.

Thanks a lot for reporting 🙏
This issue is fixed in the latest alpha version of prisma2.
You can try it out with npm i -g prisma2@alpha.
In case it’s not fixed for you - please let us know and we’ll reopen this issue!

This is a pretty important one - this is the documented path for most of GCP and especially important for any serverless deployment, as documented here GCP cloud run <==> SQL connection guide Has anybody managed to figure out a working connection string?
I'm currently trying:
DATABASE_URL="postgresql://postgres:currencyforhumankind@<IP_ADDRESS>/ying-staging?host=/cloudsql/ying-staging/.s.PGSQL.5432"

This worked for me: postgres://user:password@localhost/mydb?host=/cloudsql/myproject:us-east4:test

@nargetdev If I remember correctly, you should not append .s.PGSQL.5432 to the socket string. That will automatically get appended by the driver
Use postgresql://USER:PASS@<IP_ADDRESS>/ying-staging?host=/cloudsql/DATABASE_NAME

Thank you very much @Sytten, that worked for me as well!!

@Sytten solution works like charm, it is very hard to find a solution for this case, I almost gave up


docker deamon restart results in out of memory condition in the server (Docker 1.13)

Description
Whenever the docker daemon is retarted (systemctl restart docker), it results in a out of memory condition
Running Docker 1.13.0-rc3
During test, when docker service is restarted, for whatever reasons, it generates udev events for cdrom and the IMPORT action fails resulting in creation of many systemd-udev threads. This hogs the memory.
[root@BaseVM ~]# ps aux --sort -rss | grep systemd
root       553  0.0  0.2  45416 10464 ?        Ss   14:06   0:00 /usr/lib/systemd/systemd-journald
root         1  0.1  0.1  43592  3932 ?        Ss   14:06   0:01 /usr/lib/systemd/systemd --system --deserialize 22
dbus       856  0.0  0.0  26864  2048 ?        Ss   14:06   0:00 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root      2615  0.0  0.0  43264  1940 ?        Ss   14:11   0:00 /usr/lib/systemd/systemd-udevd
root       868  0.0  0.0  26400  1724 ?        Ss   14:06   0:00 /usr/lib/systemd/systemd-logind
root      7994  0.0  0.0  43264  1148 ?        S    14:30   0:00 /usr/lib/systemd/systemd-udevd
root      7995  0.0  0.0  43264  1148 ?        S    14:30   0:00 /usr/lib/systemd/systemd-udevd
root      7998  0.0  0.0  43264  1084 ?        S    14:30   0:00 /usr/lib/systemd/systemd-udevd
root      7999  0.0  0.0  43264  1048 ?        S    14:30   0:00 /usr/lib/systemd/systemd-udevd
root      8186  0.0  0.0 112648   976 pts/0    S+   14:30   0:00 grep --color=auto systemd
when run with 4GB memory, system runs into low memory ( due to many systemd-udev threads) resulting is systemd being shutdown along with random processes being killed.

Steps to reproduce the issue:

install docker 1.13-rc3 in the system
systemctl restart docker

Describe the results you received:
Describe the results you expected:
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
(paste your output here)

Output of docker info:
(paste your output here)

Additional environment details (AWS, VirtualBox, physical, etc.):

Could you add the information about your setup, as requested in the issue template?

fulll output of docker version
full output of docker info
any relevant (daemon) configuration flags
what are you running in your containers? How many containers are running etc

Also note that docker 1.13.0-rc4 was released with various bug fixes

Hi, We are able to reproduce this with Docker 1.13 as well.  And system-udevd hogs the memory
Here is the journalctl -xl | grep docker log file
docker.txt

dmesg.txt
Dmesg log from system

Not sure, I do notice that the minimum requirements for that image is 4GB (16GB recommended); http://docs.portworx.com/get-started-px-enterprise.html, and a lot of failed attempts to reach the plugin. The killed processes seem to all be related to mount, indeed,
Did this same setup work correctly on docker 1.12?
/cc @anusha-ragunathan for (legacy) plugins

Same setup work correctly with docker 1.12.

Feature to restore volumes on daemon restart was newly added in 1.13. Daemon log shows error in restoring volumes, because the plugin container (px-enterprise) has not been started yet. However, this shouldnt cause OOM. I'll look more.

Looks like there were 7 volumes created using the px-enterprise containerized plugin. When the daemon was restarted, daemon tried to restore the volumes, but failed because the plugin was not running. This is expected and harmless.
Note that px-enterprise failed to start as seen in daemon logs:
bin/docker inspect -f {{.State.Running}} px-enterprise
Jan 25 14:57:47 BaseVM dockerd[23430]: time="2017-01-25T14:57:47.007951337-08:00" level=error msg="Error setting up exec command in container px-enterprise: Container baca6de117364d6a43ad0cb35d736c47c26a329ba618722e7080b3e3473ca76c is not running
As @thaJeztah pointed out, the host memory is 3.703 GiB and the container minimum recommended is 4G, its natural that OOM is killing random processes.
Does the issue happen all the time? If yes, it would be good to avoid kernel overcommitment and repeat the experiment. To avoid memory overcommittment, set this in /etc/sysctl.conf and restart your VM:
vm.overcommit_memory = 2
vm.overcommit_ratio = 80
Another option is to repeat experiment on a VM with 16G RAM.
https://www.kernel.org/doc/Documentation/sysctl/vm.txt

Note that f2d384fca6fa08da13fdc01c7991e8e35b081198 fixes a deadlock here during volume restore, effectively preventing the daemon from booting up (in 1.13) in some circumstances.
This should be fixed in 1.13.1

In this issue, the daemon boots fine, the container doesnt due to memory limits.

Did the suggested workaround (avoiding memory overcommit) fix the issue?

ping @benramki did @anusha-ragunathan's suggestion help? Are you still able to reproduce this issue on docker 1.13.1 (which was released)


example for multiple transaction

Hi,
Can you give another example where there are more than 1 transaction? i.e ul, isd, ul_ack?

This library only provides the codec. Stuff like the transaction management is not covered.


[clike mode] Add C# keywords/builtins: async, await, Action, Func, Task

Thanks. Merged as e3cc695


Remove finalizer from IdentityMap to fix memory leak that was preventing them from being GC'd properly.

Signed-off-by: Dane Jensen<EMAIL_ADDRESS>

LGTM


chore(actions): user mattermost-quality as commit-author

Hello @yasserfaraazkhan, let me know if this is ready for review.
Also, do you know what were the change at data/test-cases-toc.json? I would expect no change there. Could you revert that file?

Hi @saturninoabril ,
this is up for review, the job ran successfully and pulled new Data-sync changes

Thanks Yasser. However, I would expect that the save/sync will happen once merged to main only. Is it not the case?
Another that worries me is the changes happening at data/test-cases-toc.json. Can you take a look?
In this PR, I thought the related change is only at .github/workflows/deno.yml and other files should be reverted.

Thanks Yasser. However, I would expect that the save/sync will happen once merged to main only. Is it not the case?
Another that worries me is the changes happening at data/test-cases-toc.json. Can you take a look?
In this PR, I thought the related change is only at .github/workflows/deno.yml and other files should be reverted.

For testing purpose I run it on this branch to confirm sync is automatically, but from now on it will run post merge, I'll try to remove the sync changes.
the data/test-cases-toc.json was just new line getting added. I'll take a look once

Hi @saturninoabril
this should be good to get merged now 😄

Btw, Eli is out and so I'm requesting review from Mario.


Fix gidir and use getcwd

Make both has_git_.. functions return the git directory or nil.  Use
the same check for all function and vim function getcwd instead of
opening another process.

I found that you used 4 space indent for lua ?

Sorry the formatted picked up the mixed indentation. Thanks for telling me

I will test it later. Thanks for PR

merged.


Added option to control the ScrollBar policy of each scroll bar

What should the color be for the background?

Note: these warnings still appear
Flowless/src/main/java/org/fxmisc/flowless/VirtualizedScrollPane.java:32: warning - Tag @see: can't find setHbarPolicy(ScrollBarPolicy) in org.fxmisc.flowless.VirtualizedScrollPane
Flowless/src/main/java/org/fxmisc/flowless/VirtualizedScrollPane.java:38: warning - Tag @see: can't find setVbarPolicy(ScrollBarPolicy) in org.fxmisc.flowless.VirtualizedScrollPane
Flowless/src/main/java/org/fxmisc/flowless/VirtualizedScrollPane.java:32: warning - Tag @see: can't find setHbarPolicy(ScrollBarPolicy) in org.fxmisc.flowless.VirtualizedScrollPane
Flowless/src/main/java/org/fxmisc/flowless/VirtualizedScrollPane.java:38: warning - Tag @see: can't find setVbarPolicy(ScrollBarPolicy) in org.fxmisc.flowless.VirtualizedScrollPane

I just tried this using RichTextFX and got a NPE

I can fix the problem if I use getOrElse(false). However, if the VSP is shown initially and the scroll bars should be displayed, they don't.

Looks like the first getValue() call returns a 'null' because it hasn't figured out its value yet. So, it will initially display no scroll bars. Soon after its first call, the value is updated to an actual value and the VSP lays out its children correctly this time regarding scroll bars.

The corner is much less noticeable now. I get this image:

....using this code:
import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.control.ScrollPane;
import javafx.scene.layout.Region;
import javafx.stage.Stage;
import org.reactfx.value.Val;
import org.reactfx.value.Var;

public class TestCornerColor extends Application {

    class VirtualizedImpl extends Region implements Virtualized {

        VirtualizedImpl() {
            super();
            setStyle("-fx-background-color: black;");
        }

        Var<Double> esxp = Var.newSimpleVar(0.0);
        @Override
        public Var<Double> estimatedScrollXProperty() {
            return esxp;
        }

        Var<Double> esyp = Var.newSimpleVar(0.0);
        @Override
        public Var<Double> estimatedScrollYProperty() {
            return esyp;
        }

        @Override
        public Val<Double> totalHeightEstimateProperty() {
            return esxp.map(d -> d + 30);
        }

        @Override
        public Val<Double> totalWidthEstimateProperty() {
            return esyp.map(d -> d + 30);
        }
    }

    public static void main(String[] args) {
        launch(args);
    }

    @Override
    public void start(Stage primaryStage) {
        VirtualizedImpl v = new VirtualizedImpl();
        VirtualizedScrollPane<VirtualizedImpl> root = new VirtualizedScrollPane<>(v, ScrollPane.ScrollBarPolicy.ALWAYS, ScrollPane.ScrollBarPolicy.ALWAYS);
        Scene scene = new Scene(root, 400, 400);
        scene.getStylesheets().add(TestCornerColor.class.getResource("virtualized-scroll-pane.css").toExternalForm());
        primaryStage.setScene(scene);
        primaryStage.show();
    }
}

So, it seems like the javadoc warning is caused by a bug in the javadoc. I looked around on the web and couldn't find a good reason for why it was occurring.
I did notice that the bug is reproducible in the following code pattern:
public Test {
    private final Var<BarPolicy> verticalPolicy;
    public final BarPolicy getVerticalPolicy() { return verticalPolicy.getValue(); }
    public final void setVerticalPolicy(BarPolicy value) { verticalPolicy.setValue(value); }
    public final Var<BarPolicy> verticalPolicyProperty() { return verticalPolicy; }

    public Test(BarPolicy policy) {
        this.verticalPolicy = Var.newSimpleVar(policy);
    }
}

public static enum BarPolicy {
    FAKE
}

I'll see if any difference arises when I remove the ReactFX dependency and just use a SimpleObjectProperty

So this is tangentially related, but since you're working in this area I'll ask you if you can add a feature for me: Is there a way to get a VirtualFlow for use with horizontal scrolling only?
If I want a virtual flow to control horizontal scrolling, but delegate vertical scrolling to its acenstors (--in my case actually a regular old ScrollPane), how could I do that?
Specifically I feel like this could be expressed relatively easily with:
// horizontallyScrollable {get; set; property}

-public void scrollX(double deltaX) {
+public boolean scrollX(double deltaX) {
-  content.scrollX(deltaX);
+  return if(isHorizontallyScrollable()) { context.scrollX(deltaX); }
 }

//similar for Y

 this.addEventHandler(ScrollEvent.SCROLL, se -> {
-  scrollX(-se.getDeltaX());
-  scrollY(-se.getDeltaY());
-  se.consume();
+  boolean changed = scrollX(-se.getDeltaX());
+  changed |= scrollY(-se.getDeltaY());
+  if(changed) { se.consume(); }
-});

can I ask you to make something like this part of that PR?

actually that's pretty silly, why not just have the event handler compute whether or not it should consume the scroll event based on whether or not the component has anything to scroll?

:smile: Yeah, that seems like a better approach.

Speaking of which, @TomasMikula is this PR acceptable if I cleaned it up and removed some of the unnecessary intermediate commits?
Also, I think it would be better to have ScrollPane.ScrollBarPolicy as the type to use for the scroll bars since a class displaying one or more of VirtualizedScrollPane and ScrollPane could use a single import, but I can't figure out how to stop that tag warning from appearing. So, would you rather have the warnings but single import? Or multiple imports but no warnings?

The corner issue should be handled separately (has nothing to do with scroll bar policy).
I would reuse ScrollPane.ScrollBarPolicy.
The change seems somewhat verbose. Why isn't it sufficient to define shouldDisplayHorizontal/shouldDisplayVertical as you do, bind hbar's/vbar's visibleProperty() to those (as you do) and leave the rest as is? What is achieved by changes in layoutChildren()?

The corner issue should be handled separately (has nothing to do with scroll bar policy).

True.

I would reuse ScrollPane.ScrollBarPolicy.

K.

The change seems somewhat verbose. Why isn't it sufficient to define shouldDisplayHorizontal/shouldDisplayVertical as you do, bind hbar's/vbar's visibleProperty() to those (as you do) and leave the rest as is? What is achieved by changes in layoutChildren()?

After checking... Nothing, actually. Since this was done long ago, I can't recall everything that was going through my mind. However, I do remember thinking it would be better to have shouldDisplayHorizontal/Vertical combined into shouldDisplayBoth so that only one InvalidationListener would be added that would request the layout. For some reason, I thought this would be more efficient. Looking at it now, I think it is unnecessary and makes it less efficient.
Addtionally, my understanding of the code wasn't accurate: I thought the two if statements (of whether and how to resize each scroll bar if it was visible) wouldn't work with the new set up. I thought it would need something like a switch statement: if both showing, do this; if one horizontal/vertical, do this; if none, do this.
But of course, turns out I was wrong :stuck_out_tongue_closed_eyes:

I've removed some of the comments I made after first submitting this PR as they're no longer relevant.

@TomasMikula Sorry, that should have been in there. I've updated the PR to include the aforementioned comment.

Looks good 👍

New snapshot published.

Thanks!


[BUG] During event creation, on checking and then unchecking the 'All day' checkbox, date is reset to default

Description
Checking and unchecking the 'All day' checkbox results in the date resetting to default
Acceptance Criteria
after entering a date and unchecking the All day checkbox, the date should stay the same while the time field disappears.

close.


fix (docDB): fetching dynamic foraging sessions by software name

Change the way we fetch dynamic foraging sessions (w/ or w/o FIP) from docDB
Previously we use "data_description.modality.abbreviation": "fib", which was supposed to get "fib" sessions. But because of another error in generating data_description.modality, this actually fetched all behavior sessions processed by the new FIP pipeline. As a result, we saw almost matched sessions between this query and my temporary pipeline.
However, the error in data_description.modality has been recently fixed. To continue comparing dynamic foraging sessions from both pipelines, this PR changes the query to
{
  "session.data_streams.software.name": "dynamic-foraging-task",
  "name": {"$not": {"$regex": ".*processed.*"}},  # only raw data
}

See details in https://github.com/AllenNeuralDynamics/dynamic-foraging-task/issues/1056
Two caveats:

Some older sessions may not have correct session.data_streams.software.name
The new query includes sessions uploaded by the ephys pipeline (but they won't have "processed")

Next, I'll create venn diagrams to directly compare different queries.

@rachelstephlee @alexpiet I changed the way we query the dynamic foraging sessions from docDB in Streamlit

will review before EOW.
but for the sake of kanban-- what would we do for past sessions with the wrong label? I think you asked this on the teams chat. if we don't have a plan for them, please make a ticket with our action plan for that?

Otherwise, i have design choice question (we could have it have a column that is just 'data_modality' and it lists all the possibilities (fib, behavior, video) instead of a true/false 'has_fib'.

Adding has_fib, has_video, has_ephys columns in the behavior-centered app (i.e., has_behavior is always True) is just for convenience in filtering the table. For example, if you use list, to find out all has_fib sessions, you must multiselect (fib), (fib + video), (fib + ephys), (fib + ephys + video) etc.


fix: Add support for overriding DNS suffix for cluster IAM role service principal endpoint

Description

Add support for overriding DNS suffix for cluster IAM role service principal endpoint. There appears to be an inconsistency with the EKS endpoint in China that is used by the assume role service principal. The dns_suffix provided by the AWS partition resource is returning the correct amazonaws.com.cn but the endpoint is expecting amazonaws.com. This is a bit of a temporary work around for now to let users deploy the module into China regions until the endpoints are fixed by AWS

Motivation and Context

Closes #1904

Breaking Changes

No; backwards compatibility is maintained

How Has This Been Tested?

[ ] I have tested and validated these changes using one or more of the provided examples/* projects

This PR is included in version 18.7.3 :tada:


test-branch/execute-job/comment-from-wrong-user-on-pr/1afae10b-aefd-4c16-9586-9df634a697bb

PR for testing c0296ade1c84a72a8e91ab0cfc8303042431f44a

/canonical/self-hosted-runners/run-workflows c0296ade1c84a72a8e91ab0cfc8303042431f44a


Upgrade to latest anchor (0.19.0 -> 0.20.1)

As titled.

There was a breaking change in v0.20.0 changing #[repr(packed)] to #[repr(C)] for ZeroCopy structs. You'll have to add #[repr(packed)] to every struct after #[zero_copy] and #[account(zero_copy)].

Updated. Not testednyet will attempt today, on hold


Epic: search and analyze storage worst-case scenarios

https://github.com/zenithdb/zenith/issues/906 showed that we don't have any immediate problems with the big databases. Since increasing the database size does not automatically reveals any innate architecture problems let's create such cases manually.
One scenario that I can think of is the following:

we have a small one gigabyte table t1 and a big one terabyte table t1000
test generates 1gb of updates to t1 and 100k random updates in t1000 (assuming ~100 bytes wal records, that would be about ~10mb of wal).  With 100k random updates to t1000 we should make almost all 10 mb layer/segment dirty. And updates to t1 should trigger checkpoint in the pageserver.
wait a bit until checkpoint is done
go to 2)

If I understand current pageserver algorithm correctly, that scenario would write 1TB of data on each iteration, caused by ~1gb of postgres data changes.
That is somewhat an artificially bad scenario and we need to have some quantitative metrics that tells us is any real database load is "good" or "bad". So we need to have run-time metrics in the pageserver to measure write amplification and disk bloat. In the spirit of RUM conjecture (http://daslab.seas.harvard.edu/rum-conjecture/) it makes sense to track following things in the pageserver:

read overhead -- count number of bytes we've read from the disk (and also read ops) for a give getPage@lsn
write overhead aka write amplification -- number of bytes that we've written. Taking into account that usually WAL record are compact we can divide write amplification by the size of the wal that caused this changes and get a good indicator of write amplification
memory overhead aka disk bloat -- current physical size of the all our stored files. We can divide that size by a current logical size to get a characteristic number.

I propose the following plan of actions:

[x] reproduce that write amplification scenario in a python test
we can setup a single tenant and get process-wide numbers right away. To keep the amount of data small (hence the test execution time) we can tighten up pageserver's checkpoint distance and demonstrate that one page change caused whole layer write on a one layer. If we can do that we can unboundedly grow write amplification
[ ] #1116
[x] #1117
[ ] #1137

There's a test that does something similar.
Both of these tests seem to target the fact that we write too many image layers where delta layers would have been fine? Seems fixable, but let's talk tomorrow and see if I correctly understand the layered repo (and how it becomes a bottleneck). I don't quite understand why we need a mix of dense + sparse writes. Seems like if you can fill RAM with sparse writes you can trigger an eviction with equally bad (amortized) write amplification.

Finally I was able to reproduce this worst scenario. The problem with my previous attempts was that... our checkpointer is too slow. I.e. doesn't matter what we are doing, we are generating WAL faster then checkpointer is able to write it.
So to get the largest write amplification we need:

perform minimal number of random updates with affect all big table segments
write ~256Mb data to some small table to trigger default checkpoint distance
initiate checkpoint and wait it's completion

It is achived by the following test:
def test_write_amplification(zenith_simple_env: ZenithEnv):
    env = zenith_simple_env
    env.zenith_cli(["branch", "test_write_amplification", "empty"])
    pg = env.postgres.create_start('test_write_amplification')
    n_iterations = 100
    seg_size = 10 * 1024 * 1024
    n_segments = 10 * 1024 # 100Gb
    payload_size = 100
    header_size = 28
    tuple_size = header_size + payload_size
    big_table_size = n_segments * (seg_size // tuple_size)
    checkpoint_distance = 256 * 1024 * 1024
    small_table_size = checkpoint_distance // tuple_size
    redundancy = 3

    print(f"Big table size: {big_table_size}")
    print(f"Small table size: {small_table_size}")
    print(f"Iterations: {n_iterations}")
    print(f"Random updates per checkpoint: {n_segments * redundancy}")

    with closing(pg.connect()) as conn:
        with conn.cursor() as cur:
            with closing(env.pageserver.connect()) as psconn:
                with psconn.cursor(cursor_factory=psycopg2.extras.DictCursor) as pscur:

                    # Get the timeline ID of our branch. We need it for the 'do_gc' command
                    cur.execute("SHOW zenith.zenith_timeline")
                    timeline = cur.fetchone()[0]

                    # Create a test table
                    cur.execute("CREATE TABLE Big(pk integer primary key, count integer default 0, payload text default repeat(' ', {payload_size}))")
                    # populate ~100Gb in Big
                    cur.execute(f"INSERT INTO Big (pk) values (generate_series(1,{big_table_size}))");
                    cur.execute(f"CREATE TABLE Small(pk integer, payload text default repeat(' ', {payload_size}))")
                    for i in range(n_iterations):
                        print(f"Iteration {i}")
                        # update random keys
                        for i in range(n_segments * redundancy):
                            key = random.randint(1, big_table_size)
                            cur.execute(f"update Big set count=count+1 where pk={key}")

                        # bulk insert small table to trigger checkpoint distance
                        cur.execute(f"insert into Small (pk) values (generate_series(1, {small_table_size}))")
                        # And truncate it to avoid database size explosion
                        cur.execute("truncate Small")
                        # Run GC to force checkpoint and save image layers
                        pscur.execute(f"do_gc {env.initial_tenant} {timeline} 0")
                        row = pscur.fetchone()
                        print_gc_result(row)
    # Wait forever to be able to inspect statistic
    print('Test completed')
    time.sleep(100000)

At each iteration which takes about 400 seconds it produce ~256Mb of WAL and write to the disk ~130Gb.
So write amplification is ~500 times.

Also loading of 130Gb table with primary key takes 130 minutes (1Gb/min).
It seems to be much worse than with Vanilla Postgres.
In case of uploading data to table without indexes speed was almost the same: 7  hours for 1.3Tb which corresponds to speed 3Gb/min.

After running this test, the timeline was 259MB, but 252MB of those were image files.
A simple solution is to make sure that each image file is preceded by consecutive delta files that are as big as the image in total. Tweak some logic here. This way disk bloat would be exactly 2, and read amplification would be worst case 2, but average 1.5.
In the test example above, instead of 7MB of deltas and 252MB of images, we'd have 7MB of each, 14MB in total. The test numbers will be different on bigger data (I can try @knizhnik 's test), but we'd still have disk bloat of 2 and read amplification worst case 2 and average 1.5 (because the average get_page@lsn request consumes half the deltas between two images).
Though I'm not sure if we want to exactly balance disk bloat and read amlpification because i don't know how cheap wal redo is. But we can tune a parameter k: if we set a target total_image_size = k * total_delta_size then we have:

disk bloat 1 + k
read amplification worst case 1 + 1/k
read amplification average case 1 + 1/(2k)
write amplification = disk bloat

A simple solution is to make sure that each image file is preceded by consecutive delta files that are as big as the image in total.
Unfortunately, simple solution will not work.
First of all we have to periodically perform some kind of backups and backups IMHO should contain page images. Just for simplicity and speed of apply. So even if one page of 10Mb segment is updated, then we still have to produce new image layer.

Second argument is that one page (or even one record) of a segment can be updated multiple times. Applying 10Mb of WAL to reconstruct this page will take too much time. As far as the only way to permanently store reconstructed page now is to create new image layer, then we will have to create this layer to keep reconstruct time reasonably small.
Yes, we have page cache now and ephemeral files. But both will be missed on page server restart. So we can rely on on delta and image layers.
But I agree that the only way avoid write amplification is to delay creation of new image layer. If checkoint is performed with period one second, then definitely it is not a good idea to create new image layer each second, Especially if there are only few changes in this segment. SO layer eviction/reconstruction policy should be smarter. Right now image layer is not created by checkpoint performed at shutdown (to keep shutdown time reasonably small). I think that we should add more flexible rule for creation of new image layer. It may be triggered by:

Total number of deltas (wal records) for this segment
Maximal length of update chain for page
Time since last image layer creation.

I have tried to implement smarter image layer generation policy and started with the simplest one: do not create image layer id size of delta layer is less than 1/2 of segment size (10Mb). It certainly dramatically increase speed of this test: checkpoint takes few seconds instead of 400.
But t the same time it cause  a lot of "Timed out while waiting for WAL record" errors which start to happen when duration of checkpoint exceeds 1 minute.
Elimination of such timeouts is one of the primary purposes of introducing back pressure mechanism. But for some reasons it is not considering  this lag at all, only flush=dist_consistent_lsn and apply=S3 LSN lags.
I have created PR https://github.com/zenithdb/postgres/pull/115  introducing max_prelication_write_lag.
But it doesn't help. I have only one explanation for it: without writing image layers, eviction of layer is performed so fast, that other threads (performing get_page_at_lsn) failed to get LayerMap and LayeredTimeline mutex before it reacquired by checkpointer. So during all one minute checkpointer iteration get_page_at_lsn is blocked and timeout expired.
The simplest solution is to replace std mutex with packing_lot mutex, as we already have done in some of our components.
It provides more "fair" resource acquisition.
Alternative we can build list of layers to be evicted under mutex and then process this list without holding locks.

I have tried to implement smarter image layer generation policy and started with the simplest one: do not create image layer id size of delta layer is less than 1/2 of segment size (10Mb). It certainly dramatically increase speed of this test: checkpoint takes few seconds instead of 400.
But t the same time it cause  a lot of "Timed out while waiting for WAL record" errors which start to happen when duration of checkpoint exceeds 1 minute.
Elimination of such timeouts is one of the primary purposes of introducing back pressure mechanism. But for some reasons it is not considering  this lag at all, only flush=dist_consistent_lsn and apply=S3 LSN lags.
I have created PR https://github.com/zenithdb/postgres/pull/115  introducing max_prelication_write_lag.
But it doesn't help. I have only one explanation for it: without writing image layers, eviction of layer is performed so fast, that other threads (performing get_page_at_lsn) failed to get LayerMap and LayeredTimeline mutex before it reacquired by checkpointer. So during all one minute checkpointer iteration get_page_at_lsn is blocked and timeout expired.
The simplest solution is to replace std mutex with packing_lot mutex, as we already have done in some of our components.
It provides more "fair" resource acquisition.
Alternative we can build list of layers to be evicted under mutex and then process this list without holding locks.

Yet another problem with delaying write of image layers: right now GC is able to remove old layers only if there is image layer.
So,  1 second checkpoint interval, 3 hours of work and ... delete of tenants directory (just rm -fr) takes more than one minute...

Yet another problem with delaying write of image layers: right now GC is able to remove old layers only if there is image layer.
So,  1 second checkpoint interval, 3 hours of work and ... delete of tenants directory (just rm -fr) takes more than one minute...

Nice results!
Question on your "Applying 10Mb of WAL to reconstruct this page will take too much time." comment:
Do we ever need to reconstruct just a single page? AFAIK single-page requests should be coming mostly from the compute leader, so they should be at latest LSN, which we have an image for. Requests for old LSNs will come from PITR, but we might as well reconstruct entire segments for that workload. Applying 10MB of WAL to get an entire segment doesn't seem bad. The third case is maybe requests from read-only replicas, but let me know if we've already had conversations on that topic before I start guessing how we might run them.
Another solution is maybe a "partial image" layer, which holds images for just a few pages on the segment (along with a header describing which pages it holds). The difference between a delta and a partial image is that when you're reading the busy page on a segment, you only need to find the latest partial image, not all of them since last full image. This effectively simulates temporarily having segment_size = page_size when it makes sense. I assume in general we don't want segment_size = page size because filesystem metadata overhead?

Nice results!
Question on your "Applying 10Mb of WAL to reconstruct this page will take too much time." comment:
Do we ever need to reconstruct just a single page? AFAIK single-page requests should be coming mostly from the compute leader, so they should be at latest LSN, which we have an image for. Requests for old LSNs will come from PITR, but we might as well reconstruct entire segments for that workload. Applying 10MB of WAL to get an entire segment doesn't seem bad. The third case is maybe requests from read-only replicas, but let me know if we've already had conversations on that topic before I start guessing how we might run them.
Another solution is maybe a "partial image" layer, which holds images for just a few pages on the segment (along with a header describing which pages it holds). The difference between a delta and a partial image is that when you're reading the busy page on a segment, you only need to find the latest partial image, not all of them since last full image. This effectively simulates temporarily having segment_size = page_size when it makes sense. I assume in general we don't want segment_size = page size because filesystem metadata overhead?

Do we ever need to reconstruct just a single page?
Why do we need to reconstruct single page? At least because SMGR API is dealing with pages and compute node request a page from pageserver. Definitely it doesn't mean that we can not reconstruct the whole segment at once. But what benefits we will get from it? The problem is applying 10Mb of WAL records is long delay for get_page_at_lsn execution and as a result, long time of query execution. The that we have reconstructed not the single page, but the whole segment will not help to redude this delay. More over, with the current implementation of page image reconstruction (using WAL redo postgres instance, receive queries through the pipe), there will be no large difference between execution of N request to materialize page and one request to materialize N pages.

So from my point of view, eager materialization of the whole segment will not help us to reduce page access time and may even have negative impact on performance (if segment pages are updated non uniformly).
Now concerning different workloads. There is OLAP workload, where pages are accessed mostly sequentially and when some kind of bulk load (request the whole segment instead of individual pages) may be very useful.
And there is OLTP workload when we most frequently request random B-Tree page and do not need its neighbors.
And yes, compute leader request page at latest LSN, but this page can be actually very old  (updated long time ago).
Actually I do not completely understand your proposal to reconstruct not the single page, but the whole segment, because ...
it is exactly what we are currently doing when evict in-memory layer: create delta layer and image layer. And exactly because of it we get write amplification in case of skews of  page access. I try to find out some criteria we can use to postpone image layer creation. But as I have mentioned above, there seems to be a lot of problems.

Misunderstanding: By "reconstruct" I meant "compute page by redoing wal, not reading from image". Sorry if I'm overloading an existing term.
The pageserver technically needs to serve all GetPage@LSN requests, even sparse pages at LSN != latest. But seems like no service is actually making these requests. We're either getting:
a) requests at latest LSN, or
b) requests with high locality (pages in large contiguous blocks)
For (a), performance is good because we're not doing any wal redo. We have images. Even if the page is very old, if it's the latest one we have an image on disk.
For (b), we compute entire segments. If that's what we're doing now, then no change necessary. We're redoing 8kb of wal per page amortized. This should only happen during PITR. If we need to further optimize it, there are ways.
I'll get to your storage concerns later. Also I haven't fully specified how to achieve k=1 and also "image for all segments on disk" at the same time. I'll get to that later. Now I'm focused on the following point: "we can have bad performance for non-local GetPage@LSN workload if LSN is not latest, because this workload doesn't exist" Let me know if this is not the case. I'm new to the team, I don't know everything :)

Misunderstanding: By "reconstruct" I meant "compute page by redoing wal, not reading from image". Sorry if I'm overloading an existing term.

Sorry, if I was unclear, but by "reconstruct" I also mean applying WAL records create final page image.

For (a), performance is good because we're not doing any wal redo. We have images.

We do not get images at page server (except FPI). We receive WAL and has to apply it at page server to reconstruct page.
The question is how frequently we should do it...
We can materialize page immediately after receiving each WAL record. But in this case we have to overwrite page many times.
Or we can delay materialization, hoping that just updated page will tno be requested by compute node soon. Ideally we should collect large enough set of WAL records to be applied, but reconstruct page in background (by checkpointer thread)  before it will be requested by compute node. In this case there will be no get_page_at_lsn delay.

May main concern now is that with max_replication_flush_lag=0 above test never completes successfully (terminated by page server returned error: Timed out while waiting for WAL record after some iterations), even through I have specified max_replication_write_lag=1MB.
This back pressure tunning seems to be very obscure....

So write amplification is ~500 times.
Final write amplification after finishing all 100 iteration of the test is much smaller: "just" 50 times:

table size: 100 Gb
generated WAL: 280 Gb
disk write IO: 14 Tb
I think it is because of random updates and autovacuum, which periodically update the whole table.

This trace illustrate problem with large number of files.
Checkpoint is performed each 1 second, but new image layer is created only if total size of deltas exceeds half of segment size (10Mb). So cause create of large number of small delta layers:
[ec2-user@ip-172-31-24-242 ~]$ time ls -l /data/zenith/test_output/test_write_amplification/repo/tenants/187e2ae40c5d494b86918838419ac904/timelines/f1080e663a20916c897d6ff2f26acfda/ | wc -l
1784267

real	0m36.851s
user	0m25.106s
sys	0m12.033s

According to pageserver.log, GC takes 7 seconds just to traverse all files:
2022-01-09 16:49:31.81  INFO [utils.py:75] GC duration 7415 ms
2022-01-09 16:49:31.82  INFO [utils.py:78]   REL    total: 1784297, needed_by_cutoff 323, needed_by_branches: 0, not_updated: 1783832, needed_as_tombstone 101, removed: 11, dropped: 30

Loading data in the same table in Vanilla Postgres takes only 21 minutes (vs. 130 for Zenith).
So here difference between Vanilla and Zenith is 6.5 times. As I wrote above, such difference most likely is explained by presence of index building of which requires reading of the whole table.

@kelvich I'd like to close the "Think of any changes" subtask since I think the LSM solution with a few modifications is the right way to go. My final thoughts (after many conversations with @heikki):
Let's start with the LSM tree solution (as proposed here https://github.com/zenithdb/zenith/issues/936). The worst case scenario for that solution is a 1TB wal that's writing to one single page. Answering GetPage@LSN for that page would require replaying 1TB of wal, which is not good. Read amplification for GetPage@LSN is unbounded.
So let's patch this solution by inserting an 8KB page every 8KB/k of wal, for each page individually. This increases write amplification by an additional factor of k, but places an upper bound on read amplification.
We can play with the numbers but let's analyze the case with fanout = 4 and k = 1. With fanout 4, 1TB is stored in 6 layers (because 1 + log_4(1TB / 1GB) = 6).
Disk bloat is k + 1 = 2.
Write amplification is k * num_layers = 12. This seems big but as long as it's below SSD_speed/network_speed we should be good. That number is 20 AFAIK so we have some wiggle room.
To analyze read amplification, let's look at 2 cases:
a) Overhead during recovery from s3
b) Overhead for GetPage@LSN requests
During s3 startup/recovery we need to download num_layers * fanout * 1GB = 24GB to get our first page. If the network is the bottleneck that's about 1 sec to get the first page and 40 sec to restore full 1TB dataseservice (without sharding).
For GetPage@LSN overhead we have 1 + 1/k = 2 read amplification for wal redo, and some extra overhead to find the relevant entries from the LSN. We need to look inside num_layers * fanout = 24 files. It's not bad, but with some indexing this can be reduced even further down to 1 for the common case.
Stepping back, if we compare this system against vanilla replicated postgres, here are the differences in performance I'd expect:

I'm not worried about handling write throughput because the network will be the bottleneck in our case, and in the replicated vanilla postgres case as well.
We can afford to disable checkpointing on the compute node so we might gain performance over vanilla postgres on some workloads.
I'm not worried about s3 restore. If you store wal + checkpoints of vanilla postgres on s3 you'll have to download the same amount of data. We might perform better because we can partially restore what's needed first.
I think the only thing (theoretically) left to optimize is read performance of GetPage@LSN. A vanilla postgres would just read from disk, but we have to take a network roundtrip and maybe some wal redo. The wal redo part might be possible to eliminate in 99% of the cases (when LSN=latest) but the network roundtrip is unavoidable.  So maybe it's worth adding a perf test for this.

Sorry, may be should organize some meeting about proposed improvement of the current layered approach, because frankly speaking I do not completely understand it. First of all 1Tb of WAL updating the single page never was the worst case. Checkpointer periodically (each 256Mb if it is fast enough) will produce full page images, so pageserver never needs to apply 1Tb of WAL during get_page_at_lsn. Also Postgres itself (compute node) will help to prevent such scenrio: periodically it will do checkpoints (do not mix with checkpoints performed by page server) and produce FPI.
But I still do not understand how you are going  to eliminate write amplification and large number of produce files for the scenario where just one page of segment is once updated. So we have in-memory layer containing just one WAL record.
But we can not keep  this layer open forever: pageserver needs to provide data durability and advance disk consistent LSN.
Right now we will create delta layer (very small in this case ) and image layer (10Mb). We can postpone creation of image layer  but it doesn't prevent problem with large number of small delta layers files. AndGC will not be able to collect them until image layer is constructed.
So may be it will be more efficient and convenient to discuss it by voice...

Very rough summary of our meeting: The biggest discussion point was that the LSM solution would be doing non-sequential disk reads during GetPage@LSN, which would result in (5x?) slower performance on cache miss than we have now. See Figure 3 here for reference on SSD random read performance https://www.usenix.org/system/files/conference/fast16/fast16-papers-lu.pdf

@kelvich I'd like to close the "Think of any changes" subtask since I think the LSM solution with a few modifications is the right way to go. My final thoughts (after many conversations with @heikki):

Yeah, +1 on stopping activity here. I think with https://github.com/zenithdb/zenith/issues/1076 and https://github.com/zenithdb/zenith/issues/1116 getting merged we should close that issue. And any other bloat/perf discussion can be continued in https://github.com/zenithdb/zenith/issues/936


Create CODE_OF_CONDUCT.md

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Prototype Behavior Interface

Converts all well-described behavioral data from .mat file. Makes note of fields that need more info from Schneider lab (ex. 'push' trials table)

Thanks for the detailed review @h-mayorquin!


bold don't working

This is the first time I am working with docx and I am facing a problem.
I am reading a text file. If 'the first position is numeric, I want to make it bold and normal if it is not.
from docx import Document
from docx.shared import Pt
.....
for linea in lineas:
if linea[:1].isnumeric:
p = doc.add_paragraph('', style='List')
p.add_run(linea).bold = True
else:
p = doc.add_paragraph('', style='List')
p.add_run(linea).bold = False
But it leaves all text in bold
What am I doing wrong?

if you are using a language not English,try underline instead of bold, if it works, you shall start searching in :
w:bCs
https://stackoverflow.com/questions/45627652/python-docx-add-style-with-ctl-complex-text-layout-language
I will update this answer.


Mic sound is not streamed from MacOS browsers to UE app

Originally opened by @vlobzakov
UE Version:
E.g. UE 5.1.1+
Frontend Version:
UE5.1+
Problem component
Signalling Server WebRTC
Description
Sound capture via microphone doesn't work for MacBooks
Steps to Reproduce:

Build UE application that should record voice via Microphone.
Stream UE application via Pixelstreaming.
Open Stream on Macbook in Chrome or Safari.
Speak.
Check UE app what does it recieve(record) via browser.

Expected behavior

On Windows/Linux any browser can interact with UE via Pixelstreaming doing sound recordings.

Screenshots
Platform (please complete the following information):

OS: MacOS that comes with MacBook Intel, MacBook M1 and MacBook M2.
Browser Chrome 90+ and Safari

Additional context
In case of Chrome you can record sound with help of webrtc-internals and store it locally. Then yoou can play these files and listen that Chrome does recording properly. But sound from microphone isn't streamed to UE app.
If Pixelstreaming service taken from branch 5.0 this bug is not available and UE app can record sound from browser properly.

Moved from https://github.com/EpicGames/PixelStreamingInfrastructure/issues/494

were you able to solve this? I can neither send mic input, nor receive audio output.


GLITCH: Music desyncs when one peer went to Main Menu and then reconnects with another peer

If one peer already played some rounds and then gets back to the Main Menu and plays again with some peer, the music will be desynced between peers. Is this even a problem though? I don't quite know why this is, as the music should be synced, at least I thought that.

Okay this is worse then I thought. There seem to be all sorts of issues with the music when reconnecting to new rounds after playing. I should definitely investigate this issue.

So the issue appears to be that when the player presses P (enters Matchmaking) very quickly after entering the Main Menu when his peer disconnects then the BGM doesn't start in the next round.


What about using LSP4IJ?

I noticed that you have implemented your own lsp support, but I would like to notify you that it exists now a free LSP support for IntelliJ https://github.com/redhat-developer/lsp4ij

You can see the current LSP support at https://github.com/redhat-developer/lsp4ij/blob/main/docs%2FLSPSupport.md
By reading your LSP support at https://github.com/clojure-lsp/clojure-lsp-intellij/blob/master/docs/capabilities.md it seems LSP4IJ covers the same thing than your plugin and perhaps more. I have tested quickly https://github.com/clojure-lsp/clojure-lsp-intellij/issues/51 and it seems it is working with LSP4IJ.

LSP4IJ provides an LSP console to show LSP traces and see the state of language server https://github.com/redhat-developer/lsp4ij/blob/main/docs%2FUserGuide.md

If you want to play quickly with LSP4IJ and your Clojure LS without developping an IJ plugin please read https://github.com/redhat-developer/lsp4ij/blob/main/docs/user-defined-ls/clojure-lsp.md

If you want to integrate your ls in an IJ plugin please read https://github.com/redhat-developer/lsp4ij/blob/main/docs%2FDeveloperGuide.md

You can read an article about LSP4IJ at https://idetools.dev/blog/lsp4ij-announcement/
Hope you could be interested with LSP4IJ and don't hesitate to answer me if you have any questions.
Thanks!

Hey @angelozerr, thanks for bringing that, I was not aware of that and it's awesome to see free open source alternatives like that one! I'll take a closer look, but some things I noticed:

When I hold Ctrl, and hover over a symbol, I see the whole buffer text underlined, seems a little bit weird
Do you support semantic tokens?
When I click on a code lens I see this, which I'm not entirely sure what means:

When I created this plugin, the idea is to provide LSP support but in a way intellij users don't need to setup nothing, like just works out of the box, with minor customization as possible, is it possible to have a plugin that uses lsp4ij under the hood and have all those things configured already out of the box?

When I hold Ctrl, and hover over a symbol, I see the whole buffer text underlined, seems a little bit weird

I think it is the same issue than https://github.com/redhat-developer/lsp4ij/issues/98 It is because if you have use just TextMate or TEXT language, the PsiFile is not tokenized and the range is the full file content. It is badly not possible to customize that with gotoDeclarationHandler extension point -( But if you have a plugin which tokenized the PsiFile with PsiElement you will not have this problem.

Do you support semantic tokens?

I have pushed last hours this support, see doc at https://github.com/redhat-developer/lsp4ij/blob/main/docs/UserGuide.md#semantic-tokens-support but it is experimental and it seems that there is a CPU issue https://github.com/redhat-developer/lsp4ij/issues/394 but if you want to play with this feature you can install nightly build https://github.com/redhat-developer/lsp4ij?tab=readme-ov-file#testing-nightly-builds
I have played with clojure lsp and it seems it colorizes some tokens.

When I click on a code lens I see this, which I'm not entirely sure what means:

It means that language server report Codelens with command code-lens-references, you can see those CodeLens by using LSP console https://github.com/redhat-developer/lsp4ij/blob/main/docs/UserGuide.md#lsp-console
If language server doesn't provide this command, it tries to search if it exists a command on client side, if this command doesn't exist, you have this popup error and you have a link which suggest to implement an IJ plugin which register the expected command.
I think you have implemente this LSP command in your plugin, right?

clojure-lsp-intellij has a good feedback during initialization which I don't see easily, I need to click on LanguageServers and even so have logs enabled, check the $/progress notifications

$/progress has been implemented with IntelliJ Backgroind Task, see https://github.com/redhat-developer/lsp4ij/blob/main/docs/LSPSupport.md#progress-support
What is your suggestion to improve that?

When I created this plugin, the idea is to provide LSP support but in a way intellij users don't need to setup nothing, like just works out of the box, with minor customization as possible, is it possible to have a plugin that uses lsp4ij under the hood and have all those things configured already out of the box?

See the section of our article https://idetools.dev/blog/lsp4ij-announcement/#how-to-integrate-your-language-server-in-an-intellij-plugin
It seems that you have played with https://github.com/redhat-developer/lsp4ij/blob/main/docs/user-defined-ls/clojure-lsp.md which is working just with few settings and you don't need to write an IJ plugin, but it is better to develop an IJ plugin:

to embed the clojure-lsp in the plugin
provide required LSP command like the code-lens-references command.

In other words, my suggestion is to update your IJ plugin, by removing all LSP features and  based on LSP4IJ and register the Clojure LSP with the LSP4IJ server extension point https://github.com/redhat-developer/lsp4ij/blob/main/docs/DeveloperGuide.md
The goal of your plugin is to:

embed default clojure lsp
provide an action to download last version of clojure lsp
implement required LSP command like code-lens-references command

@angelozerr thanks for the explanation, it completly makes sense!
I wish we had that plugin when I started developing this one hehe, but I agree it makes sense to leave the LSP logic to LSP4IJ indeed, especially because of semantic tokens which I know is hard to support on client side, it's just that it's a huge transition we'd need to make carefully, especially because there are lots of people already using clojure-lsp-intellij plugin.

But if you have a plugin which tokenized the PsiFile with PsiElement you will not have this problem.

clojure-lsp-intellij already has a language parser for Clojure, why that still happens?
So, just summarizing some things I have in mind so we can be aligned:

I still need to test each existing feature and check it's working or the most important ones are working
we would need to launch a 3.0.0 (next major version) of clojure-lsp-intellij, which would delegate all LSP connection to LSP4IJ
I wanna a way to download clojure-lsp automatically the same way it does ATM, is there any way to have this logic working and after downloaded manually trigger the LSP start like we do today?
about the code-lens-reference, got it, it should be easy to support it.
we would need to fix this underlined issue
Also, there is a behavior I'd like you to consider supporting, which is when Ctrl+click on a definition, we show the references, it's something I brought from Cursive which Intellij users use a lot instead the need to manually triggering the find-references shortcut, WDYT?

BTW, really cool project the LSP4IJ, it remembers me when we extracted clojure-lsp logic to https://github.com/clojure-lsp/lsp4clj 😄

clojure-lsp-intellij already has a language parser for Clojure, why that still happens?

How have you tested LSP4IJ,with your plugin? If you did that you should have duplicate completion (one from your pluin, and one from LSP4IJ), right?

I still need to test each existing feature and check it's working or the most important ones are working

Exactly, when I discovered your project, I have spend some time to check the feature that you have implemented before annoying you and I think a feature that you will loose is the code action available from the refactor menu. I have not implemented this feature. Perhaps LSP4IJ will need to provide some customization, I don't know, I'm waiting for feedback from adaptor like you if you decide to migrate to LSP4IJ.

we would need to launch a 3.0.0 (next major version) of clojure-lsp-intellij, which would delegate all LSP connection to LSP4IJ

Indeed it will be more careful if you decide to adopt LSP4IJ.

I wanna a way to download clojure-lsp automatically the same way it does ATM, is there any way to have this logic working and after downloaded manually trigger the LSP start like we do today?

Yes, LSP4IJ provide an API to enable/disable and start/stop language server. See https://github.com/redhat-developer/lsp4ij/blob/main/docs/DeveloperGuide.md#language-server-manager

about the code-lens-reference, got it, it should be easy to support it.

I think.Please note that it exists the default command https://github.com/redhat-developer/lsp4ij/blob/main/docs/DeveloperGuide.md#editoractionshowreferences perhaps you could declare you action by using the class of this action?

we would need to fix this underlined issue

Also, there is a behavior I'd like you to consider supporting, which is when Ctrl+click on a definition, we show the references, it's something I brought from Cursive which Intellij users use a lot instead the need to manually triggering the find-references shortcut, WDYT?

Today Ctrl+Click is associate to the textDocument/definition, you woule like to associate to textDocument/references? I would like to avoid doing that because references can be expensive, but if you need it we could provide a customization or perhaps you could implement on your side the GoToDeclarationHandler with references. The main thing to do is to create issues on LSP4IJ to discuss about that

How have you tested LSP4IJ,with your plugin? If you did that you should have duplicate completion (one from your pluin, and one from LSP4IJ), right?

I disabled my plugin, enabled yours and restarted intellij

a feature that you will loose is the code action available from the refactor menu.

TBH, That's not a core feature, if code actions are provided via the context menu (lightbulb), that's its good enough. It'd be nice though to support that in the future

provide an API to enable/disable and start/stop language server.

awesome, so I think that will keep working as expected

perhaps you could declare you action by using the class of this action?

Ah, yeah, that should be enough!

Today Ctrl+Click is associate to the textDocument/definition, you woule like to associate to textDocument/references?

Yeah, first time I heard about that I disliked too, but I understood later, the point is that if you are already on the definition, you don't want to go to definition, but the references of it, so users can use the same shortcut to go and come back.
IMO would be nice to have this in the core without the need to support any customization, but if you disagree, a customization would be really useful.
I'll find some time during the next week to deep dive and check if there are other bugs, thanks again for raising this.

I disabled my plugin, enabled yours and restarted intellij

Ok so it means that you have not a PsiFile for clojure, right?

TBH, That's not a core feature, if code actions are provided via the context menu (lightbulb), that's its good enough.

It is and with fast performance (it should)

It'd be nice though to support that in the future

Please create a new ossue for that with your detailled idea (ex : do you want some customization of the icon,etc)

IMO would be nice to have this in the core without the need to support any customization, but if you disagree, a customization would be really useful.

Let's create a new issue to discuss about that and see the feedback from the community.

I'll find some time during the next week to deep dive and check if there are other bugs, thanks again for raising this.

Great!

Ok so it means that you have not a PsiFile for clojure, right?

Ah makes sense 😅, so if we had clojure-lsp-intellij using lsp4ij and defining a Parser like it does today, that would work, right? BTW, with lsp4ij, would be required to change anything related to Langauge declaration, like parser, lexer etc?

Please create a new ossue for that with your detailled idea (ex : do you want some customization of the icon,etc)

Will do!

Ok so it means that you have not a PsiFile for clojure, right?

Ah makes sense 😅, so if we had clojure-lsp-intellij using lsp4ij and defining a Parser like it does today, that would worinspecter?

Yes!
BTW, with lsp4ij, would be required to change anything related to Langauge declaration, like parser, lexer etc?

No LSP4IJ works with or without a PsiFile

Please create a new ossue for that with your detailled idea (ex : do you want some customization of the icon,etc)

Will do!

@ericdallo just for your information, we will release 0.6.0 next week and we provide a cool new feature with document symbol.
Here a screenshot with clojure:

Awesome!
I still need to check the feature parity with current features so we can consider switch to LSP4IJ

Awesome!

Glad it pleases you!
For some shortcut, I think it will not the same thing than you, since I have not found a solution to override the Action menu for Find implementation for instances https://github.com/redhat-developer/lsp4ij/blob/main/docs/LSPSupport.md#implementation
I had to provide new shortcut. I don'tunderstand how you have manage this usecase without breaking this feature for standard IntelliJ plugin (ex : Java plugin which doesn't use LSP).

@ericdallo I have started the 0.8.0 release of LSP4IJ which provides type + call hierarchy. Here a sample with clojure:

See https://github.com/redhat-developer/lsp4ij/blob/main/docs/LSPSupport.md#call-hierarchy to know how to integrate call hierarchy with clojure since you are using a custom clojure language


Allow versions popup to scroll

@Blendify can you please verify this? I have no project on RTD with enough versions to check if this fixes #575.
I've successfully tested against your url with chrome devtools though.
Note that I have removed the top border from that section, since it looked bad when the versions popup was fully expanded. It also fixes something I've always found odd. If this is desired, we can put it back in.
Before:

After:

@Blendify I saw that padding and decided to leave it alone for this PR since it is unrelated. It might even be desired so that there is some space below the last entry in the navbar.
It's also set by both the theme (to 2em) and by CSS inserted on RTD (to 3em), so changing it is a multistep process.


Running tslint with files specified don't respect tslint.config exclude options

Bug Report

TSLint version: 5.11.0
TypeScript version: 3.2.2
Running TSLint via:  vue-cli

Reproduce
tslint.json
  "linterOptions": {
    "exclude": [
      "node_modules/**",
      "src/**/*.d.ts"
    ]
  },

Running the following command
npx tslint src/types/shims.d.ts
Actual behavior
The shims.d.ts file is being linted.
Expected behavior
The shims.d.ts should not be linted and tslint should respect tslint.json exclude section.
Motivation
Using git pre commit hook with lint-staged should work smoothly with tslint. Currently, files that excluded are being linted.

This should be related to https://github.com/vuejs/vue-cli/blob/dev/packages/%40vue/cli-plugin-typescript/lib/tslint.js

This should be bug of globby