🐛 [firebase_ui_oauth_google] 'AuthProvider' is imported from both

Bug report
Receiving Error 'AuthProvider' is imported from both ... on run when using firebase_ui_oauth_google version ^1.2.10.
Describe the bug
When importing firebase_ui_oauth_google: ^1.2.10 I need to use dependency_overrides and force usage of firebase_auth: "4.12.0" to avoid the error.
Steps to reproduce

Clone https://github.com/j1mmyto9/flutter-boilerplate
flutter pub add firebase_ui_oauth_google
Configure Firebase per https://firebase.google.com/codelabs/firebase-auth-in-flutter-apps#2

Expected behavior
I expect provider class names to not clash.
Sample project
TODO

Additional context
N/A

Flutter doctor
Run flutter doctor and paste the output below:
Click To Expand
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.13.8, on macOS 13.2.1 22D68 darwin-x64, locale en-US)
[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 14.3.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.3)
[✓] IntelliJ IDEA Community Edition (version 2019.3.3)
[✓] VS Code (version 1.82.2)
[✓] Connected device (2 available)
[✓] Network resources

• No issues found!

Flutter dependencies
Run flutter pub deps -- --style=compact and paste the output below:
Click To Expand
Dart SDK 3.1.4
Flutter SDK 3.13.8
my_app 1.0.0+1

dependencies:
- auto_route 6.4.0 [flutter path collection meta]
- auto_size_text 3.0.0 [flutter]
- bot_toast 4.1.3 [flutter]
- cached_network_image 3.3.0 [cached_network_image_platform_interface cached_network_image_web flutter flutter_cache_manager octo_image]
- cupertino_icons 1.0.6
- d20 0.3.1+1 [math_expressions]
- dart_json_mapper 2.2.9 [analyzer build build_config intl meta path reflectable collection pubspec_parse]
- device_info_plus 8.2.2 [device_info_plus_platform_interface ffi file flutter flutter_web_plugins meta win32]
- environment_config 3.1.0 [dart_style code_builder args yaml]
- equatable 2.0.5 [collection meta]
- firebase_ui_oauth_google 1.2.10 [firebase_auth firebase_ui_oauth flutter google_sign_in]
- flutter 0.0.0 [characters collection material_color_utilities meta vector_math web sky_engine]
- flutter_bloc 8.1.3 [bloc flutter provider]
- flutter_dotenv 5.1.0 [flutter]
- flutter_localizations 0.0.0 [flutter intl characters clock collection material_color_utilities meta path vector_math web]
- flutter_slidable 2.0.0 [flutter]
- flutter_svg 2.0.9 [flutter vector_graphics vector_graphics_codec vector_graphics_compiler]
- formz 0.5.0+1 [meta]
- get_it 7.6.4 [async collection]
- http 0.13.6 [async http_parser meta]
- logger 1.4.0
- modal_bottom_sheet 3.0.0-pre [flutter]
- package_info_plus 3.1.2 [ffi flutter flutter_web_plugins http meta path package_info_plus_platform_interface win32]
- shared_preferences 2.2.2 [flutter shared_preferences_android shared_preferences_foundation shared_preferences_linux shared_preferences_platform_interface shared_preferences_web shared_preferences_windows]
- shimmer 2.0.0 [flutter]
- url_launcher 6.2.2 [flutter url_launcher_android url_launcher_ios url_launcher_linux url_launcher_macos url_launcher_platform_interface url_launcher_web url_launcher_windows]

dev dependencies:
- auto_route_generator 6.2.0 [build source_gen analyzer path build_runner code_builder dart_style xml args glob auto_route]
- build_runner 2.4.7 [analyzer args async build build_config build_daemon build_resolvers build_runner_core code_builder collection crypto dart_style frontend_server_client glob graphs http_multi_server io js logging meta mime package_config path pool pub_semver pubspec_parse shelf shelf_web_socket stack_trace stream_transform timing watcher web_socket_channel yaml]
- flutter_lints 2.0.3 [lints]
- flutter_test 0.0.0 [flutter test_api matcher path fake_async clock stack_trace vector_math async boolean_selector characters collection material_color_utilities meta source_span stream_channel string_scanner term_glyph web]

dependency overrides:
- firebase_auth 4.12.0 [firebase_auth_platform_interface firebase_auth_web firebase_core firebase_core_platform_interface flutter meta]

transitive dependencies:
- _fe_analyzer_shared 61.0.0 [meta]
- _flutterfire_internals 1.3.16 [collection firebase_core firebase_core_platform_interface flutter meta]
- analyzer 5.13.0 [_fe_analyzer_shared collection convert crypto glob meta package_config path pub_semver source_span watcher yaml]
- args 2.4.2
- async 2.11.0 [collection meta]
- bloc 8.1.2 [meta]
- boolean_selector 2.1.1 [source_span string_scanner]
- build 2.4.1 [analyzer async convert crypto glob logging meta package_config path]
- build_config 1.1.1 [checked_yaml json_annotation path pubspec_parse yaml]
- build_daemon 4.0.1 [built_collection built_value crypto http_multi_server logging path pool shelf shelf_web_socket stream_transform watcher web_socket_channel]
- build_resolvers 2.4.2 [analyzer async build collection convert crypto graphs logging package_config path pool pub_semver stream_transform yaml]
- build_runner_core 7.2.11 [async build build_config build_resolvers collection convert crypto glob graphs json_annotation logging meta package_config path pool timing watcher yaml]
- built_collection 5.1.1
- built_value 8.8.1 [built_collection collection fixnum meta]
- cached_network_image_platform_interface 3.0.0 [flutter flutter_cache_manager]
- cached_network_image_web 1.1.0 [cached_network_image_platform_interface flutter flutter_cache_manager]
- characters 1.3.0
- checked_yaml 2.0.3 [json_annotation source_span yaml]
- clock 1.1.1
- code_builder 4.9.0 [built_collection built_value collection matcher meta]
- collection 1.17.2
- convert 3.1.1 [typed_data]
- crypto 3.0.3 [typed_data]
- dart_style 2.3.2 [analyzer args path pub_semver source_span]
- desktop_webview_auth 0.0.12 [crypto flutter http flutter_web_plugins plugin_platform_interface]
- device_info_plus_platform_interface 7.0.0 [flutter meta plugin_platform_interface]
- email_validator 2.1.17
- fake_async 1.3.1 [clock collection]
- ffi 2.1.0
- file 6.1.4 [meta path]
- firebase_auth_platform_interface 7.0.9 [_flutterfire_internals collection firebase_core flutter meta plugin_platform_interface]
- firebase_auth_web 5.8.12 [firebase_auth_platform_interface firebase_core firebase_core_web flutter flutter_web_plugins http_parser js meta]
- firebase_core 2.24.2 [firebase_core_platform_interface firebase_core_web flutter meta]
- firebase_core_platform_interface 5.0.0 [collection flutter flutter_test meta plugin_platform_interface]
- firebase_core_web 2.10.0 [firebase_core_platform_interface flutter flutter_web_plugins js meta]
- firebase_dynamic_links 5.4.8 [firebase_core firebase_core_platform_interface firebase_dynamic_links_platform_interface flutter meta plugin_platform_interface]
- firebase_dynamic_links_platform_interface 0.2.6+16 [_flutterfire_internals firebase_core flutter meta plugin_platform_interface]
- firebase_ui_auth 1.7.0 [email_validator firebase_auth firebase_core firebase_dynamic_links firebase_ui_localizations firebase_ui_oauth firebase_ui_shared flutter flutter_localizations]
- firebase_ui_localizations 1.9.0 [flutter flutter_localizations path]
- firebase_ui_oauth 1.4.10 [desktop_webview_auth firebase_auth firebase_ui_auth firebase_ui_shared flutter_svg flutter]
- firebase_ui_shared 1.4.1 [flutter]
- fixnum 1.1.0
- flutter_cache_manager 3.3.1 [clock collection file flutter http path path_provider rxdart sqflite uuid]
- flutter_web_plugins 0.0.0 [flutter characters collection material_color_utilities meta vector_math web]
- frontend_server_client 3.2.0 [async path]
- glob 2.1.2 [async collection file path string_scanner]
- google_identity_services_web 0.2.2 [js meta]
- google_sign_in 6.1.6 [flutter google_sign_in_android google_sign_in_ios google_sign_in_platform_interface google_sign_in_web]
- google_sign_in_android 6.1.20 [flutter google_sign_in_platform_interface]
- google_sign_in_ios 5.6.5 [flutter google_sign_in_platform_interface pigeon]
- google_sign_in_platform_interface 2.4.3 [flutter plugin_platform_interface]
- google_sign_in_web 0.12.2+1 [flutter flutter_web_plugins google_identity_services_web google_sign_in_platform_interface http js]
- graphs 2.3.1 [collection]
- http_multi_server 3.2.1 [async]
- http_parser 4.0.2 [collection source_span string_scanner typed_data]
- intl 0.18.1 [clock meta path]
- io 1.0.4 [meta path string_scanner]
- js 0.6.7 [meta]
- json_annotation 4.8.1 [meta]
- lints 2.1.1
- logging 1.2.0
- matcher 0.12.16 [async meta stack_trace term_glyph test_api]
- material_color_utilities 0.5.0 [collection]
- math_expressions 2.4.0 [vector_math]
- meta 1.9.1
- mime 1.0.4
- nested 1.0.0 [flutter]
- octo_image 2.0.0 [flutter]
- package_config 2.1.0 [path]
- package_info_plus_platform_interface 2.0.1 [flutter meta plugin_platform_interface]
- path 1.8.3
- path_parsing 1.0.1 [vector_math meta]
- path_provider 2.1.1 [flutter path_provider_android path_provider_foundation path_provider_linux path_provider_platform_interface path_provider_windows]
- path_provider_android 2.2.2 [flutter path_provider_platform_interface]
- path_provider_foundation 2.3.1 [flutter path_provider_platform_interface]
- path_provider_linux 2.2.1 [ffi flutter path path_provider_platform_interface xdg_directories]
- path_provider_platform_interface 2.1.1 [flutter platform plugin_platform_interface]
- path_provider_windows 2.2.1 [ffi flutter path path_provider_platform_interface win32]
- petitparser 5.4.0 [meta]
- pigeon 11.0.1 [analyzer args collection meta path yaml]
- platform 3.1.3
- plugin_platform_interface 2.1.7 [meta]
- pool 1.5.1 [async stack_trace]
- provider 6.1.1 [collection flutter nested]
- pub_semver 2.1.4 [collection meta]
- pubspec_parse 1.2.3 [checked_yaml collection json_annotation pub_semver yaml]
- reflectable 4.0.5 [analyzer build build_resolvers build_config build_runner build_runner_core dart_style glob logging package_config path source_span]
- rxdart 0.27.7
- shared_preferences_android 2.2.1 [flutter shared_preferences_platform_interface]
- shared_preferences_foundation 2.3.4 [flutter shared_preferences_platform_interface]
- shared_preferences_linux 2.3.2 [file flutter path path_provider_linux path_provider_platform_interface shared_preferences_platform_interface]
- shared_preferences_platform_interface 2.3.1 [flutter plugin_platform_interface]
- shared_preferences_web 2.2.1 [flutter flutter_web_plugins shared_preferences_platform_interface]
- shared_preferences_windows 2.3.2 [file flutter path path_provider_platform_interface path_provider_windows shared_preferences_platform_interface]
- shelf 1.4.1 [async collection http_parser path stack_trace stream_channel]
- shelf_web_socket 1.0.4 [shelf stream_channel web_socket_channel]
- sky_engine 0.0.99
- source_gen 1.5.0 [analyzer async build dart_style glob path source_span yaml]
- source_span 1.10.0 [collection path term_glyph]
- sprintf 7.0.0
- sqflite 2.3.0 [flutter sqflite_common path]
- sqflite_common 2.5.0+2 [synchronized path meta]
- stack_trace 1.11.0 [path]
- stream_channel 2.1.1 [async]
- stream_transform 2.1.0
- string_scanner 1.2.0 [source_span]
- synchronized 3.1.0+1
- term_glyph 1.2.1
- test_api 0.6.0 [async boolean_selector collection meta source_span stack_trace stream_channel string_scanner term_glyph]
- timing 1.0.1 [json_annotation]
- typed_data 1.3.2 [collection]
- url_launcher_android 6.2.0 [flutter url_launcher_platform_interface]
- url_launcher_ios 6.2.1 [flutter url_launcher_platform_interface]
- url_launcher_linux 3.1.1 [flutter url_launcher_platform_interface]
- url_launcher_macos 3.1.0 [flutter url_launcher_platform_interface]
- url_launcher_platform_interface 2.2.0 [flutter plugin_platform_interface]
- url_launcher_web 2.2.0 [flutter flutter_web_plugins url_launcher_platform_interface]
- url_launcher_windows 3.1.1 [flutter url_launcher_platform_interface]
- uuid 4.2.2 [crypto sprintf meta]
- vector_graphics 1.1.9+1 [flutter vector_graphics_codec]
- vector_graphics_codec 1.1.9+1
- vector_graphics_compiler 1.1.9+1 [args meta path_parsing xml vector_graphics_codec path]
- vector_math 2.1.4
- watcher 1.1.0 [async path]
- web 0.1.4-beta
- web_socket_channel 2.4.0 [async crypto stream_channel]
- win32 4.1.4 [ffi]
- xdg_directories 1.0.3 [meta path]
- xml 6.3.0 [collection meta petitparser]
- yaml 3.1.2 [collection source_span string_scanner]

Hi @Xerosigma, where is AuthProvider used in your app? I followed the steps you outlined above but I do not get the error when I run the app. What platform(s) do you see this issue on?

There's nothing from firebase_ui except a renaming of AuthProvider to something else which would be a breaking change without much value.
There's several ways of dealing with name conflicts, such as:
import 'package:firebase_auth/firebase_auth.dart' hide AuthProivder;

or
import 'package:firebase_auth/firebase_auth.dart' as fba;

or
import 'package:firebase_auth/firebase_auth.dart' show EmailAuthProvider;

I had this issue but I solved it by updating the dependencies. Turns out I haven't upgrade the packages for a long time and my firebase_ui_auth was version 1.4.0. and my environment was set to sdk: ">=2.17.6 <3.0.0". I updated them and with the newest version of firebase_ui_auth the problem no longer exists.


Error handling for the Agent

Although PRISM already has some errors defined and returned by our Cloud Agent RestAPI calls, we do not have any mechanism to handle and report errors when they are happening during record processing or sending/receiving DIDComm messages. It is required to implement a mechanism of error handling for Cloud Agent to log, update records in the database, and report errors in the right way.

[ ] hyperledger/identus-edge-agent-sdk-ts/issues/308

@elribonazo this is done no?

I'm in favor of close this ticket. But feel free to open it again


Zenodo?

Should we mention Zenodo somewhere? They offer DOI for datasets, and (being CERN-operated) have both super-duper large storage and are a grassrot initiative.

I mentioned it in Rule 5 if that is any help. You can see the text in PR #50

I think it's worth it as an example of where to store your data

I think we have this covered, so I'm closing.


Update to August 8

this change should be backwards-compatible with August 4, so we can just bump the minor


Reduce compilation time

 Issue by CreepySkeleton
Tuesday Jul 28, 2020 at 11:47 GMT
Originally opened as https://github.com/clap-rs/clap/issues/2037

All features
No features
The clap crate (not counting dependencies) takes about 11 seconds to compile. All dependencies put together take roughly the same amount of time.
I have a few thoughts on how to improve the situation. Generally, compile time wins these days are accomplished through "compile less code", hence:

[ ] Feature-gate #[derive(Clone, Copy, Eq...]. The features should probably be fine-grained, like impl-traits-eq enables Eq + PartialEq. Plus the "enable everything" impl-traits feature that implies all the traits.

[ ] Avoid using derives and use some sort of STRUCT! macro as shown here. It turns out that even built-in derives have impact on compilation time when you have lots of them, and every struct here has #[derive(Debug)].

[x] Optimize proc-macro-error (drop syn-mid dep). I had already done that but decided to play around a bit more and then forgot about it. Oops.
[ ] Make clap modular so developers could choose exactly the set of features they need and avoid compile the others. I'll crate a separate issue a bit later.

The list is probably not exhaustive...

 Comment by mainrs
Thursday Jul 30, 2020 at 11:31 GMT

Hey!
If you want I can tackle down the trait implementation features :) How fine-grained do you want it to be though?

 Comment by CreepySkeleton
Friday Jul 31, 2020 at 12:24 GMT

Sure, you can try, but keep in mind that the real goal is to reduce compilation time. If the measurements will show no real difference (at least half second, everything below the threshold is noise), I will say "well we tried" and move on to other ideas.

 Comment by CreepySkeleton
Friday Jul 31, 2020 at 14:23 GMT

The invocation was
cargo clean # make sure the we build from scratch
cargo +nightly build -Z timings -p clap:3.0.0-beta.1 --features "yaml unstable"

This will create cargo-timing-<HASH>.html file that you can open in your web browser and see the adorable stats you were so excited about :) Then I uploaded the raw html on gist.github.com and linked it through gistpreview.github.io so you see the rendered version instead of raw HTML guts.

 Comment by mainrs
Friday Jul 31, 2020 at 13:35 GMT

Sure thing! Could you tell me how to managed to time the compilation? And create the visuals for it as well :)


Schema changes in nested objects do not reflect in parent object

Prerequisites

[x] Are you running the latest version of the plugin? (Check releases)
[x] Are you running a supported version of Gradle? (Check the README)
[x] Are you running a supported version of Apache Avro? (Check the README)
[x] Are you running a supported version of Java? (Check the README)
[x] Did you check to see if an issue has already been submitted?
[x] Are you reporting to the correct repository?
[x] Did you perform a cursory search?

I updated the schema of a class (class B) that is used by another avro schema (class A).
Class A's SCHEMA$ variable was not updated with the new nested fields, resulting in serialization errors.

Thank you for the issue report. A sample project with detailed reproduction steps would assist greatly with ensuring that we’re discussing the same behavior.

It seems possible that you are running into problems related to the behavior introduced by AVRO-150.  Specifically, in SpecificCompiler, when asked to compile a schema to Java code, it checks whether the destination file is more recent (by file system last modified timestamp) than the source file, and if so, silently skips doing anything.
In most cases, that should be fine: you change a file (which makes the last modified newer than the previous build artifacts), you run the build, and it re-compiles.  In some cases, it can be sort of weird, though.  Some examples:

Your source files are in a version control system that messes with last modified timestamps, and you check out a different branch, resulting in source files that have changed but have older time stamps
Your system's concept of what time it is goes backwards (daylight saving time, time zone change, NTP, unreliable system clock)
You have multiple schema files that result in the generation of a given Java source file; the generation based on one of them then results in the generated file being more recent than the other source files

For many such situations, deleting the generated Java source files (such as via the clean task) and re-running the build should be sufficient to get to a consistent state.
My plugin attempts to prevent situations where unreliable results are produced by parsing all source schemas that it has been provided, checking for duplicate definitions of any types (nested or otherwise) and intentionally failing if any type definitions conflict (that is, there are two or more definitions for what the contents of a given Java source file should be).  That said, there are ways to get into this sort of situation that the plugin couldn't detect.  A couple possible solutions to this class of problem would be to ensure that all class generation for a given destination directory happens in a single GenerateAvroJavaTask run or extracting all nested types to their own source files.

CC: @tomasAlabes

Thanks for the research @davidmc24! In my case cleaning the build folder with the classes wasn't enough, I had to go for the build cache. After that it worked again.

@tomasAlabes Thanks for the explicit call-out of the build cache.  I don't have much experience with that yet, and wouldn't have thought to clear it.

@TalMoshel Actually, upon re-reading your submission, I think I understand now that you're referring to a related but different problem in this ticket than ones I've attempted to help with before.  If you have nested types extracted to separate files, but don't change the parent file, and re-compile, Avro will refuse to re-write the parent file.
This is a type of failure that will be resolved by a clean build, and that's the behavior that the upstream project seems to want to encourage by how they handle this.
In terms of ways that this plugin might be able to do better...

Previous versions of this plugin pro-actively cleaned their destination folders to work around this behavior.  This caused problems for some people who configured their destination to be a source folder; we won't be going in that direction again.
It might be possible to update the last modified timestamp of the source file immediately before calling SpecificCompiler to compile it.  I'm not sure what additional impact that would have.
We could compile the files to a temporary directory, and then pull the results into the real destination, at the cost of additional I/O.

Beyond that, I think the main way of changing it would be upstream (to remove, or at least allow disabling the option in Avro's SpecificCompiler class).

I have the same problem and for me even the cache cleanning it wasn't enought. My schema looks like this:
{
    "type": "record",
    "name": "SomeEvent",
    "namespace": "someNameSpace",
    "fields": [
        {
            "name": "data",
            "type": {
                "type": "record",
                "name": "SomeEventData",
                "fields": [
                    {
                        "name": "Boleto",
                        "type": {
                            "type": "record",
                            "name": "Boleto",
                            "fields": [
                                {
                                    "name": "data_vencimento",
                                    "type": [
                                        "string"
                                    ],
                                    "default": null
                                }
                            ]
                        }
                    }
                ]
            }
        }
    ]
}


[FIX] JPA blocking calls should be moved to bounded elastic

Before
openjpa (blocking) calls on the IMAP event loop

After
No openjpa (blocking) calls on the IMAP event loop

https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-1579/1/testReport/
Looks related

So I tracked down and detected the leak:
Spawn a dockerized JPA james locally thanks to docker and imapsync my company emails to it...
Before

GC is unable to reclaim memory, showing a leak that eventually builds up to an OOM.
After

We are waaaay better at reclaiming the damn memory.

I ran a test with the changes (though, on my local machine and not remote deployment) and it seems to be behaving better:

While simultaneously running the profiler I noticed that, it seems, James/OpenJPA doesn't use any connection pool and simply re-opens connection to the database for each query (?!):

Documentation (https://james.apache.org/server/config-system.html) states, that James delegates connectivity to commons-dbcp, but it doesn't seem to be the case. I even tried to force pool size but still - connections being opened and closed

Hmm... it seems something is still leaking:

I must confess I did only run a local derby database thus might have mess the connection churn...

While simultaneously running the profiler I noticed that, it seems, James/OpenJPA doesn't use any connection pool and simply re-opens connection to the database for each query (?!)

Confirmed by flame graphs running mariaDB...

Adding DBCP2 on the classpath seems to be enough to solve the connection churn issue...

I no longer spot obvious connects in the flae graph of james + mariaDB...
Full flame graph:
msql2.zip

With dbcp2 in classpath I was getting lot of "multithred" exceptions. I had openjpa.Multithreaded=true in james-database.properties but upon debugging I noticed it didn't take any effect (getting properties from injected EntityManager). Including it directly in JPAEntityManagerModule fixed that problem (most likely passing it via -D commandline parameter would work as well). Memory usage is way better and kept (relatively) low, though copying/appending messages is still quite slow (~2msg/s, with low CPU usage so still seems like IO-bound issue):

I had openjpa.Multithreaded=true in james-database.properties

Should be in jvm.properties

I had openjpa.Multithreaded=true in james-database.properties

Should be in jvm.properties

That's true. When I searched for it, I didn't pay enough attention and saw "...properties" and assumed it was database properties as it seemed related...

I did run some performance test on top of this branch (local mariaDB setup), 50 user simulation on top of 5 accounts with ~50 mailboxes and ~1000 emails...
Results were better than I expected:

Though SELECT and STATUS are shockingly slow (p50 -> p99) and the biggest contributor to latencies. I suspect those loads all messages...
Also this PR as it stands breaks soe tests (did not had time to dig...)

Is it possible to to get exact sql queries/parameters? To possibly try to optimise it somehow?

We did get that with glowroot in the past for Cassandra...

is it related? https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-1579/9/testReport/

@chibenwa ?


Accessibility features for markdown files

Add accessibility features for all markdown based on: https://github.blog/2023-10-26-5-tips-for-making-your-github-profile-page-accessible/
Checked all markdown files in the main folder, others are all okay :wink:

@mishmanners I attempted to make these modifications but it appears you have the Allowing changes to a pull request branch created from a fork disabled.

I'll do that today!

Should be fixed now @frankhinek

@mishmanners I tried to resolve the merge conflicts in this PR, but it appears you'll have to rebase on the current main branch and push up a commit before this PR can be merged.  The primary reason is that the "Hacktoberfest" header was already removed from README.md and this PR still contains references to that text.
Once you rebase, we should be able to merge.  Thanks again for the contribution.

So this says merged now. All good @frankhinek? Sorry for the delay I was away for a couple of days without much internet.


Lokalise: Translations update for existing languages

This pull request was initiated by Lokalise (user Do Hee Kim) at 2020-12-16 22:57:06

@adorapham Here is the new PR. Does this look okay? Thanks!


Changed expected location of ca.pem

When this commit got merged 62fe7fcd78c901957c1ceb79e410ca72e668864d the location of the ca.pem file is no longer being read from the root of the projects (where the publish command places it) but now it's expected to be inside /config directory.
I don't know if this behaviour is intended (so we should change the makefile publish) or just a side effect and should be fixed.
I could help with it and provide a fix but I'd like to know what do you think.

Good catch, that's a bug.  I'll get a fix out for this.

Merged https://github.com/Netflix/bless/pull/50


Ignore target directory that produced by cargo

This change copy .gitignore that generated by cargo new to rustler.new
task to ignore target directory produced by cargo build or rls which's
Language Server for Rust.

The CI / OTP 22.x / Elixir 1.9 / Rust nightly (pull_request) failure due to download rust nightly issue.

❤


Implement PSR-18

I've tried to implement PSR-18 to this library. But as you (@clue) told me a year ago, it might be impossible to do sync requests with this library. In guzzle we do something like:
public function sendRequest(RequestInterface $request): ResponseInterface
{
    return $this->sendAsyncRequest($request)->wait();
}

Nevertheless, I'm about to open the vote on the specification and would like to hear your comments before that.
See the specification: https://github.com/php-fig/fig-standards/blob/master/proposed/http-client/http-client.md
Read the blog post: https://medium.com/php-fig/the-http-client-psr-9c2535132980

@Nyholm Thank you for reaching out and this excellent question!
think you did an excellent job with the specification and the reasoning behind this as detailed in your previous Medium post. To me, PSR-18 looks good as-is and it makes perfect sense that async requests are of scope for this iteration. Traditional blocking and non-blocking code just don't mix well.
That being said, making any promise-based async API in ReactPHP available as a blocking API is relatively easy and works very similar to the example snippet you've posted above. By combining https://github.com/clue/reactphp-buzz#send with https://github.com/clue/reactphp-buzz#blocking, you end up with this (pseudo-code):
class Psr18Adaptor
{
    private $browser;

    public function __construct(LoopInterface $loop, Browser $browser = null)
    {
        if ($browser === null) {
            $browser = new Browser($loop);
        }

        $this->loop = $loop;
        $this->browser = $browser;
    }

    public function sendRequest(RequestInterface $request): ResponseInterface
    {
        return Block\await($this->browser->send($request), $this->loop);
    }
}

It's my understanding that having blocking code as part of a non-blocking library should usually be avoided as it may confuse newcomers. I think it makes sense to just leave this here for anybody who's interested in using an async HTTP client through the (blocking) PSR-18 API.
I hope this helps :+1:

I agree with you. Thank you.
Aslo FYI. The readiness vote is open: https://groups.google.com/forum/#!topic/php-fig/BsroPEZcr_M


Docs.MS Release Updates for azure-sdk-template

Update docs metadata and targeting for release of azure-sdk-template

Docs Build status updates of commit 2a971f3:
:clock10: Full build: the average full build time is 257 min(s) 30 sec(s), this is based on the last 881 build(s) for this repository.

Docs Build status updates of commit 2a971f3:
:warning: Validation status: warnings

File
Status
Preview URL
Details

docs-ref-services/sdk-template-readme-pre.md
:warning:Warning
View (azure-java-stable) View (azure-java-preview) View (azure-java-legacy)
Details

legacy/docs-ref-autogen/overview/EventHubs/Client.yml
:warning:Warning
View (azure-java-legacy)
Details

package.json
:white_check_mark:Succeeded

docs-ref-services/sdk-template-readme-pre.md

Line 56: [Warning-InvalidBookmark] Invalid link: 'Introduction'. The file docs-ref-services/sdk-template-readme-pre.md doesn't contain a bookmark named 'introduction'.
Line 56: [Warning-InvalidBookmark] Invalid link: 'Introduction'. The file docs-ref-services/sdk-template-readme-pre.md doesn't contain a bookmark named 'introduction'.
Line 56: [Warning-InvalidBookmark] Invalid link: 'Introduction'. The file docs-ref-services/sdk-template-readme-pre.md doesn't contain a bookmark named 'introduction'.

legacy/docs-ref-autogen/overview/EventHubs/Client.yml

[Warning-DuplicateUids] Uid(azure.java.sdk.landingpage.services.eventhub.Client) has already been defined in docs-ref-services/messaging-eventhubs-readme.md.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Added New German Formats and Adjusted French Format

Added new formats for German.
InQUOTE_HDR_REGEX:
-format for parsing German. End of customer message: 'Ende der Kundennachricht'
In _MULTI_QUOTE_HDR_REGEX:

format for parsing German. "Am 21.06.2024 um 09:49 schrieb X"
format for parsing German. "X chrieb am Do. 20. Juni 2024 um 14:49:"
format for parsing German. Original message: "-------- Ursprüngliche Nachricht --------"
format for parsing English/German. Forwarded message: "-------- Forwarded message --------"

In EMAIL_SIGNOFF_REGEX:

format for parsing German. 'Mit freundlichen Grüßen'
format for parsing German. 'Freundliche Grüße'

Also adjusted for cases when a colon isn't used for the french format in _MULTI_QUOTE_HDR_REGEX.

Separate to this it would maybe be also nice to write a test for this bit of regex r"|(-{,10})(\s*)?(Ursprüngliche|Original|Forwarded).?(Nachricht|Message)(\s*)?(-{,10})" #German/English as it is a bit scary!

I ended up updating the regex here for this after some testing:
r"|-(-*)?\s?(Ursprüngliche|Original|Forwarded).?(Nachricht|Message)\s?(-*)?-"
It was capturing just "Forwarded Message" on it's own so the regex should more accurately just capture if it's --- Forwarded Message ---.
I did write a test for this format previously: test_orginal_message_sign_off. I added a few more examples emails to the test, for each format variation.

Updated the word limit default value to 1000 for the cut_off_at_signature function as during testing larger texts were not being parsed correctly due to the previously set word limit.


Add an opaque field for necessary bindings struct

Without this, offsets of captures in closure contexts may be
incorrect if there is a non-empty necessary bindings structure
at the front.
rdar://problem/26312900

@swift-ci Please test

@swift-ci Please test


One vs all DE bug?

https://github.com/YosefLab/scVI/blob/336aeb0b1566717aea436b6837c48bd162b4d307/scvi/inference/posterior.py#L295
It's hard for me to tell if this is a bug, in particular, I believe sample_pairs should be True, otherwise it's not necessary to report permuted Bayes factors in this function. As an aside, it would be great if someone added docstrings to all DE functions :)

^ @romain-lopez

sample_pairs=False means that you do not subsample the pairs. I did not write any of the code in this section and it is extremely hard for me to write a docstring.
There are several open question to designing the API for the differential expression, notably:

Do we subsample the cells / from the variational distribution. For efficiency, we may need the first and for accuracy we might need the second. how many times then ?
Do we use importance sampling ? If yes, the code has to be substantially refactored. I partially did it in the scANVI paper repository, but there would be more work to put it in here and I am not sure yet we have a problem with our vanilla estimator.
What kind of high-level test do we want to perform ? One-VS-All, One-VS-One, etc...
How to perform DE with multiple batches, with scVI and scANVI ? A lot of code for this is on this repo and the scANVI repo, but there would be some refactorization to perform in order to make it readable. The multiple batches scenario is when the API really needs to be designed well.
Is the permutation test really informative or meaningful ? It was more of a sanity check when we wrote the scVI paper and I am not even sure it can be meaningfully used to assign a frequentist p-value for differential expression. If this is the case, might as well get rid of this code.
Do we want to reweight the rho values by the ZI weight for example ?

I was hoping we would get some partial answers from the work of Pierre and that would help us move forward on having a functional code for differential expression.

Thanks for the information about sample_pairs.
@PierreBoyeau It'd be great if you'd think about the issues Adam and Romain have raised. I'm not sure how many of the old DE functions you'll want to keep, but for those we do keep, it'd be good to docstrings.

Hey everyone,
For sure I will think about that. I can add docstrings but I think that refactoring right now could be a good idea.
@romain-lopez even if your scANVI implementation do not include all old DE functionalities, I think your new implementation is much clearer than before.
As a matter of fact I was working on refactoring the pipeline Friday based on your code. The idea would be to have a cleaner code, optional importance sampling, while keeping the same default behavior for DE.
What do you think?

yes that could be helpful !

Please let me know if you think I should not include importance sampling, if I should include gamma sampling later, if you have other remarks ...

It seems to me like importance sampling can wait as well as the gamma
sampling as we do not even know whether this is needed / useful.
On Mon, May 20, 2019 at 7:00 PM Pierre Boyeau<EMAIL_ADDRESS>wrote:

Please let me know if you think I should not include importance sampling,
if I should include gamma sampling later, if you have other remarks ...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/YosefLab/scVI/issues/342?email_source=notifications&email_token=AFCDZD33UU7UVPIOMYXBH6DPWNJVRA5CNFSM4HN2UCYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV2Q6JQ#issuecomment-494210854,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFCDZD4I6G2IGVGHOIOQM4LPWNJVRANCNFSM4HN2UCYA
.

Seems like Adam was just asking for some better docstrings. We can merge those first and close this issue. After that you may want to focus on importance sampling

Yes docstrings would be great! Also the confusion started here:
https://github.com/YosefLab/scVI/blob/master/tests/notebooks/basic_tutorial.ipynb
When one_vs_all_degenes is called you get the full table with BF and permuted BF, but it's my understanding that since sample_pairs is False that these should all have the same value. At least that's what happened when I ran it myself.

Hey Adam,
I updated the docstrings, feel free to tell me if you think they are clear enough.
I agree with you on this, I added a sample_pairs parameter to this method by default at False, I think this cannot hurt


Use 32 bits on binary encoding

This changes the binary encoding to not use *size types but *32 types. *size shouldn't be used since it is different sizes on different platforms so could lead to different issues. Also on 32 bit platforms we can't represent things like Vec to the same length as on 64 bit platforms so I think we should prefer to limit things where possible to 32 bits in length.
Relates to #70
What are your thoughts?

Agree that it would be good to have consistent behaviour across both 32-bit and 64-bit platforms. I will note that this limits documents to a maximum of 2^32 operations in their history. At the moment this is not a problem since Automerge would probably get unusably slow long before this limit is reached, but at some point in the future we may want to support longer histories. For that reason I think it's okay if the current implementation uses 32-bit variables, but I think the specification of the binary format should allow values greater than 2^32.

Is this desirable?  Under the current code a 64bit system could continue to run if it encountered a > 2^32 bit run on ops, but a 32bit system would fail.   Isn't that better than just making the 64bit system also fail?  (hobbling the 64bit system so it fails in the same way as a 32 bit one).  Is there another scenario I'm not thinking about?  I suppose its moot since we're likely never going to see a document of that size...

I think using usize should be fine then as it should allow 64 bit systems to run with larger docs, and it seems the lower level uses the variable encoding (leb128) for all number types anyway.
I'll repost this as a documentation PR.

Maybe we should put in an informative panic if usize max < the number that comes out of the leb128


Twitter example does not work on cluster

Not sure if it has anything to do with our code, is authentication error.
The same command works on the same machine when spark master is local[*]
16/03/17 08:38:28 ERROR scheduler.ReceiverTracker: Deregistered receiver for stream 0: Restarting receiver with delay 2000ms: Error starting Twitter stream - java.lang.IllegalStateException: Authentication credentials are missing. See http://twitter4j.org/en/configuration.html for details
at twitter4j.TwitterBaseImpl.ensureAuthorizationEnabled(TwitterBaseImpl.java:215)
at twitter4j.TwitterStreamImpl.sample(TwitterStreamImpl.java:161)
at org.apache.spark.streaming.twitter.TwitterReceiver.onStart(TwitterInputDStream.scala:93)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:148)
at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply$mcV$sp(ReceiverSupervisor.scala:197)
at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply(ReceiverSupervisor.scala:188)
at org.apache.spark.streaming.receiver.ReceiverSupervisor$$anonfun$restartReceiver$1.apply(ReceiverSupervisor.scala:188)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

The twitter examples pass the Authorization token via System.properties.  For some reason the system.properties values are lost when running on a cluster.  This works ok on Java, Scala, and local mode.
I think the auth is gotten from the properties only on the master node, not the workers.

Changed the example to use an alternate authorization which does require the use of System.properties.  We should still understand why the properties auth mechanism does not work in case it is a symptom of a larger underlying issue.

The java example also fails on a cluster, so there is no unique eclairjs problem


Need to parse data in WaitUntilReady.

Fixes problem when the robot is ready at startup. Otherwise calibration will not work.

Good point! Let me try that.

@paLeziart I changed the implementation based on your suggestion. The calibration works with this on the real robot as well.


no source files found

hi,
I´ve created a workflow as suggested for FreeRDP, see https://github.com/FreeRDP/FreeRDP/pull/9794
now, everything works fine until the actual clang-tidy run, where source files are not detected.
anything I´m missing here?

This check only runs on changed files of the PR.
No files were found in your PR because your PR didn't change and CPP files only workflow and cmake.

Try with 0.17.1 instead of 0.14.0. Looks like there's some trouble with the permissions, hopefully the newer versions fix that.

@bwrsandman ok, the newer version was a bit clearer with what is failing.
still, tried to add permissions: issues: write and still get
Request POST /repos/FreeRDP/FreeRDP/issues/9819/comments failed with 403: Forbidden
No warnings to report, LGTM!
Traceback (most recent call last):
  File "/usr/local/bin/review", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/clang_tidy_review/review.py", line 177, in main
    post_review(
  File "/usr/local/lib/python3.11/dist-packages/clang_tidy_review/__init__.py", line 1102, in post_review
    pull_request.post_lgtm_comment(lgtm_comment_body)
  File "/usr/local/lib/python3.11/dist-packages/clang_tidy_review/__init__.py", line 339, in post_lgtm_comment
    self.pull_request.create_issue_comment(body)
  File "/usr/local/lib/python3.11/dist-packages/github/PullRequest.py", line 437, in create_issue_comment
    headers, data = self._requester.requestJsonAndCheck("POST", f"{self.issue_url}/comments", input=post_parameters)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/github/Requester.py", line 494, in requestJsonAndCheck
    return self.__check(*self.requestJson(verb, url, parameters, headers, input, self.__customConnection(url)))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/github/Requester.py", line 629, in requestJson
    return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/github/Requester.py", line 726, in __requestEncode
    status, responseHeaders, output = self.__requestRaw(cnx, verb, url, requestHeaders, encoded_input)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/github/Requester.py", line 760, in __requestRaw
    response = cnx.getresponse()
               ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/github/Requester.py", line 174, in getresponse
    r = verb(
        ^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 935, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/github/GithubRetry.py", line 179, in increment
    raise Requester.createException(response.status, response.headers, content)  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
github.GithubException.GithubException: 403 {"message": "Resource not accessible by integration", "documentation_url": "https://docs.github.com/rest/issues/comments#create-an-issue-comment"}
0s

I'm not able to repro on a repository which already has the test, so I think the issue is that you're running it from a branch which doesn't have the required permissions.
My config looks like this btw. I don't use the cmake option, I generate the databse myself.
  clang-tidy:
    needs: build
    runs-on: ubuntu-latest
    if: startsWith(github.event_name, 'pull_request')
    steps:
      - uses: actions/checkout@v3
        with:
          ref: ${{ github.event.pull_request.head.ref }}
          repository: ${{ github.event.pull_request.head.repo.full_name }}
          submodules: recursive
      - name: Download generated compile database
        uses: actions/download-artifact@v3
        with:
          name: openblack-compile-database
          path: cmake-build-presets/ninja-multi-vcpkg
      - name: Add base repo to git config
        run: git remote add upstream ${{ github.event.pull_request.base.repo.html_url }}
        if: startsWith(github.event_name, 'pull_request')
      - uses<EMAIL_ADDRESS>        id: review
        with:
          build_dir: cmake-build-presets/ninja-multi-vcpkg
          exclude: "*ShaderIncluder.h,*json.hpp,*imgui_impl_sdl*,*stb_image_write.h,*dr_*.h"
          config_file: ""       # Don't use a single config file
          clang_tidy_checks: "" # Use checks from .clang-tidy tree
          lgtm_comment_body: "" # Don't post a lgtm message
          annotations: false
      - name: If there are any comments, fail the check
        if: steps.review.outputs.total_comments > 0
        run: exit 1

@bwrsandman ok, identified the setting that triggers this:
lgtm_comment_body: ""

and the build succeeds.

That just means it won't try to post anything if all is well. You didn't fix the underlying issue.
What's happening is that your PR comes from your own repo and not the target repo. So you don't have permission to post as the GitHub action bot on the PR.

assumed as much.
but I don´t get it why permissions: issues:write does not fix this? are there some variables missing in the post?

This is my best guess
From a security perspective, I don't want any random contributors to be able to use my repo's token. Even if they add a permission.
Since you're doing this PR from a fork, you won't have the permissions from that PTlR.

@bwrsandman I´ve merged the workflow now, but still there are issues with the run.
did I miss something or why does it still fail?

@ZedThree do you have any idea why he's getting 403 on his project when it comes to posting?

If I compare a job which works with yours, the permissions differ:
Works
GITHUB_TOKEN Permissions
  Actions: write
  Checks: write
  Contents: write
  Deployments: write
  Discussions: write
  Issues: write
  Metadata: read
  Packages: write
  Pages: write
  PullRequests: write
  RepositoryProjects: write
  SecurityEvents: write
  Statuses: write

Yours only has read so it would make sense that it can't post anything to the PR.
GITHUB_TOKEN Permissions
  Contents: read
  Metadata: read

Blindly guessing here but you probably need write permissions for PullRequests.
And remember that since it's a pull_request_target, your change needs to be in the target branch to run.

@bwrsandman thank you, found it.
I´ve had copied an example but if I leave out the permissions section altogether it works fine.


Add endian keyword argument to legacy API

[x] Closes Issue #15
[x] Tests added
[x] Changes are documented in CHANGELOG.rst

This is ready for review/merge. I'll have this sit for another few days before merging.

Rebased on latest master

@glenfletcher Better late then never. This PR adds the "endian" kwarg to the legacy api.


Missing accept-ranges from mocked S3 service API response

moto version 4.1.5 - botocore 1.29.96 - boto3 1.26.87 - s3resumable 0.0.3
Am using S3Resumable to handle my file downloads even when interrupted.
Long story short, I am able to mock s3 but it seems the output it generates when getting an object from the mocked s3 service is missing the accept-range http header. Is this something that could be easily added?
What I would have expected:
{
	'ResponseMetadata': {
		'RequestId': 'WDY36BWApXxh8Gj4PZRlE59jweQn4DQBshR34TsR2qW3zo94GyO0',
		'HTTPStatusCode': 200,
		'HTTPHeaders': {
			'content-type': 'binary/octet-stream',
			'content-md5': 'K9/JANWZCbhvUWgPjpJjBg==',
			'etag': '"2bdfc900d59909b86f51680f8e926306"',
			'last-modified': 'Thu, 23 Mar 2023 15:42:31 GMT',
			'content-length': '520',
                        'accept-ranges': 'bytes',
			'x-amzn-requestid': 'WDY36BWApXxh8Gj4PZRlE59jweQn4DQBshR34TsR2qW3zo94GyO0'
		},
		'RetryAttempts': 0
	},
        'AcceptRanges': 'bytes',
	'LastModified': datetime.datetime(2023, 3, 23, 15, 42, 31, tzinfo = tzutc()),
	'ContentLength': 520,
	'ETag': '"2bdfc900d59909b86f51680f8e926306"',
	'ContentType': 'binary/octet-stream',
	'Metadata': {}
}

What I got back:
{
	'ResponseMetadata': {
		'RequestId': 'WDY36BWApXxh8Gj4PZRlE59jweQn4DQBshR34TsR2qW3zo94GyO0',
		'HTTPStatusCode': 200,
		'HTTPHeaders': {
			'content-type': 'binary/octet-stream',
			'content-md5': 'K9/JANWZCbhvUWgPjpJjBg==',
			'etag': '"2bdfc900d59909b86f51680f8e926306"',
			'last-modified': 'Thu, 23 Mar 2023 15:42:31 GMT',
			'content-length': '520',
			'x-amzn-requestid': 'WDY36BWApXxh8Gj4PZRlE59jweQn4DQBshR34TsR2qW3zo94GyO0'
		},
		'RetryAttempts': 0
	},
	'LastModified': datetime.datetime(2023, 3, 23, 15, 42, 31, tzinfo = tzutc()),
	'ContentLength': 520,
	'ETag': '"2bdfc900d59909b86f51680f8e926306"',
	'ContentType': 'binary/octet-stream',
	'Metadata': {}
}

If you need more information or anything just let me know.

Welcome to Moto @lweijl! Marking it as an enhancement to add this header.

Thank you @bblommers, it's nice to see how fast this package is enhanced and being released!
Is this possible for the head_object as well? (for same reasons) Or should that request be in a different 'issue'?

No problem @lweijl, I'll just reopen this one to make sure we remember :+1:

Thanks for you effort @bblommers! it is greatly appreciated.
when looking at this it seems that on the head_object call the accept-ranges header is coming back without the -:
{
	'ResponseMetadata': {
		'RequestId': 'zwKVtIDJjph3DUIw160sMOtI80UDqLFfq3f23qbFOZ2UsfUl0ftR',
		'HTTPStatusCode': 200,
		'HTTPHeaders': {
			'content-type': 'binary/octet-stream',
			'content-md5': 'K9/JANWZCbhvUWgPjpJjBg==',
			'etag': '"2bdfc900d59909b86f51680f8e926306"',
			'last-modified': 'Fri, 12 May 2023 13:08:41 GMT',
			'content-length': '520',
			'acceptranges': 'bytes', --> this one
			'x-amzn-requestid': 'zwKVtIDJjph3DUIw160sMOtI80UDqLFfq3f23qbFOZ2UsfUl0ftR'
		},
		'RetryAttempts': 0
	},
	'LastModified': datetime.datetime(2023, 5, 12, 13, 8, 41, tzinfo = tzutc()),
	'ContentLength': 520,
	'ETag': '"2bdfc900d59909b86f51680f8e926306"',
	'ContentType': 'binary/octet-stream',
	'Metadata': {}
}

is that intentional?

Eh.. no. That was not intentional @lweijl, sorry about that. A typo in the first case, and a careless copy/paste the second time.
PR #6326 adds the dashes for both objects.

No problem, such things happen @bblommers. Thanks again!


Release notes for 318

Hive
----

* Improve performance of S3 Object listing. (:issue: `1232`)

https://github.com/prestosql/presto/pull/1232

SPI
---

Add `Block#isLoaded` method. (:issue: `1216`)

https://github.com/prestosql/presto/pull/1216

* Add support for ``Asia/Qostanay`` time zone. (:issue:`1221`)

https://github.com/prestosql/presto/pull/1221

SPI Changes
-------------
* Introduce ``SecurityContext`` group of classes to store security related information that is passed to plugins. (:issue:`171`)

#171

SPI Changes
-------------
* Allow connector to override minimal schedule split batch size. (:issue:`1251`)

#1251

Kafka Connector Changes
--------------------------
 * Update Kafka client library to ``1.1.1``. (:issue:`1112`)

#1112

General
-------

* Add session properties for `query_max_memory_per_node` and `query_max_total_memory_per_node`. (:issue:`1212`)

https://github.com/prestosql/presto/pull/1212

General
-------

* Fix ``DISTINCT FROM`` for ``UUID`` type. (:issue:`1180`)

#1180

General
-------

* Rename Kerberos properties to make them consistent with other HTTP server properties. (:issue:`1270`)

https://github.com/prestosql/presto/pull/1270

Jdbc-based connectors changes
---------------------------------
* Add credential provider for JDBC connectors. Thanks to that user can pass RDBMS credentials in separate file, by using ``credential-provider.type=FILE`` and ``connection-credential-file=credentials.properties``. (:issue:`1124`)

#1124

Jdbc-based connectors changes
-----------------------------
- Allow logging all calls to ``JdbcClient``. This can be enabled by turning on ``DEBUG`` logging for ``io.prestosql.plugin.jdbc.JdbcClient``. (:issue:`1274`)

#1274)

General Changes
-----------------
- Introduce Google Sheets Connector. (:issue:`1030`)

#1030

General
-----------------
- Add periodic cpu usage tracking in resource groups (:issue: `1128`)

#1128

Hive Changes
------------

* Fix query failure due to missing credentials while writing empty bucket files. (:issue: `1298`)

#1298 (fix is in PR #1303)

Hive Changes
---

* Improve performance when reading data from GCS. (:issue:`1200`)

#1200

General Changes
---

* Improve performance for queries over ``information_schema`` that have a limit. (:issue:`999`)

#999

General
---

* Add :func:`last_day_of_month` function. (:issue:`1295`)

#1295

Hive Changes
---

* Add support for reading data from S3 Requester Pays buckets. This can be enabled
  using the ``hive.s3.requester-pays.enabled`` configuration property. (:issue:`1241`)

#1241

General Changes
------------------
 - Remove ``distributed_planning_time_ms`` column from ``system.runtime.queries``. (:issue:`1084`)
 - Change the semantics of ``analysis_time_ms`` column from ``system.runtime.queries``. Previously it was showing the time spent on query planning, now it  shows time spent on semantic analysis. of query. (:issue:`1084`)
 -  Add ``planning_time_ms `` column to ``system.runtime.queries`` that shows time spent on query planning. (:issue:`1084`)

#1084

* Improve query performance when ``optimize_hash_generation`` is enabled. (:issue:`1071`)

#1071

I would recommend amending https://github.com/prestosql/presto/issues/1238#issuecomment-521727316 to the following to better explain the user impact.
- Change query CPU tracking for resource groups to update periodically while the query is running. Previously, CPU usage would only update at query completion. This improves resource management fairness when using CPU-limited resource groups. (:issue: `1128`)

General Changes
------------------
- Allow to display details of query that is in the queue or during semantic analysis. Query details are accessible from the web ui or ``system.runtime.queries``. (:issue: `1079`)
- Allow to cancel query that is in the queue or during semantic analysis. Query can be cancelled from the web ui or using builtin procedure ``system.runtime.kill_query``.(:issue: `1079`)

#1079

PostgreSQL Connector Changes
---
* Add support for PostgreSQL ``timestamp[]`` type. (:issue:`1023`)

https://github.com/prestosql/presto/pull/1023 (to be improved in https://github.com/prestosql/presto/pull/1262)

Hive Connector Changes
-----
 - Support ``INSERT INTO`` bucketed unpartitioned Hive table. (:issue:`1127`)

#1127

General
-------

* Fix ``DISTINCT FROM`` for ``UUID`` type. (:issue:`1291`)

#1291


Scrapers are hanging

For some reason scrapers are intermittently hanging. I noticed this yesterday when enough hung processes had piled up to exhaust memory, and this started showing up on the dashboard when subprocesses wouldn't fork anymore.
I spun up a fresh machine last night, and right now there are two lingering scraper processes. (see below) In the same time, six cron jobs ran successfully. strace says both processes are reading from a socket, so presumably we are doing some network access without a timeout.
    1  1127  1127  1127 ?           -1 Ss       0   0:00 /usr/sbin/cron -f
 1127 28348  1127  1127 ?           -1 S        0   0:00  \_ /usr/sbin/CRON -f
28348 28349 28349 28349 ?           -1 Ss    1000   0:00  |   \_ /bin/sh -c bash -c '...
28349 28350 28349 28349 ?           -1 S     1000   0:00  |       \_ bash -c cd ...
28350 28527 28349 28349 ?           -1 S     1000   5:23  |           \_ python igs
 1127 31346  1127  1127 ?           -1 S        0   0:00  \_ /usr/sbin/CRON -f
31346 31347 31347 31347 ?           -1 Ss    1000   0:00      \_ /bin/sh -c bash -c '...
31347 31348 31347 31347 ?           -1 S     1000   0:00          \_ bash -c cd ...
31348 31525 31347 31347 ?           -1 S     1000   5:24              \_ python igs

According to gdb, one process was reading from an SSL socket.
Here's what I got from /proc/<PID>/net/tcp:
  sl  local_address rem_address   st tx_queue rx_queue tr tm->when retrnsmt   uid  timeout inode
 269: F2391FAC:DBD0 2015E0A5:01BB 01 00000000:00000000 00:00000000 00000000  1000        0 1629135 1 ffff8800373f6900 708 4 26 2 -1

I killed one of the hung processes, and this was the tail end of the log.
https://www2.ed.gov/about/offices/list/oig/otheroigproducts.html
[other][2016-09-26][x21q0001]
        report: education/2016/x21q0001/report.pdf
        text: education/2016/x21q0001/report.txt
        data: education/2016/x21q0001/report.json
[other][2016-08-15][coveredsystems088152016]
        report: education/2016/coveredsystems088152016/report.pdf
        text: education/2016/coveredsystems088152016/report.txt
        data: education/2016/coveredsystems088152016/report.json
[other][2016-02-29][scrareport02292016]
        report: education/2016/scrareport02292016/report.pdf
        text: education/2016/scrareport02292016/report.txt
        data: education/2016/scrareport02292016/report.json
https://www2.ed.gov/about/offices/list/oig/specialreportstocongress.html
https://www2.ed.gov/about/offices/list/oig/testimon.html
https://www2.ed.gov/about/offices/list/oig/ireports.html
bash: line 1: 31525 Terminated              python igs


Integration Test - Release - 2022/01/14 08:25:30

Auto create for integration test. AnotherAuthorUpdatedArticleFlow

@prmerger-dev-2 : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Install the Windows Server Backup feature

If the Windows Server Backup feature is not installed on test server, the test cases about disk backup will be aborted due to the below error. The "New-WBPolicy" is available after installing the feature even without restarting the server.
Error:
New-WBPolicy : The term 'New-WBPolicy' is not recognized as the name of a cmdlet, function, script file, or operable program.

Test result:
...
05/22/2019 07:13:27 : [INFO ] Installing the Windows-Server-Backup ...
05/22/2019 07:13:27 : [INFO ] Install the Windows-Server-Backup successfully
05/22/2019 07:13:27 : [INFO ] Removing old backups
...
Test Run On           : 05/22/2019 07:08:03
VHD Under Test        : D:\xhx\vhds\Ubuntu_18.04.1.vhdx
Total Test Cases      : 1 (1 Passed, 0 Failed, 0 Aborted, 0 Skipped)
Total Time (dd:hh:mm) : 0:0:11
   ID TestArea             TestCaseName                                                     TestResult TestDuration(in minutes)
-------------------------------------------------------------------------------------------------------------------
    1 BACKUP               VSS-BACKUPRESTORE-MULTIFS-VHDX               PASS                 7.57


add link to gitter on mochajs.org

Are you doing this yourself, or you want someone else to handle the PR?

@dasilvacontin go for it

Solved via ec7b2fe1b2107960bd3982b7f82c281c0141b544


Remote debugging docs tasks for Fx68

See https://trello.com/c/ZLneh5mV/10-new-remote-debugging-aboutdebugging-fx-68 for original trello card
Acceptance criteria

[ ] [remote-dbg-next] about:debugging lists search webextensions by default — documented appropriately.

Closing this as a duplicate of Sprints #1392


Umbrella issus for basic function

Description
This is an umbrella issue to track some basic functions.
CI/CD

[x] Run Cargo clippy -D warning and Cargo deny in Github action  #22 @dierbei
[ ] Prepare for the physical machine: Postpone
[x] #4 @flyflypeng

Functions

[x] Privileged container in vmm-sandboxer : Supported
[x] Resource usage stats for both container and sandbox : Supported
[x] Time synchronization between Host and Guest VM @Burning1020 #50
[x] Limit VM process into cgroup filesystem @flyflypeng #51
[x] Sandbox files persist and recover @Vanient #40

Close it as it has been done.


ISSUE-960:

Describe your changes :
Fixed Integration test for Hive and updated documented with list of pip packages. This PR closes #960

Type of change :

[x] Bug fix
[x] Documentation

Checklist:

[x] I have read the CONTRIBUTING document.
[x] I have performed a self-review of my own.
[x] I have tagged my reviewers below.
[x] I have commented on my code, particularly in hard-to-understand areas.
[x] My changes generate no new warnings.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] All new and existing tests passed.

Reviewers
Ingestion: @harshach @ayush-shah @pmbrull

Hi @vijaypm, thanks for tackling this.
Regarding the is_responsive function, it is interesting to first check that the endpoint works before trying to run the tests. However, instead of trying to connect through a socket, it might be cleaner to use the healthcheck endpoint that comes with the API at http://localhost:8585/api/v1/health-check.
Wdyt? This way you can directly use python requests and check for a 200.

@pmbrull good point. I think we need two separate mechanisms - using 'requests' to test whether the metadata service http port is responsive or not, and 'socket' to test whether the hive2 port is open or not.
I'll revert the 'is_responsive' method to use requests to test for http responsiveness, and I'll add a new method 'is_port_open' to use socket to test whether a port is open or not.

Btw, I looked into doing a GET on the healthcheck api instead of just the service endpoint. The challenge with using the healthcheck is that the healthcheck fails even if ingestion is working perfectly fine. Since these integration tests are exercising the ingestion api, I assume they are sufficient for now.

Btw, I looked into doing a GET on the healthcheck api instead of just the service endpoint. The challenge with using the healthcheck is that the healthcheck fails even if ingestion is working perfectly fine. Since these integration tests are exercising the ingestion api, I assume they are sufficient for now.

Yep thats a good point. Yesterday I noticed some issues with the healthcheck although the REST was working correctly. Maybe @harshach can give us a hint here on might be wrong. If needed I can open an issue, but would be interesting to have a small troubleshooting guide on what to check if the API reports to be unhealthy.
Thanks!

Thanks @pmbrull . @vijaypm can you please use the health check api. We can merge it in after

Thanks for handling this @vijaypm . Looks Great!


ddev get requires config.yaml, config.yaml precludes addon

Is there an existing issue for this?

[X] I have searched the existing issues

Run a Diagnostic and Paste Link Here
% ddev --version
ddev version v1.19.5

Current Behavior
I want to use ddev get to create a new project, but I can't use an addon to manage config.yaml.
% mkdir foo && cd foo
% ddev get /Path/to/my/project
Unable to get project(s) [/Path/to/my/project]: could not find a project in /Users/paul/test/foo.  Please specify a project name or change directories: no .ddev/config.yaml file was found in this directory or any parent
% ddev config --project-name=foo
Creating a new ddev project config in the current directory (/Users/paul/test/foo)
Once completed, your configuration will be written to /Users/paul/test/foo/.ddev/config.yaml

Found a php codebase at /Users/paul/test/foo
Configuration complete. You may now run 'ddev start'.
% ddev get /Path/to/my/project
NOT overwriting file/directory /Users/paul/test/foo/.ddev/config.yaml. The #ddev-generated signature was not found in the file, so it will not be overwritten. You can just remove the file and use ddev get again if you want it to be replaced: signature was not found in file /Users/paul/test/foo/.ddev/config.yaml
[...]

ddev wants there to be a config file, but unless I add #ddev-generated manually to it, I can't replace it with the one from the addon.
This removes much of the convenience of having an addon that manages config.
Am I just mistaken about the use case for addons?
Expected Behavior
% mkdir foo && cd foo
% ddev get my/addon
% ddev restart
# SUCCESS!

Steps To Reproduce
Repro in 'Current Behavior'.
Anything else?
No response

Sorry, I should have said /Path/to/my/addon. I want to use an addon as a starting point.
I want to be able to tell people: "Type ddev get my/addon and you'll have the config you need."
Trying now, and ddev config --auto has the same issue: It adds a config.yaml file which then can't be managed through ddev get because it lacks the #ddev-generated tag.
If I'm not allowed to change config.yaml using the addon, that's fine, it's just not documented anywhere and this becomes a documentation issue.
Thanks!

You can certainly change config.yaml with your add-on. You can use ddev config --<whatever> in your pre-install scripts...

install.yaml is trying to add my config.yaml file from the addon, but it can't because it's disallowed from managing it.
Ideally I'd be able to have a starting point that's just a file instead of a script that modifies the file, sort of like you do with composer create-project. Then the user can modify it however they need. But if that's not possible or out of scope or whatever, then OK.
So now it's two documentation issues:

Update addon docs to point out that you shouldn't copy config.yaml using addons, or maybe even have ddev complain when you try.

Document the fact that you can say ddev config --whatever, because that's not documented anywhere.

ddev config --help will show you the many options.
If you'd like to collaborate and talk more about what you're up to, you're welcome in discord, https://discord.gg/hCZFfAMc5k
You can do some very impressive things in install.yaml, but I don't understand why you couldn't add #ddev-generated to the config.yaml. I don't understand why you can't completely replace the config.yaml if you add that in the pre-install script.

After conversation with @rfay:

Don't muck with config.yaml.
Add config.addon-name.yaml to override as needed.
Otherwise use install.yaml events/hooks to modify config.
All of this will work much better after https://github.com/drud/ddev/pull/3983
addons are KEWL.

I do think the day could come that ddev start --auto might be allowed - it could check to see if an accidental new project were being created. And that would mean that ddev get --auto might work as well, but as we've discussed here, it probably isn't necessary.


Add android client

Codecov Report

Merging #2 (87ea16a) into master (d939134) will increase coverage by 1.98%.
The diff coverage is 95.53%.

@@             Coverage Diff              @@
##             master       #2      +/-   ##
============================================
+ Coverage     92.55%   94.53%   +1.98%
- Complexity       59       67       +8
============================================
  Files             6        6
  Lines           188      238      +50
  Branches         29       28       -1
============================================
+ Hits            174      225      +51
- Misses            3        4       +1
+ Partials         11        9       -2

Flag
Coverage Δ
Complexity Δ

kotlin
94.53% <95.53%> (+1.98%)
0.00 <36.00> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ
Complexity Δ

.../main/kotlin/eventually/core/model/TaskSchedule.kt
88.05% <88.63%> (+3.15%)
24.00 <16.00> (+3.00)

.../kotlin/eventually/core/scheduling/SchedulerOps.kt
97.22% <97.22%> (ø)
15.00 <15.00> (?)

core/src/main/kotlin/eventually/core/model/Task.kt
90.00% <100.00%> (-1.31%)
6.00 <3.00> (-1.00)

...c/main/kotlin/eventually/core/model/TaskSummary.kt
100.00% <100.00%> (ø)
9.00 <2.00> (-2.00)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d939134...87ea16a. Read the comment docs.


Ores pure are not displaying Silver

I don't know how to do a pull request or I would, but line 49 of the ores display LUA needs to be changed from
{"Acanthite", {name = "Silver Pure", label = "Silver"}},
to
{"Acanthite", {name = "Pure Silver", label = "Silver"}},
Nice script!  Thanks

Thanks for submitting a ticket. I don't know when they changed it, but names are only checked on container scan so I wouldn't have noticed this until I had to build a new container databank.
This also affects all the other pures that were named with "Pure" second: Nickel, Silver, Fluorine, Scandium, Niobium, Vanadium
The fix is checked in. If any of the above don't show in your display after updating re-scan the container to pick up the new name in the databank.


Remove Suggest service and _suggest endpoint

The _suggest endpoint has been deprecated since 5.0 and the REST API spec for the Suggest service has been removed.
We should remove the Suggest endpoint/service as well and ask people to use Search instead.

Removed with the first commit in preparing elastic.v6.


sudden high battery drain.

The household has 2 iphones:

6s running iOS 14.4   battery health: 50%
6 running iOS 12.5.1  battery health: 90%

On Mon 15th Feb the 6s went flat overnight for the first time I can remember and then again on the following night.
Checking battery usage graphs on both phones showed rapid steady battery decrease overnight and the CPU running at 100% in screen-off mode.  With the phones' limited logging it is hard to say for certain, but the 10-day plot shows a definite change in background rates starting on Monday.  Overnight the battery usage by app lists Covidsafe as the only app running but attributes a "-" as the percentage which usually means less than 1%. My interpretation of this is that the remainder is in the kernel itself. When I disable location services for covidsafe then this disappears - when I re-enable them then the 100% cpu usage returns.

Please report this to the COVIDsafe team in the first instance here: https://github.com/AU-COVIDSafe/mobile-ios/issues including the version of COVIDsafe you are running (I believe they've just published another update - perhaps try that version?).
They will be able to diagnose specific issues with their app and can then come back to our team if the issue is with the Herald library. Recent Herald battery drain reports can be found on our website: https://vmware.github.io/herald/efficacy/results (Some tests are with location on, others with location off)
Thanks for the report.

Thanks, sorry for the mispost.


Documentation difficult to understand due to grammatical errors.

Hey,
Just trying out Kratos, when reading through the documentation, there are several grammatical errors within the English version of the doc.
Such as the "Overview" page, this page describes why Kratos is a tool that a developer should use. However, reading it involves some form of interpretation and may include some confusion especially to those who are learning about microservices with Golang.
I think if you could fix and add more updates to the English documentation, you could get more people to use Kratos.

Hello, we are preparing a better documents, but it may take some time. If you find any mistake in the docs, you could tell us in our discord channel, so that we can fix it as soon as possible.


Many to many participants

Is there a feature of many to many participants in PION? I mean a many to many call based on Star topology.
If so, is there any sample or just share the big idea!
thanks

check pion/ion from this repository is a SFU media server, also in ion/examples there is a basic SFU server


work locally with SERVER_PRESET=koyeb pnpm build, not work on koyeb

Environment

Reproduction

Describe the bug
✔ Server built in 25ms
[nitro] ✔ Generated public .output/public
[nitro] ℹ Building Nuxt Nitro server (preset: koyeb, compatibility date: 2024-04-03)
[nitro]  ERROR  Error: File<EMAIL_ADDRESS>does not exist.
undefined
ERROR  File<EMAIL_ADDRESS>does not exist.
at Job.emitDependency (/layers/heroku_nodejs-pnpm-install/virtual/store/@vercel+nft@0.27.6/node_modules/@vercel/nft/out/node-file-trace.js:310:23)
at async Job.maybeEmitDep (/layers/heroku_nodejs-pnpm-install/virtual/store/@vercel+nft@0.27.6/node_modules/@vercel/nft/out/node-file-trace.js:110:17)
at async Promise.all (index 1)
at async Job.emitDependency (/layers/heroku_nodejs-pnpm-install/virtual/store/@vercel+nft@0.27.6/node_modules/@vercel/nft/out/node-file-trace.js:321:9)
at async Promise.all (index 1)
at async nodeFileTrace (/layers/heroku_nodejs-pnpm-install/virtual/store/@vercel+nft@0.27.6/node_modules/@vercel/nft/out/node-file-trace.js:51:5)
at async Object.buildEnd (/layers/heroku_nodejs-pnpm-install/virtual/store/nitropack@2.10.4_typescript@5.6.3/node_modules/nitropack/dist/rollup/index.mjs:435:26)
at async Promise.all (index 0)
at async PluginDriver.hookParallel (/layers/heroku_nodejs-pnpm-install/virtual/store/rollup@4.27.3/node_modules/rollup/dist/es/shared/node-entry.js:20737:9)
at async /layers/heroku_nodejs-pnpm-install/virtual/store/rollup@4.27.3/node_modules/rollup/dist/es/shared/node-entry.js:21700:9
ERROR  File<EMAIL_ADDRESS>does not exist.
 ELIFECYCLE  Command failed with exit code 1.
[Error: heroku/nodejs-pnpm build script error]
There was an error while attempting to run a build script
from this project's package.json. The command exited with a non-zero exit code.
Details: Exit code exit status: 1
Timer: Builder ran for 40.939368851s and ended at 2024-11-19T21:46:03Z
ERROR: failed to build: exit status 1
Build failed ❌
Additional context
No response
Logs

Can you check if you can reproduce this with plain Nitro?

empty nuxt app work

always errors like
ould not resolve dependency:
npm ERR! pinia@"*" from the root project
npm ERR!
npm ERR!  After make my copy of it and update dependencies in it all work.


Iss68

Tests are implemented and functional, using jest (testing framework) and puppeteer + pixelmatch for visual regression testing. Testing is integrated with documentation generation.
A single .py script gets run by sphinx, and generates an html viewer, that is integrated as documentation in a sphinx gallery. The output of this viewer is captured by the test and compared to a previous, "blessed" version (a diff image is generated). The screenshot is integrated back as thumbnail in a sphinx gallery. Adding a test is easy: just add plot_<test_name>.py in the tests folder (that's the script that runs and generates the viewer), and plot_<test_name>_test.js as well (very short test configuration script, with the size of the webpage to be captured in the visual tests & the thumbnails).
This PR will not be merged before the documentation is also ready to go.

Tests are now functional, and deployed on circleci.  But this is not a huge deal, as it can be done again as needed in the future. Now the docs need to be written and deployed too.

closes #68 #82 #75 #77 #70 #64 #63 #59 #28


feat: use es2019 for lib and target

We are currently using node 12 in all of our node TypeScript projects,
and we are beginning to use babel to transpile front-end projects. So,
to take advantage of new features in TypeScript/JavaScript we need to
increase our tsconfig lib and target settings. Since node 12 supports
all of es2019, this seems a good version to target.
Also, as we upgrade to the current version of TypeScript (3.9.2) some of
the functionality that was allowed in previous versions of TypeScript
when targeting "es5" now require tslib. For example, array spread notation
now requires tslib to transpile to es5.

FYI @jthomerson @yokuze: This needs to be merged and deployed before we can proceed with the TypeScript upgrades.


[Feature Request]: Get KEY data (active or not) and use that info for tally

What feature would you like to see added to TallyArbiter?
Wondering if there is a way (with TSL) for TA to obtain data when keys go on and off. I would use that data to output a different color/image to my clients to let the camera ops know that there is a video/graphic that is overlaying their camera feed on stream.
This might not be possible if TSL doesn't send this data and I have a feeling it doesn't... :( But there might be another option besides TSL

That's a feature we're working on currently. What kind of switcher are you using?

Ross Carbonite Black Solo


Continue Ruby Studies

These days I'm researching & studying some Ruby Advance stuff, hoping it'll help.
I already researched on module included(object)
@digitalextremist  Can you point out some stuff (just names) I should continue researching on?

Sure.
extend Forwardable, and def_delegators
class << self and def self.method_name
class methods and variables vs. instance variables and methods
I'll add more later. ECell & Culture use ZeroMQ heavily also, in addition to Celluloid and pure Ruby Standard Library.

One question,
I don't understand this (what you did in Celluloid::Actor::Manager)
class << self

  INJECTION_POINTS = [
    :actor_created,
    :actor_removed,
    :initializer
  ]
  @@injections = {}
  # Reminder: Do not mutate these values once set!

  INJECTION_POINTS.each { |ip|
    @@injections[ip] = []
    define_method(ip) { @@injections[ip] }
    define_method(:"#{ip}!") { |*args|
    #de puts "adding injection #{ip}: #{args}";
    @@injections[ip] << args
   }
  }
end

def initialize
  self.class.initializer.map { |args|
    async.send(args.first, *args.slice(1,-1))
  }
end

def actor_created(actor)
  monitor(actor) #de Will link, and cause :actor_removed when actor is terminated.
  self.class.actor_created.map { |args|
    async.send(args.first, actor)
  }
end

Could you please tell me what is this technic name? then I can search for it.

Finished reading Ruby Standard Library Documentation. For ECell, Culture & ZeroMQ what are the areas I should refer? And for Ruby any other resources?

Now, you ought to get more familiar with Celluloid itself and Reel ... even start to fix issues in their trackers.


Arrange code so we do not have temporarily undefined classes

We have been hitting a core dump as uWSGI expires workers, where the stack trace hints at a "double free" type of error. This only happens with the default preforking behavior of uWSGI, and in some other environmental conditions that remain unclear to me.
This is one of a few changes I've identified that, combined, resolve this. Another one is https://github.com/jazzband/django-oauth-toolkit/pull/1357
This also looks like better practice to me. While I realize there are some times we have to use a variable that won't be defined until later in the file, I cannot find a reason that needs to be the case here.

I'm not sure how this is meant to help at all.
HStoreField.get_transform cannot be called until the module is fully loaded anyway so neither KeyTransformFactory or KeyTransform are temporary by any means. I don't see how organizing it this way will help unless there's a pretty critical bug in uWSGI in how it loads modules. In other words, KeyTransformFactory is resolved at run time and is not bound at module definition time so the ordering should not matter.
The source of Django contains tens (if not hundreds) of instances of objects being defined this way and Python doesn't offer any way to test against this pattern so even if we wanted to merge this change we can't have a guarantee that we wouldn't regress somehow. There are cases where it just impossible to even attempt such as when dealing with cycles of references between objects.


Solid Gauge chart losing series rendering on page/container resize [ng-highcharts + angular]

I have customized a solid gauge chart and it works fine when run on a simple html container
http://jsfiddle.net/roydondsouza/m4hzg181/
But after integrating it with angular and ng-highcharts the series data loses its rendering.
http://jsfiddle.net/roydondsouza/6hxaB/139/
Expected:

Result after resize: (Renders, but loses the rendering)

Details here:
http://stackoverflow.com/questions/27311035/solid-gauge-chart-losing-series-rendering-on-page-container-resize-ng-highchart

I'm also having this problem. @roydondsouza did you find a way around this?

https://github.com/highslide-software/highcharts.com/issues/3708 Possible fix here.

In fact, I can confirm that this fix works.

@seiyria Thanks a lot for the link. Let me check and confirm this.

@seiyria  Thanks a lot for the link. Let me check and confirm this.

https://github.com/highslide-software/highcharts.com/issues/3708 fixes this issue. Thanks @seiyria

Thanks @seiyria for looking into this and @roydondsouza for raising it!

:+1:

:) (Y)


Asset Tag link to exiternal system

Issue type
[X] Feature request
[ ] Bug report
[ ] Documentation

Environment

Python version:  3.5.3
NetBox version:  2.2.9

Description
Ability to create a link from the asset tag field of a device (or anything with an asset tag) to an external system (e.g. inventory tracking). This could be customised by having a configuration setting with a prefix and suffix that are prepended and appended to the asset tag as relevant.
For example, if ASSET_LINK_PREFIX was set to http://example.com/inventory/ and ASSET_LINK_SUFFIX was set to /view/, the asset tag 123 would generate the URL http://example.com/inventory/123/view/

I understand Netbox isn't trying to be a universal inventory tracking system, hence why having the link is probably more flexible.
On a similar note, this could also be used for a ticketing system or similar. As such, would it be more flexible to have an "external links" section with multiple links?

Example UI:

Sounds like https://github.com/digitalocean/netbox/issues/969 would solve this.

Yup, must have missed that one.


Feature/control panel phase2

This PR adds remaining Control Panel features.

Looks great overall, although (as discussed) we have to stop doing this to each other:

:wink:

Looks great overall, although (as discussed) we have to stop doing this to each other:

😉

Agreed. But 3000+ of that is regenerated protobuf files. Another 500 will be UI files :)


Encrypted comparison

Other operations, such as encrypted comparison, sorting, or regular expressions, are in most cases not feasible to evaluate on encrypted data using this technology.

Sorry, I'm new to this. Is the above implying that encrypted comparison is not possible with SEAL? As in, do I absolutely have to look elsewhere for this type of functionality?
Given a simple example, like sending two encrypted numbers to a server and subtracting them, would I somehow be able to know if the encrypted result was equal to 0 without decrypting it on the server? One way that I could think of doing this is by sending an encrypted 0 from the client, and then doing the comparison on the server. Am I thinking about this correctly?

There is a huge difference between doing equality check (is encrypted number equal to zero) and comparison (is the encrypted number positive). How these might be done depends strongly on the encryption scheme you use (SEAL implements BFV and CKKS).
BFV allows encrypting modular integers where comparison is very challenging to do but equality test is reasonable. For equality, you can subtract two encrypted integers (or vectors of them); the result is zero mod plain_modulus if they are equal and otherwise non-zero. Now, if plain_modulus is prime, using Fermat's little theorem you can exponentiate to power plain_modulus-1. The result will be an encrypted 0 precisely when the two inputs matched, and 1 otherwise. Then just compute 1-Enc(result) to flip the bit. This can make sense if plain_modulus is reasonably small, otherwise you'll end up needing very large parameters to support the exponentiation. Note that the server does not learn the unencrypted result, but only an encrypted result, which the data owner has to decrypt. In some protocols it suffices to not compute the exponentiation, and instead return the difference of the two values (possibly randomized by a random multiplicand modulo plain_modulus). In this case, however, your signal for equality is zero and non-equality a random integer modulo plain_modulus, which may be hard to use for any further encrypted computations.
CKKS allows encrypting fixed-precision real numbers where the comparison function can be approximated (see e.g. https://eprint.iacr.org/2019/1234). This is probably not very practical if you need high-precision comparison near zero.

So will the server alone never be able to branch based off of an encrypted equality check or comparison? Will the server always have to return the encrypted result to the client, where the client can decrypt the result and than make a branching decision?

Right. It's impossible to truly branch on encrypted data; otherwise the branch would leak information about the encrypted data, which is not supposed to be the case. What is possible though, is evaluating both branches and filtering out the desired one. This is used for example in https://arxiv.org/abs/1909.08362.

Ah, thanks for the insights!

Hello    @kimlaine @lastmjs , I need help to understand the equality check technique between two encrypted integer. I want to quote your statement :

BFV allows encrypting modular integers where comparison is very challenging to do but equality test is reasonable. For equality, you can subtract two encrypted integers (or vectors of them); the result is zero mod 'plain_modulus' if they are equal and otherwise non-zero.

I have two encrypted integer
a = encryptedzero
b = encryptedone
How can I get it is equal or not ? is it possible to get True of false ? If you give an small example for equality check will be highly appreciated.

It does not yield true or false. The equality check result is in encrypted form. One equality check algorithm, subtraction, gives encrypted zero (equal) or encrypted non-zero (unequal). The other algorithm, Fermat's little theorem, gives encrypted zero (equal) or encrypted one (unequal).
Subtraction algorithm:
Ciphertext encrypted_a, encrypted_b, encrypted_equality;
Evaluator::sub(encrypted_a, encrypted_b, encrypted_equality);
Plaintext equality;
Decryptor::decrypt(encrypted_equality, equality);

Now equality is 0 if a and b are equal and is non-zero otherwise.
Fermat's little theorem (following the above code block):
Evaluator::exponentiate_inplace(encrypted_equality, plaintext_modulus - 1, relin_keys);
Plaintext new_equality;
Decryptor::decrypt(encrypted_equality, new_equality);

Now new_equality is 0 if a and b are equal and is 1 otherwise.

@WeiDaiWD  Thank you for your great example and I am clear. I am working on genome data. Actually based on this equality result I will perform another computation. So I just want to know if I decrypt the new_equality variable to get the equality check result on the server-side. is it mean that I provide my secret key to the cloud side? Because I want to keep encrypted the individual's genome data on the cloud during the analysis. After analysis individuals will receive the encrypted result and able to see the result using the secret key that I can make sure a secure analysis of sensitive data. Hope your feedback will be a solution to my misunderstanding or guidelines to go ahead.

If you share the secret key with the server, there is no point to encryption anything at the beginning. And we don't recommend to have the client decrypt that and send it back to server. What you want to do is computing new_equality * X + (plain_one - new_equality) * Y which is an encrypted method to compute new_equality ? X : Y.

@WeiDaiWD  Thank you very much for your support and guidelines

Hi @WeiDaiWD @kimlaine,
Is it possible to apply the Fermat's little theorem to a vector in order to batch encrypted comparison?
Following the example, I changed encrypted_equality to a vector of ints but the process starts to consume all available memory and dies when executing exponentiate_inplace. It works fine for a scalar.

exponentiate_inplace is extremely memory inefficient (we should have explained this in comment). It creates a vector of exponent number of same ciphertexts and calls multiply_many. If you look at the implementation of multiply_many, you can see that we are simply trying to perform a series of ciphertext multiplication with the fewest depth. Potentially, it can avoid copying the ciphertext exponent times, which is what you need to get rid of the massive memory consumption. Well, still, avoid choosing a large plain_modulus.

Thanks a lot for a quick response!
In that case, do you have any tips how to vectorize equality checks? One-by-one does the job but it's gonna be extremely inefficient for larger datasets

Vectorization is done at the ciphertext layer. Each Ciphertext encrypts a vector of values. exponentiate_inplace raise all of the underlying values of one ciphertext to the given exponent. For example, if you have Ciphertext ct_x, ct_y that encrypt (x0, x1, x2, ..., x1023) and (y0, y1, y2, ..., y1023), then running evaluator::sub_inplace(ct_x, ct_y) will give you (x0-y0, x1-y1, x2-y2, ..., x1023-y1023) in ct_x. Now you need to compute evaluator::exponentiate_inplace(ct_x, plain_modulus-1) for the Fermat's little theorem. Finally the result will be a vector of powers encrypted in the single ciphertext ct_x. The catch here is that you need to implement a different evaluator::exponentiate_inplace that does not require a massive amount of memory.

Thanks for a thorough response. I've made it work all the way until the evaluator::exponentiate_inplace step and then it obviously fails. I'm new to SEAL so it'd be great to shirk from re-implementing the library. Do you happen to know any other idea for equality check that can be batched using the current implementation?

What I meant by "implement a different evaluator::exponentiate_inplace" is not re-implementing the library. They are multiple algorithms to compute exponentiation, for example, sequentially multiplying the value itself, square-and-multiply, etc. You just need to pick one that gives a low multiplicative depth, and then use Evaluator::square(...) or Evaluator::multiply(...) to implement the algorithm. I don't know another way to check equality beside the Fermat's little therom.

Hi guys, I am having some difficulties understanding this encrypted comparison still even reading those papers. I will appreciate it if you could help to understand it based on one example.

Encrypted all data, For say, I have 5 numbers {5,10,15,20,25}
Encrypted the range value, ENC_range(10)
Upload encrypted data and range value to the server
Server computes it and finds the range >=10 in this case
Return the encrypted outputs whose range >=10 only to the client

For step 4 and 5, I am confused about how does server perform the comparison over encrypted data and also return the >=10 value as output only (To me, I found Vectorize the data and then apply a broadcast operation on the range value to perform step 3). For step 4, I want the server just return to me the larger value whose range >=10 only; the client just wants a larger value, not the smaller one, I do not find anything for that; what I understand the client might need to do some filtering here to get it but Is there a way around client does not have to do it. How do I perform a comparison on all data here using SEAL or FHE or HE, is there any scheme that can help to achieve this goal? Is there any code example or implementation available to look into to understand?
Thank you.


grib save level accuracy

We can't currently save level=1.5m to grib, as the "scaling" is set to 0, allowing integers only. Change the scaling to allow 1 or 2 decimal places.
Thanks for identifying this, @chrisshort1981

Ported to iris-grib.


Add .csx as an option in the file types for opening Advanced Scripts

When you click on the open button in the Advanced Script editor the file type defaults to *.cs but in the new script repository (https://github.com/TabularEditor/Scripts) the suggested extension is *.csx

You can work around this by switching to . but it would be nice to maybe even default to .cs and csx files.  I'm happy to have a go at this and submit a pull request if you'd like, it should not be too hard to configure.

Good catch - I've got a new release coming up shortly and this is a very minor fix, so don't worry about submitting a PR (unless you really want to 😄)

Nah, that's OK. You can probably fix the issue before I could get my fork updated and find where to make the edit :)

Fixed in 2.9.8.


New category: presentations / talks / tutorials

Feature description
When I was working on the timeline I realised it would be good to have some kind of category listing out all our previous conference presentations / talks / tutorials. This will include package-specific talks (e.g. rtables trainings) as well as overall talks like PHUSE etc. It would be great to also link the materials / videos if possible.
This can be done in form of a timeline or sections by year.
I do not recommend to introduce the "upcoming events" section as this will be quite hard to maintain (i.e. requires constant updates). We can assume that it would be a retrospective tracker updated when an even already took place and there are materials available to be linked.
Code of Conduct

[X] I agree to follow this project's Code of Conduct.

Contribution Guidelines

[X] I agree to follow this project's Contribution Guidelines.

Security Policy

[X] I agree to follow this project's Security Policy.

I have created a conference tracker gdoc (shared in our internal chat room) and we can reuse it here


Fix #223, where session is loaded into Lazy window

Based on @zamotany's code here:
https://github.com/rmagatti/auto-session/issues/223#issuecomment-1666658887
If Lazy is present, we wait to automatically restore the session until we get the VeryLazy event and there's no Lazy window still up.
If Lazy isn't present, loading works as before.

I looked a little more closely at the Lazy events. I'm still testing locally, but I think we could probably use LazyDone instead of VeryLazy. It won't matter when there's an update popup (since dismissing that will take longer than either event) but during normal usage when there is no popup, I think LazyDone would be a bit faster to restore the session.

I realized we could use VimEnter so changed the code to that. Also added an option to disable the Lazy delay