Convert PHP code base to use service container.

Context

Summary
Copy the infrastructure for service containers from amp plugin. Copy infrastructure and exception namespaces from amp plugin. Spin up classes in as services.
Relevant Technical Choices
Much of this PR is just changing the name of the init method to register and adding implements to all classes.
The directories of Infrastructure and Exception are copied from the amp plugin. I had a couple of tweaks, including.

Adding get_registration_action_priority to Delayed interface to pick when a action is registered.
Lots of PHPStan fixes to phpdoc blocks.
Change namespace and added since phpdoc.
Add REST_API_Factory and Register_Widget classes, to move logic out of plugin class.
More methods in the plugin class ( to include files ) moved to namespace.php
There an odd is_needed method in SVG class. Hopeful we remove this at some point.
Check is added to see if block is already registered, as in activate it is registered twice.
decoder is now injected in dashboard and story post type classes.
Improved Unit tests, so have helpers for adding / remove capabilities now that decoder is added to constructor of story post type.
Added some backwards compatibility with a magic getter. It is possible to get class properties like this, get_plugin_instance()->customizer->render_stories(). However it is no longer possible to do this,

remove_action(  get_plugin_instance()->customizer, 'init );

For two reasons,

The method init is renamed to register.
This method is no longer register directory to action, it uses a closure.

Plugin class still needs a little work.
To-do

Add more tests.
Update documentation.

User-facing changes

Testing Instructions

QA

[ ] This is a non-user-facing change and requires no QA

This PR can be tested by following these steps:

UAT

[ ] UAT should use the same steps as above.

This PR can be tested by following these steps:

Reviews
Does this PR have a security-related impact?

Does this PR change what data or activity we track or use?

Does this PR have a legal-related impact?

Checklist

[ ] This PR addresses an existing issue and I have linked this PR to it in ZenHub
[ ] I have tested this code to the best of my abilities
[ ] I have verified accessibility to the best of my abilities (docs)
[x] I have verified i18n and l10n (translation, right-to-left layout) to the best of my abilities
[x] This PR contains automated tests (unit, integration, and/or e2e) to verify the code works as intended (docs)
[x] I have added documentation where necessary
[x] I have added a matching Type: XYZ label to the PR

Fixes #

CC @schlessera and @westonruter
I have whole sale copied your implementation of service containers in your plugin to bring uniformity to the two plugins. The implementation is solid and generic. Baring a couple of tweaks and change of namespaces, went in super smooth.
Love your thoughts and feedback on this PR.

SVG upload E2E test is constantly failing. Does it work for you? Might be a bug.

@spacedmonkey I'll see if I can get a more detailed review in soon, but I already wanted to give a few pointers about the general approach.
Timing
I see you've added a required action of init and a priority of 10 to everything. Regarding the priority, 10 is the default, so this is redundant. Regarding the default action, it doesn't make sense to me to specify that everywhere via the Delayed interface, as this is basically a "double-hook" process. You're calling the register() method on the init action, only to then decide what hooks to actually register the service functionality on. So, ideally, you'd just register() immediately by default and have the register() method decide by its hooks what the timing is supposed to be.
So, instead of:
instantiate() => {
	'init' => register() {
		'some_action' => $this->callback()
	}
}

it would be:
instantiate() => register() {
	'some_action' => $this->callback()
}

Delayed is meant for scenarios where the instantiation of the service can only happen at a certain point in time.
If all of your services require the 'init' action to be able to even instantiate, then you change the registration action here: https://github.com/google/web-stories-wp/pull/6906/files#diff-afc0298087d72ee3c18032663a02b2c9f484a7bd1262fe8956c0300b8fb0305cR49
Service Base class
I would recommend against using a shared base class for services. If you need to do something that is true for all services, it can probably be re-thought and done in some other way, like with the above timing change. And if you still want to use a base class for some shared functionality, I'd recommend naming it for what it actually provides, instead of "replacing" the interface, which can cause confusion.
Make use of the Injector
The infrastructure is mostly built around the principle that the instantiation should happen in a central, controlled way, where the ideal is to have one instantiation call instantiate the entire object graph as needed.
This can be achieved by making use of dependency injection, which has a recursive effect.
For the rest of the code, the new keyword should serve as a code smell.
Whenever you find the need to use the new keyword, you should:

Check whether you can have that object just injected via the constructor;
Otherwise, get a hold of the Injector to instantiate it (while having all dependencies be dealt with automatically).

For 2., this would look like this:
$story_post_type = new Story_Post_Type( new Experiments(), new Meta_Boxes(), new Decoder() );

you should have something like this:
$story_post_type = $injector->make( Story_Post_Type::class );

Note that you don't need to know anything about the dependencies, they are controlled via the injector.
Getting a hold of the injector is done via:

constructor injection :)
$container->get( 'injector' )
Services::get_injector()

Hope that helps, and I'm happy to answer any questions.

Hmm, looking at it a bit more closely, it seems like you're not yet using the service-based architecture...?
The core of the infrastructure is the ServiceBasedPlugin abstract class, which would need to be implemented by something like a WebStoriesPlugin class, and that would be what you trigger in your bootstrap file. Is this still WIP and I'm just too early with the feedback?

You can see the core of the AMP plugin here, for example: https://github.com/ampproject/amp-wp/blob/develop/src/AmpWpPlugin.php

$story_post_type = new Story_Post_Type( new Experiments(), new Meta_Boxes(), new Decoder() );

There are only a handful of places where this is used now, I can refactor them to use the injector.

I see you've added a required action of init and a priority of 10 to everything.

This is not true, there are a number of classes, that have different actions and properties. It defaults to these in the service, but are overridden where required.

Delayed is meant for scenarios where the instantiation of the service can only happen at a certain point in time.
If all of your services require the 'init' action to be able to even instantiate, then you change the registration action here:

All of current classes are written with this delay in mind. So to keep to the current format and not to introduce to many changes, I kept it this way. I think should maybe look into the changing this later, as this is already a big refactor.

it doesn't make sense to me to specify that everywhere via the Delayed interface, as this is basically a "double-hook" process

Why? Can you explain this a little more detail. I don't see it doubling hooking. Is it double hooking here?
foreach ( $services as $id => $class ) {
			$id    = $this->maybe_resolve( $id );
			$class = $this->maybe_resolve( $class );
			// Allow the services to delay their registration.
			if ( is_a( $class, Delayed::class, true ) ) {
				$registration_action = $class::get_registration_action();
				if ( did_action( $registration_action ) ) {
					$this->register_service( $id, $class );

					continue;
				}

				\add_action(
					$class::get_registration_action(),
					function () use ( $id, $class ) {
						$this->register_service( $id, $class );
					},
					$class::get_registration_action_priority()
				);

				continue;
			}

			$this->register_service( $id, $class );
		}

You can see the core of the AMP plugin here, for example: https://github.com/ampproject/amp-wp/blob/develop/src/AmpWpPlugin.php

Plugin has been changed to extend - ServiceBasedPlugin. Without that the code wouldn't work at all.

SVG upload E2E test is constantly failing. Does it work for you? Might be a bug.

I am seeing the error locally, but I don't see how it would be related to my change...

Why? Can you explain this a little more detail. I don't see it doubling hooking.

Yes, sure. So, consider a Service with the register() method and a process() method that needs to run on the 'wp' action. The register() method of that service will contain the add_action() call to hook the method of the service to the WordPress lifecycle: add_action( 'wp', [ $this, 'process' ] );.
If you now use a Delayed interface to that service and tell it to use the init action as the registration hook, then you are actually doing an add_action() to even call the register() method in the first place.
So instead of this:

instantiate service
execute register()
add_action( 'wp', [ $this, 'process' ] );
forward to 'wp' hook
execute [ $this, 'process' ]

You will end up with this:

instantiate service
add_action( 'init', [ $this, 'register' ] );
forward to 'init' action
execute register()
add_action( 'wp', [ $this, 'process' ] );
forward to 'wp' hook
execute [ $this, 'process' ]

Note that you have two hooks (2. & 5.) just to execute [ $this, 'process' ] at the right moment.

You are talking about the hooking the register method on init and then within that method, hooking into add other methods?
If so, I know this is not the best, but this is how this currently works in our plugin. For example
https://github.com/google/web-stories-wp/blob/05ba6811cae5633354358551fb67e91838d837b9/includes/Plugin.php#L323
https://github.com/google/web-stories-wp/blob/05ba6811cae5633354358551fb67e91838d837b9/includes/Dashboard.php#L100-L106
I know this isn't great, but there are reasons we have done this in our code base.
I really don't want to tackle this problem in this PR. I will change this in a follow up PR.

Also, more generally, there's a bit Google copyright plastered across all of the infrastructure code and tests, but not a single attribution to the real source of this: https://github.com/mwpd/basic-scaffold
A short mention for source attribution somewhere (if only in the readme) would be appreciated.

@schlessera Please can you refrain from comments out of scope of this change. This is not a full audit of our code. I am just trying to shoehorn in service containers to the codebase. I am not currently second guessing how the classes are written, just trying to get them to work as is. Later we can / will refactor to improve, that as this PR is already large, I do not wish to make this part of this PR.
Any references to Database_Upgrader and Story_Post_Type have been avoid. There are tickets to refactor these classes https://github.com/google/web-stories-wp/issues/5062 https://github.com/google/web-stories-wp/issues/5668. We will tackle changes to how those classes are spun and called in those tickets.

Also, more generally, there's a big Google copyright plastered across all of the infrastructure code and tests, but not a single attribution to the real source of this: mwpd/basic-scaffold
A short mention for source attribution somewhere (if only in the readme) would be appreciated.

Absolutely 👍  While we don't have an attribution section in the top-level readme yet, we can  add appropriate notes in the main Service + Plugin classes as a starter.

@schlessera You are right, copyright was a bad call here. I have removed it and added some props. https://github.com/google/web-stories-wp/pull/6906/commits/8d15d187314888e11b1daf3a9b8979f8ae4687b8
If anyone has any ideas on better wording or format, it is now a single find and replace.

@schlessera You are right, copyright was a bad call here. I have removed it and added some props. 8d15d18
If anyone has any ideas on better wording or format, it is now a single find and replace.

We have some guidelines:
Files that copy lines of code from third-party files, or that are directly translated or derived from third-party files, must retain any copyright notices present in the third-party file(s). A Google copyright notice can be added to the license header as a separate copyright notice, beneath any existing notices, but above the license text.
Any files that are exact copies of files taken from a third-party source must retain any existing copyright notices and must not include any new copyright notices.

Should we be copying this then?

For new files that are not found in basic-scaffold we obviously don't need to add something custom and can use the regular copyright notice.
For files originating from basic-scaffold without modification we can use:
/**
 * File description.
 *
 * @package   Google\Web_Stories
 * @author    Alain Schlesser<EMAIL_ADDRESS> * @license   MIT
 * @link      https://www.mwpd.io/
 * @copyright 2019 Alain Schlesser
 */

For files originating from basic-scaffold but modified we can use:
/**
 * File description.
 *
 * @package   Google\Web_Stories
 * @copyright 2019 Alain Schlesser
 * @license   MIT
 * @link      https://www.mwpd.io/
 */

/**
 * Original code modified for this project.
 *
 * @copyright 2020 Google LLC
 * @license   https://www.apache.org/licenses/LICENSE-2.0 Apache License 2.0
 */

Updated comments. https://github.com/google/web-stories-wp/pull/6906/commits/14c2f49a64b74cc4d7ab59a9b3d98b09d74d0fdc


Tag "master" of quay.io/coreos/package-server does not exist

quay.io/coreos/package-server in the 0.7.0. manifest file uses master tag which does not exist, which makes the deploy fail.
https://github.com/operator-framework/operator-lifecycle-manager/blob/170be9c08533b32f14d49e9829055b40af15389e/deploy/upstream/manifests/0.7.0/22-packageserver.yaml#L108

I opened #472 to fix the 0.7.0 release manifests.
Just fyi, that packages server is somewhat of a "preview" feature for now (it doesn't yet support watch apis, and console hasn't been updated to use it). If it causes problems for you, you can safely delete the deployment for now.

@ecordell Thanks for the quick fix!


Provide documentation that is easy to browse online (from github)

Hi,
One of the things I find inconvenient about google perftools is the documentation; one of the main reason is that it's not really available online easily (there are old sites caching it but you never know if the github version is newer or not)... that seems a bit weird nowadays :)
Anyway I did a little experiment forking gperftools and then:

converting main README and other HTML files under docs/ folder in MarkDown syntax.
This is useful to have Github able to render the documentation in a nice, eye-candy format

linking doc pages to the main README. This makes it easy to browse the docs online.

provide a Makefile that allows for offline conversion from MarkDown to HTML. This is useful in case anybody forks the repo and wants to browse documentation locally with his browser.

Can you take a look at  https://github.com/f18m/gperftools  and see if the new documentation format looks nice to you?
To get it into the main repo it should be enough to merge this PR

Much thanks for this work. I personally dislike markdown a bit, but since you offer this work and it is improvement over what we have, I'll take it.
But lets make it work first. I've just accepted another documentation fix and your work now needs rebasing :(
Larger issue is making sure it works w.r.t. building source tarballs (i.e. make dist) and building from those tarballs. I don't see anything that tells automake to include your md files into source tarball. And it looks like it'll try to include resultant .html files causing confusion. We should figure out if we want to ship .html in the source tarballs and what we'll install if user system lacks pandoc. I think we should still install .md files regardless.
Please test and fix your work w.r.t. those requirements. I also see some unrelated changes (e.g. comments in configure.ac). Lets get those in separately.

Ah, also I do prefer clean history. Before we merge your work, lets rebase everything such that git history is nice.

Hi Alk,
Good to see that the value of online documentation is appreciated :)
I did rebase the branch so that it now merges without conflict. I played with rebasing and I think I figured out how to stash all commits together.
Regarding "make dist": I tested and resulting tarballs build just fine with or without pandoc installed.
Regarding .md files installation, my rationale was that if you want to consult documentation locally, then you probably are going to use a browser that understands HTML, not Markdown. So I placed inclusion of .html files only under FOUND_CONVERTER_FOR_HTML conditional var.
If you prefer .md files only installed I can

remove if FOUND_CONVERTER_FOR_HTML / endif blocks
replace dist_doc_DATA += docs/something.html with  dist_doc_DATA += docs/something.md

Sounds good?

Another 2 points

my last developments are in "docbranch" of my fork:
https://github.com/f18m/gperftools/tree/docbranch
now all HTML files have been converted to Markdown.

I'd suggest to put all png/gif files into a new folder docs/images to make it easier to browse documentation

Hi Alk,
I think I've completed a review of the aesthetics of markdown documentation.
You can see the results here:
https://github.com/f18m/gperftools/tree/master/docs
I think I didn't miss something, let me know if that looks OK to you.
Then I will stash all commits together so this pull request will appear as a single big commit.
Thanks,
Francesco

Ping :)

Result looks good. Thanks. Couple more comments:
a) would be nice if configure prints warning about pandoc thingy missing
b) after I installed pandoc, make -j8 failed for me on first invocation complaining about lack of rule for readme.html or something like that
c) would be nice if we install .md docs as well. I'd consider installing html and md versions of the docs seperately. E.g. $prefix/share/doc/gperftools/html/readme.html

I am not able to reproduce 'make -j8' issue. Not sure why.
Installing .md and .html is good enough for now.
Couple more things. In cpu profiler format page I am seeing weird gap before  table in binary profile records section.
Also make rule for building .html files that includes sed 'inline' rewriting links is not actually 100% correct. Think of what happens when someone or something interrupts the build. Either pipe all of that in one invocation, or ensure make is aware of intermediate files (i'll smartly remove than afterwards).
I am also seeing non-working image links it tcmalloc.html. Looks like links are pointing into image/ but this is not how we install the files.

Hi alk,

Couple more things. In cpu profiler format page I am seeing weird gap before table in binary profile records section.

You're right. Fixed it.

Also make rule for building .html files that includes sed 'inline' rewriting links is not actually 100% correct. Think of what happens when someone or something interrupts the build. Either pipe all of that in one invocation, or ensure make is aware of intermediate files (i'll smartly remove than afterwards).

Fixed by concatenating the 2 steps into 1 GNU make step

I am also seeing non-working image links it tcmalloc.html. Looks like links are pointing into image/ but this is not how we install the files.

Good point. that required writing a custom (trivial) installation rule in "install-data-hook"...
I checked in all the changes above

I get this new warnings from automake:
docs/Makefile.am:52: warning: variable 'ADDITIONAL_IMAGE_SOURCES' is defined but no program or
docs/Makefile.am:52: library has 'ADDITIONAL_IMAGE' as canonical name (possible typo)
It seemingly just complains that XXX_SOURCES is commonly used to list sources for target XXX.
Another minor thing is page titles (i.e. thingy browsers reflect in tab name) are missing now.
I've noticed couple places where some italic or bold text isn't really so. I.e. section names in pprof_remote_servers.html. Or
CPUPROFILE_FREQUENCY=_x_

I think pprof_remote_servers should just be removed.

Another issue is that make dist doesn't put images into archive (but does put html files). Which then makes build fail.
It is nice if source tarball does have html files and we can install them when end-user system doesn't have pandoc. But not big deal if we don't.

Hi alk,
I did the following changes:

fixed autogen warnings about ADDITIONAL_IMAGE_SOURCES

attempted to get HTML titles right: that requires using the Pandoc "pandoc_title_block" extension. It works and requires basically replacing the "#" char with "%". Then when using pandoc you get correct HTML. Unfortunately that is a pandoc extension and is not rendered correctly on github:
https://github.com/f18m/gperftools/blob/master/docs/cpuprofile.md

so honestly I don't think if it's worth using it.

fixed CPUPROFILE_FREQUENCY and searched for other occurrences of sometext inside  tag but failed to find any additional occurrence
removed pprof_remote_servers.md
I think I fixed "make dist". I tried the following steps:
./autogen.sh
./configure
make dist
mv gperftools-2.6.3.tar.gz ~/tmp
cd ~/tmp
tar xvzf gperftools-2.6.3.tar.gz
cd gperftools-2.6.3
./configure
make install

and now the final "make install"  succeeds.
There is an imperfection though: image files get installed both by the normal "make install" target into /usr/local/share/doc/gperftools/ (where they don't actually need) and by "install-data-hook" in /usr/local/share/doc/gperftools/images/ (where they must be).
Unfortunately I did not quite undertstand how to tell Automake to

include files in the "dist" tarball
but do not install them in the "install" target
Anyway I think it's a minor issue since they are a few Kbs of images that get duplicated.

I am not seeing any obvious problems with current state of this change. So now is time to clean up history, rebase and get it merged at last. Thanks.

Sure.

:closed_book: MEV SW OD Build #27839 result: FAILURE

Manifest Name
Build Status
Jenkins Link
Artifacts
Unsuccessful Stages
Description

mev-ts-trunk
FAILURE
59604
 Artifacts
IPU SDK Build

mev-trunk
FAILURE
56680
 Artifacts
Simics_C0_ACC

mev-release
NOT AFFECTED

mev-ts-release
NOT AFFECTED

mev-ts-release-1.6.1
NOT AFFECTED

mev-ts-release-bitbake
NOT AFFECTED

mev-ts-release-external
NOT AFFECTED

mev-ts-trunk-bitbake
NOT AFFECTED

mev-ts-trunk-bitbake-c1
NOT AFFECTED

mev-ts-trunk-external
NOT AFFECTED

mev-upstream-idpf
NOT AFFECTED

:closed_book: MEV SW OD Build #27894 result: FAILURE

Manifest Name
Build Status
Jenkins Link
Artifacts
Unsuccessful Stages
Description

mev-trunk
FAILURE
56702
 Artifacts
IPU_IMC_6_6_4

mev-ts-trunk
SUCCESS
59633
 Artifacts

mev-release
NOT AFFECTED

mev-ts-release
NOT AFFECTED

mev-ts-release-1.6.1
NOT AFFECTED

mev-ts-release-bitbake
NOT AFFECTED

mev-ts-release-external
NOT AFFECTED

mev-ts-trunk-bitbake
NOT AFFECTED

mev-ts-trunk-bitbake-c1
NOT AFFECTED

mev-ts-trunk-external
NOT AFFECTED

mev-upstream-idpf
NOT AFFECTED

mev-ts-trunk BAT tests: success - build 59633 - it is required if PR have BAT MEV status check as mandatory. If failed, please review Tasks in provided link to Berta. For questions please contact with<EMAIL_ADDRESS>


Dependency

Hello,
You also need this package : openssl-devel .
Regards

Hi,
After some thought given to it I would not add this to the README as a requirement for multiple reasons:

It's only needed when you want to connect to MongoDB via an SSL connection (which is an option, but disabled by default), so a lot of people will not need SSL libraries when using this tool
Even if you use SSL, then the PHP PECL MongoDB extension will need SSL support and It can be solved via multiple ways, which does not invole installing a specific SSL library (e.g. statically linked extension built by compiling your own from source on an another machine)
Even if you would use your Linux distribution's package management, then it would either use OpenSSL packages (different names for different distributions in many cases, so openssl-devel is just one alternativa) or maybe use LibreSSL ones, so the name cannot be well documented

I close this issue now, but if you think there is a good way to include this part to the README.md then please reopen with a suggestion. Thanks.
Best regards,
Pal David Gergely


Support upserts

Allows to upsert object graphs with Realm.add(x, update:true) (and Realm.addAll([x,y], update:true))

There is some work in Core about this https://github.com/realm/realm-core/pull/5516

There is some work in Core about this realm/realm-core#5516

This is still in draft. I suggest we update our code once it lands.

The code seems fine to me, but I think we could improve the test coverage of that feature.
Off the top of my head, some tests that are missing:

A test that upserting an object with a collection different from the original collection is clearing the old collection before populating it with the new items.

is tested in realm_test.dart#715

A test that upserting an object with an empty collection clears the old collection.

isn't that implied by 1) , ie. [bob, carol] updated to [dan]? But yes, I can add a test for that specifically.

Tests that mix and match managed and unmanaged objects.

Doesn't the realm_test.dart#659-668 cover this? The Friends are updated through a Party (that has no PK).

A test that upserting an object with a collection where the items of the collection have changes, those changes get applied to the original objects.

Isn't this tested in realm_test.dart#664-665?

You're right - I misread some of what was going on with the party test.
Re:
2. It is implied, but I think an empty collection is somewhat special, so it deserves an explicit test.
3. Yes, that covers some of it, but I think it's a bit simple. Looking at the code, I don't think there are bugs, but it's a good idea to guard against refactoring by devising a more complex hierarchy being upserted. Something like:
@RealmModel()
class _ObjWithPK {
  @PrimaryKey()
  late int id;

  late _ObjWithPK linkWithPK;
  late List<_ObjWithPK> listWithPK;

  late _ObjWithoutPK linkWithoutPK;
  late List<_ObjWithoutPK> listWithoutPK;
}

@RealmModel()
class _ObjWithoutPK {
  late _ObjWithPK linkWithPK;
  late List<_ObjWithPK> listWithPK;

  late _ObjWithoutPK linkWithoutPK;
  late List<_ObjWithoutPK> listWithoutPK;
}

And test that a more complex graph with cycles is getting correctly inserted. Again, I don't think there's a bug in the code, but I think I can come up with buggy implementations that pass the Party test.

I have gathered all test in one test case about friends and parties, with an easy to follow narrative.


🐛 BUG: Error: write EOF at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)

Which Cloudflare product(s) does this pertain to?
Wrangler core
What version(s) of the tool(s) are you using?
3.4.0
What version of Node are you using?
18.16.0
What operating system are you using?
Windows 11
Describe the Bug
I just run to preview the nuxt3 project locally:

npx wrangler dev .output/server/index.mjs --site .output/public --local --port 8833

and got the error
Please provide a link to a minimal reproduction
No response
Please provide any relevant error logs

Node.js v18.16.0
PS D:\Work\Dev\mx_ys_shop_dawn> npx wrangler dev .output/server/index.mjs --site .output/public --local --port 8833
⛅ wrangler 3.4.0
wrangler dev now uses local mode by default, powered by 🔥 Miniflare and 👷 workerd.
To run an edge preview session for your Worker, use wrangler dev --remote
▲ [WARNING] --local is no longer required and will be removed in a future version.
wrangler dev now uses the local Cloudflare Workers runtime by default. 🎉
⎔ Starting local server...
*** Received structured exception #0xe06d7363: (unknown exception code); stack: 7ffe46bd2bdb 7ffe3635671f 7ff6f26cbc31 7ff6f26bd4a0 7ff6f26de116 7ff6f26ddd60 7ff6f1881480 7ff6f1881179 7ff6f3521ebf 7ffe488f559f 7ffe4948485a
╭──────────────────────────────────────────────────────────────────────────────> ────────────────────────────────────────╮
│ [b] open a browser, [d] open Devtools, [l] turn off local mode, [c] clear console, [x] to exit                       │
╰──────────────────────────────────────────────────────────────────────────────> ────────────────────────────────────────╯
D:\Work\Dev\mx_ys_shop_dawn\node_modules\wrangler\wrangler-dist\cli.js:30632
throw a;
^
Error: write EOF
at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)
Emitted 'error' event on Socket instance at:
at emitErrorNT (node:internal/streams/destroy:151:8)
at emitErrorCloseNT (node:internal/streams/destroy:116:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -4095,
code: 'EOF',
syscall: 'write'
}
Node.js v18.16.0

Did you find a solution to this?

I'm also facing same issue

For me it was a corrupted .DLL file. During an automatic windows update my internet went down for a moment and did not install it correctly and received the warning when I started my computer again and downloaded the .DLL file from the warning. A quick way to continue developing is using WSL as is Linux based and virtualized I could continue doing everything I needed until I solved it

This is happening for me as well. Windows 11 - Node v20.9.0.  I was simply using the Hello World example worker for testing. It works great when using "--remote" flag.

I can confirm that its now working for me on the latest version of wrangler.

The issue might be due to the lack of necessary tools to compile native modules on your system.
You can resolve by running the install_tools.bat in the directory of your nodejs. For example for nvm user, you can find in  C:\Users\[Username]\AppData\[Roaming/local]\nvm\[nodeversion]rs\[Username]\AppData\[Roaming/local]\nvm\[nodeversion]

@nghianguyen119 Thanks, It worked after running install_tools.bat

Im also facing the same error Write :EOF , while running the npm run dev  command to run the server locally
after intalling install_tools.bat it worked, now server is running on the port 8787

hey i am facing this issue
Error: write EOF
at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:95:16)
at WriteWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
errno: -4095,
code: 'EOF',
syscall: 'write'
}
can any one help me

I also faced the same issue, got resolved after running install_tools.bat...try running this .. it will be solved
or re-install nodejs, and tick the checkbox containing additional tools while installing


Add translation for Axolotls

Adds en_US translation for text.betterf3.line.axolotls

This was bugging me, so i thought i would PR to fix it. Translations will need to be done, but i'll add a few while i'm here.

I have added them. Any more changes you may have can be added to the Crowdin page for easier updating!
https://crowdin.com/project/betterf3


Use curl_multi_socket_action with libevent instead of curl_multi_perform

Differences laid out here: https://curl.se/libcurl/c/libcurl-multi.html.
Could potentially help with https://github.com/supabase/pg_net/issues/13.

The pg background worker module already has some functions that encapsulate kqueue, select, poll(ref). Maybe we should try to use that first.

Hmm not too familiar with postgres internals, but that looks like a purpose-built latch implementation for the storage manager.

Been learning postgres internals on my own, so I'm not sure either. I just got a hunch from seeing pg_cron codebase(uses WaitLatch here) and that there are events for sockets here.


[berkeleydb] build failure

Host Environment

OS: [Microsoft Windows [Version 10.0.19041.423]]
Compiler: Visual Studio 2019 Version: 16.7.0  VS Tools Version: 14.27.29110

To Reproduce
Steps to reproduce the behavior:
upgrade Visual Studio 2019 to 16.7.0
cd [vcpkg-root]
git pull updates vcpkg latest as of 20200806 at 22:47 GMT 18:57EDT
set VCPKG_DEFAULT_TRIPLET=x64-windows
set VCPKG_VISUAL_STUDIO_PATH=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community
.\bootstrap-vcpkg.bat -verbose -disableMetrics -win64 -withVSPath "'%VCPKG_VISUAL_STUDIO_PATH%'"
.\vcpkg.exe install berkeleydb --editable
Failure logs

Cut and past the appropriate build messages from the console output.
Computing installation plan...
The following packages will be built and installed:
    berkeleydb[core]:x64-windows
Detecting compiler hash for triplet x64-windows...
Starting package 1/1: berkeleydb:x64-windows
Building package berkeleydb[core]:x64-windows...
-- Using cached D:/Git/vcpkg.downloads/db-4.8.30.NC.zip
-- Using source at D:/Git/vs2019.prod/buildtrees/berkeleydb/src/db-4-c2cacfc4e9
-- Configuring x64-windows
-- Building x64-windows-dbg
CMake Error at scripts/cmake/vcpkg_execute_build_process.cmake:139 (message):
    Command failed: "C:/Program Files/CMake/bin/cmake.exe" --build . --config Debug --target install -- -v
    Working Directory: D:/Git/vs2019.prod/buildtrees/berkeleydb/x64-windows-dbg
    See logs for more information:
      D:\Git\vs2019.prod\buildtrees\berkeleydb\install-x64-windows-dbg-out.log

Call Stack (most recent call first):
  scripts/cmake/vcpkg_build_cmake.cmake:91 (vcpkg_execute_build_process)
  scripts/cmake/vcpkg_install_cmake.cmake:24 (vcpkg_build_cmake)
  ports/berkeleydb/portfile.cmake:22 (vcpkg_install_cmake)
  scripts/ports.cmake:79 (include)
Error: Building package berkeleydb:x64-windows failed with: BUILD_FAILED
Please ensure you're using the latest portfiles with .\vcpkg update, then
submit an issue at https://github.com/Microsoft/vcpkg/issues including:
  Package: berkeleydb:x64-windows
  Vcpkg version: 2020.06.15-nohash
Additionally, attach any relevant sections from the log files above.

Please attach any additional failure logs mentioned in the console output.
install-x64-windows-dbg-out.log

Additional context

install-x64-windows-dbg-out.log mentions several issues with file C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\include\atomic lines 1674, 1675, 1685, 1686

Note 1: port berkeleydb was building fine with VS 2019 version 16.6.5, but now is not building with version 16.7.0
Note 2: port berkeleydb still builds fine with VS 2017 version 15.9.25

Thanks for posting this issue. This is a source bug, I will confirm and fix it soon.

@LilyWangL Thank you very much! Have a wonderful weekend.


share this with /r/incremental_games

I'm sure someone will appreciate it. #5 is a blocker though. also, let's use this myself some more first, to see if the api feels good before a public announcement locks it in.

also, let's use this myself some more first, to see if the api feels good before a public announcement locks it in.

#7

https://www.reddit.com/r/incremental_games/comments/5nn5x5/swarmnumberformat_a_js_library_for_nicely/?st=ixv21rp6&sh=b61adfd4


Only get SQL queries instead of executing them

Problem
I am currently working with retool.com which allows me to interface with my postgres database in a dashboard. The first thing I reached for was Prisma! I was able to generate a query using intellisense pretty quickly, but the problem was I had to run the query and look in the logs to see what the sql was. This is problematic if the query is an insert/update/delete/etc.
Suggested solution
const sql = await prisma.feed.findMany({
  // complex stuff here
}).toSQL();

console.log(sql); // => SELECT .....

Additional context
One problem I noticed is that sometimes Prisma splits the findMany into two actual SQL queries instead of doing a join. This wouldn't work as the user is expecting one SQL statement so Prisma would need additional logic to stop this from happening.

Closing as duplicate of https://github.com/prisma/prisma-client-js/issues/746
Thanks for the issue!


Add endpoint dataset

This PR allows using the endpoint price_hist to download the data from Binance.
The requirements.txt is updated.

Ok, i'm going to merge


Show Folder README

Describe The Feature
Option to show the README file of the current folder if you click on the folder, like in GitHub.

Remarks
The user can still click on the arrow icon to the left of the folder to expand / collapse it.
This feature is designed for consistency with GitHub's folder system, which renders the README file of a folder at the bottom (if it has one).

Settings

Key
Value

Suggested name
explorer.previewFolderReadme

Description
Shows the README file of a folder when clicking on the folder name.

Type
bool

Default Value
false

-
-

Value
Description

true
Shows the README file of the folder if the name is clicked

false
Don't show the README file of the folder if the name is clicked

See also:

#201288
#201231

How would you expect this to work in vscode? You would click the folder and the readme file would be opened in your editor?

Yes, ideally.
This would be useful for consistency with GitHub.

I don't think this is a feature that we will take on. Our goal is to be a good editor experience and not have parity with GitHub. The little chevron is a hard click target and users much prefer to click the entire folder.

We closed this issue because we don't plan to address it in the foreseeable future. If you disagree and feel that this issue is crucial: we are happy to listen and to reconsider.
If you wonder what we are up to, please see our roadmap and issue reporting guidelines.
Thanks for your understanding, and happy coding!


Issue #546 - New API endpoint for recording stub mappings

As promised, here's a PR that integrates my wiremock-snapshot extension. It adds a  new API endpoint, /__admin/snapshot, for recording stub mappings from the request journal. The README has more details, but here's a summary of the options it accepts:

"filters" - Request patterns to use for determining the requests for which to create stub mappings.

Possible values: Identical to /__admin/requests/find
Default: no filtering.

"persist" - If set to true, persist stub mappings to disk. Otherwise, just output them.

Possible values: true, false
Default: true

"sortFields" - Array of fields in the request to use for sorting stub mappings, mainly for output.

Possible values: "url", "method", or a header name (e.g. "Accept")
Default: no sorting.

"captureHeaders" - Header matchers for including headers in the stub mapping. The request is matched against each matcher, and the associated header is added to the stub mapping if there's a match.

Possible values: Request matchers for headers.
Default: none

"outputFormat" - Determines response body.

Possible values: "ids" to return array of stub mapping IDs, "full" to return array of stub mapping objects
Default: "ids"

Design issues
Filtering
Filtering is currently request-only. I'm not sure if response filtering is desired. If so, we'd probably want to add a new ResponsePattern class (analogous to ResponsePattern) to encapsulate matchers for the response. This would be best accompanied with some refactoring to make Response into a an interface implemented by LoggedResponse .
Body file extraction/naming
Currently, the API does not extract the response bodies to a separate file. While it'd be easy to add a flag to enable this, I think more granularity is needed. Body file extraction is useful for large binary responses, but generally undesirable for small JSON responses. Perhaps we can add a "bodyFileCritera" option consisting of matchers that determine whether or not to extract the body. That way, you could use "headers": { "Content-Type": { "doesNotMatch": ".*json.*" } } } to only extract non-JSON responses. Again, we'll probably want to add a ResponsePattern class for this, and a "bodySize" matcher to allow matching by the size of the body.
Uniqueness/Deduplication
To ensure stub mappings are unique, I have StubMappingTransformer#apply() generate a MD5 hash of the RequestPattern JSON and set that as the UUID. This is pretty hacky, but it seems to work well. I considered just using hashCode(), but that's not appropriate since it has a high collision probability and the result is not guaranteed to be consistent between executions (see the JavaDoc).
This makes me wonder why StubMapping IDs are random instead of being a content-based hash. I see this was added in PR #306, but it doesn't seem uniqueness was considered.
Outputting
The options available in "outputFormat" seem sufficient to me. If we add body file extraction, we may want to add another output format to serve everything in a zip file.
Sorting
I implemented the ability to sort the stub mappings by the request URL, method, or a header. The main use case for this is to make it easier to manually edit the stub mappings. This is arguably better served by extension points, so it might be better to just remove this feature.
The class used for sorting, RequestFieldsComparator, is pretty hacky and a violation of the open-closed principle, but I couldn't think of a better implementation.
Extension points
An extension point for modifying StubMapping objects would be easy to add. Specifically, we could have WireMockApp#buildAdminRequestHandler() pass in the stub mapping extensions along with the API extensions to AdminRoutes, which could then call
router.add(POST, "/snapshot", new SnapshotTask(this.stubMappingTransformerExtensions));

instead of
router.add(POST, "/snapshot", SnapshotTask.class);

Adding the ability to customize the file names for persisted mappings would be more complex. Perhaps we could use a wrapper class, e.g. RecordedStubMapping, that would allow extensions to set the name for the mapping file and (if applicable) the body file.
Statefulness/Scenarios
It'd be easy to detect duplicate requests and then set the StubMapping scenario name, but I'm not quite sure about naming. We could just use something like UniqueFilenameGenerator to auto-generate the scenario name, but we'll probably want to have some way of customizing it. Maybe a "scenarioNameTemplate" option that accepts a Handlebars template?
edit: Minor clarifications

Hi, thanks for raising this, and apologies for taking this long to respond.
I'll address the points you've made in turn:
Filters
Looks good. Request filtering is virtually free to add so it would seem silly not to include it, but I don't think it's worth attempting to implement response filtering at this point. It's not something I've had any feedback to indicate would be useful.
One additional thing I think is essential here: being able to filter by specifying a list of request IDs.
Persist parameter
Also looks good.
Sort fields
Not sure about this. Given that stub insertion order affects precedence i.e. all other things being equal a more recently added stub mapping will be preferred, could sorting have the effect of muddying the water? Also, as per your comment further down, I'm wondering whether adding some kind of configurable/pluggable naming strategy might be better.
Capture headers
Looks good. I'm wondering whether there's a case for specifying the operator also. I can think of a few cases where contains might be better than equalTo e.g. the Accept header at the time of recording is application/json but then changes to application/json; q=0.9,*/*.
Output format
I'd be tempted to default to full here, for consistency with the rest of the API. Given my earlier comment about filtering by ID, it would be great to be able to see the mapping of request ID -> stub mapping ID here.
Body file extraction
The configuration options I'd probably want here are:

Size threshold for emitting a body file for binary content
Size threshold for emitting a body file for text content

In both cases omitting the parameter means don't write body files for any size.
WireMock already contains some code that can reasonably reliably tell binary from text content. Might be worth reusing that.
Uniqueness/Deduplication
Is there any reason StubMapping.equals() won't work here?
The reason stub IDs are generated randomly is that they're meant to be persistent (serving as a primary key). If a stub is edited, the ID would have to change, making it a harder job to keep track of it over time.
Outputting
Agree zip output would be nice to have. Perhaps not important enough to require it in the first release though.
Extension points
An extension point to transform output is really important IMO, especially as it's quite easy to implement.
Regarding file naming, perhaps the answer is that we move to setting the file name to whatever is in the stub's name field. That way the transformer can edit that if the filename needs to be altered. More generally, keeping the two the same would make it easier from a user perspective to trace between files and mappings.
Scenarios
Agree using UniqueFilenameGenerator would make sense. Perhaps the scenario gets the name generated by that, and the individual stubs get that plus a sequential number, which would also be the scenario step name.
Also agree it should be customisable. Perhaps given the previous point, the extension should have two methods that can be overridden - one for stub name, one for scenario name.

One additional thing I think is essential here: being able to filter by specifying a list of request IDs.

Yeah, that'd be useful. Added in 97a767f8cc834ed39499c81af419c87e6ff62241

Given that stub insertion order affects precedence i.e. all other things being equal a more recently added stub mapping will be preferred, could sorting have the effect of muddying the water?

I agree, this could be confusing. I removed it in 18d44f5af1c7a3f18a7c177a8b433d16b7dbcdc6

I'd be tempted to default to full here, for consistency with the rest of the API. Given my earlier comment about filtering by ID, it would be great to be able to see the mapping of request ID -> stub mapping ID here.

You're right, "full" is more consistent. I changed it to default to "full" in 47f4adcacf9afb8e65d091abd208efb2b46b69de. Having that request ID mappings does sound useful; I'll do that this weekend.

If a stub is edited, the ID would have to change, making it a harder job to keep track of it over time.

Oh, I didn't consider the case of editing a mapping. Yes, that means a surrogate key (in RDBMS terms) makes sense. One of the reasons I had it generate a unique hash is because I figured calling admin.getStubMapping(stubMapping.getId()) would be a lot faster than admin.countRequestsMatching(stubMapping.getRequest(), but looking at InMemoryRequestJournal it seems they're both O(n). I changed it to use admin.countRequestsMatching() in 7e49cfb21d48d7767ecb186131664d98d5a03efb

Agree using UniqueFilenameGenerator would make sense. Perhaps the scenario gets the name generated by that, and the individual stubs get that plus a sequential number, which would also be the scenario step name.

That sounds like a good plan for now. I'll work on that this weekend.
For body file extraction and extensions, I entered PR #682 to facilitate that.

This is all looking great so far.
I have a couple of small requests:

Could we make the route map to /recordings/snapshot?
The reason is that I'd like to further extend this at some stage to provide the option of providing a start/stop recording mechanism via /recordings/start-recording and /recordings/stop-recording. In practice this can probably just track the serve event IDs seen in the time interval between starting and stopping then snapshot on stop, but since this is what most people are used to I still think it aids usability.

Could we also have an acceptance test or two?

Again, thanks for all the effort you've put into this!

I've finished adding basic scenario support in 6eaffd841c2b11be7539a7ff8e7c9c0bd515a671. This can be enabled using the new "requestsAsScenarios" flag. The logic for generating the scenario name is at https://github.com/tomakehurst/wiremock/commit/6eaffd841c2b11be7539a7ff8e7c9c0bd515a671#diff-1513fe9344f15d8e88447a9dc21831cfR51
The following acceptance test shows how it works: https://github.com/tomakehurst/wiremock/pull/674/commits/6eaffd841c2b11be7539a7ff8e7c9c0bd515a671#diff-2735104c468d56aed57eb1adbdd3f56eR282
I need to write tests for SnapshotStubMappingScenarioHandler, but I think I'll wait until we iron out the details for that.

Could we make the route map to /recordings/snapshot

Done in ddfbf57

Could we also have an acceptance test or two?

That's kinda what the SnapshotTaskTest class is meant for. It looks like the only difference between those tests and the other Wiremock acceptance tests is that the former instantiate SnapshotTask directly, while the latter are full end-to-end tests. I'll go ahead and change SnapshotTaskTest to be end-to-end so they're consistent.

 They uncovered a few bugs, so that was a good call. I also wrote a shell script at https://github.com/MasonM/wiremock-snapshot/blob/master/test-script.sh to make testing at the command line easier.
I ended up rewriting the whole scenario handling logic because I realized my original implementation was pretty bad. Hopefully this one is a little better.
The extension point for transforming stub mappings is in 44634c5f072ecc5c1bf11de530dd3dd5e38c4097. I added two new options, "transformers" and "transformerParameters", to the endpoint for consistency with response definition transformers. The StubMappingTransformer transformers behace pretty much the  same as response definition transformers.
I'll do body file extraction next.

Wow, again this is prolific stuff!
I'll try and review everything you've done by e/o tomorrow at the latest.

FYI: The body file extraction code in 516095a and ce49104 is a WIP. I entered PR #690 to fix an issue I ran into where binary bodies were getting corrupted.

@tomakehurst I should've asked this earlier, but would it help if I rebased these commits into a logical progression, starting with a bare-bones snapshot endpoint and adding on one feature per commit? That seems like the best way of making this easier to review.

p.s. Let me know if there's anything I can do to make this or PR #690 easier to review

Sure. I'll be devoting a lot more time to this from Monday (no more client work, woo!), so expect to see some comments then.

 Would you mind taking a look?

Merge conflicts resolved in 996d723. I also did yet more refactoring for scenario handling/post-processing in 901db8d

Great, thanks for this.
I've been spending a bit of time reviewing the original spec, plus past bug tickets etc. and I've compiled a list of what I think we have left to do:

Support start recording/stop recording via the API (I think this could be a very simple case of bookmarking the start and end logged requests then just snapshotting as usual)
Add all new API calls to the DSL
Support truncation of generated filenames to a maximum length (there are a few bug tickets around complaining about very long URLs generating filenames the OS can't handle)
Set the name field of recorded stubs to be the same as the filename (without the extension)
Add the ability to configure the ignore* flags when JSON body matching

Is there anything you think I've missed, or that you'd like to see in there?
I'm intending to work on 1 and 2, which I think are the biggest but also (hopefully) the least conflicting with what you're doing.
I'll push any changes I make to my remote branch and then post a comment on here so that you can pull them into your branch, if that suits?

Two more things:

Adding IDs to generated stub mappings (so that they can be addressed later, as discussed)
Documentation (I'll start working on this)

@tomakehurst Yeah, that sounds like a good list. I'm fine handling 3-5 and changing the "full" output format to match the example I gave in https://github.com/tomakehurst/wiremock/pull/674#discussion_r125558910 (almost done, just need to update the acceptance tests). I sent an invite to give you write access to https://github.com/MasonM/wiremock/, so you're free to just push to this branch, otherwise I can just merge in your changes.

Actually, I think the full output is fine. I hadn't appreciated when reviewing the test case that you'd set it to loose JSON matching and that the IDs are already present in the stubs.
Also, I've started the DSL work and it includes a couple of refactors so please could you pull from here:
https://github.com/tomakehurst/wiremock/tree/MasonM-add-snapshot-admin-task
Thanks again!

I'm having trouble pushing to your branch for some reason. Sure it's my setup, but can't figure out what's up. Would you mind pulling from mine for now?

My last couple of commits (on my branch still) modify the capture headers behaviour based on my earlier comment.

Okay, I merged and push your changes. Github is saying you haven't accepted the invitation yet, so maybe the e-mail got lost. I think you can visit this link to accept it: https://github.com/MasonM/wiremock/invitations
For the body filename truncation fix, there's one other issue I found: Unicode characters in the path part of a URL or IRI are being included un-encoded. Here's a shell script that reproduces this by recording a request to https://en.wiktionary.org/wiki/¥:
#!/bin/bash

java -jar build/libs/wiremock*standalone*.jar &
WIREMOCK_PID=$!
trap "kill $WIREMOCK_PID" exit

until $(curl --output /dev/null --silent --head http://localhost:8080); do
        sleep 1
done

curl -s -d '{ "response": { "proxyBaseUrl": "https://en.wiktionary.org" } }' http://localhost:8080/__admin/mappings > /dev/null

curl -s 'http://localhost:8080/wiki/%C2%A5' > /dev/null

curl -s -d '{ "extractBodyCriteria": { "textSizeThreshold": "1" } }' http://localhost:8080/__admin/recordings/snapshot

The Yen symbol is not allowed in NTFS, so I think this will break on Windows, but I haven't tried it.
We could just percent-encode all Unicode characters, but that'd make the filenames hard to read when recording requests to non-English sites. Providing a"bodyFileNameTemplate" option for customizing the filename seems like the best option. Using Handlebars for the template would probably be pretty easy, since we already have a request model for templating. We'd need to add a few properties and helpers, and the default template would look something like:
{{#truncate 255}body-{{urlSlugify request.url}}-{{uniqueId}}.{{detectExtension request.body}}{{/truncate}}

I think it'd be a good idea to have a default filename generator that can produce safe filenames across all platforms even in the presence of odd unicode chars. If that means producing ugly filenames for non-latin-based languages, then I'd say that's worth it (at least in the short term) if it means we can get it released sooner.
I like your idea for a name template too though. Maybe the template should be for the stub's name field in addition to the body file (so that they're always the same)?

Sorry I haven't been active in the last few days; I'm currently on a business trip. I'll be getting back Thursday and have the weekend free to work on this.

No worries, you've contributed a huge amount already!
I think the only significant thing left to do (apart from a mountain of documentation) is the stub and file naming strategy stuff we talked about.
It'd be great to get your feedback on what I've done also though. To summarise (in no particular order):

Added start/stop recording. Essentially this just bookmarks the starting event when you hit record, then snapshots everything between that and the end event when you hit stop.
Added a very basic UI web page under /__admin/recorder to start and stop recording (since there's no CLI equivalent this time).
Added everything new in the API to the DSL.
Tweaked the behaviour of the persist flag so that it still adds the stub mapping, but with its persist flag set to false (user can always hit reset afterwards if all they wanted was to download the stubs).
Fixed a bug with the scenario recording (it wasn't setting the state progression up correctly)
Changed the capture headers behaviour as per my comment earlier (it now uses the captured value).
Added a JSON matching flags section to the spec
Moved recording related classes to the recording package, as there are now quite a lot of them.

OK, I hope you don't mind but I went ahead and wrote a basic naming implementation that hopefully satisfies OS naming restrictions. It now does the following:

Generates a name for the stub, based on the URL
Names the file after the stub name + ID, rather than the generic "saved-stub-xxxx"
Names the body file after the stub name + ID also, but with the correct file extension
Truncates names to a max of 242 (name 200 + ID 37 + extension 5), since 255 seems to be the predominant OS name length limit

I'd still like to see your idea about templating the name implemented, but I think it's worth doing a release on what we've got in the meantime.

Right, I'm going to stop changing stuff now.
I'll hold off merging this until you've had a chance to take a look.

Okay, I'm back and I've had a chance to look things over a bit. I haven't reviewed everything, but your changes look great! Just a few comments:

Yes, now that I think about it, templating for the body filename is probably overcomplicating things for the initial release. Your code to do least-common-denominator naming in SafeNames looks good.
With body file extraction, there's still the issue with duplicate files being created for identical requests. This is a problem when recording multiple times (e.g. when running a test suite), and since the extracted body files accumulate until they're manually cleared, it can be pretty annoying. I can think of two solutions to this:

Instead of using the StubMapping ID in the filename, use a hash of the file contents. I did a little research, and FarmHash is a good choice for the hash function, since it's very fast, has few collisions, and is included with Guava. For clarity, we should probably continue using the URL/extension in the name, which means it's still possible for duplicate files to be generated if different requests result in the same response, but that seems like an acceptable tradeoff. I implemented that in this branch and tested it out pretty extensively. If it looks good to you, it should merge cleanly. If not, that's fine.
Keep track of extracted response bodies and skip duplicates. Won't work for files that are already on disk and will increase memory usage, but won't be hard to implement.

The JsonMatchingFlags stuff looks good, but that only deals with the use case of customizing JSON body matching. The proposal in https://github.com/tomakehurst/wiremock/issues/707#issuecomment-312794279 is more general, since it allows overriding the valuePatternForContentType() logic for deriving the matcher from the Content-Type header. The primary use case for this is with non-standard MIME types for JSON and XML bodies. Many applications send JSON with the MIME type "text/javascript" or "text/plain" (see https://github.com/mislav/faraday-stack/pull/2 for examples), which will cause the current logic to use equalTo . Another possible issue is DTDs: the standard MIME type for DTDs is application/xml-dtd, which will cause it to use equalToXml. I implemented the proposal in this branch, except I inlined "matchOptions" to keep the deserialization code simple. It it quite a bit more complex than the current code, and I'm not sure how common the cases I mentioned above are, so I'll leave it you to decide if that added complexity is worth it. If so, that branch should merge cleanly.
The simplified start/stop recording endpoints are a good idea, but maybe it'd be worth allowing users to specify the whole ResponseDefinition instead of just the targetBaseUrl. The two use cases for this are passing additional headers via additionalProxyRequestHeaders and response templating so the proxy URL is dynamic (which I added in PR #641). I have a working implementation of this in this branch, but I'd need to update the docs and add more tests. I'm not quite sure if this is worth the added complexity.
The parameter list for the snapshot API has gotten pretty long, and may be intimidating for new users. The start/stop recording helps a lot with this, but maybe it'd be worth structuring the parameters a little more. One way of doing that would be wrap the options specific to generating StubMappings in a separate object, called something like "generatorOptions". Here's a branch implementing that, and here's what a request would look like:

POST /__admin/recordings/snapshot
{
    "filters" : {
        "urlPathPattern" : "/api/.*",
        "method" : "GET",
        "ids" : [ "40a93c4a-d378-4e07-8321-6158d5dbcb29" ]
    },
    "outputFormat" : "FULL",
    "generatorOptions": {
        "persist" : false,
        "captureHeaders" : {
            "Accept" : { },
            "Content-Type" : {
                "caseInsensitive" : true
            }
        },
        "extractBodyCriteria" : {
            "textSizeThreshold" : "2 kb",
            "binarySizeThreshold" : "1 Mb"
        },
        "repeatsAsScenarios" : false,
        "transformers" : [ "modify-response-header" ],
        "transformerParameters" : {
            "headerValue" : "123"
        },
        "jsonMatchingFlags" : {
            "ignoreArrayOrder" : false,
            "ignoreExtraElements" : true
        }
    },
}

I'll try to finish reviewing tomorrow.

Okay, I'm finished reviewing and haven't noticed any major issues. Of the 4 things I brought up in my last comment, I think the issue with duplicate body files is the only major one.
I was thinking a little more about re-structuring the parameter list, and I think it'd be clearer to additionally wrap the request and response options in objects inside generatorOptions, e.g.
POST /__admin/recordings/snapshot
{
    "filters" : {
        "urlPathPattern" : "/api/.*",
        "method" : "GET",
        "ids" : [ "40a93c4a-d378-4e07-8321-6158d5dbcb29" ]
    },
    "outputFormat" : "FULL",
    "generatorOptions": {
        "request": {
            "captureHeaders" : {
                "Accept" : { },
                "Content-Type" : {
                    "caseInsensitive" : true
                }
            },
            "jsonMatchingFlags" : {
                "ignoreArrayOrder" : false,
                "ignoreExtraElements" : true
            },
            "repeatsAsScenarios" : false
        },
        "response": {
            "extractBodyCriteria" : {
                "textSizeThreshold" : "2 kb",
                "binarySizeThreshold" : "1 Mb"
            }
        },
        "persist" : false,
        "transformers" : [ "modify-response-header" ],
        "transformerParameters" : {
            "headerValue" : "123"
        }
    }
}

The advantage to that is it's immediately obvious that extractBodyCriteria applies only to response bodies, captureHeaders only applies to requests, and transformers applies to the whole StubMapping. This does make calls to the API more verbose, so there is a tradeoff. I implemented this change in the branch I mentioned previously.

👍
I'm tempted to live with this for now. There's a bit of a tradeoff - although capturing multiple copies of the same response body isn't very space efficient, it does keep things simple. If I want to edit a stub after recording I can change the body file safe in the knowledge the edit will only affect the intended stub.
I like this, let's merge it. I'd like to add matchesJsonPath and matchesXPath support in the near future, and this will enable that nicely.
I did consider whether it would be worth giving more control over the proxy mapping when start/stop recording, but TBH for now I think it's enough
just to advise people wanting full control to use snapshotting. My main intention for including start/stop recording was to help beginners who might
find snapshotting harder to grasp than recording, so it's deliberately a bit simplified.
I can see your point, but OTOH an extra level of nesting is a bit painful for folks wanting to construct e.g. a curl request to record/snapshot.
On balance, I think this probably isn't worth the effort so I'd prefer to leave as-is.

Great thanks. I think it might be time to release!

If I want to edit a stub after recording I can change the body file safe in the knowledge the edit will only affect the intended stub.

Okay, that makes sense. The issue for me is that the mock data I'm working with largely consists of media files, so I have to delete old body files each time I re-record my test suite. That makes Docker-based deployment of Wiremock difficult, since it means I have to map the "__files" directory to the host.
Would you be open to having DELETE /mappings accept a boolean parameter called "removeBodyFiles" that deletes all body files associated with the stub mappings? That'd solve this issue and be easy to add. I can enter a PR for that in the next day or two.

I like this, let's merge it.

Okay, done. There are a couple issues with RequestBodyAutomaticPatternFactory that I added hacks for:

Bodies of type "multipart/form-data" were using equalTo, which meant it never matched. To match a m "multipart/form-data" requests requires a new matcher, since it needs to use the boundary specified in the Content-Type header. I added a hack to have it use AnythingPattern in that case, since that seems better than a stub that never matches.
Binary bodies always used equalTo, which matched by the UTF-8 decoded body. Since that corrupts non-UTF8 bodies, there needs to be a way to match against the binary body. Adding a equalToBase64 matcher would work, and I think I can get a PR in for that this weekend. In the meantime, I just added a hack to use AnythingPattern

OK, understood. I like the idea of being able to specify that deleting a mapping should delete the corresponding body file. If you're happy to PR that, that'd be great.
Form data support isn't great in WM at the moment. I've been meaning to think about adding a matcher for that for a while.
Agree binary body matching also needs sorting out. Again, if you're happy to work on that, it'd be much appreciated.
Overall I don't think these issues are worth holding up a release for. Would you agree?

Overall I don't think these issues are worth holding up a release for. Would you agree?

Sure, if you're fine with the hacks in RequestBodyAutomaticPatternFactory. It's 2AM here, so I'm off to bed. I'll work on adding a equalToBase64 matcher tomorrow.

I'll take a final glance at it. In any case I think I'm going to tweak the DSL a little before finalising.
Thanks again for all your help, sorry if I've kept you up late!

And we're live! One more time - thanks for all your help. I think this is the most awesome new feature WireMock has had for a long time, and I'm looking forward to seeing what feedback it gets.


Vsphere driver

New driver for vSphere
Description
With this driver one can manage a vSphere environment, the connection is made with a vCenter server, it might not work with just the eSXI host. This driver has a dependency called pyvmomi that can be installed with pip install pyvmomi, it is needed since the rest API of vSphere is not robust enough yet. The rest driver will work, except list_images, snapshot methods and console method which need the soap driver. Images in this driver are considered  VM templates or ovf files. The Rest driver also lacks the ability to clone a VM due to the API. The added tests check core functionality of the rest driver only.
Status

done, ready for review

Checklist (tick everything that applies)

[x] Code linting (required, can be done after the PR checks)
[x] Documentation
[x] Tests
[x] ICLA (required for bigger changes)

Unfortunately there is one last test that won't pass since the driver has async code. In production environments with VMs in the triple digits, the list_nodes method gets pretty slow, for each VM to_node gets called in a serial manner where probably more API requests are made.., this can take some notable time. I would argue that every driver could benefit from an async list_nodes, in this case it is optional, there is an argument to enable or disable the async functionality. I wonder if something can be done in regard to the test that can't be fulfilled.


Not working in Edge

The Websockets request appears to be returning a 101 code, which I assume means that it is using the HTTP fallback, which I assume is causing the streams to not work. I'm going to look deeper into this issue and see if I can provide more insight.

Actually the 101 status code is normal. You can force disabling websockets (for testing purposes) starting meteor with the env var DISABLE_WEBSOCKETS set to true

Hmm ok, interesting, i'll dig deeper to try and get more info as to why the event handler isn't firing, uniquely on edge.

I have tested all browsers with the DISABLE_WEBSOCKETS env var and it appears to not be the issue causing silent failure in Edge.


ENYO-5358: snapshot build fails with react-dom 16.4.1

Fixes the issue by using dynamic timer/console functions during react-dom modular evaluation. While evaluated in v8 snapshot, any execution will be a noop, however when the window exists, the real window-based function will execute. This is mainly for instances where the setTimeout or console functions are captured and assigned to local variables at a modular level within react-dom. In React 16.4.1, this is the case within the react scheduler section.
Enact-DCO-1.0-Signed-off-by: Jason Robitaille<EMAIL_ADDRESS>

Added safeguards. In reality, given the targetted nature of v8 snapshot (and the minimum version v8 snapshot is supported shouldn't be an issue, but I agree better to be safe and guard edgecases :+1: )


Missing component when compiling Z-Stack-firmware/coordinator/Z-Stack_3.x.0

Hi
In Z-Stack-firmware/coordinator/Z-Stack_3.x.0/COMPILE.md there are 3 different ccs projects to select
- znp_CC26X2R1_LAUNCHXL_tirtos_css
- znp_CC1352P_2_LAUNCHXL_tirtos_css
- znp_LP_CC2652RB_tirtos_ccs
However I can only find the first two on a fresh install of SIMPLELINK-CC13XX-CC26XX-SDK_<IP_ADDRESS> using Code Composer Studio 11.
I am using linux platform for both installers.
ls -1 /opt/ti/simplelink_cc13xx_cc26xx_sdk_5_40_00_40/examples/rtos
CC1312R1_LAUNCHXL
CC1352P1_LAUNCHXL
CC1352P_2_LAUNCHXL
CC1352P_4_LAUNCHXL
CC1352R1_LAUNCHXL
CC26X2R1_LAUNCHXL
LP_CC1311P3
LP_CC1312R7
LP_CC1352P7_1
LP_CC1352P7_4
LP_CC2651P3
LP_CC2652PSIP
LP_CC2652R7
LP_CC2652RB
LP_CC2652RSIP

Kind regards

I have the feeling that this may be an issue with the user interface of CCS11. I will try on a different platform

Just in case anybody comes across this again. You need to perform a >>>FULL<<< install of the first SIMPLELINK-CC13XX-CC26XX-SDK_<IP_ADDRESS> package.


Improve greenlight nginx to be equal to bbb-install.sh

location=/  is duplicated.
It is installed by BBB: /etc/bigbluebutton/nginx/greenlight-redirect.nginx

Copy from new bbb-install.sh (syntax should be holded)

Since you touch the complete nginx-vhost: could you maybe also take a look at #16 and incorporate that?


dark mode

Enable a dark mode theme choice

This will probably wait until we have the basic site done, then when we have a profile page or such for customization.


Add compatibility with "Honhan BoardView" files (.bdv)

https://www.deadlink.....
There's a copy of Honhan's Boardview.exe and a sample .bdv file (compal_la-5911p_r1.0_boardview.bdv).  I hope you're interested in adding compatibility with this file type as well.
Great job and thanks for your efforts.

It seems Dropbox has disabled that download link.

BDV-BoardView.zip
maybe this will work? dropbox got too much traffic. lol you're too popular.

Thanks, got it decrypted -- will work on integrating the format later.
compal_la-5911p_r1.0_boardview.bdv.txt

@chloridite do you have the decoding for these two formats added yet/soon?  Thanks for doing them :+1:

See here: https://github.com/piernov/OpenBoardView/commits/bdv-support

package main

import (
	"io/ioutil"
	"os"
)

func main() {
	buf, err := ioutil.ReadFile(os.Args[1])
	if err != nil {
		panic(err)
	}
	line := 1
	for i := 0; i < len(buf); i++ {
		b := buf[i]
		if b == 0xa {
			line++
			continue
		}
		if b == 0xd && i < len(buf)-1 && buf[i+1] == 0xa {
			line++
			i++
			continue
		}
		l := int8(line % 0x7f)
		if true || b <= 0x7f {
			b = byte(0x7f - int8(b) + l + 0x20)
		} else {
			b = byte(0x7f - int8(b) + 0x20 - l)
		}
		buf[i] = b
	}
	os.Stdout.Write(buf)
}

This is my decoder from reading the disassembled .exe -- might lead to a slightly better-looking decode_bdvfunction.

Are we okay to close this now?  I believe this is fully supported as far as available test files have show  as of R7.3 ?

I think it is too.


fix undefined symbol errors on macOS

This fixes errors like following:
[100%] Linking CXX shared module run_onnx_core.so
Undefined symbols for architecture x86_64:
  "_PyBaseObject_Type", referenced from:
      pybind11::detail::make_object_base_type(_typeobject*) in run_onnx_core.cc.o

I'm using Apple LLVM version 10.0.0 (clang-1<IP_ADDRESS>) on macOS Mojave.

Oh, thanks.
#761 works for me and I think it's more desirable fix.

Thanks for the check!


added automatic de-indexing with the withdraws reltype

https://github.com/dominictarr/secure-scuttlebutt/issues/42

I don't want to merge this yet, because it's possible to get inconsistent indexes if it crashes before the get completes. It would be better to leave a feature out, than to have a feature that works unreliably*,
what we need to do before we merge this is have a way to get reliable async indexes.
* This probably works nearly all the time, but that isn't good enough for a system that aspires to be as foundational as ssb.
That said, this would also have flow on effects to other datastructures that are represented across multiple messages... I think we need to aquire more experience with those (and possibly, represent deletes closer to the application layer...)
I'm gonna close this issue for now, but we should revisit this later.


Only runs with ES 5.4.0 not as stated in README with 5.4

When I use the most current 5.4. version (5.4.3) i got the error message that it only runs with 5.4.0 which is strange since 5.4.0 and 5.4.3 should be API compatible.
Please either fix in code or within the README file.
Steps to reproduce

create Dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.3

RUN elasticsearch-plugin install https://github.com/MLnick/elasticsearch-vector-scoring/releases/download/v5.4.0/elasticsearch-vector-scoring-5.4.0.zip

Build
docker build -t es_with_vector-scoring .

which results in following error
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM docker.elastic.co/elasticsearch/elasticsearch:5.4.3
 ---> 2ae8547160a7
Step 2/2 : RUN elasticsearch-plugin install https://github.com/MLnick/elasticsearch-vector-scoring/releases/download/v5.4.0/elasticsearch-vector-scoring-5.4.0.zip
 ---> Running in ca31562f69ae
-> Downloading https://github.com/MLnick/elasticsearch-vector-scoring/releases/download/v5.4.0/elasticsearch-vector-scoring-5.4.0.zip
[=================================================] 100%??
Exception in thread "main" java.lang.IllegalArgumentException: plugin [elasticsearch-vector-scoring] is incompatible with version [5.4.3]; was designed for version [5.4.0]
        at org.elasticsearch.plugins.PluginInfo.readFromProperties(PluginInfo.java:146)
        at org.elasticsearch.plugins.InstallPluginCommand.verify(InstallPluginCommand.java:428)
        at org.elasticsearch.plugins.InstallPluginCommand.install(InstallPluginCommand.java:495)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:215)
        at org.elasticsearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:199)
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
        at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
        at org.elasticsearch.cli.Command.main(Command.java:88)
        at org.elasticsearch.plugins.PluginCli.main(PluginCli.java:47)
The command '/bin/sh -c elasticsearch-plugin install https://github.com/MLnick/elasticsearch-vector-scoring/releases/download/v5.4.0/elasticsearch-vector-scoring-5.4.0.zip' returned a non-zero code: 1

Hi there - most of this plugin functionality is now supported natively in ES: https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html
Also see https://www.elastic.co/blog/text-similarity-search-with-vectors-in-elasticsearch.
See also an update to related Code Pattern on IBM Developer that illustrates a recommendation use case: IBM/elasticsearch-spark-recommender#54


bad links in javadoc

[ERROR] ...src\main\java\com\marklogic\client\expression\package-info.java:22: warning - Tag @link: reference not found: com.marklogic.row.RowManager

[ERROR] ...\src\main\java\com\marklogic\client\expression\package-info.java:22: warning - Tag @link: reference not found: com.marklogic.expression.PlanBuilder

[ERROR] ...\src\main\java\com\marklogic\client\row\package-info.java:23: warning - Tag @link: reference not found: com.marklogic.expression.PlanBuilder

The javadoc build is also behaving very strangely (on Windows) as of today. It went from taking virtually no time to sitting around contemplating its navel for 3 min. and horking up 3.2G of memory before finally deigning to generate the javadoc. I see no checkins that might explain this, but I've seen it on 2 hosts now. The JVM actually ran out of memory on the doc build machine. (Yes, I know we can give it more.)
I'm hoping (with little reason) that fixing the bad links might sooth the savage beast.

Yes, Erik added back in the type interfaces to the javadoc.  That's why it's taking longer and more memory.  Let us know if you're not able to add the extra memory to accommodate.

I cannot seem to give the JVM enough memory to make it succeed on our build machine. I'm sure it's pilot error, but I'm not familiar enough with the ins and outs to figure out what it is. Perhaps something will jump out at you.
I am doing the following, based on reading that that's how I can pass JVM options through Maven:
mvn -DjvmArgs="-Xmx4000m" -Dmaven.test.skip javadoc:javadoc
It is not maven itself that's running out of memory, btw, but javadoc:
 javadoc: error - java.lang.OutOfMemoryError: Please increase memory.
Perhaps this is not something I can fix outside of the pom?

I tried for myself and I also encountered OutOfMemoryError.  I added 3500m to the javadoc section in pom.xml and it resolved the issue for me so I committed the change: 25f6fa7a9b8df8d0d6f749ce3addce3f370cb724.  This also resolves part of #674.

Strange -- the JavaDoc builds without issue on my laptop.  Thanks for fixing it, Sam.  Kim, if the POM change that Sam made didn't fix it for you, please send the bug back to me.

It builds on mine without running out of memory, too, though it sure uses up a lot of it.
In any case, my polluting this bug with the memory issue aside, the @link issue is still real, so you're going to get this back tomorrow regardless. :)

Thanks, I'll clean up the @ link warnings

That ought to learn me.  I assumed it was bad syntax, but I had left out the .client.  So clever.
Should be good now.

Looks good. I see neither warnings nor out-of-memory errors now. Thank you both very much.


consecrating, dholes

This pull request was generated by the 'mq' tool
[test]
flake rate: 0.4
logical conflict every: 5
sleep for: 300s
[pullrequest]
requests per hour: 30
deps=[c,d]

/trunk merge


check if journald directory exists

Description of Changes
When journald input is given an invalid directory parameter, it will silently fail without error. When running journalctl manually, you get this
# journalctl -f --directory /fake
Failed to open /fake: No such file or directory

Because journalctl fails completely, we should check if the directory exists before attempting to start.
This change adds the check, and is only applied when the optional directory parameter is specified. When the directory exists, the agent starts normally. When the directory does not exist, stanza gives this error and fails to start:
{
  "level": "error",
  "timestamp": "2021-08-16T16:33:13.220Z",
  "message": "Failed to build agent",
  "error": "invalid value /fake for parameter directory, directory does not exist: stat /fake: no such file or directory"
}

Because journalctl fails instead of waiting for the directory to appear, I believe Stanza should as well.
Resolves https://github.com/observIQ/stanza/issues/378
Please check that the PR fulfills these requirements

[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)
[ ] Add a changelog entry (for non-trivial bug fixes / features)
[ ] CI passes

Log Files
Logs / Second
CPU Avg (%)
CPU Avg Δ (%)
Memory Avg (MB)
Memory Avg Δ (MB)

1
1000
1.6379576
+0.103465796
127.05846
-1.8001099

1
5000
5.3104043
+0.36204624
137.35938
-0.6030426

1
10000
11.172696
+0.3102455
147.9433
+5.6053467

1
50000
50.324192
-2.9001312
174.34738
-2.7846222

1
100000
98.727356
-1.618042
234.00189
-0.5430908

10
100
1.9483519
+1.8239021e-05
133.77007
-1.8204498

10
500
6.18964
-0.5001426
139.3385
-1.0134583

10
1000
12.275848
+0.120449066
148.4674
+2.2599792

10
5000
54.017822
-1.7080612
180.01482
+0.6058655

10
10000
108.12867
-0.6463547
219.05469
-13.366379


Caret compats

Current compats force a Requires version without https://github.com/JuliaPackaging/Requires.jl/pull/106, which is causing me lots of weird trouble in projects with RLEVector. It looks like looser compats here are harmless.

Can we have a new release with these compats?


WIP – Serve school overview JSON from endpoint

... instead of printing it all out on the page.
We'll see if this fixes our memory errors!

Pushed this work to the staging site. Pages appear to load much more slowly with the JSON endpoint route as opposed to the printing-JSON-on-page route. Possibly caused by a lack of precomputing on the staging site?

Get this error when running the precompute task in staging:
> PrecomputeStudentHashesJob.new(STDOUT).precompute_all!(Time.now)
Querying for jobs...
. . .
Found 6 jobs.
Starting precomputing...
ActiveRecord::StatementInvalid: PG::ProgramLimitExceeded: ERROR:  index row size 3568 exceeds maximum 2712 for index "index_precomputed_query_docs_on_key"
HINT:  Values larger than 1/3 of a buffer page cannot be indexed.

Hmm, the same task is working just fine on staging.


Is the default device id value uuid v4 correct?

Summary
I read the docs that Amplitude's device Id is generated as uuid V4 . docs, src/uuid.js
But i got device id like "uh0zoye4eSo3Ig8bSg1z6Z".
This is not UUID v4, isn't it? (UUID v4 have to be like "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx")
I don't know why that id value is returned. Can you help me?

Hi @grap3fruit, thanks for reaching out! Do you have a code snippet you can share on how amplitude is being used? Also which version are you using?

Hi @kevinpagtakhan. thanks for your help.
I'm using amplitude-js 8.15.1.
I used the code below in react. That's all.
useEffect(()=>{
  // 1. initilize amplitude
  amplitude.getInstance().init(process.env.NEXT_PUBLIC_AMPLITUDE_API_KEY);
  // 2. check deviceId
  console.log(amplitude.getInstance().options.deviceId); // => uh0zoye4eSo3Ig8bSg1z6Z
},[])

Hi @grap3fruit, it seems that docs are out of sync. The default value for deviceId is a randomly generated base 64 string: https://github.com/amplitude/Amplitude-JavaScript/blob/main/src/base64Id.js. It's not limited to a base 64 string though, you can still call setDeviceId() and pass with a uuid string. The only ask is the string is sufficiently unique.

Oh ok thanks. 👍


Send RAL instances

The PR proposes that all RAL instances are Send. Send-able Instances are feature-gated behind "rtfm", but it seems like a useful feature for non-RTFM users.
By enabling Send for RAL instances, most HAL peripherals will also become Send. We can then wrap HAL peripherals in Sync-able cortex_m::interrupt::Mutexes, or use them wherever else Send is necessary.
Send-able RAL instances would match what svd2rust outputs (sample of an svd2rust output). Both RAL and svd2rust instance types serve the same goal: dereference to a register block. It seems to follow that they may both be Send.
If accepted, we must re-generate of the RAL. Follow the guidance in CONTRIBUTING to re-generate the RAL. Alternatively, we may manually implement Send on peripherals in the HAL.

I like this change, and it makes sense to me


System requirements for t5-11b model

In descriptions of DeepSpeed I see that it allows training up to 13 bilions parameters model on a single GPU. What is the memory requirements were for this? I see information about a single V100 - was it v100 with 16 GB memory or 32 GB?
And a final question: is there a way to run google's t5-11b model on a GPU with 24 GB of memory (rtx3090) ? fp16 mode and cpu_offload turned on. Here is my config:
https://github.com/exelents/try_t5_qa/blob/main/ds_config.json

@exelents Thanks for using DeepSpeed.
13B parameter requires 32GB V100. Have you seen this tutorial?
The config looks fine for running in fp16 and cpu_offload. Can you please try it and share your experience?

@exelents Thanks for using DeepSpeed.
13B parameter requires 32GB V100. Have you seen this tutorial?
The config looks fine for running in fp16 and cpu_offload. Can you please try it and share your experience?

13B parameter requires 32GB V100

That's a pity. I tried to finetune T5 with 11B paramenters on 24GB RTX3090. Seems, it's not enough, because I got an Cuda OOM error.

13B parameter requires 32GB V100

That's a pity. I tried to finetune T5 with 11B paramenters on 24GB RTX3090. Seems, it's not enough, because I got an Cuda OOM error.

Yes, unfortunately, for a 11B model, just the model parameters will consume 22GB and there will nothing left for activations or gradients on 24GB GPU.

Yes, unfortunately, for a 11B model, just the model parameters will consume 22GB and there will nothing left for activations or gradients on 24GB GPU.


Create email notification when a new user is created

Done. Activation emails are sent in the post_save signal.


Additional props

[x] Card.shape #1533
[x] NavigationDestination.tooltip #1646
[x] NavigationRail.* #1644
[x] BottomSheet.* #1642

PR ready for merge.


Server erroneously reports it is only broker not dealer

When using authentication the server replies to clients with roles: broker but does not include the dealer role. Overriding clients to force them to make calls works fine, but most will not even attempt a call unless the server replies with dealer role.

@0Kills Could you provide the router code that reproduces this issue. I use the simple authentication example and the router is correctly responding with the roles:
  {
    "roles": {
      "broker": {
        "features": {
          "subscriber_blackwhite_listing": true,
          "publisher_exclusion": true,
          "subscriber_metaevents": true
        }
      },
      "dealer": {
        "features": {
          "caller_identification": true,
          "progressive_call_results": true
        }
      }
    }
  }

I believe this is corrected in the latest release by the WelcomeMessage priority fix. Please open a new issue if it is not.


[Breaking] Search parameters as first argument

Previously, I couldn't get autocomplete to work correctly for the other search params once engine was defined, i.e. { engine: "google", <autocomplete for other params> }. By setting engine as the first argument of getJson and getHtml instead, the type of search params (2nd arg) could be narrowed down to provide accurate autocomplete.
This PR explores having the search params as the first argument of getJson/getHtml while ensuring arbitrary engines/search params are accepted, autocomplete works and a way to have type safety is maintained.
Related to #5
Attempts
1st try
I re-looked into this issue and found that I needed another abstraction, EngineParameters:
export type EngineName = keyof EngineMap;
export type EngineParameters<
  E extends EngineName = EngineName,
> = {
  [K in E]: { engine: K } & EngineMap[K]["parameters"];
}[E];

...

export async function getJson<
  E extends EngineName = EngineName,
  P extends EngineParameters<E> = EngineParameters<E>,
>(
  parameters: P,
  callback?: (json: BaseResponse<E>) => void,
) { ... }

Factor
Works?

Does not show warnings for arbitrary engines
No (because EngineName must be keyof EngineMap)

Does not show warnings for arbitrary search params
Yes (because of getJson’s P extends EngineParameters<E>)

Autocomplete for engine
Yes

Autocomplete for search params
Yes

Type-checking for inline params
Yes

Types for safety
Yes via EngineParameters<"...">

2nd try
The problem still remained that an arbitrary engine shows a warning. Based on this TypeScript issue comment, there’s a way to support autocomplete whilst allowing for any string. I’ve adopted the concept here:
type AnyEngineName = string & {};
export type EngineName = (keyof EngineMap) | AnyEngineName;

EngineParameters was also modified to use the above.
export type EngineParameters<
  E extends EngineName = EngineName,
> = {
  [K in E]:
    & { engine: K }
    & (K extends keyof EngineMap ? EngineMap[K]["parameters"]
      : BaseParameters & Record<string, unknown>);
}[E];

What this does is that for all known engines (those in keyof EngineMap), the params are taken from EngineMap, while for unknown engines, params is anything that includes properties from BaseParameters.

Factor
Works?

Does not show warnings for arbitrary engines
Yes (because of AnyEngineName)

Does not show warnings for arbitrary search params
Yes (because of getJson’s P extends EngineParameters<E>)

Autocomplete for engine
No

Autocomplete for search params
Not quite (types are showing as unknown)

Type-checking for inline params
No

Types for safety
Yes via EngineParameters<"...">

However, this change causes autocomplete for engine and type-checking for inline params to not work. Also, search params autocompletions showed unknown as their type.
3rd try
To fix the issue with search params autocompletions not reflecting the correct type, removing the Record<string, unknown> in EngineParameters worked.
export type EngineParameters<
  E extends EngineName = EngineName,
> = {
  [K in E]:
    & { engine: K }
    & (K extends keyof EngineMap ? EngineMap[K]["parameters"] : BaseParameters);
}[E];

Factor
Works?

Does not show warnings for arbitrary engines
Yes (because of AnyEngineName)

Does not show warnings for arbitrary search params
Yes (because of getJson’s P extends EngineParameters<E>)

Autocomplete for engine
No

Autocomplete for search params
Yes

Type-checking for inline params
No

Types for safety
Yes via EngineParameters<"...">

Conclusion
I think what we have in the 3rd try is good enough. Users can supply any arbitrary engine/search params without seeing warnings, while still being able to receive hints as to what search params are supported and their types. For users who want more type safety, they can use EngineParameters<"..."> which only allows the explicitly defined parameters.
I can’t seem to find a way to get autocomplete for engine to work while taking care of all other factors. I think it’s an acceptable tradeoff as it’s still better to not show warnings for arbitrary engines. Perhaps we can revisit this in future.
Demos:
Autocomplete for search params, accepts arbitrary engines/search params
ts-params-first-arg.webm
Using EngineParameters<...> for type safety
ts-params-satisfies.webm

This is a breaking change, I suggest we release it as 2.0.0. Note that this is based on the work in #8 which should be merged first as a 1.x.x version.

Thanks @zyc9012, I've made the changes while maintaining autocomplete for both variants. There's not too much extra code which is good.
I tried function overloads, similar to how I did it previously, but the intellisense outputs aren't super helpful for users. I opted to use a union type for the ..args instead:
  ...args:
    | [parameters: P1, callback?: (json: BaseResponse<E>) => void]
    | [engine: string, parameters: P2, callback?: (json: BaseResponse<E>) => void]

I also made sure that engine is typed as a string rather than EngineName to ensure that parameters continue to have autocomplete,

while allowing arbitrary types (in this case start can be a string)

Please take a look and let me know if there's anything amiss! Thanks :)


Tool to visualize memory maps

It should be pretty straightforward to visualize addresses, such as by displaying increasing address location vs time. This could help de-fugly some of the really gross parsing routines, such as d8041ah

this is done https://github.com/SiliconAnalysis/zorrom/blob/master/vizlayout.py


Vue-Quill-Editor into an other component does'nt work

Vue-Quill-Editor into an other component does'nt work
Version Vue-Quill-Editor : 3.0.6
I use vue-quill-editor but it does'nt work into the component draggable from vue-draggable plugin.
It's possible to add a style (like bold), but it's not possible to remove it. Outside the draggable, it works properly.
There is a problem of passing an event on the vue-quill-editor outside a component.

I had this same issue, a fix was to use the handle option on sortableJs (vue-draggable): https://github.com/SortableJS/Vue.Draggable/issues/455

Did you manage to get this working? I'm having the exact same problem. Can't really get up and running with the provided docs link.


QUART_AUTH cookie missing.

When running the following locally (on Windows 10) with quart 0.18.4, quart-wtforms 1.0.0, and quart-auth 0.9.0, a QUART_AUTH cookie appears, but when running it remotely (on CentOS), the QUART_AUTH cookie isn't populated in the browser. The form loads, successfully submits, and after the line login_user(AuthUser(UserType.viewer.value)), I'd expect to have the cookie. I've swapped return redirect(url_for('home')) for return 123 and the same thing happens -- cookie is populated when running quart locally, but not remotely. I am not using a reverse-proxy or anything. Just hypercorn (hypercorn main:app -b <IP_ADDRESS>:8080 -w 2) and the dev server (py main.py).
The firewall is open on port 8080, and the page loads from the remote server. I've also set this up with Docker, and the same issue occurs.
@app.route("/login", methods=['GET', 'POST'])
async def login():
    form = await LoginForm.create_form()
    if await form.validate_on_submit():
        if form.username.data == UserType.viewer.value and compare_digest(form.password.data, os.getenv('PASSWORD_VIEWER')):
            login_user(AuthUser(UserType.viewer.value))
            return redirect(url_for('home'))
        else:
            form.form_errors.append('Incorrect credentials')
    return await render_template('login.html', form=form)

Any ideas as to why this is happening?

So, I wasn't using https. To get https working with hypercorn, I generate certs with
openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out cert.pem

and ran hypercorn with
sudo ./virt_env/bin/hypercorn --certfile cert.pem --keyfile key.pem -b <IP_ADDRESS>:8080 -w 1 main:app

See also docs on disabling secure cookies for dev


[BUG] Test TestStreamCompression.testPagedStreams fails

Environment
Operating System: FreeBSD 13
Version: / Commit SHA: 8.1.0
Other: clang-11
Describe the bug

[ RUN      ] TestStreamCompression.testPagedStreams
/disk-samsung/freebsd-ports/misc/openvdb/work/openvdb-8.1.0/openvdb/openvdb/unittest/TestStreamCompression.cc:422: Failure
Expected equality of these values:
  ss.tellp()
    Which is: 4452
  std::streampos(5400)
    Which is: 5400
/disk-samsung/freebsd-ports/misc/openvdb/work/openvdb-8.1.0/openvdb/openvdb/unittest/TestStreamCompression.cc:455: Failure
Expected equality of these values:
  ss.tellg()
    Which is: 4452
  std::streampos(5400)
    Which is: 5400
/disk-samsung/freebsd-ports/misc/openvdb/work/openvdb-8.1.0/openvdb/openvdb/unittest/TestStreamCompression.cc:548: Failure
Expected equality of these values:
  fileout.tellp()
    Which is: 42724
  std::streampos(51480)
    Which is: 51480
[  FAILED  ] TestStreamCompression.testPagedStreams (35 ms)

I think this error likely means that you're using a version of Blosc that we don't officially support (we recommend 1.5.0)? If you'd like to submit a patch to either update these values for FreeBSD or for a later version of Blosc, we'd gladly accept.

I use c-blosc-1.21.0 from FreeBSD ports.
Repology shows  that  c-blosc-1.5.0 isn't used anywhere any more: https://repology.org/project/c-blosc/versions
It seems to be a very old version. Github shows (https://github.com/Blosc/c-blosc) that blosc-1.5 was released in 2014.

@yurivict yeah this is a problem - VDB is locked to blosc 1.5.0  mainly due to incompatibility issues between various blosc versions: https://www.blosc.org/posts/new-forward-compat-policy/
The short of it is that we really need to investigate which version of blosc we can switch to safely without breaking the reader for files compressed with 1.5.0. On top of that, as you point out in this issue, newer versions of blosc are changing the compression ratios so we really need to check that these changes are legitimate. We need to come up with a compatible subset of blosc versions and fix the tests accordingly, but we need to be careful with this!

Just a note that as of OpenVDB 9 we've bumped our new proposed blosc minimum to 1.17.0. 1.17.0 and 1.18.0 pass our unit tests. I still get test failures with 1.19.0, 1.20.0 and 1.21.0 so this issue is still valid.


doc: drop the code flow page and topic branch references

Hi, counter proposal for https://github.com/zephyrproject-rtos/zephyr/pull/56580:

drop the references to topic branches in the getting started page, at this stage that's just confusing
drop the page describing how topic branch works entirely, it does not reflect any current workflow and have to be rethinked anyway

Topic branches are not in widespread use and the details on how are managed are being reworked and do not reflect the current documentation.
Drop the current "code flow" page that tries to define them (it's not linked anywhere anyway). And clear the reference to topic branches in the contributor guidelines.

DNM until this is finalised by the Process WG.

Process WG:

OK with removing mention of this type of branch from contribution guidelines
restore code flow document since we do have a recent TSC vote to approve the creation of such branches (currently termed topic branches)