Performance and sequencing characteristics of loader thread

Related to nodejs/node#44710 but I want to extract here so it can outlive that PR
It would be great to have written down the expected sequencing behavior of the loader thread.  I'm not sure if "sequencing" is the correct term, but I would like to understand how the thread's workload tasks run concurrently or sequentially.
Past revisions of the off-thread PR forced certain tasks to run sequentially, where you would otherwise assume they are happening concurrently.  Hopefully that is no longer the case.  I'm hoping to verify by asking this question.
Scenarios
sync request from one thread blocking sync request from another

Given a main thread, a user thread, and the ESM loader thread
the main thread makes a blocking import.meta.resolve calls to the loader thread which must communicate with some-remote-process://takes-100ms-to-load/package.json
the user thread makes a blocking import.meta.resolve calls to the loader thread which must communicate with some-remote-process://also-takes-100ms-to-load/package.json
Do those tasks execute concurrently within the esm loader thread?  Or sequentially?

When (future) main thread makes 20x synchronous require.resolve calls, can each run synchronously in the loader thread?  Or does the loader thread need to spin its event loop, handling a single request every tick?

today, require.resolve behavior can be tweaked in-thread, synchronously (fast)
will future require.resolve tweaks require spinning loader thread's event loop (slow) or can that also be done synchronously?

When loader thread is blocked by atomics because it has no work, can a message posted to globalPreload port wake it up?

I don’t think anything on that thread affects require.resolve; perhaps you meant to say import.meta.resolve there?
As far as I understand things, import.meta.resolve on the off-thread branch functions kind of like require.resolve or fs.readSync etc., where it synchronously blocks the main thread while it runs. So that sounds like it would be sequential.
I’m not sure what you mean by “the user thread;” normal Node apps have only one thread, and if --loader is used they’ll have two, but that’s it unless user code creates additional Worker threads.
For the rest of it, I think the best way to get to the bottom of these questions would be for you to add some tests to the off-thread branch. Cover the various scenarios you’re discussing here, and have the expected result of the test match what you think should happen, and if that isn’t what occurs then we can discuss.

Given a main thread, a user thread, and the ESM loader thread

I think the scenario cannot happen with the current PR, as the user thread would have its own ESM loader thread. It's still interesting to discuss the outcomes in case this is ever implemented (but I suppose would need to be opt-in).

the main thread makes a blocking import.meta.resolve calls to the loader thread which must communicate with some-remote-process://takes-100ms-to-load/package.json
the user thread makes a blocking import.meta.resolve calls to the loader thread which must communicate with some-remote-process://also-takes-100ms-to-load/package.json

Currently the worker assumes there can be at most one sync request at a time, so we would need to find some clever way to work around that.

When (future) main thread makes 20x synchronous require.resolve calls, can each run synchronously in the loader thread? Or does the loader thread need to spin its event loop, handling a single request every tick?

Currently we are using "message" events, so it would have the same limitation as with other events (I'm not an expert here, but I think if there are 20+ messages in the queue, there would all be handled in the same tick)

When loader thread is blocked by atomics because it has no work, can a message posted to globalPreload port wake it up?

That's an interesting use case, could you write a test for that?

I don’t think anything on that PR affects require.resolve; perhaps you meant to say import.meta.resolve there?

I'm thinking ahead to the future where CJS can be handled by loaders, too.  This is confusing, I'll try to update the OP to make this clearer.

I think the scenario cannot happen with the current PR, as the user thread would have its own ESM loader thread.

I see.  Is a future change going to collapse the multiple loader threads into a single loader thread, shared across all main and user threads?  What's the current plan?

Is a future change going to collapse the multiple loader threads into a single loader thread, shared across all main and user threads? What's the current plan?

Currently, you can specify whichever loaders for you worker thread, it doesn't inherit from the parent thread. This feature is not going away, as it seems a common enough use case to have a different loading mechanism for a Worker. That being said, we could have an opt-in option to share loaders, this only needs a champion I think.

Ok, I don't necessarily want it to be one way or the other, so long as the docs mention its behavior, whatever that is.

When loader thread is blocked by atomics because it has no work, can a message posted to globalPreload port wake it up?

That's an interesting use case, could you write a test for that?

The use-case is to avoid creating 2x copies of a transpiler and static analysis service.
This service is used to transpile both require()d CJS and import-ed ESM, and also provide static analysis diagnostics for all those files, where diagnostics in one file often require analyzing the source of other files.  This is how typechecking works.
require() can be hooked by require.extensions.  If we are also running an ESM loader, then we have a typechecker service running in the loader thread.  To avoid creating a duplicate typechecker in the main thread, we delegate all typechecking and transpilation to the ESM loader thread.  require.extensions hooks make blocking calls to the loader thread, requesting both transpiled source and diagnostic messages.  This requires atomics and globalPreload port messages.  These messages must wake up the loader thread.
I'm also curious if quibble needs this: https://github.com/nodejs/loaders/discussions/124#discussioncomment-4506736

When loader thread is blocked by atomics because it has no work, can a message posted to globalPreload port wake it up?

FYI the version that has landed is a bit different from the time we discussed this, now the worker never locks itself. Having a test would still be interesting I think, if someone wants to contribute that, that would be awesome.


[CI] Pin sqlite3 conan recipe version

The latest recipe requires Conan 1.51.3, which introduces this bug:
https://github.com/conan-io/conan/issues/11822
Hopefully this is a band-aid until 1.52 is out and we can create a proper conan lockfile.

@feltech Do you prefer the Revert "<commit>" approach for the commit title of the conan revert, or better to just retitle to [CI] Revert Conan version to 1.50.2?

Closing, as we decide to postpone this till it becomes a problem (it's only an issue when rebuilding, and the pre-built packages work on CI platforms).


Technology and Innovation Fellowship page is off...

I sent the T&I fellowship page to a @antislice and all the JS, CSS, and images are broken. :(
Current behavior

Expected behavior

It should look something like this. http://web.archive.org/web/20160308233312/http://www.consumerfinance.gov/jobs/technology-innovation-fellows/

Steps to replicate behavior

Visit http://www.consumerfinance.gov/jobs/technology-innovation-fellows/

This was fixed in #2547 and is now fixed on the live site. Thank you for reporting this @KimberlyMunoz!


adding save/load functionality to funcalign/srm

Adding save/load functionality to funcalign/srm to resolve issue #428

@me-sh, thank you for the PR! Please add tests. See the other SRM tests:
https://github.com/brainiak/brainiak/blob/master/tests/funcalign/test_srm.py
You can use the Pytest tmp_path fixture to obtain an output directory:
https://docs.pytest.org/en/latest/tmpdir.html#the-tmp-path-fixture

I made some minor adjustments and added a test of the SRM save/load functionality here: https://github.com/me-sh/brainiak/pull/1 (not sure how to commit changes more directly to this PR)

Thanks, @snastase! You should be able to push your branch to @me-sh's.
git remote add me-sh https://github.com/me-sh/brainiak
git fetch me-sh
git push me-sh

Because we are planning to make a release very soon, I will add @snastase's commits to this PR and merge it.

Thanks @snastase and @mihaic !

No, thank you, @me-sh! :)
And thanks for the quick help, @snastase.


Fail to encrypt using OpenMP, and a possible solution

When using encrypt inside a omp parallel for loop, the error ippMontExp: set Mont input error. is very likely to occur, especially when the plaintext is large, e.g., close to N.
Here is the simplest code to reproduce the problem.
int main()
{
	ipcl::KeyPair keyPair = ipcl::generateKeypair(2048);
	std::cout << "N=" << *keyPair.pub_key.getN() << std::endl;
	std::cout << "N^2=" << *keyPair.pub_key.getNSQ() << std::endl;
	std::vector<BigNumber> bignums;
	for (size_t i = 0; i < 100; i++)
	{
		bignums.push_back(BigNumber(uint32_t(i + 1)));
	}

	for (size_t t = 0; t < 100; t++)
	{
		std::cout << "Start to encrypt 100 BigNumbers ==== time " << t << std::endl;
		BigNumber N = *keyPair.pub_key.getN();
		BigNumber moded = N - bignums[t];
#pragma omp parallel for
		for (size_t i = 0; i < 100; i++)
		{
			keyPair.pub_key.encrypt(ipcl::PlainText(moded));
		}
		std::cout << "Encrypted 100 BigNumbers" << std::endl;
	}
}

The error seems to be caused by the ippsMod_BN function, which somehow modifies the data of modulus (the m_nquare field) during its computation. Hence it is not thread-safe.
However, I notice that copying m_square in raw_encrypt probably fixes the problem.
std::vector<BigNumber> PublicKey::raw_encrypt(const std::vector<BigNumber>& pt,
                                              bool make_secure) const {
  std::size_t pt_size = pt.size();
  BigNumber nsq = *m_nsquare; // This code copies the data of m_nsquare
  std::vector<BigNumber> ct(pt_size);

  for (std::size_t i = 0; i < pt_size; i++)
    // ct[i] = (*m_n * pt[i] + 1) % (*m_nsquare);
    ct[i] = (*m_n * pt[i] + 1) % nsq;

  if (make_secure) applyObfuscator(ct);

  return ct;
}

Hi @zfscgy
Thank you very much for raising this issue and pointing out improvement plans. This issue has been fixed in this PR #72
You can update the development branch to try it out. If you find any other issue, please feel free to raise them.

Thanks!


Remove some .NET specific logic from serialization/deserialization

There was some .NET specific logic where we were attempting to only store sanitized request bodies as JSON if the serialization would leave the bytes unchanged. If reserializing would have reformatted/re-encoded the body, we instead serialized as string. This could happen if characters such as & were included in the body. This behavior made recordings inconsistent and harder to read. Instead of doing this, we will always serialize as JSON, but use the STJ RelaxedJsonEncoding, to prevent excessive escaping of special characters.
The PR also always re-normalizes the sanitized request before matching using STJ RelaxedJsonEncoding, as well as when loading the requests from stored session records. This prevents matching issues that could arise from reformatting that occurs while sanitizing. Now we are always reformatting in a consistent way.
This change is an alternative to the new setting proposed in https://github.com/Azure/azure-sdk-tools/pull/2427

Let's add a test that ensures that re-normalization works when matching and loading.

Can you also check if this solves the number formatting problem and add a test if it does?

Do we need to update content-length values after normalization, especially during the record loading?


p2p: Improve peer clock diff detection

The clock diff detection now takes into account not only when the handshake message was sent but also when a reply was received. That way, we get a more precise idea regarding peer's time and can act accordingly.
Not exactly sure how to unit-test this, it needs a way to simulate a delay in peer's reply. Should be doable from functional tests but sadly don't have the time to do it now.

The code is fine, but the algorithm... I'm not sure this is what we would use to minimize the damage from latency.
As far as I understand, the issue previously was that we're taking the handshake timestamp starting point, call it A, and learning the difference in time (with the max diff tolerance) by reading the timestamp the peer sent to us, call it B. Now |A-B| does not account for network latency, and hence it contains the network latency as an error in the measurement. So there's no way to know the relation between A and the true point in time the peer recorded timestamp B.
My understanding of the improvement solution would be to register local time at two points, once when we send the handshake and when we receive the response. Say the two points are A1 and A2. Now assuming the bidirectional latency is exactly equal (which is not absolutely true in real life, but again, probably a good approximation), and that computation time is negligible, we can assume that sending the handshake and receiving a response takes the same time, then the best approximation of the time instant, at which the peer recorded the timestamp and responded, would be the average of A1 and A2, hence (A1+A2)/2. In other words, this approximation makes the timestamp at which the peer recorded B, be equivalent to (A1+A2)/2, as much as possible, by minimizing network latency effects.
However, what I'm seeing in the code currently doesn't attempt to minimize the error using that logic. It tries, on the other hand, to be more strict in ensuring that timestamp of the peer is not too far from the time recorded when receiving the message. I may be wrong, but this does not really contribute to minimizing the error caused by network latency.
Please let me know if I'm misunderstanding something. Perhaps there's another angle I'm not seeing.

I pushed a commit with my solution based on my explanation. No pressure. It can also be considered just for preview. Let's discuss it.

The code is fine, but the algorithm... I'm not sure this is what we would use to minimize the damage from latency.
As far as I understand, the issue previously was that we're taking the handshake timestamp starting point, call it A, and learning the difference in time (with the max diff tolerance) by reading the timestamp the peer sent to us, call it B. Now |A-B| does not account for network latency, and hence it contains the network latency as an error in the measurement. So there's no way to know the relation between A and the true point in time the peer recorded timestamp B.
My understanding of the improvement solution would be to register local time at two points, once when we send the handshake and when we receive the response. Say the two points are A1 and A2. Now assuming the bidirectional latency is exactly equal (which is not absolutely true in real life, but again, probably a good approximation), and that computation time is negligible, we can assume that sending the handshake and receiving a response takes the same time, then the best approximation of the time instant, at which the peer recorded the timestamp and responded, would be the average of A1 and A2, hence (A1+A2)/2. In other words, this approximation makes the timestamp at which the peer recorded B, be equivalent to (A1+A2)/2, as much as possible, by minimizing network latency effects.
However, what I'm seeing in the code currently doesn't attempt to minimize the error using that logic. It tries, on the other hand, to be more strict in ensuring that timestamp of the peer is not too far from the time recorded when receiving the message. I may be wrong, but this does not really contribute to minimizing the error caused by network latency.
Please let me know if I'm misunderstanding something. Perhaps there's another angle I'm not seeing.

The idea was to take the latency into account in a bit more robust way, not relying on the assumption that the peer sampled the time in the middle of the time interval between sending the handshake message and receiving the reply. That way, assuming the peer is not making up the timestamp, we know that the peer is max 10s off (given 10s time offset limit).
When treating the midpoint as the peer's sample time, we can get out of this range. E.g. if the peer takes 4s to reply, we remain connected to the peer even if it may be up to 12s off with the 10s max time offset setting. So if nothing else, it makes the semantics of the max time offset setting make more sense.
I'm not saying my approach is better, there may be facets of the problem I'm missing. However, I think the response time probably should be taken into account somehow rather than just assuming the turnaround times are symmetric.

So I managed to find a way to test the timestamp negotiation, including networking delays (response timeout) albeit not particularly elegant.


Fixing dynamic reconfig

Solved errors include:

warning from accessing locked parameters
not having min/max exposure/gain for auto exposure mode available

Fix for #23


EF 6.2.0 migrate.exe ERROR: Object reference not set to an instance of an object

I'm setting up migrate.exe to run DB migration during VSTS release:
migrate.exe DataAccess.dll /connectionProviderName="System.Data.SqlClient" /connectionString="Data Source=SQLXXX\DEV01;Initial Catalog=XXXXX;Integrated Security=true;" /verbose
Output:
VERBOSE: Target database is: 'XXXXX' (DataSource: SQLXXX\DEV01, Provider: System.Data.SqlClient, Origin: Explicit).
No pending explicit migrations.
Running Seed method.
System.Data.Entity.Migrations.Design.ToolingException: Object reference not set to an instance of an object.
   at System.Data.Entity.Migrations.Design.ToolingFacade.Run(BaseRunner runner)
   at System.Data.Entity.Migrations.Console.Program.Run()
   at System.Data.Entity.Migrations.Console.Program.Main(String[] args)
ERROR: Object reference not set to an instance of an object.

We have no code in Seed method. No pending explicit migrations is expected but we need the script to finish without an error for the release process to continue. How to fix this error?
Same question on Stack Overflow

To make it work had to provide /startupConfigurationFile instead of /connectionString.


CLI accepts any value as authentication type

Describe the bug
The CLI asks for a text input for the authentication type and accepts any value. The generation succeeds, but the resulting application can't be compiled.

Expected behavior
Let the user select an authentication type like in the base generator:

Added a bug bounty to get this fixed.

@mraible
Cool, thanks! I'm working on it.


Unable to use in client side rendering

Hi,
have you used this in single page app what renders the map on client side?
On the second time when I render the map (for instance visiting in other page and coming back), I get following error and the map tiles don't show up at all:

Have to refresh the page to get it work again.
I manage to get it working without errors properly by doing following changes, but I'm not familiar enough with the codebase to understand is there some other concequences with these changes:
		GoogleMapsLoader.load(function (_google) {
			google = _google;
			console.log('I just loaded google maps api');
			self._ready = true; // <---- ADDED THIS!
			self._initMapObject();
			self._update();

			//this._ready = google.maps.Map !== undefined;
			//if (!this._ready) L.Google.asyncWait.push(this);
		});

	_initMapObject: function () {
		if (!this._ready || !this._container) return; // <--- ADDED _container check!

		this._google_center = new google.maps.LatLng(0, 0);
		var map = new google.maps.Map(this._container, {
			center: this._google_center,

         // CODE CONTINUES....

Any help?

Hi, I know this error but I didn't have time to find a solution to this, since my map was loaded successfully.  I will check out and I will release a new version when I will found a little bit time. If the above code works for now use it, I will come back as soon as possible.

I can confirm that OP's solution fixes the issue. Thank you, @ernoaapa for this walkthrough, and thank you, @Charmatzis for at great job on this very needed component.

I have fixed this error, as @ernoaapa suggested.
THX, @ernoaapa


Bug: (AuthorizationCode.exchangeAuthCodeForTokens)

Prerequisites

[X] I have searched the repository’s issues and Kinde community to ensure my issue isn’t a duplicate
[X] I have read the contributing guidelines
[X] I agree to the terms within the code of conduct

Describe the issue
nuxt-kinde-auth-sample % npm run dev

dev
nuxt dev

Nuxt 3.7.4 with Nitro 2.6.3                                                                                9:20:34 PM
9:20:34 PM
➜ Local:    http://localhost:3000/
➜ Network:  use --host to expose
➜ DevTools: press Shift + Option + D in the browser (v1.0.0)                                                                                9:20:35 PM
ℹ Vite client warmed up in 533ms                                                                                9:20:36 PM
✔ Nitro built in 279 ms                                                                                nitro 9:20:36 PM
[nuxt] [request error] [unhandled] [500] Authentication flow state not found
at AuthorizationCode.exchangeAuthCodeForTokens (./node_modules/@kinde-oss/kinde-typescript-sdk/dist/sdk/oauth2-flows/AuthorizationCode.js:75:19)
at async AuthorizationCode.handleRedirectFromAuthDomain (./node_modules/@kinde-oss/kinde-typescript-sdk/dist/sdk/oauth2-flows/AuthCodeAbstract.js:36:24)
at async Object.handleRedirectToApp (./node_modules/@kinde-oss/kinde-typescript-sdk/dist/sdk/clients/server/authorization-code.js:55:9)
at  (./node_modules/@nuxtjs/kinde/dist/runtime/server/api/callback.get.mjs:6:3)
at async Object.handler (./node_modules/h3/dist/index.mjs:1630:19)
at async Server.toNodeHandle (./node_modules/h3/dist/index.mjs:1840:7)
Starter kit URL
https://github.com/kinde-starter-kits/nuxt-starter-kit
Operating system(s)
macOS
Operating system version(s)
MacOS Sonoma 14.0
Further environment details
Nuxt 3.7.4 with Nitro 2.6.3
Reproducible test case URL
http://localhost:3000/api/callback?code=QbaIcMAOCYO8LVZJhSsyhRPuOuzWE4k_vrf3hnLlH9w.PaBCglsfpnAwHIJAzgZopB73-eB7RTDpP9-39HGbvQI&scope=openid profile email offline&state=21bf5d2a7964214db226f26105a1743f90ab369a8c878f70c0559af4
Additional information
No response

Hey @malwaremanu thanks for raising this one.
I haven't been able to reproduce it as yet, from the error message I'm assuming the state query parameter has gone out of sync.
Silly question but I assume all the environment variables are set correctly either via .env or runtime config?
Is this happening in all browsers?

Yes it is happening in Firefox and Safari Both.
So, Here's my .env file.
NUXT_KINDE_CLIENT_ID="client_id copied from portal..."
NUXT_KINDE_CLIENT_SECRET="secret copied from portal..."
NUXT_KINDE_AUTH_DOMAIN="http://xyz.kinde.com"
NUXT_KINDE_REDIRECT_URL="http://localhost:3000/api/callback"
NUXT_KINDE_LOGOUT_REDIRECT_URL="http://localhost:3000"
NUXT_KINDE_POST_LOGIN_REDIRECT_URL="http://localhost:3000/dashboard"
The moment i am logged in from http://xyz.kinde.com, it is redirecting me to this url
http://localhost:3000/api/callback?code=5tpK4VE7HgffqU6szKHYEjJDGrmAjIeG3j3gtt-gLhQ.Ud7KmhpYlL58nDJXe-gh_nKZbCdcNHcRnw34xFe87_U&scope=openid profile email offline&state=9e4ab3fa75e47797d7229b4ea609bd78e28e85d4a92320e6e6408a97
which gives me this json response.
{
"url": "/api/callback?code=5tpK4VE7HgffqU6szKHYEjJDGrmAjIeG3j3gtt-gLhQ.Ud7KmhpYlL58nDJXe-gh_nKZbCdcNHcRnw34xFe87_U&scope=openid%20profile%20email%20offline&state=9e4ab3fa75e47797d7229b4ea609bd78e28e85d4a92320e6e6408a97",
"statusCode": 500,
"statusMessage": "",
"message": "fetch failed",
"stack": "at Object.fetch (node:internal/deps/undici/undici:13220:11)\nat process.processTicksAndRejections (node:internal/process/task_queues:95:5)\nat async AuthorizationCode.fetchTokensFor (./node_modules/@kinde-oss/kinde-typescript-sdk/dist/sdk/oauth2-flows/AuthCodeAbstract.js:136:26)\nat async AuthorizationCode.exchangeAuthCodeForTokens (./node_modules/@kinde-oss/kinde-typescript-sdk/dist/sdk/oauth2-flows/AuthorizationCode.js:85:20)\nat async AuthorizationCode.handleRedirectFromAuthDomain (./node_modules/@kinde-oss/kinde-typescript-sdk/dist/sdk/oauth2-flows/AuthCodeAbstract.js:36:24)\nat async Object.handleRedirectToApp (./node_modules/@kinde-oss/kinde-typescript-sdk/dist/sdk/clients/server/authorization-code.js:55:9)\nat  (./node_modules/@nuxtjs/kinde/dist/runtime/server/api/callback.get.mjs:6:3)\nat async Object.handler (./node_modules/h3/dist/index.mjs:1630:19)\nat async Server.toNodeHandle (./node_modules/h3/dist/index.mjs:1840:7)"
}
I can share me repo, if you can spare sometime.
I may be doing some silly mistake, still i would like to learn something new.
and BTW, kinde.com is the easiest to implement so far.
I have done it using fastapi python. Just struggling on frontend part.
Thanks,
Manupal

So,
just to check if i am missing something, i again cloned the repo, changed .env.example to .env and pasted my env data.
and then logged in again.
still i got a json response,
{"url":"/api/auth/callback?code=1tjaZZOL1x6lUaLfZrLrZYlj0NvvZpzWoKEBF4r9NJE._U3fzHA7hBCSj7js6czmFckcroFZ2EoI7C4EyQ6vsjA&scope=openid profile email offline&state=538616164f2d3b46d27e90c596b96e70ac3be461a1a4b96cd8928d78","statusCode":404,"statusMessage":"Page not found: /api/auth/callback?code=1tjaZZOL1x6lUaLfZrLrZYlj0NvvZpzWoKEBF4r9NJE._U3fzHA7hBCSj7js6czmFckcroFZ2EoI7C4EyQ6vsjA&scope=openid+profile+email+offline&state=538616164f2d3b46d27e90c596b96e70ac3be461a1a4b96cd8928d78","message":"Page not found: /api/auth/callback?code=1tjaZZOL1x6lUaLfZrLrZYlj0NvvZpzWoKEBF4r9NJE._U3fzHA7hBCSj7js6czmFckcroFZ2EoI7C4EyQ6vsjA&scope=openid+profile+email+offline&state=538616164f2d3b46d27e90c596b96e70ac3be461a1a4b96cd8928d78","stack":""}
I hope, i am doing my best to express an error.

Hi @malwaremanu, there have been a number of enhancements to the Nuxt Module over the last few weeks, are you able to give it another go with the latest module version and see if you're still experiencing any issues?


[Question] How to make private vaults for per authenticated user?

Hi @soarecostin,
I would like to know which approach is the best for my question?
Thanks in advance.
Cheers.

You may use the key($key) function in the facade to set an encryption key.
You can ask the user password and use pbkdf2 to generate a safe hash to use as key : https://www.php.net/manual/en/function.hash-pbkdf2.php

@leonekmi but with this technique if the user change his password you have to re-encrypt all the files, no ?

I would love also to know if with this technique it's required to re-encrypt all files when the user changes its password.

I would love also to know if with this technique it's required to re-encrypt all files when the user changes its password.

Sure, but you can probably create a background task after the user changes his password. Assuming you ask the old one for stealth checks, you can decrypt the data and encrypt it again with a new key (aka the new password).

Sure, but you can probably create a background task after the user changes his password. Assuming you ask the old one for stealth checks, you can decrypt the data and encrypt it again with a new key (aka the new password).

Right but if the user own big amount of files, that could be a heavy task.
Why not use a hash of a user data that won't change, as user uuid for example ;)

Right but if the user own big amount of files, that could be a heavy task.
Why not use a hash of a user data that won't change, as user uuid for example ;)

It kinda defeats the purpose of having an encrypted vault if the uuid or any other "non changing informations" is stored into your database "as is", why encrypt it then?
It's only useful then if you don't trust the external provider for your data (S3 for example), but it would be misleading to say to your users that data is encrypted, because if you have the keys available with a simple request as an admin, the data is not really encrypted in a way that would actually be protecting user data.

It kinda defeats the purpose of having an encrypted vault if the uuid or any other "non changing informations" is stored into your database "as is", why encrypt it then?
It's only useful then if you don't trust the external provider for your data (S3 for example), but it would be misleading to say to your users that data is encrypted, because if you have the keys available with a simple request as an admin, the data is not really encrypted in a way that would actually be protecting user data.

What about generating a random key to encrypt the data, and then use pbkdf2 to store that key? That way if the user changes the password, we only have to re-encrypt the stored key again with pdbkdf2, not all the data. Would that be a good aproach (from a security point of view)?

What about generating a random key to encrypt the data, and then use pbkdf2 to store that key? That way if the user changes the password, we only have to re-encrypt the stored key again with pdbkdf2, not all the data. Would that be a good aproach (from a security point of view)?

Hi,
I come back on that question because it's my dev time to shield my app.
After skimming the crypto and security forums about per user encryption, I realised that my first proposition (with user id) was far from reality ^^
@underdpt your is a good choice I think, the one well described there https://security.stackexchange.com/questions/157422/store-encrypted-user-data-in-database
The user and only can decrypt his encrypted data.
The only thing blocking for me now is if the user loses his password, he loses the key too, no way...
I have to let the user access a forget password process in my app.
A solution could be to also pbkdf2 store the initial random key with another user passphrase or question/answer asked at user registration like the password.
My app could ask the passphrase to allow user to reset his password if he forgot it and get the encryption key.
Would that be a lack of security ?
Any idea ?


setMorphClass on MorphToMany relation class

Laravel Version: 5.6
PHP Version: 7.2

Description:
I'm building my application on top of laravel orchid package, and they have the Orchid\Platform\Core\Models\Post class that they use internally in their core to save posts, add categories and tags, etc. When saving tags, the taggable type in database is being set to Orchid\Platform\Core\Models\Post.
I also my own post class App\Post that inherits from the base post class.
The problem occurs when I try to get tags on a post (App\Post):
$post->tags; // nothing is returned

that's obviously because in the database the namespace of a parent post is being stored.
I can resolve this issue by setting the getMorphClass on my App\Post class:
    public function getMorphClass()
    {
        return parent::class;
    }

but in case I would want to create some new polymorphic relations on my App\Post class, things would break.
Suggestion
Add a setMorphClass on a MorphToMany relation so that for some polymorphic relations it is possible to use parent class.
Example of usage:
class Post extends BasePost
{
    public function tags()
    {
         return $this->morphToMany(static::$tagsModel, 'taggable', 'tagged', 'taggable_id', 'tag_id')
                ->setMorphClass(parent::class)
         ;
    }
}

Please let me know if there is a way already to accomplish this.
Otherwise I believe this would be a useful feature to have.
Thanks.

@tillkruss
Can I please know why this issue was closed?
Is it already resolved?
Thanks :)

You didn't read the text when opening a new issue. This is not a place to ask for help or suggest new features.

Ah, my bad. Sorry about that.
Thanks


Fix code editor search match inconsistencies

Closes #68650
Fixes code editor's search match inconsistencies which display an incorrect number of matches while filtering for “Whole Words”, with the following:

Continue searching line for whole word matches on mismatch
Using searched text length to offset

Results of custom editor build with fixes using provided example project of #68650 :

Code looks good, can't think of anything that could go wrong.

I would suggest squashing the commits, the two changes are very closely related and we don't go in such granularity usually. See PR workflow for instructions.

@akien-mga done, I've squashed the commits 👍

Thanks! And congrats for your first merged Godot contribution :tada:


Support for NodePort services

Services with type=NodePort should have all node IPs that match the nodeSelector defined in DNS. Depends on https://github.com/kubernetes-incubator/external-dns/issues/239.

Follow up with https://github.com/kubernetes-incubator/external-dns/issues/49#issuecomment-308137512 once this is merged.

Can we close this issue as #559 was merged?

This was done thanks to @grimmy. Will be released tomorrow as v0.5.3.

@linki Thanks for shepherding this last year. Do you know if there are doc's anywhere about how this is supposed to work? The main page still says Support for Services with type=NodePort is yet to be defined and I haven't found any docs about it.
Also haven't observed any DNS Zone changes being made when adding NodePort services so wondering if it needs to be configured.


Update README.md

spring code moved

@SychevIgor thanks for the PR.
@mrm9084 can you please review? The sample links are broken too. Not sure what they should be updated to.

spring-cloud-azure-feature-management and spring-cloud-azure-feature-management-web also need to be updated. They are in the same folder in the SDK repo as the other App Config Spring Libraries.

@mrm9084 @zhenlan links on feature management updated, but where microsoft moved samples- it's unknown.

@SychevIgor Here are the sample links
https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/spring/azure-spring-boot-samples/azure-spring-cloud-appconfiguration-sample
https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/spring/azure-spring-boot-samples/azure-spring-cloud-feature-management-sample
https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/spring/azure-spring-boot-samples/azure-spring-cloud-feature-management-web-sample

@mrm9084 included links on samples

@mrm9084 ping. (I know that it's time of Ignite, but still)


Check In

As User I want to check in to my drains so that I can report that I have cleaned them.
Minimally, I can use the honor system to click a button that records the check in.
Ideally, I can upload before and after photos.
See https://github.com/Code4HR/adopt-a-drain for Check In Feature
Checkin Model
Pull Request

This will be closed with a pull request from the Code4HR repo


P4_16: Move text on reading uninit vals and writing to invalid headers

I moved this text into its own section, and added cross-references to
it from where the text was moved from, Section 8.14 "Operations on
headers", and another in Section 8.13 "Operations on structs", even
though it really does apply to reading uninitialized of any types.
Motivation for this movement: The moved text specifially describes the
behavior when an uninitialized value of type enum or error is
read, even though a header is not allowed to contain fields with those
types.  With the text in Section 8.14, one could be easily be confused
into believing that it implied that a header could contain fields of
those types.
The only changes I made to the moved text are as follows:

Changed the first sentence to "The result of reading a field" to
"The result of reading a value", since reading uninitialized values
applies not only to fields of headers and structs, but also to local
variables with types like bit<W> or bool.

Added a few more explicit examples of situations where an
uninitialized value can be read, changing this text: "reading any
other value that has not been initialized, e.g. a field from a
struct." to this: "reading any other value that has not been
initialized, e.g. a field from a struct, any uninitialized
variable inside of an action or control, or an out parameter
of a control or action you have called, which was not assigned a
value during the execution of that control or action (this list
of examples is not intended to be exhaustive)."

Added this paragraph at the end of the new section:
"For a top level parser or control in an architecture, it is up
to that architecture to specify whether control parameters with
direction in or inout are initialized when the control is
called, and under what conditions they are initialized, and if so,
what their values will be."

Intended to address issue https://github.com/p4lang/p4-spec/issues/726

Do you think we should review this (quickly) at a LDWG? I'm tempted to classify it as just a superficial re-organization of the existing text, but there are a few non-trivial changes.

It seems perfectly reasonable for anything but minor typo fixes to go through the LDWG, to me, so I don't mind at all if it is brought up there.

Since we have 2 reviews I will merge this as well.


RBAC support via AWS IAM roles

Feature name
RBAC support via AWS IAM roles
Describe the feature you'd like to see implemented
As a customer I would like to control who can view what workload within AWS Perspective depending e.g. on AWS IAM roles so that not everybody has view on everything.
Describe the value this feature will add to AWS Perspective
More fine granular control of permissions to view workloads within AWS Perspective

This is something that's come up with other customers too, the fine grained permissions this would allow would definitely be very useful.

We've evaluated this request and added it to our backlog and has been prioritised on the roadmap for the solution.


[BugFix] Restore primary key table cause meta inconsistent when restart BE(#30129)

Problem:
This problem is introduced by commit cfba8973f4718c0a4f2d6846d2bdaa473ba0be49 In this pr, we create a new tablet meta with snapshot tablet_schema to avoid the crash if the backup table has been done schema change.
But the problem is that, the _updates in new tablet meta is nullptr. It forget to reset it to tablet->updates(). So when remove expired versions for primary key table, tablet_meta.save_meta() will save the wrong meta data for this tablet. When we restart be, it will cause the meta inconsistent problem.
Solution:
reset the tablet_schema directly for the tablet_meta
Fixes #30129
What type of PR is this:

[x] BugFix
[ ] Feature
[ ] Enhancement
[ ] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Does this PR entail a change in behavior?

[ ] Yes, this PR will result in a change in behavior.
[x] No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

[ ] Interface/UI changes: syntax, type conversion, expression evaluation, display information
[ ] Parameter changes: default values, similar parameters but with different default values
[ ] Policy changes: use new policy to replace old one, functionality automatically enabled
[ ] Feature removed
[ ] Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

[x] I have added test cases for my bug fix or my new feature
[ ] This pr needs user documentation (for new or modified features or behaviors)

[ ] I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

[x] I have checked the version labels which the pr will be auto-backported to the target branch

[ ] 3.1
[ ] 3.0
[ ] 2.5
[ ] 2.4

[FE Incremental Coverage Report]
:heart_eyes: pass : 0 / 0 (0%)

[BE Incremental Coverage Report]
:disappointed: fail : 11 / 16 (68.75%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
src/runtime/snapshot_loader.cpp
0
5
00.00%
[538, 541, 543, 544, 548]

:large_blue_circle:
src/storage/tablet_meta.cpp
11
11
100.00%
[]


fix: stream instruction output in chunks of 3Mb

Description:
Previously we would try to send all of it at once as the max message size is inifnite. User would get something like -
Caused by: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (17391758 vs. 4194304)

Barnabas ran into it. I ran into it several times mostly with execs.
Is this change user facing?
YES

@tedim52 this might be something we want to take in consideration for your instruction "output streaming" feature you're working on.

@gbouv this is still in progress. need to fix a few new lines. Will tag you now but review when I ping y ou!


About aggregation and interpolation

hello, zhijian,

i want to do a  aggregation and interpolation in sparse voxel,
there are my code:

# aggregate + upsampling
def aggregate(x:SparseTensor,  voxel_size:int):
    z = PointTensor(x.F, x.C.float())

    x0 = initial_voxelize(z, 1, voxel_size)          # voxelization

    z0 = voxel_to_point(x0, z, nearest=True)   # voxel to point

    x1 = SparseTensor(z0.F, x.C, 1)

    print(x1.C.equal(x.C))
    print(x1.F.equal(x.F))
    return x1

in x1 = SparseTensor(z0.F, x.C, 1), i use x.C and z0.F to construct output x1, did I do something wrong?

in this case, the kernel == voxel_size and stride == voxel_size, could I understand in this way?

thanks a lot! : )

Hi @suyunzzz, I don't fully understand your need. Could you please elaborate a bit more?

Hi @suyunzzz, I don't fully understand your need (what the goal is for your operator). Could you please elaborate a bit more?

Hello, zhijian,
In the MinkowskiEngine, there are some LocalPooling function, i.e. MinkowskiAvgPooling, not global_avg_pool, i want to know how to implement it(local_avg_pool) in torchsparse.
it just used to a multi-scale aggregate module ^^,like this:

i am not sure i expline clearly, thanks a lot

I haven't checked the details of your implementation, but I think you are on the right track.
From my understanding, you might need to pass x.s to reconstruct the sparse tensor rather than 1.

Adding @Xiuyu-Li and @kentangSJTU to this thread. We might need to support upsampling (nearest / trilinear) and downsampling in the future.

I haven't checked the details of your implementation, but I think you are on the right track.
From my understanding, you might need to pass x.s to reconstruct the sparse tensor rather than 1.

Because it just aggregate on input resolution, so i pass 1.
i will check again, thanks a lot

As i understand "nearest" mean use the nearest point interpolation .you should use trilinear interpolation . But I need help  to implement linear interpolation.

As i understand the "nearest" mean use the nearest interpolation ,you should ues trilinear interpolation.But i need help for set value for  "get_kernel_offsets" function's  stride . Or maybe I misunderstood?

Hi, @suyunzzz @zhijian-liu , sorry to bother you. I am looking for the torchsparse counterpart for MinkowskiInterpolation(). Any update on this issue? Or any possible suggestions for me to investigate will be greatly appreciated, thank you.

@HaFred, thanks for your interest! Unfortunately, we haven't been able to look into this issue yet.


When creating a new project on disk, files in sub directories are missing

Describe the bug
Originally reported by someone else on Discord. I went ahead and tested it myself and they are indeed missing.
Link to the blitz that caused the error
https://stackblitz.com/edit/node-xfumvg
It's a freshly generated Nodejs server using express-generator.
To Reproduce
Steps to reproduce the behavior:

Go to https://stackblitz.com/local
Click on the "Start a new Node project with your local files" button
Pick your directory/files. Make sure you have a file within two parent directories
Once the upload is done, you'll notice the two folders but no file within the sub folder. Interestingly enough, sub sub folders (folders within sub folders) get uploaded. And files within sub sub folders are also nonexistent.

Expected behavior
I expected all my files and directories to be uploaded.
Desktop (please complete the following information):

OS: Ubuntu 20.04
Browser Chrome
Version 90.0.4430.72 (Official Build) (64-bit)

This is an issue indeed!
The local fs integration is very alpha atm. Will be fixed asap!
🙏 for reporting.

@R4meau what was the extension of the files in those subfolders?
There was a bug where .ts files were not uploaded properly. I'm thinking it's gonna be related to that since I can't repro this issue with .js files.

@KwintenP It was all kinds of files.

This should be fixed now :tada: Thanks for the report.


DNS+dialup report incorrect

I'm getting the following values in a default report (with the latest build) when testing my site:
Detailed Report:

	DNS+dialup:
  		Average:<PHONE_NUMBER>.0216 secs
  		Fastest:	0.1668 secs
  		Slowest:<PHONE_NUMBER>.8548 secs

	DNS-lookup:
  		Average:	0.0041 secs
  		Fastest:	0.0000 secs
  		Slowest:	0.0821 secs

	Request Write:
  		Average:	0.0001 secs
  		Fastest:	0.0000 secs
  		Slowest:	0.0003 secs

	Response Wait:
  		Average:	0.0820 secs
  		Fastest:	0.0256 secs
  		Slowest:	0.1665 secs

	Response Read:
  		Average:	0.0033 secs
  		Fastest:	0.0002 secs
  		Slowest:	0.0219 secs

I believe the DNS+dialup math is wrong

That doesn't seem right, can you share the parameters of the command?
Are you able to reproduce the same result ?

$ hey -more -h2 https://alexbilbie.com/
Result:
All requests done.

Summary:
  Total:	0.3446 secs
  Slowest:	0.3185 secs
  Fastest:	0.0217 secs
  Average:	0.1578 secs
  Requests/sec:	290.1685

Detailed Report:

	DNS+dialup:
  		Average:<PHONE_NUMBER>.5014 secs
  		Fastest:	0.1317 secs
  		Slowest:<PHONE_NUMBER>.8548 secs

	DNS-lookup:
  		Average:	0.0281 secs
  		Fastest:	0.0000 secs
  		Slowest:	0.0582 secs

	Request Write:
  		Average:	0.0001 secs
  		Fastest:	0.0000 secs
  		Slowest:	0.0011 secs

	Response Wait:
  		Average:	0.0637 secs
  		Fastest:	0.0171 secs
  		Slowest:	0.1065 secs

	Response Read:
  		Average:	0.0200 secs
  		Fastest:	0.0003 secs
  		Slowest:	0.0723 secs

Status code distribution:
  [200]	100 responses

Response time histogram:
  0.022 [1]	|∎∎
  0.051 [0]	|
  0.081 [1]	|∎∎
  0.111 [21]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.140 [18]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.170 [20]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.200 [15]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.229 [14]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.259 [8]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.289 [1]	|∎∎
  0.318 [1]	|∎∎

Latency distribution:
  10% in 0.0959 secs
  25% in 0.1134 secs
  50% in 0.1576 secs
  75% in 0.1980 secs
  90% in 0.2317 secs
  95% in 0.2428 secs
  99% in 0.3185 secs

After some testing it's when the -h2 flag is passed into the command

Confirmed
Problem with HTTP/2 and Httptrace, working on it.
Thanks for reporting

HTTP/2 transport and connection is being reused. Only GetConn (httptrace) is never called except for the first request. To fix this we can either use new transport for each request or simply check if the GetConn was called.
@rakyll
Any suggestions?

I was wondering whether we should show average for DNS latency from this tool. On a separate issue, we were thinking about warming up the request to avoid DNS latency and totally avoid displaying DNS stats.
@JRonak, I don't think we should use a new transport. Each transport will require its own connection pool and we won't be able to reuse the idle connections. On average, every HTTP client has support for pooling, the current stats are probably more realistic.

Actually, this issue isn't directly related to DNS latency which we discussed earlier. I personally feel we should give the user an option if they would like to skip DNS.
And you are right about the transport, I will open a PR to fix this issue by adding the time elapsed only if a new connection is established.

Seems that this is still an ongoing issue. Did anything ever happen with a pull request?
liam:$ hey -n 10000 -c 200 -h2 -more https://liam.sh
67 requests done.
[... truncated ...]
All requests done.

Summary:
  Total:	22.8757 secs
  Slowest:	15.7546 secs
  Fastest:	0.0381 secs
  Average:	0.2960 secs
  Requests/sec:	430.3700

Detailed Report:

	DNS+dialup:
  		Average:<PHONE_NUMBER>.0840 secs
  		Fastest:	0.1292 secs
  		Slowest:<PHONE_NUMBER>.8548 secs

	DNS-lookup:
  		Average:	0.0002 secs
  		Fastest:	0.0000 secs
  		Slowest:	0.0119 secs

	Request Write:
  		Average:	0.0000 secs
  		Fastest:	0.0000 secs
  		Slowest:	0.0035 secs

	Response Wait:
  		Average:	0.2051 secs
  		Fastest:	0.0362 secs
  		Slowest:	2.2112 secs

	Response Read:
  		Average:	0.0242 secs
  		Fastest:	0.0001 secs
  		Slowest:	0.8178 secs

Status code distribution:
  [200]	9845 responses

Response time histogram:
  0.038 [1]	|
  1.610 [9740]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  3.181 [18]	|
  4.753 [52]	|
  6.325 [0]	|
  7.896 [18]	|
  9.468 [6]	|
  11.040 [0]	|
  12.611 [0]	|
  14.183 [0]	|
  15.755 [10]	|

Latency distribution:
  10% in 0.0603 secs
  25% in 0.1028 secs
  50% in 0.1740 secs
  75% in 0.3338 secs
  90% in 0.5137 secs
  95% in 0.6661 secs
  99% in 1.6881 secs

Error distribution:
  [148]	Get https://liam.sh: unexpected EOF
  [7]	Get https://liam.sh: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

@lrstanley, AFAIK, no one is working on a fix right now.

@alexbilbie @lrstanley Can you check with latest hey + go versions? I can't reproduce the issue.

Details (average, fastest, slowest):
  DNS+dialup:<PHONE_NUMBER>.8257 secs, 0.1265 secs,<PHONE_NUMBER>.8548 secs
  DNS-lookup:	 0.0001 secs, 0.0000 secs, 0.0178 secs
  req write:	 0.0002 secs, 0.0000 secs, 0.0304 secs
  resp wait:	 0.0269 secs, 0.0200 secs, 0.2676 secs
  resp read:	 0.0004 secs, 0.0000 secs, 0.0100 secs

Status code distribution:
  [200]	1013 responses
  [503]	8857 responses

Error distribution:
  [130]	Get https://liam.sh: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Specifically:
DNS+dialup:<PHONE_NUMBER>.8257 secs, 0.1265 secs,<PHONE_NUMBER>.8548 secs

I found the reason.  Because hey reuses connections, GetConn was only called on the first request, and  for others connStart was never set  (and statyed at initial value 0). I added a check and if connections are reused we now do not calculate connection time (This is not ideal, but better than the erroneous values).
@lrstanley can you verify?

Confirmed that DNS+dialup now reports correctly as of b633e22873b9e2c543774978051ff2fec713ab84 per my testing.
This issue can now be closed. Thanks!


[compiler service] Support test and fixture hooks

https://testcafe.io/documentation/402831/guides/basic-guides/organize-tests#fixture-hooks
https://testcafe.io/documentation/402831/guides/basic-guides/organize-tests#test-hooks

https://github.com/DevExpress/testcafe/commit/31f478fec9aeafa570ee92f250eac952dcc6f015


Congress update

Here's what I ran, for reference:
cd /usr/local/aclu/elections-api/sources
rm -rf congress_legislators
make congress_legislators
cd ..
python scripts/data_congress_legislators.py

I also double-checked that the updates from https://github.com/aclu-national/elections-api/pull/20 are still reflected here.
Because this adds 3 new records, we should remember to run make aclu_ids for those new IDs to get indexed.

FYI I added one more commit to this PR before merging. Here is what I ran:
python scripts/source_congress_photos_bioguide.py
python scripts/source_congress_photos_wikipedia.py
./scripts/data_congress_photos.sh


CORS problem

I can not make post call, I get this error all the time:

Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

This is my code:
       superagent             .post(url, body)             .use(tokenPlugin)             .withCredentials()             .then(responseBody),
I have CORS enabled on my ASP.NET Core Web API:
        app.UseCors(x => x                 .AllowAnyOrigin()                 .AllowAnyMethod()                 .AllowAnyHeader()                 .AllowCredentials());

This looks like a message from your browser, and if the browser says CORS is not sent by the server correctly, then it's not sent by the server correctly. Inspect the headers.

I have this problem too, can u show me how to solve it ? Thank


[CELEBORN-610][FLINK] Eliminate pluginconf and merge its content to CelebornConf.

What changes were proposed in this pull request?
Pluginconf might be hard to understand why Celeborn needs to config class.
Why are the changes needed?
Ditto.
Does this PR introduce any user-facing change?
NO.
How was this patch tested?
UT.

Shall we categorize the Flink-specific configurations added here for easier reference? Do you have any ideas? @waitinfuture @pan3793 @AngersZhuuuu

Shall we categorize the Flink-specific configurations added here for easier reference? Do you have any ideas? @waitinfuture @pan3793 @AngersZhuuuu

+1 for these, we can have 3 categories for client conf
client-common client-spark client-flink

I am inclined to distinguish them by the configuration name.
celeborn.client.
celeborn.client.spark.
celeborn.cleint.flink.

the configuration "doc" is too rough
buildConf("celeborn.client.flink.inputGate.memory")
      .doc("Memory reserved for a input gate. For Flink plugin only.")
      ...

For example, the above "doc" does not help, users can not gain more information from "doc" statement than the configuration key itself. "input gate" is a new concept, it should be explained briefly at least in one of the keys' doc.


 For example, in DeltaOptions:
src/org/tukaani/xz/DeltaOptions.java:46: warning: [this-escape]
possible 'this' escape before subclass is fully initialized
    [javac]         setDistance(distance);
    [javac]                    ^

I fixed one this-escape warning by making a the internal class IndexDecoder final. I wonder what is the best way to fix the public classes: XZOutputStream, DeltaOptions, and LZMA2Options.
The fix to XZOutputStream in this PR feels fairly good already in sense that I don't really see any alternatives.
The filter options classes could be fixed in a few ways:

Make them final like currently in this PR. This is simple to do but will break things if someone has extended the options classes at all.

Make many (all?) public method in the options classes final. This will break less as the classes can still be extended if the functions aren't overridden. But, for example, someone could in theory have created a custom override that prevents setting a too high LZMA2 dictionary size. It sounds unlikely though.

Add private methods that can be called from the constructors and public methods. That is, make the public methods trivial wrappers for the private methods. This should break backward compatibility even less. It's still not perfect if there was a subclass with a constructor that relied on the current (somewhat broken) behavior.
A downside is that this makes the code uglier. I would like to avoid this solution. It is OK in XZOutputStream but LZMA2Options would need many such functions.

Add static functions to do the argument verification, for example, void validateDistance(int distance). Use these in the constructors and public methods. The actual assignments of the validated values would be duplicated (done in both constructors and in the public setter methods).
This makes the code even uglier than in (3).

I suppose any fix is at least a slight API/ABI breakage. I think even with (1) very few users of the package would be affected though.
@bokken: I would like to solve this fairly quickly (a few days) to get the 1.10 release out.

I pushed all four variants of the filter options change ideas. The version 3 is the latest one (without a number).
4 is indeed too ugly. I suppose it's about choosing between 1 and 3, assuming that this can be changed at all.
The benefit of 1 is that overriding can be done so that it messes up internals of the package. For example, if LZMA2Options.getPb returned 5 it would certainly break the encoder.
Once this is decided, I think 1.10 is ready. There is a draft of NEWS.md in the news_draft branch.

I think I lean towards making the classes final.
Extending these classes and overriding methods allows for too many bad/non-sensical things to happen. For example, several of the "setter" methods enforce valid values - but one could simply extend the class and override the getter to return anything.

Hi @Larhzu
Just to help a little with the warnings (I hope I don't upset anyone or defend them)...
The warning on line 46 is something that the Java compiler frequently throws when an object calls a public method in its constructor, indicating that this method may be overridden in some inherited subclade and that the behavior may change.
There are several ways to solve this warning, although setting the class as 'final' solves it, you can also do the following trick:
public class DeltaOptions extends FilterOptions {
    ...
    public DeltaOptions(int distance) throws UnsupportedOptionsException {
        DeltaOptions.this.setDistance(distance);
    }
    ...
}

In this way we avoid creating unnecessary methods to encapsulate other public methods, which will only be called in the constructor.
The key is in this line of code:
Object.this.method(args)

@bokken:

I think I lean towards making the classes final.
Extending these classes and overriding methods allows for too many bad/non-sensical things to happen. Which is the same thing you point out.

I cannot find code on the web that extends the options classes. So the likelyhood of breakage is pretty low if they are made final.

And I struggle to find a reasonable/useful thing to do with subclasses.

The main thing I can imagine is overriding setDictSize or setPreset to limit the settings to some value lower than the default maximums. Perhaps there could be some debugging use cases if getOutputStream or getInputStream can be overridden but that's getting very specific and then one perhaps can compile a custom version of the code too.
I tried to check what other situtations there are where public functions are called from other functions within the class:

In output streams, close calls finish. This perhaps could be documented in the API docs.

XZOutputStream.endBlock() is called in finish(). This too could be documented.

SeekableXZInputStream.seekToBlock calls getBlockPos but there is no guarantee that it will do so in the future too. So I don't want to put that in the API docs. Perhaps such a call should be avoided.

I read that there is a recommendation that one should either design a class to be extensible or make it final. I get why but on the other hand sensible extensions are low risk, and preventing them feels a bit silly (it could even be really annoying in some situations).

@JNightRider:

DeltaOptions.this.setDistance(distance);

I tried it and it makes no difference. It doesn't fix the warning. If setDistance is overridden, the method in a subclass will be called. Sorry if I'm missing something.

@bokken:

I think I lean towards making the classes final.

I have done this. I hope the master branch is now about ready for 1.10. I updated the news_draft branch as well.
Earlier releases in Maven Central don't include debugging information. I noticed that at least some packages do though. I don't remember anyone requesting it but I wonder if it should be added. Building xz.jar with OpenJDK 22:

Size
Options

126 KiB
-g:none

147 KiB
-g:source,lines

165 KiB
-g or -g:source,lines,vars

Thoughts?

@JNightRider:

Just to help a little with the warnings (I hope I don't upset anyone)...

I forgot to say that it's great that more people comment and give feedback. Thanks!

Earlier releases in Maven Central don't include debugging information. I noticed that at least some packages do though. I don't remember anyone requesting it but I wonder if it should be added.

I have observed that including the debug info is more and more common (pretty sure this is because of the maven-compiler-plugin default behavior of -g).

Ah, so it hasn't always been so common. I suppose nowadays package sizes matter less and less. Having the debug info available without extra steps likely is convenient. I did enable debug info in 1.10 binaries in Maven Central.
While it's unclear if Ant will be used in this project a year from now, I played around a bit and got reproducible builds working. I used OpenJDK 21 for the binaries because that has long support, which makes it easier for people to verify the 1.10 binaries a year or two later.
I read that there's work going on in Maven plugins to make more builds reproducible. I feel it's a good goal. It seems that OpenJDK's compiler output doesn't often change outside of major releases, which helps a lot.


[Draft] Clean build: Only integrate module dependencies once

Not sure if this change is safe, or if it needs to go further to impact incremental builds as well.
Currently this is shaving close to 2.5 minutes off of a clean build for a project with >150 modules and ~350 swift files in the main target.
Issue before was this was reading and integrating the module dependency information (M) for each input swift file (N), so (N*M), now it's only happening M times.
It needs more testing, but there is probably an incremental build situation where dependency information is being read and integrated N * M times where M is modules that have changed since last build.

@swift-ci test

Please don't merge this yet. I'm in the midst of analysis and I suspect it may be introducing a bug when incremental imports is disabled. Hope to know for sure very soon!

Closing in favor of #750


Merge origin/develop into main

Description
Preparations for version 1.8.0
Screenshot (if UI-related)
To-Dos

[ ] Successful build yarn build
[ ] Translation keys yarn i18n:extract
[ ] Database migration (if required)

Issues Fixed or Closed

Fixes #XXXX

:tada: This PR is included in version 1.8.0 :tada:
The release is available on:

v1.8.0
GitHub release

Your semantic-release bot :package::rocket:


[GameTest] [Bug] GameTests sometimes fail to run on servers and realms

Describe the bug
Seemingly randomly, valid GameTests will simply not run on a server/realm. This happens consistently on a random basis, and doesn't happen to everyone. The GameTests have been updated to respect the 1.19 change of using the type value of script.
To Reproduce
It seems finicky. Personally have been unable to reproduce, and only have the notes from attempts at doing so. There's also a bug report on Mojira, which seems more targeted towards realms.

Most reports have been using Ubuntu (assuming realms also are based off that distribution). Would make sense given the past disparities, but my tests succeeded on Ubuntu.
Double-modules (data + script types) didn't affect the outcome, nor did multiple packs.
Other experimental toggles didn't appear to cause an effect.
World settings, different scripts under the same UUID/world, and conversions between 1.18.30 and 1.19 appeared to fail.

My current guess is the machine. Maybe the OS version (For example: Focal Fossa vs Jammy Jellyfish)? Or maybe hardware, like CPU and GPU? Pure randomness seems unlikely, and it appears that realms are consistently affected, suggesting something to do with the machine
Update 6 June, 2022 12:27 CDT - I've now tested Ubuntu 18.04, 21.10, and 22.04 (in addition to 20.04), all of which I have been unable to reproduce. Not sure of the reason that this happens
Expected behavior
The GameTests are supposed to run their script, but do not. No errors appear to be thrown.
Desktop (please complete the following information):

Server: BDS/Realms
Version: Release <IP_ADDRESS>, BDS <IP_ADDRESS>

Additional context
There's an opened bug report about this on Mojira, which contains steps to reproduce specifically for realms: REALMS-10333

Based on further information gathered, it appears this is more user error regarding mojang-net. Closing this issue with some final thoughts

Realms won't work with the mojang-net module, due to security concerns
Server owners need to enable this module in their permissions.json file, as it's disabled by default
Some that don't use mojang-net also seem to have pasted every GameTest dependency, so if the module isn't a key part, remove it


AWS Lambda custom runtime

A simple custom runtime for nodejs with pnp.

This was completed with the yarn bundle command


I wrote an implementation

GoRx

We have a branch (rx-interfaces) where we are rewriting (refactoring) RxGo, I invite you to take a look at the path we are following and I count (very) with your and all community contribution

@langhuihui that looks great. We are making an overhaul of RxGo and could use your help / contributions.

@langhuihui is there a way to check out this work and to contribute to it as the moment?

what can I do for you?

Track the work that is being done in RxGo 2 (new version with interface support)
https://github.com/ReactiveX/RxGo/milestone/1
roadmap: https://github.com/ReactiveX/RxGo/issues/99#issuecomment-443816758


Use normalizedContextUrl

Description
If we can, use the normalized context URL we get now.

It was discussed normalizedContextUrl will be removed so it shouldn't be used at all, instead we should use clone url  and git status, ref

Thanks @jeanp413 🙏


Feedback: Using Zapier in the introductory tutorial notebook is a bad UX

I am really excited to use this project, I love the design decisions and if all goes well I would like to be a regular user and contributor.
I know in the tutorial notebook it says zapier is optional, but I would really like to learn as much as I can so I thought I would try, and I have never had such a frustrating experience trying to get an API key. I think they have changed their UI as all the bots are wrong, perplexity is wrong, your chatbot is wrong, all the instructions I can find are wrong. There is no option to create an app, their is a section that says apps, but when you create new you can create several things, none of which is an app. And yes I am using the developer platform. The default option to create a zap from the developer platform is an AI workflow that tries to use AI to build your zap, so using a value like "Agency Swarm Integration" as suggested by the Swarm Agent Chatbot doesnt work. There is an option to switch to manually create Zap, but its not even save-able until I configure at least one trigger and 1 action. The NLA Instructions say to connect your account to a custom action, but it is not usable until you select an available existing action from their list, I cannot find any way around it.
I have used a LOT of API's from a lot of sites, gotten a lot of api keys, never had much trouble until zapier, and I know it says the option can be skipped but I think putting something like this which can easily lead to frustration in the introductory tutorial which has a high risk of making people frustrated before they can even try agency swarm, and the problem has nothing to do with agency swarm.
I am not trying to insist that something be changed for my benefit, just offering a suggestion as I think not using the zapier integration in the introductory tutorial and maybe making it a separate example would be a better user experience and create a better assurance that new users have a good experience when first trying out agency swarm.
Thank you!

For clarity the link to the introductory tutorial notebook I am using is from the video, not on this repo. The link provided is https://colab.research.google.com/drive/1qGVyK-vIoxZD0dMrMVqCxCsgL1euMLKj#scrollTo=sY8FY_UwLKy_&uniqifier=2
Thanks!

Thanks for feedback @afewell. I totally agree with you. This notebook was created a long time ago, and needs an update. Do you have any suggestions for which tools to replace them with?

Get zapier key here : https://nla.zapier.com/credentials/

Wow Thanks for such a quick response ... honestly I wasnt expecting such a rapid response and I'd hoped to already have some updates to commit. I am still finding my way around but I will plan to make a PR with some revised suggestions for the notebook soon. If you would prefer to close this please feel free or if you want to leave it open I will reference it in my PR when I am able to propose an update.
Seeing such a supportive community here is extremely encouraging! Thanks again so much!

Im going to go ahead and close this as the feedback has been received, additional guidance for finding the zapier key was provided (thanks arnavsaxena17), although even with the API key it did not work for me, it seems like additional setup of zapier may be required. In any case I know Arsenii mentioned he has hired someone for documentation, I also created a introductory notebook and tutorial, so I think its fine to close this issue. Thanks!


[Suggestion] Skiplagged / pokemongomap map usage

I think it would be perfect if you could use those maps as background.
Best way to catch good pokemon and level up is new york in my experience, and skiplagged has a nice and updated map :)
Or let it use pokemongomap desktop so you can scan the area your in :)
Is this possible ? :)

Hi @Ezoteric im sorry for late respon, if you means a snipe its already at now.
I dont know how skiplagged doing with their api, if you have link with any document from their api i'll read it.

skiplagged, was implement for new release..
but its not for human walking snipe.. if you need more information just create new issue


Fix SDL_PATH in CMakeLists.txt

Use PATH instead of FILEPATH for setting SDL folder.
Fix VMA compile error by explicitly using C++17

Thanks for the fix


How to add RuntimeWiring while creating GraphQLSchema object using GraphQLSchemaGenerator class in SPQR V0.9.8?

This is how we do in graphql-java project. How can it be achieved via SPQR?
static GraphQLSchema buildSchema() {
String sdlSpec = "" +
        "type Query {\n" +
        "    dateField : String @dateFormat \n" +
        "}";

TypeDefinitionRegistry registry = new SchemaParser().parse(sdlSpec);

RuntimeWiring runtimeWiring = RuntimeWiring.newRuntimeWiring()
        .directive("dateFormat", new DateFormatting())
        .build();

return new SchemaGenerator().makeExecutableSchema(registry, runtimeWiring);

}

Please don't open duplicate issues. Merged with #167 .

Hi,
I am just trying to find different ways of achieving my goal. Adding RuntimeWiring is one of the way which I found (and it is not only used to create custom directives, there are other purposes as well). It is not my intention to create duplicate issues, if you feel so, apologies. Anyhow, thanks a lot for your response.

RuntimeWiring is used to wire behavior into a schema defined in SDL. There's no SDL in SPQR, and all behavior is already defined in the Java code you map.
You can customize various aspects, like type mapping, using custom TypeMappers, value conversion, using custom Input/OutputConverters etc.

If your use-case is better suited for starting from an SDL-defined schema, then using graphql-java directly is probably the best option.


socket.recovered is not always true

We have an intermittent test failure around our connection recovery after adding the redis streams adapter to our socket.io system.
The test procedure is:

Connect a socket to our server
Trigger a reconnection via socket.io.engine.close() on the client
Wait for the reconnection to occur
Check that socket.recovered is true

Step 3 reliably occurs, but Step 4 fails approximately 30% of the time. No errors seem to occur in the socket library, it seemingly just doesn't recognize the reconnection as a recovery (or that recovery silently fails). There were no failures using the default in-memory adapter.

Hi! I think you should not expect the connection state recovery feature to always succeed, as it won't be the case in production. Successful recovery must really be seen as the happy path.
That being said, there might be two explanations for the recovery failure:

either the reconnection is too fast (in that case, the server may not have stored the session context in Redis yet)
or the client is disconnected without having received a first packet (which is necessary to init the offset on the client side)

Ah, yeah the test was triggering a reconnect rather quickly - adding a delay seems to have removed the flakiness! Thank you for the suggestion


Use withCredentials=false and xdomain could be removed

like here https://github.com/wolframkriesing/es6-react-workshop/blob/master/src/_external-deps/http-get.js#L7

is this a "good first bug"?
as I see, one has to replace the usage of atomic with the node built-ins - http and url as is used in the linked file.
I know many things will probably break in its wake ( I hope there are tests to catch them ).
if that can be done successfully, can we remove atomicjs as a dependency and the use of the xdomain script in index.html?
is that all there is to get this done?


Add helper function for checking if a channel is open

Similary as for #3466, create an IsOpen function in the Channel type that does the open check:
if !channel.IsOpen() {
  return errorsmod.Wrapf(channeltypes.ErrInvalidChannelState, "expected %s, got %s", channeltypes.Open, channel.State)
}

func (channel Channel) IsOpen() bool {
  return channel.State == OPEN
}

And use it everywhere where we check if the channel is open or not.

reminder comment: needs to be changed in 04-channel branch too.


Setting initial datetime does not work in react-flatpickr

Using react-flatpickr, try setting up an initial value containing date and time like 05/14/2019 14:25.
Use exactly the same config in the pickr:
{
dateFormat: 'm/d/Y H:i',
format: 'm/d/Y H:i',
time_24hr: true,
enableTime: true
}

Flatpickr does not pick up and display the initial date at all. It works only when passing date string only like 05/14/2019.

Hi @nevendyulgerov, that does sound like a bug, but I'd like to 100% confirm it. Would you mind creating a JS fiddle showcasing the problem so we can verify and the take a decision accordingly? You can fork this fiddle: https://jsfiddle.net/tj2Lo4z7/

Hey @viorelsfetea this is reproducible against the react wrapper only. When you try the same thing against a bare-boned flatpickr, everything works as expected: https://jsfiddle.net/0ymg71h5/.

Unfortunately, that means that you need to open an issue downstream on the repo of the react wrapper: https://github.com/haoxins/react-flatpickr . Sorry we couldn't help you more


error with repo under a project naming convention during config transfer

Describe the bug
Problem Description: During the config transfer phase a of a migration it fails with a 400 error. See example below:
"errors": [
{

  "status": 400,

  "message": "The repository key 'andrewr-test' should start with the project key and a dash: 'andrewr-te-'\n"

}

]
Current behavior
fails with 400 error
Reproduction steps
Set up a self hosted and SaaS instance and configure them for migration. As described here.
In the self hosted instance create a repository. Give it a name that does not include a projkect name(ie andrewr-test)
In self hosted create a project. Assign the repository you created to the project.
Run the config transfer. Should see the error message mentioned above.
Expected behavior
The JFrog CLI continues to support migrating repositories, even if they belong to a project. If migrating Projects to the Cloud is unsupported, we should instead update our documentation to describe this major limitation up front.
JFrog CLI version
2.44, 2.45
Operating system type and version
mac, windows
JFrog Artifactory version
on prem 7.47.5, saas 7.64.7
JFrog Xray version
No response

Iam also getting the similar error while doing this activity. any idea how its resolved for you.
om-prem version is 7.59.9 on windows

@rajani-git-maker - this problem should be fixed in JFrog CLI version 2.4.6 through https://github.com/jfrog/jfrog-cli-core/pull/906.
What version of JFrog CLI are you currently using?

Thanks. issue been resolved after upgrading the jfrog cli to 2.46.0 version or latest.

@yahavi : how to pass the access token because its throwing 401 props authentication failed : i used access token as token id - gives me error then tried passing api key still error.
what is the value to be used while adding jf c add source-server -> access token arrow or username/password or api key token because any 1 can only be used.

@rajani-git-maker
The token needs to be an admin token. Please take a look at Generating Admin Tokens to learn how to generate the necessary admin token. Remember to use this token as access token, and make sure that the platform URL is accurate as well.

@yahavi : I did all correctly but still its giving error : while doing the jf c add source-server, for username/password or apikey set with same OR given the access token - in both scenarios its throwing error that jf rt transfer-config is not safe to do with admin/password warning throwing ireespective of how stronger token or apikey used.
e.g. Make sure that you're using strong credentials in your source platform (for example - having the default admin:password credentials isn't recommended).


Added notification messages.

Coverage decreased (-0.3%) to 65.799% when pulling 82989cc5844454af6ab1c82ac0dd9c9a577dac8d on notification into 72d132ed9d40dccef8fea4b16786bdb9e0794f6a on master.

Coverage decreased (-0.3%) to 65.799% when pulling 82989cc5844454af6ab1c82ac0dd9c9a577dac8d on notification into 72d132ed9d40dccef8fea4b16786bdb9e0794f6a on master.


What are the correct formats for hashes in ResourceSync

The ResourceSync specification says very little about the formats of the hashes but in table 3 says that hash is defined by Atom Link Extensions. This spec then defines the hashes to be simple hex representations. So, for a zero-length file we would have:
hash="md5:d41d8cd98f00b204e9800998ecf8427e
      sha-1:da39a3ee5e6b4b0d3255bfef95601890afd80709
      sha-256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

Now, in the spec we have some entries which are not valid according to this format:
cat resourcesync.html | perl -ne '$l++; print "$l: $_" if (/hash="(md5|sha-1|sha-256):([^"]+)"/ and $2!~/^[0-9a-f]+$/)'
2967:              hash="sha-256:f4OxZX_x_DFGFDgghgdfb6rtSx-iosjf6735432nklj"
2972:              hash="sha-256:y66dER_t_HWEIKpesdkeb7rtSc-ippjf9823742opld"
2979:              hash="sha-256:f4OxZX_x_FO5LcGBSKHWXfwtSx-j1ncoSt3SABJtkGk"
2984:              hash="sha-256:h986gT_t_87HTkjHYE76G558hY-jdfgy76t55sadJUYT"

but presumably use another digest presentation. However, the current implementation in this code creates MD5 sums using the format of RFC2616, Sec. 14.15 which was specified in version 0.1 of ResourceSync, i.e:
hash="md5:1B2M2Y8AsgTpgAmY7PhCfg==
      sha-1:da39a3ee5e6b4b0d3255bfef95601890afd80709
      sha-256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

So, appears like the actions are:

fix code here to calculate MD5 correctly
address issue in spec

Code issue fixed by #26.

Now produces output for md5 as expected:
> mkdir x
> touch x/a
> resync --hash=md5 --hash=sha-1 --hash=sha-256 --resourcelist http://x/ x
<?xml version='1.0' encoding='UTF-8'?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:rs="http://www.openarchives.org/rs/terms/">
<rs:md at="2017-03-20T23:13:10.422681Z" capability="resourcelist" completed="2017-03-20T23:13:10.423589Z" />
<url><loc>http://x/a</loc><lastmod>2017-03-20T23:12:30Z</lastmod><rs:md hash="md5:d41d8cd98f00b204e9800998ecf8427e sha-1:da39a3ee5e6b4b0d3255bfef95601890afd80709 sha-256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" length="0" /></url>
</urlset>

Wrote note about spec on ResourceSync group: https://groups.google.com/forum/#!topic/resourcesync/jpOHcT2vMYo


Multiple languages (i.e. spanish, german) versions of the  prf directives

Is there any possibility of having several versions of the prf directives?  In jupyter-book, with the sphinxcontrib_proof we can set a global variable "locale=es" and get the titles of this directives to change from rendering "Theorem (...) " to rendering "Teorema (...)"
as well as in the referencing part: See Teorema 1

This is still on the list for needed enhancements, so not yet possible. Here's the tracking issue for this topic overall:

https://github.com/jupyter-book/mystmd/issues/166

Want to head over there and voice support?

thank you for the reply, I wasn't able to find any related issue.


Add sorting

people
batting
pitching
fielding
fieldingOF
fieldingOFSplit
salaries
appearances

Batting

ID
playerID
yearID
stint
teamID
team_ID
lgID
G
G_batting
AB
R
H
2B
3B
HR
RBI
SB
CS
BB
SO
IBB
HBP
SH
SF
GIDP

Pitching

ID
playerID
yearID
stint
teamID
team_ID
lgID
W
L
G
GS
CG
SHO
SV
IPouts
H
ER
HR
BB
SO
BAOpp
ERA
IBB
WP
HBP
BK
BFP
GF
R
SH
SF
GIDP


Investigate PR Assistant bot use in our repos

1ES supports an AI-backed PR reviewer under the Merlinbot that should be useful for catching security and other issues earlier in the development cycle. The feature is enabled at the repository level (e.g., see https://github.com/1ES-microsoft/GitOps.Defaults/blob/main/app-primitives/pull-request-copilot-activation.yml#L912 for the start of DevDiv's repo list) but can be customized in a number of ways. For example, we may want to enable the bot only for internal/release/* branches in our mirror repos.
Features include the above-mentioned PR reviews as well as description updates and responding to Copilot commands. PR Assistant docs provides the full docs.

Release Note Category

[ ] Feature changes/additions
[ ] Bug fixes
[x] Internal Infrastructure Improvements

Release Note Description
Enabled the PR Assistant bot in some internal DNCEng repositories.

Looks like it's running on our internal-only repos, so we're considering this done. :)


Set solution

Don't merge this, just getting it in and getting an account registration of JS code ;)

I took the liberty of moving your changes to solutions/parhaml.js. I'm good with it if you are.

Doh. Thanks @ericwburden


Docker usage and fix on issue #39

Hello,
as indicated here are changes for

issue #39
docker compatibility (configurable config directory, example docker file)

Hope to help,
Lars

I'm unsure if the WORKDIR change is superfluous, doesn't matter though.
One last thing: We should provide a default config. In /opt/config/ you won't have any and the folder name doesn't make much sense. I'd suggest . as a config will exist there and the user would be ready to roll.

I use sys.path[0] as a default for --config_dir if not set (see line 38). I don't see a contradiction ? For Docker, the default storage for /config is within the container. But that would not make a lot of sense as data within the container gets deleted when e.g. upgrading to a new version.
Directory /opt/config was just meant as an example where someone might place his config.ini. Mine is in /srv/docker/miflora (I have all data and config stuff for my docker stack under /srv/docker), but that is also more of a personal choice.

Added a line of documentation for reporting_method == homeassistant-mqtt to config.ini.dist also.

Thanks @LarsAC, your addition is very much appreciated!


Affinity for MultiGPU optimization in a single run

The affinity stuff was a bit confusing, I wasn't able to figure out how to correctly set it up for using multiple GPUs for optimization. Note that I am doing only a single run. Here's my setup:
    affinity = make_affinity(
        n_cpu_core=12,
        n_gpu=4,
        gpu_per_run=3,
        sample_gpu_per_run=1,
        async_sample=True,
        n_socket=1,
        run_slot=0
    )

I want to use 4 GPUs for the run (3 for optimization, 1 for sampling) in a single run with asynchronous sampling.
I get the error
    n_worker = len(affinity["workers_cpus"]) if n_worker is None else n_worker
KeyError: 'workers_cpus'

What am I doing wrong with the configuration?
Sidenote: It would be great to have simpler configuration settings for single run experiments (which I imagine would be a bulk of the use case)

I was able to fix this by switching to AsyncRLEval, but I get an unrelated error now:
Process Process-6:
Traceback (most recent call last):
  File "/opt/conda/envs/pytorch-py3.6/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/conda/envs/pytorch-py3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/v-ankean/.local/lib/python3.6/site-packages/rlpyt/runners/async_rl.py", line 603, in memory_copier
    replay_buffer.append_samples(samples_to_buffer(sample_buffer))
AttributeError: 'NoneType' object has no attribute 'append_samples'
2020-03-15 04:26:53.188146  | dqn_asterix_0 Evaluation reached max num trajectories (5).
Process Process-7:
Traceback (most recent call last):
  File "/opt/conda/envs/pytorch-py3.6/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/opt/conda/envs/pytorch-py3.6/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/v-ankean/.local/lib/python3.6/site-packages/rlpyt/runners/async_rl.py", line 603, in memory_copier
    replay_buffer.append_samples(samples_to_buffer(sample_buffer))
AttributeError: 'NoneType' object has no attribute 'append_samples'

Here's my complete script for posterity:
from rlpyt.agents.dqn.atari.atari_catdqn_agent import AtariCatDqnAgent
from rlpyt.algos.dqn.cat_dqn import CategoricalDQN
from rlpyt.samplers.async_.gpu_sampler import AsyncGpuSampler
from rlpyt.samplers.parallel.gpu.sampler import GpuSampler
from rlpyt.samplers.parallel.gpu.collectors import GpuWaitResetCollector
from rlpyt.envs.atari.atari_env import AtariEnv, AtariTrajInfo
from rlpyt.algos.dqn.dqn import DQN
from rlpyt.agents.dqn.atari.atari_dqn_agent import AtariDqnAgent
from rlpyt.runners.minibatch_rl import MinibatchRlEval
from rlpyt.utils.launching.affinity import encode_affinity, make_affinity, quick_affinity_code, affinity_from_code
from rlpyt.utils.logging.context import logger_context
import wandb

from src.rlpyt_models import MinibatchRlEvalWandb, AsyncRlEvalWandb

def build_and_train(game="ms_pacman", run_ID=0):
    affinity = make_affinity(
        n_cpu_core=12,
        n_gpu=4,
        gpu_per_run=3,
        n_socket=1,
        run_slot=0,
        sample_gpu_per_run=1,
        async_sample=True
    )
    config = dict(
        env=dict(game=game),
        algo=dict(batch_size=128, min_steps_learn=1e4, prioritized_replay=True),
        sampler=dict(batch_T=2, batch_B=32),
    )
    sampler = AsyncGpuSampler(
        EnvCls=AtariEnv,
        TrajInfoCls=AtariTrajInfo,
        env_kwargs=dict(game=game),
        eval_env_kwargs=dict(game=game),
        eval_n_envs=10,
        eval_max_steps=int(10e3),
        eval_max_trajectories=5,
        # batch_T=4,  # Get from config.
        # batch_B=1,
        **config["sampler"]  # More parallel environments for batched forward-pass.
    )
    algo = CategoricalDQN(**config["algo"])  # Run with defaults.
    agent = AtariCatDqnAgent()
    runner = AsyncRlEvalWandb(
        algo=algo,
        agent=agent,
        sampler=sampler,
        n_steps=50e6,
        log_interval_steps=1e3,
        affinity=affinity,
    )
    name = "dqn_" + game
    log_dir = "example"
    with logger_context(log_dir, run_ID, name, config):
        runner.train()

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    parser.add_argument('--game', help='Atari game', default='pong')
    args = parser.parse_args()
    wandb.init(project='rlpyt')
    build_and_train(
        game=args.game,
        run_ID=args.run_ID
    )

Let me know if you want me to create a separate issue for this error.

Turns out this is due to a bug in CategoricalDQN with async sampling and has a quick fix as mentioned by @bmazoure in #30. I have created a PR to fix this: https://github.com/astooke/rlpyt/pull/126

Seems you were you able to get everything running OK?
Yes sorry for the complication in the affinity, and the most complicated is in the async affinity.  Hopefully the options you need are available?  I'm not 100% sure I tested all the cases there but had enough to get R2D1 running as fast as it can.

Yeah, I was able to get it working. Maybe the docs could use info on how to set affinity appropriately, and workarounds around common issues?


Upgrade React's peer dependency to 0.13.0

:+1:

Any reason this wouldn't work with React 0.14?

There would be deprecation warnings from not going through ReactDOM for the rendering - not sure if that matters to you.

Warnings aren't as important as the installation error I see with unresolved peerDependencies from NPM.
Where in Marty does it Call render()?

Application.renderToString and related methods.

It looks like renderToString() takes a render argument.
Seems possible to pass in from react-dom.
I'm not to familiar with then inner workings of marty and marty-lib, but Perhaps it would be helpful to cut a final 1.0 with support for React 0.14.


Fix bug where first ScrollViewer was not focusable; Closes #160;

Fixed bug where the first ScrollViewer in the "ScrollViewer (preview)" page was not focusable.
Description

Added a ' IsTabStop="True" ' to the ScrollViewer.
Motivation and Context

Fixes #160.
How Has This Been Tested?

Tested by tabbing through page.
Screenshots (if appropriate):
Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Can you please confirm that arrowing while the Scrollviewer has focus causes the scrollviewer to move as expected?

Using the ScrollViewer using arrow keys works just as exspected.


fix: replace vue router exact prop

Moved exact prop functionality to be handled manually
For more info: https://router.vuejs.org/guide/migration/#removal-of-the-exact-prop-in-router-link

Will update it and then create a new PR


Custom Resources in root orchestrator

Short
We want to be able to dynamically add new endpoints to new api objects and automatically handle their storage.
Proposal
For further extending the system, this issue will handle dynamically creating new api objects, provide api endpoints for manipulating them and handle their storage.
Add the logic in resource abstractor to also benefit from the hooks.
Flow:

Adding a new resource should create a new endpoint at /api/custom_resources/<new_resource>.
There will be a database called custom_resources that would hold the resources name along with their schema for validation - think of it as a collection of metadata.
Actual data of a resource should be in a collection of custom_resources.

Ratio
This is useful for further extending the system.
Impact
Resource abstractor
Checklist

[x] Discussed
[ ] Documented
[x] Implemented
[x] Tested

🥀 This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 10 days.

🥀 This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 10 days.


Add validate number without region code

Connection with issue(s)

Close #64
Solution description
If don't pass region code to validator, try validate a complete number
Video demo
https://user-images.githubusercontent.com/21011641/182707477-f1a37f4b-345b-402d-8288-d7b6c43c4ba5.mp4
To Do

[x] Read contributing guide
[x] Check the original issue to confirm it is fully satisfied
[x] Add solution description to help guide reviewers
[ ] Add unit test to verify new or fixed behaviour
[x] If apply, add documentation to code properties and package readme

That looks great! If I do not pass a region code, still validate if the number is valid!
Thank you for digging into this. I can try to test it as well.

I understand that it's a WIP, still have added these comments as reference

I tested in Android with random numbers and works fine

When will this get pushed in a version bump? Thanks for fixing!

When finish the readme refactor

This breaks format() on IOS
https://github.com/flutter-form-builder-ecosystem/phone_number/blob/688e85155106a505385cab7c010fe750c64d6d7b/ios/Classes/SwiftPhoneNumberPlugin.swift#L26-L42
Passing string as +5518988012364 and region as US and getting the error
PlatformException(InvalidArgument, The 'string' argument is missing., null, null)

@deandreamatias Looks like the IOS Swift Code needs modification as well.

Open a issue please

Done, Issue #82


Does simplecrawler support distributed queuing?

Hello!
I'm wondering if it is possible to run simplecrawler as a distributed cache, to reduce download time?
That means to manually inject and retrieve items from a distributed queue, e.g. Redis.  I am just wondering if this is the right way to proceed?

This has been a long-standing ambition - to be able to run multiple crawler instances that share a queue that's backed by Redid, for instance. The architecture should support it, but you will need to write a queue implementation that uses Redis (or something similar) yourself.
One alternative might be too look into https://github.com/yujiosaka/headless-chrome-crawler - they support distributed crawling if I'm not mistaken


Add Hover effect on dropdown menu ✨

Pull Request
Title

Add Hover effect on dropdown menu ✨
Description

Related Issues

#754
Checklist

[x] I have tested the changes locally
[ ] Documentation has been updated (if necessary)
[x] Changes are backward-compatible

Screenshots (if applicable)

Additional Notes

it is disappearing before clicking on drop down options..

@manikumarreddyu Okay I check

@manikumarreddyu Done

if it possible please remove that arrows ...please @AnushkaChouhan25
do this under new issue

Sure @manikumarreddyu I made new issue for it.


chore(docs): fix-typo-in-options-reference

Just fixes a small typo.
Thank you for you work 🙏🏼

Can you also do the same change in es docs?

Can you also do the same change in es docs?
yeah sure. I thought i dropped that first commit during a rebase 🤔
just squash merge this if you want


Facial-tracked bones are not displayed correctly

This occurs for both VRM and normal .glb models. In the Model view, left column, bones should be displayed as selected if they are indeed selected.

probably fixed by commit 7cfbbbe5f5a56172a670093816a7c5f46d10b73b


Starting mixing produces "Input is banned for 42550 Minutes Details:: (Windows 10)

When I start mixing, after about 5 seconds, I get an error message, and mixing stops. IT seems I am banned, as starting produces this error no matter what now.
i was also receving another message about 12 hours ago, after 3-15mins after starting mixing.

The first message came up for me, too lately. I must revise that part of the code, because it's been done in a complicated way.
The input ban is quite a problem, you've been banned for one month, because your client didn't do what the coordinator expected.
It became apparent our broadcasting mechanism stopped working reliably at scale, which can cause a lot of nasty, seemingly unrelated issues, like this. This way you didn't get notified when phase change happened and so you didn't act on that, therefore your coins were banned. This is my assumption, it can be something else, but I doubt.
Also I am planning to implement opt-in logging in the future, so we get more correct error reports, unfortunately I completely dumped logging back then, saying "logs in a privacy oriented wallet is dangerous".

There is a new version of the wallet, called Wallet Wasabi. The mixing is not yet implemented, but will be refactored.