Validate the contents of identity centric metadata

Currently if I'm looking at a project on PyPI, it can be difficult to determine if it's "real" or not. I can look and see the user names that are publishing the project as well as certain key pieces of metadata such as the project home page, the source repository, etc.
Unfortunately, there's no way to verify that a project that has say.. https://github.com/pypa/pip in it's home page, is actually the real pip, and isn't a fake imposter pip. The same could go for other URLs, or email addresses etc. Thus it would be useful if there was some way to actually prove ownership of those URLs/emails, and either differentiate them in the UI somehow, or hide them completely unless they've been proven to be owned by one of the publishing users.

I recall issues like this coming up at least once, if not a few times, over in pypi-support. Someone would fork a repository, change the name in setup.py, and then upload it to PyPI, with or without any other changes. All of the documentation and other links would remain pointed to the originals. This was confusing for users and frustrating to owners of the original package.
So I'm 👍 for some sort of blue verified checkmark or something from that perspective.
With my publisher hat on, though, I would hope this would be completely automated and I wouldn't have to do anything special to earn that blue checkmark.

One idea: we could add a blue checkmark for all links in the sidebar that contain a link back to the project's pypi page or pip install <project-name>. This would for us to load those links, but it would be zero-effort for most packages.
That being said, it wouldn't help if they point to forked versions, but in that case, the github star count might be a tell.

👍
Any progress on this issue?
I've been looking at malware from PyPI and it is common for the author_email to be "spoofed" (either pointing to nowhere, or using somebody else's email address).
Some related context is this HN discussion: https://news.ycombinator.com/item?id=33438678 Many commenters are asking about providing this sort of information.
I see some considerations that need discussion:

how do you support small personal projects (one or two maintainers) and large projects (big team, company) at the same time?
how do you provide verification signals without creating a false sense of security? (e.g. hide unvalidated data from the UI versus showing a blue checkmark against it)

Some validation is easier than others as well - e.g. email validation is pretty straightforward, but homepage validation would require something like the ACME protocol.

Haha, rereading my 2-year-old comment above about a blue check marks seems to resonate strangely in today's term :sweat_smile:
Who would have guessed...

My general thoughts here is that for metadata that we can 'verify', we should probably elevate that metadata in the UI over 'unverified' metadata.
We can already validate email addresses that correspond to verified emails of maintainers. That won't include the ability to verify mailinglist-style emails, but that could potentially be added to organizations once that feature lands.
With #12465, we'll be able to 'validate' the source repository as well, so any metadata that either references the given upstream source repository can be considered verified as well.
I agree that domains/urls will need to use the ACME protocol or something similar. I think there's probably a UX question on how these would be done per-project, if we wanted to go that route.

Mastodon has a link verification system, that might be nice.
That's never going to be foolproof though.

Related: #8462 #10917

From attempting to perform identity-assurance checks on packages manually: bidirectional references can be a reassuring indicator.
In context here: when a PyPi package points to a GitHub repository as its source code, then that's interpretable as a useful but as-yet-untrusted statement.  When up-to-date references are inspected within the contents of the cloned linked repository and they point back to the same original package on PyPi, then confidence in the statement increases.
For reproducible-build-compliant packages the situation improves further: any third party can confirm not only that the source origin and package destination are in concordance, but also whether the published artifact from the destination is bit-for-bit genuine by comparing it a build-from scratch of the corresponding raw origin source materials.  This can be verified on both a historic and ongoing basis.
So that's two orthogonal identity validation mechanisms:

Is-where-it-claims-to-be (it's possible to navigate a graph from the content distribution location A to the source at B and then from B back again to A).
Is-what-it-claims-to-be (the claimed source code for a package and version builds into the bit-for-bit identical content found at the content distribution location for the same package and version).

These don't prevent an attacker copying the source in entirety and creating a duplicate under a different name with an internally-consistent reference graph.  Given widespread free communication I think it's reasonable to expect that enough of the package consumer population will be (or become) aware of and gravitate towards the authentic package to solve that problem.

Following on to my previous comment, here's a mockup of what I'm imagining to separate the metadata we can verify today (source repository, maintainer email, GitHub statistics, Owner/Maintainers) from the unverifiable metadata:

"Source" should only be verified as a project link if the project has a Trusted Publisher with a matching URL, and the publisher has been used at least once to publish to this project
"Metadata" should only be emails that are included in Author-Email or Maintainer-Email that are also verified user emails for any collaborator on the project
"GitHub Statistics" should only be included if "Source" is verified
"Owner"/"Maintainers" should always be included.

Over time we can move things from below the fold to above it, but this should be a big improvement as-is for now.
I pushed the diff for the mockup here, there's some hacky stuff in there just to get the mockup to look good, but it could be a good starting point.

I'm starting working on this for creating the verified session and adding "Owner"/"Maintainers" on that

I wonder if it makes more sense to have verified details and then unverified details or to have each category with a verfified sub-section and a non-verified sub-section. It feels weird to break the project links apart from one another. When your eyes have reached the place where the repository is, it's not very clear that if the documentation isn't there, you have to look a some place else entirely to find a different link section that might contain the link to the docs.
I'd even argue that in this case, the whole thing would look more readable if the project doesn't use trusted publishers, which is
What about something like this ? (not arguing it's better, just a suggestion for the discussions)
(Would @nlhkabu have an opinion on the matter ?)

#16205 Starts marking URLs as verified, this now just needs to be surfaced in the UI.

#16205 Starts marking URLs as verified, this now just needs to be surfaced in the UI.

I'm working on the UI part now

Reopening this, we have solved this for a subset of project urls that relate to Trusted Publishing, but these two remain:

"Metadata" should only be emails that are included in Author-Email or Maintainer-Email that are also verified user emails for any collaborator on the project
"GitHub Statistics" should only be included if "Source" is verified

I think we also want to think about a solution for validating non-trusted publisher project URLs (e.g., ACME).

I think we also want to think about a solution for validating non-trusted publisher project URLs (e.g., ACME).

I am a massive fan of this idea 🙂
To spitball a little bit, a given FQDN's (foo.example.com) resources could be considered verified for a project example if:

foo.example.com is a secure origin (HTTPS)
foo.example.com/.well-known/pypi exists and is JSON
The contents of the pypi JSON resource are something like this:

{
  "version": 1,
  "packages": ["example"]
}

(where packages is plural since a FQDN may host links/resources for multiple projects.)
Another lower-intensity option would be the rel="me" approach that Mastodon and similar services use. This approach has the benefit of being per-resource (meaning that the entire FQDN isn't considered valid, since the user may not control the entire FQDN), at the cost of requiring the user to tweak their HTML slightly. Using example again:

foo.example.com is a secure origin
/some/resource on foo.example.com is (X)HTML
/some/resource contains a <link ...> as follows:

<head>
  <link rel="me" href="https://pypi.org/p/example">
</head>

Like with the .well-known approach, the user could include multiple <link> tags to assert ownership of multiple PyPI projects.
Alternatively, this could use meta instead to prevent implicit resolution of links:
<head>
  <meta rel="me" namespace="pypi.org" package="example">
</head>

...or even multiple in the same tag:
<head>
  <meta rel="me" namespace="pypi.org" package="example another-example">
</head>

Ref on Mastodon's link verification: https://docs.joinmastodon.org/user/profile/#verification

Using .well-known would be interesting but I think this wouldn't cover enough use cases (I'm thinking things like Read the Docs pages, etc, where the user doesn't control the FQDN).
Using rel="me" makes sense to me. We probably also need to think about verifying these outside the upload loop via a task.

Using .well-known would be interesting but I think this wouldn't cover enough use cases (I'm thinking things like Read the Docs pages, etc, where the user doesn't control the FQDN).

Yeah -- my thinking was that it'd be most useful for projects/companies with full-blown domains, e.g. company $foo might want to blanket-verify all of their PyPI project names without having to add them to each docs page/homepage. But that's probably a much more niche case than Read The Docs etc. 🙂

Using rel="me" makes sense to me. We probably also need to think about verifying these outside the upload loop via a task.

Makes sense! That reminded me: do we also want to consider periodic reverification? My initial thought is "no" since URLs are verified on a per-release basis, but I could see an argument for doing it as well (or at least giving project owners the ability to click a "reverify" button once per release or similar).

I think I would want to stick to the "verified at time of release" model, at least for now (not trying to reinvent Keybase here).

If we specifically plan for this to be used on ReadTheDocs, it makes sense to ensure that whatever format we decide on is easy to use with Sphinx & mkdocs.
I've made a small test with Sphinx:
.. meta::
   :rel=me namespace=pypi.org package=example: pypi

produces
<meta content="pypi" namespace="pypi.org" package="example" rel="me" />

Note: it's going to be much harder to have multiple values in package separated by a space as it would break the meta directive parsing.
It's easy to have multiple meta tags though:
.. meta::
   :rel=me namespace=pypi.org package=example: pypi
:rel=me namespace=pypi.org package=other: pypi

Also, it's impossible to not have a content element, as far as I can tell. But then maybe instead of package= we should list the packages in content, which would solve both issues at once:
.. meta::
   :rel=me namespace=pypi.org: example other

<meta content="example other" namespace="pypi.org" rel="me" />

On mkdocs, it looks like one would need to extend the theme, which is a bit cumbersome. But I'm sore someone would make a plugin soon enough to solve this (it's probably the same with sphinx, realistically)

Thanks for looking into that @ewjoachim! Using content for the package(s) makes sense to me, given that 🙂

I'm working on this. One thing that we should be aware of is that implementing this kind of verification, where each URL is accessed and parsed to see if it contains the meta tag specified above, means that warehouse will start doing a lot more outgoing requests.
I'm not sure of how many files are uploaded per second (let's call it N), but assuming (conservatively) that each of those releases has 1 URL in its metadata, that means that PyPI will have at least N new outgoing requests per second, to arbitrary URLs.
We can have restrictions to reduce network activity and protect against DoS attacks (like what Mastodon does, limiting the response size to 1 MB), but we'll still need to handle all the new outgoing requests.
Since I don't know the number of files uploaded per second, I'm leaving the question open here to see if PyPI's infrastructure can handle the extra network activity downloading those webpages would cause.

Yeah, thanks for calling that out! I think there are a few things PyPI could do to keep the degree of uncontrolled outbound traffic to a minimum:

Limit the number of URLs allowed per release. I suspect that there are no (legit?) projects that expect to attach more than a dozen unique URLs to each release, but we should probably confirm that 🙂
Limit the number of unique FQDNs during verification, regardless of number of metadata URLs.
Limit the amount of allowed network traffic, like you said. In practice I think we can even do well under 1MB, since the relevant <meta> tag should be in the <head>.
"Collate" verified URLs, i.e. don't re-perform URL verification if another file in the same release has already verified the URL within a particular time window (~15-60 minutes?)

PyPI could do some or all of them; I suspect limiting unique FQDNs might be a little too extreme compared to the others.

Also, as far as I can tell, pypi.org's DNS point to fastly. If we were to easily know the IPs of the real servers beneath fastly, DDoS attacks could become easier. We need to make sure that the outgoing ip used to connect to the website is not same as the inbound IP for the website. It's probably the case as the worker run on different machines, but it's worth mentally checking this by anyone who knows the infrastructure well enough. (Also, ensure that inbound traffic that's not coming from fastly of firewalled, it's probably already the case but in case it's not, it's probably worth doing it)

Limit the number of unique FQDNs during verification, regardless of number of metadata URLs.

Also, limit the number of underlying IPs after DNS resolving, and/or domain names, because if a server has a wildcard DNS, we can generate infinite FQDNs that will hit the same server.

Oh, and we may want to ensure we put a reasonable timeout for the requests (for us).
Also, we may want to control (& advertise) the user agent we use to make those requests.
And maybe keep a log of all the requests we made and what release it's linked to ?
Also, could we end up in trouble if someone makes us make a request to prohibited content ?
Oh, we also need to protect ourselves from SSRF, even though in this case we're not displaying what we requested back to the user, in the hopefully inexistant but possible case that an internal GET request can have side effect, this could be catastrophic. e.g. we're on AWS, if the user publishes a package with URL http://<IP_ADDRESS>/latest/meta-data/, then suddenly the page we manipulate contains an usable AWS token from the machine that made the request. We need to make sure that the URL contains a proper domain (not an IP, not localhost)
I guess this is the kind of question everyone should ask when they implement a webhook or a crawler or anything. There surely is a resource out there from people who have solved these headaches already.

Just for completeness: if the page is cut due to being more than 1MB, but we still want to check the headers, we'll need a HTML parser that accept partial content.

Also, we may want to control (& advertise) the user agent we use to make those requests.

Yes, this makes a lot of sense to do!

And maybe keep a log of all the requests we made and what release it's linked to ? Could we end up in trouble if someone makes us make a request to illegal content ?

This log would presumably just be the list of URLs listed in the project's JSON API representation/latest release page on PyPI, no? I'm personally wary of PyPI retaining any more data than absolutely necessary, especially since in this case we're not actually storing any data from the URL, only confirming that the URL is serving HTML with a particular <meta> tag at a particular point in time.

Oh, we also need to protect ourselves from SSRF, even though in this case we're not displaying what we requested back to the user, in the hopefully inexistant but possible case that an internal GET request can have side effect, this could be catastrophic.

For SSRF, I think the main thing we'll need to do is prevent server-controlled redirects. In other words: if the URL itself doesn't serve the <meta> tag itself, we won't allow it to redirect us anywhere else. I don't think PyPI should worry about GETs being non-idempotent -- any web service that allows that is simultaneously thoroughly out of spec and would be immediately broken by the first spider to crawl the project on PyPI anyways 🙂

We need to make sure that the URL contains a proper domain (not an IP, not localhost)

I could be convinced that we should add this restriction as a practical matter, but I'm not sure it's that important in terms of security? If the URL has an IP as its host but otherwise matches the secure origin rules (i.e. HTTPS), is there a reason we shouldn't validate it?

Oh and MITM of course. We should only try to validate HTTPS urls, validating an HTTP URL would only lead to an untrustable result.

FWIW, this one at least is covered under "must be a secure origin" in https://github.com/pypi/warehouse/issues/8635#issuecomment-2289617568.

For SSRF, I think the main thing we'll need to do is prevent server-controlled redirects. In other words: if the URL itself doesn't serve the  tag itself, we won't allow it to redirect us anywhere else. I don't think PyPI should worry about GETs being non-idempotent -- any web service that allows that is simultaneously thoroughly out of spec and would be immediately broken by the first spider to crawl the project on PyPI anyways 🙂

The danger of SSRF is internal urls. The request will be made from within the PyPI infrastucture and may have access to network-protected endpoints that might not be accessible to random spiders

This could make sense indeed.

Hm, thinking again, if someone uses https://<IP_ADDRESS>, this means we ARE going to make the request and if it just so happens that this IP is listening on 443, the request will go through, and we will evaluate the result. It's probably not a high attack vector, but I'm not at all comfortable that the PyPI server will be able to request any internal HTTPS URL (with a GET request where the path is the only thing controlled by the attacker)

Oh, btw, should we make sure the port is not overridden (or force it ourselves to 443) ? I don't know if there are protocols out there where we could do nasty things just by opening a TCP connection. I hope not.

The danger of SSRF is internal urls. The request will be made from within the PyPI infrastucture and may have access to network-protected endpoints that might not be accessible to random spiders

Ah, I see what you mean. Yeah, I think the expectation here would be that we deny anything that resolves to a local/private/reserved address range. IP addresses would be allowed only insofar as they represent public ranges (and serve HTTPS, per above).

Oh, btw, should we make sure the port is not overridden (or force it ourselves to 443) ? I don't know if there are protocols out there where we could do nasty things just by opening a TCP connection. I hope not.

I think this falls under the "PyPI isn't responsible if your thing breaks after you stuff a URL to it in your metadata," but this is another datapoint in favor of making our lives easy and simply not supporting anything other than HTTPS + domain names + port 443, with zero exceptions 🙂

"Collate" verified URLs, i.e. don't re-perform URL verification if another file in the same release has already verified the URL within a particular time window (~15-60 minutes?)

What if we only perform this kind of verification once per release? As in, during the upload of the first file that creates the release.
The reason why we currently re-verify URLs for each file upload of the same release is because Trusted Publisher verification means that some file uploads might come from the Trusted Publisher URL and some not. So it makes sense to re-verify: the first file upload might not come from a relevant Trusted Publisher, but subsequent ones might.
However, this is not the case for this type of verification, since we're accessing resources independent of the upload process and authentication. So checking the URLs once during release creation might be a simple way of limiting the amount of requests we make.

The reason why we currently re-verify URLs for each file upload of the same release is because Trusted Publisher verification means that some file uploads might come from the Trusted Publisher URL and some not. So it makes sense to re-verify: the first file upload might not come from a relevant Trusted Publisher, but subsequent ones might.

But the pages might change. I agree that once a page has been verified, it's probably fair to trust it for some amount of time, but if someone already has a URL set up, and learns about this feature and adds their meta tag and pushes a new version, we should recheck even if we've checked before.

but if someone already has a URL set up, and learns about this feature and adds their meta tag and pushes a new version, we should recheck even if we've checked before.

Yes, that's what I meant with my comment, we should do this kind of verification (meta tag) once per release:

So checking the URLs once during release creation might be a simple way of limiting the amount of requests we make.

Maybe the confusion is because I'm using "release" to refer to a new version of a package, so I'm saying we should recheck every time the user uploads a new version.

Ah no my bad, you said it right, I misunderstood. It's just that as far as I had understood, we had dismissed the idea of re-verifying a link, so it was already what I thought we were at. So I thought you suggested once per project, but it's my bad :)

So how does this work for institutional projects or are there any plans to implement it?
For instance, there's a team of developers and one devops staffer is tasked with releasing. After the staffer email is verified (email-click_link) that allows PyPi to publish that username as "Verified details" under the "Maintainers" header. But what about project URLs that point to a specific Web site not associated with the repository provider? Is there any mechanism to get the actual project Web site verified? (Concrete example: Repo is at github.com; staffer is at<EMAIL_ADDRESS>project contact is at<EMAIL_ADDRESS>project site is at project.biginstitution.com. The project site and project contact will be reported as "Unverified details".)
Apparently to get a project-focused email address reported as verified, an account must be opened at PyPi; it must be verified (email-click_link); and then the project-focused email address must be tied to the project. If it's a legitimate email and shared amongst the team, well that feels a little security-crunchy -- I wouldn't want bunches of people with capacity to reset a password with PyPi rights but that seems the only approach right now that might work.
Anyway, what about License details? "Unverified" when it's a custom institutional license. Project keywords in Setuptools are "Unverified". Python version (e.g. Requires) is unverified.
Maybe I missed something in docs...

I know there is a lot of work here to automate detection of "Verified details", and that's great. However, this may not cover all scenarios, specifically as described in my previous comment. A simple solution would be to provide text fields on a project basis to add verified URLs and/or email addresses. Since the account holder has to be verified, shouldn't this be an acceptable addition? Better yet, restrict additional details to the parent domain of the account holder, e.g<EMAIL_ADDRESS>can only add emails and URLs that are at biginstitution.com or its subdomains.

I see that ACME-style verification is the approach in this ticket's description but I should add it is not always easy to get metadata injected into top-level institutional URLs which may be maintained by divergent teams.

Yeah, that's why we haven't gone forwards with full URL verification yet -- the design space is somewhat open, and there are tradeoffs to an ACME well-known approach, a <meta> approach, etc.

Apparently to get a project-focused email address reported as verified, an account must be opened at PyPi; it must be verified (email-click_link); and then the project-focused email address must be tied to the project. If it's a legitimate email and shared amongst the team, well that feels a little security-crunchy -- I wouldn't want bunches of people with capacity to reset a password with PyPi rights but that seems the only approach right now that might work.

Could you say more about what you're expecting here? The point of the "verified" badge next to an email in the project view is only to provide a visual indicator that the email in the project's metadata matches one on an owning account, i.e. one entitled to publish for the account. PyPI could do non-user verifications of emails, but that would have some consequences:

Email verification is fraught under good conditions; doing it outside of a user flow means needing to consider things like periodic reverification since there'll be no consistent delivery signal (from project emails)
We'd need to figure out how to appropriately signal to users that some verified emails are verified because they're user emails (i.e. because they're empowered to publish that project) and others are verified because they were 'just' verified.

If what you want to do is share a contact non-owner email for a project, perhaps you could link to it from an already-verified page? That would accomplish the same thing in terms of transitive verification, I think.

Anyway, what about License details? "Unverified" when it's a custom institutional license. Project keywords in Setuptools are "Unverified". Python version (e.g. Requires) is unverified.

Everything that isn't currently "verifiable" (i.e. attributable to a verifiable external resource) is currently marked as "unverified." It would perhaps make sense to have a third category here for things that are "not verified because verifying them doesn't make sense," although it's not immediately clear where we'd order that in the current metadata pane 🙂

A simple option would be to provide text fields on a project basis to add verified URLs and/or email addresses.

This sounds simple, but it's not that simple in practice 🙂 -- PyPI's project view is generally careful not to conflate "project originated" and "index originated" metadata, except where the latter can explicitly verify the former. We could add a notion of free-form index-originated project metadata, but doing so would need to still verify those things, since verification is meant to be two-way.

Better yet, restrict additional details to the parent domain of the account holder, e.g<EMAIL_ADDRESS>can only add emails and URLs that are at biginstitution.com or its subdomains.

PyPI isn't actually aware of domains or subdomains at all, really -- there's some very basic specialized stuff in place around github.com and some other well-known URLs, but PyPI doesn't have a "structured" view of a domain or its subdomains, its email inboxes, etc.
There's been some discussion on that topic recently (and how it merges with PyPI's org feature), which you might find interesting + could benefit from a potential user's perspective.


Integrating with create-react-app

I'm trying to use wasm-loader with create-react-app and I'm having some trouble getting it to work.
I'm using react-app-rewired to override the configuration:
module.exports = function override(config, env) {
  config.module.rules.push({
    test: /\.wasm$/,
    include: '/code/wasm-test/src',
    use: [require.resolve('wasm-loader')]
  });
  config.module.rules.forEach(rule => console.dir(rule));
  return config;
}

I've tried several different variants of this, got the same result for all of them (instead of use, reference loader directly, have an object with loader and options inside use, omit include).
The registration seems to work fine, and checking the output from the rules it seems that the rule for wasm-loader is added correctly:
{ test: /\.(js|jsx)$/,
  enforce: 'pre',
  use:
   [ { options: [Object],
       loader: '/code/wasm-test/node_modules/eslint-loader/index.js' } ],
  include: '/code/wasm-test/src' }
{ exclude:
   [ /\.html$/,
     /\.(js|jsx)$/,
     /\.css$/,
     /\.json$/,
     /\.bmp$/,
     /\.gif$/,
     /\.jpe?g$/,
     /\.png$/ ],
  loader: '/code/wasm-test/node_modules/file-loader/index.js',
  options: { name: 'static/media/[name].[hash:8].[ext]' } }
{ test: [ /\.bmp$/, /\.gif$/, /\.jpe?g$/, /\.png$/ ],
  loader: '/code/wasm-test/node_modules/url-loader/index.js',
  options: { limit: 10000, name: 'static/media/[name].[hash:8].[ext]' } }
{ test: /\.(js|jsx)$/,
  include: '/code/wasm-test/src',
  loader: '/code/wasm-test/node_modules/babel-loader/lib/index.js',
  options:
   { babelrc: false,
     presets: [ '/code/wasm-test/node_modules/babel-preset-react-app/index.js' ],
     cacheDirectory: true } }
{ test: /\.css$/,
  use:
   [ '/code/wasm-test/node_modules/style-loader/index.js',
     { loader: '/code/wasm-test/node_modules/css-loader/index.js',
       options: [Object] },
     { loader: '/code/wasm-test/node_modules/postcss-loader/lib/index.js',
       options: [Object] } ] }
{ test: /\.wasm$/,
  use: [ '/code/wasm-test/node_modules/wasm-loader/index.js' ] }

However, when trying to load wasm files I get not found failures:
import Counter from './wasm/counter';
// import fails on module not found
// if I replace with './wasm/counter.wasm' I get the file name as the value of the import
// non-existent file names (e.g. './wasm/counter2.wasm') also trigger not found, as expected
const wasmHelloWorld = () => {
  console.dir(Counter);
}
window.onload = wasmHelloWorld;

Any idea what might be the issue?
Thanks for building this by the way, I can see it being really useful!

I managed to get it working by adding the .wasm extension to config.resolve.extensions and adding the wasm rule to the fileLoader exclusions. I ended up with this in config-overrides.js:
// through this file we can override the default cra configuration
// see: https://github.com/timarney/react-app-rewired
const path = require('path')

module.exports = function override(config, env) {
  const wasmExtensionRegExp = /\.wasm$/;

  config.resolve.extensions.push('.wasm');
  // make the file loader ignore wasm files
  const fileLoader = config.module.rules
    .find(rule => rule.loader && rule.loader.endsWith(`${path.sep}file-loader${path.sep}index.js`));
  fileLoader.exclude.push(wasmExtensionRegExp);

  // and add a dedicated loader for them
  config.module.rules.push({
    test: wasmExtensionRegExp,
    include: path.resolve(__dirname, "src"),
    use: [{loader: require.resolve('wasm-loader'), options: {}}]
  });
  return config;
}

Now I'm hitting another issue (Chrome 59):

WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.

This seems to be unrelated to the loader, so resolving this issue.

@cimi I'm having this error:

./src/wasm/testingwasm.wasm
Module parse failed: magic header not detected
You may need an appropriate loader to handle this file type.
Error: magic header not detected

Sounds like a corrupted module. It could happen because of a number of different reasons.
wabt the WebAssembly binary toolkit has a few very useful tools available online, one of which allows you to upload a wasm module, validate it and convert to the text format: wasm2wat.
Upload your file there to test whether or not the file is actually valid to begin with.

I spent forever looking or a solution to this, so hoping this saves someone some time. New version of CRA seems to have rearranged the file-loader rule. My hack is:
// through this file we can override the default cra configuration
// see: https://github.com/timarney/react-app-rewired
const path = require('path');

module.exports = function override(config, env) {
  const wasmExtensionRegExp = /\.wasm$/;

  config.resolve.extensions.push('.wasm');
  // make the file loader ignore wasm files
  let fileLoader = null;
  config.module.rules.forEach(rule => {
    (rule.oneOf || []).map(oneOf => {
      if (oneOf.loader && oneOf.loader.indexOf('file-loader') >= 0) {
        fileLoader = oneOf;
      }
    });
  });
  fileLoader.exclude.push(wasmExtensionRegExp);

  // Add a dedicated loader for them
  config.module.rules.push({
    test: wasmExtensionRegExp,
    include: path.resolve(__dirname, 'src'),
    use: [{ loader: require.resolve('wasm-loader'), options: {} }],
  });
  return config;
};

Note: that needs to go into config-overrides.js, same dir as package.json. You also need to npm install --save-dev react-app-rewired and replace the scripts in package.json:
    "start": "react-app-rewired start",
    "build": "react-app-rewired build",
    "test": "react-app-rewired test --env=jsdom",

I read a bit that CRA 2+ can do this, so unsure if react-app-rewired is strictly necessary?

For building WASM that can be consumed, easiest way is to use wasm-pack, which will create a pkg directory. I put all the Rust stuff off the root dir into native. Then in package.json you can import pkg (which is a npm module) via:
  "dependencies": {
    "native": "file:../native/pkg",
  }

Finally, this must be async loaded, so my index.tsx is:
import './index.css';

// The wasm must load async. Just wait to load anything else until it's done.
async function loadNative() {
  await import('native');
  await import('./app');
}

loadNative();

This blog post might be helpful for future visitors (using @AThilenius' answer above!): https://prestonrichey.com/blog/react-rust-wasm/

I encountered with
> react-app-rewired start
Starting the development server...

Failed to decode custom "name" section @50458; ignoring (integer representation too long).
Failed to compile.

./recognizer/pkg/recognizer_bg.wasm
Module parse failed: Unexpected section: 0x86
You may need an appropriate loader to handle this file type.
Error: Unexpected section: 0x86

I am following @prichey 's  solution
The .wasm file is valid, can be disasm into .wast, and can be compiled directly in browser.
what should I do now ?

seems this is related to version issue of wasm-bindgen. change the line to =0.2.34 and it works.

Note for future people @AThilenius :
As of 4/15/20 this solution still works. Here is a complete code example.


How to fetch multiple URLs (non-constant number) at once?

Say I need to fetch a list of remote resources, while the length on the list depends on the current props. (How) can I achieve that with use-http? As far as I understand the documentation of react, the number of hooks in a component needs to be constant.

I'm assuming you're talking about pagination? You could do something like
const SomeComp = ({ page, pageLength }) => {
  const [list, setList] = useState([])
  const { get, loading } = useFetch('url')

  const fetchList = async () => {
    const theList = await get(`?page=${page}&pageLength=${pageLength}`)
    setList(theList)
  }

  const mounted = useRef(false)
  useEffect(() => { // componentDidMount
    if (!mounted.current) {
      fetchList()
      mounted.current = true
    }
  }, []

  return
{list.map(item =>
{item.text}
)}

}

Does this make sense? Or are you asking something else?

Sorry for being unclear. What I actually am trying to do is load multiple URLs at once:
const SomeComp = ({urls}) => {
  const images = // fetch all the urls
  return images.map(/* do something with the results*/)
}

However I understand now that I can probably solve it like this:
const [results, setResults] = useState([])
const request = useFetch()
async function fetchStuff() {
    for (const url of urls) {
        const result = await request.get(url)
        setResults([...results, result])
    }
  }
useEffect(() => { fetchStuff() })

You can do that, however it's always recommended that you don't have api calls/awaits in loops. Why do you need to make an api call for image urls? If you're displaying images, why not just do imageUrls.map(url => )

In my application I am fetching  data, not images – that was just for the example.

You could do something like
const Comp = (props) => {
  const [imagesData, setImagesData] = useState([])
  const [loading, setLoading] = useState(false)
  const { get } = useFetch('url')

  const getImageData = useCallback(async () => {
    setLoading(true)
    const imgData = await images.map(async image =>  await get('/image?id=${image.id}'))
    setImagesData(imgData)
    setLoading(false)
  }, [props.images, imagesData, setImagesData, setLoading])

  const mounted = useRef(false)
  useEffect(() => {
    if (!mounted.current) { // on mount
      getImagesData()
    }
  }, [getImagesData])

  return (
    <>
      {loading && 'Loading...'}
      {imagesData.length > 0 && imagesData.map(img => )}
    </>
  )
}

Is this something like what you're looking for?

@supermihi is your question answered? Gonna close this if it is :)

@alex-cory thanks for your answer. The above should work (though I'd remove the mounted construct, which would prevent re-loading if props.images changes, right?).
However I don't really see the benefit of useFetch here, as opposed to calling fetch directly?

Oh yeah, that makes sense about the mounted part. In this case, the only benefit would be if you used the <Provider url='https://example.com'><App /></Provider>. Kind of like how axios works.  Also, if you have authentication, you could handle that at the top level in addition to the syntax is a bit cleaner


Needs documentation to add "App Engine Admin API"

When we ran the terraforming scripts using the README, we ended up with
* google_project.opsman_service_account_project: Error applying IAM policy for project "<redacted>": googleapi: Error 403: The caller does not have permission, forbidden

We fixed it by enabling the App Engine Admin API, which the README does not reference as needing to be enabled.

@chendrix was this actually a correct diagnosis of the issue or was there more?


what differents with react-window and virtual-list

What motivation to create your own solution?
link: react-window

🤔

rc-virtual-list is used for rc-component which support motion insert & remove. This is not face to upper developer. Pls feel free to use react-window.


Private Repositories - API Token

Hello!
First I'd like to express my thanks for this awesome project so thank you :)
Is there a reason why I cannot search private repos after providing gitleaks with a read-only API token? It still asks me for my github.com username and password after passing in my --token.
e.g., command:
./gitleaks --temp --token= --verbose --org https://github.com/myOrg

This issue will be fleshed out in the next version of gitleaks. This might be broken. I'll take a look and get back to you

Released new version, please follow examples in the wiki and try now


Speed of time vars and z output

Added support for speed of time variables as in matlab, as well as added z to output struct.

Looks good! It would be nice to have an example or test using this feature. Maybe you can rebase after merging #51

Yep will rebase and retarget to main after #51 goes in


sep service helm-chart initial

This PR adds a helm chart for SEP service.    This helm chart has been tested by publishing the chart to a helm chart repository and then deploying it to a aws EKS, MSK test environment.  The Stellar validator service has run after deploying this chart successfully.    This has not yet been tested in SDF infrastructure.
next step for this PR is to test helm chart (currently published/hosted by s3 bucket ) using these instruction and enable cicd for updates (helm package, helm repo index and publish)

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

this is awesome! question regarding the values.yaml file, looks like ~80% of the configmap values (and looks like some settings are not made configurable, ex: data-access) are coming from this values.yaml file, why not just have a separate configmap for each deployment?

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/

Reference Server Preview is available here:https://anchor-ref-pr345.previews.kube001.services.stellar-ops.com/SEP Server Preview is available here:https://anchor-sep-pr345.previews.kube001.services.stellar-ops.com/


[Component] Carousel

Wrapper of swiperjs.com

done with https://github.com/juntossomosmais/atomium/pull/235


Test samplesheet needed

Add test samplesheet that doesn't point to internal cdc data storage.

Test data already merged, hosted on s3.


python3Packages.rdkit: init at 2020.09.3

Motivation for this change
RDKit is a very widely used package in computational chemistry.
Things done

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[X] Fits CONTRIBUTING.md.

@rmcgibbo please fix the eval error of ofborg.

Thanks, @SuperSandro2000. Looks like I didn't structure the package correctly for ofborg given the fact that it needs sources from multiple repos. I'll try to follow this design pattern: https://discourse.nixos.org/t/how-to-create-package-with-multiple-sources/9308/3

@jonringer: Sure, will do.
I'm not totally sure how to reproduce the ofborg failure locally. First attempts to move the external sources into a single 'srcs' are failing with
error: cannot coerce a set to a string, at /home/mcgibbon/projects/nixpkgs/pkgs/development/interpreters/python/mk-python-derivation.nix:111:5
(use '--show-trace' to show detailed location information)

so i'll try to fix that.

you pass it a list, not a set

Looks like I'm still hitting an ofborg failure, where the builtins.fetchTarball (one of the srcs) won't pull down in restricted mode.
while evaluating the attribute 'srcs' of the derivation 'python3.7-rdkit-2020.09.3' at /var/lib/ofborg/checkout/repo/38dca4e3aa6bca43ea96d2fcc04e8229/mr-est/ofborg-evaluator-3/pkgs/development/interpreters/python/mk-python-derivation.nix:111:5:
access to URI 'http://sourceforge.net/projects/avalontoolkit/files/AvalonToolkit_1.2/AvalonToolkit_1.2.0.source.tar' is forbidden in restricted mode

Result of nixpkgs-review pr 107268 run on x86_64-darwin 1

3 packages failed to build:

python37Packages.rdkit
python38Packages.rdkit
python39Packages.rdkit

2 packages built:

coordgenlibs
maeparser

<1xyqxdqxwviq7b8jjxssxjlkldk01ms8dzqdjgvjs8n3fh7w0l11/details>
  100   391  100   391    0     0     73      0  0:00:05  0:00:05 --:--:-- 18619                                                                                100 2530k  100 2530k    0     0   378k      0  0:00:06  0:00:06 --:--:-- 4571k                                                                                unpacking source archive /private/tmp/nix-build-source.drv-0/AvalonToolkit_1.2.0.source.tar                                                                                error: zip file must contain a single file or directory.                                                                                hint: Pass stripRoot=false; to fetchzip to assume flat list of files

Thanks for the reviews. I think I've addressed all of the comments with the latest force-push.

Can you please put the changes from the review commit into the corresponding init commits?

Result of nixpkgs-review pr 107268 run on x86_64-darwin 1

3 packages failed to build:

python37Packages.rdkit
python38Packages.rdkit
python39Packages.rdkit

2 packages built:

coordgenlibs
maeparser

Fails to build on darwin because inchi failds to build:
these derivations will be built:
  /nix/store/93dwx0vanag5scwr3yf5818lzv8zzb0j-inchi-1.05.drv
building '/nix/store/93dwx0vanag5scwr3yf5818lzv8zzb0j-inchi-1.05.drv'...
unpacking sources
unpacking source archive /nix/store/7j8zpnzw37y1g3pwyv9ds718ljgjqjc9-INCHI-1-SRC.zip
source root is INCHI-1-SRC
setting SOURCE_DATE_EPOCH to timestamp<PHONE_NUMBER> of file INCHI-1-SRC/INCHI_BASE/src/runichi2.c
patching sources
configuring
no configure script, doing nothing
building
build flags: -j4 -l4 SHELL=/nix/store/drz9yxh2v1swn9yl88c8w5g0x69b8wk2-bash-4.4-p23/bin/bash
gcc -fPIC -DTARGET_API_LIB -DCOMPILE_ANSI_ONLY -ansi -O3 -c ../../../INCHI_BASE/src/ichican2.c
gcc -fPIC -DTARGET_API_LIB -DCOMPILE_ANSI_ONLY -ansi -O3 -c ../../../INCHI_BASE/src/ichicano.c
gcc -fPIC -DTARGET_API_LIB -DCOMPILE_ANSI_ONLY -ansi -O3 -c ../../../INCHI_BASE/src/ichi_io.c
gcc -fPIC -DTARGET_API_LIB -DCOMPILE_ANSI_ONLY -ansi -O3 -c ../../../INCHI_BASE/src/ichierr.c
/nix/store/drz9yxh2v1swn9yl88c8w5g0x69b8wk2-bash-4.4-p23/bin/bash: gcc: command not found
/nix/store/drz9yxh2v1swn9yl88c8w5g0x69b8wk2-bash-4.4-p23/bin/bash: gcc: command not found
make: *** [makefile:196: ichicano.o] Error 127
make: *** Waiting for unfinished jobs....
make: *** [makefile:196: ichican2.o] Error 127
/nix/store/drz9yxh2v1swn9yl88c8w5g0x69b8wk2-bash-4.4-p23/bin/bash: gcc: command not found
make: *** [makefile:196: ichi_io.o] Error 127
/nix/store/drz9yxh2v1swn9yl88c8w5g0x69b8wk2-bash-4.4-p23/bin/bash: gcc: command not found
make: *** [makefile:196: ichierr.o] Error 127
builder for '/nix/store/93dwx0vanag5scwr3yf5818lzv8zzb0j-inchi-1.05.drv' failed with exit code 2
error: build of '/nix/store/93dwx0vanag5scwr3yf5818lzv8zzb0j-inchi-1.05.drv' failed

Build fixed on darwin, and previous review comments incorporated into the per-package commits.

@jonringer: Thanks. Fixes applied.

Result of nixpkgs-review pr 107268 run on x86_64-darwin 1

6 packages built:

coordgenlibs
inchi
maeparser
python37Packages.rdkit
python38Packages.rdkit
python39Packages.rdkit

Result of nixpkgs-review pr 107268 run on x86_64-linux 1

6 packages built:

coordgenlibs
inchi
maeparser
python37Packages.rdkit
python38Packages.rdkit
python39Packages.rdkit

(Friendly ping on this PR)

(Friendly ping on this PR)

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 107268 run on x86_64-linux 1

6 packages built:

coordgenlibs
inchi
maeparser
python37Packages.rdkit
python38Packages.rdkit
python39Packages.rdkit

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 107268 run on x86_64-linux 1

6 packages built:

coordgenlibs
inchi
maeparser
python37Packages.rdkit
python38Packages.rdkit
python39Packages.rdkit

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 107268 run on x86_64-darwin 1

6 packages built:

coordgenlibs
inchi
maeparser
python37Packages.rdkit
python38Packages.rdkit
python39Packages.rdkit

This is a semi-automatic executed nixpkgs-review which does not build all packages (e.g. lumo, tensorflow or pytorch)
If you find some bugs or got suggestions for further things to search or run please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 107268 run on x86_64-darwin 1

6 packages built:

coordgenlibs
inchi
maeparser
python37Packages.rdkit
python38Packages.rdkit
python39Packages.rdkit

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 107268 run on x86_64-linux 1

5 packages built:

coordgenlibs
inchi
maeparser
python37Packages.rdkit
python38Packages.rdkit

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 107268 run on x86_64-darwin 1

2 packages failed to build and are new build failures:

python37Packages.rdkit: log https://termbin.com/dei9
python38Packages.rdkit: log https://termbin.com/luxg

Thanks for the report on Darwin, Sando. I think I've fix it now with some more darwin-specific fixups in libinchi.

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 107268 run on x86_64-darwin 1

2 packages built:

python37Packages.rdkit
python38Packages.rdkit


Commit skipping logic is hardcoded and yields subtle bugs in release notes content

Problem
For generating release notes with number of commits and contributors, we skip "automated" commits. It is an important feature because the number of commits is off if we consider commits made by release automation. The automated skip message is configurable by the user and Mockito project uses it. However, the implementation of shipkit is hardcoded to ignore commits with [ci skip] - see the code.
Even though the skip message is hardcoded, the recent Mockito release notes seems to calculate commits correctly. Why? :)
Suggested change
For the time being, let's ignore all commits with "[ci skip]" and also all commits that contain releasing.git.commitMessagePostfix. Code hints:

Code that contains hardcoded "[ci skip]"
Code that contains commitMessagePostfix configuration

@NagRock, do you want to take a stab at it? :)

Yea! You can assign me


DataTransformCell crashes due to incompatibility with support for datetimes with timezones added in Explorer 0.8.3

The following change described in Explorer's v0.8.3 changelog

Add new data type for datetimes with timezones: {:datetime, precision, time_zone} The old dtype is now {:naive_datetime, precision}.

lets DataTransformCell.normalize_dtypes crash with "1st argument: not an atom" from Atom.to_string(v) in the last pattern
    |> Enum.map(fn
      # TODO: {:datetime, _} is removed on v0.9+
      {k, {:datetime, :millisecond}} -> {k, "datetime[ms]"}
      {k, {:datetime, :microsecond}} -> {k, "datetime[μs]"}
      {k, {:datetime, :nanosecond}} -> {k, "datetime[ns]"}
      {k, {:naive_datetime, :millisecond}} -> {k, "datetime[ms]"}
      {k, {:naive_datetime, :microsecond}} -> {k, "datetime[μs]"}
      {k, {:naive_datetime, :nanosecond}} -> {k, "datetime[ns]"}
      {k, {:f, 32}} -> {k, "float"}
      {k, {:f, 64}} -> {k, "float"}
      {k, {:s, 8}} -> {k, "integer"}
      {k, {:s, 16}} -> {k, "integer"}
      {k, {:s, 32}} -> {k, "integer"}
      {k, {:s, 64}} -> {k, "integer"}
      {k, {:u, 8}} -> {k, "integer"}
      {k, {:u, 16}} -> {k, "integer"}
      {k, {:u, 32}} -> {k, "integer"}
      {k, {:u, 64}} -> {k, "integer"}
      {k, {:list, _}} -> {k, "list"}
      {k, {:struct, _}} -> {k, "struct"}
      {k, v} -> {k, Atom.to_string(v)}
    end)

when the data frame contains f.i. a column with a datetime with µs precision in UTC.

Could you very kindly submit a PR? Then I can cut a new release. :)

Could you very kindly submit a PR? Then I can cut a new release.

@josevalim, that would require a fix which I don't have. ;) I just started using LiveBook and (Kino) Explorer two days ago.
At first glance it looks like this is just a matter of "map the datatype to a string used for presentation". I'll see if I can find the time to confirm that and create a PR.
If anyone more experienced is faster with a PR, I won't be mad, though.

At first glance it looks like this is just a matter of "map the datatype to a string used for presentation". I'll see if I can find the time to confirm that and create a PR.

@sesamzoo yes, exactly, that should be all that is necessary!
Probably we can print it as datetime[μs, #{zone}].

Can you please try the main branch and let us know if it works? Thank you :heart:

Data transform cell for data frame with UTC datetimes work with the main branch (4f79ece) and Explorer 0.8.3.
Thanks a lot, @josevalim.

New version released!


Blacklist to exclude/ignore specific contacts

I would like to request a reconsideration of #254, "allow to mark excludes from backup".
I would prefer to have a blacklist, so that all but specific contacts are logged. This way I can exclude security tokens and other sensitive communications.
Thanks!

Incorporated in  #984


Error casting byte[] to Guid when using IncludeOptimized with Oracle and Entities with Guid Primary Keys

Hi,
Thanks for the great library!
I'm trying to use IncludeOptimized with Entities that have GUID Primary Keys. If I do this against SQL Server it works just fine, however if I do it against Oracle12c I get the following error:
System.InvalidOperationException : The specified cast from a materialized 'System.Byte[]' type to the 'System.Guid' type is not valid. at System.Data.Entity.Core.Common.Internal.Materialization.Shaper.ErrorHandlingValueReader1.GetValue(DbDataReader reader, Int32 ordinal) in
C:\systems\EntityFramework6\src\EntityFramework\Core\Common\internal\materialization\Shaper.cs:line 951 at System.Data.Entity.Core.Common.Internal.Materialization.Shaper.GetColumnValueWithErrorHandling[TColumn](Int32 ordinal) in
C:\systems\EntityFramework6\src\EntityFramework\Core\Common\internal\materialization\Shaper.cs:line 577 at lambda_method(Closure , Shaper ) at System.Data.Entity.Core.Common.Internal.Materialization.Coordinator1.ReadNextElement(Shaper shaper) in
C:\systems\EntityFramework6\src\EntityFramework\Core\Common\internal\materialization\Coordinator.cs:line 132 at System.Data.Entity.Core.Common.Internal.Materialization.Shaper1.SimpleEnumerator.MoveNext() in
C:\systems\EntityFramework6\src\EntityFramework\Core\Common\internal\materialization\Shaper.cs:line 320 at Z.EntityFramework.Plus.QueryFutureValue1.SetResult(DbDataReader reader) in C:\systems\EntityFramework-Plus\src\Z.EntityFramework.Plus.EF6\QueryFuture\QueryFutureValue.cs:line 71 at
Z.EntityFramework.Plus.QueryFutureBatch.ExecuteQueries() in C:\systems\EntityFramework-Plus\src\Z.EntityFramework.Plus.EF6\QueryFuture\QueryFutureBatch.cs:line 102 at Z.EntityFramework.Plus.QueryFutureValue1.get_Value() in
C:\systems\EntityFramework-Plus\src\Z.EntityFramework.Plus.EF6\QueryFuture\QueryFutureValue.cs:line 57 at Z.EntityFramework.Plus.QueryIncludeOptimizedProvider1.Execute[TResult](Expression expression) in
C:\systems\EntityFramework-Plus\src\Z.EntityFramework.Plus.EF6\QueryIncludeOptimized\QueryIncludeOptimizedProvider.cs:line 203 at OracleEntityFrameTest.Tests.Select_GivenIncludeOptimized_ShouldIncludeChildren() in C:\systems\OracleEntityFrameTest\OracleEntityFrameTest\Tests.cs:line 58

I used the following entities to produce that error:
public class Company
{
    public Guid Id { get; set; }
    public string Name { get; set; }
    public List<Employee> Employees { get; set; }
}

public class Employee
{
    public Guid Id { get; set; }
    public string Name { get; set; }

    public Guid CompanyId { get; set; }
    [ForeignKey(nameof(CompanyId))]
    public Company Employer { get; set; }
}

And the following code:
 var company = context.Companies
    .IncludeOptimized(c => c.Employees)
    .FirstOrDefault();

If I do a normal EntityFramework Include with those entities and context it works both on SQL Server and Oracle. It might have something with Oracle storing Guids as RAW(16) since it does not have an equivelant to SQL Server's uniqueidentifier type. However that didn't make a lot of sense because as far as I can tell EntityFramework-Plus isn't doing the conversion, EntityFramework is.
I don't know if this helps at all but I did notice that when EntityFramework reads the next element with built Include it uses different Enumerators to with IncludeOptimized.
Here is the stack track for the normal Include:
EntityFramework.dll!System.Data.Entity.Core.Common.Internal.Materialization.Coordinator<OracleEntityFrameTest.Company>.ReadNextElement(System.Data.Entity.Core.Common.Internal.Materialization.Shaper shaper) Line 115	C#
EntityFramework.dll!System.Data.Entity.Core.Common.Internal.Materialization.Shaper<OracleEntityFrameTest.Company>.RowNestedResultEnumerator.MaterializeRow() Line 498	C#
EntityFramework.dll!System.Data.Entity.Core.Common.Internal.Materialization.Shaper<OracleEntityFrameTest.Company>.RowNestedResultEnumerator.MoveNext() Line 430	C#
EntityFramework.dll!System.Data.Entity.Core.Common.Internal.Materialization.Shaper<OracleEntityFrameTest.Company>.ObjectQueryNestedEnumerator.TryReadToNextElement() Line 720	C#
EntityFramework.dll!System.Data.Entity.Core.Common.Internal.Materialization.Shaper<OracleEntityFrameTest.Company>.ObjectQueryNestedEnumerator.MoveNext() Line 585	C#
EntityFramework.dll!System.Data.Entity.Internal.LazyEnumerator<OracleEntityFrameTest.Company>.MoveNext() Line 69	C#
[External Code]
EntityFramework.dll!System.Data.Entity.Core.Objects.ELinq.ObjectQueryProvider.GetElementFunction.AnonymousMethod__14_1(System.Collections.Generic.IEnumerable<OracleEntityFrameTest.Company> sequence) Line 233	C#
EntityFramework.dll!System.Data.Entity.Core.Objects.ELinq.ObjectQueryProvider.ExecuteSingle<OracleEntityFrameTest.Company>(System.Collections.Generic.IEnumerable<OracleEntityFrameTest.Company> query, System.Linq.Expressions.Expression queryRoot) Line 217	C#
EntityFramework.dll!System.Data.Entity.Core.Objects.ELinq.ObjectQueryProvider.Execute<OracleEntityFrameTest.Company>(System.Linq.Expressions.Expression expression) Line 126	C#
EntityFramework.dll!System.Data.Entity.Internal.Linq.DbQueryProvider.Execute<OracleEntityFrameTest.Company>(System.Linq.Expressions.Expression expression) Line 90	C#
OracleEntityFrameTest.dll!OracleEntityFrameTest.Tests.Select_GivenInclude_ShouldIncludeChildren() Line 43	C#
[External Code]

Here is the stack trace for IncludeOptimized:
EntityFramework.dll!System.Data.Entity.Core.Common.Internal.Materialization.Coordinator<OracleEntityFrameTest.Company>.ReadNextElement(System.Data.Entity.Core.Common.Internal.Materialization.Shaper shaper) Line 115	C#
EntityFramework.dll!System.Data.Entity.Core.Common.Internal.Materialization.Shaper<OracleEntityFrameTest.Company>.SimpleEnumerator.MoveNext() Line 320	C#
Z.EntityFramework.Plus.EF6.dll!Z.EntityFramework.Plus.QueryFutureValue<OracleEntityFrameTest.Company>.SetResult(System.Data.Common.DbDataReader reader) Line 71	C#
Z.EntityFramework.Plus.EF6.dll!Z.EntityFramework.Plus.QueryFutureBatch.ExecuteQueries() Line 102	C#
Z.EntityFramework.Plus.EF6.dll!Z.EntityFramework.Plus.QueryFutureValue<OracleEntityFrameTest.Company>.Value.get() Line 57	C#
Z.EntityFramework.Plus.EF6.dll!Z.EntityFramework.Plus.QueryIncludeOptimizedProvider<OracleEntityFrameTest.Company>.Execute<OracleEntityFrameTest.Company>(System.Linq.Expressions.Expression expression) Line 203	C#
OracleEntityFrameTest.dll!OracleEntityFrameTest.Tests.Select_GivenIncludeOptimized_ShouldIncludeChildren() Line 58	C#
[External Code]

Thanks in advance for any help you can offer. I really tried to solve this myself before posting, was even hoping to do a pull request, but I've run out of time to try and figure it out on my own.

Hello @brendon1982 ,
Thank you for reporting this issue. I remember a similar issue about byte[] and Guid when I have added the support of Oracle to Entity Framework Extensions.
I will try to look at it very soon.
Best Regards,
Jonathan

Hello @brendon1982 ,
We successfully found a fix by using our own DbDataReader instead of the one provided by Oracle to fix the Guid issue.
However, we will still look to find a fix without using this workaround.
A fix should be available on Monday or Tuesday.
Best Regards,
Jonathan

Hello @JonathanMagnan,
I see it is already up on nuget, I'll try to update and give it another bash in the project that was having the issue tomorrow.
Again thanks so much :D

Hello @brendon1982 ,
The v1.4.23 has been released. This one should contains the fix for Oracle ;)
Let me know if that's working.
Best Regards,
Jonathan

Hi @JonathanMagnan,
Thanks for the update. I've tried the update and no longer get my original error, however I am now getting the following InvalidOperationException:
Additional information: The 'IsOpen' property on 'Company' could not be set to a 'System.Int16' value. You must set this property to a non-null value of type 'System.Boolean'
When I run the following query:
 var company = context.Companies
    .IncludeOptimized(c => c.Employees)
    .ToList();

Using the following entities:
public class Company
{
    public Guid Id { get; set; }
    public string Name { get; set; }
    public bool IsOpen { get; set; }
    public List<Employee> Employees { get; set; }
}

public class Employee
{
    public Guid Id { get; set; }
    public string Name { get; set; }

    public Guid CompanyId { get; set; }
    [ForeignKey(nameof(CompanyId))]
    public Company Employer { get; set; }
}

In Oracle my Companies table looks like this:
 COL,    TYPE,                NULLABLE
 ---------------------------------------------------
 Id,     RAW                 Not Nullable
 Name,   NVARCHAR(255 CHAR), Nullable
 IsOpen, NUMBER(1, 0),       Not Nullable

Thanks again for all the help :D

Damn!
I guess the custom OracleDataReader is not the way to go. I will make more test then to find out how to manage this.
Best Regards,
Jonathan

Damn indeed!
Sorry for taking so long to respond. I've been finishing up a project so life has been a bit hectic.
Everything is working with the standard includes however we got quite  a performance boost on SQL ever when we used the IncludeOptimized instead. I look forward to giving it another bash when it is all figured out.
Again thank you for the great library and all the effort.
Regards,
Brendon


[Direct Line] API Europe outage?

Hi,
Can anyone provide information whether DirectLine API should be running 24/7.  I experience issues when sending messages to/from a bot instance.
Most notably yesterday between 6 pm and 10 pm BST.
Here is the type of message I was receiving in my bot:
Error: Request to 'https://europe.directline.botframework.com/v3/conversations/LjU2yO53wRB67lMlHNU6Dw/activities/LjU2yO53wRB67lMlHNU6Dw%7C0000001' failed: [503] Service Unavailable
    at Request._callback (...\ChatConnector.js:452:46)
    at Request.self.callback (...request.js:186:22)
    at emitTwo (events.js:106:13)
    at Request.emit (events.js:191:7)
    at Request.<anonymous> (...request.js:1081:10)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:188:7)
    at IncomingMessage.<anonymous>(...\request.js:1001:12)
    at IncomingMessage.g (events.js:291:16)
    at emitNone (events.js:91:20)

Do you know of any planned outage? Do you expect any further outages?
This is linked to another issue I have commented on: https://github.com/Microsoft/BotBuilder/issues/507
I also experience occasional issues with calls returning 500 status when calling:
POST https://directline.botframework.com/v3/directline/conversations//activities
or
POST https://directline.botframework.com/v3/directline/conversations

We had a temporary interruption on one server in our North Europe datacenter yesterday. This server generated HTTP 500 and 503 errors for less than 30 minutes, and has been repaired. We don’t have any further interruptions planned.

@nwhitmont  The outage is aparent again for the europe service: Request to
https://europe.directline.botframework.com/v3/conversations/40Yu0ECXBGvGshq5S2p11K/activities/40Yu0ECXBGvGshq5S2p11K|0000001' failed: [503] Service Unavailable
This is the type of error I am getting. The failures seem to be fairly frequent having the last one 20 days ago.

Still getting the same error after an hour of trying? Is there any word on the European data centres outages?
Thanks

All services had interruptions between 6:30am and 7:30am.
@vasilkrastev90 @McGern You are commenting on a closed issue. In the future, please open a new issue to report problems/questions.

Sorry, thought because it's the same unresolved issue that you might open this one back up rather than having a glut of tickets that are the same,  making it harder to search and parse for the relevant issue


[NodeJS] Managing Dialog Data

Hi,
I have a tree of Dialogs:

LUIS Dialog => session.beginDialog('/sub')
1.1 Prompts + Command Dialog => session.beginDialog('/sub/sub')
1.1.1 Prompts
1.1.2 Prompts

At each level Prompts put answers in session.dialogData
When a sub Dialog ends, I call endDialog({response : subdata}) that merge with parent's dialogData
BUG ?

It seems merged subdata is overrided by current dialog data (it works if there is no further steps)
I want to populate my dialogData with my LUIS Entities but it seems session.beginDialog() put a new dialogData.

What should I do ? Should I backup everything in a userData and lost Dialog Scope ?

At last I use userData to merge data of Dialog Flow it works well. I also reset data at the top of the flow (triggered by LUIS)

Can I see a simplified version of your code? There's a issue right now where your merged dialogData may not get saved if you don't send the user a message which I suspect is what's happening.

Trying to close out some of our issues so please re-open if you're still having issues.


Show warning/disable

Display a warning, and disable functionality, on browsers non-WebRTC capable.

I'll see what I can do without having to use Peer
PR welcome
Thanks!

On Dec 7, 2015, at 1:12 PM, Fastidious<EMAIL_ADDRESS>wrote:
Display a warning, and disable functionality, on browsers non-WebRTC capable.
—
Reply to this email directly or view it on GitHub.

Isn't Safari the only one not supporting it? If so, checking for it would do it.

Old browser also don't support WebRTC, simple-peer offer a simple way to do it, but it's only referenced within WT I think

Done :)

I just exposed this as a new property: WebTorrent.WEBRTC_SUPPORT


Add basic collections data loading and table display

Description
Adds a bare-bones table for collections display on the landing page.
As an aside, given the recent UX changes to the "Append collections" functionality, I don't think there is value in making the link behavior, etc. configurable for this component from the start. The new mocks are sufficiently different enough that I think they should be separate components.
Follow ups

Implement sorting by column
Implement a table toolbar with pagination and filtering
Implement correct selection handling and delete functionality
File a PatternFly issue for the cut off checkbox layout in an empty table with the variant='compact' prop

Checklist

[ ] Investigated and inspected CI test results
[ ] Unit test and regression tests added
[ ] Evaluated and added CHANGELOG entry if required
[ ] Determined and documented upgrade steps
[ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

If any of these don't apply, please comment below.
Testing Performed
(The following tests were performed with a mock server running locally to provide data.)
Visit the collection page, and see a spinner while data loads:

If an error occurs, display a brief message to the user. (This message is the descriptive message returned by the server.)

After loading, the collections data should be displayed in the table:

If no data is returned by the request, and EmptyState is displayed with a possible user action. (No effect right now).

The table menu should display three options, with the "delete" option disabled if the collection is in use:

Selecting the "edit" option should route the user to the edit page for that collection.( No effect right now other than URL change.)

Selecting the "clone" option should route the user to the clone page for that collection.( No effect right now other than URL change.)

Clicking on the name of a collection should route the user to that collection's page, by ID. (No effect right now other than URL change.)

Current dependencies on/for this PR:

master

PR #3152

PR #3173

PR #3181   👈

PR #3155
PR #3157
PR #3154

This comment was auto-generated by Graphite.

@dvail Why did you requested a review from stackrox/merlin team?

@janisz I didn't request a review myself. I'm guessing the GitGuardian security checks added the merlin team automatically?
Nothing related to certificates was changed in this PR but it seems to be triggering on commits https://github.com/stackrox/stackrox/commit/2de65f09db1a67bd73025a584c340e0f9df92b91 and https://github.com/stackrox/stackrox/commit/fbb2196e997764ef8506a57132d0a93d6bf3af12 for some reason.
I'll remove the merlin team from review.

/retest-required


copy python dev container

PR adds repo.yaml rules to copy the python dev container.

Repoman Generation Results
Repoman pushed changes to remotes for the following projects:
Project: todo-python-mongo-terraform
Remote: azure-samples-staging
Branch: pr/575
You can initialize this project with:
azd init -t Azure-Samples/todo-python-mongo-terraform -b pr/575

View Changes | Compare Changes


How do I use this library in my project?

'using ParLibrary.Converter;' results in an error "could not be found"

Thanks for the reply!
I got the project to recognize the library but now I have an error,
in ParLibrary/Sllz/Compressor.cs line 115, I get this error:

cannot convert from 'System.IO.SeekOrigin' to 'Yarhl.IO.SeekMode' [ParLibrary]

How can I fix this?
also another question, how close did you work on RyuModManager?, fyi I'm attempting to compile that project to work with linux

The Yarhl API changed some time ago and I use the newer one in this project, but it is in alpha state, so you can't get it from nuget. You have to follow the instructions to get the preview release.
And about RyuModManager, I'm not involved with it at all.


docker compose 启动mysql 报错

系统:Centos7
docker版本:18.06.1-ce
docker-compose版本:1.20.1, build 5d8c71b
错误:
cat-mysql | Initializing database
cat-mysql | mysqld: Can't create/write to file '/var/lib/mysql/is_writable' (Errcode: 13 - Permission denied)
cat-mysql | 2019-03-15T01:13:23.527287Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
cat-mysql | 2019-03-15T01:13:23.530942Z 0 [ERROR] --initialize specified but the data directory exists and is not writable. Aborting.
cat-mysql | 2019-03-15T01:13:23.530970Z 0 [ERROR] Aborting

这个上次也有童鞋反馈 mysqld: Can’t create/write to file '/var/lib/mysql/is_writable' (Errcode: 13 - Permission denied)，之后加的 user: "1000:50" 解决。

我这边确实报错了，您还是复现一下吧。上次的同学是 win或者mac平台？

系统:Centos7
docker版本:18.06.1-ce
docker-compose版本:1.20.1, build 5d8c71b
错误:
[root@localhost docker]# docker-compose up
Building cat
Traceback (most recent call last):
File "bin/docker-compose", line 6, in
File "compose/cli/main.py", line 71, in main
File "compose/cli/main.py", line 127, in perform_command
File "compose/cli/main.py", line 1085, in up
File "compose/cli/main.py", line 1081, in up
File "compose/project.py", line 527, in up
File "compose/service.py", line 360, in ensure_image_exists
File "compose/service.py", line 1084, in build
File "site-packages/docker/api/build.py", line 159, in build
File "site-packages/docker/utils/build.py", line 31, in tar
File "site-packages/docker/utils/build.py", line 79, in create_archive
File "tarfile.py", line 1833, in gettarinfo
FileNotFoundError: [Errno 2] No such file or directory: '/proc/10/exe'
[37612] Failed to execute script docker-compose


The get vulnerabilities subcommand does not respect global flags (e.g. --kubeconfig)

What steps did you take and what happened:
The global flags to configure client-go are not respected by the get vulnerabilities subcommand:
$ starboard --kubeconfig /path/to/kube/config \
    get vulnerabilities pod/nginx-tn2wq \
    -o yaml

What did you expect to happen:
The --kubeconfig flag and other global flags to be passed to the get vulnerabilities subcommand.
Anything else you would like to add:
N/A
Environment:

Starboard version (use starboard version): 0.2.6
Kubernetes version (use kubectl version): Any
OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Any

/assign

@krisctl Are you available to work on this one? If not please let us know if we should leave it as unassigned.

@danielpacak I tried this out, and it seems like the ordering of the kubeconfig flag is an issue with your example:
kubectl starboard --kubeconfig=/path/to/my-kubeconfig get vulnerabilities pod/pod-xyz -o yaml
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

But the below one seems to work with passing the kubeconfig flag after the vulnerabilities subcommand.
kubectl starboard get vulnerabilities --kubeconfig=/path/to/my-kubeconfig pod/pod-xyz -o yaml
apiVersion: v1
items:
- apiVersion: aquasecurity.github.io/v1alpha1
  kind: VulnerabilityReport
  metadata:
    creationTimestamp: "2020-10-06T23:04:00Z"
    generation: 1
    labels:
-----

@danielpacak It seems like the example you have shared is missing the namespace flag. I tried to run the same command by providing the flag and I seem to get some output. Can you please share logs of what output/error you get when you run the command?
./bin/starboard --kubeconfig=/path/to/kubeconfig get vulnerabilities pod/pod-xyz -n=ns1 -o yaml

@danielpacak can you please share the logs from when you run the command?

We don't have meaningful logs output as this command fails immediately due to connection error with Kubernetes API. To reproduce you can do the following:
$ mv ~/.kube/config ~/.kube/config2
./bin/starboard get vuln deploy/nginx
error: an error on the server ("") has prevented the request from succeeding

Here's the actual issue, i.e. even if we specify the right location of kubeconfig file it's ignored by starboard get vulns command. If you check the code you'll see that the implementation is a bit hacky because we spawn kubectl get command, but we do not pass all flags such as --kubeconfig:
$ ./bin/starboard --kubeconfig ~/.kube/config2 get vuln deploy/nginx
error: exit status 1

$ kubectl get pod --kubeconfig ~/.kube/config2
NAME                         READY   STATUS    RESTARTS   AGE
nginx-db749865c-xfw7h        1/1     Running   1          21h
wordpress-855854854c-c684g   1/1     Running   2          40h

/assign


A small suggestion about Radxa Zero 3

A small suggestion, pins 27 and 28 on the Radxa Zero 3 can be used as USB pins to communicate with the network card. When designing buttons or other functions, could you avoid using these pins? Just like I2C, UART, or USB, etc.

see my recommend

Future builds will move the recording button function. To be determined, probably to be compatible with RubyFPV.


fix: 10.12.0 patch

DO NOT MERGE
Removes any references to referrer in android code. Hopefully this will reduce the number of AORs.
Relevant issue: https://github.com/react-native-device-info/react-native-device-info/issues/1377

@Gongreg I believe you should add lib folder to gitignore, in you PR it is commented out.

@ashwanihundwani1986 , simce this patch is not published anywhere and we use it as a git dependency, we included lib directory into commit so that the project is built.

是个狠人


Issue Linking TraceViewer packages

When trying to link packages 'traceviewer-base' and 'traceviewer-react-components' from 'theia-trace-extension', the resources view throws the following error:

It could be because of a mismatch in React and React-DOM versions in theia-trace-extension and vscode-trace-extension. Another reason could be that packages were installed at a different level in the following PR: https://github.com/eclipse-cdt-cloud/theia-trace-extension/pull/886

We believe this issue should be fixed, with the latest traceviewer-react-components, that's being built by CI right now [1]. If it's not the case, please re-open:
[1]  https://github.com/eclipse-cdt-cloud/theia-trace-extension/actions/runs/6318318877/job/17157047891

The change doesn't seem to fix all the problems. I tried it with the latest traceviewer-base/traceviewer-react-components but and I get the following error:
vscode-trace-webviews:   ../vscode-trace-common/lib/server/trace-server-url-provider.js 572 bytes [built] [code generated]
vscode-trace-webviews: ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-explorer/available-views/index.tsx
vscode-trace-webviews: ./src/trace-explorer/available-views/index.tsx 7:13-37
vscode-trace-webviews: [tsl] ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-explorer/available-views/index.tsx(7,14)
vscode-trace-webviews:       TS2786: 'TraceExplorerViewsWidget' cannot be used as a JSX component.
vscode-trace-webviews:   Its instance type 'TraceExplorerViewsWidget' is not a valid JSX element.
vscode-trace-webviews:     The types returned by 'render()' are incompatible between these types.
vscode-trace-webviews:       Type 'React.ReactNode' is not assignable to type 'import("/home/user/git/theia-training/theia-trace-extension/node_modules/@types/react/ts5.0/index").ReactNode'.
vscode-trace-webviews: ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-explorer/opened-traces/index.tsx
vscode-trace-webviews: ./src/trace-explorer/opened-traces/index.tsx 8:13-38
vscode-trace-webviews: [tsl] ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-explorer/opened-traces/index.tsx(8,14)
vscode-trace-webviews:       TS2786: 'TraceExplorerOpenedTraces' cannot be used as a JSX component.
vscode-trace-webviews:   Its instance type 'TraceExplorerOpenedTraces' is not a valid JSX element.
vscode-trace-webviews:     The types returned by 'render()' are incompatible between these types.
vscode-trace-webviews:       Type 'React.ReactNode' is not assignable to type 'import("/home/user/git/theia-training/theia-trace-extension/node_modules/@types/react/ts5.0/index").ReactNode'.
vscode-trace-webviews: ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-explorer/properties/index.tsx
vscode-trace-webviews: ./src/trace-explorer/properties/index.tsx 13:13-36
vscode-trace-webviews: [tsl] ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-explorer/properties/index.tsx(13,14)
vscode-trace-webviews:       TS2786: 'TraceExplorerProperties' cannot be used as a JSX component.
vscode-trace-webviews:   Its instance type 'TraceExplorerProperties' is not a valid JSX element.
vscode-trace-webviews:     The types returned by 'render()' are incompatible between these types.
vscode-trace-webviews:       Type 'React.ReactNode' is not assignable to type 'import("/home/user/git/theia-training/theia-trace-extension/node_modules/@types/react/ts5.0/index").ReactNode'.
vscode-trace-webviews: ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-explorer/shortcuts/index.tsx
vscode-trace-webviews: ./src/trace-explorer/shortcuts/index.tsx 8:13-36
vscode-trace-webviews: [tsl] ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-explorer/shortcuts/index.tsx(8,14)
vscode-trace-webviews:       TS2786: 'ChartShortcutsComponent' cannot be used as a JSX component.
vscode-trace-webviews:   Its instance type 'ChartShortcutsComponent' is not a valid JSX element.
vscode-trace-webviews:     The types returned by 'render()' are incompatible between these types.
vscode-trace-webviews:       Type 'React.ReactNode' is not assignable to type 'import("/home/user/git/theia-training/theia-trace-extension/node_modules/@types/react/ts5.0/index").ReactNode'.
vscode-trace-webviews: ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-explorer/time-range/index.tsx
vscode-trace-webviews: ./src/trace-explorer/time-range/index.tsx 7:13-32
vscode-trace-webviews: [tsl] ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-explorer/time-range/index.tsx(7,14)
vscode-trace-webviews:       TS2786: 'TimeRangeDataWidget' cannot be used as a JSX component.
vscode-trace-webviews:   Its instance type 'TimeRangeDataWidget' is not a valid JSX element.
vscode-trace-webviews:     The types returned by 'render()' are incompatible between these types.
vscode-trace-webviews:       Type 'React.ReactNode' is not assignable to type 'import("/home/user/git/theia-training/theia-trace-extension/node_modules/@types/react/ts5.0/index").ReactNode'.
vscode-trace-webviews: ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-viewer/index.tsx
vscode-trace-webviews: ./src/trace-viewer/index.tsx 8:13-33
vscode-trace-webviews: [tsl] ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-viewer/index.tsx(8,14)
vscode-trace-webviews:       TS2786: 'TraceViewerContainer' cannot be used as a JSX component.
vscode-trace-webviews:   Its instance type 'TraceViewerContainer' is not a valid JSX element.
vscode-trace-webviews:     The types returned by 'render()' are incompatible between these types.
vscode-trace-webviews:       Type 'React.ReactNode' is not assignable to type 'import("/home/user/git/theia-training/theia-trace-extension/node_modules/@types/react/ts5.0/index").ReactNode'.
vscode-trace-webviews:         Type 'ReactElement<any, string | JSXElementConstructor<any>>' is not assignable to type 'ReactNode'.
vscode-trace-webviews:           Property 'children' is missing in type 'ReactElement<any, string | JSXElementConstructor<any>>' but required in type 'ReactPortal'.
vscode-trace-webviews: ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-viewer/vscode-trace-viewer-container.tsx
vscode-trace-webviews: ./src/trace-viewer/vscode-trace-viewer-container.tsx 465:21-42
vscode-trace-webviews: [tsl] ERROR in /home/user/git/vscode-trace-extension/vscode-trace-webviews/src/trace-viewer/vscode-trace-viewer-container.tsx(465,22)
vscode-trace-webviews:       TS2786: 'TraceContextComponent' cannot be used as a JSX component.
vscode-trace-webviews:   Its instance type 'TraceContextComponent' is not a valid JSX element.
vscode-trace-webviews:     The types returned by 'render()' are incompatible between these types.
vscode-trace-webviews:       Type 'Element' is not assignable to type 'ReactNode'.
vscode-trace-webviews:         Property 'children' is missing in type 'Element' but required in type 'ReactPortal'.
vscode-trace-webviews:  @ ./src/trace-viewer/index.tsx 3:0-67 6:32-52
vscode-trace-webviews: webpack 5.88.2 compiled with 7 errors in 8501 ms
error Command failed with exit code 1.

Picking this issue back up.

The cause of the issue: React Issue #14257 - Hooks don't work with yarn link
When we use yarn-link there are two sources of react and react-dom being used.  One from theia-trace-extension and one from vscode-trace-extension.  I believe the error comes from a hook call in the @mui/material package which was added in the feature that @hriday-panchasara mentioned in the description here.
I was able to fix this issue by linking the directory for react and react-dom used in theia-trace-extension to vscode-trace-extension and /node_modules/@mui/:
cd theia-trace-extension/node_modules/react
yarn link
cd ../../node_modules/react-dom
yarn link

Navigate to vscode-trace-extension...
cd vscode-trace-extension
yarn link react
yarn link react-dom
cd /node_modules/@mui
yarn link react
yarn link react-dom

This stops the error from occurring while developing both repos simultaneously.
That being said, I think there are a lot of opportunities to optimize the development process of this repository :).

There may be other parts of the code that will throw this issue, and react + react-dom will need to be linked as well. There is probably a way to configure the webpack.js or something to automatically point all react + react-dom imports to the same file.
I'll try to look into that at some point.

@williamsyang-work thanks for the investigation and finding this work-around. Let's keep this issue open, until we have a programmatic fix.


FormBuilder.attempt_mapping is taking > 1 second

Environment

Ruby 3.1.2
Rails 7.0.4
Simple Form 5.1.0

Current behavior
This may be related to our Gemfile, and some additional gems that are extending the exception message processing, however what we are seeing is that in FormBuilder
    def attempt_mapping(mapping, at)
      return if SimpleForm.inputs_discovery == false && at == Object

      begin
        at.const_get(mapping)
      rescue NameError => e
        raise if e.message !~ /#{mapping}$/
      end
    end

the raise if e.message !~ /#{mapping}$/ is taking over 1000ms for the first lookup of that type of field (e.g. NumericInput) on a form. I believe that it's attempting to check if the exception message contains the name of the class that was searched for, e.g. (from my hacking about earlier to see what the issue was)
14:54:44 web.1                        | "NumericInput"
14:54:44 web.1                        | Object
14:54:44 web.1                        | #<NameError: uninitialized constant NumericInput
14:54:44 web.1                        |
14:54:44 web.1                        |         res = at.const_get(mapping)
14:54:44 web.1                        |                 ^^^^^^^^^^>
14:54:44 web.1                        |
14:54:44 web.1                        |
14:54:44 web.1                        | SJH attempt_mapping RAISE TIME = 1159.357387

If I change the above line of code to raise unless e.is_a? NameError then the execution time drops to < 1 ms, e.g.
15:42:17 web.1                        | SJH attempt_mapping RAISE TIME = 0.008567

I believe that this keeps the same behaviour, as it will raise the exception if there's anything other than the NameError that was being checked for.
I don't believe the slow down is due to simple_form, the line of code in question hasn't changed in 10 years, and we can see the same slow down if we just log out the exception message in the function, e.g.
puts "SJH attempt_mapping #{e.inspect}"

also takes 1000 ms. But I believe the change from raise if e.message !~ /#{mapping}$/ to raise unless e.is_a? NameError should be relatively low risk to implement in simple_form and would prevent others from tripping over this. We aren't experiencing this behaviour elsewhere in our application as we handling exceptions based on their type rather than their contents.

I'm reverting the merged change, as it caused a regression. See #1824 for more info.
Happy to review something else that fixes the original problem without the regression, but for now we'll have to live with it.


Add depthwise conv2d for Theano and CNTK

This PR enables DepthwiseConv2D, initially designed for TensorFlow only, to work on Theano and CNTK. This PR includes the following:

moving class DepthwiseConv2D from keras/applications/mobilenet.py to keras/layers/convolutional.py without modification,
adding depthwise_conv2d for Theano and CNTK,
revising test_depthwise_conv_2d in backend_test.py with respect to existing backend test styles,
adding test_depthwise_conv_2d in convolutional_test.py.

Sounds great! Have you checked that MobileNet inference with imagenet weights gives the expected results with Theano and CNTK?
Merging your other PR  has introduced a merge conflict with this one, please fix.

@fchollet, The conflict has been resolved. The inference results are the following: Rather, the outputs of TensorFlow look strange to me.
 $ cat test_mobilenet.py
from keras.applications.mobilenet import MobileNet
from keras.preprocessing import image
from keras.applications.mobilenet import preprocess_input, decode_predictions
import numpy as np

model = MobileNet(weights='imagenet')

img = image.load_img('cat.png', target_size=(224, 224))
x = image.img_to_array(img)
x = np.expand_dims(x, axis=0)
x = preprocess_input(x)

print('Predicted:', decode_predictions(model.predict(x), top=3)[0])

 $ KERAS_BACKEND=cntk python test_mobilenet.py
Using CNTK backend
Selected GPU[0] Tesla P100-PCIE-16GB as the process wide default device.
('Predicted:', [(u'n02123159', u'tiger_cat', 0.25886607), (u'n02124075', u'Egyptian_cat', 0.14433344), (u'n02123045', u'tabby', 0.06473238)])

 $ KERAS_BACKEND=tensorflow python test_mobilenet.py
Using TensorFlow backend.
('Predicted:', [(u'n03482405', u'hamper', 0.21035318), (u'n02123159', u'tiger_cat', 0.11360751), (u'n04074963', u'remote_control', 0.09508653)])

 $ KERAS_BACKEND=theano python test_mobilenet.py
Using Theano backend.
('Predicted:', [(u'n02123159', u'tiger_cat', 0.2602814), (u'n02124075', u'Egyptian_cat', 0.14234008), (u'n02123045', u'tabby', 0.06521355)])

It's orthogonal to this PR, but we should try to determine at what commit the discrepancy arose. A possible candidate would be some recent modifications to BN in TF. It could also be a preprocessing issue.

I think that this PR is enough by itself because all the backend and layer tests were passed. As you said @fchollet, I'll try to investigate the discrepancy for each application. Currently,

TensorFlow differs in MobileNet, ResNet50
Theano differs in InceptionV3
OK in VGG16, VGG19, DenseNet121
Versions: Keras 2.1.3, TensorFlow 1.3.0, Theano 1.0.1, CNTK 2.3.1

The results are the following:
 $ KERAS_BACKEND=theano python test_resnet50.py
Using Theano backend.
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.17173238), (u'n02123597', u'Siamese_cat', 0.10258683), (u'n02130308', u'cheetah', 0.0565808)])

 $ KERAS_BACKEND=cntk python test_resnet50.py
Using CNTK backend
Selected GPU[0] Tesla P100-PCIE-16GB as the process wide default device.
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.17055313), (u'n02123597', u'Siamese_cat', 0.11803201), (u'n03482405', u'hamper', 0.055440076)])

 $ KERAS_BACKEND=tensorflow python test_resnet50.py
Using TensorFlow backend.
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.116303556), (u'n02123597', u'Siamese_cat', 0.075559795), (u'n02130308', u'cheetah', 0.048775144)])

 $ KERAS_BACKEND=cntk python test_inception3.py
Using CNTK backend
Selected GPU[0] Tesla P100-PCIE-16GB as the process wide default device.
('Predicted:', [(u'n02123045', u'tabby', 0.69501376), (u'n02123159', u'tiger_cat', 0.11184513), (u'n02356798', u'fox_squirrel', 0.069066964)])

 $ KERAS_BACKEND=tensorflow python test_inception3.py
Using TensorFlow backend.
('Predicted:', [(u'n02123045', u'tabby', 0.70286405), (u'n02123159', u'tiger_cat', 0.110841505), (u'n02356798', u'fox_squirrel', 0.067352444)])

 $ KERAS_BACKEND=theano python test_inception3.py
Using Theano backend.
('Predicted:', [(u'n02123045', u'tabby', 0.6191356), (u'n02123159', u'tiger_cat', 0.14174756), (u'n02124075', u'Egyptian_cat', 0.10547265)])

 $ KERAS_BACKEND=cntk python test_vgg16.py
Using CNTK backend
Selected GPU[0] Tesla P100-PCIE-16GB as the process wide default device.
('Predicted:', [(u'n01687978', u'agama', 0.12963562), (u'n01675722', u'banded_gecko', 0.09029942), (u'n02317335', u'starfish', 0.086037084)])

 $ KERAS_BACKEND=tensorflow python test_vgg16.py
Using TensorFlow backend.
('Predicted:', [(u'n01687978', u'agama', 0.12963554), (u'n01675722', u'banded_gecko', 0.09029918), (u'n02317335', u'starfish', 0.08603718)])

 $ KERAS_BACKEND=theano python test_vgg16.py
Using Theano backend.
('Predicted:', [(u'n01687978', u'agama', 0.12963551), (u'n01675722', u'banded_gecko', 0.090299465), (u'n02317335', u'starfish', 0.086037494)])

 $ KERAS_BACKEND=cntk python test_vgg19.py
Using CNTK backend
Selected GPU[0] Tesla P100-PCIE-16GB as the process wide default device.
('Predicted:', [(u'n02137549', u'mongoose', 0.16966635), (u'n03124170', u'cowboy_hat', 0.088505425), (u'n01496331', u'electric_ray', 0.08285297)])

 $ KERAS_BACKEND=tensorflow python test_vgg19.py
Using TensorFlow backend.
('Predicted:', [(u'n02137549', u'mongoose', 0.16966705), (u'n03124170', u'cowboy_hat', 0.088505276), (u'n01496331', u'electric_ray', 0.082852826)])

 $ KERAS_BACKEND=theano python test_vgg19.py
Using Theano backend.
('Predicted:', [(u'n02137549', u'mongoose', 0.16966648), (u'n03124170', u'cowboy_hat', 0.088506155), (u'n01496331', u'electric_ray', 0.082852475)])

 $ KERAS_BACKEND=cntk python test_densenet121.py
Using CNTK backend
Selected GPU[0] Tesla P100-PCIE-16GB as the process wide default device.
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.30215406), (u'n02123394', u'Persian_cat', 0.08100653), (u'n02123045', u'tabby', 0.04857644)])

 $ KERAS_BACKEND=tensorflow python test_densenet121.py
Using TensorFlow backend.
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.2955615), (u'n02123394', u'Persian_cat', 0.08364624), (u'n02123045', u'tabby', 0.048363034)])

 $ KERAS_BACKEND=theano python test_densenet121.py
Using Theano backend.
('Predicted:', [(u'n02124075', u'Egyptian_cat', 0.29556265), (u'n02123394', u'Persian_cat', 0.08364609), (u'n02123045', u'tabby', 0.048363138)])

These predictions seem all in line with each other. There are small numerical differences between backends (as well as a potential small implementation difference with pooling windows for Theano), which can result with small prediction differences for one specific picture.
The TF MobileNet seems broken though. The predictions are not random, but it looks like we have a serious issue.