doc: restructure module documentation

Summary
Go module documentation is spread out over several sources. The canonical source is go help, but it's difficult to read in a terminal, and the HTML equivalent (golang.org/cmd/go) is hard to discover and lacks structure and examples. Many developers seem to have difficulty finding solutions to common problems and discovering reference material.
This issue proposes restructuring module documentation into three main areas.

A specification should be added as one or more documents linked from the References section of golang.org/doc. The specification should be written in the main Go repository, and updates should be linked to the Go release cycle.
A quick reference should remain in the go command with a brief description of each command and each flag. go help modules and other pages should be shortened, pointing to the specification where appropriate. The quick reference should be mirrored at golang.org/cmd/doc, and updates should be linked to the Go release cycle.
A series of guide articles should be published to help developers learn how to start using modules and how to solve specific problems with modules. Guide articles should be linked from the Learning Go section of golang.org/doc. Guide articles may be published outside the Go release cycle and should be kept up to date with current features and best practices.

Specification
A complete reference for everything related to modules should be added as one or more structured documents linked from golang.org/doc under the References section. This is analogous to the Language specification or The Go memory model.
The specification should be written as HTML documents within the main Go repository. Updates on the website will be tied to the Go release cycle. This is desirable since things like the go.mod format and the proxy protocol are tied to the current release. The latest version of the documentation will always be visible from tip.golang.org.
The specification should include the following sections. Note that this is simply a rough list of headings to establish scope, not an outline, which will come later.

Introduction

Motivation, problems solved (based on https://research.swtch.com/vgo-intro) - very brief description of what modules are and why they exist.
Preliminary module support (from go help modules) - how to enable or disable module aware mode.
Defining a module (from go help modules) - overview of using modules, with an example go.mod file and mention of go mod init.
The main module and the build list (from go help modules) - defining terminology.
Pseudo-versions (from go help modules) - definition and format.
Module queries (from go help modules) - list of @ suffixes accepted by go get, go mod download, etc.
Upgrading and downgrading with go get - more detailed description of go get flags and use of module queries. go help get will link here.
Module compatibility and semantic versioning - explanation of semantic versioning, need for major version suffixes, +incompatible version suffix.

Mechanics of downloading, verifying, and storing modules

Module code layout (from go help modules) - how modules are extracted from vcs repo. Currently a pointer to https://research.swtch.com/vgo-module
Module download and verification (from go help modules) - description of how modules are downloaded and verified against go.sum.
Module replace semantics
Modules and vendoring (from go help modules) - go mod vendor and -mod=vendor. We will need to define vendoring without relying on prior knowledge or GOPATH documentation.
Minimal version selection (based on https://research.swtch.com/vgo-mvs) - how versions are chosen
Module authentication - go.sum and checksum database.
Private modules (from go help module-private) - hosting and accessing private modules. Currently, this is just about configuring the go command to avoid leaking information about private modules to the proxy and sumdb. We should include GOAUTH (when implemented; see #26232) and other mechanisms (.netrc, git config) for connecting to and authenticating with private servers.

Formats and protocols

go.mod format (from go help go.mod)
go.sum format
GOPROXY protocol (from go help goproxy) - Should describe fallback behavior.
GOSUMDB protocol.

Quick reference
A quick reference for the Go command should be preserved in go help and golang.org/cmd/go/, which is generated from the same text. The quick reference should briefly describe each command and each flag. Quick reference pages may link to sections in the specification or occasionally guide articles, but they should not link to the wiki or external sources that may change or disappear.
As with the specification, updates to the quick reference on the website will be tied to the Go release cycle.
The following changes should be made to go help:

environment - should provide a brief description of GOPROXY family variables, referring to 'go help goproxy' for more details.
get (module-get) - Should be slightly more concise. Flags should be indented and descriptions should be shortened (like go help build) to make them easier to find quickly. Reference documentation should be linked for details.
mod download - No change needed.
mod edit - Flags should be indented (like go help build). Content is already concise.
mod graph - No change needed.
mod init - Should define the module path and explain what it should look like. Consider linking to an introductory guide article.
mod tidy - No change needed.
mod vendor - Should link to "Modules and vendoring" in reference documentation.
mod verify - Should link to "Module authentication".
mod why - Flags should be indented. Content is already concise.
goproxy - Should provide a brief description of all proxy environment variables. Should link to reference documentation for proxy protocol, sumdb protocol, Module authentication, etc.
modules - Should provide a concise introduction (~200-500 words) and link to reference documentation.
modules-auth - Should link to "Module authentication" in reference documentation.
modules-private - Should link to "Private modules" in reference documentation.

Guide articles
Guide articles should teach developers how to accomplish specific tasks. Each article should focus on a specific topic or small group of related topics. Articles may be more verbose than reference documentation and should include more realistic examples.
Unlike reference documentation, guide articles may be published outside of the Go release cycle. Unlike blog articles, guide articles should be updated after they are published to include new features and best practices. Blog articles may be converted to guide articles after they are published.
Guide articles should be linked from the Learning Go section of golang.org/doc. See the Hosting section below for possible hosting locations.
The following is a non-exhaustive list of articles to be written.

Using Go Modules - An introduction to modules. Should be copied mostly as-is with dates and versions removed or updated.
How to Write Go Code (#28215) - Currently is an introduction to GOPATH, aimed at new Go developers. Should be rewritten as an introduction to modules. This has some overlap with Using Go Modules, but the audience is new Go developers, rather than experienced Go developers who are new to modules.
Publishing Go modules (blog post in progress) - a guide for releasing a new version of a module. Semantic versions, semantic import versioning, tag format for modules not at repository root. How to push new major version.
Migrating to Go modules - how to migrate from dep and other systems. Discusses  migration with go mod init and limitations. Considerations for v2+ modules. Compatibility with GOPATH (minimal module compatibility).
v2 and beyond (blog post planned) - Migration and publishing considerations for v2+ modules.
Tools for understanding modules - how to answer common questions using go list [-m] [-json], go mod why [-m], go mod graph.
Debugging Go Modules - a guide for how to debug module failures. See https://github.com/golang/go/wiki/Modules#faqs--possible-problems for some existing topics.
Multi module repositories - a guide for managing multi-module repositories. (see https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories)
Maintaining module requirements - a guide for managing dependencies with modules, loosely based on go help modules. Explains how to accomplish common tasks with go get and go mod tidy in more detail than the above documents.
How to upgrade to a new major version of a dependency - Should cover any details left out of Migrating to Go modules. Should keep up with tools that assist with this.
Local modifications to dependencies - walks through creating a short-lived fork of a dependency, fixing an issue, submitting a PR, upgrading and unforking once the PR is merged. May be combined with Maintaining module requirements, depending on level of detail.
Managing private modules - authentication with private repos, private proxies, -insecure flag.

Hosting
The golang.org website is served using golang.org/x/website/cmd/golangorg. Currently, this binary has some static content built into the content/static package. This includes some HTML, JS, CSS, and templates. Most content is extracted from the Go repository itself. For example, golang.org/doc is doc/docs.html.
The blog is served on a separate site, blog.golang.org using the binary golang.org/x/blog. Content is written in the content directory in golang.org/x/tools/present format.
At the moment, there is no place for non-blog content to be published outside the Go release cycle. We are considering a few locations:

The doc/articles directory of the main Go repository. x/website would need to serve pages in this directory at master instead of from the latest release.
A directory within golang.org/x/website itself. Currently, this repository has not content (just templates, CSS, and JS). This would reduce the need to serve pages from multiple versions of the Go repository, but we'd at least need to serve doc/docs.html from tip in order to update links outside the release cycle.
A separate documentation site, to be determined.

Whereever we end up hosting articles, authors should be able to write in Markdown or a similar convenient format, rather than raw HTML.

cc @bcmills @jadekler @rsc @tbpg @thepudds

@myitcv noticed that go help modules does mention GOPROXY's special off value, but go help goproxy doesn't. I think the latter help page is where one would look first.

Is there any reason for this issue to go through the proposal process?  It seems like something to be decided by the cmd/go maintainers.

@ianlancetaylor, mostly I think we'd like feedback on where to put new documentation that isn't tied to the release cycle: in doc/articles in the main go repo, in x/website, or some new documentation repository (such as x/doc)?
I think our preference would be a new repository, so that the main go repo can remain synchronized to the release cycle and x/website can remain focused on mechanism rather than content.

OK.
The latter point is related to #29206.

godoc -http=localhost:6060 -goroot=$HOME/sdk/go1.14
Errors:
2020/02/26 09:13:59 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:14:02 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:14:03 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:14:03 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:14:07 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:14:08 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:14:08 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:14:08 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:16:46 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:16:48 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:16:49 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:16:49 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData
2020/02/26 09:16:50 template: root.tmpl:6:2: executing "root" at <.AnalyticsHTML>: can't evaluate field AnalyticsHTML in type blog.rootData

I downloaded go get golang.org/x/website how do I now view the doc.html?

@AlphaHot Can you please file a new issue and fill in the answers to “what did you do”, “what did you expect to see”, and “what happened instead”? It will be easier to investigate with more information. Thanks.

description of how modules are downloaded

Please also document in one place the algorithm go get uses for getting a new module and for installing a module already specified in go.mod (probably a subset of former).

From #38985, we should have some documentation on branching models, mapping between versions and branches, and dealing with pseudo-versions in development. In particular, there are some tricks that may need to be employed when tagging releases off release branches (not master) like this comment.

From a comment in CL 243077, we need an article on how and when to use replace directives. It should explain what the implications are for other modules that depend on a module with replace, how to test with and without replace, and how to avoid replace in released versions.

We are pretty aggressive internally about not allowing replace in non-main modules, and have found it is easier to just disallow it. Valid use cases are either hacking/testing deps, something that should not be released, or upstream bug fixes, where we prefer temporary, internal forks. Would it be reasonable for gorelease to warn or error on any?

I's not necessarily harmful to keep replace directives around, as long as you've tested that your module works without them. But I suspect that's rarely the case, so I'm inclined to agree: gorelease should report a warning.
If we decide to go forward with #40276, there's a much stronger argument: go get -b will report an error when building an executable from a module with a replace. You really wouldn't want to release a version with a replace left over if anyone might go get -b something inside it.

I have proposed some additions to the base documentation that pertains, I think, to this issue, I have not approached the writing of a tutorial, as is seems that the nature of the public module proxies are rather subjectively dependant upon the system that a user is working with. In the case of many novice programmers, many will be unaware that they are using a cached proxy, and even less so, of the permanent nature of a version creation.
As such I have made some suggestive additions to the modules documentation to bring some aspects of the module and versioning system to the users attention sooner.
https://github.com/golang/website/pull/51
https://github.com/golang/go/pull/46192

I think this is now go.dev/ref/mod


HDDS-1232 : Recon Container DB service definition.

The attached patch contains the Recon container DB service definition (Interface + Impl). It uses a RocksDB instance to store the container-Key information. The interface is still evolving at this point of time, and maybe updated when the end to end work is done to read the OM DB snapshot and write the container-key data into this DB.
Unit testing done. Manual testing will be done along with HDDS-1233 and HDDS-1234.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
25
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
60
Maven dependency ordering for branch

+1
mvninstall
1087
trunk passed

+1
compile
967
trunk passed

+1
checkstyle
179
trunk passed

+1
mvnsite
116
trunk passed

+1
shadedclient
927
branch has no errors when building and testing our client artifacts.

+1
findbugs
133
trunk passed

+1
javadoc
78
trunk passed

_ Patch Compile Tests _

0
mvndep
20
Maven dependency ordering for patch

+1
mvninstall
80
the patch passed

+1
compile
876
the patch passed

+1
javac
876
the patch passed

+1
checkstyle
184
the patch passed

+1
mvnsite
92
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
1
The patch has no ill-formed XML file.

+1
shadedclient
636
patch has no errors when building and testing our client artifacts.

-1
findbugs
42
hadoop-ozone/ozone-recon generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)

+1
javadoc
78
the patch passed

_ Other Tests _

-1
unit
65
common in the patch failed.

+1
unit
33
framework in the patch passed.

+1
unit
24
ozone-recon in the patch passed.

+1
asflicense
33
The patch does not generate ASF License warnings.

5802

Reason
Tests

FindBugs
module:hadoop-ozone/ozone-recon

Found reliance on default encoding in org.apache.hadoop.ozone.recon.spi.impl.ContainerDBServiceProviderImpl.getCountForForContainerKeyPrefix(ContainerKeyPrefix):in org.apache.hadoop.ozone.recon.spi.impl.ContainerDBServiceProviderImpl.getCountForForContainerKeyPrefix(ContainerKeyPrefix): String.getBytes()  At ContainerDBServiceProviderImpl.java:[line 79]

Found reliance on default encoding in org.apache.hadoop.ozone.recon.spi.impl.ContainerDBServiceProviderImpl.getKeyPrefixesForContainer(long):in org.apache.hadoop.ozone.recon.spi.impl.ContainerDBServiceProviderImpl.getKeyPrefixesForContainer(long): new String(byte[])  At ContainerDBServiceProviderImpl.java:[line 102]

Found reliance on default encoding in org.apache.hadoop.ozone.recon.spi.impl.ContainerDBServiceProviderImpl.getKeyPrefixesForContainer(long):in org.apache.hadoop.ozone.recon.spi.impl.ContainerDBServiceProviderImpl.getKeyPrefixesForContainer(long): String.getBytes()  At ContainerDBServiceProviderImpl.java:[line 97]

Found reliance on default encoding in org.apache.hadoop.ozone.recon.spi.impl.ContainerDBServiceProviderImpl.storeContainerKeyMapping(ContainerKeyPrefix, Integer):in org.apache.hadoop.ozone.recon.spi.impl.ContainerDBServiceProviderImpl.storeContainerKeyMapping(ContainerKeyPrefix, Integer): String.getBytes()  At ContainerDBServiceProviderImpl.java:[line 61]

Failed junit tests
hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-569/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/569

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux d0fa55847bd2 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 1bc282e

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/1/artifact/out/new-findbugs-hadoop-ozone_ozone-recon.html

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/1/artifact/out/patch-unit-hadoop-hdds_common.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/1/testReport/

Max. process+thread count
445 (vs. ulimit of 5500)

modules
C: hadoop-hdds/common hadoop-hdds/framework hadoop-ozone/ozone-recon U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/1/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
26
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
25
Maven dependency ordering for branch

+1
mvninstall
1025
trunk passed

+1
compile
973
trunk passed

+1
checkstyle
193
trunk passed

+1
mvnsite
133
trunk passed

+1
shadedclient
1046
branch has no errors when building and testing our client artifacts.

+1
findbugs
169
trunk passed

+1
javadoc
119
trunk passed

_ Patch Compile Tests _

0
mvndep
22
Maven dependency ordering for patch

+1
mvninstall
96
the patch passed

+1
compile
941
the patch passed

+1
javac
941
the patch passed

+1
checkstyle
187
the patch passed

+1
mvnsite
103
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
1
The patch has no ill-formed XML file.

+1
shadedclient
612
patch has no errors when building and testing our client artifacts.

-1
findbugs
45
hadoop-ozone/ozone-recon generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)

+1
javadoc
90
the patch passed

_ Other Tests _

-1
unit
68
common in the patch failed.

+1
unit
38
framework in the patch passed.

+1
unit
34
ozone-recon in the patch passed.

+1
asflicense
35
The patch does not generate ASF License warnings.

6054

Reason
Tests

FindBugs
module:hadoop-ozone/ozone-recon

Found reliance on default encoding in org.apache.hadoop.ozone.recon.spi.impl.ContainerDBServiceProviderImpl.getKeyPrefixesForContainer(long):in org.apache.hadoop.ozone.recon.spi.impl.ContainerDBServiceProviderImpl.getKeyPrefixesForContainer(long): new String(byte[])  At ContainerDBServiceProviderImpl.java:[line 120]

Failed junit tests
hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-569/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/569

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux 773b2c016654 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 1bc282e

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/2/artifact/out/new-findbugs-hadoop-ozone_ozone-recon.html

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/2/artifact/out/patch-unit-hadoop-hdds_common.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/2/testReport/

Max. process+thread count
412 (vs. ulimit of 5500)

modules
C: hadoop-hdds/common hadoop-hdds/framework hadoop-ozone/ozone-recon U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/2/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
28
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
23
Maven dependency ordering for branch

+1
mvninstall
1016
trunk passed

+1
compile
942
trunk passed

+1
checkstyle
197
trunk passed

+1
mvnsite
130
trunk passed

+1
shadedclient
1056
branch has no errors when building and testing our client artifacts.

+1
findbugs
175
trunk passed

+1
javadoc
115
trunk passed

_ Patch Compile Tests _

0
mvndep
21
Maven dependency ordering for patch

+1
mvninstall
92
the patch passed

+1
compile
909
the patch passed

+1
javac
909
the patch passed

+1
checkstyle
199
the patch passed

+1
mvnsite
112
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
1
The patch has no ill-formed XML file.

+1
shadedclient
608
patch has no errors when building and testing our client artifacts.

+1
findbugs
175
the patch passed

+1
javadoc
85
the patch passed

_ Other Tests _

-1
unit
67
common in the patch failed.

+1
unit
39
framework in the patch passed.

-1
unit
25
ozone-recon in the patch failed.

+1
asflicense
41
The patch does not generate ASF License warnings.

5990

Reason
Tests

Failed junit tests
hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-569/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/569

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux 29efa507691f 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 064f38b

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/3/artifact/out/patch-unit-hadoop-hdds_common.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/3/artifact/out/patch-unit-hadoop-ozone_ozone-recon.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/3/testReport/

Max. process+thread count
444 (vs. ulimit of 5500)

modules
C: hadoop-hdds/common hadoop-hdds/framework hadoop-ozone/ozone-recon U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/3/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
38
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
80
Maven dependency ordering for branch

+1
mvninstall
1128
trunk passed

+1
compile
1101
trunk passed

+1
checkstyle
202
trunk passed

+1
mvnsite
140
trunk passed

+1
shadedclient
1089
branch has no errors when building and testing our client artifacts.

+1
findbugs
168
trunk passed

+1
javadoc
105
trunk passed

_ Patch Compile Tests _

0
mvndep
60
Maven dependency ordering for patch

+1
mvninstall
91
the patch passed

+1
compile
933
the patch passed

+1
javac
933
the patch passed

+1
checkstyle
198
the patch passed

+1
mvnsite
116
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
2
The patch has no ill-formed XML file.

+1
shadedclient
691
patch has no errors when building and testing our client artifacts.

+1
findbugs
184
the patch passed

+1
javadoc
105
the patch passed

_ Other Tests _

-1
unit
91
common in the patch failed.

+1
unit
42
framework in the patch passed.

+1
unit
42
ozone-recon in the patch passed.

+1
asflicense
52
The patch does not generate ASF License warnings.

6600

Reason
Tests

Failed junit tests
hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-569/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/569

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux 59fde664740b 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 064f38b

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

findbugs
v3.1.0-RC1

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/4/artifact/out/patch-unit-hadoop-hdds_common.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/4/testReport/

Max. process+thread count
340 (vs. ulimit of 5500)

modules
C: hadoop-hdds/common hadoop-hdds/framework hadoop-ozone/ozone-recon U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/4/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

0
reexec
27
Docker mode activated.

_ Prechecks _

+1
@author
0
The patch does not contain any @author tags.

+1
test4tests
0
The patch appears to include 3 new or modified test files.

_ trunk Compile Tests _

0
mvndep
19
Maven dependency ordering for branch

-1
mvninstall
20
root in trunk failed.

-1
compile
21
root in trunk failed.

-0
checkstyle
18
The patch fails to run checkstyle in root

-1
mvnsite
21
common in trunk failed.

-1
mvnsite
20
framework in trunk failed.

-1
mvnsite
20
ozone-recon in trunk failed.

+1
shadedclient
100
branch has no errors when building and testing our client artifacts.

-1
findbugs
21
common in trunk failed.

+1
javadoc
58
trunk passed

_ Patch Compile Tests _

0
mvndep
59
Maven dependency ordering for patch

+1
mvninstall
84
the patch passed

-1
compile
575
root in the patch failed.

-1
javac
575
root in the patch failed.

+1
checkstyle
194
the patch passed

+1
mvnsite
85
the patch passed

+1
whitespace
0
The patch has no whitespace issues.

+1
xml
1
The patch has no ill-formed XML file.

-1
shadedclient
657
patch has errors when building and testing our client artifacts.

+1
findbugs
160
the patch passed

+1
javadoc
70
the patch passed

_ Other Tests _

-1
unit
76
common in the patch failed.

+1
unit
31
framework in the patch passed.

+1
unit
23
ozone-recon in the patch passed.

+1
asflicense
31
The patch does not generate ASF License warnings.

2599

Reason
Tests

Failed junit tests
hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient

Subsystem
Report/Notes

Docker
Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/569

Optional Tests
dupname  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient  findbugs  checkstyle  xml

uname
Linux cf85033f1891 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
personality/hadoop.sh

git revision
trunk / 373705f

maven
version: Apache Maven 3.3.9

Default Java
1.8.0_191

mvninstall
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/artifact/out/branch-mvninstall-root.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/artifact/out/branch-compile-root.txt

checkstyle
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/artifact/out//testptch/patchprocess/maven-branch-checkstyle-root.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/artifact/out/branch-mvnsite-hadoop-hdds_common.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/artifact/out/branch-mvnsite-hadoop-hdds_framework.txt

mvnsite
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/artifact/out/branch-mvnsite-hadoop-ozone_ozone-recon.txt

findbugs
v3.1.0-RC1

findbugs
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/artifact/out/branch-findbugs-hadoop-hdds_common.txt

compile
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/artifact/out/patch-compile-root.txt

javac
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/artifact/out/patch-compile-root.txt

unit
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/artifact/out/patch-unit-hadoop-hdds_common.txt

Test Results
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/testReport/

Max. process+thread count
411 (vs. ulimit of 5500)

modules
C: hadoop-hdds/common hadoop-hdds/framework hadoop-ozone/ozone-recon U: .

Console output
https://builds.apache.org/job/hadoop-multibranch/job/PR-569/5/console

Powered by
Apache Yetus 0.9.0 http://yetus.apache.org

This message was automatically generated.


chore: cleanup examples removal

Description

Cleans up remaining references to examples

Tasks
KNO-6150

Merge activity

Jun 6, 4:50 PM CDT: @kylemcd started a stack merge that includes this pull request via Graphite.


Documentation for testing WiPPLPy

I've added some documentation for how to go about writing tests for WiPPLPy. It is very similar to the documentation for PlasmaPy.

I've also updated the pre-commit to ignore URLs and added simple install instructions for pre-commit.


Perf Client Failed while inferencing request on the loaded bert model !!

Description
I have used pretrained BertForSequenceClassfication Model to load into triton server. Here, for initial setup, i am only running on my System CPU.
I started the server with the following command,
docker run -it --rm --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 -p8000:8000 -p8001:8001 -p8002:8002 -v ./triton-inference-server/docs/examples/model_repository:/models nvcr.io/nvidia/tritonserver:20.06-py3 tritonserver --model-repository=/models --log-verbose=2
Server Side Log
=============================
== Triton Inference Server ==
NVIDIA Release 20.06 (build 13333626)
Copyright (c) 2018-2020, NVIDIA CORPORATION.  All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION.  All rights reserved.
NVIDIA modifications are covered by the license terms that apply to the underlying
project or file.
WARNING: The NVIDIA Driver was not detected.  GPU functionality will not be available.
Use 'nvidia-docker run' to start this container; see
https://github.com/NVIDIA/nvidia-docker/wiki/nvidia-docker .
2020-08-12 05:45:01.186032: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
I0812 05:45:01.325936 1 server.cc:120] Initializing Triton Inference Server
E0812 05:45:01.326063 1 pinned_memory_manager.cc:192] failed to allocate pinned system memory: CUDA driver version is insufficient for CUDA runtime version
I0812 05:45:01.326134 1 graphdef_backend_factory.cc:48] Create GraphDefBackendFactory
I0812 05:45:01.326145 1 savedmodel_backend_factory.cc:48] Create SavedModelBackendFactory
I0812 05:45:01.326154 1 netdef_backend_factory.cc:46] Create NetDefBackendFactory
I0812 05:45:01.326159 1 plan_backend_factory.cc:48] Create PlanBackendFactory
I0812 05:45:01.326170 1 plan_backend_factory.cc:55] Registering TensorRT Plugins
I0812 05:45:01.326215 1 logging.cc:52] Registered plugin creator - ::GridAnchor_TRT version 1
I0812 05:45:01.326231 1 logging.cc:52] Registered plugin creator - ::NMS_TRT version 1
I0812 05:45:01.326243 1 logging.cc:52] Registered plugin creator - ::Reorg_TRT version 1
I0812 05:45:01.326256 1 logging.cc:52] Registered plugin creator - ::Region_TRT version 1
I0812 05:45:01.326279 1 logging.cc:52] Registered plugin creator - ::Clip_TRT version 1
I0812 05:45:01.326293 1 logging.cc:52] Registered plugin creator - ::LReLU_TRT version 1
I0812 05:45:01.326312 1 logging.cc:52] Registered plugin creator - ::PriorBox_TRT version 1
I0812 05:45:01.326324 1 logging.cc:52] Registered plugin creator - ::Normalize_TRT version 1
I0812 05:45:01.326334 1 logging.cc:52] Registered plugin creator - ::RPROI_TRT version 1
I0812 05:45:01.326343 1 logging.cc:52] Registered plugin creator - ::BatchedNMS_TRT version 1
I0812 05:45:01.326355 1 logging.cc:52] Registered plugin creator - ::FlattenConcat_TRT version 1
I0812 05:45:01.326366 1 logging.cc:52] Registered plugin creator - ::CropAndResize version 1
I0812 05:45:01.326377 1 logging.cc:52] Registered plugin creator - ::DetectionLayer_TRT version 1
I0812 05:45:01.326392 1 logging.cc:52] Registered plugin creator - ::Proposal version 1
I0812 05:45:01.326405 1 logging.cc:52] Registered plugin creator - ::ProposalLayer_TRT version 1
I0812 05:45:01.326419 1 logging.cc:52] Registered plugin creator - ::PyramidROIAlign_TRT version 1
I0812 05:45:01.326433 1 logging.cc:52] Registered plugin creator - ::ResizeNearest_TRT version 1
I0812 05:45:01.326446 1 logging.cc:52] Registered plugin creator - ::Split version 1
I0812 05:45:01.326457 1 logging.cc:52] Registered plugin creator - ::SpecialSlice_TRT version 1
I0812 05:45:01.326468 1 logging.cc:52] Registered plugin creator - ::InstanceNormalization_TRT version 1
I0812 05:45:01.326482 1 onnx_backend_factory.cc:53] Create OnnxBackendFactory
I0812 05:45:01.331247 1 libtorch_backend_factory.cc:46] Create LibTorchBackendFactory
I0812 05:45:01.331263 1 custom_backend_factory.cc:46] Create CustomBackendFactory
I0812 05:45:01.331271 1 ensemble_backend_factory.cc:47] Create EnsembleBackendFactory
I0812 05:45:01.331836 1 model_repository_manager.cc:667] AsyncLoad() 'bert'
I0812 05:45:01.331851 1 model_repository_manager.cc:729] TriggerNextAction() 'bert' version 1: 1
I0812 05:45:01.331884 1 model_repository_manager.cc:767] Load() 'bert' version 1
I0812 05:45:01.331889 1 model_repository_manager.cc:786] loading: bert:1
I0812 05:45:01.332012 1 model_repository_manager.cc:839] CreateInferenceBackend() 'bert' version 1
I0812 05:45:01.752939 1 libtorch_backend.cc:217] Creating instance bert_0_cpu on CPU using model.pt
I0812 05:45:02.337176 1 dynamic_batch_scheduler.cc:216] Starting dynamic-batch scheduler thread 0 at nice 5...
I0812 05:45:02.359768 1 model_repository_manager.cc:967] successfully loaded 'bert' version 1
I0812 05:45:02.359830 1 model_repository_manager.cc:729] TriggerNextAction() 'bert' version 1: 0
I0812 05:45:02.359854 1 model_repository_manager.cc:744] no next action, trigger OnComplete()
I0812 05:45:02.359995 1 model_repository_manager.cc:564] VersionStates() 'bert'
I0812 05:45:02.363198 1 grpc_server.cc:549] Ready for RPC 'ServerLive', 0
I0812 05:45:02.363220 1 grpc_server.cc:549] Ready for RPC 'ServerReady', 0
I0812 05:45:02.363227 1 grpc_server.cc:549] Ready for RPC 'ModelReady', 0
I0812 05:45:02.363235 1 grpc_server.cc:549] Ready for RPC 'ServerMetadata', 0
I0812 05:45:02.363243 1 grpc_server.cc:549] Ready for RPC 'ModelMetadata', 0
I0812 05:45:02.363260 1 grpc_server.cc:549] Ready for RPC 'ModelConfig', 0
I0812 05:45:02.363274 1 grpc_server.cc:549] Ready for RPC 'ModelStatistics', 0
I0812 05:45:02.363286 1 grpc_server.cc:549] Ready for RPC 'SystemSharedMemoryStatus', 0
I0812 05:45:02.363302 1 grpc_server.cc:549] Ready for RPC 'SystemSharedMemoryRegister', 0
I0812 05:45:02.363319 1 grpc_server.cc:549] Ready for RPC 'SystemSharedMemoryUnregister', 0
I0812 05:45:02.363333 1 grpc_server.cc:549] Ready for RPC 'CudaSharedMemoryStatus', 0
I0812 05:45:02.363345 1 grpc_server.cc:549] Ready for RPC 'CudaSharedMemoryRegister', 0
I0812 05:45:02.363356 1 grpc_server.cc:549] Ready for RPC 'CudaSharedMemoryUnregister', 0
I0812 05:45:02.363368 1 grpc_server.cc:549] Ready for RPC 'RepositoryIndex', 0
I0812 05:45:02.363383 1 grpc_server.cc:549] Ready for RPC 'RepositoryModelLoad', 0
I0812 05:45:02.363396 1 grpc_server.cc:549] Ready for RPC 'RepositoryModelUnload', 0
I0812 05:45:02.363433 1 grpc_server.cc:680] Thread started for CommonHandler
I0812 05:45:02.363614 1 grpc_server.cc:2623] New request handler for ModelInferHandler, 1
I0812 05:45:02.363651 1 grpc_server.cc:517] Thread started for ModelInferHandler
I0812 05:45:02.365107 1 grpc_server.cc:2860] New request handler for ModelStreamInferHandler, 3
I0812 05:45:02.365135 1 grpc_server.cc:517] Thread started for ModelStreamInferHandler
I0812 05:45:02.365152 1 grpc_server.cc:3199] Started GRPCInferenceService at <IP_ADDRESS>:8001
I0812 05:45:02.366743 1 http_server.cc:2546] Started HTTPService at <IP_ADDRESS>:8000
I0812 05:45:02.409279 1 http_server.cc:2565] Started Metrics Service at <IP_ADDRESS>:8002
I try to run triton client with the following command,
docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:20.06-py3-clientsdk
Then i am using perf_client for making inference,
/workspace# perf_client -m bert --shape INPUT__0:4
*** Measurement Settings ***
Batch size: 1
Measurement window: 5000 msec
Using synchronous calls for inference
Stabilizing using average latency
Request concurrency: 1
Failed to maintain requested inference load. Worker thread(s) failed to generate concurrent requests.
Thread [0] had error: failed to run model 'bert_0_cpu
Meantime my serverside log is,
I0812 05:45:02.365152 1 grpc_server.cc:3199] Started GRPCInferenceService at <IP_ADDRESS>:8001
I0812 05:45:02.366743 1 http_server.cc:2546] Started HTTPService at <IP_ADDRESS>:8000
I0812 05:45:02.409279 1 http_server.cc:2565] Started Metrics Service at <IP_ADDRESS>:8002
I0812 05:47:59.083256 1 http_server.cc:1185] HTTP request: 0 /v2/models/bert
I0812 05:47:59.083327 1 model_repository_manager.cc:608] GetInferenceBackend() 'bert' version -1
I0812 05:47:59.083363 1 model_repository_manager.cc:564] VersionStates() 'bert'
I0812 05:47:59.084391 1 http_server.cc:1185] HTTP request: 0 /v2/models/bert/config
I0812 05:47:59.084444 1 model_repository_manager.cc:608] GetInferenceBackend() 'bert' version -1
I0812 05:48:15.671811 1 http_server.cc:1185] HTTP request: 0 /v2/models/bert
I0812 05:48:15.671850 1 model_repository_manager.cc:608] GetInferenceBackend() 'bert' version -1
I0812 05:48:15.671864 1 model_repository_manager.cc:564] VersionStates() 'bert'
I0812 05:48:15.672311 1 http_server.cc:1185] HTTP request: 0 /v2/models/bert/config
I0812 05:48:15.672330 1 model_repository_manager.cc:608] GetInferenceBackend() 'bert' version -1
I0812 05:48:15.673012 1 http_server.cc:1185] HTTP request: 0 /v2/models/bert/stats
I0812 05:48:15.673032 1 model_repository_manager.cc:564] VersionStates() 'bert'
I0812 05:48:15.673043 1 model_repository_manager.cc:608] GetInferenceBackend() 'bert' version 1
I0812 05:48:15.673109 1 http_server.cc:1185] HTTP request: 2 /v2/models/bert/infer
I0812 05:48:15.673125 1 model_repository_manager.cc:608] GetInferenceBackend() 'bert' version -1
I0812 05:48:15.673153 1 infer_request.cc:347] add original input: [0x0x7f0fd80071e0] request id: , model: bert, requested version: -1, actual version: 1, flags: 0x0, correlation id: 0, batch size: 0, priority: 0, timeout (us): 0
original inputs:
[0x0x7f0fd800dc68] input: INPUT__0, type: INT32, original shape: [1,4], shape: []
override inputs:
inputs:
original requested outputs:
requested outputs:
I0812 05:48:15.673189 1 infer_request.cc:480] prepared: [0x0x7f0fd80071e0] request id: , model: bert, requested version: -1, actual version: 1, flags: 0x0, correlation id: 0, batch size: 1, priority: 0, timeout (us): 0
original inputs:
[0x0x7f0fd800dc68] input: INPUT__0, type: INT32, original shape: [1,4], shape: [4]
override inputs:
inputs:
[0x0x7f0fd800dc68] input: INPUT__0, type: INT32, original shape: [1,4], shape: [4]
original requested outputs:
OUTPUT__0
requested outputs:
OUTPUT__0
I0812 05:48:15.673240 1 libtorch_backend.cc:550] Running bert_0_cpu with 1 requests
W0812 05:48:15.673264 1 pinned_memory_manager.cc:103] failed to allocate pinned system memory: no pinned memory pool, falling back to non-pinned system memory
I0812 05:48:15.673273 1 pinned_memory_manager.cc:130] non-pinned memory allocation: size 16, addr 0x7f10180016f0
I0812 05:48:15.947926 1 libtorch_backend.cc:772] The following operation failed in the TorchScript interpreter.
Traceback of TorchScript, serialized code (most recent call last):
File "code/torch.py", line 7, in forward
def forward(self: torch.WrappedModel,
input_ids: Tensor) -> Tuple[Tensor]:
_0 = ((self.model).forward(input_ids, ),)
~~~~~~~~~~~~~~~~~~~ <--- HERE
return _0
File "code/torch/transformers/modeling_bert.py", line 10, in forward
input_ids: Tensor) -> Tensor:
_0 = self.classifier
_1 = (self.dropout).forward((self.bert).forward(input_ids, ), )
~~~~~~~~~~~~~~~~~~ <--- HERE
return (_0).forward(_1, )
class BertModel(Module):
File "code/torch/transformers/modeling_bert.py", line 35, in forward
_12 = torch.to(extended_attention_mask, 6, False, False, None)
attention_mask0 = torch.mul(torch.rsub(_12, 1., 1), CONSTANTS.c0)
_13 = (_3).forward((_4).forward(input_ids, input, ), attention_mask0, )
~~~~~~~~~~~ <--- HERE
return (_2).forward(_13, )
class BertEmbeddings(Module):
File "code/torch/transformers/modeling_bert.py", line 59, in forward
position_ids = torch.arange(annotate(number, seq_length), dtype=4, layout=0, device=torch.device("cpu"), pin_memory=False)
input0 = torch.expand(torch.unsqueeze(position_ids, 0), [_20, _21], implicit=False)
_22 = (_18).forward(input_ids, )
~~~~~~~~~~~~ <--- HERE
_23 = (_17).forward(input0, )
_24 = (_16).forward(input, )
File "code/torch/torch/nn/modules/sparse.py", line 7, in forward
def forward(self: torch.torch.nn.modules.sparse.Embedding,
input_ids: Tensor) -> Tensor:
inputs_embeds = torch.embedding(self.weight, input_ids, 0, False, False)
~~~~~~~~~~~~~~~ <--- HERE
return inputs_embeds
lib/python3.7/site-packages/torch/jit/init.py(875): trace
saved_pytorch_model.py(13):
RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got CPUIntType instead (while checking arguments for embedding)
I0812 05:48:15.948065 1 pinned_memory_manager.cc:157] non-pinned memory deallocation: addr 0x7f10180016f0
I0812 05:48:21.675463 1 http_server.cc:1185] HTTP request: 0 /v2/models/bert/stats
I0812 05:48:21.675595 1 model_repository_manager.cc:564] VersionStates() 'bert'
I0812 05:48:21.675669 1 model_repository_manager.cc:608] GetInferenceBackend() 'bert' version 1
Triton Information
What version of Triton are you using?
20.06
Are you using the Triton container or did you build it yourself?
Triton container - nvcr.io/nvidia/tritonserver:20.06-py3
To Reproduce
Model Name: BertForSequenceClassfication Model
I pulled the suggested PyTorch image:
docker pull nvcr.io/nvidia/pytorch:20.06-py3
I installed transformers
My transformers version is 3.0.2 while the PyTorch version is 1.5.1
I ran the following script to export the torchscript model:
import torch
from transformers import *

class WrappedModel(torch.nn.Module):
    def __init__(self):
        super(WrappedModel, self).__init__()
        self.model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
    def forward(self, data):
        return self.model(data)

example = torch.zeros((4,128), dtype=torch.long) # bsz , seqlen
pt_model = WrappedModel().eval()
traced_script_module = torch.jit.trace(pt_model, example)
traced_script_module.save("model.pt")

Then I ran the following script to test if the torchscript model can do inference as expected:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
pt_example_inputs = tokenizer.encode_plus("This is a sample input", return_tensors="pt")
example_inputs = {}
example_inputs['input_ids'] = pt_example_inputs['input_ids']
print(example_inputs)

import torch
model = torch.jit.load('model.pt')
example_outputs = model(example_inputs['input_ids'])

print(example_outputs[0].shape)

Output:
{'input_ids': tensor([[ 101, 2023, 2003, 1037, 7099, 7953,  102]])}
torch.Size([1, 2])

config.pbtxt

name: "bert"
platform: "pytorch_libtorch"
max_batch_size: 256
input {
  name: "INPUT__0"
  data_type: TYPE_FP32
  dims: [-1]
}
output {
  name: "OUTPUT__0"
  data_type: TYPE_FP32
  dims: [2]
}
dynamic_batching {
  preferred_batch_size: [1,2,4,8,16,32,64,128,256]
  max_queue_delay_microseconds: 10000
}
version_policy: {
  all {}
}

And then pulled triton server image and ran the container:
docker pull nvcr.io/nvidia/tritonserver:20.06-py3
And then I started server with the following command,
docker run -it --rm --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 -p8000:8000 -p8001:8001 -p8002:8002 -v ./triton-inference-server/docs/examples/model_repository:/models nvcr.io/nvidia/tritonserver:20.06-py3 tritonserver --model-repository=/models --log-verbose=2
And then i pulled the triton client image as well,
docker pull nvcr.io/nvidia/tritonserver:20.06-py3-clientsdk
`docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:20.06-py3-clientsdk`

For initial check inside container  i ran this,
perf_client -m bert --shape INPUT__0:4 -v -v
I got the following log,

Trying <IP_ADDRESS>:8000...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 8000 (#0)

GET /v2/models/bert HTTP/1.1
Host: localhost:8000
User-Agent: libcurl-agent/1.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 194
<
Connection #0 to host localhost left intact
{"name":"bert","versions":["1"],"platform":"pytorch_libtorch","inputs":[{"name":"INPUT__0","datatype":"INT32","shape":[-1,-1]}],"outputs":[{"name":"OUTPUT__0","datatype":"FP32","shape":[-1,2]}]}
Trying <IP_ADDRESS>:8000...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 8000 (#0)

GET /v2/models/bert/config HTTP/1.1
Host: localhost:8000
User-Agent: libcurl-agent/1.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 535
<
Connection #0 to host localhost left intact
{"name":"bert","platform":"pytorch_libtorch","version_policy":{"all":{}},"max_batch_size":256,"input":[{"name":"INPUT__0","data_type":"TYPE_INT32","dims":["-1"]}],"output":[{"name":"OUTPUT__0","data_type":"TYPE_FP32","dims":["2"]}],"instance_group":[{"name":"bert","count":1,"kind":"KIND_CPU"}],"default_model_filename":"model.pt","dynamic_batching":{"preferred_batch_size":[1,2,4,8,16,32,64,128,256],"max_queue_delay_microseconds":"10000"},"optimization":{"input_pinned_memory":{"enable":true},"output_pinned_memory":{"enable":true}}}
*** Measurement Settings ***
Batch size: 1
Measurement window: 5000 msec
Using synchronous calls for inference
Stabilizing using average latency

Request concurrency: 1
inference request: {"id":"","inputs":[{"name":"INPUT__0","datatype":"INT32","shape":[1,4],"parameters":{"binary_data_size":16}}],"outputs":[{"name":"OUTPUT__0","parameters":{"binary_data":true}}]}

Trying <IP_ADDRESS>:8000...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 8000 (#0)

POST /v2/models/bert/infer HTTP/1.1
Host: localhost:8000
User-Agent: libcurl-agent/1.0
Accept: /
Inference-Header-Content-Length: 177
Content-Type: application/octet-stream
Content-Length: 193

We are completely uploaded and fine
Trying <IP_ADDRESS>:8000...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 8000 (#0)

GET /v2/models/bert/stats HTTP/1.1
Host: localhost:8000
User-Agent: libcurl-agent/1.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 339
<
Connection #0 to host localhost left intact
{"model_stats":[{"name":"bert","version":"1","last_inference":0,"inference_count":0,"execution_count":0,"inference_stats":{"success":{"count":0,"ns":0},"fail":{"count":2,"ns":189531825},"queue":{"count":0,"ns":0},"compute_input":{"count":0,"ns":0},"compute_infer":{"count":0,"ns":0},"compute_output":{"count":0,"ns":0}},"batch_stats":[]}]}
Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Content-Length: 43
<
Connection #0 to host localhost left intact
inference response: {"error":"failed to run model 'bert_0_cpu"}
Trying <IP_ADDRESS>:8000...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 8000 (#0)

GET /v2/models/bert/stats HTTP/1.1
Host: localhost:8000
User-Agent: libcurl-agent/1.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 339
<
Connection #0 to host localhost left intact
{"model_stats":[{"name":"bert","version":"1","last_inference":0,"inference_count":0,"execution_count":0,"inference_stats":{"success":{"count":0,"ns":0},"fail":{"count":3,"ns":203056216},"queue":{"count":0,"ns":0},"compute_input":{"count":0,"ns":0},"compute_infer":{"count":0,"ns":0},"compute_output":{"count":0,"ns":0}},"batch_stats":[]}]}
Pass [1] Error: No valid requests recorded within time interval. Please use a larger time window.
Failed to maintain requested inference load. Worker thread(s) failed to generate concurrent requests.
Thread [0] had error: failed to run model 'bert_0_cpu

Some Notes
Perf_client can able to make inference on other models which is already in the triton repository, (like resnet50_netdef, simple ..)
Here, I am only starting server and making inference on my local system CPU.
Questions:

What do you think would cause this problem?
Thanks in Advance :)

By default, perf_client sends random data in the input tensors. BERT models have been known to have trouble with random data so that could be the issue. You can ask perf_client to sends just zeros for the input data and see if that resolves the issue.

Yeah, I tried that too, But sadly that not helps.There is no change in the output log..
perf_client -m bert --shape INPUT__0:4 --input-data zero -v -v

Trying <IP_ADDRESS>:8000...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 8000 (#0)

GET /v2/models/bert HTTP/1.1
Host: localhost:8000
User-Agent: libcurl-agent/1.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 194
<
Connection #0 to host localhost left intact
{"name":"bert","versions":["1"],"platform":"pytorch_libtorch","inputs":[{"name":"INPUT__0","datatype":"INT32","shape":[-1,-1]}],"outputs":[{"name":"OUTPUT__0","datatype":"FP32","shape":[-1,2]}]}
Trying <IP_ADDRESS>:8000...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 8000 (#0)

GET /v2/models/bert/config HTTP/1.1
Host: localhost:8000
User-Agent: libcurl-agent/1.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 535
<
Connection #0 to host localhost left intact
{"name":"bert","platform":"pytorch_libtorch","version_policy":{"all":{}},"max_batch_size":256,"input":[{"name":"INPUT__0","data_type":"TYPE_INT32","dims":["-1"]}],"output":[{"name":"OUTPUT__0","data_type":"TYPE_FP32","dims":["2"]}],"instance_group":[{"name":"bert","count":1,"kind":"KIND_CPU"}],"default_model_filename":"model.pt","dynamic_batching":{"preferred_batch_size":[1,2,4,8,16,32,64,128,256],"max_queue_delay_microseconds":"10000"},"optimization":{"input_pinned_memory":{"enable":true},"output_pinned_memory":{"enable":true}}}
*** Measurement Settings ***
Batch size: 1
Measurement window: 5000 msec
Using synchronous calls for inference
Stabilizing using average latency

Request concurrency: 1
inference request: {"id":"","inputs":[{"name":"INPUT__0","datatype":"INT32","shape":[1,4],"parameters":{"binary_data_size":16}}],"outputs":[{"name":"OUTPUT__0","parameters":{"binary_data":true}}]}

Trying <IP_ADDRESS>:8000...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 8000 (#0)

GET /v2/models/bert/stats HTTP/1.1
Host: localhost:8000
User-Agent: libcurl-agent/1.0
Accept: /

Trying <IP_ADDRESS>:8000...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 8000 (#0)

POST /v2/models/bert/infer HTTP/1.1
Host: localhost:8000
User-Agent: libcurl-agent/1.0
Accept: /
Inference-Header-Content-Length: 177
Content-Type: application/octet-stream
Content-Length: 193

We are completely uploaded and fine
Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 339
<
Connection #0 to host localhost left intact
{"model_stats":[{"name":"bert","version":"1","last_inference":0,"inference_count":0,"execution_count":0,"inference_stats":{"success":{"count":0,"ns":0},"fail":{"count":3,"ns":187081060},"queue":{"count":0,"ns":0},"compute_input":{"count":0,"ns":0},"compute_infer":{"count":0,"ns":0},"compute_output":{"count":0,"ns":0}},"batch_stats":[]}]}
Mark bundle as not supporting multiuse
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< Content-Length: 43
<
Connection #0 to host localhost left intact
inference response: {"error":"failed to run model 'bert_0_cpu"}
Trying <IP_ADDRESS>:8000...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 8000 (#0)

GET /v2/models/bert/stats HTTP/1.1
Host: localhost:8000
User-Agent: libcurl-agent/1.0
Accept: /

Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: application/json
< Content-Length: 339
<
Connection #0 to host localhost left intact
{"model_stats":[{"name":"bert","version":"1","last_inference":0,"inference_count":0,"execution_count":0,"inference_stats":{"success":{"count":0,"ns":0},"fail":{"count":4,"ns":203684333},"queue":{"count":0,"ns":0},"compute_input":{"count":0,"ns":0},"compute_infer":{"count":0,"ns":0},"compute_output":{"count":0,"ns":0}},"batch_stats":[]}]}
Pass [1] Error: No valid requests recorded within time interval. Please use a larger time window.
Failed to maintain requested inference load. Worker thread(s) failed to generate concurrent requests.
Thread [0] had error: failed to run model 'bert_0_cpu

And same time the server side log also same,

Were you able to successfully run the torchscript model using the C++ API? The issue here is not with the perf_client but with the improper trace produced by the model. A specific op in the model expected scalar type Long; but got CPUIntType
Instead of using torch.jit.trace please use script = torch.jit.script(model) and repeat the same experiment. Just because the models runs fine using the python API (should but) does not guarantee that the traced model will work using the C++ API.

While using torch.jit.script i got the following error,
RuntimeError:
Expected a default value of type Tensor (inferred) on parameter "input_ids". Because "input_ids" was not annotated with an explicit type it is assumed to be type 'Tensor'.:
    def forward(self, input_ids=None, token_type_ids=None, position_ids=None, inputs_embeds=None):
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...  <--- HERE
        # print(input_ids)

I ran the following script:
import torch
from transformers import *

class WrappedModel(torch.nn.Module):
    def __init__(self):
        super(WrappedModel, self).__init__()
        self.model = BertForSequenceClassification.from_pretrained('bert-base-uncased')
    def forward(self, input_ids):
        return self.model(input_ids)
example = torch.zeros((4,128), dtype=torch.long) # bsz , seqlen
pt_model = WrappedModel().eval()
traced_script_module = torch.jit.script(pt_model, example)
traced_script_module.save("model.pt")

@VJAYSLN  Is this issue resolved? If not please reopen.


Test FB 2017 JUL 26

This pull request addresses the problems described at issue #1413
I'll be writing tests with pytest in the next few days (probably until this Friday) so that we can assure all of our testsuite is in good shape. I actually already started to find bugs in the testsuite, so it is already a winner :-)
I'll merge this into master by the end of today's work-day.
@graphicore you may want to follow the commits I push to this branch today. It may bring insides on ways to improve our pytest strategy.

@graphicore How should we ideally deal with conditions in this testing framework? Should we craft fake data manually? Or should we reuse the implementation of the condition functions from the spec file?
For instance, on test/003 we have a description condition (which ends up being the contents of a DESCRIPTION.en_us.html file) and on test/006 we have a metadata condition (which is a protobuf instance created by parsing a METADATA.pb file).
One good outcome of reusing the condition functions from the spec file is that we'd be validating them as well. Should I then simply import those condition functions and use them?

@graphicore How should we ideally deal with conditions in this testing framework? Should we craft fake data manually?

I would rather craft it manually and test the conditions separately. Testing these things implicitly may cause hard to find bugs.
We could test the conditions explicitly as well, but this is much more like testing a normal function: known input -> expected output.
All the dependency resolution stuff should probably also be tested one day, but rather with a fake spec also crafted for the test.
The good thing with crafted data is that is very good controllable, not so many moving parts. The bad thing is that it is probably a lot to write, which can (will?) become a burden as well.
Just a thought: turning off caching in a testRunner could be a good feature for unit tests. When we always get fresh results from the conditions we can still manipulate the values for the tests, but we don't produce side effects.

Or should we reuse the implementation of the condition functions from the spec file?

Thinking about this, I'll look at nicer ways to get conditions via a spec. Our goal should be to have not so much boilerplate. Calling all conditions in a dependency tree manually is cumbersome as well as crafting the data for each case...

Should I then simply import those condition functions and use them?

That's definitely OK. Just trying to come up with something less verbose.

By the way, to keep the code clean I only plan to use  the Message class on tests that have more than a single yield statement  with same result type (i.e. for tests that have more than a single FAIL  scenario, since that could lead to ambiguity on the meaning of the  pytest assert statements).

Yeah, that's fine. What we should never do though is looking directly at a message text to validate a result. In that case we should have an object with message.code

@graphicore you may want to follow the commits I push to this branch today. It may bring insides on ways to improve our pytest strategy.

Looking good so far, at  "pytest: adapt codestyle for test/029" 95b49ae

Thanks!
I think that, as long as we don't update the data/test/* font files, we can use them as a reference for know-good/known-bad scenarios, to reduce the verbosity of the crafting of fake font-data for testing.

Yeah, also I think the tests read well enough. Thus, if something spooky happens, it shouldn't be too hard to adopt.

yes, I'm happy with this approach.


Composer support

Composer is so popular in php community these day. Thus, Does it possible to define composer.json and publish to packagist.org for sake of package accessibility?

+1, please also publish a release
thanks for a great piece of software!

+1 I will be so great :)

To be honest I do almost zero PHP development these days. But if someone wants to send a pull request (with instructions on how to use and test it with Composer), I'd be glad to review it.

hi @elidickinson i have made it just now

I cannot install it

diego3/php-export-data v1.0.0 requires elidickinson/php-export-data dev-psr4 -> no matching package found.

exatctly @DarkAllMan , i think it happen because i forked the elidickinson rep, so may he need register or accept my pull request to have it working.
As an alternative you can download my version and set up manually into your composer directory,
what you think @elidickinson  ?

In case this isn't obvious: You can use @diego3's current dev build to bypass the dependency:
composer require diego3/php-export-data:@dev


QEMU/OVMF improvements

A few improvements to cargo xtask run to make it more likely to work well across various Linux distros and Windows.

The paths where OVMF files get installed are annoyingly inconsistent between each OS/distro. I tried various methods to be clever about how the paths get searched, but what ended up being clearest in the end is just having separate functions to get the expected paths for each host type. This covers Windows and Linux (Arch, CentOS, Debian/Ubuntu, and Fedora). I only tested the results on Fedora and Windows since that's what I have installed, but I got paths for the others by using online package search tools.
Changed the CLI interface to take separate --ovmf-code and --ovmf-vars options rather than --ovmf-dir. This works better because the file names of OVMF can vary a lot. For that reason, the uefi-test-runner directory is no longer automatically searched.
On Windows, the QEMU installer doesn't add the QEMU directory to the PATH. Add the default directory to the search path to make it more likely that cargo xtask run will just work without manually modifying the PATH.
The vars file is now always copied to a temporary location so that it can be made read+write. Previously we did this just for aarch64, but I think it's a little simpler (and more like a real UEFI env) to just always make it r+w.

Fixes https://github.com/rust-osdev/uefi-rs/issues/469

The paths where OVMF files get installed are annoyingly inconsistent between each OS/distro.

It's alright, they're sometimes even inconsistent across different versions of the same distro 😒 I've gotten used to copying them to the uefi-rs directory to avoid issues.


运行时出现错误， '_compile_builder' it contains a nested function with free variables？

CentOS 7.5
错误内容如下：试了很多机子都不行
Traceback (most recent call last):
File "wsgi.py", line 28, in
from rrd import app
File "/dashboard/dashboard/rrd/init.py", line 19, in
from flask import Flask, request
File "/dashboard/dashboard/env/lib/python2.7/site-packages/flask/init.py", line 21, in
from .app import Flask, Request, Response
File "/dashboard/dashboard/env/lib/python2.7/site-packages/flask/app.py", line 23, in
from werkzeug.routing import BuildError, Map, RequestRedirect, Rule
File "/dashboard/dashboard/env/lib/python2.7/site-packages/werkzeug/routing.py", line 948
exec(code, globs, locs)
SyntaxError: unqualified exec is not allowed in function '_compile_builder' it contains a nested function with free variables
这个该如何解决，试了两天没找到解决方法

我也是，这个到底是什么问题啊？费劲死了。对flask真的不了解。

升级
Werkzeug==0.15.4


Build failure: Build step not linking against GraphicsMagick on macOS

macos branch fails to build awtfdb with on macOS, graphicsmagick is installed via brew:
$ brew info graphicsmagick
==> graphicsmagick: stable 1.3.43 (bottled), HEAD
Image processing tools collection
http://www.graphicsmagick.org/
Installed
/opt/homebrew/Cellar/graphicsmagick/1.3.43 (506 files, 16.8MB) *
  Poured from bottle using the formulae.brew.sh API on 2024-08-07 at 07:55:57

$ zig build
.zig-cache/o/<id>/cimport.h:1:10: error: 'GraphicsMagick/wand/magick_wand.h' file not found
#include <GraphicsMagick/wand/magick_wand.h>

Build Summary: 6/10 steps succeeded; 1 failed (disable with --summary none)
install transitive failure
├─ install wrapper-awtfdb transitive failure
│  └─ zig build-exe wrapper-awtfdb Debug native 3 errors
└─ link the utils transitive failure
   └─ zig build-exe wrapper-awtfdb Debug native (+8 more reused dependencies)

System details

OS: macOS
System version: 14.6
Xcode Command Line Tools version: 2408
Zig version: 12.1
Arch: arm64

Closing in favor of https://github.com/lun-4/awtfdb/issues/67


Code 404

My problem is that the webserver is not pulling from the userdata.js I currently have
var users = ["email"];
and the error I am getting is https://gyazo.com/ed8b50a0085ad83c9a1ee4493ee687b8

Username should be email address, also see this #1246


Comments in-line with JSX elements cause Invariant Violation error

[ X ] Review the documentation: https://facebook.github.io/react-native
[ X ] Search for existing issues: https://github.com/facebook/react-native/issues
[ X ] Use the latest React Native release: https://github.com/facebook/react-native/releases

Environment
React Native Environment Info:
    System:
      OS: macOS 10.14.1
      CPU: x64 Intel(R) Core(TM) i9-8950HK CPU @ 2.90GHz
      Memory: 3.58 GB / 32.00 GB
      Shell: 5.3 - /bin/zsh
    Binaries:
      Node: 10.7.0 - /usr/local/bin/node
      npm: 6.4.0 - /usr/local/bin/npm
      Watchman: 4.9.0 - /usr/local/bin/watchman
    SDKs:
      iOS SDK:
        Platforms: iOS 12.1, macOS 10.14, tvOS 12.1, watchOS 5.1
    IDEs:
      Xcode: 10.1/10B61 - /usr/bin/xcodebuild
    npmPackages:
      react: 16.6.0-alpha.8af6728 => 16.6.0-alpha.8af6728
      react-native: 0.57.4 => 0.57.4
    npmGlobalPackages:
      react-native-cli: 2.0.1
      react-native-git-upgrade: 0.2.7
      react-native-vector-icons: 4.6.0

Description
In line comments break react-native 56+, as shown in the code and repo below. Removing "This comment breaks the app" allows the app to run. Comments on lines by themselves seem to be ok.
export default class App extends Component<Props> {
  render() {
    return (
      <View style={styles.container}>
        {/* This comment is ok */}
        <Text style={styles.welcome}>Welcome to React Native!</Text> {/* This comment breaks the app */}
        <Text style={styles.instructions}>To get started, edit App.js</Text>
        <Text style={styles.instructions}>{instructions}</Text>
      </View>
    );
  }
}

Possibly linked to:  #20084
Reproducible Demo
https://github.com/kernel-sanders/ReactNative56TextBug

This is most likely a Babel bug. See how the compiled JS looks like

Repro: https://snack.expo.io/@jkcooper/rn-afterline-comments

This happens because of the whitespace before the comment:
</Text> {/*

The whitespace is recognized as a string as per JSX's parsing rules which causes an error in RN due to being outside Text.
If you remove this whitespace, it works:
</Text>{/*

Closing since this is the expected behaviour.


[stack-view] stack-block-label that has tooltip when ellipsised

Select one ...  (check one with "x")
[ ] bug
[ ] feature request
[x] enhancement

Expected behavior
A stack view checks if its labels overflow and adds tooltip attribute to the ones that have ellipsis. The user can see the whole content of the "ellipsis"-ed label simply by hovering it.
The value of the added tooltip is the innerHTML of the label.
This behavior is activated by an attribute directive(e.g. className) added to the clr-stack-view.
Actual behavior
The current behavior of stack-view's labels is to wrap to a new line.
Every use of stack-views in the H5C is overriding those labels to be:

white-space: nowrap
overflow:hidden + text-overflow: ellipsis
This leads to the need to add tooltips to the labels that overflow.

Reproduction of behavior
https://embed.plnkr.co/kX8rSykzWrplA09SGkrG/
This Clarity Plunker example demos a simple check of the label's scrollWidth and offsetWidth
and adding a title attribute when scrollWidth is less than the offsetWidth.
Checking the size change may be a bit more challenging and expensive.
I've found a solution using element-resize-detector that may be promising.
Even without it though if this attribute directive simply does:

modifies the styles of the labels to be one line and ellipsis-ed
adds tooltips to all labels

It will still eliminate the need to add a title attribute to each stack-block-label.

We need to support touch screens, so any pattern that requires hover to get the information cannot be accepted. You'll notice that even our tooltips activate on click, for instance.
We have labels wrap on purpose, this is what we recommend and support. If you decide to override this and force an ellipsis on your side, you'll need to find a way to display the full information somehow yourselves when it wraps, we cannot implement it for you there.

Btw, there is something that I don't quite understand in this issue.
Assigning overflow hidden with ellipsis only changes the screen rendering of the node. In the DOM, it has the complete text. So, a reader should not have problem reading the whole data, without the need to hover over the ellipsis.
As far as I know, the overflow will not affect readability. It is even an often used technique to create content for screenreader only, to make it with size of 0 or 1px and overflow hidden.


useSquiggle and useSquigglePartial hooks - components refactorings

Changes:

extracted some common code to the useSquiggle and useSquigglePartial hooks, including setting div.value for observablehq (I couldn't find any reasons why it it'd be bad to set it every time, so there's no observable: true flag)
fixed some minor bugs with useEffect deps (previously run wasn't executed on binding/environment changes, which could cause stale data if you tried to chain together multiple SquigglePartials)
minor refactorings

renderSquigglePartialToDom and renderSquiggleEditorToDom are trivial now, but I haven't removed them yet. (Maybe I should, no real reason to provide renderXToDom for every component, can be done by lib users.)
I don't like that I couldn't unify SquiggleEditor and SquigglePartial more with a HOC (which was my initial intention), due to run/runPartial discrepancy, which is annoying. It'd be still possible to do this with generics but not worth the complexity.
I'd strongly prefer to have a single useSquiggle hook, but run converts its output with createTsExport, which is a one way road (AFAICS) so it can't be used for bindings. Ugh. I'll have to take a closer look at why squiggle-lang is implemented (I've already read #351 and #333) before fixing this.
PS: Minor backward-incompatible changes:

I removed some useless SquigglePartial props, I don't think anyone used them
onChange gets called with an undefined value on errors, it wasn't called on errors before (which I think is good, errors should trickle down instead of being hidden)

Maybe I should, no real reason to provide renderXToDom for every component, can be done by lib users.

Thought about this some more, it's not that easy until we rethink our bundling story. Right now squiggle's webpack bundle includes React, which forbids using it with externally loaded React. (Solving #494 might help, though.)

Yeah! I'll try and review this soon

Actually, this PR created this bug, which is pretty bad/blocking.  (Just found out when I merged).
Please fix soon.

Not entirely sure if this breaks observable support, but it looks like it's going to break anyway in the near future, so I don't mind.

It should be ok. I haven't checked it with observable, but I did check that .value gets set and updated correctly.


SearchAdapter empty

Hi. I use library version 3.2.1 and add suggestion in a thread after server request.
The problem is searchAdapter.getItemCount = 0 conversely suggestionList.size contain results of server request.
List<SearchItem> suggestionList=new ArrayList<>();
mSearchView.setOnQueryTextListener(new SearchView.OnQueryTextListener() {
                @Override
                public boolean onQueryTextChange(String newText) {
                    if (newText.length() > 3) {
                        taskAutoComplete(newText); //look down
                    }
                    return false;
                }

private void taskAutoComplete(final String query) {
new Thread() {
            public void run() {

                mPlaceAutocomplete.autocomplete(query); //look down

                mHandler.post(new Runnable() {
                    @Override
                    public void run() {
                        suggestionsList = mPlaceAutocomplete.getSuggestionList();
                        Log.d(TAG, "suggestion list " + suggestionsList.size());

                        mSearchAdapter = new SearchAdapter(getApplicationContext(), suggestionsList);
                        Log.d(TAG, "searchAdapter " + mSearchAdapter.getItemCount());

                        mSearchView.setAdapter(mSearchAdapter);
                    }
                });
            }
        }.start();
    }

autocomplete(String query)
.
.
.
for (int i = 0; i < predsJsonArray.length(); i++)
                suggestionsList.add(new SearchItem(suggestion));
.
.
public List<SearchItem> getSuggestionList() {
        return suggestionsList;
    }
.
.

any news?

Use please last version 3.3.1. About your bug I did not have time yet to check it, sorry.

Please use last version of the library. And there is method setAdapter( RecyclerView.Adapter ). You should write your own adapter.


WIP: changes for V+jets processor

Many changes related to my V+Jets processor. Maybe worth synchronizing with main fork

Wrong branch


🐛 [FIREBASE APP CHECK]  IntegrityService : Unbind from service.

Bug report
Describe the bug
I have implemented the app check as instructed in then  documentation
Steps to reproduce
Steps to reproduce the behavior:

Install the firebase app check
Use the same example provided in the installation documentation
See the following steps by app check and the error message

Expected behavior
Expected to receive integrity token
Sample project
Please link to the public repository URL.
https://github.com/elirehema/sampleintegrity

Aditional infos
This is what i get as an error

Flutter doctor
Run flutter doctor and paste the output below:

Click To Expand
PASTE OUTPUT INSIDE HERE

Flutter dependencies
Run flutter pub deps -- --style=compact and paste the output below:
Click To Expand
![image](https://github.com/firebase/flutterfire/assets/29277048/1fbf9223-89fa-4a7c-87c7-2048468a8f53)

Hello, have the same issue. Does anyone have any updates connected with this issue?

Hi, is there any update for this issue? I'm still facing it.

waiting to solve the issue...

Same here, I leaved a question here: https://stackoverflow.com/questions/76667281/android-firebase-app-check-error-while-retrieving-information-from-server-df-df

any update on this issue ?

Hey @elirehema - if in debug mode, it is likely because the debug token has expired and you need to update. If in release mode, you need to have the app in playstore for app integrity provider to work along with the requisite app signin SHA key in your Firebase console.

It still doesn't work in release mode(provided the sha is added and app is uploaded in play store(internal test))

Facing the same issue, provided the sha is added and app is uploaded in play store(internal test)

Same issue here, does anyone have a fix for it?

same issue, any solution?

In my case this solution worked:
If .getToken() function is working in your app, comment it out. (Only .activate() command should work in your app)
After opening your Flutter app, wait 10-20 seconds, the console will show the following text.
Enter this debug secret into the allow list in the Firebase Console for your project: 0c3e59b1-9f49-4bc5-a656-b3c2037cbb07
Add the debug key here to firebase > app check > apps > select app > manage debug tokens.


Add ability to edit application configuration

What would you like to be added:
Why is this needed:

It would also be better to add ability to edit Piped configuration.


Gracefully handle devices that do not support A/V info.

Breaking change

Proposed change

Type of change

[ ] Dependency upgrade
[x] Bugfix (non-breaking change which fixes an issue)
[ ] New integration (thank you!)
[ ] New feature (which adds functionality to an existing integration)
[ ] Breaking change (fix/feature causing existing functionality to break)
[ ] Code quality improvements to existing code or addition of tests

Example entry for configuration.yaml:

# Example configuration.yaml
media_player:
  - platform: onkyo
    host: <IP_ADDRESS>
    name: Amplifier

Additional information

This PR fixes or closes issue: fixes #
This PR is related to issue: #46130
Link to documentation pull request:

Checklist

[ ] The code change is tested and works locally.
[x] Local tests pass. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.
[x] I have followed the development checklist
[x] The code has been formatted using Black (black --fast homeassistant tests)
[ ] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated for www.home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[ ] The manifest file has all fields filled out correctly.
Updated and included derived files by running: python3 -m script.hassfest.
[ ] New or updated dependencies have been added to requirements_all.txt.
Updated by running python3 -m script.gen_requirements_all.
[ ] Untested files have been added to .coveragerc.

The integration reached or maintains the following Integration Quality Scale:

[ ] No score or internal
[ ] 🥈 Silver
[ ] 🥇 Gold
[ ] 🏆 Platinum

@OnFreund I think this will fix the issue regarding the queries for audio and video info. However I cannot test it here because my devices don't give the errors. Do you know who can test this?

Maybe @danieljkemp can help?

I was able to test it! All it took was a k8s ConfigMap, and a lifecycle.postStart.exec.command entry.
Anyhow, I was able to load it in HA without any errors, and all of my expected controls are working.

Perfect, then we can merge this into the main PR! 🙂


feat(openapi): Select schemas based on Operation params

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#995
#1001  👈
#1000
#999
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Cobalt0s and the rest of your teammates on  Graphite


Use npm scripts for a dev cleaner workflow

Related to #34. I didn't touch linting as that's up to you to decide what you want to use (there's barely anything in .jshintrc right now anyway).
Tests run on pre-commit and Travis CI will include browser tests with Karma.

Great stuff. But it says conflicts... Probably because I merged another PR. Why do we need phantom by the way? I have it installed, but it perfectly runs with chrome - I suspect many people may not have it installed. I suggest removing it.

@epoberezkin Karma can't run on Travis with Chrome, you'll need either Firefox or PhantomJS.

Ah... Got it.

I'll resolve the conflicts now too :+1:

@epoberezkin Tests passed


cursor pointer for labels

It would be good to have cursor:pointer style for labels by default.
Here is the example of the form with label to test -
http://twitter.github.com/bootstrap/examples/signin.html
There are already such styles:
button,
html input[type="button"],
input[type="reset"],
input[type="submit"] {
cursor: pointer;
-webkit-appearance: button;
}
It would be good also to have such styles by default:
label[for], select, .pointer {
cursor: pointer;
}

We want a cursor when hovering a label used on a custom-file-input, because clicking it opens Choose file dialog.


ClimateMapped: Improve data page navigation labels

Improve the labelling on Navigations in the explore page in ClimateMapped
Request Link: Feature 6

Implemented this as tooltip when hovering over the buttons (similar to how it's done in original Wazimap).


feat(sfn): use DataFrame between zipper and sfn

This change is

streamfunction/client.go, line 65 at r1 (raw file):

	}

	return stream.Write(data.Encode())

for sending multiple Frames in goroutines on the same QUIC-stream, suggest we add Lock.


configure_polaris_envs.py fails to find software from conda on a generic mac/linux machine.

When attempting to create the polaris dev environment on a generic mac or linux computer (rather than one of the supported machines), this error is thrown:
A software compiler or a spack base was not defined so required software was not installed with spack.

which is a bug because the software should come entirely from conda.

@altheaden, thanks for posting this!


Update aws-java-sdk-core to 1.12.505

About this PR
📦 Updates com.amazonaws:aws-java-sdk-core from 1.12.459 to 1.12.505
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.amazonaws", artifactId = "aws-java-sdk-core" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "com.amazonaws", artifactId = "aws-java-sdk-core" }
}]

labels: library-update, early-semver-patch, semver-spec-patch, commit-count:1

Superseded by #574.


Add the whole serial with one button

Closes #5

Codecov Report

Merging #12 into master will not change coverage.
The diff coverage is 0.00%.

@@          Coverage Diff           @@
##           master     #12   +/-   ##
======================================
  Coverage    0.00%   0.00%
======================================
  Files          16      16
  Lines         587     594    +7
  Branches       82      83    +1
======================================
- Misses        587     594    +7

Impacted Files
Coverage Δ

web/app/pages/now-playing/index.js
0.00% <0.00%> (ø)

web/app/pages/serial/index.js
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f11dac...c512f3c. Read the comment docs.


Error: division by zero

hotel year she ntact first search world so data you no good ove ...
thread 'main' panicked at 'attempt to dmik/.cargo/registry/src/github.com-1ecc6299db9ec823/tui:480:22

│note: run with `RUST_BACKTRACE=1` environment variablee

On finishing a test with only two words typed one incorrect.
I can investigate this issue and get back with a pr soon.

There's already some work on this in #42, so I'm closing this as a duplicate. A PR would be awesome!


Render issue even while trying the example

Hi,
Even when i use the example to render an image i get this issue -
Uncaught Error: ImageZoom.render(): A valid React element (or null) must be returned. You may have returned undefined, an array or some other invalid object.
Pasting code here -

import React, {Component} from 'react';
import ReactMarkdown from 'react-markdown'
import markdown from '../../../data/Work/sourceapp/sourceapp.md'
import styles from './ProjectPage.scss'
import MediumImage from '../../utils/MediumImage'
import ImageZoom from 'react-medium-image-zoom'

class ProjectPage extends Component {
  render() {
    return (

Image Zoom

          Trust fund seitan chia, wolf lomo letterpress Bushwick before they
          sold out. Carles kogi fixie, squid twee Tonx readymade cred typewriter
          scenester locavore kale chips vegan organic. Meggings pug wolf
          Shoreditch typewriter skateboard. McSweeney's iPhone chillwave,
          food truck direct trade disrupt flannel irony tousled Cosby sweater
          single-origin coffee. Organic disrupt bicycle rights, tattooed
          messenger bag flannel craft beer fashion axe bitters. Readymade
          sartorial craft beer, quinoa sustainable butcher Marfa Echo Park Terry
          Richardson gluten-free flannel retro cred mlkshk banjo. Salvia
          90's art party Blue Bottle, PBR&B cardigan 8-bit.

          Meggings irony fashion axe, tattooed master cleanse Blue Bottle
          stumptown bitters authentic flannel freegan paleo letterpress ugh
          sriracha. Wolf PBR&B art party aesthetic meh cliche. Sartorial
          before they sold out deep v, aesthetic PBR&B craft beer
          post-ironic synth keytar pork belly skateboard pour-over. Tonx cray
          pug Etsy, gastropub ennui wolf ethnic Odd Future viral master cleanse
          skateboard banjo. Pitchfork scenester cornhole, whatever try-hard
          ethnic banjo +1 gastropub American Apparel vinyl skateboard Shoreditch
          seitan. Blue Bottle keffiyeh Austin Helvetica art party. Portland
          ethnic fixie, beard retro direct trade ugh scenester Tumblr readymade
          authentic plaid pickled hashtag biodiesel.

          <ImageZoom
            image={{
              src: 'http://res.cloudinary.com/aswin/image/upload/v1511706514/Cover_g8mjcw.png',
              alt: 'Golden Gate Bridge',
              className: 'img'
            }}
            zoomImage={{
              src: 'http://res.cloudinary.com/aswin/image/upload/v1511706514/Cover_g8mjcw.png',
              alt: 'Golden Gate Bridge',
              className: 'img--zoomed'
            }}
          />

          Thundercats freegan Truffaut, four loko twee Austin scenester lo-fi
          seitan High Life paleo quinoa cray. Schlitz butcher ethical Tumblr,
          pop-up DIY keytar ethnic iPhone PBR sriracha. Tonx direct trade
          bicycle rights gluten-free flexitarian asymmetrical. Whatever drinking
          vinegar PBR XOXO Bushwick gentrify. Cliche semiotics banjo retro squid
          Wes Anderson. Fashion axe dreamcatcher you probably haven't heard
          of them bicycle rights. Tote bag organic four loko ethical selfies
          gastropub, PBR fingerstache tattooed bicycle rights.

          <ImageZoom
            image={{
              src: 'http://res.cloudinary.com/aswin/image/upload/v1511706514/Cover_g8mjcw.png',
              alt: 'Picture of Mt. Cook in New Zealand',
              className: 'img',
              style: {
                width: '20em'
              }
            }}
            zoomImage={{
              src: 'http://res.cloudinary.com/aswin/image/upload/v1511706514/Cover_g8mjcw.png',
              alt: 'Picture of Mt. Cook in New Zealand',
              className: 'img--zoomed'
            }}
          />

          Ugh Portland Austin, distillery tattooed typewriter polaroid pug
          Banksy Neutra keffiyeh. Shoreditch mixtape wolf PBR&B, tote bag
          dreamcatcher literally bespoke Odd Future selfies 90's master
          cleanse vegan. Flannel tofu deep v next level pickled, authentic Etsy
          Shoreditch literally swag photo booth iPhone pug semiotics banjo.
          Bicycle rights butcher Blue Bottle, actually DIY semiotics Banksy
          banjo mixtape Austin pork belly post-ironic. American Apparel
          gastropub hashtag, McSweeney's master cleanse occupy High Life
          bitters wayfarers next level bicycle rights. Wolf chia Terry
          Richardson, pop-up plaid kitsch ugh. Butcher +1 Carles, swag selfies
          Blue Bottle viral.

          Keffiyeh food truck organic letterpress leggings iPhone four loko
          hella pour-over occupy, Wes Anderson cray post-ironic. Neutra retro
          fixie gastropub +1, High Life semiotics. Vinyl distillery Etsy freegan
          flexitarian cliche jean shorts, Schlitz wayfarers skateboard tousled
          irony locavore XOXO meh. Ethnic Wes Anderson McSweeney's
          messenger bag, mixtape XOXO slow-carb cornhole aesthetic Marfa banjo
          Thundercats bitters. Raw denim banjo typewriter cray Tumblr, High Life
          single-origin coffee. 90's Tumblr cred, Terry Richardson occupy
          raw denim tofu fashion axe photo booth banh mi. Trust fund locavore
          Helvetica, fashion axe selvage authentic Shoreditch swag selfies
          stumptown +1.

          <ImageZoom
            image={{
              src: 'http://res.cloudinary.com/aswin/image/upload/v1511706514/Cover_g8mjcw.png',
              alt: 'Gazelle Stomping',
              title: "Don't exceed original image dimensions...",
              className: 'img',
              style: {
                width: '250px'
              }
            }}
            shouldRespectMaxDimension={true}
          />

          Scenester chambray slow-carb, trust fund biodiesel ugh bicycle rights
          cornhole. Gentrify messenger bag Truffaut tousled roof party pork
          belly leggings, photo booth jean shorts. Austin readymade PBR plaid
          chambray. Squid Echo Park pour-over, wayfarers forage whatever
          locavore typewriter artisan deep v four loko. Locavore occupy Neutra
          Pitchfork McSweeney's, wayfarers fingerstache. Actually
          asymmetrical drinking vinegar yr brunch biodiesel. Before they sold
          out sustainable readymade craft beer Portland gastropub squid Austin,
          roof party Thundercats chambray narwhal Bushwick pug.

          Literally typewriter chillwave, bicycle rights Carles flannel
          wayfarers. Biodiesel farm-to-table actually, locavore keffiyeh hella
          shabby chic pour-over try-hard Bushwick. Sriracha American Apparel
          Brooklyn, synth cray stumptown blog Bushwick +1 VHS hashtag. Wolf
          umami Carles Marfa, 90's food truck Cosby sweater. Fanny pack
          try-hard keytar pop-up readymade, master cleanse four loko trust fund
          polaroid salvia. Photo booth kitsch forage chambray, Carles scenester
          slow-carb lomo cardigan dreamcatcher. Swag asymmetrical leggings,
          biodiesel Tonx shabby chic ethnic master cleanse freegan.

          Raw denim Banksy shabby chic, 8-bit salvia narwhal fashion axe.
          Ethical Williamsburg four loko, chia kale chips distillery Shoreditch
          messenger bag swag iPhone Pitchfork. Viral PBR&B single-origin
          coffee quinoa readymade, ethical chillwave drinking vinegar
          gluten-free Wes Anderson kitsch Tumblr synth actually bitters. Butcher
          McSweeney's forage mlkshk kogi fingerstache. Selvage scenester
          butcher Shoreditch, Carles beard plaid disrupt DIY. Pug readymade
          selvage retro, Austin salvia vinyl master cleanse flexitarian deep v
          bicycle rights plaid Terry Richardson mlkshk pour-over. Trust fund
          try-hard banh mi Brooklyn, 90's Etsy kogi YOLO salvia.

    )
  }
}

export default ProjectPage

Hi, @aswinckr, I'll copy this code and see what I can find! Be back with you shortly

Hi @rpearce , I'm sorry i just figured out the problem. I was using the latest build ^3.0.2 of react-medium-zoom but my react was v15.4.2. Didn't take note of the incompatibility you mentioned in the docs.
Thank you, I'll close this issue

@aswinckr great news! Thanks for making my job easy :)


Fix a problem of file append test on Windows

This is a pull request that was discussed on #1352.
Types of changes
What types of changes does your code introduce?
Put an x in the boxes that apply

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)

Proposed changes
The test script 50-file_spec.js fails only on Windows environment.
On the Windows environment, the unlinked file won't be completely deleted until its file handle is closed.  Under this condition, to recreate the same file fails. So I explicitly call close.
Checklist
Put an x in the boxes that apply

[x] I have read the contribution guidelines
[ ] For non-bugfix PRs, I have discussed this change on the mailing list/slack team.
[x] I have run grunt to verify the unit tests pass
[ ] I have added suitable unit tests to cover the new/changed functionality

Coverage decreased (-0.03%) to 86.398% when pulling 0a575fbcb0fb72e7765f04754950a3bba06650eb on Kazuki-Nakanishi:filetest into 9134d8841d940fc122d88df0dd00e574aed2958f on node-red:master.

Although all tests passed on Windows, Ubuntu and mac OS in my environment, they somehow failed on the Travis environment.
Sorry, but please let me investigate it.

Coverage increased (+1.02%) to 87.451% when pulling c7eb130ad5f7af0a9186485969a0518821258263 on Kazuki-Nakanishi:filetest into 9134d8841d940fc122d88df0dd00e574aed2958f on node-red:master.

The reason that this problem occurs only on Linux was that the behavior of fs-extra is difference between Windows and Linux.
The following is a sample code to see the difference.
var fs = require("fs-extra");
var filename='a.txt';
try { fs.unlinkSync(filename); } catch(e) {}

var wstream = fs.createWriteStream(filename, { flags:'a', autoClose:true }); console.log(fs.statSync(filename).ino);

fs.unlinkSync(filename);
wstream.end(); // comment out this line.
setTimeout(function() {
  fs.writeFileSync(filename,"");
  console.log(fs.statSync(filename).ino);
}, 500);

Windows

Run the above code as is: Node.js returns a different inode.
Run with commenting out wstream.end();: An error occurs. (This is the case of an original problem.)

Linux

Run the above code as is: Node.js returns a same inode. (This is the case that causes a Travis error.)
Run with commenting out wstream.end();: Node.js returns a different inode.

I thought that close function in a Node-RED node might be called from somewhere. But I've reconsidered that close function is called only when Node-RED instance stops. If this is true, I should not call close directly. And it means that this test case cannot be run on Windows. In conclusion, I just simply skip this test case when running on Windows.
If my understanding is not correct, please let me know.

Need to check what impact this difference in behaviour has on the functionality of the node. Don't want to disable the test if it will hide a bug

@knolleary
Thank you for your comment!
Let me organize whether the difference impacts the Node-RED behavior or not.

Run the above code as is:
Even if the file that is being appended some strings by a file node was deleted from the other process, the file node does not close the file handle. At the time, when the other process creates a new file with the same filename, new inode is always assigned to the file. So the problem of Linux does not actually occur.

Run with commenting out wstream.end()
After a file node appends strings to a file, if the other process tries to open the file, an error occurs.
As the message is like "file is locked", the other process can recognize that the file cannot be modified.
With regard to a file node itself, this problem should not happen because a file node always calls wstream.end(); when necessary.

I found that the behavior is different between del command and deleting a file on Windows explorer (GUI).

del command
1-1. Append strings to a.txt with a file node.
1-2. Run del a.txt on command prompt. (or fs.unlink on Node.js)
1-3. Run echo abc > a.txt on command prompt.
Then Access is denied. error occurs.

deleting a file on Windows explorer
2-1. Append strings to a.txt with a file node.
2-2. Delete a.txt on Windows explorer.
2-3. Create a.txt on Windows explorer.
2-4. Append strings to a.txt with a file node.
This case works fine.

When 2-2 has done, a temporary file whose file size is same as a.txt is shown on explorer.
I assumed that the file is renamed from a.txt (although this file actually cannot be opened).

Based on this information, I changed a test sequence from unlink to rename and unlink, then the test passed without error on both Windows 8.1 and Windows 10.
I think that this is the correct test sequence for deleting a file on Windows explorer.

Coverage increased (+1.1%) to 87.576% when pulling b3af75d295ad97e6d956a6af7d50346c2f9f7f66 on Kazuki-Nakanishi:filetest into 9134d8841d940fc122d88df0dd00e574aed2958f on node-red:master.

Great - thanks for investigating that.