Seeking advice for using deepcopy-gen

I tried to use deepcopy-gen on custom resource, but it doesn't work: 1. no xxx.deepcopy.go file was generated; 2. saw many confusing errors in log message.
Created this issue to seek help and advice for usng deepcopy-gen.
Here's what I did:

Add tag to my custom resource:// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// EtcdClusterList is a list of etcd clusters.
type EtcdClusterList struct {
	metav1.TypeMeta `json:",inline"`
	// Standard list metadata
	// More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []EtcdCluster `json:"items"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

type EtcdCluster struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`
	Spec              ClusterSpec   `json:"spec"`
	Status            ClusterStatus `json:"status"`
}

For more information, check out https://github.com/coreos/etcd-operator/blob/master/pkg/apis/etcd/v1beta1/cluster.go .
Compile deepcopy-gen from code-generator repo at a8e1019df0df00506fe1db5232a2af30ff4c87aa
Run command:deepcopy-gen -i github.com/coreos/etcd-operator/pkg/apis/etcd/v1beta1 -v=4 --logtostderr --bounding-dirs "github.com/coreos/etcd-operator/pkg/apis" --output-file-base zz_generated.deepcopy

No generated file comes out.
Here's the log message:
https://gist.github.com/hongchaodeng/2a8725a8c5b314aa2e401c72d119f0ce
Let me know if you need more details.

Hi, I have learning about code generator recently. I think I know the issue. I saw that etcd operator uses client-go 4.0.0. This repo works with client-go master. You need to use code-generators from the Kubernetes 1.7 branch.
This is what I have done with client-go 4.0 and worked for me:

Build a docker image from Kube 1.7 https://github.com/appscode/gengo-builder/blob/master/Dockerfile
Run code-gen using docker image https://github.com/appscode/voyager/blob/master/hack/codegen.sh

@tamalsaha
Thanks for your help.
We want to bump up dependencies to align with upcoming k8s 1.8 .
So I update dependencies in etcd-operator to latest client-go:
- package: k8s.io/client-go
  version: c6f8cf2c47d21d55fa0df928291b2580544886c8
- package: k8s.io/api
  version: 926789af7ddda62752e08bc9b93f1d1ebbc7b2de
- package: k8s.io/apimachinery
  version: a28c9b0b532638dcfb2c8b1b6f8f1e4a21041752

Still the same error.

@tamalsaha
One question:
I'm confused how to ensure the dependency is deterministic here:
https://github.com/appscode/gengo-builder/blob/f1e9a54e21dc712442a61fb7674d46410ab3b01a/Dockerfile#L4

My knowledge is limited regarding dependency. What I understood was that you need to use the correct version of the generator so that the generated code uses the correct dependencies. To compile the generator you only need the kubernetes project.

Still failed..
I tried to generate a docker image from the same Dockerfile: https://github.com/appscode/gengo-builder/blob/f1e9a54e21dc712442a61fb7674d46410ab3b01a/Dockerfile
Then run similar commands:

docker run --rm -ti \
  -v "${GOPATH}/src/github.com/coreos/etcd-operator":"${DOCKER_REPO_ROOT}" \
  -w "$DOCKER_REPO_ROOT" \
  "${IMAGE}" deepcopy-gen \
  --go-header-file="hack/docker/k8s_codegen/boilerplate.go.txt" \
  --input-dirs="github.com/coreos/etcd-operator/pkg/apis/etcd/v1beta1" \
  --v=4 --logtostderr \
  --bounding-dirs "github.com/coreos/etcd-operator/pkg/apis" \
  --output-file-base zz_generated.deepcopy

still fail:
I0919 04:17:51.887042       1 parse.go:417] type checker: /usr/local/go/src/crypto/tls/cipher_suites.go:18:2: could not import golang_org/x/crypto/chacha20poly1305 (unable to import "golang_org/x/crypto/chacha20poly1305": cannot find package "golang_org/x/crypto/chacha20poly1305" in any of:
	/go/src/github.com/coreos/etcd-operator/vendor/golang_org/x/crypto/chacha20poly1305 (vendor tree)
	/usr/local/go/src/golang_org/x/crypto/chacha20poly1305 (from $GOROOT)
	/go/src/golang_org/x/crypto/chacha20poly1305 (from $GOPATH))
I0919 04:17:51.900739       1 parse.go:417] type checker: /usr/local/go/src/crypto/tls/key_agreement.go:20:2: could not import golang_org/x/crypto/curve25519 (unable to import "golang_org/x/crypto/curve25519": cannot find package "golang_org/x/crypto/curve25519" in any of:
	/go/src/github.com/coreos/etcd-operator/vendor/golang_org/x/crypto/curve25519 (vendor tree)
	/usr/local/go/src/golang_org/x/crypto/curve25519 (from $GOROOT)
	/go/src/golang_org/x/crypto/curve25519 (from $GOPATH))
I0919 04:17:51.918757       1 parse.go:366] type checking encountered some issues in "crypto/tls", but ignoring.
I0919 04:17:51.968790       1 parse.go:417] type checker: /usr/local/go/src/net/http/h2_bundle.go:46:2: could not import golang_org/x/net/http2/hpack (unable to import "golang_org/x/net/http2/hpack": cannot find package "golang_org/x/net/http2/hpack" in any of:
	/go/src/github.com/coreos/etcd-operator/vendor/golang_org/x/net/http2/hpack (vendor tree)
	/usr/local/go/src/golang_org/x/net/http2/hpack (from $GOROOT)
	/go/src/golang_org/x/net/http2/hpack (from $GOPATH))
I0919 04:17:51.970830       1 parse.go:417] type checker: /usr/local/go/src/net/http/h2_bundle.go:47:2: could not import golang_org/x/net/idna (unable to import "golang_org/x/net/idna": cannot find package "golang_org/x/net/idna" in any of:
	/go/src/github.com/coreos/etcd-operator/vendor/golang_org/x/net/idna (vendor tree)
	/usr/local/go/src/golang_org/x/net/idna (from $GOROOT)
	/go/src/golang_org/x/net/idna (from $GOPATH))
I0919 04:17:51.976932       1 parse.go:417] type checker: /usr/local/go/src/net/http/h2_bundle.go:48:2: could not import golang_org/x/net/lex/httplex (unable to import "golang_org/x/net/lex/httplex": cannot find package "golang_org/x/net/lex/httplex" in any of:
	/go/src/github.com/coreos/etcd-operator/vendor/golang_org/x/net/lex/httplex (vendor tree)
	/usr/local/go/src/golang_org/x/net/lex/httplex (from $GOROOT)
	/go/src/golang_org/x/net/lex/httplex (from $GOPATH))

I got it worked..
I create a doc.go file under the same dir:
// +k8s:deepcopy-gen=package

package v1beta1

Same command.
Thanks @tamalsaha for your help all the same. Appreciated!

I tried to use deepcopy-gen on custom resource with k8s 1.8.3, but get some errors. I have created the doc.go.  I'm seeking for some advice
here is the error info:
I1205 11:28:10.736185    6980 parse.go:417] type checker: E:\Go\src\runtime\trace.go:150:30: cannot use (traceBufHeader literal) (value of type traceBufHeader) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I1205 11:28:10.757187    6980 parse.go:417] type checker: E:\Go\src\runtime\trace.go:150:7: array length 64 << 10 - unsafe.Sizeof((traceBufHeader literal)) (value of type uintptr) must be constant
I1205 11:28:10.757187    6980 parse.go:417] type checker: E:\Go\src\runtime\mgcwork.go:303:37: cannot use (workbufhdr literal) (value of type workbufhdr) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I1205 11:28:10.757187    6980 parse.go:417] type checker: E:\Go\src\runtime\mgcwork.go:303:7: array length (_WorkbufSize - unsafe.Sizeof((workbufhdr literal))) / sys.PtrSize (value of type uintptr) must be constant
I1205 11:28:10.758187    6980 parse.go:417] type checker: E:\Go\src\runtime\chan.go:27:28: cannot use (hchan literal) (value of type hchan) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I1205 11:28:10.758187    6980 parse.go:417] type checker: E:\Go\src\runtime\chan.go:27:66: cannot use (hchan literal) (value of type hchan) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I1205 11:28:10.759189    6980 parse.go:417] type checker: E:\Go\src\runtime\chan.go:27:14: unsafe.Sizeof((hchan literal)) + uintptr(-int(unsafe.Sizeof((hchan literal))) & (maxAlign - 1)) (value of type uintptr) is not constant
I1205 11:28:10.761186    6980 parse.go:417] type checker: E:\Go\src\runtime\hashmap.go:80:31: cannot use (struct{b bmap; v int64} literal).v (value of type int64) as unsafe.ArbitraryType value in argument to unsafe.Offsetof
I1205 11:28:10.761186    6980 parse.go:417] type checker: E:\Go\src\runtime\hashmap.go:80:15: unsafe.Offsetof((struct{b bmap; v int64} literal).v) (value of type uintptr) is not constant
I1205 11:28:10.763186    6980 parse.go:417] type checker: E:\Go\src\runtime\mheap.go:148:47: cannot use (mcentral literal) (value of type mcentral) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I1205 11:28:10.764189    6980 parse.go:417] type checker: E:\Go\src\runtime\mheap.go:148:13: array length sys.CacheLineSize - unsafe.Sizeof((mcentral literal)) % sys.CacheLineSize (value of type uintptr) must be constant
I1205 11:28:10.765187    6980 parse.go:417] type checker: E:\Go\src\runtime\mfinal.go:28:65: cannot use (finalizer literal) (value of type finalizer) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I1205 11:28:10.765187    6980 parse.go:417] type checker: E:\Go\src\runtime\mfinal.go:28:11: array length (_FinBlockSize - 2 * sys.PtrSize - 2 * 4) / unsafe.Sizeof((finalizer literal)) (value of type uintptr) must be constant
I1205 11:28:10.767187    6980 parse.go:417] type checker: E:\Go\src\runtime\mheap.go:1535:41: cannot use (gcBitsHeader literal) (value of type gcBitsHeader) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I1205 11:28:10.768188    6980 parse.go:417] type checker: E:\Go\src\runtime\mheap.go:1535:27: unsafe.Sizeof((gcBitsHeader literal)) (value of type uintptr) is not constant
I1205 11:28:10.768188    6980 parse.go:417] type checker: E:\Go\src\runtime\mstats.go:436:39: cannot use memstats.by_size (variable of type [67]struct{size uint32; nmalloc uint64; nfree uint64}) as unsafe.ArbitraryType value in argume
nt to unsafe.Offsetof
I1205 11:28:10.769186    6980 parse.go:417] type checker: E:\Go\src\runtime\mstats.go:436:76: cannot use memstats.by_size[0] (variable of type struct{size uint32; nmalloc uint64; nfree uint64}) as unsafe.ArbitraryType value in argumen
t to unsafe.Sizeof
I1205 11:28:10.769186    6980 parse.go:417] type checker: E:\Go\src\runtime\panic.go:120:34: cannot use (_defer literal) (value of type _defer) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I1205 11:28:10.770190    6980 parse.go:417] type checker: E:\Go\src\runtime\panic.go:120:20: unsafe.Sizeof((_defer literal)) (value of type uintptr) is not constant
I1205 11:28:10.772187    6980 parse.go:417] type checker: E:\Go\src\runtime\sema.go:51:42: cannot use (semaRoot literal) (value of type semaRoot) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I1205 11:28:10.773188    6980 parse.go:417] type checker: E:\Go\src\runtime\sema.go:51:8: array length sys.CacheLineSize - unsafe.Sizeof((semaRoot literal)) (value of type uintptr) must be constant
I1205 11:28:10.781187    6980 parse.go:366] type checking encountered some issues in "runtime", but ignoring.
I1205 11:28:10.782187    6980 parse.go:417] type checker: E:\Go\src\sync\map.go:69:31: cannot convert new(interface{}) (value of type *interface{}) to unsafe.Pointer
I1205 11:28:10.783187    6980 parse.go:417] type checker: E:\Go\src\sync\pool.go:68:27: cannot use (poolLocalInternal literal) (value of type poolLocalInternal) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I1205 11:28:10.783187    6980 parse.go:417] type checker: E:\Go\src\sync\pool.go:68:7: array length 128 - unsafe.Sizeof((poolLocalInternal literal)) % 128 (value of type uintptr) must be constant
I1205 11:28:10.783187    6980 parse.go:366] type checking encountered some issues in "sync", but ignoring.
I1205 11:28:19.704699    6980 parse.go:417] type checker: E:\Go\src\crypto\cipher\xor.go:12:36: cannot use uintptr(0) (constant 0 of type uintptr) as unsafe.ArbitraryType value in argument to unsafe.Sizeof
I1205 11:28:19.705698    6980 parse.go:417] type checker: E:\Go\src\crypto\cipher\xor.go:12:18: int(unsafe.Sizeof(uintptr(0))) (value of type int) is not constant
I1205 11:28:19.706699    6980 parse.go:366] type checking encountered some issues in "crypto/cipher", but ignoring.
I1205 11:28:21.787401    6980 parse.go:417] type checker: E:\Go\src\crypto\tls\cipher_suites.go:18:2: could not import golang_org/x/crypto/chacha20poly1305 (unable to import "golang_org/x/crypto/chacha20poly1305": cannot find package
"golang_org/x/crypto/chacha20poly1305" in any of:
E:\Go\src\golang_org\x\crypto\chacha20poly1305 (from $GOROOT)
E:\workspace\src\golang_org\x\crypto\chacha20poly1305 (from $GOPATH))
I1205 11:28:21.840404    6980 parse.go:417] type checker: E:\Go\src\crypto\tls\key_agreement.go:20:2: could not import golang_org/x/crypto/curve25519 (unable to import "golang_org/x/crypto/curve25519": cannot find package "golang_org/
x/crypto/curve25519" in any of:
E:\Go\src\golang_org\x\crypto\curve25519 (from $GOROOT)
E:\workspace\src\golang_org\x\crypto\curve25519 (from $GOPATH))
I1205 11:28:21.845403    6980 parse.go:366] type checking encountered some issues in "crypto/tls", but ignoring.
I1205 11:28:22.522440    6980 parse.go:417] type checker: E:\Go\src\net\http\h2_bundle.go:46:2: could not import golang_org/x/net/http2/hpack (unable to import "golang_org/x/net/http2/hpack": cannot find package "golang_org/x/net/http
2/hpack" in any of:
E:\Go\src\golang_org\x\net\http2\hpack (from $GOROOT)
E:\workspace\src\golang_org\x\net\http2\hpack (from $GOPATH))
I1205 11:28:22.523439    6980 parse.go:417] type checker: E:\Go\src\net\http\h2_bundle.go:47:2: could not import golang_org/x/net/idna (unable to import "golang_org/x/net/idna": cannot find package "golang_org/x/net/idna" in any of:
E:\Go\src\golang_org\x\net\idna (from $GOROOT)
E:\workspace\src\golang_org\x\net\idna (from $GOPATH))
I1205 11:28:22.523439    6980 parse.go:417] type checker: E:\Go\src\net\http\h2_bundle.go:48:2: could not import golang_org/x/net/lex/httplex (unable to import "golang_org/x/net/lex/httplex": cannot find package "golang_org/x/net/lex/
httplex" in any of:
E:\Go\src\golang_org\x\net\lex\httplex (from $GOROOT)
E:\workspace\src\golang_org\x\net\lex\httplex (from $GOPATH))
I1205 11:28:22.575442    6980 parse.go:417] type checker: E:\Go\src\net\http\transport.go:32:2: could not import golang_org/x/net/proxy (unable to import "golang_org/x/net/proxy": cannot find package "golang_org/x/net/proxy" in any of
:
E:\Go\src\golang_org\x\net\proxy (from $GOROOT)
E:\workspace\src\golang_org\x\net\proxy (from $GOPATH))
I1205 11:28:22.593443    6980 parse.go:366] type checking encountered some issues in "net/http", but ignoring.
I1205 11:28:28.073469    6980 main.go:81] Completed successfully.

Hit the same issue after two years.


Use EPoll transport for Netty.

This should reduce GC churn from various byte[] allocations in the C* driver.
Tested on stage, basic heap profiling says it's good at getting rid of the crazy amounts of byte[] allocations we have, and GC gets a bit better. As far as I can tell, there are no downsides.

:+1:

LGMT


feat: add documentation💡

Implementation PR
No response
Reference Issues
No response
Summary
Add documentation again, current version (docs) is outdated (made for version v3.x.x).
Basic Example
n/a
Drawbacks
n/a
Unresolved questions
No response

New documentation posted at https://paperplane.ijskoud.dev/


Topic #269 fc gfortran

Replaces find_or_install mpich invocation with `find_or_install gcc."  Also, this pull request improves the installation script stack management and testing.

@rouson looking like you're making some great progress, and adding some unit tests for the shell scripts, which is great!
It looks like ./install.sh exits early on OS X and linux when invoked as
./install.sh --yes-to-all -i "$HOME/opt/opencoarrays" -j 4 -f "$HOME/bin/gfortran" -c "$HOME/bin/gcc" -C "$HOME/bin/g++"

Also, it seems that CMake/CTest is appending an extra .sh for the shell script tests. This looks quite easy to test.
The only reason this branch is out of sync is because of some documentation fixes I've made, and those shouldn't conflict with anything we're doing here so a simple git pull origin master should do the trick to sync them up.

@zbeekman
Feel free to fix the issues you've mentioned above however you see fit.  I'll focus on editing find_or_install.sh to launch CMake with CC=gcc FC==gfortran cmake ... as we discussed.  I assume you won't need to modify find_or_install.sh so there is little chance of a conflict between our edits.

Feel free to fix the issues you've mentioned above however you see fit. I'll focus on editing find_or_install.sh to launch CMake with CC=gcc FC==gfortran cmake ... as we discussed. I assume you won't need to modify find_or_install.sh so there is little chance of a conflict between our edits.

sounds good. Push up any changes you may once you're happy with them. I'll rebase any changes on top of yours so that git pulls should be easy fast forward merges.

Current coverage is 50.38% (diff: 100%)

No coverage report found for master at 635f4db.

Powered by Codecov. Last update 635f4db...99f104f

Also, FYI, as soon as we're ready to merge this, I'll probably interactively rebase it to clean up the commit history with some squashed commits, etc. so that there aren't any work in progress (WIP) commits. (Yes, I was the one who introduced the non-functional commit.)

@rouson I've added my changes to CMake to add the shell script tests... Right now all of the CI tests pass, but I think you said that there was more work to do, no? IIRC there are some other cases that might not be correctly handled in terms of getting FC and CC into build_opencoarrays.sh

LGTM via @rouson above

doh! I meant to reword/squash the commit with WIP in it... gah. Now my push git hook is going to yell at me all the time... I'm strongly contemplating reverting master.... but it's probably not worth it.


Add support for RWKV

Not sure if this RNN counts as a LLM, but if so would be nice to have it, let me know what needs to be done with packaging.
https://www.rwkv.com/

Looking at the config file (config.json · RWKV/rwkv-5-world-1b5 at main), it looks like its an architecture not supported by llama.cpp, and thus we are not able to support it yet.
We can leave the issue open to track supporting Rwkv5ForCausalLM

This seems to be where discussion of RWKV support in Lllama.cpp is happening. So far it's really just discussion.

Especially important now that the 1.0 release (v5) and Eagle 7B is out https://blog.rwkv.com/p/eagle-7b-soaring-past-transformers

Seems like some work has already been done on this https://github.com/RWKV/rwkv.cpp

https://github.com/ggerganov/llama.cpp/pull/8980

And it is in the llamacpp synced in #6621 :)

@technovangelist :) rwkv6 is in llama.cpp now. let me if you can run it

Any update on this? What else needs to be done for a model to be added other than llama.cpp support?

#7313
This should make RWKV v6 support to be in a useable state


Issue: Unable to resolve host address ‘downloads.apache.org’

Replace:
from
https://downloads.apache.org/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
to
http://apache.mirrors.pair.com/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz

This is weird, the first link is still working for me, and I remember that Apache recommends now to use downloads.apache.org which will automatically find the best mirror. Are you sure we have to change this? By the way what is this mirror? Finally I'm a priori against using HTTP instead of HTTPS.

It's not even listed in the Apache Maven mirror list: https://maven.apache.org/download.cgi

I think below is the reason why it is working and why not:
Please have a look at availability report: http://www.apache.org/mirrors/

Agreed with @rouazana. Thereis plenty of mirrors using https, which is safer than plain http. downloads.apache.org is up in your link by the way. Might have been just a temporary issue with the node (can happen), I don't see the point of changing this.
But thanks for your concern either way :)

Https works.
I don't see a reason why we should downgrade to http.
Thanks for the concern.


Added specs for Inline Blocks

EquationFormat formatting was missing.

Hi @Devorein, thank you a lot for the PR! Mind rebasing your branch? I just added a Github check.

Thanks for updating. It would be great if you can add an example or just yields of how an equation can be represented ["e", string] doesn't really show how an equation looks like. How the string will likely be formatted?

Good idea. On it.

@phuctm97 Can you please direct me on how to show an example for equation format in the readme. This is a special type of formatting like Inline Date, Person, Page, and I don't see any examples for those as well.

@phuctm97 Can you please direct me on how to show an example for equation format in the readme. This is a special type of formatting like Inline Date, Person, Page, and I don't see any examples for those as well.

Sure, sorry for the confusion. There's really no standard for examples in this specification. The easiest way is to capture a screen with several equations and log an object how these equations are represented in your proposals. Add that next to the current example or how you feel easier for readers to understand.
Thanks for pointing out that PageFormat, UserFormat, and ExternalLinkFormat don't have examples yet, will add these soon!

How was it @Devorein? Just want to check in if you need more helps to merge this.

@phuctm97 I've added examples and new specs for inline content. Could you please check and report?

This is really awesome. However, look like you were using npm, it generated package-lock.json and unexpected change to yarn.lock. Please help revert changes of these 2 files and use yarn instead.

@phuctm97 I've replaced npm with yarn and distinguished between two different types for contents in the spec.

Look amazing! Merging.

Thank you for contributing @Devorein. Adding you to the list of contributors in the frontpage.


Added API Gateway multi-origin CORS example

Issue #, if available:
Description of changes:
Created an example of using Lambda to provide CORS headers on API Gateway endpoints given a list of allowed origins. The Lambda function checks the origin of incoming requests against a list, and only provides CORS headers if there is a match.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Looking good. The only other things (as communicated offline) is to split this into multiple Lambda Functions instead of implementing routing. Alternatively, if you prefer to use a single function, use aws-serverless-express (and in this case you could probably just use the cors middleware)

Improved the routing


Select Command does not work on displayed room list

The logic for the select command applies to the guest list even though the view is a room list.
(1) The selection is displayed correctly, but the message says "selected guest".

(2) As there are only 4 guests, selecting with an index >4 creates a command exception.

The logic is checking the validity of selection using the guest list.

fixed in #211


Update dependencies, use ClimaCore operators

This PR:

Updates the dependencies (notably ClimaCore)
and uses some of the recently added ClimaCore operators

bors r+


Support Streaming in Spark Structured Streaming Runner

The current version of the Spark Structured Streaming Runner only support Bounded IO. This ticket is to address the support of full Streaming.
This issue is blocked because of issues on Spark with multiple aggregations SPARK-26655
Imported from Jira BEAM-9933. Original Jira may contain additional context.
Reported by: iemejia.

Might become an option later in 2023 again once Project Lightspeed is completed, closing for now


HTML rewriting  - POC

A merge request, mostly to see how big this got

We're pretty sure for now this isn't the direction we are going in

The initial spike has been done and work has moved into the viahtml branch. Related Slack thread: https://hypothes-is.slack.com/archives/C1MA4E9B9/p1600697110022400.


Hidden readline not remove in fasta header

Pretty self explanatory. Seems you need chrom = chrom.strip() to prevent readlines from messing with the fastq file. Bowtie2 handles it well, but any other parsing of the fastq file throws an error since probe position is on a new line.
Right now producing:
@NZ_CP00
:54-94
TAAATTGAAGAGTTTGATCATGGCTCAGATTGAACGCTGGC
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

And not:
@NZ_CP00:54-94
TAAATTGAAGAGTTTGATCATGGCTCAGATTGAACGCTGGC
+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Hi Andrew,
Thanks for reaching out about this!
I am trying to reproduce this issue locally; would you kindly provide:

a small fasta file that results in this fastq output when run through blockParse.py
the output of python --version command
your operating system

I have also started an issue_12_dev branch and attempted to implement the change you suggested. To test this change on your end:
$ git pull
$ git checkout issue_12_dev

and then you should be able to run the modified blockParse.py and see if it resolves the issue on your files.
Once I'm able to reproduce the issue and test a solution, I'll go ahead and merge the changes. Thanks again!
Best,
Conor Camplisson


Added lang/pt_br

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@FITFC please sign the CLA that I can merge your contribution.


[Feature] Typescript support for route path and route path params(Auto complete)

Description
It's so good to have Typescript support for route path an path params, just like Tanstack Router 🔥
With this, everyone will have a better DX.
Suggested solution

Instead of using a static declaration file, we should generate one.
Wrapping react-router-dom/vue-router API to using types from generated declaration file. Then exports them.

Example:
// Generated declaration file

declare module '~react-pages' {
  import type { RouteObject } from 'react-router'
  import type { NavigateOptions, LinkProps } from 'react-router-dom'

  // Generated route paths
  export type RoutePaths = "/posts" | "/posts/:id";

  // Declare function generatePath
  type Split<T extends string> = T extends `${infer P}/`
    ? Split

      : T extends `/${infer P}`
    ? Split

      : T extends `${infer PL}/${infer PR}`
    ? Split<PL> | Split<PR>
      : T;

  type PathParamsKey<T extends string, K = Split<T>> = K extends `:${infer P}`
    ? P
      : K extends `...${infer P}`
    ? P
      : // eslint-disable-next-line @typescript-eslint/no-unused-vars
      K extends `${infer _P}*`
    ? "slug"
      : never;

  type PathParams<T extends string> = {
    [P in PathParamsKey<T>]: string | number;
  };

  export type GeneratePathOptions<
    T extends string,
    Params extends object = PathParams<T>
  > = {
    path: T;
  } & (keyof Params extends never ? { params?: Params } : { params: Params });

  export function generatePath<T extends string = RoutePaths>({
    path,
    params,
  }: GeneratePathOptions<T>): string;

  // Declare hook useNavigate
  export function useNavigate(): <T extends string = RoutePaths>(
    pathOptions: GeneratePathOptions<T>,
    options?: NavigateOptions
  ) => void

  // Declare component Link
  export function Link<T extends string = RoutePaths>(
    props: GeneratePathOptions<T> &
  Omit<LinkProps, "to">
  ): void

  // Export routes
  const routes: RouteObject[]
  export default routes
}

declare module 'virtual:generated-pages-react' {
  import type { RouteObject } from 'react-router'
  const routes: RouteObject[]
  export default routes
}

// Function generatePath

export default function generatePath<T extends string>({
  path,
  params,
}: GeneratePathOptions<T>): string {
  let result: string = path;

  if (params) {
    for (const key in params) {
      if (Object.prototype.hasOwnProperty.call(params, key)) {
        const value = params[key as keyof typeof params] as string;
        result = result.replace(`:${key}`, value);
      }
    }

    if ("slug" in params && result.endsWith("*")) {
      result = result.replace(/\*$/, params.slug as string);
    }
  }

  return result;
}

Then, we should exports generatePath, useNavigate and Link in virtual module, which currently only exports routes.
Alternative
No response
Additional context
No response
Validations

[X] Follow the Code of Conduct
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Out of scope and not planning to do.
Maybe someone can make a react version of unplugin-vue-router.


Option for removing the generation time at top of docs.go

Is your feature request related to a problem? Please describe.
I add the docs dir to my git repo, and I want add the docs generation step to git pre commit hook for automation.
But every time the hook executes, the timestamp at the top of docs.go changes, even no changes in other parts of docs.go
Describe the solution you'd like
A cli option for removing the timestamp.
Describe alternatives you've considered

Check the docs.go changes, if the content is not change, then no for the timestamp too.
Completely remove the timestamp, since we can always get the doc modify time and author from git.

For 2, there are some other tools used for generation won't add timestamp at their generated text.
For examples:

https://github.com/golang/mock
https://github.com/go-kit/kit

Additional context
None.

If i understand well , your git pre commit hook should force the user to not be able to commit if the latest commited docs.go file is different from the one generated  via hook. Correct ?

Yes, and thanks for your reply.
I'm using the pre-commit and it won't let me commit if the code is modified by the hooks, but it does not matter, what I am saying is the timestamp should not be added to git.
The hooks is simple:
- repo: local
  hooks:
  - id: generate-docs
    name: Generate docs
    pass_filenames: false
    entry: swag init --generalInfo cmd/server.go
    files: \.go$
    language: system
    types: [text]

I was thinking to a strategy like update-only , that will change the entire file if local file is different than the generated one (excluding the first 3 lines which is the header).
Unfortunately i was busy and i had no time to advance with this.

This feature is useful for my project where we have a Makefile target/CI job that checks if annotations have been changed without updating the swagger docs. It would be helpful if swag init had a flag to ignore/skip the timestamp.

Isn't this solved by --generatedTime and it already defaults to false?


Read Node not showing devices

Only shows count.  No errors in the log file.   I was able to have discovery save the config (see other bug reported today).  I can even export it or import config.
OS: Windows11 Pro
Browser: Edge

This is a known issue and will there will be an update release shortly

Hi @m2b
Can you please update your module to the latest version and try again ? Removing all the nodes from a flow and restarting node-red would be ideal.
Thanks !

I upgrade and have been testing.

So far, no exception on discovery. But where is it saving saving the config file?  I could not locate it.
It is however, showing 3 less devices on discovery than before.
As far as read, same problem.  Shows only count.  One time it showed on device with a count of 7 (there are 10 devices that Yabe can see). At the next discovery the device disappeared.
I am investigating if there is an issue since other application on the same host is reading the devices correctly.
I am working on setting up a separate host to run node-red and then try again.

My requirement is not discovery.  My limited experience is that it does not always work.  I have had device issues where rebooting the device is necessary to restart discovery.   Even Yabe has had issues discovering.    The other issue with discovery has been with Routers.  They don't always show their attached MSTP devices on discovery.
My preference is to have the device vendor to provide me a list of items in the device and then just setup the Bacnet read to use read those points.  Any chance that you can setup read to not depend on a discovered devices list?
In case you are interested the devices I am testing are Moxa gateway (Modbus to Bacnet), Accuenergy Power Meters, Flexim Flow meters, and JACE Controllers setup as a routers to Bacnet/MSTP Flexim Flow Meters.

ok, just a few quick points, make sure you don't have 2 gateway nodes on the same flow, the gateway is to be common and you can have multiple read and write nodes into the same gateway. You also can't run node bacnet on the same device that another BACnet application is running on

Hi @m2b  - could you please try again with the latest module version ? Please follow the updating steps in the readme if possible.

Please try again with latest version.
Closing this issue due to inactivity, it can be re opened at any time or a new issue created alternatively.


Fixes for asar archive

There were a few dev live reload regressions when using it in dev mode and atom/atom isn't cloned locally.
The issues:

Relied on fixtures no longer shipped with core, those fixtures are now included in this package
Tried to watch files included in asar archive which throws an error.

Closes #13

:+1:


Feature Requests

[x] Add an option for canceling colorized text output;
[x] Re-fromat output text as following fomat:

STEP TOTEN ...
   1 xx.xxx ...

Finished in https://github.com/Ionizing/usefultools-for-vasp/commit/f8e1005d311d9badeaf9cbdfe6fe95a1bc42b22b .


environment: transfer and adapt Env and SenseEnv from physic into Weather and SenseWeather

As part of this work, make it use context.Context.
The addition of the new packages can be done seamlessly in v3.5.0, but the removal of physic.Env and physic.SenseEnv is a breaking change.

Ported to https://github.com/periph/conn/issues/28.


Add whitelist string and array policies, including SNS Raw Attribute Delivery

This (partially) closes https://github.com/mj1618/serverless-offline-sns/issues/41.
These changes only support whitelist string filtering (including OR matching), which seems to cover the reported use cases. If needed, we can revisit and support the full range of supported filters
It also skips the SNS envelope with RawMessageDelivery (for SQS->SNS delivery). Closing #55 in favor of this PR

Thanks @mj1618!

@mj1618 do you know how long this will take to get merged and released? I'd like to set a reminder to update our local tooling to point back to a new version of Serverless Offline SNS, instead of my fork.
Thanks!

Done @astuyve

Thanks @mj1618!

Hi, i have this filter policy
    filterPolicy:
      operation:
      - "INSERT"
      newItemType:
      - prefix: "response"

and this is my code to publish sns data
/* eslint-disable no-console, import/no-extraneous-dependencies */
const AWS = require('aws-sdk');

const sns = new AWS.SNS({
  endpoint: 'http://<IP_ADDRESS>:4001',
  region: 'ap-southeast-2',
});

const stage = process.argv[2];

sns.publish(
  {
    Message: JSON.stringify({
      default: JSON.stringify({
        newItem: {
          ITEM_TYPE: 'response/onboarding/name',
          ITEM_VALUE:
            '{"S":"{\\"type\\":\\"name\\",\\"value\\":\\"shierro\\"}"}',
          ID: 'ResimaxVIP#1234',
        },
      }),
    }),
    MessageAttributes: {
      operation: {
        DataType: 'STRING_VALUE',
        StringValue: 'INSERT',
      },
      newItemType: {
        DataType: 'STRING_VALUE',
        StringValue: 'response',
      },
    },
    MessageStructure: 'json',
    TopicArn: `arn:aws:sns:ap-southeast-2:xxx:trm-conversation-${stage}`,
  },
  (err, result) => {
    console.log('err', err);
    console.log('new user published', result);
  },
);

but doesn't work. any advice on how to make the filter work offline?

Hey @shierro -
Right now prefix filters are not supported. We'd need to modify a fair bit of logic to support that.
Currently I'm not able to work on it, as I'm traveling for the next 30 days.


feat(batch): add batchMaxChunkSize option and change default to 200kb

Codecov Report

Merging #771 (95f8d46) into master (8087e18) will increase coverage by 0.00%.
The diff coverage is 60.00%.


Fix issues found by md-nits

Trivial stuff.  md-nits should be added to the CI (#15459)

This pull request is ready to merge

Merged
3e3ad3c54855dd534437871a0c78858de5e3d246 Fix issues found by md-nits


Bugfix: Reduce log noise

Reduce the severity of certain unused Neovim events to verbose (originally was warning)
Prettify oni-plugin-markdown-preview's package.json

Benefit: reduce clutter for a user that wants to view logs and report the actual error.

Thanks for doing this, @TalAmuyal ! 👍 , makes sense to downgrade these - they tend to clutter up error reports (or act as red herrings for real issues).


Remove postgres Python dependency

psycopg mucks up pipenv install for me on macOS and Linux, so let's remove it before the TAs try to build our project?

@reelmatt Good call, pushed another commit


bossac

For programming my Due with a Raspberry Pi I reset it with
stty -F /dev/ttyACM0 1200
Then I can flash it with bossac.

That's good to know, thanks.

You're welcome.


fix: add null comparison

hasValue test was failing because there was no null comparison.

Thank you so much for the pull request!
I think this issue has already been resolved with https://github.com/ipavlic/apex-lambda/commit/8912711f796336f365428ef959fe0fc21f09400d#diff-d905d470f8df4874878eb61379605e4e73d427d8b73fc189473d9b33ca0fa4c4 .
I think the logic is a bit off in the pull request. If both sides are null, then they are the same. That's fine. But otherwise, left side is always considered to be "greater than" right side.
This is a problem because null > 'a' and simultaneously 'a' > null, which doesn't make sense. Correct me if I'm wrong please. Thank you again!

Hi @wesley-prado, I'm going to go ahead and close this if there are no additional comments. Thank you!


Fix/12150 regexprep err

Which issue does this PR close?

Closes #12150 & #11912
Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Closing


Introspection query no longer works with GET

When used with GET, the introspection query sends variables=undefined. undefined being not valid JSON, it makes the request somewhat invalid.

fixed above


Make explicit list of fields required for portal to function, make missing entries invalidate the data

For example, principal_investigator_name was marked optional in the JSON schema but is required for the portal to function.
I believe this involves marking more things as required in the schema, hence I'm putting this issue on the metadata repo.

@jeffbaumes Many of these schema changes require data that I don't have access to.
@dehays can we discuss this at the Wed April 28 call?

@wdduncan  Yes, we can discuss this tomorrow.  The fields I think you are probably referring to are the ones on study that are not available in GOLD.
@jeffbaumes There are a number of these that I don't believe can be made required in the schema.  DOE, PI web sites, objective.  If values for these are not available from NMDC sources (GOLD, NCBI, etc.) then requiring them in the schema does not make sense.  In the cases where we do have values - we need to consider how they would be included.  I do believe that should happen upstream of search portal ingest; i.e. not provided to Kitware out-of-band in various files.

One possible course of action is an nmdc-portal LinkML file that imports nmdc.yaml, i.e.
# in e.g. nmdc-schema/src/schema/extensions/nmdc-portal.yaml
...
imports:
  - nmdc
...

Then, the portal schema can use the slot_usage facility to make certain inherited slots required, add slots, etc.
This way, the nmdc-portal requirements can be verified via tests in the nmdc-schema repo to be a proper extension of the base nmdc schema (e.g. no required fields are made optional, etc.), though I think for this issue such an error is unlikely - the portal only makes optional fields required and adds new fields.

@jeffbaumes There are a number of these that I don't believe can be made required in the schema. DOE, PI web sites, objective. If values for these are not available from NMDC sources (GOLD, NCBI, etc.) then requiring them in the schema does not make sense. In the cases where we do have values - we need to consider how they would be included. I do believe that should happen upstream of search portal ingest; i.e. not provided to Kitware out-of-band in various files.

@dehays if we know what must remain optional (i.e. we can't expect to require it on ingest or submission) then we should make that explicit and adapt our portal DB and UI to accommodate so it is resilient to those fields being absent. Basically @jbeezley is reporting what is required to drive the portal we see today. If we don't require these things some aspects of the interface will suffer (i.e. we can't expect to have PI/study pages that are all as nice as what is there today), which is fine but we need to judge with some finality what are the minimum requirements for inclusion in NMDC.

Adding notes exchanged via email here for reference:
Agree with your comments here David (and yours Kjiersten). I commented in parallel on GitHub, but the gist is that we need to resolve which fields we can and can't expect to require. Many "required" things for the portal could be made optional if needed.
https://github.com/microbiomedata/nmdc-schema/issues/41
On Wed, Apr 28, 2021 at 1:31 PM Kjiersten Fagnan<EMAIL_ADDRESS>wrote:
I support the approach David laid out for what fields are required vs optional. I can add the following comments to the ticket, but we seem to be getting into this via email.
Could we create some default values for the portal to populate the page - avatar, URL, DOI and scientific objectives would be harder if not impossible.
In the future, when contributors are providing data to NMDC, could we also collect - photo, website URL, etc as part of the submission process - or perhaps give the PI the ability to add this themselves? This depends on having some level of access controls (different roles in the data portal than exist right now). Maybe this is part of working directly with the PIs to get their help on the study/data landing pages?
Kjiersten
On Wed, Apr 28, 2021 at 10:20 AM David Hays<EMAIL_ADDRESS>wrote:
Bill and Emiley said:

https://github.com/microbiomedata/nmdc-schema/issues/41 I do not have access to the data for some of the fields that Jeff is requesting. I can give an estimate until we track the data.

@Emiley Eloe-Fadrosh any idea of who to follow up on with to get access to the data needed?

https://github.com/microbiomedata/nmdc-schema/issues/19 Again, I do not have access to images of PIs.

@Emiley Eloe-Fadrosh any idea of who to follow up on with to get access to the PI images needed?

David should be able to address #41, seems like a GOLD database dump issue?
For #19, as was indicated in the github ticket, these were all manually collected by me. Not sure the best solution here, but this could tie into the more general discussion of the study pages (and some items from #41 like scientific objectives that are not part of the GOLD db dump). Not everything can be fully automated.... just my two cents.

On #41, I believe the fields that Bill is referring to are the ones that are NOT available from GOLD; i.e. those listed in https://github.com/microbiomedata/nmdc-metadata/issues/301 and  https://github.com/microbiomedata/nmdc-schema/issues/19 such as PI web site, PI image, scientific objective, publication DOIs.  Basically, the ones that Emiley collected manually and provided to Kitware.
For these, Bill could add these to the schema as non required attributes.  He has no way of making the GOLD ETL populate these because they do not exist in GOLD.
Jon states that the portal UI depends on these fields - but I believe the portal UI will need to treat them as optional fields as well because normally they will not be available.  If we add 10K studies tomorrow from GOLD or NCBI - we will not be waiting for Emiley to collect values for these fields before they can be displayed in the UI.  The portal UI needs to be flexible enough to handle cases where these values are not available.
I also believe it should not be the responsibility of search portal development to merge additional metadata for studies to extend what was made available for ingest.  So that implies the need for an curate/annotate procedure that is available between GOLD or NCBI ETL and search portal ingest.  And in the case of images - in addition to a way to edit the study json docs to add PI image URLs, there is also the need to add and manage the image files to a location associated with the metadata URL.
So for #41, there are a number of fields that are always available (We will always have a PI for a study.) that can be made required in the schema.  But for those for which there is no available source except manual curation, at best these could be optional fields in the schema. Make sense?
-David
On Tue, Apr 27, 2021 at 1:51 PM Emiley Eloe-Fadrosh<EMAIL_ADDRESS>wrote:
For these two:

https://github.com/microbiomedata/nmdc-schema/issues/41 I do not have access to the data for some of the fields that Jeff is requesting. I can give an estimate until we track the data.

@Emiley Eloe-Fadrosh any idea of who to follow up on with to get access to the data needed?

https://github.com/microbiomedata/nmdc-schema/issues/19 Again, I do not have access to images of PIs.

@Emiley Eloe-Fadrosh any idea of who to follow up on with to get access to the PI images needed?

David should be able to address #41, seems like a GOLD database dump issue?
For #19, as was indicated in the github ticket, these were all manually collected by me. Not sure the best solution here, but this could tie into the more general discussion of the study pages (and some items from #41 like scientific objectives that are not part of the GOLD db dump). Not everything can be fully automated.... just my two cents.

Yes.  What @jeffbaumes said was my intent.
We need to ensure that the UI design takes into account information that may not be available.  Especially if the the common case is going to be that this information is missing, the current study page won't make much sense.  Where this information is missing on a study, it might be better to link to a page inviting PI's to submit the data to us directly.

Based on the meeting today with @dehays, @emileyfadrosh, @dwinston, @wdduncan, and @jbeezley, @wdduncan will add the missing fields into the schema for this sprint.

@wdduncan I'll move this to in progress per the last comment but let me know if you won't be able to add the missing fields this week or maybe it's already done?

@dehays and @wdduncan have discussed this and @dehays will close this issue and open several new related issues for June.

Closing this issue. Replaced by #50 and #51


Unable to run tests successfully as per documentation

I am trying to follow the documentation to setup my VM and successfully run the test suite, yet I'm facing multiple hurdles along the way.
Expected results

follow the instructions only,
run the test suite successfully.

Actual results
I'm facing some issues:

one OS dependency is not specified,
one needs an admin actually named admin to load examples
examples can't be all loaded in one go,
tests fail

How to reproduce the bug
On a new VM running Fedora 32:

OSError: mysql_config not found

# Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development

git clone git@github.com:apache/superset.git
cd superset
git checkout 1.0.1 # to make things reproducible

sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies

python3.8 -m venv venv
source venv/bin/activate
pip install -r requirements/local.txt

This throws an OSError: mysql_config not found
As per this SO question, the solution is to install mariadb-devel — what actually solves the issue.

Fail to load the examples

# Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development

git clone git@github.com:apache/superset.git
cd superset
git checkout 1.0.1 # to make things reproducible

sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies
sudo dnf -y install mariadb-devel # FIX as per https://stackoverflow.com/q/22949654/5433628

python3.8 -m venv venv
source venv/bin/activate
pip install -r requirements/local.txt
pip install .

superset fab create-admin --username adent --firstname arthur --lastname dent --email<EMAIL_ADDRESS>--password 42
superset db upgrade
superset init
superset load-examples

This returns the following error:
INFO:superset.utils.core:Creating database reference for examples
Loading examples metadata and related data into examples
Traceback (most recent call last):
  File "/home/fedora/superset/venv/bin/superset", line 11, in <module>
    load_entry_point('apache-superset==1.0.1', 'console_scripts', 'superset')()
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/cli.py", line 187, in load_examples
    load_examples_run(load_test_data, only_metadata, force)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/cli.py", line 120, in load_examples_run
    from superset import examples
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/examples/__init__.py", line 18, in <module>
    from .birth_names import load_birth_names
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/examples/birth_names.py", line 46, in <module>
    raise NoDataException(
superset.exceptions.NoDataException: Admin user does not exist. Please, check if test users are properly loaded (`superset load_test_users`).

We can then explicitely create an admin user:
# Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development

git clone git@github.com:apache/superset.git
cd superset
git checkout 1.0.1 # to make things reproducible

sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies
sudo dnf -y install mariadb-devel # FIX as per https://stackoverflow.com/q/22949654/5433628

python3.8 -m venv venv
source venv/bin/activate
pip install -r requirements/local.txt
pip install .

superset fab create-admin --username admin --firstname Ad --lastname Ministrator --email<EMAIL_ADDRESS>--password secret # FIX explicitely create an admin user with username exactly `admin`
superset db upgrade
superset init
superset load-examples

But the following error is still returned
[…]
INFO:superset.models.helpers:Updating table_columns gender
Loading data inside the import transaction
INFO:superset.datasets.commands.importers.v1.utils:Loading data inside the import transaction
Traceback (most recent call last):
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlite3.OperationalError: too many SQL variables

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/commands/importers/v1/examples.py", line 67, in run
    self._import(db.session, self._configs, self.overwrite, self.force_data)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/commands/importers/v1/examples.py", line 99, in _import
    dataset = import_dataset(
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/datasets/commands/importers/v1/utils.py", line 119, in import_dataset
    load_data(data_uri, dataset, example_database, session)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/datasets/commands/importers/v1/utils.py", line 146, in load_data
    df.to_sql(
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/core/generic.py", line 2605, in to_sql
    sql.to_sql(
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/io/sql.py", line 589, in to_sql
    pandas_sql.to_sql(
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/io/sql.py", line 1406, in to_sql
    raise err
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/io/sql.py", line 1398, in to_sql
    table.insert(chunksize, method=method)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/io/sql.py", line 830, in insert
    exec_insert(conn, keys, chunk_iter)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/pandas/io/sql.py", line 758, in _execute_insert_multi
    conn.execute(self.table.insert(data))
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
    return meth(self, multiparams, params)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
    ret = self._execute_context(
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) too many SQL variables
[SQL: INSERT INTO "FCC 2018 Survey" ("ID", is_software_dev, is_first_dev_job, months_job_search, job_pref, job_intr_fllstck, job_intr_backend, job_intr_frntend, job_intr_mobile, job_intr_devops, job_intr_datasci, job_intr_teacher, job_intr_qa_engn, job_intr_ux_engn, job_intr_projm, job_intr_gamedev, job_intr_infosec, job_intr_dataengn, job_intr_other, when_appl_job, expected_earn, job_lctn_pref, job_relocate, reasons_to_code, reasons_to_code_other, rsrc_fcc, rsrc_mdn, rsrc_so, rsrc_edx, rsrc_coursera, rsrc_khan_acdm, rsrc_pluralsght, rsrc_codeacdm, rsrc_udacity, rsrc_udemy, rsrc_code_wars, rsrc_treehouse, rsrc_hackerrank, rsrc_frntendmstr, rsrc_lynda, rsrc_egghead, rsrc_css_tricks, rsrc_other, codeevnt_fcc, codeevnt_hackthn, codeevnt_confs, codeevnt_workshps, codeevnt_startupwknd, codeevnt_nodeschl, codeevnt_womenwc, codeevnt_girldevit, codeevnt_coderdojo, codeevnt_meetup, codeevnt_railsbrdg, codeevnt_gamejam, codeevnt_railsgrls, codeevnt_djangogrls, codeevnt_wkndbtcmp, codeevnt_other, podcast_fcc, podcast_codenewbie, podcast_changelog, podcast_sedaily, podcast_js_jabber, podcast_syntaxfm, podcast_ltcwm, podcast_fullstckrd, podcast_frnthppyhr, podcast_codingblcks, podcast_shoptalk, podcast_devtea, podcast_progthrwdwn, podcast_geekspeak, podcast_hanselmnts, podcast_talkpythonme, podcast_rubyrogues, podcast_codepenrd, podcast_seradio, podcast_other, yt_mit_ocw, yt_fcc, yt_computerphile, yt_devtips, yt_cs_dojo, yt_engn_truth, yt_learncodeacdm, yt_lvluptuts, yt_funfunfunct, yt_codingtuts360, yt_codingtrain, yt_derekbanas, yt_simplilearn, yt_simpleprog, yt_mozillahacks, yt_googledevs, yt_other, hours_learning, months_programming, bootcamp_attend, bootcamp_name, bootcamp_finished, bootcamp_have_loan, bootcamp_recommend, money_for_learning, age, gender, gender_other, country_citizen, country_live, live_city_population, is_ethnic_minority, lang_at_home, school_degree, school_major, marital_status, has_finance_depends, has_children, num_children, do_finance_support, debt_amt, home_mrtg_has, home_mrtg_owe, student_debt_has, student_debt_amt, curr_emplymnt, curr_emplymnt_other, curr_field, last_yr_income, communite_time, is_self_employed, has_served_military, is_recv_disab_bnft, has_high_spd_ntnet, time_start, time_end, network_id, time_total_sec) VALUES

[…]

(Background on this error at: http://sqlalche.me/e/13/e3q8)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fedora/superset/venv/bin/superset", line 11, in <module>
    load_entry_point('apache-superset==1.0.1', 'console_scripts', 'superset')()
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/flask/cli.py", line 426, in decorator
    return __ctx.invoke(f, *args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/cli.py", line 187, in load_examples
    load_examples_run(load_test_data, only_metadata, force)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/cli.py", line 171, in load_examples_run
    examples.load_from_configs(force)
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/examples/utils.py", line 30, in load_from_configs
    command.run()
  File "/home/fedora/superset/venv/lib64/python3.8/site-packages/superset/commands/importers/v1/examples.py", line 71, in run
    raise self.import_error()
superset.commands.exceptions.CommandException

Weirdly, the solution seems to be… running superset load-examples again.

Test suite not passing

# Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development

git clone git@github.com:apache/superset.git
cd superset
git checkout 1.0.1 # to make things reproducible

sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies
sudo dnf -y install mariadb-devel # FIX as per https://stackoverflow.com/q/22949654/5433628

python3.8 -m venv venv
source venv/bin/activate
pip install -r requirements/local.txt
pip install .

superset fab create-admin --username admin --firstname Ad --lastname Ministrator --email<EMAIL_ADDRESS>--password secret # FIX explicitely create an admin user with username exactly `admin`
superset db upgrade
superset init
superset load-examples || superset load-examples # FIX

pip install -r requirements/testing.txt # FIX as, btw, this isn't specified but otherwise `tox` isn't installed
tox -e py38

and this fails:
[…]
======================================================= test session starts =======================================================
platform linux -- Python 3.8.7, pytest-6.1.2, py-1.9.0, pluggy-0.13.1
cachedir: .tox/py38/.pytest_cache
rootdir: /home/fedora/superset, configfile: pytest.ini, testpaths: tests
plugins: cov-2.10.1, celery-4.4.7
collected 1194 items
[…]
=========================== short test summary info ============================
FAILED tests/cache_tests.py::TestCache::test_no_data_cache - redis.exceptions...
FAILED tests/cache_tests.py::TestCache::test_slice_data_cache - redis.excepti...
FAILED tests/celery_tests.py::test_run_async_cta_query[TABLE] - sqlalchemy.ex...
FAILED tests/celery_tests.py::test_run_async_cta_query[VIEW] - sqlalchemy.exc...
FAILED tests/celery_tests.py::test_run_async_cta_query_with_lower_limit[TABLE]
FAILED tests/celery_tests.py::test_run_async_cta_query_with_lower_limit[VIEW]
FAILED tests/core_tests.py::TestCore::test_annotation_json_endpoint - Excepti...
FAILED tests/core_tests.py::TestCore::test_cache_logging - AttributeError: 'N...
FAILED tests/core_tests.py::TestCore::test_explore_json - AssertionError: 500...
FAILED tests/core_tests.py::TestCore::test_explore_json_async - AssertionErro...
FAILED tests/core_tests.py::TestCore::test_explore_json_async_results_format
FAILED tests/core_tests.py::TestCore::test_explore_json_data - AssertionError...
FAILED tests/core_tests.py::TestCore::test_explore_json_data_invalid_cache_key
FAILED tests/core_tests.py::TestCore::test_explore_json_dist_bar_order - KeyE...
FAILED tests/druid_tests.py::TestDruid::test_client - Exception: http request...
FAILED tests/druid_tests.py::TestDruid::test_druid_time_granularities - Excep...
FAILED tests/import_export_tests.py::TestImportExport::test_import_new_dashboard_slice_reset_ownership
FAILED tests/import_export_tests.py::TestImportExport::test_import_override_dashboard_slice_reset_ownership
FAILED tests/import_export_tests.py::TestImportExport::test_import_slices_override
FAILED tests/query_context_tests.py::TestQueryContext::test_cache - redis.exc...
FAILED tests/query_context_tests.py::TestQueryContext::test_csv_response_format
FAILED tests/query_context_tests.py::TestQueryContext::test_query_object_unknown_fields
FAILED tests/query_context_tests.py::TestQueryContext::test_samples_response_type
FAILED tests/query_context_tests.py::TestQueryContext::test_sql_injection_via_columns
FAILED tests/query_context_tests.py::TestQueryContext::test_sql_injection_via_groupby
FAILED tests/query_context_tests.py::TestQueryContext::test_sql_injection_via_metrics
FAILED tests/security_tests.py::TestRolePermission::test_alpha_permissions - ...
FAILED tests/security_tests.py::TestRolePermission::test_gamma_permissions - ...
FAILED tests/security_tests.py::TestRolePermission::test_gamma_permissions_basic
FAILED tests/security_tests.py::TestRolePermission::test_gamma_user_schema_access_to_tables
FAILED tests/security_tests.py::TestRolePermission::test_granter_permissions
FAILED tests/security_tests.py::TestRolePermission::test_hybrid_perm_database
FAILED tests/security_tests.py::TestRolePermission::test_hybrid_perm_druid_cluster
FAILED tests/security_tests.py::TestRolePermission::test_is_admin_only - Attr...
FAILED tests/security_tests.py::TestRolePermission::test_is_alpha_only - Attr...
FAILED tests/security_tests.py::TestRolePermission::test_is_gamma_pvm - Attri...
FAILED tests/security_tests.py::TestRolePermission::test_public_permissions_basic
FAILED tests/security_tests.py::TestRolePermission::test_public_sync_role_builtin_perms
FAILED tests/security_tests.py::TestRolePermission::test_public_sync_role_data_perms
FAILED tests/security_tests.py::TestRolePermission::test_schemas_accessible_by_user_admin
FAILED tests/security_tests.py::TestRolePermission::test_schemas_accessible_by_user_datasource_access
FAILED tests/security_tests.py::TestRolePermission::test_schemas_accessible_by_user_datasource_and_schema_access
FAILED tests/security_tests.py::TestRolePermission::test_schemas_accessible_by_user_schema_access
FAILED tests/security_tests.py::TestRolePermission::test_set_perm_database - ...
FAILED tests/security_tests.py::TestRolePermission::test_set_perm_druid_cluster
FAILED tests/security_tests.py::TestRolePermission::test_set_perm_slice - Att...
FAILED tests/security_tests.py::TestRolePermission::test_set_perm_sqla_table
FAILED tests/security_tests.py::TestRolePermission::test_sql_lab_permissions
FAILED tests/security_tests.py::TestRolePermission::test_sqllab_gamma_user_schema_access_to_sqllab
FAILED tests/security_tests.py::TestRolePermission::test_views_are_secured - ...
FAILED tests/utils_tests.py::TestUtils::test_log_this - Exception: http reque...
FAILED tests/annotation_layers/api_tests.py::TestAnnotationLayerApi::test_get_list_annotation_layer
FAILED tests/cachekeys/api_tests.py::test_invalidate_existing_cache - redis.e...
FAILED tests/cachekeys/api_tests.py::test_invalidate_existing_caches - redis....
FAILED tests/databases/api_tests.py::TestDatabaseApi::test_database_schemas
FAILED tests/databases/api_tests.py::TestDatabaseApi::test_update_database - ...
FAILED tests/databases/api_tests.py::TestDatabaseApi::test_update_database_uri_validate
FAILED tests/databases/commands_tests.py::TestExportDatabasesCommand::test_export_database_command
FAILED tests/tasks/async_queries_tests.py::TestAsyncQueries::test_load_chart_data_into_cache
FAILED tests/tasks/async_queries_tests.py::TestAsyncQueries::test_load_explore_json_into_cache
ERROR tests/access_tests.py::TestRequestAccess::test_clean_requests_after_schema_grant
ERROR tests/base_api_tests.py::TestBaseModelRestApi::test_default_missing_declaration_get
ERROR tests/base_api_tests.py::TestBaseModelRestApi::test_default_missing_declaration_put
ERROR tests/celery_tests.py::test_run_async_cta_query[TABLE] - sqlalchemy.exc...
ERROR tests/celery_tests.py::test_run_async_cta_query[VIEW] - sqlalchemy.exc....
ERROR tests/celery_tests.py::test_run_async_cta_query_with_lower_limit[TABLE]
ERROR tests/celery_tests.py::test_run_async_cta_query_with_lower_limit[VIEW]
ERROR tests/core_tests.py::TestCore::test_feature_flag_serialization - sqlalc...
ERROR tests/dashboard_tests.py::TestDashboard::test_save_dash_with_filter - s...
ERROR tests/dashboard_tests.py::TestDashboard::test_save_dash_with_invalid_filters
ERROR tests/import_export_tests.py::TestImportExport::test_export_2_dashboards
ERROR tests/import_export_tests.py::TestImportExport::test_import_1_slice - s...
ERROR tests/import_export_tests.py::TestImportExport::test_import_2_slices_for_same_table
ERROR tests/import_export_tests.py::TestImportExport::test_import_dashboard_1_slice
ERROR tests/schedules_test.py::TestSchedules::test_dashboard_chrome_like - sq...
ERROR tests/schedules_test.py::TestSchedules::test_deliver_dashboard_as_attachment
ERROR tests/schedules_test.py::TestSchedules::test_deliver_dashboard_inline
ERROR tests/schedules_test.py::TestSchedules::test_deliver_email_options - sq...
ERROR tests/schedules_test.py::TestSchedules::test_deliver_slice_attachment
ERROR tests/schedules_test.py::TestSchedules::test_deliver_slice_csv_attachment
ERROR tests/schedules_test.py::TestSchedules::test_deliver_slice_csv_inline
ERROR tests/schedules_test.py::TestSchedules::test_deliver_slice_inline_image
ERROR tests/security_tests.py::TestRolePermission::test_admin_permissions - s...
ERROR tests/security_tests.py::TestRolePermission::test_gamma_user_schema_access_to_charts
ERROR tests/security_tests.py::TestRolePermission::test_gamma_user_schema_access_to_dashboards
ERROR tests/security_tests.py::TestRolePermission::test_set_perm_druid_datasource
ERROR tests/utils_tests.py::TestUtils::test_build_extra_filters - sqlalchemy....
ERROR tests/charts/api_tests.py::TestChartApi::test_admin_gets_filtered_energy_slices
ERROR tests/charts/api_tests.py::TestChartApi::test_bulk_delete_chart_with_report
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_annotations - ...
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_applied_time_extras
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_async - redis....
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_async_invalid_token
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_async_results_type
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_cache - redis....
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_cache_key_error
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_cache_no_login
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_cache_run_failed
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_csv_result_format
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_default_row_limit
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_default_sample_limit
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_incorrect_request
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_incorrect_result_format
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_incorrect_result_type
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_invalid_form_data
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_jinja_filter_request
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_limit_offset
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_mixed_case_filter_op
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_no_data - redi...
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_prophet - redi...
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_query_missing_filter
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_query_result_type
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_rowcount - red...
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_simple - redis...
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_timegrains - r...
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_with_invalid_datasource
ERROR tests/charts/api_tests.py::TestChartApi::test_chart_data_with_invalid_enum_value
ERROR tests/charts/api_tests.py::TestChartApi::test_create_chart - redis.exce...
ERROR tests/charts/api_tests.py::TestChartApi::test_create_chart_validate_datasource
ERROR tests/charts/api_tests.py::TestChartApi::test_create_chart_validate_owners
ERROR tests/charts/api_tests.py::TestChartApi::test_create_chart_validate_params
ERROR tests/charts/api_tests.py::TestChartApi::test_create_simple_chart - red...
ERROR tests/charts/api_tests.py::TestChartApi::test_delete_bulk_chart_admin_not_owned
ERROR tests/charts/api_tests.py::TestChartApi::test_delete_bulk_chart_bad_request
ERROR tests/charts/api_tests.py::TestChartApi::test_delete_bulk_chart_not_owned
ERROR tests/charts/api_tests.py::TestChartApi::test_delete_bulk_charts - redi...
ERROR tests/charts/api_tests.py::TestChartApi::test_delete_bulk_charts_not_found
ERROR tests/charts/api_tests.py::TestChartApi::test_delete_chart - redis.exce...
ERROR tests/charts/api_tests.py::TestChartApi::test_delete_chart_admin_not_owned
ERROR tests/charts/api_tests.py::TestChartApi::test_delete_chart_not_owned - ...
ERROR tests/charts/api_tests.py::TestChartApi::test_delete_chart_with_report
ERROR tests/charts/api_tests.py::TestChartApi::test_delete_not_found_chart - ...
ERROR tests/charts/api_tests.py::TestChartApi::test_export_chart - redis.exce...
ERROR tests/charts/api_tests.py::TestChartApi::test_export_chart_gamma - redi...
ERROR tests/charts/api_tests.py::TestChartApi::test_export_chart_not_found - ...
ERROR tests/charts/api_tests.py::TestChartApi::test_get_chart - redis.excepti...
ERROR tests/charts/api_tests.py::TestChartApi::test_get_chart_no_data_access
ERROR tests/charts/api_tests.py::TestChartApi::test_get_chart_not_found - red...
ERROR tests/charts/api_tests.py::TestChartApi::test_get_charts - redis.except...
ERROR tests/charts/api_tests.py::TestChartApi::test_get_charts_changed_on - r...
ERROR tests/charts/api_tests.py::TestChartApi::test_get_charts_custom_filter
ERROR tests/charts/api_tests.py::TestChartApi::test_get_charts_favorite_filter
ERROR tests/charts/api_tests.py::TestChartApi::test_get_charts_filter - redis...
ERROR tests/charts/api_tests.py::TestChartApi::test_get_charts_no_data_access
ERROR tests/charts/api_tests.py::TestChartApi::test_get_charts_page - redis.e...
ERROR tests/charts/api_tests.py::TestChartApi::test_get_current_user_favorite_status
ERROR tests/charts/api_tests.py::TestChartApi::test_get_filter_related_owners
ERROR tests/charts/api_tests.py::TestChartApi::test_get_ids_related_owners - ...
ERROR tests/charts/api_tests.py::TestChartApi::test_get_related_fail - redis....
ERROR tests/charts/api_tests.py::TestChartApi::test_get_related_owners - redi...
ERROR tests/charts/api_tests.py::TestChartApi::test_get_repeated_ids_related_owners
ERROR tests/charts/api_tests.py::TestChartApi::test_get_time_range - redis.ex...
ERROR tests/charts/api_tests.py::TestChartApi::test_import_chart - redis.exce...
ERROR tests/charts/api_tests.py::TestChartApi::test_import_chart_invalid - re...
ERROR tests/charts/api_tests.py::TestChartApi::test_import_chart_overwrite - ...
ERROR tests/charts/api_tests.py::TestChartApi::test_info_security_chart - red...
ERROR tests/charts/api_tests.py::TestChartApi::test_not_update_chart_none_dashboards
ERROR tests/charts/api_tests.py::TestChartApi::test_query_exec_not_allowed - ...
ERROR tests/charts/api_tests.py::TestChartApi::test_update_chart - redis.exce...
ERROR tests/charts/api_tests.py::TestChartApi::test_update_chart_new_dashboards
ERROR tests/charts/api_tests.py::TestChartApi::test_update_chart_new_owner - ...
ERROR tests/charts/api_tests.py::TestChartApi::test_update_chart_not_owned - ...
ERROR tests/charts/api_tests.py::TestChartApi::test_update_chart_validate_datasource
ERROR tests/charts/api_tests.py::TestChartApi::test_update_chart_validate_owners
ERROR tests/charts/api_tests.py::TestChartApi::test_user_gets_none_filtered_energy_slices
ERROR tests/dashboards/api_tests.py::TestDashboardApi::test_export - sqlalche...
ERROR tests/dashboards/api_tests.py::TestDashboardApi::test_get_dashboard_not_found
ERROR tests/dashboards/commands_tests.py::TestExportDashboardsCommand::test_append_charts
ERROR tests/dashboards/commands_tests.py::TestExportDashboardsCommand::test_export_dashboard_command
ERROR tests/dashboards/commands_tests.py::TestExportDashboardsCommand::test_export_dashboard_command_invalid_dataset
ERROR tests/dashboards/commands_tests.py::TestExportDashboardsCommand::test_export_dashboard_command_key_order
ERROR tests/dashboards/commands_tests.py::TestExportDashboardsCommand::test_export_dashboard_command_no_access
ERROR tests/dashboards/dao_tests.py::TestDashboardDAO::test_set_dash_metadata
ERROR tests/databases/api_tests.py::TestDatabaseApi::test_get_database_related_objects
ERROR tests/datasets/commands_tests.py::TestImportDatasetsCommand::test_import_v0_dataset_cli_export
ERROR tests/datasets/commands_tests.py::TestImportDatasetsCommand::test_import_v0_dataset_ui_export
ERROR tests/reports/commands_tests.py::test_email_chart_report_schedule - sql...
ERROR tests/reports/commands_tests.py::test_email_dashboard_report_schedule
ERROR tests/reports/commands_tests.py::test_slack_chart_report_schedule - sql...
ERROR tests/reports/commands_tests.py::test_report_schedule_not_found - sqlal...
ERROR tests/reports/commands_tests.py::test_report_schedule_working - sqlalch...
ERROR tests/reports/commands_tests.py::test_report_schedule_working_timeout
ERROR tests/reports/commands_tests.py::test_report_schedule_success_grace - s...
ERROR tests/reports/commands_tests.py::test_report_schedule_success_grace_end
ERROR tests/reports/commands_tests.py::test_alert_limit_is_applied[alert1] - ...
ERROR tests/reports/commands_tests.py::test_alert_limit_is_applied[alert2] - ...
ERROR tests/reports/commands_tests.py::test_alert_limit_is_applied[alert3] - ...
ERROR tests/reports/commands_tests.py::test_alert_limit_is_applied[alert4] - ...
ERROR tests/reports/commands_tests.py::test_alert_limit_is_applied[alert5] - ...
ERROR tests/reports/commands_tests.py::test_alert_limit_is_applied[alert6] - ...
ERROR tests/reports/commands_tests.py::test_alert_limit_is_applied[alert7] - ...
ERROR tests/reports/commands_tests.py::test_email_dashboard_report_fails - sq...
ERROR tests/reports/commands_tests.py::test_slack_chart_alert[alert1] - sqlal...
ERROR tests/reports/commands_tests.py::test_slack_chart_alert[alert2] - sqlal...
ERROR tests/reports/commands_tests.py::test_slack_chart_alert[alert3] - sqlal...
ERROR tests/reports/commands_tests.py::test_slack_chart_alert[alert4] - sqlal...
ERROR tests/reports/commands_tests.py::test_slack_chart_alert[alert5] - sqlal...
ERROR tests/reports/commands_tests.py::test_slack_chart_alert[alert6] - sqlal...
ERROR tests/reports/commands_tests.py::test_slack_chart_alert[alert7] - sqlal...
ERROR tests/reports/commands_tests.py::test_email_chart_no_alert[alert1] - sq...
ERROR tests/reports/commands_tests.py::test_email_chart_no_alert[alert2] - sq...
ERROR tests/reports/commands_tests.py::test_email_chart_no_alert[alert3] - sq...
ERROR tests/reports/commands_tests.py::test_email_chart_no_alert[alert4] - sq...
ERROR tests/reports/commands_tests.py::test_email_chart_no_alert[alert5] - sq...
ERROR tests/reports/commands_tests.py::test_email_chart_no_alert[alert6] - sq...
ERROR tests/reports/commands_tests.py::test_email_chart_no_alert[alert7] - sq...
ERROR tests/reports/commands_tests.py::test_email_chart_no_alert[alert8] - sq...
ERROR tests/reports/commands_tests.py::test_email_chart_no_alert[alert9] - sq...
ERROR tests/reports/commands_tests.py::test_email_mul_alert[alert1] - sqlalch...
ERROR tests/reports/commands_tests.py::test_email_mul_alert[alert2] - sqlalch...
ERROR tests/reports/commands_tests.py::test_invalid_sql_alert[alert1] - sqlal...
ERROR tests/reports/commands_tests.py::test_invalid_sql_alert[alert2] - sqlal...
= 60 failed, 1113 passed, 16 skipped, 229 warnings, 152 errors in 1043.72s (0:17:23) =
ERROR: InvocationError for command /home/fedora/superset/.tox/py38/bin/pytest -s (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   py38: commands failed

One could also run the shell script to run tests faster:
# Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development

git clone git@github.com:apache/superset.git
cd superset
git checkout 1.0.1 # to make things reproducible

sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies
sudo dnf -y install mariadb-devel # FIX as per https://stackoverflow.com/q/22949654/5433628

python3.8 -m venv venv
source venv/bin/activate
pip install -r requirements/local.txt
pip install .

superset fab create-admin --username admin --firstname Ad --lastname Ministrator --email<EMAIL_ADDRESS>--password secret # FIX explicitely create an admin user with username exactly `admin`
superset db upgrade
superset init
superset load-examples || superset load-examples # FIX

pip install -r requirements/testing.txt # FIX as, btw, this isn't specified but otherwise `tox` isn't installed
scripts/tests/run.sh

but then it misses some OS dependencies:
------------------------------------
DB_URI=postgresql+psycopg2://superset:superset@localhost/test
Superset config module=tests.superset_test_config
Run init procedures=1
Run reset DB=1
Test to run:tests
------------------------------------
--------------------
Reseting test DB
--------------------
scripts/tests/run.sh: line 29: docker-compose: command not found
scripts/tests/run.sh: line 40: docker: command not found

So after installing it and setting it up:
# Following https://github.com/apache/superset/blob/master/CONTRIBUTING.md#setup-local-environment-for-development

git clone git@github.com:apache/superset.git
cd superset
git checkout 1.0.1 # to make things reproducible

sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies
sudo dnf -y install mariadb-devel # FIX as per https://stackoverflow.com/q/22949654/5433628

# FIX install docker as per https://docs.docker.com/engine/install/fedora/
sudo dnf -y install dnf-plugins-core
sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
sudo dnf install docker-ce docker-ce-cli containerd.io
sudo systemctl start docker
sudo systemctl enable docker
sudo groupadd docker
sudo usermod -aG docker $USER
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

sudo reboot

python3.8 -m venv venv
source venv/bin/activate
pip install -r requirements/local.txt
pip install .

superset fab create-admin --username admin --firstname Ad --lastname Ministrator --email<EMAIL_ADDRESS>--password secret # FIX explicitely create an admin user with username exactly `admin`
superset db upgrade
superset init
superset load-examples || superset load-examples # FIX

pip install -r requirements/testing.txt # FIX as, btw, this isn't specified but otherwise `tox` isn't installed
scripts/tests/run.sh

But I get
scripts/tests/run.sh
------------------------------------
DB_URI=postgresql+psycopg2://superset:superset@localhost/test
Superset config module=tests.superset_test_config
Run init procedures=1
Run reset DB=1
Test to run:tests
------------------------------------
--------------------
Reseting test DB
--------------------
Error: No such container: superset_db

Environment
(please complete the following information):

superset version: 1.0.1 (tested with 667eb83 — latest commit passing CI at 2021-04-09T14:58:35Z — as well)
python version: 3.8.7
node.js version: n/a

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional information
It may well be that I'm not reading the instructions correctly!

So I've been able to narrow down things a little.
Using following setup on a Fedora32 VM:
sudo dnf install -y git
git clone git@github.com:apache/superset.git
cd superset
git heavy_check_markout 1.1.0 # to make things reproducible
sudo dnf install -y gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel # as per https://superset.apache.org/docs/installation/installing-superset-from-scratch#os-dependencies
sudo dnf -y install mariadb-devel # FIX as per https://stackoverflow.com/q/22949654/5433628
python3.8 -m venv venv
source venv/bin/activate
pip install -r requirements/testing.txt
pip install .
superset fab create-admin --username admin --firstname Ad --lastname Ministrator --email<EMAIL_ADDRESS>--password secret # FIX explicitely create an admin user with username exactly `admin`
superset db upgrade
superset init
superset load-examples || superset load-examples # FIX
sudo dnf -y install redis
sudo systemctl enable --now redis

I can detail the result of various tests, that I run using:
tox -e py38 -- tests/<filename>.py 2>&1 | tee tox38_<filename>.log

Status
Test file
Result
Log file

:warning:
access_tests.py
9P, 9W, 1E
tox38_access_tests.log

:x:
alerts_tests.py
20F, 6P, 6W
tox38_alerts_tests.log

:warning:
base_api_tests.py
3P, 9W, 2E
tox38_base_api_tests.log

:heavy_check_mark:
base_tests.py

tox38_base_tests.log

:heavy_check_mark:
cache_tests.py

tox38_cache_tests.log

:x:
celery_tests.py
4F, 5P, 12W
tox38_celery_tests.log

:heavy_check_mark:
commands_test.py
2P, 7W
tox38_commands_tests.log

:heavy_check_mark:
config_tests.py
6P, 8W
tox38_config_tests.log

:gear:
conftest.py
6W
tox38_conftest.log

:x:
core_tests.py
61P, 38W, 1E
tox38_core_tests.log

:heavy_check_mark:
csv_upload_tests.py
2P, 2S, 10W
tox38_csv_upload_tests.log

:warning:
dashboard_tests.py
17P, 7W, 2E
tox38_dashboard_tests.log

:gear:
dashboard_utils.py

:heavy_check_mark:
dataframe_test.py
2P, 7W
tox38_dataframe_test.log

:x:
datasource_tests.py
1F, 2P, 7W
tox38_datasource_tests.log

:heavy_check_mark:
dict_import_export_tests.py
13P, 7W
tox38_dict_import_export_tests.log

:heavy_check_mark:
druid_func_tests.py
33P, 7W
tox38_druid_func_tests.log

:heavy_check_mark:
druid_func_tests_sip38.py
33P, 7W
tox38_druid_func_tests_sip38.log

:heavy_check_mark:
druid_tests.py
9P, 1S, 63W
tox38_druid_tests.log

:heavy_check_mark:
email_tests.py
8P, 7W
tox38_email_tests.log

:heavy_check_mark:
event_logger_tests.py
5P, 7W
tox38_event_logger_tests.log

:heavy_check_mark:
feature_flag_tests.py
3P, 7W
tox38_feature_flag_tests.log

:heavy_check_mark:
form_tests.py
2P, 7W
tox38_form_tests.log

:x:
import_export_tests.py
3F, 15P, 7W, 4E
tox38_import_export_tests.log

:gear:
__init__.py

:gear:
insert_chart_mixin.py

:heavy_check_mark:
jinja_context_tests.py
30P, 7W
tox38_jinja_context_tests.log

:heavy_check_mark:
log_api_tests.py
5P, 7W
tox38_log_api_tests.log

:heavy_check_mark:
logging_configurator_tests.py
1P, 6W
tox38_logging_configurator_tests.log

:heavy_check_mark:
migration_tests.py
1P, 7W
tox38_migration_tests.log

:heavy_check_mark:
model_tests.py
18P, 7W
tox38_model_tests.log

:heavy_check_mark:
pandas_postprocessing_tests.py
24P, 1S, 7W
tox38_pandas_postprocessing_tests.log

:x:
query_context_tests.py
2F, 13P, 61W
tox38_query_context_tests.log

:heavy_check_mark:
result_set_tests.py
15P, 7W
tox38_result_set_tests.log

:warning:
schedules_test.py
5P, 7W, 8E
tox38_schedules_test.log

:x:
security_tests.py
24F, 13P, 7W, 4E
tox38_security_tests.log

:heavy_check_mark:
sqla_models_tests.py
9P, 7W
tox38_sqla_models_tests.log

:heavy_check_mark:
sqllab_tests.py
30P, 8W
tox38_sqllab_tests.log

:heavy_check_mark:
sql_parse_tests.py
42P, 6W
tox38_sql_parse_tests.log

:heavy_check_mark:
sql_validator_tests.py
11P, 7W
tox38_sql_validator_tests.log

:heavy_check_mark:
stats_logger_tests.py
2P, 6W
tox38_stats_logger_tests.log

:heavy_check_mark:
strategy_tests.py
9P, 6W
tox38_stats_logger_tests.log

:gear:
superset_test_config.py

:gear:
superset_test_config_sqllab_backend_persist.py

:gear:
superset_test_config_thumbnails.py

:gear:
superset_test_custom_template_processors.py

:heavy_check_mark:
tagging_tests.py
1P, 1S, 7W
tox38_tagging_tests.log

:gear:
test_app.py

:x:
thumbnails_tests.py
2F, 10S, 7W
tox38_thumbnails_tests.log

:warning:
utils_tests.py
51P, 11W, 1E
tox38_thumbnails_tests.log

:heavy_check_mark:
viz_tests.py
55P, 20W
tox38_viz_tests.log

Where:

:x: tests failed (possibly with errors as well),
:warning: : tests didn't failed, yet had errors,
:heavy_check_mark: : tests ran fine (possibly with warnings),
:gear: : not a test file.

I still need to investigate the faulty ones more closely to understand why they fail.

@EBoisseauSierra looking at your logs, it appears that the example database is not being properly populated, causing the errors throughout.
Perhaps it could be easier to debug this from within superset itself. Can you run the flask and npm dev server and inspect the example database in SQL Lab? That could help indicate what's properly functioning vs. what's broken

Thanks @etr2460 for having had a deeper look.
I have uploaded here (:warning: 91MiB file) the SQLite db generated (in /home/fedora/.superset/superset.db) by the script detailed above.
I'm afraid I won't be able to prioritise investigating this (i.e. why running the tests fails when installing Superset bare metal) further at the moment, as I've been able to solve my actual issue (i.e. “how to run the test suite locally, so that if it passes there, the Github CI will be ok too?”).
To do so, I've given up on the “bare metal” install, and went for docker instead. The process is detailed here. I now intend to improve the related PR a bit further and close the whole matter once done.
How does that sound to you?

I hadn't seen #13970 , what is a great initiative. I suppose that helps too!


Intégration Error

Hello and thank you for this development
I have an error during the integration of the device and I cannot configure this one.
When I follow these steps :
1 - Go to your Home Assistant instance
2 - Go to "Settings" -> "Devices & Services" -> Click "+"
3 - Search for "Intex Spa" -> Select it
I have the following error message :
Error
The configuration stream could not be loaded : {"message":"Invalid handler specified"}
What can I do to solve this pb ?
Thanks in advance

Hi @Chezoimsvc
Please mention your home assistant version and provide logs

Hello Mathieu,
My version is :
Home Assistant 2023.7.2
Supervisor 2023.07.1
Operating System 10.3
Interface utilisateur : 20230705.1 - latest
And I have the following error in the log :
Logger: homeassistant.config_entries
Source: config_entries.py:874
First occurred: 17:08:18 (2 occurrences)
Last logged: 17:08:26
Error occurred loading flow for integration intex_spa: cannot import name 'IntexSpa' from 'intex_spa' (/usr/local/lib/python3.11/site-packages/intex_spa/init.py)

Hi @Chezoimsvc
I assume you've already tried reinstalling the integration from HACS.
So I'll check the behavior of initializing an intex_spa instance with the software versions you mentioned, as it could be a lead.

Hi @mathieu-mp
Yes, I redid the whole installation from HACS. (deleting and reinstalling with all the restarts between the different steps)
I also redid the installation of the intex-spa 0.8.4 package but I still have the same problem.

HI @mathieu-mp
I just updated Home Assistant:
Home Assistant 2023.7.3
Supervisor 2023.07.1
Operating System 10.3
Interface utilisateur : 20230705.1 - latest
Good news, the problem has disappeared !
Thank you for your help and for great dev

Great, that's good news!
And thank you for your follow-up


multi-ingressgateway installation by Istio operator will create a default ingressgateway?

Hi guys, I'm creating multi ingress gateways for staging and demo env in the same cluster by Istio operator, the operator yaml is as below. I named the two ingress gateway as  istio-ingressgateway-staging and  istio-ingressgateway-demo, but I found that the operator create a default ingress gateway named istio-ingressgateway. How can I disable the default name ingress gateway, or is this a feature of istio operator (to create a default name ingressgateway when the component is enabled)?
  components:
    base:
      enabled: true
    cni:
      enabled: false
    egressGateways:
    - enabled: false
      name: istio-egressgateway
    ingressGateways:
    - enabled: true
      k8s:
        env:
        - name: ISTIO_META_REQUESTED_NETWORK_VIEW
          value: network1
        service:
          ports:
          - name: status-port
            port: 15021
            targetPort: 15021
          - name: tls
            port: 15443
            targetPort: 15443
          - name: tls-istiod
            port: 15012
            targetPort: 15012
          - name: tls-webhook
            port: 15017
            targetPort: 15017
      label:
        app: istio-ingressgateway-staging
        istio: ingressgateway-staging
        topology.istio.io/network: network1
      name: istio-ingressgateway-staging
    - enabled: true
      k8s:
        env:
        - name: ISTIO_META_REQUESTED_NETWORK_VIEW
          value: network1
        service:
          ports:
          - name: status-port
            port: 15021
            targetPort: 15021
          - name: tls
            port: 15443
            targetPort: 15443
          - name: tls-istiod
            port: 15012
            targetPort: 15012
          - name: tls-webhook
            port: 15017
            targetPort: 15017
      label:
        app: istio-ingressgateway-demo
        istio: ingressgateway-demo
        topology.istio.io/network: network1
      name: istio-ingressgateway-demo

Thanks for your time!

Thanks, it works. @zirain