Blocked login and "challenge required"

Hi,
there's a problem when Instagram detects a "suspicious login attempt" and forces the user to receive a code via email to confirm that the user is actually the person that is logging in. I don't know what triggered this double check in the first place, but now it is basically impossible to login with Instagram4j.
This is the output:
2018-08-24 16:58:52 [main] INFO  org.brunocvcunha.instagram4j.Instagram4j  - Setup...
2018-08-24 16:58:52 [main] INFO  org.brunocvcunha.instagram4j.Instagram4j  - Device ID is: android-896793c78ea8383f, random id: fa7226d0-a7be-11e8-9b00-0242a06024bd
2018-08-24 16:58:52 [main] INFO  org.brunocvcunha.instagram4j.Instagram4j  - Logging with user ***** and password ***************
2018-08-24 16:58:52 [main] INFO  org.brunocvcunha.instagram4j.Instagram4j  - Sending request: org.brunocvcunha.instagram4j.requests.internal.InstagramReadMsisdnHeaderRequest
2018-08-24 16:58:52 [main] INFO  org.brunocvcunha.instagram4j.Instagram4j  - Sending request: org.brunocvcunha.instagram4j.requests.internal.InstagramFetchHeadersRequest
2018-08-24 16:58:53 [main] INFO  org.brunocvcunha.instagram4j.requests.InstagramRequest  - Reading StatusResult from {"status": "ok"}
2018-08-24 16:58:54 [main] INFO  org.brunocvcunha.instagram4j.requests.InstagramRequest  - Reading StatusResult from {"status": "ok"}
2018-08-24 16:58:54 [main] INFO  org.brunocvcunha.instagram4j.Instagram4j  - Sending request: org.brunocvcunha.instagram4j.requests.internal.InstagramSyncFeaturesRequest
2018-08-24 16:58:55 [main] INFO  org.brunocvcunha.instagram4j.requests.InstagramRequest  - Reading InstagramSyncFeaturesResult from {"ig_search_client_h1_2017_holdout": {"behavior": "pass"}, "ig_android_video_captions_universe": {"mode": "only_muted"}, "ig_and
2018-08-24 16:58:55 [main] INFO  org.brunocvcunha.instagram4j.Instagram4j  - Sending request: org.brunocvcunha.instagram4j.requests.internal.InstagramZeroRatingTokenRequest
2018-08-24 16:58:55 [main] INFO  org.brunocvcunha.instagram4j.Instagram4j  - Sending request: org.brunocvcunha.instagram4j.requests.internal.InstagramLogAttributionRequest
2018-08-24 16:58:56 [main] INFO  org.brunocvcunha.instagram4j.requests.InstagramRequest  - Reading StatusResult from {"status": "ok"}
2018-08-24 16:58:56 [main] INFO  org.brunocvcunha.instagram4j.Instagram4j  - Sending request: org.brunocvcunha.instagram4j.requests.InstagramLoginRequest
2018-08-24 16:58:57 [main] INFO  org.brunocvcunha.instagram4j.requests.InstagramRequest  - Reading InstagramLoginResult from {"message": "challenge_required", "challenge": {"url": "https://i.instagram.com/challenge/2456900006/yIgSw8q5bv/", "api_path": "
2018-08-24 16:58:57 [main] INFO  org.brunocvcunha.instagram4j.Instagram4j  - Challenge required: InstagramChallenge(url=https://i.instagram.com/challenge/********/, api_path=/challenge/*******/)

(The last line is the relevant one)
I've redacted some sensitive information, but when going to the challenge URL there is a button to send a confirmation code to the email that then has to be entered. The code has to be entered from the same device that is trying to login, so it would be useful if the API would provide a way to enter the confirmation code and send it back to Instagram, in order to be able to login again. I would be very glad if somebody could a workaround to this issue, if there is any available with the current implementation.

Thing is that I'm using the same IP address for two other accounts (without any proxy) and they work without issues (at least they are working now). It could be as you say, but I think that if there was a way to send the confirmation code from the bot it would whitelist the device  and allow it to login (it already asked me on my phone if that login attempt was mine and I clearly said yes, so the confirmation code should clear all doubts).

Yeah it's random, I've used IP addresses that work on one account but the same IP address doesn't work for the other, as you said in your original post it gets the challenge required, I accept it, but the subsequent login request gets sent back to the challenge required infinite loop. The only way I have been able to solve this is to use a residential proxy. This has been the only solution for me.

how about this https://github.com/mgp25/Instagram-API/wiki/FAQ#what-does-checkpoint_required-error-mean ?
let us know if this resolved your issue.

It gives this challenge required when running this locally too - no proxy or external server involved

Yeah. Same problem.
Even if i login to my account from web-browser and confirm, that this was me - i still get the error.
Someone has a solution?

I do not know if it will help you but basically what I did was find the bottleneck, put a breakpoint lo method login () and left the program execution stopped at line 201, allowed the device in the browser checking the option: "yes, it was me "and then I returned to the execution of the program, it worked ... Now just make an implementation.

@Vorkytaka you need to do full verification challenge process from the API from the same session.  See my comment here: https://github.com/brunocvcunha/instagram4j/issues/308#issuecomment-536248658


Segment merge lineage data structure

Added segment merge lineage that is a wrapper class of ZNRecord
Added segment merge group that will be used by broker during segment
selection process
Added a unit test

Codecov Report

Merging #3543 into master will decrease coverage by 0.04%.
The diff coverage is 79.64%.

@@             Coverage Diff              @@
##             master    #3543      +/-   ##
============================================
- Coverage     70.89%   70.85%   -0.05%
  Complexity        4        4
============================================
  Files           998     1001       +3
  Lines         45669    45836     +167
  Branches       6108     6132      +24
============================================
+ Hits          32379    32479     +100
- Misses        11176    11222      +46
- Partials       2114     2135      +21

Impacted Files
Coverage Δ
Complexity Δ

...edin/pinot/common/metadata/ZKMetadataProvider.java
65.21% <0%> (-4.79%)
0 <0> (ø)

...ommon/lineage/SegmentMergeLineageAccessHelper.java
0% <0%> (ø)
0 <0> (?)

...om/linkedin/pinot/common/lineage/SegmentGroup.java
100% <100%> (ø)
0 <0> (?)

...edin/pinot/common/lineage/SegmentMergeLineage.java
85.4% <85.4%> (ø)
0 <0> (?)

...core/operator/dociditerators/AndDocIdIterator.java
51.72% <0%> (-31.04%)
0% <0%> (ø)

...a/manager/realtime/RealtimeSegmentDataManager.java
75% <0%> (-25%)
0% <0%> (ø)

...lix/EmptyBrokerOnlineOfflineStateModelFactory.java
86.66% <0%> (-13.34%)
0% <0%> (ø)

...impl/dictionary/FloatOffHeapMutableDictionary.java
87.27% <0%> (-5.46%)
0% <0%> (ø)

...pinot/core/operator/docidsets/OrBlockDocIdSet.java
94.33% <0%> (-3.78%)
0% <0%> (ø)

... and 25 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce6a53f...e553144. Read the comment docs.

#2715


chore: Single component setup

This PR changes release-please manifest setup to single-component setup.

I thought "generation diff / release-please-config" would fail.

"generation diff / root-pom" filed:
Running sed command. It may take few minutes.
xargs: cannot fit single argument within argument list size limit

I thought "generation diff / release-please-config" would fail.

Now it is failing. But it is not required, we don't need it anyway now. We can delete it later on.

Right. These files are no longer needed


Fresh versions

Pull request to update dependencies to their latest versions.

Looks good to me. Tests passing and clean build works.


toascii - Add a Metadata Header to JSON Lines Format

It would be useful to have some metadata in every JSON file output.
Fields to Include

Version number
Commit-id tool was built against

Also a bool for uncommitted changes? (Only of interest to gfxr devs)

Source trace filename
Set of flags passed to tool (any command-line switches and env vars affecting output)

Maybes

is_trimmed: bool
Frame range captured
Number of lines [optional]

! It probably can't be added at the front of the file until the whole file is generated so perhaps not worthwhile.

List of functions in capture

is_trimmed: bool

I'm not sure we know that until after we've started parsing the capture file.

is_trimmed: bool

I'm not sure we know that until after we've started parsing the capture file.

Good point. Curse the lexicographer who didn't make sure "true" and "false" occupied the same number of characters. We could always seek back and overwrite a default false with the help of a cheeky space:
"false,"
"true, "

@bradgrantham-lunarg How much of the source file's path do we want to expose in the header?

Just the filename?

(-) Might lose useful information encoded in the directory structure for some users (e.g. /captures/windows/ihv_red/driver_version/trimmed/optimised/game_one/gfxrecon_capture.20220928T155218.gfxr )

Full path to capture file?

(-) Could expose sensitive info in path. E.g. (/secret_project_for_client_x/feature_john_is_blocking_but_ill_get_in_during_his_pto/captured_by_employee_first_name_second_name/gfxrecon_capture.20220928T155218.gfxr)

@brad I have some dummy values being output on the first line of the file. They look like this:
{"header":{"source":"gfxrecon_capture.20220928T155218.gfxr","version":"0.8.0","commit":"2ab193321131ab2d120b46bdc0723c0c8704e95c"}}


Perf tracking

Opening this as a meta-issue to track low hanging fruit for perf wins.
### Tasks
- [ ] https://github.com/chainguard-dev/apko/issues/772
- [ ] https://github.com/chainguard-dev/apko/issues/773
- [ ] https://github.com/chainguard-dev/apko/issues/775
- [ ] https://github.com/chainguard-dev/apko/issues/776
- [ ] https://github.com/chainguard-dev/apko/issues/777
- [ ] https://github.com/chainguard-dev/apko/issues/778
- [ ] https://github.com/chainguard-dev/apko/issues/779
- [ ] https://github.com/chainguard-dev/apko/issues/780
- [ ] https://github.com/chainguard-dev/go-apk/pull/78
- [ ] https://github.com/chainguard-dev/apko/pull/793
- [ ] https://github.com/chainguard-dev/apko/issues/799
- [ ] https://github.com/chainguard-dev/go-apk/pull/85
- [ ] https://github.com/chainguard-dev/go-apk/pull/87
- [ ] https://github.com/chainguard-dev/go-apk/pull/86
- [ ] https://github.com/chainguard-dev/go-apk/pull/82

State of the world before we start optimizing things too much:
$ pwd
/Users/jonjohnson/src/github.com/chainguard-images/images

$ time apko publish --arch amd64 images/go/configs/1.19.apko.yaml localhost:8081/go
...
apko publish --arch amd64 images/go/configs/1.19.apko.yaml localhost:8081/go  27.45s user 4.15s system 242% cpu 13.009 total

After https://github.com/chainguard-dev/apko/pull/782
apko publish --arch amd64 images/go/configs/1.19.apko.yaml localhost:8081/go  14.16s user 3.83s system 151% cpu 11.892 total

Mild speedup, but huge reduction in CPU usage.

You can see the relative length of BuildLayer shrinking vs buildImage.

After https://github.com/chainguard-dev/go-apk/pull/74

After https://github.com/chainguard-dev/go-apk/pull/75
apko publish --keyring-append  --repository-append  --arch amd64    13.46s user 4.56s system 227% cpu 7.917 total

Shaved off a little under a second.

Been a while since an update...
Here's a cold cache:
apko publish --keyring-append  --repository-append  --arch amd64    12.49s user 3.26s system 123% cpu 12.779 total

Here's warm:
apko publish --keyring-append  --repository-append  --arch amd64    9.03s user 1.56s system 216% cpu 4.882 total

Got another half second from some bufio here and there.
Cold:
apko publish --keyring-append  --repository-append  --arch amd64    12.20s user 2.79s system 132% cpu 11.289 total

Hot:
apko publish --keyring-append  --repository-append  --arch amd64    9.06s user 1.56s system 251% cpu 4.223 total

I have a branch that gets us down to ~3s on the hot path, but it's a bit of a dead end because it mostly just makes the work we're already doing a little bit more concurrent, which doesn't actually help that much in a build-the-world scenario.
This is HEAD:

This is my branch:

At least in these two flamegraphs, the exact same 9.26 seconds of CPU time is getting done.
Looking at where we're spending that time...
About a third of our CPU time is in pgzip compressing the final layer. Since we're doing a parallel compression, this only takes ~850ms, so that's about the speed of light for us on a hot path:

We spend 1.5 serially writing things to disk and walking the filesystem to read them back from disk:

Meanwhile, we are gunzipping the data section of each APK, so that we are paying that time 2x (just concurrently):

A bit of a surprising result is that we spent a third of a second just cleaning up the temporary directory we created:

Then we spend a surprising amount of time pushing images, but that's mostly because docker desktop won't stop touching my config file 🙄

cat ~/.docker/config.json | jq .credsStore
"desktop"

If we drop that, things look a little better here:

The SBOM generation is still pretty slow. I'm going to see if I can shift some of that left, but I managed to cut 1/3 of it in https://github.com/chainguard-dev/apko/pull/801
Anyway, looking back at where we're spending our time, very roughly:
1s writing a bunch of files to disk
0.5s reading a bunch of files from disk
0.5s cleaning up a bunch of files from disk
1s compressing things and other stuff
I have a plan to index the data section of APKs (really, just extract the tar headers) when we download them for the first time, then use that to avoid writing everything to disk. Instead, we can figure out what we would write to disk, what files would get overwritten by what would be written to disk by subsequent APKs, and also what files would be affected by the apko config stuff (chmod and whatnot)... then we just (in parallel) compress the subset of files from each APK that would have ended up in the final layer and at the very end append all these gzipped tarballs together with a bunch of metadata we compute along the way.
It should look something like this:
~1s writing a bunch of files to disk~
~0.5s reading a bunch of files from disk~
~0.5s cleaning up a bunch of files from disk~
1s compressing things and other stuff
That last bit of compressing stuff will now happen even more concurrently than with pgzip, so I'm guessing that will bring us well under a second (on a hot path).
The next step after that would be to write some fun software that takes advantage of some details in DEFLATE to much more efficiently modify/recompress the existing APK's data section, which would shrink that latency by ~4-5x and get us closer to 250ms, at which point it will make some sense to revisit where we are spending our time.

So I'm not sure that we have the same performance constraints here, but you may find pantsbuild/pex#2175 interesting, especially the medusa-zip tool to rearrange zip files really fast at https://github.com/cosmicexplorer/medusa-zip. It's not quite the same thing as taking advantage of DEFLATE, but one extreme crime I have performed is the hackery to read out the contents of a zip archive into another one without touching the file stream at all: https://github.com/cosmicexplorer/zip/blob/94c21b77b21db4133a210f335e0671f4ea85d6a0/src/read.rs#L331-L392. The zip format was made for messing around like this; I would love to see more crimes against DEFLATE too.

This isn't exactly relevant except that I happened to be working on it at the same time as the above, but in https://github.com/pypa/pip/issues/12184#issuecomment-1676002698 I demonstrate the performance impact of creating a local index for pip which gets lazily updated as it crawls dependencies. Since I believe we discussed one result of this being the publication of indices for positions referencing some other compressed targz stream, I wanted to note instead that in a related but different application, I was able to generate local indices for resources as they were crawled, amortizing that transformation per-node. I would recommend trying that approach first here if you haven't solved the problem already by now.

It would also seem very much within the scope of something like medusa-zip to handle the creation of such indices in a streaming manner when a targz is first downloaded.
The following is mostly a note to self:
One additional concern that arose from the zip merging solution investigated in pantsbuild/pex#2158 and pantsbuild/pex#2175 was that (as initially proposed) merging zip files from a shared cache would also take up more disk space than before (to create the cached zips). While handling that cache is in one sense an application-specific issue (see pantsbuild/pex#2201), if we also expand this medusa-zip archive library's capabilities to cover targz merging via creation of decompressed indices (and therefore hand over responsibility for the lifecycle of filesystem cache entries to that library/service), we could have it handle cache eviction/etc for the local entries it handles.
I'll create a separate issue if I have further thoughts on any of this and stop derailing this thread.

Although, regarding this approach in particular:

I have a plan to index the data section of APKs (really, just extract the tar headers) when we download them for the first time, then use that to avoid writing everything to disk. Instead, we can figure out what we would write to disk, what files would get overwritten by what would be written to disk by subsequent APKs, and also what files would be affected by the apko config stuff (chmod and whatnot)... then we just (in parallel) compress the subset of files from each APK that would have ended up in the final layer and at the very end append all these gzipped tarballs together with a bunch of metadata we compute along the way.

In order to execute build processes in isolated chroots that can be cached and executed remotely via the bazel remexec api, pants maintains a virtual filesystem consisting of merkle trees stored in an LMDB content-addressed store, which can be efficiently synced against a remote database (since the db only contains a mapping of (checksum) -> (byte string), and entries are stored as encoded protobufs). It exposes this to build tasks with a pretty novel API.
Your problem here can be solved without the global deduplication that pants performs, but I wanted to mention how encoding directory contents into merkle trees is a useful general approach for performing (as you said) "figure out what we would write to disk, what files would get overwritten by what would be written to disk by subsequent APKs, and also what files would be affected by the apko config stuff (chmod and whatnot)...". This act of normalization into a db-friendly format (in pants's case, converting directory trees into protobufs referencing other entities by checksum) may be the link that lets us meaningfully generalize this into a library, one which:

efficiently reads/normalizes zips/targz into a local LMDB store

encoded into protobufs like pants

efficiently computes the result of superimposing/transforming a sequence of normalized directory trees,

without filesystem operations; this is what pants does with Digest and Snapshot

has methods to efficiently export a normalized directory tree into zip/targz

not done in pants, but see e.g. pantsbuild/pants#19049 for similar optimizations

After https://github.com/chainguard-dev/go-apk/pull/98
Cold
~11s -> 4.9s
This came mostly from being able to fetch and decompress in parallel, which speeds up the installation phase.

Hot
~4.2s -> 2.6s
We still have that faster install phase but we get to skip the fetch phase entirely.

After https://github.com/chainguard-dev/apko/pull/860
Using --offline flag (can't do this cold) on the hot path saves ~200ms mostly from avoiding TLS handshake at the beginning.

With https://github.com/chainguard-dev/apko/pull/867
Building cgr.dev/chainguard/go for both amd64 and arm64...
cold
4.9s -> 3.7s
We are mostly limited here by how quickly we can fetch and decompress each APK.

hot
2.6s -> 1.5s
We spend most of our time now in pgzip with a bit of time burned doing TLS handshakes at the beginning and SBOM generation (giant JSON document rendering) at the end.

offline
2.4s -> 1.3s

The next phase is to take this (CPU) hungry hungry pgzippopotamus and replace it with something that can go faster with less CPU. I'd even be with with a slightly slower implementation that would allow use much less CPU.
There is a particularly ambitious optimization we can perform where we could stitch together pre-existing DEFLATE streams when we know that their decompressed contents are identical, which would let us reuse the CPU-intensive parts of compressing all these files.
Ensuring that the decompressed contents are identical is very difficult in the general case, but we can skip that difficulty by taking advantage of APK checksums where we already do know that the contents are identical. This requires writing a custom DEFLATE encoder, which might be out of reach for the amount of time I have here, but I want to write it here for posterity in case I come back to it.


coverage

bring back coverage reports for tests

https://github.com/threepointone/glamor/commit/59ce83a6afbc3ef6463a29481432944bed814d9b


HTTPS support

Provide a way to access Local Beach projects via HTTPS (https://foo-bar.localbeach.net).

✅


database connection issue with v1.22.6

Is there an existing issue for this?

[X] I have searched the existing issues

Output of ddev debug test
Expand `ddev debug test` diagnostic information
Running bash [-c /var/folders/bd/w600ldtx14z4fdjhmk_3dj1m0000gn/T/test_ddev.sh]
======= Existing project config =========
These config files were loaded for project tsad8: [/Users/james/PhpStormProjects/tsad8_gov_build/.ddev/config.yaml]
name: tsad8
type: drupal10
docroot: docroot
php_version: 8.2
webserver_type: apache-fpm
webimage: ddev/ddev-webserver:v1.22.6
router_http_port: 80
router_https_port: 443
xdebug_enabled: true
additional_hostnames: []
additional_fqdns: []
database: {mariadb 10.4}
hooks: map[post-import-db:[map[exec:drush cr] map[exec:drush updb -y] map[exec:drush cr]] post-start:[map[composer:install --no-interaction]]]
upload_dir: docroot/sites/default/files
upload_dirs: [docroot/sites/default/files]
webimage_extra_packages: [exiftool]
project_tld: ddev.site
use_dns_when_possible: true
composer_version: 2
nodejs_version: 16
default_container_timeout: 120
======= Creating dummy project named  tryddevproject-7233 in ../tryddevproject-7233 =========
OS Information: Darwin Jamess-MacBook-Pro-2.local 22.3.0 Darwin Kernel Version 22.3.0: Mon Jan 30 20:42:11 PST 2023; root:xnu-8792.81.3~2/RELEASE_X86_64 x86_64
ProductName:		macOS
ProductVersion:		13.2.1
BuildVersion:		22D68
User information: uid=501(james) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae),701(com.apple.sharepoint.group.1)
DDEV version:  ITEM             VALUE
 DDEV version     v1.22.6
 architecture     amd64
 db               ddev/ddev-dbserver-mariadb-10.4:v1.22.6
 ddev-ssh-agent   ddev/ddev-ssh-agent:v1.22.6
 docker           24.0.7
 docker-compose   v2.23.3
 docker-platform  orbstack
 mutagen          0.17.2
 os               darwin
 router           ddev/ddev-traefik-router:v1.22.6
 web              ddev/ddev-webserver:v1.22.6
PROXY settings: HTTP_PROXY='' HTTPS_PROXY='' http_proxy='' NO_PROXY=''
======= DDEV global info =========
Global configuration:
instrumentation-opt-in=true
omit-containers=[]
performance-mode=mutagen
router-bind-all-interfaces=false
internet-detection-timeout-ms=3000
disable-http2=false
use-letsencrypt=false
letsencrypt-email=
table-style=bright
simple-formatting=false
use-hardened-images=false
fail-on-hook-fail=false
required-docker-compose-version=v2.23.3
use-docker-compose-from-path=false
project-tld=
xdebug-ide-location=
no-bind-mounts=false
router=traefik
wsl2-no-windows-hosts-mgt=false
router-http-port=80
router-https-port=443
mailpit-http-port=8025
mailpit-https-port=8026
traefik-monitor-port=10999

======= DOCKER info =========
docker location: lrwxr-xr-x  1 james  admin  34 Nov 22 12:27 /usr/local/bin/docker -> ../Cellar/docker/24.0.7/bin/docker
Docker Desktop Version: Print: Entry, ":CFBundleShortVersionString", Does Not Exist
Print: Entry, ":CFBundleVersion", Does Not Exist
Docker Desktop for Mac File build Doesn'tDocker Desktop for Mac Exist, build WillDocker Desktop for Mac Create: build /Applications/Docker.app/Contents/Info.plistDocker Desktop for Mac File build Doesn'tDocker Desktop for Mac Exist, build WillDocker Desktop for Mac Create: build /Applications/Docker.app/Contents/Info.plist
docker version:
Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.21.3
 Git commit:        afdd53b4e3
 Built:             Thu Oct 26 07:06:42 2023
 OS/Arch:           darwin/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:05:28 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.8
  GitCommit:        8e4b0bde866788eec76735cc77c4720144248fb7
 runc:
  Version:          1.1.9
  GitCommit:        82f18fe0e44a59034f3e1f45e475fa5636e539aa
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
DOCKER_DEFAULT_PLATFORM=notset
======= Mutagen Info =========
Mutagen is installed in ddev, version=0.17.2
--------------------------------------------------------------------------------
Name: tsad8
Identifier: sync_7CCGypSalx7q57scwLiaHn85edXexWcXOoMU3h9jvIk
Labels:
	com.ddev.config-hash: 654a7799e053320033f4baa45c0f9c345f2546e4
	com.ddev.volume-signature: Users-james-orbstack-run-1707336854
Configuration:
	Synchronization mode: Two Way Resolved
	Hashing algorithm: Default (SHA-1)
	Maximum allowed entry count: Default (2⁶⁴−1)
	Maximum staging file size: Default (18 EB)
	Symbolic link mode: POSIX Raw
	Ignore VCS mode: Default (Propagate)
	Ignores:
		/.git
		/.tarballs
		/.ddev/db_snapshots
		/.ddev/.importdb*
		.DS_Store
		.idea
		/docroot/sites/default/files
	Permissions mode: Default (Portable)
Alpha:
	URL: /Users/james/PhpStormProjects/tsad8_gov_build
	Configuration:
		Watch mode: Default (Portable)
		Watch polling interval: Default (10 seconds)
		Probe mode: Default (Probe)
		Scan mode: Default (Accelerated)
		Stage mode: Neighboring
		File mode: Default (0600)
		Directory mode: Default (0700)
		Default file/directory owner: Default
		Default file/directory group: Default
	Connected: No
Beta:
	URL: docker://ddev-tsad8-web/var/www/html
		DOCKER_HOST=unix:///Users/james/.orbstack/run/docker.sock
	Configuration:
		Watch mode: Default (Portable)
		Watch polling interval: Default (10 seconds)
		Probe mode: Default (Probe)
		Scan mode: Default (Accelerated)
		Stage mode: Neighboring
		File mode: Default (0600)
		Directory mode: Default (0700)
		Default file/directory owner: Default
		Default file/directory group: Default
		Compression: Default (DEFLATE)
	Connected: No
Status: [Paused]
--------------------------------------------------------------------------------
Name: tryddevproject-1452
Identifier: sync_XfiXzMUJDrKqXNTbaxuK9dKZ3PhmvYdbEMatq1PmNTa
Labels:
	com.ddev.config-hash: 0311359b0f6eedb428c2679de60b5d021c2d717b
	com.ddev.volume-signature: Users-james-orbstack-run-1707337210
Configuration:
	Synchronization mode: Two Way Resolved
	Hashing algorithm: Default (SHA-1)
	Maximum allowed entry count: Default (2⁶⁴−1)
	Maximum staging file size: Default (18 EB)
	Symbolic link mode: POSIX Raw
	Ignore VCS mode: Default (Propagate)
	Ignores:
		/.git
		/.tarballs
		/.ddev/db_snapshots
		/.ddev/.importdb*
		.DS_Store
		.idea
	Permissions mode: Default (Portable)
Alpha:
	URL: /Users/james/PhpStormProjects/tryddevproject-1452
	Configuration:
		Watch mode: Default (Portable)
		Watch polling interval: Default (10 seconds)
		Probe mode: Default (Probe)
		Scan mode: Default (Accelerated)
		Stage mode: Neighboring
		File mode: Default (0600)
		Directory mode: Default (0700)
		Default file/directory owner: Default
		Default file/directory group: Default
	Connected: Yes
	Synchronizable contents:
		28 directories
		76 files (106 kB)
		0 symbolic links
Beta:
	URL: docker://ddev-tryddevproject-1452-web/var/www/html
		DOCKER_HOST=unix:///Users/james/.orbstack/run/docker.sock
	Configuration:
		Watch mode: Default (Portable)
		Watch polling interval: Default (10 seconds)
		Probe mode: Default (Probe)
		Scan mode: Default (Accelerated)
		Stage mode: Neighboring
		File mode: Default (0600)
		Directory mode: Default (0700)
		Default file/directory owner: Default
		Default file/directory group: Default
		Compression: Default (DEFLATE)
	Connected: Yes
	Synchronizable contents:
		28 directories
		76 files (106 kB)
		0 symbolic links
Status: Watching for changes
--------------------------------------------------------------------------------
======= Docker Info =========
Docker platform: orbstack
Using Docker context: default (unix:///Users/james/.orbstack/run/docker.sock)
docker-compose: v2.23.3
Using DOCKER_HOST=unix:///Users/james/.orbstack/run/docker.sock
Docker version: 24.0.7
Able to run simple container that mounts a volume.
Able to use internet inside container.
Docker disk space:
Filesystem                Size      Used Available Use% Mounted on
overlay                 162.7G     66.3G     96.4G  41% /

 Container ddev-tryddevproject-1452-db  Stopped
 Container ddev-tryddevproject-1452-web  Stopped
 Container ddev-tryddevproject-1452-web  Stopped
 Container ddev-tryddevproject-1452-db  Stopped
 Container ddev-tryddevproject-1452-db  Removed
 Container ddev-tryddevproject-1452-web  Removed
 Network ddev-tryddevproject-1452_default  Removed
Project tryddevproject-1452 has been stopped.
Stopped Mutagen daemon
The ddev-ssh-agent container has been removed. When you start it again you will have to use 'ddev auth ssh' to provide key authentication again.
Network ddev_default removed
Existing docker containers:
CONTAINER ID   IMAGE          COMMAND     CREATED      STATUS    PORTS     NAMES
6c750dfb9524   17b365050b12   "/bin/sh"   7 days ago   Created             phpstorm_helpers_PS-223.8836.42
Network ddev_default created
Starting tryddevproject-7233...
Network ddev-tryddevproject-7233_default created
 Container ddev-ssh-agent  Created
 Container ddev-ssh-agent  Started
ssh-agent container is running: If you want to add authentication to the ssh-agent container, run 'ddev auth ssh' to enable your keys.
Building project images...
Project images built in 0s.
 Container ddev-tryddevproject-7233-web  Created
 Container ddev-tryddevproject-7233-db  Created
 Container ddev-tryddevproject-7233-web  Started
 Container ddev-tryddevproject-7233-db  Started
You have Mutagen enabled and your 'php' project type doesn't have `upload_dirs` set.
For faster startup and less disk usage, set upload_dirs to where your user-generated files are stored.
If this is intended you can disable this warning with `ddev config --disable-upload-dirs-warning`.
Starting Mutagen sync process...
Mutagen sync flush completed in 1s.
For details on sync status 'ddev mutagen st tryddevproject-7233 -l'
Waiting for web/db containers to become ready: [web db]
Starting ddev-router if necessary...
 Container ddev-router  Created
 Container ddev-router  Started
Waiting for additional project containers to become ready...
All project containers are now ready.
Successfully started tryddevproject-7233
Project can be reached at https://tryddevproject-7233.ddev.site https://<IP_ADDRESS>:32852
======== Curl of site from inside container:
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 07 Feb 2024 20:23:37 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding

======== curl -I of http://tryddevproject-7233.ddev.site from outside:
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Date: Wed, 07 Feb 2024 20:23:37 GMT
Server: nginx
Vary: Accept-Encoding

======== full curl of http://tryddevproject-7233.ddev.site from outside:
Success accessing database... db via TCP/IP
ddev is working. You will want to delete this project with 'ddev delete -Oy tryddevproject-7233'
======== Project ownership on host:
drwxr-xr-x  4 james  staff  128 Feb  7 13:22 ../tryddevproject-7233
======== Project ownership in container:
drwxr-xr-x 1 james dialout 16 Feb  7 20:23 /var/www/html
======== In-container filesystem:
Filesystem     Type  1K-blocks     Used Available Use% Mounted on
/dev/vdb1      btrfs 170573192 69685296 100946440  41% /var/www
======== curl again of tryddevproject-7233 from host:
Success accessing database... db via TCP/IP
ddev is working. You will want to delete this project with 'ddev delete -Oy tryddevproject-7233'
Thanks for running the diagnostic. It was successful.
Please provide the output of this script in a new gist at gist.github.com
Running ddev launch in 5 seconds

Please run cleanup after debugging with 'ddev debug testcleanup'

Expected Behavior
no database issues
Actual Behavior
PHP loses connection to MySQL during transactions. This occurs when running som Drupal update hooks,
PHP Fatal error:  Uncaught Error: Call to a member function inTransaction() on null in /var/www/html/docroot/core/modules/mysql/src/Driver/Database/mysql/Connection.php:483
Stack trace:
#0 /var/www/html/docroot/core/modules/mysql/src/Driver/Database/mysql/Connection.php(415): Drupal\mysql\Driver\Database\mysql\Connection->doCommit()
#1 /var/www/html/docroot/core/lib/Drupal/Core/Database/Connection.php(1447): Drupal\mysql\Driver\Database\mysql\Connection->popCommittableTransactions()
#2 /var/www/html/docroot/core/lib/Drupal/Core/Database/Transaction.php(71): Drupal\Core\Database\Connection->popTransaction('drupal_transact...')
#3 [internal function]: Drupal\Core\Database\Transaction->__destruct()
#4 {main}
  thrown in /var/www/html/docroot/core/modules/mysql/src/Driver/Database/mysql/Connection.php on line 483

Fatal error: Uncaught Error: Call to a member function inTransaction() on null in /var/www/html/docroot/core/modules/mysql/src/Driver/Database/mysql/Connection.php on line 483

Error: Call to a member function inTransaction() on null in /var/www/html/docroot/core/modules/mysql/src/Driver/Database/mysql/Connection.php on line 483

Call Stack:
    9.9992   78603384   1. Drupal\Core\Database\Transaction->__destruct() /var/www/html/docroot/core/lib/Drupal/Core/Database/Transaction.php:0
    9.9992   78603384   2. Drupal\Core\Database\Connection->popTransaction($name = 'drupal_transaction') /var/www/html/docroot/core/lib/Drupal/Core/Database/Transaction.php:71
    9.9992   78603384   3. Drupal\mysql\Driver\Database\mysql\Connection->popCommittableTransactions() /var/www/html/docroot/core/lib/Drupal/Core/Database/Connection.php:1447
    9.9992   78603760   4. Drupal\mysql\Driver\Database\mysql\Connection->doCommit() /var/www/html/docroot/core/modules/mysql/src/Driver/Database/mysql/Connection.php:415

Steps To Reproduce
As best as I can determine problem is that sometime after the database connection is made the Connection variable gets set to NULL.
I upgraded to DDEV v1.22.5 to get access to PHP 8.3, then to v1.22.6 to get access to xdebug in PHP 8.3. I began to have problems in December seeing the "call to member function on null" error. I spent some time struggling with it before a co-worker commented about having problems. Comparing notes we determined that we were having the same problem, talked to a third co-worker and when he wasn't having problem working on the same project determined that the DDEV version was the issue. Downgrading back to v1.22.5 solved the problem, but I lose access to xdebug in PHP 8.3 ☹.
I am experiencing this problem with at least two different websites and ddev configurations (granted one DDEV config is a copy of the other). Different code bases (both Drupal 10) and databases.
I would think it was a Drupal 10 or a module issue if it weren't for the fact that downgrading DDEV fixes the problem.
Anything else?
No response

This sure looks like a problem with your PHP code. I don't think you can get this error without a problem in your PHP code.
I'm sure you know that different versions of DDEV have different patch version of PHP 8.3. What do you have?
You should be able to use xdebug just fine with webimage_extra_packages: [php8.3-xdebug] which gets you the latest xdebug version. Also, DDEV v1.22.7 will be out today or tomorrow with an updated image.

DDEV v1.22.7 is now available, which has PHP 8.3.2 and Xdebug 3.3.1. Could you please test with it?
What you're seeing is almost certainly a code problem, but could have something to do with PHP patch version I supposed.

I have verified that the problem still exists for me in DDEV v1.22.7... I will now begin comparing projects and try to see which code is shared between them all that might be causing the problem.

In my investigating I found this Drupal.org issue Transaction autocommit during shutdown relies on unreliable object destruction order (xdebug 3.3+ enabled) which makes it appear to be an xdebug change that caused the problem. This is why changing the PHP version from 8.3 to 8.2 or 8.2 didn't fix the issue, only rolling back to ddev v1.22.5... I assume that DDEV v1.22.5 has a different xdebug version,

Yes, thanks. that issue is actually mentioned in the v1.22.7 release notes and has been widely shared as well. Thanks for reporting back.
I believe xdebug for PHP 8.3 wasn't available until 1.22.6, and that it became available because of xdebug 3.3.*

Xdebug 3.3.2 has been released and claims to solve some of the crashes. It is in the upstream deb.sury.org packages as of yesterday. You may get relief with
webimage_extra_packages: ["php${DDEV_PHP_VERSION}-xdebug"]

Do a ddev restart to pick it up.
Please report back on your results

Hi @rfay
Thank you for the fast response.
I tried adding the line and ddev restart
But have same error.
Stack trace:
#0 /var/www/html/public/core/modules/mysql/src/Driver/Database/mysql/Connection.php(402): Drupal\mysql\Driver\Database\mysql\Connection->doCommit()
#1 /var/www/html/public/core/lib/Drupal/Core/Database/Connection.php(1678): Drupal\mysql\Driver\Database\mysql\Connection->popCommittableTransactions()
#2 /var/www/html/public/core/lib/Drupal/Core/Database/Transaction.php(71): Drupal\Core\Database\Connection->popTransaction('drupal_transact...')
#3 [internal function]: Drupal\Core\Database\Transaction->__destruct()
#4 {main}
  thrown in /var/www/html/public/core/modules/mysql/src/Driver/Database/mysql/Connection.php on line 470

Fatal error: Uncaught Error: Call to a member function inTransaction() on null in /var/www/html/public/core/modules/mysql/src/Driver/Database/mysql/Connection.php on line 470

Error: Call to a member function inTransaction() on null in /var/www/html/public/core/modules/mysql/src/Driver/Database/mysql/Connection.php on line 470

Call Stack:
    5.0119   61282616   1. Drupal\Core\Database\Transaction->__destruct() /var/www/html/public/core/lib/Drupal/Core/Database/Transaction.php:0
    5.0119   61282616   2. Drupal\Core\Database\Connection->popTransaction($name = 'drupal_transaction') /var/www/html/public/core/lib/Drupal/Core/Database/Transaction.php:71
    5.0119   61282616   3. Drupal\mysql\Driver\Database\mysql\Connection->popCommittableTransactions() /var/www/html/public/core/lib/Drupal/Core/Database/Connection.php:1678
    5.0119   61282992   4. Drupal\mysql\Driver\Database\mysql\Connection->doCommit() /var/www/html/public/core/modules/mysql/src/Driver/Database/mysql/Connection.php:402

PHP 8.1.27 (cli) (built: Apr 11 2024 22:37:31) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.27, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.27, Copyright (c), by Zend Technologies
    with Xdebug v3.3.2, Copyright (c) 2002-2024, by Derick Rethans

I'm currently restricted to use 9.5.11

Thanks for the help @rfay

I was unaware that the  issue affected Drupal 9, which is EOL anyway. So you definitely won't get the fix.
I'm no expert on the problem, but it appears that it was discovered in 10.2.x 5 months ago.
You should be able to use the workarounds in https://stackoverflow.com/questions/77582344/how-can-i-install-xdebug-on-ddev-for-php-8-3-before-its-ready-in-a-release/77583024#77583024 to downgrade xdebug to 3.2.x as mentioned in the Drupal.org issue.
The maintainer of Xdebug believes that these issues are resolved in 3.3.2, https://github.com/xdebug/xdebug/releases/tag/3.3.2


Adds developer guide

Signed-off-by: Ashwin Pc<EMAIL_ADDRESS>Description

Adds a developer guide to the repo
fixes liniting errors

Issues Resolved
#290
#307 (Partial fix) Left a todo for one of the changes that that was commented out after speaking to the developer
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Sorry for checking this PR late. But we do have a developer guide under contributing md. https://github.com/opensearch-project/opensearch-dashboards-functional-test/blob/main/CONTRIBUTING.md#developer-guide which contains more detailed instructions.
Probably you can add the instructions over there. Then I can do a bulk move over to the proper folder structure.

@tianleh Ah missed the testing and onboarding sections. Would you rather have me move those sections from contributing here instead? That way we dont need to duplicate the work. If you prefer to do the move yourself, i can add my sections in the contributing doc.Let me know which one is ideal.

@tianleh Ah missed the testing and onboarding sections. Would you rather have me move those sections from contributing here instead? That way we dont need to duplicate the work. If you prefer to do the move yourself, i can add my sections in the contributing doc.Let me know which one is ideal.

Let me take care of the bulk restructure. You can add your sections into the existing doc https://github.com/opensearch-project/opensearch-dashboards-functional-test/blob/main/CONTRIBUTING.md#developer-guide I am in the middle of creating another PR which may address some of the linting issues. Once you sync my latest change, your PR will be much simpler.
Wish I checked this earlier to avoid the convenience. @ashwin-pc

No problem! I'll hold off on this for a lil then.

https://github.com/opensearch-project/opensearch-dashboards-functional-test/pull/317 is merged. Please rebase from main and update PR accordingly

I will sync up with @ashwin-pc about the plan to move forward.

closing this since its superseded by changes made in #403


Wait For Email: UID SEARCH Failure due to backend error

Wait For Email is failing if specify sender argument.
${latest}=    Wait For Email    subject=Welcome!    timeout=30
Works fine but if you specify sender then it starts failing.
${latest}=    Wait For Email<EMAIL_ADDRESS>  subject=Welcome!    timeout=30
Fails with error:

FAIL : imap.search error: NO, [b'[UNAVAILABLE] UID SEARCH Failure due to backend error.

I'm using @lasselindqvist ImapLibrary2 as it has the latest changes but unfortunately couldn't report the issue there.

I have now enabled issues in the fork: https://github.com/lasselindqvist/robotframework-imaplibrary2/issues

Thanks @lasselindqvist I will move the issues to your repository.

Issue is moved to https://github.com/lasselindqvist/robotframework-imaplibrary2/issues/2


Didn't understand this line in the code

Hi Yuji, Thanks a lot for creating this simple tutorial for webrtc,I had a question for this snippet below, you are passing connection.csdo as parameter to CreateOffer, csdo is scoped_refptr but the parameter  which CreateOffer accept  is void PeerConnection::CreateOffer(CreateSessionDescriptionObserver observer,*  , we are passing scoped_refptr as a parameter why is it  working ? shouldn't we pass connection.csdo.get() as a parameter ?
Problem with this line below:
connection.peer_connection->CreateOffer(connection.csdo, webrtc::PeerConnectionInterface::RTCOfferAnswerOptions());

Hi PunitSalian.
The answer to your question is the definition below in scoped_refptr.
at include/api/scoped_refptr.h

operator T*() const { return ptr_; }

scoped_refptr  will act as a simple pointer when necessary without get() or explicit cast.
The way the value is passed in here changes, depending on the version of the WebRTC.
When I updated from an earlier version, scoped_refptr was necessary, but we may not have to use the scoped_refptr recently version.

Thanks Yuji


panic in net.forEachRoute when starting container via proxy

$ eval $(weave env)
$ docker run --name=foo -ti ubuntu
root@foo:/# exit
exit
$ docker run --name=foo1 --hostname=foo.weave.local -ti ubuntu
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x20 pc=0x458102]

goroutine 1 [running]:
github.com/weaveworks/weave/net.forEachRoute(0xc8200a00f0, 0xc820039be8, 0x0, 0x0)
	/go/src/github.com/weaveworks/weave/net/route.go:49 +0x232
github.com/weaveworks/weave/net.CheckRouteExists(0x55c068, 0x5, 0xc820039c68, 0x10, 0x10, 0x0)
	/go/src/github.com/weaveworks/weave/net/route.go:63 +0xa8
github.com/weaveworks/weave/net.EnsureInterfaceAndMcastRoute(0x55c068, 0x5, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/weave/net/if.go:56 +0x338
main.checkNetwork(0x0, 0x0)
	/go/src/github.com/weaveworks/weave/prog/weavewait/check_network_iface_mcast.go:10 +0x38
main.main()
	/go/src/github.com/weaveworks/weave/prog/weavewait/main.go:20 +0x6b

goroutine 6 [chan send]:
github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink.LinkSubscribe.func2(0xc820014120, 0xc82000a0e0)
	/go/src/github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink/link_linux.go:847 +0x2e6
created by github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink.LinkSubscribe
	/go/src/github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink/link_linux.go:850 +0x107

goroutine 17 [runnable]:
syscall.Syscall(0x3, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/asm_linux_amd64.s:18 +0x5
syscall.Close(0x4, 0x0, 0x0)
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:296 +0x4a
github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink/nl.(*NetlinkSocket).Close(0xc82009e220)
	/go/src/github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:325 +0x24
github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink.RouteSubscribe.func1(0xc8200b6060, 0xc82009e220)
	/go/src/github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink/route_linux.go:306 +0x44
created by github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink.RouteSubscribe
	/go/src/github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink/route_linux.go:307 +0xe6

goroutine 18 [runnable]:
syscall.Syscall6(0x2d, 0x4, 0xc8200ca000, 0x1000, 0x0, 0xc82001ec48, 0xc82001ec44, 0x6c, 0x1000, 0x0)
	/usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
syscall.recvfrom(0x4, 0xc8200ca000, 0x1000, 0x1000, 0x0, 0xc82001ec48, 0xc82001ec44, 0x40e819, 0x0, 0x0)
	/usr/local/go/src/syscall/zsyscall_linux_amd64.go:1712 +0x91
syscall.Recvfrom(0x4, 0xc8200ca000, 0x1000, 0x1000, 0x0, 0x1000, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/syscall/syscall_unix.go:244 +0xb8
github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink/nl.(*NetlinkSocket).Receive(0xc82009e220, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink/nl/nl_linux.go:341 +0xae
github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink.RouteSubscribe.func2(0xc8200b6000, 0xc82009e220)
	/go/src/github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink/route_linux.go:312 +0x61
created by github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink.RouteSubscribe
	/go/src/github.com/weaveworks/weave/vendor/github.com/vishvananda/netlink/route_linux.go:324 +0x115
Error response from daemon: Failure during network configuration for container 08f8447ca51cf71a1c6ec5b0612fdc4212bc977740dcda0522fd30accd24204c:
nsenter: cannot open /hostproc/15566/ns/net: No such file or directory
```

I got this once and have not been able to reproduce it since.

Might be related to #1898, though I am running a 3.19 kernel.

To reproduce, apply
diff --git a/net/route.go b/net/route.go
index 3424826..988a7d7 100644
--- a/net/route.go
+++ b/net/route.go
@@ -3,6 +3,7 @@ package net
 import (
        "fmt"
        "net"
+       "time"

        "github.com/vishvananda/netlink"
 )
@@ -39,6 +40,7 @@ func forEachRoute(ignoreIfaceNames map[string]struct{}, check
        if err != nil {
                return err
        }
+       time.Sleep(1 * time.Minute)
        for _, route := range routes {
                link, err := netlink.LinkByIndex(route.LinkIndex)
                if err == nil {

rebuild weave, and then
$ weave launch
(this will take ~1 minute due to the above patch)
$ eval $(weave env)
$ docker run --name=foo -ti ubuntu
...
in another window...
$ sudo bin/docker-ns foo ip link del ethwe

ATM I cannot think of a way for this bug to occur that doesn't involve some interface going away. Which, while possible in general, should not have happened in the scenario where I encountered the problem. So there may well be another issue here.

I've just run into this again. The proxy logs contain
INFO: 2016/01/19 14:11:37.257649 Attaching container cb0e24a3cf39fa3788a73b2ea20a3f1a6ae1118a4462d477988de152b9798e3a with WEAVE_CIDR "" to weave network
WARN: 2016/01/19 14:11:37.979612 Attaching container cb0e24a3cf39fa3788a73b2ea20a3f1a6ae1118a4462d477988de152b9798e3a to weave network failed: Failure during network configuration for container cb0e24a3cf39fa3788a73b2ea20a3f1a6ae1118a4462d477988de152b9798e3a:
nsenter: cannot open /hostproc/13221/ns/net: No such file or directory
nsenter: cannot open /hostproc/13221/ns/net: No such file or directory

The container logs only have the stacktrace.
I've also had another failure, this time with just one nsenter complaint.

Further investigation shows that CheckRoute is tripping over the route {Ifindex: 343 Dst: <IP_ADDRESS>/16 Src: <IP_ADDRESS> Gw: <nil> Flags: []}, returning Link not Found from netlink.LinkByIndex. The ifindex corresponds to eth0, which does exist:
$ sudo bin/docker-ns 29ec ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
343: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
    link/ether 02:42:ac:11:00:04 brd ff:ff:ff:ff:ff:ff
345: ethwe: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1410 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 3e:f7:a1:b7:2e:89 brd ff:ff:ff:ff:ff:ff

So it really looks like netlink.LinkByIndex is lying.


ImportError: cannot import name 'check_argument_types' from 'typeguard'

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/setuptools/depends.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/special/orthogonal.py:81: DeprecationWarning: np.int is a deprecated alias for the builtin int. To silence this warning, use int by itself. Doing this will not modify any behavior and is safe. When replacing np.int, you may wish to use e.g. np.int64 or np.int32 to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
from numpy import (exp, inf, pi, sqrt, floor, sin, cos, around, int,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/sparse/sputils.py:16: DeprecationWarning: np.typeDict is a deprecated alias for np.sctypeDict.
supported_dtypes = [np.typeDict[x] for x in supported_dtypes]
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/linalg/init.py:217: DeprecationWarning: The module numpy.dual is deprecated.  Instead of using dual, use the functions directly from numpy or scipy.
from numpy.dual import register_func
Traceback (most recent call last):
File "/home/aistudio/.data/webide/pip/bin/paddlespeech", line 8, in
sys.exit(_execute())
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/cli/entry.py", line 40, in _execute
exec("from {} import {}".format(module, cls))
File "", line 1, in
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/cli/text/init.py", line 14, in
from .infer import TextExecutor
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/cli/text/infer.py", line 29, in
from paddlespeech.text.models.ernie_linear import ErnieLinear
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/text/models/init.py", line 15, in
from .ernie_linear import ErnieLinear
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/text/models/ernie_linear/init.py", line 16, in
from .ernie_linear_updater import *
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/text/models/ernie_linear/ernie_linear_updater.py", line 24, in
from paddlespeech.t2s.training.extensions.evaluator import StandardEvaluator
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/t2s/init.py", line 19, in
from . import models
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/t2s/models/init.py", line 14, in
from .ernie_sat import *
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/t2s/models/ernie_sat/init.py", line 14, in
from .ernie_sat import *
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/t2s/models/ernie_sat/ernie_sat.py", line 21, in
from paddlespeech.t2s.modules.activation import get_activation
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/t2s/modules/init.py", line 16, in
from .losses import *
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/t2s/modules/losses.py", line 23, in
from paddlespeech.t2s.modules.nets_utils import make_non_pad_mask
File "/home/aistudio/.data/webide/pip/lib/python3.7/site-packages/paddlespeech/t2s/modules/nets_utils.py", line 21, in
from typeguard import check_argument_types
ImportError: cannot import name 'check_argument_types' from 'typeguard' (/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/typeguard/init.py)

I believe this is due to a breaking update of typeguard: it removed the check_argument_types API.

BACKWARD INCOMPATIBLE Removed the check_argument_types() and check_return_type() functions (use @typechecked instead)
https://typeguard.readthedocs.io/en/stable/versionhistory.html

Thank you for your answer, which successfully solved my problem

I believe this is due to a breaking update of typeguard: it removed the check_argument_types API.

BACKWARD INCOMPATIBLE Removed the check_argument_types() and check_return_type() functions (use @TypeChecked instead)
https://typeguard.readthedocs.io/en/stable/versionhistory.html

downgrade to typeguard==2.13.3 works for me.

I believe this is due to a breaking update of typeguard: it removed the check_argument_types API.

BACKWARD INCOMPATIBLE Removed the check_argument_types() and check_return_type() functions (use @TypeChecked instead)
https://typeguard.readthedocs.io/en/stable/versionhistory.html

downgrade to typeguard==2.13.3 works for me.

OK for me

It's @typechecked, not @TypeChecked.

这个错误是因为typeguard 库的版本问题。
typeguard 3.0版本中check_argument_types 已被移除，更改 typeguard==2.13.3 即可


upgrade:Upgrade to druid 1.1.23

Ⅰ. Describe what this PR did
Upgrade to druid 1.1.23 #2901
Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

Codecov Report

Merging #2969 into develop will decrease coverage by 0.32%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             develop    #2969      +/-   ##
=============================================
- Coverage      50.22%   49.90%   -0.33%
+ Complexity      3054     3051       -3
=============================================
  Files            599      599
  Lines          19460    19506      +46
  Branches        2392     2368      -24
=============================================
- Hits            9774     9734      -40
- Misses          8702     8760      +58
- Partials         984     1012      +28

Impacted Files
Coverage Δ
Complexity Δ

...obuf/convertor/BranchRegisterRequestConvertor.java
90.47% <0.00%> (-9.53%)
3.00% <0.00%> (ø%)

...otobuf/convertor/BranchReportRequestConvertor.java
90.90% <0.00%> (-9.10%)
3.00% <0.00%> (ø%)

...obuf/convertor/BranchRollbackRequestConvertor.java
92.00% <0.00%> (-8.00%)
3.00% <0.00%> (ø%)

...otobuf/convertor/GlobalBeginResponseConvertor.java
92.59% <0.00%> (-7.41%)
3.00% <0.00%> (ø%)

...otobuf/convertor/GlobalCommitRequestConvertor.java
93.75% <0.00%> (-6.25%)
3.00% <0.00%> (ø%)

...otobuf/convertor/GlobalStatusRequestConvertor.java
93.75% <0.00%> (-6.25%)
3.00% <0.00%> (ø%)

...obuf/convertor/GlobalRollbackRequestConvertor.java
93.75% <0.00%> (-6.25%)
3.00% <0.00%> (ø%)

...protobuf/convertor/RegisterTMRequestConvertor.java
94.73% <0.00%> (-5.27%)
3.00% <0.00%> (ø%)

...protobuf/convertor/RegisterRMRequestConvertor.java
90.47% <0.00%> (-4.77%)
3.00% <0.00%> (ø%)

...tobuf/convertor/BranchReportResponseConvertor.java
95.45% <0.00%> (-4.55%)
3.00% <0.00%> (ø%)

... and 28 more

be reviewed before the release of 1.4


Provide a means to block/ignore the crawler from crawling any illegal/adult pages

This is more of a feature request. Can you please look at some means to block the crawler from pulling in any illegal or adult content if any of the target URLs contain/links to them? Do you recommend using the LLM extraction strategy and go with a prompt for it?
In my application, I collect a URL/landing page from the users and use LLM to write up a marketing brief, which will further be used for grounding the model. So just in case someone maliciously tries to enter a hazardous URL I want to catch it at the crawling stage itself.

I added this to our backlog, and already found some good solution for that, so for now I close this issue


Reflect and EqDec utils

I draft this PR because I'm not sure exactly if I'm on the right tracks regarding reflect and EqDec.

I updated the other files to comply with this new definition (and I cleaned my proof of EqDec term a bit). I think it can be merged now.


Fix Picasso crash with invalid image value

Fixes #285

@vollkorn1982 I tested this fix locally but if you would like to confirm it works for you that would be great.

Hi @justin-stephenson, I tested your fix and it works fine for me.


Distributed coordination failure

We are running two orientdb nodes (odb1 and odb2) in distributed mode and observed a confusing failure scenario recently. I know we should be running some odd number of nodes >= 3, but I don't believe that our setup was responsible for this particularly failure.
Here is my attempted recreation of what went wrong. All log messages are from odb logfiles, except for the second one below, which is from /var/syslog.
2015-11-29 01:58:45:451 - odb1 reports no response from odb2 in 120s
2015-11-29 01:59:29 prod-orientdb1 kernel: [987840.192067] INFO: task jbd2/xvda1-8:164 blocked for more than 120 seconds.
2015-11-29 01:58:30:832 - odb2 reports no response from odb1 in 120s

odb2 spits out a bunch of hazelcast errors, seemingly related to attempting to coordinate with odb1

2015-11-29 02:01:47:475 - odb2 attempts to remove odb1 from cluster, though still gets quorum errors; continues handling requests (I believe)
2015-11-29 02:01:47:501 - odb2 updates/logs new distributed config, with itself as the only node

odb2 continues to log errors related to coordination with odb1, even though AFAICT odb1 is no longer part of its cluster, e.g. 2015-11-29 02:02:18:607 WARNI [prod-orientdb2] timeout (30001ms) on waiting for synchronous responses from nodes=[prod-orientdb2, prod-orientdb1] responsesSoFar=[prod-orientdb2] request=id=1167444 from=prod-orientdb2 task=tx [OHazelcastDistributedDatabase]

nothing happens on odb2 until 2015-11-29 02:21:26:866, when odb2 establishes a socket with odb1, then:

2015-11-29 02:24:06:359 - odb1 removes odb2 after no heartbeat for 300s
2015-11-29 02:24:06:359 - odb1 removes odb2 from cluster
2015-11-29 02:24:06:364 - odb1 gets conn attempt from odb2, which fails due to timeout
2015-11-29 02:24:07:837 - odb1 has removed odb2 from cluster
2015-11-29 02:24:14:305 - odb1 appears to be attempting to merge its cluster with odb2 cluster
2015-11-29 02:24:14:319 - odb2 re-establishes connection with odb1, adds to cluster
2015-11-29 02:24:14:319 - odb1 re-establishes connection with odb2, adds it to cluster
2015-11-29 02:24:20:317 - odb2 now logs that odb1 is part of its cluster, though it seems to have thought this all along. the new server configuration is never updated and logged, however, as it is when a node successfully joins the cluster.
2015-11-29 02:24:22:320 - odb1 believes it is merged to odb2

from here on, odb1 logs lots of errors about failure to reach quorum, detecting 1 node in timeout/conflict, and lots of these: Op: com.hazelcast.queue.impl.PollOperation@2a518de2, com.hazelcast.spi.exception.WrongTargetException: WrongTarget! this:Address[<IP_ADDRESS>]:2434, target:Address[<IP_ADDRESS>]:2434, partitionId: 226, replicaIndex: 0, operation: com.hazelcast.spi.impl.WaitNotifyServiceImpl$WaitingOp, service: hz:impl:queueService [WaitNotifyServiceImpl$WaitingOp]
odb2, meanwhile, seems to be handling requests just fine, AFAICT. it has a readquorum and writequorum of 1, so any requests *should* succeed.

2015-11-29 02:25        - this is when we start seeing lots of odb-related application errors

odb1 is manually restarted

2015-11-29 10:18:43:535 - odb2 loses connection with odb1, removes it from cluster
2015-11-29 10:18:51:191 - odb1 comes back up after restart by Jai
2015-11-29 10:18:53:414 - odb2 re-establishes connection with odb1, adds to cluster
2015-11-29 10:20:15:463 - odb1 advertises itself as back online

Happy to provide more details if they would be useful. Thanks.

Could you please provide logs?

Sure, here are truncated logs for our two nodes for the relevant time frame. I've truncated repetitive parts, as they would otherwise add up to about 20 megs.
node 1: https://gist.github.com/lukeasrodgers/6205f83e468e4035054f (this is the one we eventually restarted)
node 2: https://gist.github.com/lukeasrodgers/142701f10e0e3b4d016b
thanks

By reading the logs you have problems with timeout: hazelcast can't communicate to other nodes. What kind of network do you have?

Thanks for looking into this @lvca.
They are ec2 nodes on AWS, in general network issues are rare and transient. In any case, shouldn't hazelcast be able to recover from such issues?

Yes, it should be able. Are you using v2.1.6? We updated HZ in that release. Also, are you using AWS auto discovering plugin?

Yes, we are using AWS auto-discover plugin. We just recently upgraded to 2.1.5. If possible, could you point me to the relevant changes in the HZ upgrade? We will look into upgrading shortly. Thanks.

@lukeasrodgers it's been a while. Does the upgrade solved the problem?

We have not upgraded yet, and are back to running in standalone mode, which seems more stable. We are evaluating the feasibility of replacing orientdb with neo4j, and so I think we aren't putting any more efforts into upgrading orientdb unless it looks like switching to neo4j won't be possible for us.

Closing this issue because there is no way to reproduce it. All other users on AWS don't report such problem. In case anybody is experiencing the same problem, please reopen/comment it.


Kudu doesn't honor DNX version

In /common/Constants.cs #54, Kudu currently sets DnxDefaultVersion to "1.0.0-beta5" and uses this setting when deploying my ASP.NET 5 Beta 6 app:

Using the following command to generate deployment script: 'azure site deploymentscript -y --no-dot-deployment -r "D:\home\site\repository" -o "D:\home\site\deployments\tools" --aspNet5 "D:\home\site\repository\src\buggr.web\project.json" --aspNet5Version "1.0.0-beta5" --aspNet5Runtime "CLR" --aspNet5Architecture "x86"'.
Project file path: .\src\buggr.web\project.json
Generating deployment script for ASP.NET 5 Application

Questions:

Should this not be updated to Beta6 now that Beta6 is released (and kept in-sync with new versions as released)?
Could the ASP.NET 5 SDK deployment helper gather the required version of DNX from the deployed app's global.Jason file:

{
  "sdk": {
    "version": "1.0.0-beta6"
  }
}

Or am I missing something? ;)

That constant is only used if we fail to determine the version from global.json. That being said, we should update the constant to beta6.
Do you have a global.json that kudu isn't using?

FYI - I just updated the constant in b31e6aee952cf983c813ebb6e131adfb0e886c01

@ahmelsayed Yes, I do have a global.json, the contents of which are the blob of JSON above.

I can't reproduce this. Here is a test repo that works as expected https://github.com/ahmelsayed-test/HelloWeb as you can see global.json it's the same as yours and kudu installs beta6 correctly.
Do you mind sharing a repo that reproduces the issue you're seeing?

@bitcrazed Please see https://github.com/projectkudu/kudu/wiki/Using-a-git-repo-to-report-an-issue

@davidebbo Completely agree - will share my repro as a GitHub repo. Working on it as I type.

@davidebbo @ahmelsayed Can I run the step that generates the Azure deployment script locally, without needing to deploy Azure via GitHub?

You don't have to deploy to azure via github to repro this. You can just git push from your local repo
git push https://<YourSiteName>.scm.azurewebsites.net master

Unfortunately, you can repro this locally because the tool that generates the script asks for the dnx version in the parameters. Normally you would run it like this
npm install -g kuduscript
kuduscript -y --no-dot-deployment -o <outputFolder> --aspNet5 <path to project.json> --aspNet5Version <Dnx version> --aspNet5Runtime <CLR or CoreCLR> --aspNet5Architecture <x86/x64>

So as you can see it asks for the <Dnx version> on the command line. Kudu is the one that is supposed to supply that to the tool that generates the script.

Okay, can't share offending repo (originally created using Yeoman yo aspnet) so I created a new project to get a baseline:
Created a new project via VS 2015 File | New Project | Web - ASP.NET Web Application -> ASP.NET 5 app. Modified nothing within the project and built it. Ran locally to test app was well. A-OK.
Created a local Git repo & remote GitHub repo. Pushed project to repo: https://github.com/bitcrazed/HelloWebApp
Created Azure Web App (using new Azure portal) & configured CI via GitHub.
Waited until deployment complete, then browsed to site (http://bitcrazed-HelloWebApp.azurewebsites.net) and all looks good.
Checked deployment script - was as expected (noting expected version of ASP.NET 5):

Using the following command to generate deployment script: 'azure site deploymentscript -y --no-dot-deployment -r "D:\home\site\repository" -o "D:\home\site\deployments\tools" --aspNet5 "D:\home\site\repository\src\HelloWebApp\project.json" --aspNet5Version "1.0.0-beta6" --aspNet5Runtime "CLR" --aspNet5Architecture "x86"'.
Project file path: .\src\HelloWebApp\project.json
Generating deployment script for ASP.NET 5 Application

Then I removed global.json from filesystem and solution, and pushed the updated project to GitHub.
Once deployment complete, confirmed that the current Kudu engine is assuming Beta5 (see thread above) which @ahmelsayed has already fixed (I guess awaiting deployment).

Using the following command to generate deployment script: 'azure site deploymentscript -y --no-dot-deployment -r "D:\home\site\repository" -o "D:\home\site\deployments\tools" --aspNet5 "D:\home\site\repository\src\HelloWebApp\project.json" --aspNet5Version "1.0.0-beta5" --aspNet5Runtime "CLR" --aspNet5Architecture "x86"'.
Project file path: .\src\HelloWebApp\project.json
Generating deployment script for ASP.NET 5 Application

So, what's different between my offending solution and this one?
It appears that the offending solution has a global.json file in the web app project root (I.e. .../src/project_name/global.json), rather than in the solution root (I.e. .../). Does Kudu assume that global.json is in the solution root or does it lookup the file's location as specified in the solution file?

@ahmelsayed Thanks for the info re. deployment. In my steps above, I created a GitHub repo so you guys could see what I'd done and so that I could repro the environment I see in my offending app.
Re. Script gen':
Was wondering if I could kick Kudu locally to generate the output script - this way I could repro & debug locally. No problem if not - the environment works for now ;)

I see. Yes we assume that global.json will be in the solution root, at least that's how I thought project layout is supposed to be. I'll try yoman and see what layout it generates.
About debugging locally, you can run kudu locally and then you'll be able to debug this part locally. you can take a look at our Getting Started wiki for that

Thanks - will take a look at running Kudu locally - it's a :top: tool :grinning:
When creating a project via Yeoman, no global.json is created, nor any VS solution / project files since ASP.NET 5 is aiming to support non-MS platforms. Here's the list of files created using the current template:
create test\gulpfile.js
create test\.bowerrc
create test\.gitignore
create test\bower.json
create test\config.json
create test\hosting.ini
create test\package.json
create test\project.json
create test\README.md
create test\Startup.cs
create test\Controllers\HomeController.cs
create test\Views\_ViewImports.cshtml
create test\Views\_ViewStart.cshtml
create test\Views\Home\About.cshtml
create test\Views\Home\Contact.cshtml
create test\Views\Home\Index.cshtml
create test\Views\Shared\_Layout.cshtml
create test\Views\Shared\Error.cshtml
create test\wwwroot\css\site.css
create test\wwwroot\favicon.ico
create test\wwwroot\images\ASP-NET-Banners-01.png
create test\wwwroot\images\ASP-NET-Banners-02.png
create test\wwwroot\images\Banner-01-Azure.png
create test\wwwroot\images\Banner-02-VS.png
create test\wwwroot\js\site.js

Looks like my offending project had global.json added to the wrong location - will move for now, but it might be worth adjusting Kudu to look for the global.json location as specified in the solution file.

As far as I know global.json should be in the root of your dnx solution as in
\---DnxApp
    |   global.json
    |
    \---src
        +---app.library1
        |       Class.cs
        |       project.json
        |
        +---app.library2
        |       AnotherClass.cs
        |       project.json
        |
        \---app.web
                Controller.cs
                project.json

because it describes the runtime that all the sub projects will use.

Agreed. Just moved my global.json to what I'd called the solution root - the root of the app as you illustrated above. Still, may be worth looking-up the file's location from the solution in case someone wants to move it for some reason.

Can also (happily) confirm since moving my global.json, indicating my site runs atop DNX 1.0.0-Beta6, the site now runs as expected (Was seeing a 5xx error previously).
Thanks for your help on this.

Can also (happily) confirm that creating a global.json with the required runtime in the project root solves the problem. But shouldn't the yeoman generator add a global.json? I'll see if this is addressed. Big thanks for finding the problem though. I was diffing 2 projects created using Visual Studio and yeoman to find the differences but as the layout and contents are different it was a bit daunting... This was much more efficient.

Had the same problem trying to deploy to azure with beta7, the problem was not having the solution as the root directory of the repo, having a structure like this:
\---github - root
     |
     |
     \---Dnx.App
          |  global.json
          |
          +---app.library1
          |       Class.cs
          |       project.json
          |
          \---app.web
                  Controller.cs
                  project.json

Because the global.json file isn't in the root directory of the repo, kudu doesn't find it and falls back to beta6, my solution was to move the global.json file to the root of the repo.
As @bitcrazed said, it may be good to look for the global.json file instead of expecting a standard position.

Thx @roysanchez. It seems, just like you suggest, that the problem is not in the generator-aspnet itself but rather in how kudu interprets the project structure in azure. As long as you introduce the added folder structure you're fine. Keep global.json at the top and project.json in the subfolder. If the two files are at the same level they will confuse kudu's build/make process. I have no idea whether this is intended behaviour or a bug but you made my day. And another shout out to @bitcrazed who suggests that one shouldn't expect things to be in one place. Smells like hard-coded paths somewhere which I'm of course guilty of a long time ago.
Anyways, I've successfully deployed from VS Code via GitHub to Azure which was my aim. Thanks for all helping with this issue. What a change from 25 years ago!

@ahmelsayed Many thanks for taking our feedback on board and committing a fix for this issue :grinning:

The publish issue was fixed in beta8

@davidfowl is beta8 out?

Nope, I mean to say, it will be fixed in beta8 :smile: (it's already fixed in ci builds)


[python] Fix Cluster objects datatypes on Python 3.11

It seems that Python 3.11 requires the custom data type Nullable to implement the __hash__() function, otherwise initializing dataclass annotated objects fail with the following stack trace:
  File "/home/sag/projects/project-chip/connectedhomeip-v1.1/.environment/pigweed-venv/lib/python3.11/site-packages/chip/clusters/Objects.py", line 385, in Commands
    @dataclass
     ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1223, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1213, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 958, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 815, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'chip.clusters.Types.Nullable'> for field capacity is not allowed: use default_factory

Since all instances of Nullable are the same "value", we can return a static 0 as hash value.

@bzbarsky-apple do you happen to know if the test errors are related to this change or are those general errors we are seeing?
It feels that this change could be at fault, but before diving into it, I'd rather ask.
[2023-05-23 12:29:25.192742][TEST][STDERR]  File "/__w/connectedhomeip/connectedhomeip/.environment/pigweed-venv/lib/python3.9/site-packages/chip/clusters/ClusterObjects.py", line 56, in _PutSingleElementToTLV
[2023-05-23 12:29:25.192746][TEST][STDERR]    if issubclass(elementType, ClusterObject):
[2023-05-23 12:29:25.192750][TEST][STDERR]TypeError: issubclass() arg 1 must be a class
[2023-05-23 12:29:25.193115][TEST][STDERR]Testfail: Exception occurred when running tests.
[2023-05-23 12:29:25.193172][TEST][STDERR]CRITICAL:PythonMatterControllerTEST:Testfail: Exception occurred when running tests.

[2023-05-23 12:29:25.192750][TEST][STDERR]TypeError: issubclass() arg 1 must be a class

@agners I have never seen that failure before, so there is a good chance it's due to this change...

It seems that adding the annotation @dataclass causes the issues. I am not quite clear on why that is, it seems that the annotation changes the behavior of that class slightly, which triggers some code paths to use a dict type instead of the  Nullable type. This then triggers TypeError: issubclass() arg 1 must be a class, since the dict type is passed as argument one which is not a class...


Alpine-powered docker container

It would be awesome if we have an official Alpine Linux container. It would be way smaller than the one we've got today.
Could you think of any issues that would prevent KeyDB from running in Alpine?
Note: I would appreciate if the builds in Docker hub are automated, giving us somewhat guarantee that this is the exact Dockerfile that was used for creating the image.

I haven't tested building with MUSL and I don't have the bandwidth to actively test Alpine Linux.  That said I will fix any issues that may be encountered if someone wants to take on support and testing for that distro.  Using Musl instead of glibc is likely to be the biggest issue.
Having docker do the build is a possibility but since we have multiple release targets we wanted to ensure they all had the same build pipeline for simplicity.  We can revisit that in the future.


NXP FRDM-MCXA156: no UART RX

LPUART0, used for the debug console and shell, does not receive characters.  It does transmit, and will print to the terminal.
To replicate, the sample samples/drivers/uart/echo_bot shows the issue.  It prints to the terminal, but will not echo back any characters.  Perhaps there is an issue with the RX pin.
I first noticed this trying to run the flash_shell sample.  And then found that I did not see UART RX work in any sample I tested.  I tested with commit https://github.com/zephyrproject-rtos/zephyr/commit/74bb5b334db1749d9ff30218d89e922f1409849f when this board was first introduced, and it has the same issue.
@NeilChen93 , can you review this issue?

Hi @DerekSnell , get it, I will check and fix this issue.

@DerekSnell  @NeilChen93 I am bringing up an MCXA156 (BGA version)  now (but saw this on the FRDM board as well.
The issue is is that the input buffer enable bit is not getting set.
#define A15X_MUX(port, pin, mux)		\
	(((((port) - '0') & 0xF) << 28) |	\
	(((pin) & 0x3F) << 22) |		\
	(((mux) & 0xF) << 8))

Bit 12 needs set to enable the input buffer.
I did a quick overlay like this:
`&pinctrl {
pinmux_lpuart0: pinmux_lpuart0 {
group0 {
pinmux = <(LPUART0_RXD_P0_2 | (1<<12))>,
<LPUART0_TXD_P0_3>;
drive-strength = "low";
slew-rate = "fast";
};
};
};`
for a quick fix.

Hi @ehughes ,
@NeilChen93 already has a PR submitted at https://github.com/zephyrproject-rtos/zephyr/pull/81752.
Best regards


Better error when trying to .json a Resource that's not JSON.

It would be nice if there were a better error when trying to .json a Resource that's not JSON.
Currently in two implementations (this polyfill and Chrome Service Workers) trying to fetch and .json() a non JSON resource throws a not particularly helpful error:-
fetch("//some.thing/not-a-json-resource")
  .then(function(res) {
    return res.json();
  })
  .catch(function(err) {
    console.log(err); // => Unexpected token c
  });

It would make for more less unpleasant debugging / error aggregation if it threw an error along the lines of “Failed to parse {{url}} as JSON”… or something like that…
I know errors aren't spec'd as such so it's not something that should/could be fixed in the spec — but I will raise as an issue in Chrome as well so that GitHub & Chrome's fetches can stay aligned but I think it's something that should be improved and I'm sure you have an idea of what it might look like…

Have any specific ideas for how we could improve this? Or should we just wait it out until Chrome makes the first move?

I think this error is pretty good. "SyntaxError" and "unexpected token" are good clues that's it's invalid JSON. Then you can inspect the response and see it's HTML or other non-JSON data.

for what it's worth i'm playing with ideas here:-
https://github.com/matthew-andrews/fetchres#usage-node


CSPR Burning feature

Node enhancement proposal to enable native token burning feature.  The proposal objective is to make the existing internal CSPR token burn mechanism accessible to Smart Contract developers to let our community limit the Total Supply of CSPR tokens. This will allow us to introduce the deflationary mechanism to Casper Network.
Please find the detailed proposal in the CEP request here

The linked PRs are;
https://github.com/casper-network/casper-node/pull/4573
https://github.com/casper-network/casper-node/pull/4640
And the related CEP is https://github.com/casper-network/ceps/pull/92

Completed, therefore closing.


[DRAFT] Update tool to ignore minipools and miniblock_candidates for archiver

Used to migrate our omega archiver nodes, this PR doesn't need to be committed.


deleting nupic.cerebro

in favor of nupic.cerebro2 ? @chetan51 @rhyolight

:+1: I don't think anyone has used it in over a year. @yakticus didn't you attempt to use it last year?

Although I'm not in favor of complete deletion. I'd rather mark as deprecated so the source is still around for future reference.

I'm in favor of deprecation.

Yes - I also favor deprecation over deletion.


ESP32S3 RMT rmt_sync_reset introduces 100µs delay (IDFGH-9034)

Is your feature request related to a problem?
I'm not really sure if that's a feature request, a bug report or just merely a question. I've recently started to use the RMT peripheral on an ESP32S3 on two synchronized outputs. From my observations I suspect that rmt_sync_reset must be called between every transmission to keep both channels in sync otherwise they quickly diverge.
I've tried to queue multiple requests and call rmt_sync_reset inside the transmission complete callback but that doesn't seem to work.
Instead I now only start a single transmission and notify a high priority task to restart transmission as soon as possible. This works fine, but it introduces a 100µs delay before the transmission is actually restart.
Here's a signal I've captured which shows this behavior:
yellow = channel 0
cyan = channel 1
magenta = transmission complete callback
blue = notified high priority task

Describe the solution you'd like.
Is there a technical reason why the transmission is delayed those 100µs?
Can I currently somehow lower that time? Apparently the task and data would be ready way sooner than that.
Rephrasing that question one could also ask:
"Is there a way to do continuous transmissions with multiple synchronized channels?"
Describe alternatives you've considered.
No response
Additional context.
No response

About the rmt_sync_reset function:

It is not allowed to run in the ISR context, the critical section used inside that function is not ISR version.
calling this function in any channel's tx done callback sounds not reasonable to me. A sync manager usually manages multiple channels, you can't reset it in channelA's callback while channelB might still not finished.

About the 100us delay in between each transaction:
The driver fetches the transaction descriptor from a queue, reconfigures the RMT channel, and start the RMT encoder.
All of this will consume negligible time (plus interrupt latency ~2us). If the RMT encoder is time consuming, this will cost more time I think.

Is there a way to do continuous transmissions with multiple synchronized channels?

From my previous test, the sync can be lost after one transaction is finished. We have to resync them manually, that's why the rmt_sync_reset is provided.

Thank you for your answer. Do you have any suggestions how to improve those times?
Any chance we might get an asynchronous way to pre-encode symbols in the future? E.g. by splitting rmt_transmit into rmt_encode and rmt_start.

yes, you can pre-encode the symbols asynchronously, but can't write the symbols to the RMTMEM (a dedicated hardware memory) until the last transaction is done.  To copy the pre-encoded symbols into the RMTMEM, you need to use the RMT copy encoder. This encoder is simple and fast, just a wrapper of "memcpy".

I can't figure out how to reliably re-trigger a transmission at a certain point. The delay itself isn't really an issue, I have up to ~480µs to prepare the next command in advance, but when it's time to transmit it it must be sent immediately and without any delay. The jitter introduced through the driver layer is a real problem for my use-case.
Any suggestions how I can work around the issue without hardware changes? (My current understanding is that a single output wouldn't suffer from the issue as I wouldn't have to call rmt_sync_reset in between.)


feat(demo): add react-instantsearch

Implementation of the instant search demo using React InstantSearch. You can see a live version here: http://elite-cord.surge.sh/
Some comments:

I replaced the button that was clearing the facets and the query by our  widget that only clear the facets. We probably need to add another widget on the RIS side.
I used the rc-slider component for the range. There's no built-in range with RIS. You need to use a connector for that.
The SFFV searchbox placeholder can't be changed. I added a bug on RIS, this will be fixed.

Nice pull request @mthuret
Are you thinking to give to the implementation the same look and feel as what has been done on the InstantSearch.js demo?

@bobylito What do we do in the InstantSearch.js demo when there is a active facet that matches the search query? Do we display it part of the list or not?

@seafoox here are my answers ;)

What do we do in the InstantSearch.js demo when there is a active facet that matches the search query? Do we display it part of the list or not?

Yes.

When selecting a facet value, does that one should stick to the top of the list when there is not search query?

I'm not super sure about to understand based on the GIF. I only see a weird thing with insignia. I checked with @mthuret and it looks like a bug :)

@mthuret the magnifying glass icon is missing. 🎶 CSS 🎶

Thanks @bobylito
Regarding the look'n'feel, do we want

to add magnifying glass icon
to add cross button when queryLength > 0, in order to delete the current query
to replace “No matching brand” with “No results”
to replace the current highlighting style with something more font-weight: bold
What do you think?

Otherwise, there seems to be a margin bug when there is no results: margin bottom is then too small.

to replace “No matching brand” with “No results”

No matching brand is consistent with specifying a placeholder that is precise, no?
Here I would also suggest to use a font size consistent with the size used with items (13px) , and add some padding (5px 0 20px 0).

@LucasC do you think it's better to display No matching brand or No result?

No matching brand 👍 But as @bobylito said, with a more consistent font size.

I updated the PR with your feedbacks and you can reuse the surge link to see them.
There still a bug with the refinement list SFFV sorting values but as soon as we release another RIS version with the fix, it will be ok.
@seafoox: can you be more precise on what you mean by "Are you thinking to give to the implementation the same look and feel as what has been done on the InstantSearch.js demo?" ?  Was it concerning the SFFV or do you have other things in mind?

Looks great @mthuret.

Are you thinking to give to the implementation the same look and feel as what has been done on the InstantSearch.js demo?

Yes, was talking about the SFFV widget look'n'feel. But I think it looks the same now that you've made all those updates.
There might be one thing missing: the X to clear the current search query. Is that already under your radar?

@seafoox: the X only appear when the query is not empty. Do you have trouble seeing it?

that's perfect @mthuret

https://github.com/algolia/examples/pull/34#issuecomment-273488862 is fixed in https://github.com/algolia/instantsearch.js/pull/1970 (v3.2.0) @seafoox


WIP: OIDC provider

This adds support for a Github Actions compatible OIDC provider.
For more info, see:
https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/about-security-hardening-with-openid-connect
It depends on a few changes in act and act_runner:
https://gitea.com/gitea/act_runner/pulls/272
https://gitea.com/gitea/act/pulls/73

Still work-in-progress. I have a few more things I want to test before it's ready.

I think this is ready for review, but obviously it's useless without the act and act_runner changes.

@sorenisanerd Thank you for your work, I have to say you did a great job, good idea, good code.
However, this featrue has been blocked some trivial things.

Where to keep the code for parsing permissions? Gitea, the forked act, the upstream act, or using actionlint indead?
The permissions field is only only for OIDC, but the permissions granted to the GITHUB_TOKEN/GITEA_TOKEN (probably more often), personally, I think the latter should be supported first.
Is it possible for contributors to send a PR that updates the permissions field and runs a malicious script?

You know, once permissions are involved, we need to be checked more carefully. Please give us some time to discuss and decide.

Thank you for your kind comments :)
I'm very open to porting this to gitea or actionlinter or wherever. I don't have any strong opinions on it.
I agree it would have made more sense to first use permissions to adjust the capabilities of the GitHub token. I happened to need this oidc-provider functionality, so that's what I focused on. I included methods for merging permissions as well as definitions matching GitHub's defaults so I or someone else could add that later.
Great point about how permissions are affected when permissions apply for pull requests. Is that supposed to be safeguarded by the need to approve workflow runs for new contributors? I honestly don't know what GitHub does.

I've pushed changes to the act and act_runner PR's on gitea.com. The resulting diffs are very modest. The rest of the functionality has been pulled into Gitea here.
Is this approach more agreeable? I'll add some tests once we're happy with the overall approach.

I rebased this and reworked it into a set of more reviewable commits. If you like, I can submit them as individual PR's, but each one depends on the previous ones, which makes it a little awkward to manage as separate PR's.

Nothing new, just rebased and folded https://github.com/go-gitea/gitea/commit/ded82c30abd033695d7a6edae9a1295f792063a0 (the formatting changes based on the linting errors) into the relevant patches.

I took the liberty of renaming my migration to just permissions.go. At this point, I must have had to rename it 5-7 times locally and it's getting kinda annoying. Once this gets closer to being approved, I'll try to pick a number for it.

I'm also working on a more complete permissions implementation that should probably be merged before this one, so I'm setting this one back to WIP. I expect the permissions patch to be ready some time this week.

Hello, are there any updates on this? is there something a contributor can help out with? Would love to see this move along

I am also very interested om seeing this moving along. Any updates on this?


Performance problem: -Name slows down Get-ChildItem -Recurse

-Name is a convenient switch to only output item names  (without -Recurse) or relative paths (with -Recurse).
Given that only string instances rather than System.IO.File/DirectoryInfo instances are output, one would expect -Name to speed things up (or at least not to slow things down); however, with -Recurse present, -Name slows down the command noticeably.
Steps to reproduce (on Windows)
# Warm up the cache
$null = Get-ChildItem -Recurse 'C:\Program Files'

# Without -Name
(Measure-Command { Get-ChildItem -Recurse 'C:\Program Files' }).TotalSeconds

# With -Name
(Measure-Command { Get-ChildItem -Name -Recurse 'C:\Program Files' }).TotalSeconds

Expected behavior
The command with -Name should be faster (or at least not slower).
Actual behavior
Sample timings:
1.544098   # without -Name
4.5759553  # with -Name

That is, adding -Name made the command about 3 times slower.
Environment data
PowerShell Core 6.2.0-preview.4

I expect #8995 will address this.

It looks like this issue is more pronounced in Powershell 7.0.
Running the following command from a nodeJS project:
Get-ChildItem -Recurse -Name

Returns 12886 files.
PowerShell 6.2.3
Without -Name
IsRunning Elapsed          ElapsedMilliseconds ElapsedTicks
--------- -------          ------------------- ------------
    False 00:00:00.7503959<PHONE_NUMBER>41

With -Name
IsRunning Elapsed          ElapsedMilliseconds ElapsedTicks
--------- -------          ------------------- ------------
    False 00:00:01.3749722                1374<PHONE_NUMBER>

PowerShell 7.0.0-rc.1
Without -Name
IsRunning Elapsed          ElapsedMilliseconds ElapsedTicks
--------- -------          ------------------- ------------
    False 00:00:00.6832990<PHONE_NUMBER>55

With -Name
IsRunning Elapsed          ElapsedMilliseconds ElapsedTicks
--------- -------          ------------------- ------------
    False 00:00:09.0078861                9007<PHONE_NUMBER>

I see similar results for both macOS and Windows.

I guess this perf regressions comes from https://github.com/PowerShell/PowerShell/pull/9250
If previously I thought Kaspersky AV slow down file operations, now I tested with Microsoft AV.

@iSazonov that method should be called in all cases, and not just when -Name is used

Seems like we should expose an overload to GetChildNames() to accept recursion.  Then the FileSystemProvider can implement this and simply use EnumerateFileSystemEntries() to quickly return just the names.   In SessionStateContainer.cs, it would try the overload first, but not all providers implement it and fallback to current behavior.  Since this is not a mainline issue, not going to implement this for now.

that method should be called in all cases, and not just when -Name is used.

@SteveL-MSFT For dir -Recurse it is called 3 times for root directory, for dir -Recurse -Name - 3 times for every subdirectory and 1 time for every file.

@iSazonov yes, because when -Recurse and -Name is used together, the recursion is done outside of the FileSystemProvider so each call into the provider results in a call to GetCorrectCasedPath().  However, there's actually a lot of other work happening because of the manual recursion than just that API call.  The FileSystemProvider can leverage the EnumerateFileSystemEntries() to just get the names, but the current interface doesn't expose such a call into the provider.

I found -Name makes things slower; @jborean93 looked into it and suggested this possible cause:

The strings are decorated with .PSPath, .PSParentPath, .PS... properties.

Normally a FileInfo is returned from the FileSystem Provider, the Provider Base adds these decorators.

With -Name, a FileInfo is made but only the name string is returned, the Provider Base then resolves the string to a path and another FileInfo is made to add these extended properties.

-Name ends up doing more work.

This code may be involved:

FileSystemProvider.cs writing either a FsInfo or FsInfo.Name to the output
ProviderBase.cs wrapper code


[Feature Request] Allow different models for assistant, user, task_specify_agent, and critic etc in the RolePlaying

Required prerequisites

[X] I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
[ ] Consider asking first in a Discussion.

Motivation
As number of LLMs is increasing, many LLMs are designed for different tasks and even have different price. So, it's useful to let user specify different models for each agent in RolePlaying. Followings are some examples:
Example 1
User may have cost concern for RolePlaying when they use for example GPT4 for all assistant, user and critic. One way to lower cost might be using GPT3.5 for assistant and user but GPT4 for critic. In this way user can have some flexibility to control the cost of RolePlaying but still maintain some quality of RolePlaying.
Example 2
Now there are many LLMs are trained to optimize some specific tasks. In the future, it's possible that user can use a specific LLM for assistant but another LLM for user during RolePlaying collaboration. Maybe in this way the RolePlaying can have better performance.
Solution
Allow different models for assistant, user, task_specify_agent, and critic etc in the RolePlaying
Alternatives
No response
Additional context
No response

Thanks @zechengz. This is a great idea. This can be done by setting model_type in *_agent_kwargs. So let's say if *_agent_kwargs includes model_type, we override the default one. WDYT?

If we want fine grained control over the model backends, we'd better implement the dependency injection pattern: feed every agent with a pre-constructed model backend. See https://stackify.com/dependency-injection/

I checked that current model backend is defined by model_type and model_config_dict.
If we just have these two params for model backend, IMO we can just allow user to put them to *_agent_kwargs. And we construct each agent by getting model_type and model_config_dict from each *_agent_kwargs (if they are specified).
As current model backend is constructed within each Agent, for now IMO it's easier to pass user specified model_type and model_config_dict (from *_agent_kwargs) directly.


Installation not available on VenusOS v3.40?

Hi!
Thank you for the plugin - sounds like just what I need to monitor the temperature of my Pi and the bus bar of the Inverter.
However on an fresh beta installation I cannot select "Install".
VenusOS v3.40~1 (small)
SetupHelper 7.12
Is this just some sanity check or do I miss something?

Worked for me on 3.42 just fine.


Update README document

We should update README doc with how to get started with this repo instead of default template doc

Sorry my bad, looks like we havent committed any code here yet


improve endianness validation strategy

Fixed #9

Replacing runtime exceptions with compile-time #error in C++
Adding JavaScript-side endianness validation pre-WASM load
Removing redundant runtime checks from C++ implementation
Updating TypeScript initialization with early validation

Looks like you forgot to attach the actual cpp code files