https://www.google.com/about/careers/applications/ 403's from IR

Hi @mosajjal,
I was helping someone from Iran access a google domain website and they reported to me that they are getting 403, even after setting their DNS to the sni proxy I have in Vultr.
My experiment confirms that: I setup a fresh instance of sniproxy in Vultr. Now from my machine outside Iran, websties like udemy.com do load after setting DNS to my sniproxy, but when I experiment with a datacenter in Iran, I get the 403 back:

Setting DNS to my vultr instance IP <IP_ADDRESS> from Iran:

cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# <IP_ADDRESS> is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
# nameserver <IP_ADDRESS>
# nameserver <IP_ADDRESS>
# nameserver <IP_ADDRESS>
# nameserver <IP_ADDRESS>
nameserver <IP_ADDRESS>

Digging to make sure that my DNS has actually changed:

dig udemy.com

; <<>> DiG 9.18.1-1ubuntu1.3-Ubuntu <<>> udemy.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25881
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;udemy.com.			IN	A

;; ANSWER SECTION:
udemy.com.		3600	IN	A	<IP_ADDRESS>

;; Query time: 87 msec
;; SERVER: <IP_ADDRESS>#53(<IP_ADDRESS>) (UDP)
;; WHEN: Tue Apr 04 00:11:38 UTC 2023
;; MSG SIZE  rcvd: 52

Access to udemy.com is still blocked from the machine in IR:

wget udemy.com
--2023-04-04 00:14:43--  http://udemy.com/
Resolving udemy.com (udemy.com)... <IP_ADDRESS>, <IP_ADDRESS>
Connecting to udemy.com (udemy.com)|<IP_ADDRESS>|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-04-04 00:14:43 ERROR 403: Forbidden.

Access to udemy.com is also blocked from the Vultr machine itself!

root@sniproxy:~# wget https://www.udemy.com/
--2023-04-04 00:22:04--  https://www.udemy.com/
Resolving www.udemy.com (www.udemy.com)... 2606:4700::6812:a05a, 2606:4700::6810:e25b, <IP_ADDRESS>, ...
Connecting to www.udemy.com (www.udemy.com)|2606:4700::6812:a05a|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-04-04 00:22:04 ERROR 403: Forbidden.

The original website that my tester was trying to access was a google domain, but I figured if we solve it for udemy.com it will be solved for other websites too.
Is this expected behavior?

Another experiment with the URL that my tester tried:

google.com/about/careers/applications/signin from Vultr succeeds:

wget google.com/about/careers/applications/signin
--2023-04-04 00:24:19--  http://google.com/about/careers/applications/signin
Resolving google.com (google.com)... 2a00:1450:400e:810::200e, <IP_ADDRESS>
Connecting to google.com (google.com)|2a00:1450:400e:810::200e|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.google.com/about/careers/applications/signin [following]
--2023-04-04 00:24:19--  https://www.google.com/about/careers/applications/signin
Resolving www.google.com (www.google.com)... 2a00:1450:400e:801::2004, <IP_ADDRESS>
Connecting to www.google.com (www.google.com)|2a00:1450:400e:801::2004|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://www.google.com/about/careers/applications/apply [following]
--2023-04-04 00:24:19--  https://www.google.com/about/careers/applications/apply
Reusing existing connection to [www.google.com]:443.
HTTP request sent, awaiting response... 302 Found
Location: https://accounts.google.com/ServiceLogin?passive=1209600&continue=https://www.google.com/about/careers/applications/apply&followup=https://www.google.com/about/careers/applications/apply [following]
--2023-04-04 00:24:20--  https://accounts.google.com/ServiceLogin?passive=1209600&continue=https://www.google.com/about/careers/applications/apply&followup=https://www.google.com/about/careers/applications/apply
Resolving accounts.google.com (accounts.google.com)... 2a00:1450:400e:800::200d, <IP_ADDRESS>
Connecting to accounts.google.com (accounts.google.com)|2a00:1450:400e:800::200d|:443... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://accounts.google.com/InteractiveLogin?continue=https://www.google.com/about/careers/applications/apply&followup=https://www.google.com/about/careers/applications/apply&passive=1209600&ifkv=AQMjQ7TvHtXxi5L2S5ebl-KSIISHYbPI9JyCKvPZ0Hf5TGD2n_-k1il7-hgvzkzT6yaXo313FUY0 [following]
--2023-04-04 00:24:20--  https://accounts.google.com/InteractiveLogin?continue=https://www.google.com/about/careers/applications/apply&followup=https://www.google.com/about/careers/applications/apply&passive=1209600&ifkv=AQMjQ7TvHtXxi5L2S5ebl-KSIISHYbPI9JyCKvPZ0Hf5TGD2n_-k1il7-hgvzkzT6yaXo313FUY0
Reusing existing connection to [accounts.google.com]:443.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: https://accounts.google.com/v3/signin/identifier?dsh=S1393193365%3A1680567860089479&continue=https%3A%2F%2Fwww.google.com%2Fabout%2Fcareers%2Fapplications%2Fapply&followup=https%3A%2F%2Fwww.google.com%2Fabout%2Fcareers%2Fapplications%2Fapply&ifkv=AQMjQ7Q-PcS1-qUMys5bHSsB4R_mH0XzZumEOGsBJrRGql1ZcsKBZy4PjlN5E50Og300yBJvfi6A&passive=1209600&flowName=WebLiteSignIn&flowEntry=ServiceLogin [following]
--2023-04-04 00:24:20--  https://accounts.google.com/v3/signin/identifier?dsh=S1393193365%3A1680567860089479&continue=https%3A%2F%2Fwww.google.com%2Fabout%2Fcareers%2Fapplications%2Fapply&followup=https%3A%2F%2Fwww.google.com%2Fabout%2Fcareers%2Fapplications%2Fapply&ifkv=AQMjQ7Q-PcS1-qUMys5bHSsB4R_mH0XzZumEOGsBJrRGql1ZcsKBZy4PjlN5E50Og300yBJvfi6A&passive=1209600&flowName=WebLiteSignIn&flowEntry=ServiceLogin
Reusing existing connection to [accounts.google.com]:443.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘signin’

signin                                                      [ <=>                                                                                ] 149.48K  --.-KB/s    in 0.04s

2023-04-04 00:24:20 (3.71 MB/s) - ‘signin’ saved [153068]

google.com/about/careers/applications/signin from Iran 403s:

wget google.com/about/careers/applications/signin
--2023-04-04 00:25:00--  http://google.com/about/careers/applications/signin
Resolving google.com (google.com)... <IP_ADDRESS>, <IP_ADDRESS>
Connecting to google.com (google.com)|<IP_ADDRESS>|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.google.com/about/careers/applications/signin [following]
--2023-04-04 00:25:00--  https://www.google.com/about/careers/applications/signin
Resolving www.google.com (www.google.com)... <IP_ADDRESS>, <IP_ADDRESS>
Connecting to www.google.com (www.google.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-04-04 00:25:00 ERROR 403: Forbidden.

I'll look into it. can you please also share the sniproxy logs (in debug mode).

sniproxy.log I got this with journalctl -u sniproxy.
How do I get more verbose logs?

so there are a few interesting bits here..

IPv6 connectivity might be causing the issue for Google. can you please disable ipv6 temporarily on the client and test again

there's a potential bug related to juhannusjemut.vilkkuv.fi that I'm looking into as well.

if you're keen to test the experimental ipv6 support and you got Go installed, you can install the alpha version of sniproxy using:
go install github.com/mosajjal/sniproxy@240d9153096668a508a5fd7aba753c7fa4e6b223

IPv6 connectivity might be causing the issue for Google. can you please disable ipv6 temporarily on the client and test again

I created another Vultr instance with IP V6 disabled, installed sniproxy and still get the 403 err for udemy.com:

looks like your DNS is not working properly. how are you setting your DNS? use dig or nslookup to make sure your current DNS always returns the correct IP. keep in mind that the default list that I've shipped, doesn't have google.com as a proxied domain so it actually becomes direct. if you want sniproxy to forward all traffic, add --allDomain when running sniproxy

looks like your DNS is not working properly. how are you setting your DNS? use dig or nslookup to make sure your current DNS always returns the correct IP.

I'm setting the DNS by changing /etc/resolv.conf:
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# <IP_ADDRESS> is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
# nameserver <IP_ADDRESS>
# nameserver <IP_ADDRESS>
# nameserver <IP_ADDRESS>
# nameserver <IP_ADDRESS>
# nameserver <IP_ADDRESS>
nameserver <IP_ADDRESS>

Then when I dig from the machine in Tehran:
dig  google.com

; <<>> DiG 9.18.1-1ubuntu1.3-Ubuntu <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44548
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com.			IN	A

;; ANSWER SECTION:
google.com.		418	IN	A	<IP_ADDRESS>

;; Query time: 3 msec
;; SERVER: <IP_ADDRESS>#53(<IP_ADDRESS>) (UDP)
;; WHEN: Thu Apr 06 23:35:04 UTC 2023
;; MSG SIZE  rcvd: 44

The DNS response contains the IP of sniproxy (<IP_ADDRESS>), but I was expecting the A record to also be <IP_ADDRESS>. That's why I am guessing that DNS tampering is happening.
Now, when I dig the actual URL of interest:
dig google.com/about/careers/applications
;; communications error to <IP_ADDRESS>#53: connection refused

dig google.com/about/careers/applications @<IP_ADDRESS>
;; communications error to <IP_ADDRESS>#53: connection refused

Telecom is dropping the DNS request.

keep in mind that the default list that I've shipped, doesn't have google.com as a proxied domain so it actually becomes direct. if you want sniproxy to forward all traffic, add --allDomain when running sniproxy

I expected all domains to be proxied, because when installing via install.sh, I got this prompt:
sniproxy can proxy all HTTPS traffic or only specific domains, if you have a domain list URL, enter it below, otherwise press Enter to proxy all HTTPS traffic and I pressed Enter.
The prompt implies that the default domain list includes every domain.
I did end up adding --alldomains to /etc/systemd/system/sniproxy.service and restarted the sniproxy service.

looks like your DNS is not working properly. how are you setting your DNS? use dig or nslookup to make sure your current DNS always returns the correct IP.

I'm setting the DNS by changing /etc/resolv.conf:
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# <IP_ADDRESS> is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.
# nameserver <IP_ADDRESS>
# nameserver <IP_ADDRESS>
# nameserver <IP_ADDRESS>
# nameserver <IP_ADDRESS>
# nameserver <IP_ADDRESS>
nameserver <IP_ADDRESS>

Starting with --allDomains:
Apr 06 23:44:04 ipv4sni systemd[1]: Started sniproxy.
Apr 06 23:44:05 ipv4sni sniproxy[52880]: time=2023-04-06T23:44:05.166Z level=WARN msg="Domain list (--domainListPath) is not specified, routing ALL domains through the SNI proxy"
Apr 06 23:44:05 ipv4sni sniproxy[52880]: time=2023-04-06T23:44:05.167Z level=INFO msg="server info" public_ip=<IP_ADDRESS>
Apr 06 23:44:05 ipv4sni sniproxy[52880]: time=2023-04-06T23:44:05.168Z level=INFO msg="Certificate was not provided, using a self signed cert"
Apr 06 23:44:05 ipv4sni sniproxy[52880]: time=2023-04-06T23:44:05.168Z level=INFO msg="Started UDP DNS" service=dns host=<IP_ADDRESS> port=53
Apr 06 23:44:26 ipv4sni sniproxy[52880]: time=2023-04-06T23:44:26.254Z level=INFO msg="returned sniproxy address for domain" service=dns fqdn=google.com/about/careers/applications.
Apr 06 23:45:39 ipv4sni sniproxy[52880]: time=2023-04-06T23:45:39.409Z level=INFO msg="returned sniproxy address for domain" service=dns fqdn=google.com/about/careers/applications.

dig on my Tehran machine:
dig google.com/about/careers/applications

; <<>> DiG 9.18.1-1ubuntu1.3-Ubuntu <<>> google.com/about/careers/applications
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51440
;; flags: qr rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;google.com/about/careers/applications.
However:
wget google.com/about/careers/applications/signin
--2023-04-06 23:45:49--  http://google.com/about/careers/applications/signin
Resolving google.com (google.com)... <IP_ADDRESS>, <IP_ADDRESS>
Connecting to google.com (google.com)|<IP_ADDRESS>|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.google.com/about/careers/applications/signin [following]
--2023-04-06 23:45:49--  https://www.google.com/about/careers/applications/signin
Resolving www.google.com (www.google.com)... <IP_ADDRESS>, <IP_ADDRESS>
Connecting to www.google.com (www.google.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-04-06 23:45:49 ERROR 403: Forbidden.

And I think it's because wget is not using the system DNS, as it's resolving into <IP_ADDRESS>.
 So I think `google.com` DNS response is being tampered with.

WDYT?

Ah!!
It's Iran enforcing safesearch: https://support.opendns.com/hc/en-us/articles/227986807-How-to-Enforcing-Google-SafeSearch-YouTube-and-Bing
<IP_ADDRESS> is the IP address of google's safesearch.

two things:

editing /etc/resolve.conf usually doesn't work very well. use the following command instead:

sed -i 's/#DNS=/DNS=<IP_ADDRESS>/; s/#DNSStubListener=yes/DNSStubListener=no/' /etc/systemd/resolved.conf
systemctl restart systemd-resolved

this disables the stub listener, and changes your DNS to <IP_ADDRESS>. replace it with the current public IP of sniproxy and then restart systemd-resolved

if that doesn't work, try hard-coding google.com in your /etc/hosts with the IP of sniproxy. your file will look like this:

<IP_ADDRESS> google.com

this will bypass DNS altogether so you can test where the issue is. If editing /etc/hosts actually works, the midboxes in Iran are rewriting your DNS and "correcting" it back.
there is a neat way to avoid having DNS query and response altogether using another pet project of mine here.  Use it like this:

install it using:

go install github.com/mosajjal/go-exp/fakedns@latest

set up a simple rules.csv file with only one line:

$ cat /tmp/rules.csv
.,suffix,<IP_ADDRESS>

disable the default stub resolver (as mentioned above) and change your DNS server to <IP_ADDRESS>

start fakedns as root:

sudo fakedns -rule /tmp/rules.csv

if fakedns is not in your PATH, you'll get a file not found error. if that's the case, find the full path and replace fakedns with it. Full path is usually /home/USER/go/bin/fakedns
hopefully this helps.

Thank you @mosajjal for the instructions on changing DNS and bypassing the safesearch DNS rewrite.
Some progress but not quite there: I added <IP_ADDRESS> google.com to my /etc/hosts and now the URL does resolve to the sniproxy as expected, however, it gets rerouted to https://www.google.com/about/careers/applications/apply, and then gets stuck:
wget google.com/about/careers/applications/signin
--2023-04-09 19:08:40--  http://google.com/about/careers/applications/signin
Resolving google.com (google.com)... <IP_ADDRESS>
Connecting to google.com (google.com)|<IP_ADDRESS>|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.google.com/about/careers/applications/signin [following]
--2023-04-09 19:08:40--  https://www.google.com/about/careers/applications/signin
Resolving www.google.com (www.google.com)... <IP_ADDRESS>
Connecting to www.google.com (www.google.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://www.google.com/about/careers/applications/apply [following]
--2023-04-09 19:08:41--  https://www.google.com/about/careers/applications/apply
Reusing existing connection to www.google.com:443.
HTTP request sent, awaiting response...

On the sniproxy machine itself, wget google.com/about/careers/applications/signin is able to download the signin page.
And when on my Tehran machine, I just go to .../apply, I get stuck again:
wget https://www.google.com/about/careers/applications/apply
--2023-04-09 19:12:26--  https://www.google.com/about/careers/applications/apply
Resolving www.google.com (www.google.com)... <IP_ADDRESS>
Connecting to www.google.com (www.google.com)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response...

Server logs are as follows for google.com/about/careers/applications/signin:
Apr 09 19:25:57 ipv4sni sniproxy[52880]: time=2023-04-09T19:25:57.652Z level=INFO msg="rejected request" service=http ip=<IP_ADDRESS>:60116
Apr 09 19:25:57 ipv4sni sniproxy[52880]: time=2023-04-09T19:25:57.653Z level=INFO msg=REQ service=http method=GET host=google.com url=/about/careers/applications/signin
Apr 09 19:25:57 ipv4sni sniproxy[52880]: time=2023-04-09T19:25:57.683Z level=INFO msg="http response" service=http status_code="301 Moved Permanently"
Apr 09 19:25:57 ipv4sni sniproxy[52880]: time=2023-04-09T19:25:57.911Z level=INFO msg="establishing connection" service=https remote_ip=<IP_ADDRESS> source_ip=<IP_ADDRESS>:39084 host=www.google.com

And for https://www.google.com/about/careers/applications/apply:
Apr 09 19:26:45 ipv4sni sniproxy[52880]: time=2023-04-09T19:26:45.815Z level=INFO msg="rejected request" service=http ip=<IP_ADDRESS>:40346
Apr 09 19:26:45 ipv4sni sniproxy[52880]: time=2023-04-09T19:26:45.815Z level=INFO msg=REQ service=http method=GET host=google.com url=/about/careers/applications/apply
Apr 09 19:26:45 ipv4sni sniproxy[52880]: time=2023-04-09T19:26:45.827Z level=INFO msg="http response" service=http status_code="301 Moved Permanently"
Apr 09 19:26:46 ipv4sni sniproxy[52880]: time=2023-04-09T19:26:46.045Z level=INFO msg="establishing connection" service=https remote_ip=<IP_ADDRESS> source_ip=<IP_ADDRESS>:38020 host=www.google.com

So it looks sniproxy is not handling 301 moves?

so this confirms that /etc/hosts solves the google.com domain issue, I bet if you add www.google.com to your /etc/hosts that'll solve the 301 issue as well.
but obviously this won't be a long-term solution since you have to literally add every single FQDN to /etc/hosts and also /etc/hosts doesn't support wildcards :P
my suggestion for you is to use fakedns as mentioned above and configure your host to use it as a DNS resolver.
if that doesn't work or it's too hard to implement or the performance is too slow (possible since I haven't actually tested fakedns properly) let me know so I can share you a coredns config that'll leverage DNS-over-TCP which usually goes undetected by midboxes :)

Actually, /etc/hosts does have www.google.com mapped out to the sniproxy ip:
<IP_ADDRESS> google.com
<IP_ADDRESS> www.google.com
<IP_ADDRESS> accounts.google.com
<IP_ADDRESS> localhost

sniproxy logs indicate that the accounts.google.com call does reach the proxy:
Apr 11 22:14:06 ipv4sni sniproxy[52880]: time=2023-04-11T22:14:06.109Z level=INFO msg="establishing connection" service=https remote_ip=<IP_ADDRESS> source_ip=<IP_ADDRESS>:48118 host=www.google.com
Apr 11 22:14:07 ipv4sni sniproxy[52880]: time=2023-04-11T22:14:07.101Z level=INFO msg="establishing connection" service=https remote_ip=<IP_ADDRESS> source_ip=<IP_ADDRESS>:48130 host=accounts.google.com
Apr 11 22:15:05 ipv4sni sniproxy[52880]: time=2023-04-11T22:15:05.614Z level=INFO msg="establishing connection" service=https remote_ip=<IP_ADDRESS> source_ip=<IP_ADDRESS>:48060 host=www.google.com

Thanks for the hinsts on fakedns and coredns. I'd like to get this working with manual settings first before adding other tools into the mix.

as my experience on vultr services, we need to start cloudflare warp on vultr vps, then route the traffics from specific sites into cloudflare warp connection.
for example i had a problem while usuing chat.openai.com, installed warp-go created by chinese developer, then route traffics from openai.com site to cloudflare warp app instead of default main network on vultr vps.
i wish we had a speech as persian language 👯

yeah you might be onto something. looks like something new in the midboxes that block a legitimate TLS connection. have you tried setting /etc/hosts and then using an actual browser to visit the site? lately I've seen "non-standard" TLS clients being blocked.

I tried the same experiment on a Chrome browser running on a Windows server machine in IR and got pretty much the same results.
ERR_TIMED_OUT on the browser, and windows wget both times out.
Do you have benchmark experiments or probers running in Iran or plans to have them in the future? That would show whether this problem is local to my setup or not.

as my experience on vultr services, we need to start cloudflare warp on vultr vps, then route the traffics from specific sites into cloudflare warp connection.
for example i had a problem while usuing chat.openai.com, installed warp-go created by chinese developer, then route traffics from openai.com site to cloudflare warp app instead of default main network on vultr vps.
i wish we had a speech as persian language

assuming WARP overrides the fake IP that sniproxy needs so it won't work, but needs more testing.
the reason that Farsi is not used here is the fact that other countries also use the proxy for different purposes and it'd be good to include everyone into these conversations. Farsi limits the amount of people that can read the comments and discussions.

yeah you might be onto something. looks like something new in the midboxes that block a legitimate TLS connection. have you tried setting /etc/hosts and then using an actual browser to visit the site? lately I've seen "non-standard" TLS clients being blocked.

I tried the same experiment on a Chrome browser running on a Windows server machine in IR and got pretty much the same results.
ERR_TIMED_OUT on the browser, and windows wget both times out.
Do you have benchmark experiments or probers running in Iran or plans to have them in the future? That would show whether this problem is local to my setup or not.
Would be happy to help test out the proxy in the future using my IR nodes. Thank you for helping with this issue so far. v

I think we identified the root cause didn't we? the DNS looks to be the culprit.
you can potentially run a tcpdump against port 53 (tcpdump -nni any port 53) and conduct your tests that give you the timeout, and look at the DNS packets. happy to take a look myself too.

I had exact same issue with my DNS server which provided by sniproxy.
I have several VPS located all around the Europe and US and believe me, it is not location related issue. All results from different servers are the same.
The google's sign-in page returns 403 forbidden if I use <IP_ADDRESS> as upstream DNS resolver. However, if I use other resolvers like NextDNS or AdGuard this issue will not appear again. Please be noted that you should use DoT or DoH for upstream connection.

since this doesn't look like a bug in sniproxy and more of a functionality discussion, I'll convert the issue to a discussion and close this out.


Setting blob tags updates blob's ETag

Which service(blob, file, queue, table) does this issue concern?
Blob.
Which version of the Azurite was used?
3.26.0.
Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)
DockerHub.
What's the Node.js version?
N/A.
What problem was encountered?
Setting a blob's tags via the Set Blob Tags API endpoint changes the blob's ETag.
The official documentation states that setting a blob's tags should not modify the blob's ETag:

This operation doesn't update the ETag or last modified time of the blob.

We are building a solution that relies on the fact that we can update a blob's tags without changing the blob's ETag. A real storage account adheres to this, but Azurite seems not to.
Steps to reproduce the issue?

Run Azurite locally.
Create a container and a blob.
Get blob properties and inspect ETag.
Set blob tags.
Get blob properties and inspect ETag. The ETag is now different from the one in step three.

I've written a Go program that showcases the issue
package main

import (
	"context"
	"fmt"
	"os"

	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob"
	"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/appendblob"
	blob2 "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
)

func main() {
	if err := run(); err != nil {
		fmt.Println(err)
		os.Exit(1)
	}

	os.Exit(0)
}

func run() error {
	cred, err := azblob.NewSharedKeyCredential("devstoreaccount1", "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==")
	if err != nil {
		return err
	}
	client, err := azblob.NewClientWithSharedKeyCredential("http://<IP_ADDRESS>:10000/devstoreaccount1", cred, &azblob.ClientOptions{})
	if err != nil {
		return err
	}

	_, err = client.CreateContainer(context.Background(), "flamingo", &azblob.CreateContainerOptions{})
	if err != nil {
		return err
	}

	blob := client.
		ServiceClient().
		NewContainerClient("flamingo").
		NewAppendBlobClient("flamingo.txt")

	createResponse, err := blob.Create(context.Background(), &appendblob.CreateOptions{})
	if err != nil {
		return err
	}

	fmt.Println(fmt.Sprintf("Etag after creation: %s", *createResponse.ETag))

	propertiesResponse, err := blob.GetProperties(context.Background(), &blob2.GetPropertiesOptions{})
	if err != nil {
		return err
	}

	fmt.Println(fmt.Sprintf("Etag before setting tags: %s", *propertiesResponse.ETag))

	fmt.Println("Setting tags")

	_, err = blob.SetTags(context.Background(), map[string]string{"colour": "pink"}, &blob2.SetTagsOptions{})
	if err != nil {
		return err
	}

	propertiesResponse, err = blob.GetProperties(context.Background(), &blob2.GetPropertiesOptions{})
	if err != nil {
		return err
	}

	fmt.Println(fmt.Sprintf("Etag after setting tags: %s", *propertiesResponse.ETag))

	return nil
}

The program outputs:
Etag after creation: "0x2382396F57036C0"
Etag before setting tags: "0x2382396F57036C0"
Setting tags
Etag after setting tags: "0x244E35F69E95700"

Debug logs
2023-12-21T13:13:10.977Z 	 info: Azurite Blob service is starting on <IP_ADDRESS>:10000
2023-12-21T13:13:10.977Z 	 info: AccountDataStore:init() Refresh accounts from environment variable AZURITE_ACCOUNTS with value undefined
2023-12-21T13:13:10.977Z 	 info: AccountDataStore:init() Fallback to default emulator account devstoreaccount1.
2023-12-21T13:13:10.990Z 	 info: BlobGCManager:start() Starting BlobGCManager. Set status to Initializing.
2023-12-21T13:13:10.990Z 	 info: BlobGCManager:start() Trigger mark and sweep loop. Set status to Running.
2023-12-21T13:13:10.991Z 	 info: BlobGCManager:markSweepLoop() Start next mark and sweep.
2023-12-21T13:13:10.991Z 	 info: BlobGCManager:markSweep() Get all extents.
2023-12-21T13:13:10.991Z 	 info: BlobGCManager:start() BlobGCManager successfully started.
2023-12-21T13:13:10.992Z 	 info: BlobGCManager:markSweep() Got 0 extents.
2023-12-21T13:13:10.992Z 	 info: BlobGCManager:markSweep() Get referred extents.
2023-12-21T13:13:10.992Z 	 info: BlobGCManager:markSweep() Got referred extents, unreferenced extents count is 0.
2023-12-21T13:13:10.992Z 	 info: BlobGCManager:markSweepLoop() Mark and sweep finished, taken 1ms.
2023-12-21T13:13:10.992Z 	 info: BlobGCManager:markSweepLoop() Sleep for 600000ms.
2023-12-21T13:13:10.993Z 	 info: Azurite Blob service successfully listens on http://<IP_ADDRESS>:10000
2023-12-21T13:13:18.543Z d89c1764-53c9-460c-ac54-bed89c54c634 info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://<IP_ADDRESS>/devstoreaccount1/flamingo?restype=container RequestHeaders:{"host":"<IP_ADDRESS>:10000","user-agent":"azsdk-go-azblob/v1.1.0 (go1.21.0; darwin)","content-length":"0","accept":"application/xml","authorization":"SharedKey devstoreaccount1:AZSZ0jbOqVXOlPXTiSlrhjSUEMmoRHA7QutNP/5K06w=","x-ms-date":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-version":"2020-10-02","accept-encoding":"gzip"} ClientIP=<IP_ADDRESS> Protocol=http HTTPVersion=1.1
2023-12-21T13:13:18.544Z d89c1764-53c9-460c-ac54-bed89c54c634 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=flamingo Blob=
2023-12-21T13:13:18.545Z d89c1764-53c9-460c-ac54-bed89c54c634 verbose: DispatchMiddleware: Dispatching request...
2023-12-21T13:13:18.549Z d89c1764-53c9-460c-ac54-bed89c54c634 info: DispatchMiddleware: Operation=Container_Create
2023-12-21T13:13:18.551Z d89c1764-53c9-460c-ac54-bed89c54c634 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2023-12-21T13:13:18.551Z d89c1764-53c9-460c-ac54-bed89c54c634 info: PublicAccessAuthenticator:validate() Start validation against public access.
2023-12-21T13:13:18.551Z d89c1764-53c9-460c-ac54-bed89c54c634 debug: PublicAccessAuthenticator:validate() Getting account properties...
2023-12-21T13:13:18.552Z d89c1764-53c9-460c-ac54-bed89c54c634 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: flamingo, blob:
2023-12-21T13:13:18.559Z d89c1764-53c9-460c-ac54-bed89c54c634 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container flamingo
2023-12-21T13:13:18.559Z d89c1764-53c9-460c-ac54-bed89c54c634 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2023-12-21T13:13:18.582Z d89c1764-53c9-460c-ac54-bed89c54c634 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 21 Dec 2023 13:13:18 GMT\nx-ms-version:2020-10-02\n/devstoreaccount1/devstoreaccount1/flamingo\nrestype:container"
2023-12-21T13:13:18.582Z d89c1764-53c9-460c-ac54-bed89c54c634 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:AZSZ0jbOqVXOlPXTiSlrhjSUEMmoRHA7QutNP/5K06w=
2023-12-21T13:13:18.582Z d89c1764-53c9-460c-ac54-bed89c54c634 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2023-12-21T13:13:18.582Z d89c1764-53c9-460c-ac54-bed89c54c634 verbose: DeserializerMiddleware: Start deserializing...
2023-12-21T13:13:18.584Z d89c1764-53c9-460c-ac54-bed89c54c634 info: HandlerMiddleware: DeserializedParameters={"options":{"metadata":{},"containerCpkScopeInfo":{}},"restype":"container","version":"2020-10-02"}
2023-12-21T13:13:18.585Z d89c1764-53c9-460c-ac54-bed89c54c634 verbose: SerializerMiddleware: Start serializing...
2023-12-21T13:13:18.586Z d89c1764-53c9-460c-ac54-bed89c54c634 info: EndMiddleware: End response. TotalTimeInMS=43 StatusCode=201 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.26.0","etag":"\"0x1F883A7A98CF660\"","last-modified":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-request-id":"d89c1764-53c9-460c-ac54-bed89c54c634","x-ms-version":"2023-08-03"}
2023-12-21T13:13:18.589Z 1ae162c9-c012-4315-b673-4c36adc23bd7 info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://<IP_ADDRESS>/devstoreaccount1/flamingo/flamingo.txt RequestHeaders:{"host":"<IP_ADDRESS>:10000","user-agent":"azsdk-go-azblob/v1.1.0 (go1.21.0; darwin)","content-length":"0","accept":"application/xml","authorization":"SharedKey devstoreaccount1:OmZobbzFTzbZpqJZCJMqVefc+GsQQuHCrUiZ4rFtY+U=","x-ms-date":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-blob-type":"AppendBlob","x-ms-version":"2020-10-02","accept-encoding":"gzip"} ClientIP=<IP_ADDRESS> Protocol=http HTTPVersion=1.1
2023-12-21T13:13:18.590Z 1ae162c9-c012-4315-b673-4c36adc23bd7 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=flamingo Blob=flamingo.txt
2023-12-21T13:13:18.590Z 1ae162c9-c012-4315-b673-4c36adc23bd7 verbose: DispatchMiddleware: Dispatching request...
2023-12-21T13:13:18.590Z 1ae162c9-c012-4315-b673-4c36adc23bd7 info: DispatchMiddleware: Operation=AppendBlob_Create
2023-12-21T13:13:18.591Z 1ae162c9-c012-4315-b673-4c36adc23bd7 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2023-12-21T13:13:18.591Z 1ae162c9-c012-4315-b673-4c36adc23bd7 info: PublicAccessAuthenticator:validate() Start validation against public access.
2023-12-21T13:13:18.591Z 1ae162c9-c012-4315-b673-4c36adc23bd7 debug: PublicAccessAuthenticator:validate() Getting account properties...
2023-12-21T13:13:18.591Z 1ae162c9-c012-4315-b673-4c36adc23bd7 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: flamingo, blob: flamingo.txt
2023-12-21T13:13:18.592Z 1ae162c9-c012-4315-b673-4c36adc23bd7 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container flamingo
2023-12-21T13:13:18.592Z 1ae162c9-c012-4315-b673-4c36adc23bd7 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2023-12-21T13:13:18.592Z 1ae162c9-c012-4315-b673-4c36adc23bd7 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n\n\n\n\n\n\n\n\n\nx-ms-blob-type:AppendBlob\nx-ms-date:Thu, 21 Dec 2023 13:13:18 GMT\nx-ms-version:2020-10-02\n/devstoreaccount1/devstoreaccount1/flamingo/flamingo.txt"
2023-12-21T13:13:18.593Z 1ae162c9-c012-4315-b673-4c36adc23bd7 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:OmZobbzFTzbZpqJZCJMqVefc+GsQQuHCrUiZ4rFtY+U=
2023-12-21T13:13:18.593Z 1ae162c9-c012-4315-b673-4c36adc23bd7 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2023-12-21T13:13:18.593Z 1ae162c9-c012-4315-b673-4c36adc23bd7 verbose: DeserializerMiddleware: Start deserializing...
2023-12-21T13:13:18.593Z 1ae162c9-c012-4315-b673-4c36adc23bd7 info: HandlerMiddleware: DeserializedParameters={"options":{"metadata":{},"blobHTTPHeaders":{},"leaseAccessConditions":{},"cpkInfo":{},"cpkScopeInfo":{},"modifiedAccessConditions":{}},"contentLength":0,"version":"2020-10-02","blobType":"AppendBlob"}
2023-12-21T13:13:18.594Z 1ae162c9-c012-4315-b673-4c36adc23bd7 verbose: SerializerMiddleware: Start serializing...
2023-12-21T13:13:18.594Z 1ae162c9-c012-4315-b673-4c36adc23bd7 info: EndMiddleware: End response. TotalTimeInMS=5 StatusCode=201 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.26.0","etag":"\"0x2382396F57036C0\"","last-modified":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-request-id":"1ae162c9-c012-4315-b673-4c36adc23bd7","x-ms-version":"2023-08-03","date":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-request-server-encrypted":"true"}
2023-12-21T13:13:18.596Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 info: BlobStorageContextMiddleware: RequestMethod=HEAD RequestURL=http://<IP_ADDRESS>/devstoreaccount1/flamingo/flamingo.txt RequestHeaders:{"host":"<IP_ADDRESS>:10000","user-agent":"azsdk-go-azblob/v1.1.0 (go1.21.0; darwin)","accept":"application/xml","authorization":"SharedKey devstoreaccount1:gIkJxXyOMTXYdZFC0T66tjn8mbYtQsuQr1Zx/77npVQ=","x-ms-date":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-version":"2020-10-02"} ClientIP=<IP_ADDRESS> Protocol=http HTTPVersion=1.1
2023-12-21T13:13:18.596Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=flamingo Blob=flamingo.txt
2023-12-21T13:13:18.596Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 verbose: DispatchMiddleware: Dispatching request...
2023-12-21T13:13:18.596Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 info: DispatchMiddleware: Operation=Blob_GetProperties
2023-12-21T13:13:18.596Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2023-12-21T13:13:18.596Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 info: PublicAccessAuthenticator:validate() Start validation against public access.
2023-12-21T13:13:18.596Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 debug: PublicAccessAuthenticator:validate() Getting account properties...
2023-12-21T13:13:18.596Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: flamingo, blob: flamingo.txt
2023-12-21T13:13:18.596Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container flamingo
2023-12-21T13:13:18.596Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2023-12-21T13:13:18.597Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"HEAD\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 21 Dec 2023 13:13:18 GMT\nx-ms-version:2020-10-02\n/devstoreaccount1/devstoreaccount1/flamingo/flamingo.txt"
2023-12-21T13:13:18.597Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:gIkJxXyOMTXYdZFC0T66tjn8mbYtQsuQr1Zx/77npVQ=
2023-12-21T13:13:18.597Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2023-12-21T13:13:18.597Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 verbose: DeserializerMiddleware: Start deserializing...
2023-12-21T13:13:18.597Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 info: HandlerMiddleware: DeserializedParameters={"options":{"leaseAccessConditions":{},"cpkInfo":{},"modifiedAccessConditions":{}},"version":"2020-10-02"}
2023-12-21T13:13:18.597Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 verbose: SerializerMiddleware: Start serializing...
2023-12-21T13:13:18.598Z 01b8a9da-1ea8-4a2e-87df-18acb0863533 info: EndMiddleware: End response. TotalTimeInMS=2 StatusCode=200 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.26.0","last-modified":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-creation-time":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-blob-type":"AppendBlob","x-ms-lease-state":"available","x-ms-lease-status":"unlocked","content-length":"0","content-type":"application/octet-stream","etag":"\"0x2382396F57036C0\"","x-ms-request-id":"01b8a9da-1ea8-4a2e-87df-18acb0863533","x-ms-version":"2023-08-03","date":"Thu, 21 Dec 2023 13:13:18 GMT","accept-ranges":"bytes","x-ms-blob-committed-block-count":"0","x-ms-server-encrypted":"true"}
2023-12-21T13:13:18.599Z b0b8671f-5fbf-4164-b954-720ddefbc1fc info: BlobStorageContextMiddleware: RequestMethod=PUT RequestURL=http://<IP_ADDRESS>/devstoreaccount1/flamingo/flamingo.txt?comp=tags RequestHeaders:{"host":"<IP_ADDRESS>:10000","user-agent":"azsdk-go-azblob/v1.1.0 (go1.21.0; darwin)","content-length":"116","accept":"application/xml","authorization":"SharedKey devstoreaccount1:8fEyaWIZCA4ZWUpWvO8Cjii8R4KPk/MZMFIITVBrIUs=","content-type":"application/xml","x-ms-date":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-version":"2020-10-02","accept-encoding":"gzip"} ClientIP=<IP_ADDRESS> Protocol=http HTTPVersion=1.1
2023-12-21T13:13:18.599Z b0b8671f-5fbf-4164-b954-720ddefbc1fc info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=flamingo Blob=flamingo.txt
2023-12-21T13:13:18.599Z b0b8671f-5fbf-4164-b954-720ddefbc1fc verbose: DispatchMiddleware: Dispatching request...
2023-12-21T13:13:18.600Z b0b8671f-5fbf-4164-b954-720ddefbc1fc info: DispatchMiddleware: Operation=Blob_SetTags
2023-12-21T13:13:18.600Z b0b8671f-5fbf-4164-b954-720ddefbc1fc verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2023-12-21T13:13:18.600Z b0b8671f-5fbf-4164-b954-720ddefbc1fc info: PublicAccessAuthenticator:validate() Start validation against public access.
2023-12-21T13:13:18.600Z b0b8671f-5fbf-4164-b954-720ddefbc1fc debug: PublicAccessAuthenticator:validate() Getting account properties...
2023-12-21T13:13:18.600Z b0b8671f-5fbf-4164-b954-720ddefbc1fc debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: flamingo, blob: flamingo.txt
2023-12-21T13:13:18.601Z b0b8671f-5fbf-4164-b954-720ddefbc1fc debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container flamingo
2023-12-21T13:13:18.601Z b0b8671f-5fbf-4164-b954-720ddefbc1fc info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2023-12-21T13:13:18.601Z b0b8671f-5fbf-4164-b954-720ddefbc1fc info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"PUT\n\n\n116\n\napplication/xml\n\n\n\n\n\n\nx-ms-date:Thu, 21 Dec 2023 13:13:18 GMT\nx-ms-version:2020-10-02\n/devstoreaccount1/devstoreaccount1/flamingo/flamingo.txt\ncomp:tags"
2023-12-21T13:13:18.601Z b0b8671f-5fbf-4164-b954-720ddefbc1fc info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:8fEyaWIZCA4ZWUpWvO8Cjii8R4KPk/MZMFIITVBrIUs=
2023-12-21T13:13:18.601Z b0b8671f-5fbf-4164-b954-720ddefbc1fc info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2023-12-21T13:13:18.601Z b0b8671f-5fbf-4164-b954-720ddefbc1fc verbose: DeserializerMiddleware: Start deserializing...
2023-12-21T13:13:18.601Z b0b8671f-5fbf-4164-b954-720ddefbc1fc debug: deserialize(): Raw request body string is (removed all empty characters) <?xmlversion="1.0"encoding="UTF-8"?><Tags><TagSet><Tag><Key>colour</Key><Value>pink</Value></Tag></TagSet></Tags>
2023-12-21T13:13:18.605Z b0b8671f-5fbf-4164-b954-720ddefbc1fc info: HandlerMiddleware: DeserializedParameters={"options":{"modifiedAccessConditions":{},"leaseAccessConditions":{},"tags":{"blobTagSet":[{"key":"colour","value":"pink"}],"TagSet":{"Tag":{"Key":"colour","Value":"pink"}}}},"comp":"tags","version":"2020-10-02","body":"ReadableStream"}
2023-12-21T13:13:18.605Z b0b8671f-5fbf-4164-b954-720ddefbc1fc verbose: SerializerMiddleware: Start serializing...
2023-12-21T13:13:18.606Z b0b8671f-5fbf-4164-b954-720ddefbc1fc info: EndMiddleware: End response. TotalTimeInMS=7 StatusCode=204 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.26.0","x-ms-request-id":"b0b8671f-5fbf-4164-b954-720ddefbc1fc","x-ms-version":"2023-08-03","date":"Thu, 21 Dec 2023 13:13:18 GMT"}
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 info: BlobStorageContextMiddleware: RequestMethod=HEAD RequestURL=http://<IP_ADDRESS>/devstoreaccount1/flamingo/flamingo.txt RequestHeaders:{"host":"<IP_ADDRESS>:10000","user-agent":"azsdk-go-azblob/v1.1.0 (go1.21.0; darwin)","accept":"application/xml","authorization":"SharedKey devstoreaccount1:gIkJxXyOMTXYdZFC0T66tjn8mbYtQsuQr1Zx/77npVQ=","x-ms-date":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-version":"2020-10-02"} ClientIP=<IP_ADDRESS> Protocol=http HTTPVersion=1.1
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 info: BlobStorageContextMiddleware: Account=devstoreaccount1 Container=flamingo Blob=flamingo.txt
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 verbose: DispatchMiddleware: Dispatching request...
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 info: DispatchMiddleware: Operation=Blob_GetProperties
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 verbose: AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 info: PublicAccessAuthenticator:validate() Start validation against public access.
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 debug: PublicAccessAuthenticator:validate() Getting account properties...
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 debug: PublicAccessAuthenticator:validate() Retrieved account name from context: devstoreaccount1, container: flamingo, blob: flamingo.txt
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 debug: PublicAccessAuthenticator:validate() Skip public access authentication. Cannot get public access type for container flamingo
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 info: BlobSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 info: BlobSharedKeyAuthenticator:validate() [STRING TO SIGN]:"HEAD\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 21 Dec 2023 13:13:18 GMT\nx-ms-version:2020-10-02\n/devstoreaccount1/devstoreaccount1/flamingo/flamingo.txt"
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 info: BlobSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:gIkJxXyOMTXYdZFC0T66tjn8mbYtQsuQr1Zx/77npVQ=
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 info: BlobSharedKeyAuthenticator:validate() Signature 1 matched.
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 verbose: DeserializerMiddleware: Start deserializing...
2023-12-21T13:13:18.607Z 94218039-01f7-4630-bd77-534715a1f3e4 info: HandlerMiddleware: DeserializedParameters={"options":{"leaseAccessConditions":{},"cpkInfo":{},"modifiedAccessConditions":{}},"version":"2020-10-02"}
2023-12-21T13:13:18.608Z 94218039-01f7-4630-bd77-534715a1f3e4 verbose: SerializerMiddleware: Start serializing...
2023-12-21T13:13:18.608Z 94218039-01f7-4630-bd77-534715a1f3e4 info: EndMiddleware: End response. TotalTimeInMS=1 StatusCode=200 StatusMessage=undefined Headers={"server":"Azurite-Blob/3.26.0","last-modified":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-creation-time":"Thu, 21 Dec 2023 13:13:18 GMT","x-ms-blob-type":"AppendBlob","x-ms-lease-state":"available","x-ms-lease-status":"unlocked","content-length":"0","content-type":"application/octet-stream","etag":"\"0x244E35F69E95700\"","x-ms-request-id":"94218039-01f7-4630-bd77-534715a1f3e4","x-ms-version":"2023-08-03","date":"Thu, 21 Dec 2023 13:13:18 GMT","accept-ranges":"bytes","x-ms-blob-committed-block-count":"0","x-ms-server-encrypted":"true","x-ms-tag-count":"1"}

For reference, I am running Azurite locally with this command:
docker run -p 10000:10000 -v /Users/lasse.hels/azurite:/workspace mcr.microsoft.com/azure-storage/azurite:3.26.0 azurite-blob --blobHost <IP_ADDRESS> -l /workspace -d /workspace/debug.log

Have you found a mitigation/solution?
No.
This issue is not a dealbreaker for us, as our solution does work when run against a real storage account. Azurite's behaviour is preventing us from writing automated tests for some functionality that we would like to test.
We are pleased that Azurite exists and are happy to use it in general. Thanks.

@LasseHels
Thanks for raising this issue!
I believe the fix need change the following 2 files. I will do more test and raise a fix, might be in next week.
Delete line 3356-3357 in:
https://github.com/Azure/Azurite/blob/43f92995615564161cb2d0419cc4984d0eee9be0/src/blob/persistence/LokiBlobMetadataStore.ts#L3356
Delete line 3357-3358 in:
https://github.com/Azure/Azurite/blob/43f92995615564161cb2d0419cc4984d0eee9be0/src/blob/persistence/SqlBlobMetadataStore.ts#L3358

@blueww Thanks for the swift reply. We're looking forward to the fix!

PR https://github.com/Azure/Azurite/pull/2328 is raised to fix this issue.

Close as the fix PR is merged.
The fix will be in the next Azurite release.

@blueww I'm closing this as the change was released in v3.30.0. Thanks!


Is it possible to receive items from other components from component that using swap plugin?

When you use the swap plugin, you lose the ability to receive items from other components(in the same group).
Is it possible to receive items from other components and in the same time have the ability to swap in own component?

@ParsaJR Not at this point, something we want to get implemented. I'll tag this as an enhancement.


where can I find v1.2.0 documentation site?

Is this project still maintained?
The v2 is very buggy so I switched to v1.2.0(also a lot of bug but at least it can run)
But the doc site https://www.docz.site/ is for v2 only, where can I find the old doc site?

find doc site repo https://github.com/pedronauck/docz-website. Master branch, fresh install, still can not run...

I can't run v2 either.

some problem and V2 is too large for me~

Really looking for the docs for 1.2 as well! V2 is still very new and all examples of projects using docz live right now are on 1.2 or less. We are definitely interested in eventually migrating to v2, but are finding that it is much easier to get off the ground using 1.2 and looking at examples that are live. Any chance we can see the docs for 1.2 or less somewhere?

I deployed the v1.2 docs to : https://docz-v1.surge.sh/
Hope that helps !

Added a link to the v1 docs on the README


[FIX JENKINS-39857_Display_steps_durations_in_Blue_Ocean]

… not use humanize() format but a custom one
Description

See JENKINS-39857.

Allow that ResultItem to accept an object as well (not only a string as before). Allow timeDuration to not use humanize() format but a custom one.
Submitter checklist

[ ] Link to JIRA ticket in description, if appropriate.
[ ] Change is code complete and matches issue description
[ ] Appropriate unit or acceptance tests or explanation to why this change has no tests

Reviewer checklist

[ ] Run the changes and verified the change matches the issue description
[ ] Reviewed the code
[ ] Verified that the appropriate tests have been written or valid explanation given

@reviewbybees

@scherler What's wrong with the moment format we use everywhere right now?

@i386 nothing, we only need to allow that we can use a custom format as requested in the ticket

https://github.com/jenkinsci/blueocean-plugin/pull/743 is the user

@scherler oh I think the existing moment format is fine - the mockup may be a bit out of date

@i386 it does not hurt neither. liberty to the people ;)

@scherler I think a user defined that mockup - not sure what happened to our one

this should be similar to the header - but I think the header says "a few seconds" - I am pretty sure there was tickets I may have accidentally closed that assumed the header was different.
The user did not request it be customisable, they just suggested a short hand display, but whatever it is, should be the same as the header right @i386? Why not make it consistent

@michaelneale actually there is a ticket in the backlog to adjust it from "A few seconds" to "X seconds". Yes, consistency is key to making our UI great :)

@scherler if you are in the area https://issues.jenkins-ci.org/browse/JENKINS-41282 changes "A few seconds" to "X seconds".

right, so if the header is updated as per that ticket, then the steps can use the same logic right? (and it will be closer to what Julian wanted). Win win!
Nice one @scherler !

@michaelneale thats right!

yeah unfortunately, I think https://issues.jenkins-ci.org/browse/JENKINS-41282 should be done at the same time (combine the two) so that the same code can be used in both.

@i386 and @michaelneale https://issues.jenkins-ci.org/browse/JENKINS-41282 is now part of this fix

Nice one :)


Taking constant damage on < 1/32 size with the up, down, and all around mod (broken hitbox)

When ever i try to scale lower than 1/32, i start taking damage in survival. How do I fix this?
Expected behavior
to be able to scale at least to 1/64 without taking damage
Additional information:

Forge version: 1.12.2-2838
Mod version:1.12.2-<IP_ADDRESS>

Wow, this is new.
Well I cant reproduce, obviously.

Can you test it while being in the air?
Anyway, not enough info, and I am not working on the mod currently.
Big rework expected to be done this summer, so as with most bugs this may get randomly fixed or simply won't exist in new code.

I figure it out, it is a bug with the up, down, and all around mod.

Just tested this issue, basically the hitbox doesn't get changed when you resize and Minecraft thinks that you are being suffocated by a block (confirmed in fixed-mixin version)

Ok while latest mixin version has no hitbox issues, the gravity simply does not work (the camera is rotated and thats it).
This is due to 1.12 Forge+Mixin limitations not allowing udaa coremod to work, nothing I can do about that.
So yeah wontfix I guess, should work (or at least no fail as completely until I fix it) if both mods get updated to 1.13+ at some point.


ES2015 modules

On the latest 3.1.2 version you are compiling modules as es2015 instead of commonjs modules.
I guess it's a simple mistake.

Hi @ghetolay,
inside the /bundles/ you may find umd format modules to be used in commonJs. I just updated the Readme to showcase possible usage of these files.

Hi,
Thks for your update for systemJS users.
But I was being silly because I'm using webpack and es2015 modules are fine. I just got an unexpected token export error and I assumed things too fast.
Anyway I don't know what was the real cause of that error but latest version seems to fix it. (maybe this was due to typings folder being published).

Hi ghetolay,
Can you tell me how you to use the scroll with webpack? Just import the module in my app.module or do I have to do additional stuff?

Yes nothing more to do with version 3.2.1. Do you have any trouble ?

Thanks!!!
I'm about to try it out. Just looking around for things I'll need. Like scrolling and ways to animate components in and out (if you have a link for the latter, please let me know!).


:+1: Add localStorage and sessionStorage

It's experimental. See https://github.com/vim-denops/denops.vim/pull/135#issuecomment-939959196

Some said "localStorage is quite limited (5MB for entire deno project, doesn't work on Worker thread, etc.) so we should not rely on it" and I agree with that, at least for now.

OK. We should not rely on sessionStorage or localStorage.


Fixes alias TypeError issue (#4104)

This issue was fixed on the 3.0.x support branch, but still present on 3.1

This is going to be fixed with backport from 3.0.x branch


Quark Wooden Pressure Plates aren't supported

Quark's Obsidian Pressure Plate is supported, but not the wood variants the mod adds.

I will work on implementing this over the weekend. Apologies for not having initial support for them. This project had been in development before this repository was made, and it was still being tested against a Quark version from December 2017, so I had no idea you had added wooden variants heh.


add more RBAC permissions for dashboard leader election

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (7bd692b) 35.20% compared to head (105494c) 35.20%.

:exclamation: Current head 105494c differs from pull request most recent head 7dbe46f. Consider uploading reports for the commit 7dbe46f to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #809   +/-   ##
=======================================
  Coverage   35.20%   35.20%
=======================================
  Files          37       37
  Lines        4550     4550
=======================================
  Hits         1602     1602
  Misses       2795     2795
  Partials      153      153

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Avoid re-emitting docgen for re-exported components

In our codebase we will often define a component in one file, then re-export it from index.ts. In some cases, the re-export uses a name that does not match the original component. In these cases, react-docgen-loader emits invalid code that tries to assign docgen info to the actual name from the original component, which fails.
// components/Button/Button.tsx
import React from 'react';
export const Button = () => (<button type='submit'/>);
export default Button;

// components/Button/index.ts
import ButtonComponent from './Button';
export default ButtonComponent;

// components/Bouton/index.ts
import {Button as Bouton} from '../Button/Button';
export {Bouton};

// App.tsx
import Button from './components/Button';
import Bouton from './components/Bouton';

const App = () => (
  <form><Button/><Bouton/></form>
);

Currently invalid code is written into components/Button/index.ts and components/Bouton/index.ts that tries to assign to Button.__docgenInfo, when Button is not a valid identifier in either file.
The react-webpack implementation of react-docgen-loader avoids this problem by checking that the resolved component definition comes from the initial source file. See https://github.com/storybookjs/storybook/blob/e812f77def55aa7aeda70dddcc378c736412472d/code/presets/react-webpack/src/loaders/react-docgen-loader.ts#L25
This PR adds the same logic here, which fixes the issue.

Thanks for the sync up with storybook's latest code. 🫶

Thanks you!


Random cylinder coordinates not uniform

Uniformly sampling from possible radii and angles will result in greater density near the cylinder's axis. This could be resolved by weighting the radius sampling (proportional to radius?) to favor greater radii.

The current (wrong) implementation uses the circular uniform distribution


support all protocol numbers

support https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml protocol numbers for aws sg and nacl

Should also check if in IBM VPCs, setting protocols to all allows protocols which are not TCP, UDP or ICMP.

Made an experiment with live VPCs (thanks @kyorav), sending SCTP traffic when "all" protocols are allowed in both NACLs and SGs.
Conclusions:

In AWS: SCTP traffic is allowed to flow - SCTP connections can be established.
In IBM Cloud: Attempts to make SCTP connection timed out.

So indeed, the two clouds interpret "all" protocols differently.


Automatic testing with small open-source models

See https://www.reddit.com/r/LocalLLaMA/comments/12vu0uz/minimal_ggml_file_or_are_there_any_tinytoy_models/
It would be nice to use these models to automatically test that inference works consistently across changes.

With our new support for other model architectures, this should be much more possible. Not sure if we can support LLaMA, though.


[Feature Request] Custom links based on tags

A common issue I have is linking events across different monitoring systems. For example, a very common scenario would be me having to copy a trace id from Haystack, open a new tab, open some centralized logging tool like and paste the traceid there to see the logs for that particular trace.
I propose to add a new feature to the Haystack UI where we can configure custom links.
These could be defined in a json file where we would have a title and the rules to build the link itself based on placeholders where we could resolve the values of different tags and potentially time windows as well.
Example below:
[
    {
        "text": "Kibana",
        "url": "http://kibana.mycompany.com/?query=foo_bar=${traceId}"
    },
    {
        "text": "Grafana",
        "url": "http://grafana.mycompany.com/?dashboard=foo_bar=${serviceName}&startTime=${timestamp-5m}&endTime=${timestamp+5m}"
    }
]

The following is a good read on this, and how the same/similar feature works in Jaeger https://itnext.io/custom-links-in-jaeger-ui-3479f72bb815
And a similar feature in Zipkin: https://github.com/openzipkin/zipkin-classic#how-do-i-find-logs-associated-with-a-particular-trace

@worldtiki  this was a feature in our previous UI - we lost it when we did the UI.  In fact had the same templating like you suggested above with startTime, endTime, serviceName, operationName and traceId. Shouldnt be too hard to add it back in.  Both from trends view (start/endtime/servicename and operation) and in trace view (traceid) to any external system.


Minhash improvements

The main change here are consolidating two lines into one by removing tokens.
Besides simplification, this helps with memory consumption since set consumes much more memory (due to the ratio between the content and hashes, almost twice a list)
Since we are using the set datastructure for deduplication and is later converted into a list, we do not have to preserve that structure.
Although I have witnessed some speed acceleration as well, it is not consistent and seems to be within statistical variance. I did not witness any regressions though.

I don't understand the justification here.
A document usually has N shingles and M unique shingles (N >> M). In the PR, you create an np.array with hash_func called for each one of the N shingles first. I can't see how this changes anything in any meaningful way.
One could, for example:
hashvalues: np.ndarray = np.array([
    hash_func(token) for token in {
        bytes(" ".join(t), "utf-8")
        for t in ngrams(NON_ALPHA.split(content.lower()),
        ngram_size, min_length)
    }
], dtype=dtype)

skip the variable by creating the set on the fly if memory is concerned.

I tested the PR's embed_func with the file content from text_dedup/minhash.py. Here is the result:
Before:
3.95 ms ± 65.1 µs per loop (mean ± std. dev. of 3 runs, 1,000 loops each)

After:
4.18 ms ± 24.4 µs per loop (mean ± std. dev. of 3 runs, 1,000 loops each)

Skip `tokens` but keep set:
3.77 ms ± 191 µs per loop (mean ± std. dev. of 3 runs, 1,000 loops each)

Code snippet to reproduce:
from pathlib import Path

content = Path("/Users/chenghao/Downloads/text-dedup/text_dedup/minhash.py").read_text()
idx = 0
num_perm = 250
ngram_size = 5
hashranges = [(i, i + 25) for i in range(0, 250, 25)]
max_hash = np.uint32((1 << 32) - 1)
modulo_prime = np.uint32((1 << 32) - 5)
PERMUTATIONS = np.array(
    [
        (
            RNG.randint(1, np.uint32((1 << 32) - 5), dtype=np.uint32),
            RNG.randint(0, np.uint32((1 << 32) - 5), dtype=np.uint32),
        )
        for _ in range(num_perm)
    ],
    dtype=np.uint32,
).T

%%timeit -n 1000 -r 3
res = embed_func(
    content,
    idx,
    num_perm=num_perm,
    ngram_size=ngram_size,
    min_length=0,
    hashranges=hashranges,
    permutations=PERMUTATIONS,
    hash_func=xxh3_32hash,
    dtype=np.uint32,
    max_hash=max_hash,
    modulo_prime=modulo_prime
)

Thanks for the reproduction code. it will come in handy going forward.
I might even rearrange it into some microbenchmark and add it as a future PR.
As for my part of the testing. (PC is AMD Ryzen 7950x)
(for mor extensive testing i've done this)
%%timeit -n 2000-r 7
Original :
3.24 ms ± 73 µs per loop (mean ± std. dev. of 7 runs, 2,000 loops each)
This PR :
3.24 ms ± 41.1 µs per loop (mean ± std. dev. of 7 runs, 2,000 loops each)
This PR but operating on list directly without making np.array in the middle.:
3.23 ms ± 23 µs per loop (mean ± std. dev. of 7 runs, 2,000 loops each)
ChenghaoMa Version Suggested in PR :
3.24 ms ± 50.2 µs per loop (mean ± std. dev. of 7 runs, 2,000 loops each)
np.fromiter (Not ever making a list at all and building up with iters)
3.29 ms ± 58.4 µs per loop (mean ± std. dev. of 7 runs, 2,000 loops each)

What minor differences shown above are statistical variances.
I've tested much more under different settings but each version all come within 3.2ms~3.3ms upon repeated testing
It seems that under these test all these choices are neutral at my machine, it could very well be a regression in others.
I feel that whatever gains I might have witnessed on operating on OSCAR datasets might have been statistical variances as well.
Multiple other methods I've thought of do not show any tangible reproducible benefits either.
My motivation to exploit vectorized advanced numpy functions was a great underestimation of python native implementations.
One side benefit i was hoping for was enhanced readability but that is also controversial.
I think atleast for now, reverting embed_func back to the original version is the best way to go.
An underlying reason I have been trying to use np native functions and datastructures as much as possible was to rewrite as many functions as possible to support vectorization, but it seems to overcomplicate the code and make it difficult to read and understand with questionable benefits at the moment.
Also, even if i achieve some level of vectorization that benefits the above micro bench, it is unclear if it would be beneficial when instance of embed_func is also launched in a multiprocessing manner with dataset.map()
I added a revert commit to revert the tokens and hashvalues changes.


[test][core] update mocha version to ^10.0.0

and @types/mocha to ^10.0.0

API change check
API changes are not detected in this pull request.

overriding, known failure in arm-compute playback test

/check-enforcer override


fix timestamp appending if basePath is not defined

Q
A

Is bugfix?
yes

New feature?
no

Breaks BC?
no

Tests pass?
yes

Fixed issues
#16268

Alike fix is in master now. https://github.com/yiisoft/yii2/commit/96cd8bcb687d66ae86fba01ac6a3e9c54687d616


feat(docs): move Theme-toggle to Navbar

Is your feature request related to a problem? Please describe.
The theme toggle is currently in the right sidebar when browsing docs, maybe we should move it to the navbar to be more standardized?

Describe the solution you'd like to see

Desribe alternate solutions

Additional information
No response

I kinda like it there its different.

navbar would be better imo

I made a PR with the change, to check how it looks

navbar would be better imo

I think after seeing it in @gabrielelpidio PR, I'm of the mind it looks better in the Nav bar as well.

it's in the navbar now, we can move it back if we don't like it. what do you think @nexxeln


Implement Thread deletion

It is now available on the Net API, so let's use it!

Fixed with https://github.com/textileio/js-threads/pull/51/


Add an option to quiesce ONNX tests

Description
Adds a flag to the pom file which quiesces the ONNX tests.
Motivation
The ONNX tests don't pass on Apple Silicon as I've only just added the support for it to the upstream release.

Yeah, though I've been considering making it a default part of the modules which use native code, as there will always be some CI environment they fail in.

I actually edited my original comment to add “think if we want to” for exactly this reason.


Add doctrine/commons 3.0 support

It was a bit more than just Compose changes. See #2159.
Bumping doctrine/annotations and removing the conflict is a good idea. 👍


[swift-format] Set exec path in initializer (NFC)

Rather than define SwiftFormatInvocation::setMainExecutablePath() and use it in the one spot where a SwiftFormatInvocation is initialized, have SwiftFormatInvocation take an executable path in its initializer. No functional change, but slightly fewer lines of code.
+cc @danielmartin

LGTM.

Thanks for the review! :)
@swift-ci please smoke test

⛵

Thanks!


Changes to container networking post -- service discovery, compose, m…

…ultiple NAT networks, transparent networking caveat.

LGTM


Add ONNX test cases

This PR pulls in all of the ONNX Backend Compliance tests and adds tests for some of the nodes. Backend tests ensure we are fully compliant with the ONNX standard. This will end up driving a few changes in Axon as well, but overall I think this will make importing/exporting a breeze with ONNX :)

Backend tests are programmatically generated with a script, but they don't populate the test cases locally, I'm trying to see if there's a way we can generate the test cases in the CI/CD so we don't have to keep them in the repository

Amazing!!!


Fix the link in dask documentation

Hi All,
I have found a page that would be similar to the missing link in #478. You could find it here.
Hope this is of any value.
Thanks
Michael

Looks like there are unrelated commits in this PR. You probably need to
fetch upstream, merge master, and push again.
On Sun, Mar 10, 2019 at 6:02 PM MichaelSchroter<EMAIL_ADDRESS>wrote:

Hi All,
I have found a page that would be similar to the missing link in #478
https://github.com/dask/dask-ml/issues/478. You could find it here
https://scikit-learn.org/0.15/modules/scaling_strategies.html.
Hope this is of any value.
Thanks
Michael
You can view, comment on, or merge this pull request online at:
https://github.com/dask/dask-ml/pull/479
Commit Summary

update dask-sphinx-theme (#361)
TST: Try numba RC (#363)
Update README.rst
RLS: 0.10.0
ENH: IncrementalSearch (#356)
Examples update (#369)
Add IncrementalSearch to api.rst (#371)
Adds sklearn version check for ColumnTransformer import (#374)
[skip ci] Change links to dask.org (#375)
Auto-rechunk input arrays (#377)
Fix CI issues (#382)
dask.pydata.org -> dask.org
Update joblib documentation for scikit-learn 0.20 (#387)
Bump sklearn min version to 0.20 (#392)
Support dataframes for k-means (#393)
Fixes ShuffleSplit random seed generation bug (#381)
Support dataframes in _partial.py::fit/predict (#395)
Don't use auto chunking with unknown chunk sizes (#398)
Pass models in predict rather than their methods (#400)
Allow compute=False in ParallelPostFit.score (#402)
API: rename IncrementalSearch => IncrementalSearchCV
API: rename history_results_, and format differently
API: add cv_results_
MAINT: allow _additional_calls to return multiple models
[MRG+1] Poly trans: Issue #347 (#367)
TST, MAINT: clean stopping on plateau (see notes below)
BUG: ∞ loop in IncrementalSearchCV if decay_rate=0
TST: perform basic search (decay_rate=0) in test_search_basic
MAINT: collapse _adapt and _stop_on_plateau into one function
Fix test formatting
Closes https://github.com/dask/dask-ml/issues/385 (#407)
DOC: update changelog
Use scipy to rank
Replace cv_results asserts with sanity checks
IncrementalSearch edge cases (#373)
Filter warning from dask dataframe concat (#408)
Merge branch 'search-api' into search-bug
TST: if passive, return highest scoring model else sanity checks
API: improve and clean IncrementalSearch API (#404)
DOC: IncrementalSearchCV (#405)
Merge remote-tracking branch 'upstream/master' into
stsievert-search-bug
Merge remote-tracking branch 'upstream/master' into whatsnew
lint
fix link
DOC: update changelog (#409)
failing test
maybe fix?
Merge pull request #411 from TomAugspurger/decay-loop
Merge remote-tracking branch 'upstream/master' into
stsievert-search-bug
run tests
Merge pull request #406 from stsievert/search-bug
doc fixup
Merge pull request #413 from TomAugspurger/doc-fixup
RLS: 0.11.0
typo [skip ci]
Merge pull request #414 from TomAugspurger/typo
Replace get with scheduler
Roll back changes to test_scheduler_param
try pinning CPython
fix warnings
Specify dask array chunksizes
Merge pull request #418 from jrbourbeau/fix_get_tests
Bug-Fix in Polynomial-Features
make transformer params more general
Minor LogisticRegression updates
Merge pull request #422 from jrbourbeau/logistic_reg_cleanup
Merge pull request #417 from datajanko/poly-bug-fix
API: Lazy score, predict for IncrementalSearchCV
lint
Add version number to conf.py
Use X.Y.Z version format
Merge pull request #426 from jrbourbeau/docs_version
Fix typo in docstring
Merge pull request #429 from rmsare/docs-split-typo
Bug: Handle a value not being passed for Y in euclidian distance.
RFC: Adjusts regex test
STY: Flake8
RFC: Adjusts for pytest 4.0.0
BUG: Changes order for make_column_transformer
format conf.py
ignore conf.py
RFC: Comment out version
RFC: Resolves FutureWarning
Merge pull request #431 from thomasjpfan/issue/427
allow_unknown_chunksizes=True in
dask_ml.compose.ColumnTransformer._hstack
Use high-level graphs
Include our dsk in the graphs
avoid assert_true
Changes for PR 437 comments.
Trigger CI
Merge pull request #439 from TomAugspurger/ci-fix
Merge remote-tracking branch 'upstream/master' into ZEFR-INC-master
black formatting
isort
removed dead code path, updated test to provide confirmation.
updated test with sklearn 1-D check
Merge pull request #437 from ZEFR-INC/master
COMPAT: fix warning
Merge pull request #441 from TomAugspurger/collections-warning
Merge remote-tracking branch 'upstream/master' into
asgersoerensen-patch-1
Added test for no Y
Merge remote-tracking branch 'upstream/master' into ppf-incsearchcv
lint
Merge pull request #430 from asgersoerensen/patch-1
Merge pull request #424 from TomAugspurger/ppf-incsearchcv
fix typo: model_selectoin -> model_selection (#442)
Update preprocessing.rst
Merge pull request #445 from teoguso/master
Updated joblib.rst
Decrease n_splits
Add SparseDtype case to test
Merge pull request #450 from jrbourbeau/fix-sklearn-dev
Merge pull request #449 from suamin/patch-1
Fix input to sharedict.merge
Merge pull request #455 from jrbourbeau/fix_partial_fit
Add graphviz to dev environment
Merge pull request #459 from jrbourbeau/add_graphviz
Add oldest_supported CircleCI job
Add conda list
Add package uninstalls
Update minimum versions
Fix pandas SparseDtype failures
"oldest supported" -> "earliest supported"
Remove defaults conda channel
Uninstall pypi numpy
Remove type comments
Update earliest scikit-learn from 0.20 to 0.20.0
MAINT: loud warning if ImportError with IncrementalSearchCV
TST: add circleci test for no distributed
Revert "TST: add circleci test for no distributed"
Pin NumPy version to avoid FutureWarnings in sklearn
Update test_column_transformer
TST: rework tests to standalone file
Typo
Change deprecated da.atop to da.blockwise
Revert trying to make distributed a soft dependency
MAINT: add distributed as a dependency
MAINT: require specific version
Change da.atop for da.blockwise in data.py
remove failing tests
Merge pull request #469 from TomAugspurger/test-fixup
Merge branch 'master' into import-incremental
Merge branch 'master' into patch-1
Merge pull request #466 from stsievert/import-incremental
Merge pull request #468 from jjerphan/patch-1
Merge remote-tracking branch 'upstream/master' into update_ci
Fix blockwise failing tests
Add defaults channel back
Move blockwise to _compat
Sort imports with isort v4.3.8
Merge pull request #461 from jrbourbeau/update_ci
Minor updates to .gitignore [skip ci]
Merge pull request #472 from jrbourbeau/update_gitignore
RLS: 0.12.0
Fix sklearn dev tests (#474)
Fix imports for isort 4.3.10 (#476)
update indexable() to just yield dask dataframes (issue #324) (#471)
Fix #378: high is out of bounds for int32 for k_means (#462)
Use scikit-learn nightly wheels (#477)

File Changes

M .circleci/config.yml
https://github.com/dask/dask-ml/pull/479/files#diff-0 (39)
M .gitignore
https://github.com/dask/dask-ml/pull/479/files#diff-1 (2)
M .travis.yml
https://github.com/dask/dask-ml/pull/479/files#diff-2 (1)
M README.rst
https://github.com/dask/dask-ml/pull/479/files#diff-3 (4)
M ci/environment-2.7.yml
https://github.com/dask/dask-ml/pull/479/files#diff-4 (11)
M ci/environment-3.6.yml
https://github.com/dask/dask-ml/pull/479/files#diff-5 (17)
M ci/install-circle.sh
https://github.com/dask/dask-ml/pull/479/files#diff-6 (1)
M dask_ml/init.py
https://github.com/dask/dask-ml/pull/479/files#diff-7 (2)
M dask_ml/_compat.py
https://github.com/dask/dask-ml/pull/479/files#diff-8 (15)
M dask_ml/_partial.py
https://github.com/dask/dask-ml/pull/479/files#diff-9 (20)
M dask_ml/_utils.py
https://github.com/dask/dask-ml/pull/479/files#diff-10 (16)
M dask_ml/cluster/init.py
https://github.com/dask/dask-ml/pull/479/files#diff-11 (2)
M dask_ml/cluster/k_means.py
https://github.com/dask/dask-ml/pull/479/files#diff-12 (18)
M dask_ml/cluster/spectral.py
https://github.com/dask/dask-ml/pull/479/files#diff-13 (6)
M dask_ml/compose/_column_transformer.py
https://github.com/dask/dask-ml/pull/479/files#diff-14 (24)
M dask_ml/decomposition/pca.py
https://github.com/dask/dask-ml/pull/479/files#diff-15 (11)
M dask_ml/decomposition/truncated_svd.py
https://github.com/dask/dask-ml/pull/479/files#diff-16 (2)
M dask_ml/feature_extraction/text.py
https://github.com/dask/dask-ml/pull/479/files#diff-17 (16)
M dask_ml/impute.py
https://github.com/dask/dask-ml/pull/479/files#diff-18 (10)
M dask_ml/linear_model/init.py
https://github.com/dask/dask-ml/pull/479/files#diff-19 (6)
M dask_ml/linear_model/glm.py
https://github.com/dask/dask-ml/pull/479/files#diff-20 (30)
M dask_ml/metrics/init.py
https://github.com/dask/dask-ml/pull/479/files#diff-21 (7)
M dask_ml/metrics/pairwise.py
https://github.com/dask/dask-ml/pull/479/files#diff-22 (22)
M dask_ml/metrics/regression.py
https://github.com/dask/dask-ml/pull/479/files#diff-23 (2)
M dask_ml/metrics/scorer.py
https://github.com/dask/dask-ml/pull/479/files#diff-24 (18)
M dask_ml/model_selection/init.py
https://github.com/dask/dask-ml/pull/479/files#diff-25 (13)
M dask_ml/model_selection/_incremental.py
https://github.com/dask/dask-ml/pull/479/files#diff-26 (575)
M dask_ml/model_selection/_search.py
https://github.com/dask/dask-ml/pull/479/files#diff-27 (140)
M dask_ml/model_selection/_split.py
https://github.com/dask/dask-ml/pull/479/files#diff-28 (18)
M dask_ml/model_selection/methods.py
https://github.com/dask/dask-ml/pull/479/files#diff-29 (3)
M dask_ml/model_selection/utils.py
https://github.com/dask/dask-ml/pull/479/files#diff-30 (53)
M dask_ml/model_selection/utils_test.py
https://github.com/dask/dask-ml/pull/479/files#diff-31 (8)
M dask_ml/naive_bayes.py
https://github.com/dask/dask-ml/pull/479/files#diff-32 (2)
M dask_ml/preprocessing/init.py
https://github.com/dask/dask-ml/pull/479/files#diff-33 (25)
M dask_ml/preprocessing/_encoders.py
https://github.com/dask/dask-ml/pull/479/files#diff-34 (28)
M dask_ml/preprocessing/data.py
https://github.com/dask/dask-ml/pull/479/files#diff-35 (114)
M dask_ml/preprocessing/label.py
https://github.com/dask/dask-ml/pull/479/files#diff-36 (1)
M dask_ml/wrappers.py
https://github.com/dask/dask-ml/pull/479/files#diff-37 (141)
M docs/source/changelog.rst
https://github.com/dask/dask-ml/pull/479/files#diff-38 (31)
M docs/source/conf.py
https://github.com/dask/dask-ml/pull/479/files#diff-39 (24)
M docs/source/examples.rst
https://github.com/dask/dask-ml/pull/479/files#diff-40 (35)
A docs/source/examples/.gitignore
https://github.com/dask/dask-ml/pull/479/files#diff-41 (0)
D docs/source/examples/tensorflow.ipynb
https://github.com/dask/dask-ml/pull/479/files#diff-42 (410)
D docs/source/examples/text-vectorization.ipynb
https://github.com/dask/dask-ml/pull/479/files#diff-43 (202)
M docs/source/hyper-parameter-search.rst
https://github.com/dask/dask-ml/pull/479/files#diff-44 (112)
M docs/source/incremental.rst
https://github.com/dask/dask-ml/pull/479/files#diff-45 (25)
M docs/source/index.rst
https://github.com/dask/dask-ml/pull/479/files#diff-46 (20)
M docs/source/joblib.rst
https://github.com/dask/dask-ml/pull/479/files#diff-47 (70)
M docs/source/modules/api.rst
https://github.com/dask/dask-ml/pull/479/files#diff-48 (18)
A
docs/source/modules/generted/dask_ml.compose.ColumnTransformer.rst
https://github.com/dask/dask-ml/pull/479/files#diff-49 (16)
A
docs/source/modules/generted/dask_ml.compose.make_column_transformer.rst
https://github.com/dask/dask-ml/pull/479/files#diff-50 (6)
M docs/source/preprocessing.rst
https://github.com/dask/dask-ml/pull/479/files#diff-51 (5)
M setup.cfg
https://github.com/dask/dask-ml/pull/479/files#diff-52 (6)
M setup.py https://github.com/dask/dask-ml/pull/479/files#diff-53
(7)
M tests/compose/test_column_transformer.py
https://github.com/dask/dask-ml/pull/479/files#diff-54 (152)
M tests/linear_model/test_stochastic_gradient.py
https://github.com/dask/dask-ml/pull/479/files#diff-55 (2)
M tests/metrics/test_metrics.py
https://github.com/dask/dask-ml/pull/479/files#diff-56 (25)
M tests/model_selection/dask_searchcv/test_model_selection.py
https://github.com/dask/dask-ml/pull/479/files#diff-57 (9)
M
tests/model_selection/dask_searchcv/test_model_selection_sklearn.py
https://github.com/dask/dask-ml/pull/479/files#diff-58 (62)
M tests/model_selection/test_incremental.py
https://github.com/dask/dask-ml/pull/479/files#diff-59 (290)
M tests/model_selection/test_split.py
https://github.com/dask/dask-ml/pull/479/files#diff-60 (26)
M tests/preprocessing/test_data.py
https://github.com/dask/dask-ml/pull/479/files#diff-61 (108)
M tests/preprocessing/test_encoders.py
https://github.com/dask/dask-ml/pull/479/files#diff-62 (36)
M tests/test_impute.py
https://github.com/dask/dask-ml/pull/479/files#diff-63 (20)
M tests/test_incremental.py
https://github.com/dask/dask-ml/pull/479/files#diff-64 (12)
M tests/test_kmeans.py
https://github.com/dask/dask-ml/pull/479/files#diff-65 (15)
M tests/test_parallel_post_fit.py
https://github.com/dask/dask-ml/pull/479/files#diff-66 (42)
M tests/test_partial.py
https://github.com/dask/dask-ml/pull/479/files#diff-67 (25)
M tests/test_pca.py
https://github.com/dask/dask-ml/pull/479/files#diff-68 (52)

Patch Links:

https://github.com/dask/dask-ml/pull/479.patch
https://github.com/dask/dask-ml/pull/479.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask-ml/pull/479, or mute the thread
https://github.com/notifications/unsubscribe-auth/ABQHIsGdUoeUomOwqctX4ASKBX2a5Ciwks5vVY8ggaJpZM4bnkAA
.

Superseded by #483


Can this be configured to work with other global cookie lawas like COPPA, e Privacy, POPPIA, with Geo-blocking, Do Not Track (DNT), etc?

Can this be configured to work with other global cookie lawas like COPPA, e Privacy, POPPIA, with Geo-blocking, Do Not Track (DNT), etc? Is there a way I can achieve these with this software? I need to configure global/ regional lwa, geo-blocking, dnt, COPPA, etc, to make my website truly global. How can I go about doing this?
Thanks for your help.

Geoblocking and region specificity requires development of a server-side component to lookup the IP address against a geocoding database, biscuitman is only a client-side solution which restricts and does not address this, but a developer could integrate with this by only serving the cookie notice depending on the IP.
There is a feature that allows you to autoconsent if the users timezone is European, but this is not as reliable as geocoding but is a bit of a hack.
The way you a developer could do this, on your server would be something like:

load geolite2 geocoding db into memory
check IP on first visit
if: location/region matches a cookie requirement, then include biscuitman formatted  trackers + biscuitman config + biscuitman js
else: include trackers normally, don't include biscuitman

Okay.  Tnx for your prompt response.  Is it possible to install it via Google Tag Manager and link from there to a custom javascript of my own, containing all those functions configured into it.
Kindly advise on this please.  Tnx

With biscuitman scenario, you wouldn't even load google tag manager on your site until you got consent because Google's code kindof does what wants. There is a consent feature with Google Analytics that can be set up, but its complicated, and this project sidesteps all that.
If you need geo-coded cookie consent, there is no shortcut, you must either get a developer to implement something like I suggested or pay for one of those cookie consent management platforms to handle it for you.
What software is your website using?

I have three websites I wish to implement it on, using sofwares, namely WordPress, phpbb and Modesy.
If this can't be implemented via gtm, how else can it be done? Through a direct implementation in the header script?

I would recommend, you slow down, stop spamming all these consent managers github issues and wasting peoples time, and just read the documentation to understand how these consent managers work and pick one that meets your needs. I've told you explicitly what you'd need to get what you want in my first response, server-side development.. geolite integration.

 Why would you say that I am spamming? Please, I would advise that you moderate your use of words. Don't I have the right to ask questions about anything that catches my interest? That I asked a question should not automatically subordinate me to you or subject me to your insult!!

I told you explicitly how to implement what you wanted. You are welcome. Then you ask more questions without even acknowledging my answer, as if you didn't read it. This combined with your copy paste responses on every solution manager on github (when the answers are literally in the READMEs) tells me that you expect other people to solve the problem for you, and that is what prompted my comment. Nothing personal.


Can I use your code to enhance the quality of the compressed images?

Please I have images compressed in JPEG with different quality can I use your code to enhance the quality of the compressed images? if yes, is there any steps to do that please?

Unfortunately, we do not support restoration for JPEG :(


Link to Discord is broken

The link to Discord in Fractal's left nav bar currently points to a dead link. Fractal's Discord has been deprecated, so we should either:

remove this link entirely
direct to Decent's Discord server

My take is that we remove it for now as a bandaid to remove a top-level broken link in the app, and re-add it back when our Discord server has been revamped and we have a better user onboarding flow.

Closing, fixed.


Add process signatures to some integrations

What does this PR do?
Add process signatures to some integrations' manifest.json file.
Motivation
Use this new field in the configuration to build a process matching library that would allow to detect integrations from a process command line
Review checklist

[ ] PR has a meaningful title or PR has the no-changelog label attached
[ ] Feature or bugfix has tests
[ ] Git history is clean
[ ] If PR impacts documentation, docs team has been notified or an issue has been opened on the documentation repo

Additional Notes
Anything else we should know when reviewing?

Thanks for the reviews ! Just added the new field to the documentation and updated couchbase signature

oh sorry i missed it

@DataDog/baklava I think @sfluor is waiting on your approval :)


How about superclass_string has two token ids?

Hi Author,
If the superclass_string has two token ids, should we still get the only first token id?
ids = tokenize([superclass_string])**[0]**
 assert ids.shape[-1] == 1, f'your superclass concept string must map exactly one token id'
 superclass_embed_id = ids**[0].**item()

Like the index with bold text, should we still get only the first token id? Thanks

yeah, so that may be the drawback to this technique for now
you need superclass concepts that are only one token

maybe for the follow up paper, if this technique holds up

Thank you！


join statement causes panic

Describe the bug
offset overflow may occur when I perform an inner join
To Reproduce
❯ CREATE EXTERNAL TABLE sea STORED AS PARQUET LOCATION 'file:///tmp/data/sea/';
0 rows in set. Query took 0.009 seconds.

❯ CREATE EXTERNAL TABLE air STORED AS PARQUET LOCATION 'file:///tmp/data/air/';
0 rows in set. Query took 0.003 seconds.

❯ select count(1) from sea;
+-----------------+
| COUNT(Int64(1)) |
+-----------------+
| 56642           |
+-----------------+
1 row in set. Query took 0.010 seconds.

❯ select count(1) from air;
+-----------------+
| COUNT(Int64(1)) |
+-----------------+
| 56642           |
+-----------------+
1 row in set. Query took 0.003 seconds.

❯ SELECT * FROM air INNER JOIN sea ON sea.station = air.station;
thread 'tokio-runtime-worker' panicked at 'offset overflow', /Users/yukkit/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-select-48.0.0/src/take.rs:424:41
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'tokio-runtime-worker' panicked at 'offset overflow', /Users/yukkit/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-select-48.0.0/src/take.rs:424:41

The reason is that there is an overflow during type conversion.
https://github.com/apache/arrow-rs/blob/48.0.0/arrow-select/src/take.rs#L424
Expected behavior
No response
Additional context
git commit: 4881b5db3
data.zip
I don't have time to deal with this problem at the moment, can anyone help solve it

please assign to me

Thanks @guojidan .    @alamb Could you please assign it?

Thank you @guojidan  and thank you @yukkit  for a nice reproducer.
FWIW we have seen this error when trying to create String arrays with "very large" strings
Basically since a StringArray uses i32 as an offset, it means the maximum amount of string data that any particular StringValue can have is 2GB
So if somehow your query output tries to make a column with more than 2GB of string data it will panic
In this case the SELECT * FROM air INNER JOIN sea ON sea.station = air.station; join is many/many (so the output cardinality is going to be very large)
I ran this reproducer locally and got a stack trace
$ RUST_BACKTRACE=1 datafusion-cli -c 'SELECT * FROM air INNER JOIN sea ON sea.station = air.station;'
DataFusion CLI v32.0.0
thread 'tokio-runtime-worker' panicked at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-select-48.0.0/src/take.rs:424:41:
offset overflow
stack backtrace:
   0: rust_begin_unwind
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/std/src/panicking.rs:595:5
   1: core::panicking::panic_fmt
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/panicking.rs:67:14
   2: core::panicking::panic_display
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/panicking.rs:150:5
   3: core::panicking::panic_str
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/panicking.rs:134:5
   4: core::option::expect_failed
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/option.rs:1988:5
   5: core::option::Option<T>::expect
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/option.rs:898:21
   6: arrow_select::take::take_bytes
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-select-48.0.0/src/take.rs:424:41
   7: arrow_select::take::take_impl
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-select-48.0.0/src/take.rs:142:25
   8: arrow_select::take::take
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-select-48.0.0/src/take.rs:92:5
   9: datafusion_physical_plan::joins::hash_join::equal_rows_arr
             at /Users/alamb/Software/arrow-datafusion/datafusion/physical-plan/src/joins/hash_join.rs:876:20
  10: datafusion_physical_plan::joins::hash_join::build_equal_condition_join_indices
             at /Users/alamb/Software/arrow-datafusion/datafusion/physical-plan/src/joins/hash_join.rs:840:5
  11: datafusion_physical_plan::joins::hash_join::HashJoinStream::poll_next_impl::{{closure}}
             at /Users/alamb/Software/arrow-datafusion/datafusion/physical-plan/src/joins/hash_join.rs:955:46
  12: core::task::poll::Poll<T>::map
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/task/poll.rs:53:43
  13: datafusion_physical_plan::joins::hash_join::HashJoinStream::poll_next_impl
             at /Users/alamb/Software/arrow-datafusion/datafusion/physical-plan/src/joins/hash_join.rs:947:14
  14: <datafusion_physical_plan::joins::hash_join::HashJoinStream as futures_core::stream::Stream>::poll_next
             at /Users/alamb/Software/arrow-datafusion/datafusion/physical-plan/src/joins/hash_join.rs:1056:9
  15: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  16: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  17: datafusion_physical_plan::coalesce_batches::CoalesceBatchesStream::poll_next_inner
             at /Users/alamb/Software/arrow-datafusion/datafusion/physical-plan/src/coalesce_batches.rs:226:42
  18: <datafusion_physical_plan::coalesce_batches::CoalesceBatchesStream as futures_core::stream::Stream>::poll_next
             at /Users/alamb/Software/arrow-datafusion/datafusion/physical-plan/src/coalesce_batches.rs:204:20
  19: <core::pin::Pin<P> as futures_core::stream::Stream>::poll_next
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.28/src/stream.rs:120:9
  20: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1632:9
  21: <futures_util::stream::stream::next::Next<St> as core::future::future::Future>::poll
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/next.rs:32:21
  22: datafusion_physical_plan::stream::RecordBatchReceiverStreamBuilder::run_input::{{closure}}
             at /Users/alamb/Software/arrow-datafusion/datafusion/physical-plan/src/stream.rs:242:50
  23: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/core.rs:328:17
  24: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/loom/std/unsafe_cell.rs:16:9
  25: tokio::runtime::task::core::Core<T,S>::poll
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/core.rs:317:13
  26: tokio::runtime::task::harness::poll_future::{{closure}}
             at /Users/alamb/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.33.0/src/runtime/task/harness.rs:485:19
  27: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/panic/unwind_safe.rs:271:9

Perhaps
https://github.com/apache/arrow-datafusion/blob/a9d66e2b492843c2fb335a7dfe27fed073629b09/datafusion/physical-plan/src/joins/hash_join.rs#L725
is trying to make a single giant array batch with a single call to take 🤔

Another (a bit artificial, intentionally generating a Cartesian product) MRE:
❯ CREATE EXTERNAL TABLE test STORED AS CSV LOCATION 'file:///tmp/test-21mb.csv';
0 rows in set. Query took 0.105 seconds.

❯ SELECT repeat(lf1.column_2, 15) FROM test lf1 join test lf2 on lf1.column_1 = lf2.column_1 join test lf3 on lf2.column_1 = lf3.column_1 join test lf4 on lf3.column_1 = lf4.column_1
;
thread 'tokio-runtime-worker' panicked at /Users/brew/Library/Caches/Homebrew/cargo_cache/registry/src/index.crates.io-6f17d22bba15001f/arrow-select-49.0.0/src/take.rs:423:41:
offset overflow

stack backtrace
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::option::expect_failed
   3: arrow_select::take::take_bytes
   4: arrow_select::take::take_impl
   5: arrow_select::take::take
   6: datafusion_physical_plan::joins::utils::build_batch_from_indices
   7: <datafusion_physical_plan::joins::hash_join::HashJoinStream as futures_core::stream::Stream>::poll_next
   8: <datafusion_physical_plan::coalesce_batches::CoalesceBatchesStream as futures_core::stream::Stream>::poll_next
   9: <datafusion_physical_plan::projection::ProjectionStream as futures_core::stream::Stream>::poll_next
  10: datafusion_physical_plan::stream::RecordBatchReceiverStreamBuilder::run_input::{{closure}}
  11: tokio::runtime::task::core::Core<T,S>::poll
  12: tokio::runtime::task::harness::Harness<T,S>::poll
  13: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  14: tokio::runtime::scheduler::multi_thread::worker::Context::run
  15: tokio::runtime::context::runtime::enter_runtime
  16: tokio::runtime::scheduler::multi_thread::worker::run
  17: tokio::runtime::task::core::Core<T,S>::poll
  18: tokio::runtime::task::harness::Harness<T,S>::poll
  19: tokio::runtime::blocking::pool::Inner::run
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

query output tries to make a column with more than 2GB

In fact, it's not the case for this example, the query result does not generate such large data (but intermediate results can be really big - I mean row, not a column)
Not sure if it may help though
test file: test-21mb.csv

This may be related to https://github.com/apache/arrow-datafusion/issues/7848 (which makes giant single output RecordBatches for certain joins) that @korowa  has been working on
We just merged https://github.com/apache/arrow-datafusion/pull/8658 which is a significant step towards avoiding this kind of error so I am hopeful the situation on this PR will improve shortly

I suppose this issue may be closed -- the reproducer based on data files from the description

Reproducer code
use datafusion::error::Result;
use datafusion::prelude::*;

#[tokio::main]
async fn main() -> Result<()> {
  let ctx = SessionContext::new();

  ctx.register_parquet(
      "sea",
      "/home/eduard/Downloads/sea",
      ParquetReadOptions::default(),
  )
  .await?;

  ctx.register_parquet(
      "air",
      "/home/eduard/Downloads/air",
      ParquetReadOptions::default(),
  )
  .await?;

  let sql = r##"
  SELECT count(1) FROM air INNER JOIN sea ON sea.station = air.station;
  "##;

  let df = ctx.sql(sql).await?;
  df.show().await?;

  Ok(())
}

is able to complete with current main version(https://github.com/apache/arrow-datafusion/commit/840499fbd3c25bf82a6ee0641ef6888079e096da), while running it with 35.0.0 / 34.0.0 ends up with OOM

Nice work @korowa (again)

I can conform it seems to work fine, I had another scenario with the same issue and it works against main branch.
Thank you @korowa!


AI-900-AIFundamentals/instructions/02a-create-regression-model - Incorrect Python code

AI-900-AIFundamentals/instructions/02a-create-regression-model.html
Description of issue: Python incorrect on Create and run an inference pipeline Step 9 is incorrect.  Code should have an indented return:
import pandas as pd

def azureml_main(dataframe1=None, dataframe2=None):
    scored_results = dataframe1[['Scored Labels']]
    scored_results.rename(columns={'Scored Labels': 'predicted_price'}, inplace=True)
    return scored_results

I've already left a comment on the commit where the change was done. Tagging: @sherzyang

Hello @OscarValerock
Thanks for flagging. I created a PR with a fix for this issue.
Regards,
Mihai

Thanks @Mihai-Ac, @OscarValerock, I've merged the fix.