[WIP] Path to upgrade to 2.13 (RC2)

I suggest we keep this open to track dependencies that can get us to RC2, or to any scala past 2.13.0-M5.
Circe, doobie, tsec, and http4s are the hangups again.
Reference PRs:
#180 #142
io.circe#circe-generic_2.13.0-RC2;0.12.0-M1: not found
io.circe#circe-literal_2.13.0-RC2;0.12.0-M1: not found
io.circe#circe-generic-extras_2.13.0-RC2;0.12.0-M1: not found
io.circe#circe-parser_2.13.0-RC2;0.12.0-M1: not found
io.circe#circe-java8_2.13.0-RC2;0.12.0-M1: not found
io.circe#circe-config_2.13.0-RC2;0.6.1: not found
org.tpolecat#doobie-core_2.13.0-RC2;0.7.0: not found
org.tpolecat#doobie-h2_2.13.0-RC2;0.7.0: not found
org.tpolecat#doobie-scalatest_2.13.0-RC2;0.7.0: not found
org.tpolecat#doobie-hikari_2.13.0-RC2;0.7.0: not found
com.beachape#enumeratum-circe_2.13.0-RC2;1.5.21: not found
org.http4s#http4s-blaze-server_2.13.0-RC2;0.20.1: not found
org.http4s#http4s-circe_2.13.0-RC2;0.20.1: not found
org.http4s#http4s-dsl_2.13.0-RC2;0.20.1: not found
org.http4s#http4s-blaze-client_2.13.0-RC2;0.20.1: not found
io.github.jmcardon#tsec-common_2.13.0-RC2;0.1.0: not found
io.github.jmcardon#tsec-password_2.13.0-RC2;0.1.0: not found
io.github.jmcardon#tsec-mac_2.13.0-RC2;0.1.0: not found
io.github.jmcardon#tsec-signatures_2.13.0-RC2;0.1.0: not found
io.github.jmcardon#tsec-jwt-mac_2.13.0-RC2;0.1.0: not found
io.github.jmcardon#tsec-jwt-sig_2.13.0-RC2;0.1.0: not found
io.github.jmcardon#tsec-http4s_2.13.0-RC2;0.1.0: not found

@zakpatterson nice!


TEP-0101 PodTemplate should support environment variables.

Changes

Envs i.e. environment variables specified in podTemplate for PipelineRun and TaskRun, will take precdence over envs defined in step and StepTemplate.
config-defaults' default-pod-template can contain global default envs applicable for all PipelineRun and taskRun. These are overriden by user defined podTemplate.
Support forbidden envs.
a. A new config-default configuration: forbidden-envs
b. Envs added in the forbidden-envs are filtered while updating from a podTemplate.

Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[x] Has Docs included if any changes are user facing
[x] Has Tests included if any functionality added or changed
[x] Follows the commit message standard
[ ] Meets the Tekton contributor standards (including
functionality, content, code)
[x] Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
[x] Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
[x] Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes
PodTemplate can be used to update environment variables globally as well as PipelineRun and TaskRun level.

/kind tep

/retest

/cc @afrittoli

Hi @JeromeJu, @vdemeester and @afrittoli  Running codegen after updating to latest main branch adds a whole bunch of stuff not related to this commit.

/kind feature

opened #5764 to fix codegen out of sync issue.

Thank you @XinruZhang ! for taking a look, please take a look again

/test .*

/test .*

Thank you @ScrapCodes, looks great, please update the release notes to include a short description for default-forbidden-envs

/hold

It looks like the TEP for this feature is not implementable

It looks like the TEP for this feature is not implementable

arf.. we need to mark it as implementable 😅

It looks like the TEP for this feature is not implementable

Good catch @lbernick , thank you. It is implementable now

/hold cancel

Hi @chitrangpatel, do you think - we can add the webhook validations in a separate pull request?

Hi @chitrangpatel, do you think - we can add the webhook validations in a separate pull request?

Yes, that's completely fine.

Thank you @chitrangpatel !

@chitrangpatel ok to merge this? @ScrapCodes implemented both v1beta1 and v1 validation.

@jerop this should be ready to merge and be included in v0.43

@chitrangpatel ok to merge this? @ScrapCodes implemented both v1beta1 and v1 validation.

Yes! All good!

/lgtm


[chore] Remove docfx as unnecessary

Fixes #987
Changes
Docfx was introduced in the spec repo a while ago - https://github.com/open-telemetry/opentelemetry-specification/pull/742
with the intention to detect broken links.
Docfx v3 we used since https://github.com/open-telemetry/opentelemetry-specification/pull/2285 was in preview and is no longer accessible and not recommended to be used even if built from sources.
Docfx is a site-generator, not a linting tool, there is no (?) documentation on checks against docs it does.
We already have individual markdown lint and link checks.
With all of this I suggest removing docfx since it seems to be unnecessary.
Merge requirement checklist

[x] CONTRIBUTING.md guidelines followed.
[x] Change log entry added, according to the guidelines in When to add a changelog entry.

If your PR does not need a change log, start the PR title with [chore]

PS: I'll send a similar PR to the spec.

/cc @reyang in case you have any context on docfx checks we might still need

/cc @reyang in case you have any context on docfx checks we might still need

IIRC, relative links in markdown is verified by docfx, probably something that can be covered or already covered by markdownlint.

IIRC, relative links in markdown is verified by docfx, probably something that can be covered or already covered by markdownlint.

thanks! let me check

thanks @reyang for the pointer, neither markdownlint nor markdown-link-check check fragment part in a different file ([see foo](./bar#baz)) and there seem to be no other common checker for it.
I'll try to resurrect docfx v2 then

FYI - https://github.com/theoludwig/markdownlint-rule-relative-links seems promising (although the download number seems low). Probably give it a try as it seems to be blessed by the owner of markdownlint https://github.com/DavidAnson/markdownlint/issues/586#issuecomment-1465328977.

you sent me to the rabbit hole @reyang  😅
apparently markdownlint-rule-relative-links does not support absolute links (e.g. /docs/http/http-spans.md#something). It does support validation for ./docs/http/http-spans.md#something.
If my JavaScript was any good, I'd consider contributing something.
The TL;DR:
docfx v2 does not support checking anchors in other files (only within the same file), which is already covered by markdownlint.
I'll merge this PR and will create an issue to figure out how to do cross-file anchor checks.

thanks @reyang for the pointer, neither markdownlint nor markdown-link-check check fragment part in a different file ([see foo](./bar#baz)) and there seem to be no other common checker for it. I'll try to resurrect docfx v2 then

in Java, we're pinning an older version of markdown-link-check (v3.11.2) before anchor link checking broke, looks like a couple other otel repos have linked to that markdown-link-check issue too 😅

thanks @reyang for the pointer, neither markdownlint nor markdown-link-check check fragment part in a different file ([see foo](./bar#baz)) and there seem to be no other common checker for it. I'll try to resurrect docfx v2 then

in Java, we're pinning an older version of markdown-link-check (v3.11.2) before anchor link checking broke, looks like a couple other otel repos have linked to that markdown-link-check issue too 😅

cool, I'll check and send the PR - I have some other changes for that thing staged.

@trask I checked, it's a slightly different problem. The thing we're missing is #1009 and markdown-link-check never supported it :(

oh no, I thought we (Java) were getting cross-file anchor link checking, but did a quick test and we're not 😢


Tor and Cloudflare

I would like to share my communication with Cloudflare's support in relation to problems with accessing sites using Cloudflare through Tor proxy.
There is definitely something which Tor Browser can do and which (ungoogled) Chromium cannot which results in the problems discussed below. I don't know what it is but I hope ungoogled-chromium will be able to provide a solution as CF's "protection" system right now is pretty much denying the possibility of using Tor + ungoogled-chromium for CF sites.
====================
From: **** <>
To: Cloudflare <>
Subject: Using Tor without Tor Browser
Hi,
I am facing the following problem:
I have tor systemd service installed and running and browser (Chromium) set to use the Tor Socks proxy. I also have separately Tor Browser Bundle (TBB).
Opening sites which use Cloudlflare often present the well known captcha challenge of Clouldflare. But that does not happen in TBB. All this has nothing to do with a particular Tor circuit - it happens always: all circuits work in TBB, no circuit works in any other HTTP client.
I even tried to mimic TBB by copying literally the HTTP request headers in a curl script (attached) but even that didn't work - the curl script also hits the captcha. You can test it with https://cloudflare.com as an argument.
Why is this happening and what can I do so that I can open Cloudflare'd sites with a curl script or with a browser different from TBB?
Please when answering consider that I don't want to enable JavaScript or to reveal my IP address to Google through recaptcha as it contradicts the essential reason for using Tor. Neither I am willing to install any extension to bypass this.
#!/bin/bash

if [ -z "$1" ]; then
  echo "Supply URL as argument"
  exit 0
fi

# Scheme: "http://user:pw@sub.example.com:80/"
host=$(echo "${1}" | sed -e "s/[^/]*\/\/\([^@]*@\)\?\([^:/]*\).*/\2/")

curl "${1}" \
-H "Host: ${host}" \
-H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'DNT: 1' \
-H 'Connection: keep-alive' \
-H 'Upgrade-Insecure-Requests: 1' \
-L \
--proto-redir -all,http,https \
--compressed \
--proxy socks5h://<IP_ADDRESS>:9050/

====================
****, Apr 7, 8:18 PM PDT
Hi ****,
Thanks for contacting Cloudflare support.
You may refer to the following article for existing possible solution to proceed with,
Does Cloudflare block Tor?
Cheers,
******** | Cloudflare Support Engineer
====================
From: **** <>
To: Cloudflare <>
Subject: Re: [Cloudflare Support] **** - Using Tor without Tor Browser
Hi ****,
That article has nothing to do with my actual question. I am not asking how to tune my own website but how not to hit the captcha with an HTTP client different from TBB (as TBB doesn't hit it).
Could you please answer that?
====================
****, Apr 9, 8:11 PM PDT
Hi there,
In that case we will need more details on the exact steps for us to reproduce the issue faced, in case issue persist, please share with us:

what is the expected behavior
how does that differ from what you observe now
FULL steps to reproduce - e.g. what exactly has been run for you to test the behavior

Please respond with that information if you run into the issue again so we can continue to work with you to resolve your issue.
Best regards,
******** | Cloudflare Support Engineer
====================
From: **** <>
To: Cloudflare <>
Subject: Re: [Cloudflare Support] **** - Using Tor without Tor Browser
Hi ****,
I am sharing the full STR, expected and actual behavior below. Please kindly pay equal attention to both (A) and (B).
STR:

Install tor service

I am using tor-<IP_ADDRESS>-lp150.22.1.x86_64

Use the following /etc/tor/torrc

SocksPolicy accept <IP_ADDRESS>
SocksPolicy reject *
ControlPort 9051
HashedControlPassword ████

To generate HashedControlPassword use command:
$ tor --hash-password PASSWORD

Replace ████ with the output of the command in /etc/torrc

Create the following bash script and name it tornc. It requests new tor circuit:

#!/bin/bash
password='PASSWORD' # the same password used above

cat <<EOF | nc <IP_ADDRESS> 9051
AUTHENTICATE "${password}"
SIGNAL NEWNYM
quit
EOF

Run as root:

# torctl restart

Create the following bash script (shared in previous reply) and name it tor-curl:

#!/bin/bash

if [ -z "$1" ]; then
  echo "Supply URL as argument"
  exit 0
fi

# Scheme: "http://user:pw@sub.example.com:80/"
host=$(echo "${1}" | sed -e "s/[^/]*\/\/\([^@]*@\)\?\([^:/]*\).*/\2/")

curl -s "${1}" \
-H "Host: ${host}" \
-H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'Accept-Encoding: gzip, deflate' \
-H 'DNT: 1' \
-H 'Connection: keep-alive' \
-H 'Upgrade-Insecure-Requests: 1' \
-L \
--proto-redir -all,http,https \
--compressed \
--proxy socks5h://<IP_ADDRESS>:9050/

Run:

$ tornc
$ ./tor-curl https://ipinfo.io/ip
$ ./tor-curl https://cloudflare.com | grep '<****>'

Repeat 5

EXPECTED:
(A) tor-curl should be able to access the actual web page because it uses HTTP headers similar to those of Tor Browser (which can open the page)
(B) A security system using a challenge to protect the website should be GDPR compliant
ACTUAL:
(A) For every circuit without exception tor-curl hits the captcha and this happens for any website which uses this Cloudlflare feature (i.e. not only with https://cloudflare.com):
$ tornc
250 OK
250 OK
250 closing connection
$ ./tor-curl https://ipinfo.io/ip
<IP_ADDRESS>
$ ./tor-curl https://cloudflare.com | grep '<****>'
<****>
$ tornc
250 OK
250 OK
250 closing connection
$ ./tor-curl https://ipinfo.io/ip
<IP_ADDRESS>
$ ./tor-curl https://cloudflare.com | grep '<****>'
<****>

As you see: there is nothing malicious in using a script like tor-curl but Cloudflare is blocking ("challenging") it. Tor Browser does open the sites using Clouldflare though (and I don't know why).
(B) The current protection system of Cloudflare is non-GDPR compliant because:

It doesn't follow the principle of data minimisation:
Article 5(1)(c) https://gdpr-info.eu/art-5-gdpr/
"1. Personal data shall be:
(c) adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed (‘data minimisation’);"
The purpose here is to protect a particular website from malicious activity. There is nothing relevant in sharing the fact of the visit with a third party (Google) or in otherwise reducing user's privacy (through enforcing JavaScript). So this doesn't minimize the data, on the contrary. To make clear: IP address is personal data, especially in combination with HTTP user agent. So every HTTP request should be processed in a GDPR compliant way.

The current challenge system contradicts also Article 6(1)(f) of the GDPR because the "legitimate interest" of the controller is overriding the fundamental right of the data subject, one of which is the right to data protection (defined in Article 8 of the Charter of fundamental rights of the EU). The current challenge system doesn't even give a possibility for consent to data processing beyond what is technically necessary. With JS enabled and without extensions blocking 3rd party requests (e.g. uMatrix) it just instantly shares personal data with Google. The web browser extension of Cloudflare does not resolve the issue because even the very process of installing an extension is additional data processing - totally unnecessary and an additional open door to various vulnerabilities.

A secure, privacy respecting and GDPR compliant challenge system would be simple, secure and privacy respecting. For example: A simple image or text/ASCII-art based captcha without 3rd party HTTP requests, cookies or JavaScript.
====================
****, Apr 11, 9:15 PM PDT
Hi ****,
You mentioned For every circuit without exception tor-curl hits the captcha, can you share with us the rayID you have observed in plain text as well as the zone name where you were running your tests to, so we can help look further at your firewall event logs from there too.
Best regards,
******** | Cloudflare Support Engineer
====================
From: **** <>
To: Cloudflare <>
Subject: Re: [Cloudflare Support] **** - Using Tor without Tor Browser

can you share with us the rayID you have observed in plain text

Here are a few Ray IDs. For full command output including IP addresses check the attached file.
4c645a9a3c9358df
4c645ae56d639de5
4c645b317e739cbf
4c645b7d1cfa5ae5
4c645bc30dd3841e
4c645c0d6aa7c67d
4c645c56ad409499
4c645ccb3f11c859
4c645d107e20bf89
4c645d74b90dd113

as well as the zone name where you were running your tests to,

I have given you full STR. I don't know what "zone name" means. Please clarify.

so we can help look further at your firewall event logs from there too.

This is not about anything mine (firewall/sites) but a general issue with all sites which use Cloudflare's challenge mechanism.
$ tornc > /dev/null 2>&1; wait; tor-curl https://cloudflare.com | grep -E 'Your IP|Ray ID' | sed -r 's/<[^>]+>//g'
    Cloudflare Ray ID: 4c645a9a3c9358df
    Your IP: <IP_ADDRESS>
$ tornc > /dev/null 2>&1; wait; tor-curl https://cloudflare.com | grep -E 'Your IP|Ray ID' | sed -r 's/<[^>]+>//g'
    Cloudflare Ray ID: 4c645ae56d639de5
    Your IP: <IP_ADDRESS>
$ tornc > /dev/null 2>&1; wait; tor-curl https://cloudflare.com | grep -E 'Your IP|Ray ID' | sed -r 's/<[^>]+>//g'
    Cloudflare Ray ID: 4c645b317e739cbf
    Your IP: <IP_ADDRESS>
$ tornc > /dev/null 2>&1; wait; tor-curl https://cloudflare.com | grep -E 'Your IP|Ray ID' | sed -r 's/<[^>]+>//g'
    Cloudflare Ray ID: 4c645b7d1cfa5ae5
    Your IP: <IP_ADDRESS>
$ tornc > /dev/null 2>&1; wait; tor-curl https://cloudflare.com | grep -E 'Your IP|Ray ID' | sed -r 's/<[^>]+>//g'
    Cloudflare Ray ID: 4c645bc30dd3841e
    Your IP: <IP_ADDRESS>
$ tornc > /dev/null 2>&1; wait; tor-curl https://cloudflare.com | grep -E 'Your IP|Ray ID' | sed -r 's/<[^>]+>//g'
    Cloudflare Ray ID: 4c645c0d6aa7c67d
    Your IP: <IP_ADDRESS>
$ tornc > /dev/null 2>&1; wait; tor-curl https://cloudflare.com | grep -E 'Your IP|Ray ID' | sed -r 's/<[^>]+>//g'
    Cloudflare Ray ID: 4c645c56ad409499
    Your IP: <IP_ADDRESS>
$ tornc > /dev/null 2>&1; wait; tor-curl https://cloudflare.com | grep -E 'Your IP|Ray ID' | sed -r 's/<[^>]+>//g'
    Cloudflare Ray ID: 4c645ccb3f11c859
    Your IP: <IP_ADDRESS>
$ tornc > /dev/null 2>&1; wait; tor-curl https://cloudflare.com | grep -E 'Your IP|Ray ID' | sed -r 's/<[^>]+>//g'
    Cloudflare Ray ID: 4c645d107e20bf89
    Your IP: <IP_ADDRESS>
$ tornc > /dev/null 2>&1; wait; tor-curl https://cloudflare.com | grep -E 'Your IP|Ray ID' | sed -r 's/<[^>]+>//g'
    Cloudflare Ray ID: 4c645d74b90dd113
    Your IP: <IP_ADDRESS>

====================
****, Apr 14, 8:32 PM PDT
Hi ****,
Zone name is referring to the domain you are running the test to, I understand that you believe this is a wider issue, but we can only discuss with you if you are running test to your own zone(s) and encountering issue, as we can only discuss zone settings with owners.
I would suggest to share with us the rayID that is associated with a zone you own, from there, we can verify the issue and run tests based on your own setting and move on from there
Best regards,
******** | Cloudflare Support Engineer
====================
From: **** <>
To: Cloudflare <>
Subject: Re: [Cloudflare Support] **** - Using Tor without Tor Browser
On Mon, 15 Apr 2019 03:32:31 +0000 Cloudflare wrote:

I understand that you believe this is a wider issue, but we can only discuss with you if you are running test to your own zone(s) and encountering issue, as we can only discuss zone settings with owners.

I don't believe - I am showing facts. With the full STR you can reproduce the issue for yourself, with your own zone or with any other zone which has this "challenge" for Tor.
I don't see why you should ignore a wider issue yet suggest assistance for a limited manifestation of it.
====================
****, Apr 15, 7:59 PM PDT
Hi ****,
All rayID is associated with a specific zone, please let us know which zone you are testing with, that will help us with our investigation with the teams.
On the other hand, if these rayID are from your own zone, it can allow us to communicate with you down the road as our goal is to ensure the security and privacy of all our customers, including yourself. We will only be able to discuss in details with the zone owner, and having verify that will definitely allow us to communicate more freely upon checking with our teams.
Best regards,
******** | Cloudflare Support Engineer
====================
From: **** <>
To: Cloudflare <>
Subject: Re: [Cloudflare Support] **** - Using Tor without Tor Browser
On Tue, 16 Apr 2019 02:59:06 +0000 Cloudflare wrote:

All rayID is associated with a specific zone, please let us know which zone you are testing with, that will help us with our investigation with the teams.

I have given you complete STR and the result of it. You keep asking for
additional info - there is no such info.

On the other hand, if these rayID are from your own zone, it can allow us to communicate with you down the road as our goal is to ensure the security and privacy of all our customers, including yourself.

There is neither security, nor privacy in a captcha which shares user's IP address (and other details) with Google. I have explained that.

We will only be able to discuss in details with the zone owner, and having verify that will definitely allow us to communicate more freely upon checking with our teams.

The shared STR show that the "zone owner" is you - cloudflare.com. So you have all details to investigate and fix the issue which applies to every site using this captcha system.
====================
****, Apr 16, 5:32 PM PDT
Hi ****,
I think we're just going in circle here and I should have better explain why we would need to focus on your own zone and check from there.
The key is, Cloudflare employees can not remove the captcha. The captcha is showing specifically because of security settings the site owner has turned on, due to whatever setting they might have on their zone, may that be cloudflare.com or any other zones behind Cloudflare.
And as shared, we will not be able to discuss what is the current setting of any zone that is not yours.
To share with you the generic information, a site visitor to a website that Cloudflare is protecting can see a captcha for the following reasons:

The IP address you are on has shown problematic activity online recently in one of our data sources. If you would like to look your IP up, then please look your IP up at Project Honeypot. If the IP address shows data for malicious activity, you can see why there. You can also attempt to whitelist your IP directly on that page by connecting from that IP.

The site owner decided to block the country you are visiting from.

Your actions are triggering a Web Application Firewall (WAF) rule that the website owner has turned on.

What to do:

You can still enter the site by passing the captcha. You can also request to the site owner that the whitelist your IP address. You must have cookies and JavaScript enabled in your browser settings to pass the captcha. If you are running any anti-virus or firewall programs on your computer, then you may also need to check to make sure that they aren't blocking access to the captcha image.

If your IP is showing that you are distributing malware or spam, your computer may have an infection and we would recommend running an anti-virus scan.

Having said that, we will not be able to change the setting of any zone, nor the infrastructure of Cloudflare WAF rule for any specific visitor.
However, if this is your own zone, and you believe you have already got settings that should allow TBB (or any other TOR user) to bypass any challenge but your visitors are still observing difficulties, then, we can make use of the test rayID from your zone, and look further into your firewall setting.
Useful resource

Does Cloudflare block Tor?

Why am I getting security captcha or challenge for normal administration posts/submits?

Best regards,
******** | Cloudflare Support Engineer
====================
From: **** <>
To: Cloudflare <>
Subject: Re: [Cloudflare Support] **** - Using Tor without Tor Browser
On Wed, 17 Apr 2019 00:32:54 +0000 Cloudflare wrote:

I think we're just going in circle here

I agree.

and I should have better explain why we would need to focus on your own zone and check from there.

I am afraid you are missing the actual big issue here and trying to turn it into a limited (zone) issue, then from that work on fixing the limited issue. Please understand:
This has nothing to do with having a particular country restriction, or specific IP address being marked by Cloudflare (or by a zone owner) as problematic. These factors are generic and common sense. What I am discussing as an issue is an overall pattern which has nothing to do with any of these factors.
To prove that I have recorded for you a screencast showing the actual issue with the same Tor exit node and the same IP address but with different browsers:
[Screencast]

As you can see - TBB (on the left) opens cloudflare.com, Chromium (on the right) doesn't. There is the Ray ID too.
So:
████ Issue 1 (Tor and browsers)
TBB opens all sites, other browsers face captcha. And that is regardless of the particular IP address of the Tor exit node. Also as demonstrated in the STR earlier: that does not depend on the HTTP request headers.
████ Issue 2 (Captcha implementation)

The key is, Cloudflare employees can not remove the captcha.

But they can make a more privacy respecting captcha which does not:

use 3rd party HTTP requests
require JavaScript

I hope that clarifies actual primary and secondary issues.
====================
******, Apr 19, 1:24 AM PDT
Hello there

the curl script also hits the captcha.

Just for clarification purpose, are you trying to find a way to use the command line to query via TOR on Cloudflare website without solving the initial Captcha?
Best regards,
Mark
Cloudflare | Senior Support Engineer
====================
From: **** <>
To: Cloudflare <>
Subject: Re: [Cloudflare Support] **** - Using Tor without Tor Browser
On Fri, 19 Apr 2019 08:24:11 +0000 Cloudflare wrote:

Just for clarification purpose, are you trying to find a way to use the command line to query via TOR on Cloudflare website without solving the initial Captcha?

I am willing to be able to access through Tor all websites which use Cloudflare with any modern browser without being forced to reduce the privacy of a legitimate visit through:

JavaScript
cookies
sharing the fact of the visit a 3rd party

The curl example was aimed to demonstrate that the captcha doesn't depend on the HTTP request headers.
====================
***** M, Apr 20, 2:56 PM PDT
Hello ****,
Looking back over this issue it appears that Cloudflare's recently released "Onion Service"was meant to address some of your concerns. That is why you don't have issues with the Tor Browser.
https://blog.cloudflare.com/cloudflare-onion-service/
I don't believe that it's been adopted universally, but their are other options we just don't have insight into the status of all of them. And again as **** mentioned this is not something that we control, individual site owners can opt-out of this and specifically block Tor users if they want to.
Again we apologize for any inconvenience, please let us know if you have further questions.
Best regards,
*****- Cloudflare
====================
From: **** <>
To: Cloudflare <>
Subject: Re: [Cloudflare Support] **** - Using Tor without Tor Browser
Hi *****,
You guys asked for and received from me quite detailed information and repeated clarifications about the actual issues and the goal.
How exactly does any of the replies given so far provide a practical solution?
You keep repeating that the captcha is a feature which you don't control and it is up to the site owners to enable it or not. But the enabling of the captcha is not the issue here. The issue is the captcha itself - the way it is implemented, the fact that it reduces privacy and all the rest of it (as discussed). So that is something that only you (Cloudflare) are in control of. That is the actual issue here, no what your clients do or not.
====================
***** M, Apr 21, 12:03 PM PDT
Hello ****,
Thanks for clarifying. I do apologize for the privacy trade offs that are involved with use of Google's re-captcha service.
Google's recaptcha is an industry leader and an effective tool for preventing the automated or spam access of sites for many site owners. Unfortunately simple methods are no longer sufficient to inhibit maliscous requests.
For end users with a browser including Tor they can have a secure experience and mitigate user disturbance.
Unfortunately as you noted this does not include all browsers, and can be problematic when not using a browser. This may or may not be the site owner's intent. We can't speak to that, but imagine different cases would have different intent.
We don't strictly enforce the use of any captcha it's strictly up to the site owner's discretion. We now offer our Edge Worker, and apps platform and so users and/or site owners can implement their own solutions for cases where our's don't fit.
https://blog.cloudflare.com/introducing-apps-with-workers/
https://blog.cloudflare.com/tag/workers/
https://developers.cloudflare.com/workers/
Again we apologize for the friction this has caused. I hope that the sites can adjust their configurations and/or assist with adjusting your use-case.
We recommend that you reach out to the administrators for the specific services or sites that you run into issue with and share your experience.
Please let us know if you have further questions or concerns we can assist with.
Best regards,
*****- Cloudflare
====================
From: **** <>
To: Cloudflare <>
Subject: Re: [Cloudflare Support] **** - Using Tor without Tor Browser
Hi *****,

Google's recaptcha is an industry leader and an effective tool for preventing the automated or spam access of sites for many site owners.

Google is an industry leader in making technically effective tools to break into everyone's privacy.

We don't strictly enforce the use of any captcha it's strictly up to the site owner's discretion.

It is against the GDPR principles of data protection and minimisation to share data with other parties when it is possible not to do it. So although you may not enforce it, you don't take care to fix it either. It is practically a privacy back door - indeed a very effective one.
Additionally - in the process of enabling it there is no clear warning that informs the enabler that this tool is not GDPR compliant and about the consequences of if for the site visitor.
The final result: you are helping Google, not the web.

Unfortunately simple methods are no longer sufficient to inhibit maliscous requests.

A simple method for the user doesn't necessarily mean a simple implementation. A complex image captcha without 3rd party requests and without JS is surely possible and can work fine. It is also possible to present an animated GIF as a more complex challenge or even an HTML video. None of that requires JS or "industry leaders".

We now offer our Edge Worker, and apps platform and so users and/or site owners can implement their own solutions for cases where our's don't fit.
https://blog.cloudflare.com/introducing-apps-with-workers/
https://blog.cloudflare.com/tag/workers/
https://developers.cloudflare.com/workers/

So to answer the concern that JavaScript is a way to reduce user's privacy you suggest even more JS. Then what's the point of saying:

Please let us know if you have further questions or concerns we can assist with.

?

We recommend that you reach out to the administrators for the specific services or sites that you run into issue with and share your experience.

 I can assure you that nobody is going to contact a few million sites (or even just a few) because of this. I have seen people rage on forums and mailing lists against Clouldflare's recaptcha because of the concerns I am sharing with you. If you really care about the web and not just about corporate interest and industry leadership, you should approach this more seriously, rather than justify it.
If you don't - perhaps this whole communication is a waste of time. I already regret that I wasted a lot of mine just to read more that you are not going to do anything. I realize I am just one of thousands of non-paying users and perhaps of zero importance to you profit-wise. Yet the concerns I am raising are perfectly valid technically and I am quite disappointed that you are ignoring them.
====================
******, Apr 21, 10:12 PM PDT
Hello there
We appreciate the feedback. We take customer suggestions and input seriously and I have forwarded this to the appropriate team for consideration. Thank you again for sharing your ideas with us.
However, at this time, we do not have an immediate resolution for the concern you have brought up.
Best regards,

Cloudflare | Senior Support Engineer
====================
From: **** <>
To: Cloudflare <>
Subject: Re: [Cloudflare Support] **** - Using Tor without Tor Browser
You are welcome. I hope they will consider fixing it!
Thanks.
====================
***** M, Apr 22, 12:30 PM PDT
Hi ****,
One point of clarification, with Workers the JS runs on our servers not in the browser. It's used because it's a common language for web developer. It doesn't require JS execution on the client.
Thanks again for all your useful feedback please don't hesitate to reach out if you have further questions.
Best regards,
*****- Cloudflare
====================

I don't see how this is related to the ungoogled chromium project but I personally appreciate your effort on this issue.

Google's recaptcha is an industry leader and an effective tool for preventing the automated or spam access of sites for many site owners.

Google is an industry leader in making technically effective tools to break into everyone's privacy.

Oof!

wow! seemingly they cannot understand that people just wanna use curl and wget with sites on cloudflare. Instead of clearly stating that that is impossible due to their anti-malicious-actor policy they produce all that verbiage.
I have set up my cloudflare site to max. friendliness to tor users and it works in torbrowser-bundle, but curl always gets the challenge.
To achive curl compatibility I would have to switch off cloudflare entirely or at least take out those subdomains which I want to have curl-friendly.

Isn't the issue with curl and ungoogled-chromium in that they do not support .onion alt-svc headers described in the article mentioned by CF employee (https://blog.cloudflare.com/cloudflare-onion-service/) ? Tor Browser does support them, according to the article. If other tools don't, then that's probably why they are handled differently by CF.

indeed cloudflare improved things for users of TBB but not for curl-users.

Beyond applying firewall filters to Tor traffic, Cloudflare users can improve the Tor user experience by enabling Onion Routing.

Onion Routing allows Cloudflare to serve your website’s content directly through the Tor network, without requiring exit nodes required. This improves Tor browsing as follows:
Tor users no longer will access your site via exit nodes, which can sometimes be compromised, and may snoop on user traffic.
Human Tor users and bots can be distinguished by our Onion services, such that CAPTCHA challenges are only served to malicious bot traffic.

You can toggle this setting on or off in the Onion Routing panel of the Cloudflare dashboard Crypto app.

the onion chain in TBB looks quite normal with an guard and exit node like always.
That's probably due to this issue in TBB, which is also mentioned in that article: https://trac.torproject.org/projects/tor/ticket/27590

Isn't the issue with curl and ungoogled-chromium in that they do not support .onion alt-svc headers described in the article mentioned by CF employee (https://blog.cloudflare.com/cloudflare-onion-service/) ?

It seems chromium does support that header:
https://bugs.chromium.org/p/chromium/issues/detail_ezt?id=392575
But the issue has nothing to do with that. Opening clouldflare.com doesn't show any alt-svc response headers.

But the issue has nothing to do with that. Opening clouldflare.com doesn't show any alt-svc response headers.

After changing circuits in TBB and refreshing a page a few times, it seems like they are available only for some exit nodes.

After changing circuits in TBB and refreshing a page a few times, it seems like they are available only for some exit nodes.

I don't think this has anything to do with particular exit nodes at all. In my tests (in which I have changed circuits at least 50 times) TBB could always open cloudflare.com and ungoogled-chromium (or curl) could never do it.
I suspect it may be something in the crypto layer that allows TBB to open CF sites. I wonder where we can read about it (or ask for info).

they want to make a curl with support for Alt-Svc
https://github.com/curl/curl/pull/3498
e.g. try with   i.xerx.xyz/222   (https too)

status quo is better
I prefer the status quo.  It's useful for Ungoogled Chromium to show the CAPTCHA when entering a CF site because I do not want to be an enabler for those sites by giving them my traffic.  The instant "Attention Required" banner serves as an instant "avoid this site" flag.  Circumventing the CAPTCHA would actually help Tor users unwittingly support an adversary.
make it a user-controlled option
If someone makes U/C behave like TB w.r.t CloudFlare, then I would appreciate it if it could be switchable.  And if that functionality is enabled, I would hope to still see some indicator that I'm on a site that's normally CF-CAPTCHA encumbered.  Note that Firefox has a useful extension that automatically redirects CloudFlared sites to an archived version of the page.  We a bit crippled on Chromium-based browsers where that extension does not exist.
fingerprinting
It's also worth noting that Torproject is pushing hard to get everyone on TB because their fingerprinting countermeasure relies on many people using the same TB fingerprint.  If U/C fixes #783, it will poach from TB users.  At the same time, U/C does nothing to mitigate fingerprinting.  So it will hinder Torproject's anti-fingerprinting strategy.

status quo is better
I prefer the status quo.

"is better" != "I prefer"
Others may not prefer it.

It's useful for Ungoogled Chromium to show the CAPTCHA when entering a CF site because I do not want to be an enabler for those sites by giving them my traffic.

Then the question is: do you want to give Google information about the sites you visit?

The instant "Attention Required" banner serves as an instant "avoid this site" flag.

But when you have JS disabled (as a privacy aware user would, especially through Tor) it is not "avoid" but really block. So it becomes an enforcement for reducing privacy. The "the status quo" (what TBB offers) does not enforce that.
Also it may be more important to the user to be able to access the site rather than avoid it. Through Tor it is not such a big privacy issue to visit CF enabled site.

Circumventing the CAPTCHA would actually help Tor users unwittingly support an adversary.

Not circumventing it would help Google correlate your visits across websites because Google receives the IP address of your visit through the captcha (without your consent). So if you visit multiple sites, even through Tor, the 3rd party HTTP requests to Google still help fingerprinting. So there are actually 3 issues here:

Cloudflare (and its centralization)
captcha (and its relation to Google)
the combination of these two when using Tor

The first two are to be solved regardless of Tor as they are privacy (data correlation) issues with or without Tor.

make it a user-controlled option
If someone makes U/C behave like TB w.r.t CloudFlare, then I would appreciate it if it could be switchable.  And if that functionality is enabled, I would hope to still see some indicator that I'm on a site that's normally CF-CAPTCHA encumbered.  Note that Firefox has a useful extension that automatically redirects CloudFlared sites to an archived version of the page.  We a bit crippled on Chromium-based browsers where that extension does not exist.

Having an optional functionality is good. However having it through an extension is questionable because one has to trust yet another extension too. Having to base privacy on trust is not really the best thing.

fingerprinting
It's also worth noting that Torproject is pushing hard to get everyone on TB because their fingerprinting countermeasure relies on many people using the same TB fingerprint.  If U/C fixes #783, it will poach from TB users.

How do you "poach from users" and what exactly are you poaching?

At the same time, U/C does nothing to mitigate fingerprinting.

You are wrong. U/C adds specific flags about it. Additionally considering that the user-agent string of Chrome is statistically more popular than that of TBB, I think one may be better with U/C.
As a whole fingerprinting is all about statistics - you stand out less if you are closer to the most popular. So the question is - how do you know if you are close = how do you get actual statistics in order to minimize the fingerprint.

So it will hinder Torproject's anti-fingerprinting strategy.

How exactly?
And what is that strategy which cannot be implemented in U/C? Personally I think that using Tor with U/C could provide better privacy because unlike TBB which still keeps some of the background connections to third parties U/C comes out of the box with no such connections. Not having those connections at all is much better than having them through a Tor IP address.

"is better" != "I prefer"
Others may not prefer it.

The status quo is better w.r.t privacy, but worse w.r.t convenience.  Users who want convenience choose Google Chromium, not U/C.

Then the question is: do you want to give Google information about the sites you visit?

You've misunderstood.  I'm not suggesting going beyond the CAPTCHA.  Google does not know what CloudFlare site you're visiting if you do not execute Google's j/s.  You see the CAPTCHA (for which the puzzle is withheld due to not running Google j/s), and you hit control-w.  End of story.  What you're proposing ruins that and compromises privacy.  Users unwittingly feed traffic to CloudFlare.

But when you have JS disabled (as a privacy aware user would, especially through Tor) it is not "avoid" but really block. So it becomes an enforcement for reducing privacy.

It's the other way around.  It's accessing the website that compromises privacy (either by running j/s and solving the CAPTCHA, or circumventing the CAPTCHA in order to feed traffic to CloudFlare and feed a privacy abuser).

The "the status quo" (what TBB offers) does not enforce that.

The status quo is for TB to circumvent the CAPTCHA, and for U/C to display the CAPTCHA.

Also it may be more important to the user to be able to access the site rather than avoid it.

That's less secure and it harms everyone.  When you feed a privacy abuser (a CloudFlare site in this case), it enables the privacy abuser to thrive and continue to damage everyone.
If a user insists on feeding a privacy abuser, they can use Tor Browser.  It's not good, and worse to expand on it by creating more tools to help privacy abusers.

Through Tor it is not such a big privacy issue to visit CF enabled site.

Nonsense.  Sharing all traffic including unhashed passwords is a huge privacy issue, most particularly when users don't even know it's going on.  It happens without users knowledge or consent.  U/C displays an SSL padlock.  Most users are mislead by this, and believe they have a secure tunnel all the way to the web ownership (when really the tunnel terminates at CloudFlare).  You're advocating to expose more users to that unwitting compromise.

Not circumventing it would help Google correlate your visits across websites because Google receives the IP address of your visit through the captcha (without your consent). So if you visit multiple sites, even through Tor, the 3rd party HTTP requests to Google still help fingerprinting. So there are actually 3 issues here:

That's only if you execute Google j/s, which you should not do in the first place.

How do you "poach from users" and what exactly are you poaching?

You poach users by giving them convenience.  ATM, Tor Browser is the only way to bypass CF CAPTCHAs.  So Tor users who traffic CF sites are limited to TB, but once U/C becomes an option for that use-case, some of them will switch.  They will shrink the number of users emitting the shared fingerprint of TB thus increasing uniqueness.

You are wrong. U/C adds specific flags about it.  Additionally considering that the user-agent string of Chrome is statistically more popular than that of TBB, I think one may be better with U/C.

No, you do not know how fingerprinting works.  It's a collection of metadata and it goes well beyond user agent strings.  I cannot give you all the knowledge you need in this post, but very briefly, you cannot simply "add a flag" and make fingerprinting go away.  Every browser has a fingerprint and because it's a collection of metadata inherent in a browser's behavior, you cannot eliminate it.  What you can do is investigate all the possible pieces of metadata that are usually taken for browser fingerprinting and hardcode the browser to project the same static metadata for all instances of that browser (privacy in numbers).  Controlling the user-agent string doesn't even begin to control the problem.  It's a huge effort that only TB developers have undertaken.  Most importantly, U/C does not protect you from browser fingerprinting in the slightest.  By moving users from TB to U/C, you are shrinking the number of users who share the single TB fingerprint, and you're moving them on to a platform that does nothing to control all the fingerprinting variables.

And what is that strategy which cannot be implemented in U/C?

It could be implemented, but it's not.  You should implement that first before implementing #783.

@bruceleerabbit

It's a huge effort that only TB developers have undertaken

Sounds like you know about this. I'm curious to know what the Tor devs have done; could you describe what changes they've done?
For your reference, ungoogled-chromium currently has these three flags to directly mitigate fingerprinting (from docs/flags.md):

--fingerprinting-canvas-image-data-noise (Added flag to Bromite feature) - Implements fingerprinting deception for Canvas image data retrieved via JS APIs. In the data, at most 10 pixels are slightly modified.
--fingerprinting-canvas-measuretext-noise (Added flag to Bromite feature) - Scale the output values of Canvas::measureText() with a randomly selected factor in the range -0.0003% to 0.0003%, which are recomputed on every document initialization.
--fingerprinting-client-rects-noise (Added flag to Bromite feature) - Implements fingerprinting deception of JS APIs getClientRects() and getBoundingClientRect() by scaling their output values with a random factor in the range -0.0003% to 0.0003%, which are recomputed for every document instantiation.

The status quo is better w.r.t privacy, but worse w.r.t convenience.

Using Tor proxy and U/C - how exactly does the captcha page improve your privacy? The IP address is torified but how does sharing it with Google (through the captcha) improve my privacy? Private means "not shared". So how does sharing more info with more parties give better privacy?

You've misunderstood.  I'm not suggesting going beyond the CAPTCHA.  Google does not know what CloudFlare site you're visiting if you do not execute Google's j/s.  You see the CAPTCHA (for which the puzzle is withheld due to not running Google j/s), and you hit control-w.  End of story.

Google's captcha is an iframe and even with JS completely disabled, loading the page which shows you "Attention Required" still sends an HTTP request to Google = Google knows your IP address, can set a cookie (if 3rd party cookies are enabled) etc. So neither the fact that it is "Ungoogled" chromium nor the fact that you may be using Tor protects you from that. What can protect you is uMatrix or uBO with blocking of 3rd party HTTP requests. (but perhaps that may be impossible too considering #662)

It's the other way around.  It's accessing the website that compromises privacy (either by running j/s and solving the CAPTCHA, or circumventing the CAPTCHA in order to feed traffic to CloudFlare and feed a privacy abuser).

Accessing the site does not compromise privacy when you use Tor proxy. All that Cloudflare will know is the torified IP address and which site it contacted - that is the case with TBB.
When you are blocked from accessing it you have only 2 options:

to disable Tor (and expose your real IP address)
to enable JS and 3rd party requests to Google

Both reduce your privacy (and anonymity) much more than the case in which you would simply access the Cloudflare'd website through a torified IP address.

Nonsense.  Sharing all traffic including unhashed passwords is a huge privacy issue, most particularly when users don't even know it's going on.  It happens without users knowledge or consent.  U/C displays an SSL padlock.  Most users are mislead by this, and believe they have a secure tunnel all the way to the web ownership (when really the tunnel terminates at CloudFlare).  You're advocating to expose more users to that unwitting compromise.

You are simply becoming aggressive without understanding what I am actually talking about. Just because you don't want to visit CF sites and want to have an option to be warned that a site uses CF before visiting it does not mean the workaround you use improves anything.
I am in no way claiming that CF is very good. And you are not "sharing all traffic" with them. They cannot possibly see your traffic to other (non-CF) sites. But if you have to pass the Google captcha and you also visit other sites using the same Tor circuit (as unlike TBB U/C has no way to request a new circuit) then you surely inform Google that someone is using the same circuit to visit sites A, B, C, D... which creates a much bigger possibility for data correlation and fingerprinting.

You poach users by giving them convenience.  ATM, Tor Browser is the only way to bypass CF CAPTCHAs.  So Tor users who traffic CF sites are limited to TB, but once U/C becomes an option for that use-case, some of them will switch.  They will shrink the number of users emitting the shared fingerprint of TB thus increasing uniqueness.

This is a statistical fallacy. U/C has far less users than TBB. Those who use U/C + Tor are even less.
Even if we suppose people start switching from TBB to U/C+Tor there will be a positive result because with U/C their browsers won't make background connections to other hosts (what Firefox->TBB does).
Also why are you assuming that switching to U/C would be "increasing uniqueness"? Assuming a disabled JS and the fact that Chrome is a more popular browser (and thus statistically less fingerprintable) - how would switching to U/C "increase uniqueness"? Makes no sense. Instead of suggesting to keep U/C crippled, I would rather hope to have a more controllable fingerprint (protocol wise) along with making it possible to open CF sites through Tor. Then you would be able to appear identical to TBB, without all the Mozilla garbage.

No, you do not know how fingerprinting works.  It's a collection of metadata and it goes well beyond user agent strings.

Did you even read the explanations I sent to CF and my tests? Or are you simply assuming that you know everything just because you like to keep a workaround which currently gives you a false sense of added privacy?

It could be implemented, but it's not.  You should implement that first before implementing #783.

I am not against reducing the fingerprinting and I am not saying what should be first. This issue is about something else. You can open a separate one to discuss fingerprinting.
Here is also a reference (JS required):
https://bugs.chromium.org/p/chromium/issues/detail_ezt?id=807614

Using Tor proxy and U/C - how exactly does the captcha page improve your privacy?

It discourages users from accessing a privacy-abusing website.  CloudFlare gets less information about users under the status quo than with the convenience feature you're proposing.

The IP address is torified but how does sharing it with Google (through the captcha) improve my privacy?  Private means "not shared". So how does sharing more info with more parties give better privacy?

This is a begging the question fallacy.  The question itself relies on an bogus assumption.

Google's captcha is an iframe and even with JS completely disabled, loading the page which shows you "Attention Required" still sends an HTTP request to Google = Google knows your IP address

Nonsense.

Accessing the site does not compromise privacy when you use Tor proxy.  All that Cloudflare will know is the torified IP address and which site it contacted - that is the case with TBB.

You've fallen for the "Tor will keep me safe" blanket misconception.  Tor network coupled with various tools will prevent some disclosures of some information in some situations, but when you are on a CloudFlare reverse proxied site CloudFlare sees every single packet, including any unhashed passwords or HTTP PUT form data that you may transmit, and everything you click on, even when you have an HTTPS connection.

When you are blocked from accessing it you have only 2 options:

to disable Tor (and expose your real IP address)
to enable JS and 3rd party requests to Google

Actually you have omitted the most important option: to walk, which is actually the only privacy conscious option.  A CAPTCHA barrier facilitates that choice especially when the CAPTCHA is broken.  Suggesting removal of that barrier to send more users traffic to CloudFlare is obviously a lousy idea w.r.t. privacy.

Just because you don't want to visit CF sites and want to have an option to be warned that a site uses CF before visiting it does not mean the workaround you use improves anything.

We're talking about user privacy, which is incidentally what I "want".  I could just as well say this about what you "want" (convenience).  The status quo is better for privacy.  You're proposing hiding CloudFlare's very loud presence and at the same time making privacy-abusing websites more functionally convenient, which ultimately leads to more unwitting users feeding more information to a privacy abuser.

This is a statistical fallacy. U/C has far less users than TBB. Those who use U/C + Tor are even less.

It's statistically correct to realize that TB (btw, stop calling it TBB that's an obsolete term) that TB uses a single fingerprint across all instances.  Two users of stock TB have the same fingerprint, whereas two users of U/C have distinct fingerprints.  Because you cannot guarantee two U/C users have the same fingerprint, you also cannot guarantee that hundred of thousands of U/C users have the same fingerprint.  It makes no sense to look at population size of a platform that does not ensure fingerprint uniformity.

Also why are you assuming that switching to U/C would be "increasing uniqueness"?

Uniqueness is a measure of the number of users sharing the same fingerprint.  Obviously if fewer users share a common fingerprint, uniqueness of that common fingerprint increases.  Uniqueness is indirectly proportional to user count.

Did you even read the explanations I sent to CF and my tests? Or are you simply assuming that you know everything just because you like to keep a workaround which currently gives you a false sense of added privacy?

Actually your cURL example proves my point.  The list of headers you've attempted to mimic are insufficient.  Your fingerprint differs from that of TB enough for CF to treat your traffic differently.  FYI, plugins/extensions are enough to change a browser fingerprint.  BTW, your second question again makes use of begging the question fallacy.

@Eloston There are a lot of variables to fingerprinting.  I've not worked close enough to anti-fingerprinting development efforts to have the whole map of it in my head.  It is nice that U/C has taken some steps to mitigate fingerprinting.  I would look through TB docs and code to review what all they've done.  Note as well that TB may be the best for fingerprint masking but it may not be absolute either.  Certainly plugins are known to disturb that and create a unique fingerprint.  We also know that Tor users are getting deanonymized and we couldn't rule out fingerprinting of variables not controlled.

Using Tor proxy and U/C - how exactly does the captcha page improve your privacy?

It discourages users from accessing a privacy-abusing website.  CloudFlare gets less information about users under the status quo than with the convenience feature you're proposing.

Similarly you could simply stop using GitHub because it is owned by another big tech - Microsoft. You could as well block the whole IP address ranges owned by big tech companies which are part of PRISM and not communicate with anyone who uses any services by these companies - just because you don't want your privacy to be abused (which would be pretty much equivalent to not using Internet at all). However this issue here is about making Cloudflare sites work with U/C over Tor, not about your personal preferences. So please kindly stop flooding it with contentious off-topic comments. If you would like U/C to have the functionality which you want (and which currently you think you have although you don't) as an additional option - please open a separate issue where it would be fine to discuss it. Personally I would welcome such option. But again - that is a separate issue.

Google's captcha is an iframe and even with JS completely disabled, loading the page which shows you "Attention Required" still sends an HTTP request to Google = Google knows your IP address

Nonsense.

What I said is a fact which you cannot negate with arrogance. Here is a screenshot that proves it (clean U/C with JS and cookies disabled and no extensions):
https://www.picpasteplus.com/view.php?i=2p4pe45424l4ujp4zjtjjpti.jpg

It's statistically correct to realize that TB (btw, stop calling it TBB that's an obsolete term) that TB uses a single fingerprint across all instances.  Two users of stock TB have the same fingerprint, whereas two users of U/C have distinct fingerprints.  Because you cannot guarantee two U/C users have the same fingerprint, you also cannot guarantee that hundred of thousands of U/C users have the same fingerprint.  It makes no sense to look at population size of a platform that does not ensure fingerprint uniformity.

This is also wrong but I won't explain why because I don't want to encourage you to post more off-topic comments.
As I said: Please open a separate issue if you want to discuss fingerprint reduction.

Actually your cURL example proves my point.  The list of headers you've attempted to mimic are insufficient.  Your fingerprint differs from that of TB enough for CF to treat your traffic differently.  FYI, plugins/extensions are enough to change a browser fingerprint.  BTW, your second question again makes use of begging the question fallacy.

It is not "your point". It is my question to CF which they didn't answer.
Considering the HTTP headers are the same and there is no JavaScript in curl, AFAIK the only thing left (which may introduce a difference) is the crypto layer (i.e. below HTTP but closer to TCP). You can call that "fingerprint" if you want as practically any HTTP client's dimension can be used for fingerprinting but again - this issue is not about fingerprinting per se.
I may still be wrong about the crypto though. Unfortunately I don't know where we could ask (Some Tor community? Suggestions are welcome).

Similarly you could simply stop using GitHub because it is owned by another big tech - Microsoft. [...] So please kindly stop flooding it with contentious off-topic comments.

Take your own advice.  Github is off-topic here but I've opened #795 so you can discuss it in a more appropriate place.

this issue here is about making Cloudflare sites work

It is not the duty of UC to facilitate CloudFlare Inc, work around CF bugs, or ultimately to expand on CF privacy abuses.  You have filed your ticket in the wrong place.  #783 should not have been filed as an UC issue, and you're the only one to pollute the thread with off-topic content.
You're asking for a convenience feature that compromises the security of all U/C users.  The request is inappropriate and to the extent that there is any discussion at all it's quite perverse of you to suggest that discussion of the privacy compromises entailed by your feature request should be silenced as "off-topic".

Here is a screenshot that proves it (clean U/C with JS and cookies disabled and no extensions):

Your screenshot only shows that you've failed to prevent google code from executing in your browser (evidenced by the status 200).  My capture shows what it looks like when a U/C user properly mitigates google code from executing, and it proves that google contact is avoidable without implementing #783.

As I said: Please open a separate issue if you want to discuss fingerprint reduction.

Do your own homework.  It is not my responsibility to put in place preconditions to implementing #783.  Why should I?  I will however expose missing preconditions that would otherwise compromise everyone's security as a consequence of implementing your reckless and self-entitled feature request.

this issue is not about fingerprinting per se.

Citation needed.  Unless you can prove that CloudFlare does not use fingerprinting to accommodate TB sessions you can't rule it out.  It's a very likely possibility.  Otherwise you've misunderstood CF's motivation behind the CAPTCHA in the first place.  If the client-side could be coded to circumvent the CAPTCHA it would defeat CF's anti-bot purpose, as any bot could simply be coded.  ATM the only known way to systematically circumvent the CAPTCHA is using Selenium code (or the like) to control a TB instance.

It is not the duty of UC to facilitate CloudFlare Inc, work around CF bugs, or ultimately to expand on CF privacy abuses.

Similarly you could assert that the whole web should be blocked without option for unblocking because big tech companies which abuse privacy also own large network backbones and undersea cables and surely track the IP traffic on it.

You have filed your ticket in the wrong place.  #783 should not have been filed as an UC issue, and you're the only one to pollute the thread with off-topic content.

You are the only one who claims that. I leave it to @Eloston to decide.

You're asking for a convenience feature that compromises the security of all U/C users.

No. I am asking for an option to be able open CF sites through Tor with UC and I welcome that option coming with a visual warning and with additional measures to reduce fingerprinting (which itself is a whole separate issue which you didn't open for discussion). Ideally we should be able to have the same or smaller fingerprint than TB + a visual warning. This is my suggestion. Yours so far is: "do nothing and believe in non-facts".

The request is inappropriate and to the extent that there is any discussion at all it's quite perverse of you to suggest that discussion of the privacy compromises entailed by your feature request should be silenced as "off-topic".

Again: I invited you to open a separate issue to discuss fingerprinting reduction because it is a big one and not specific to Cloudflare only. You didn't do that. Instead you accuse me of "silencing" you and increase the arrogance and aggression in your replies.

Your screenshot only shows that you've failed to prevent google code from executing in your browser (evidenced by the status 200).

No. It shows what I said earlier: that Google's recaptcha is an iframe and that a successful HTTP request to Google is made. There is no "code execution" because JS is disabled completely (also visible on screenshot).
Maybe read what HTTP 200 means:
https://tools.ietf.org/html/rfc2616#section-10.2.1

Similarly you could assert that the whole web should be blocked without option for unblocking because big tech companies which abuse privacy also own large network backbones and undersea cables and surely track the IP traffic on it.

That's a fallacy by analogy because the backbone providers are not centralizing just one portion, then blocking access contingent on privacy compromise in exchange for access.  Furthermore, even if they were it would still not be the duty of UC to implement a privacy compromise in exchange for convenient access anyway.  The analogy is broken no matter how you look at it.

You're asking for a convenience feature that compromises the security of all U/C users.
No. I am asking for an option to be able open CF sites through Tor with UC

Whether it's an "option" or not, it's still a privacy compromise in exchange for convenience nonetheless.

I welcome that option coming with a visual warning and with additional measures to reduce fingerprinting (which itself is a whole separate issue which you didn't open for discussion). Ideally we should be able to have the same or smaller fingerprint than TB + a visual warning. This is my suggestion. Yours so far is: "do nothing and believe in non-facts".

Actually I was the first to state the need to make the feature user-controllable and mindful of the fingerprinting issues -- an idea which you now claim as your own in your goalpost-shifting attempt.

Again: I invited you to open a separate issue to discuss fingerprinting reduction because it is a big one and not specific to Cloudflare only. You didn't do that. Instead you accuse me of "silencing" you and increase the arrogance and aggression in your replies.

Again, do your own homework.  It's your problem not mine that #783 is blocked by underdeveloped anti-fingerprinting mechanisms.  Every submitter is limited by their own motivation.  I've gone as far as to raise lack of anti-fingerprinting as a barrier to #783, given rationale, and I'm content with that level of effort ATM.
Support your own initiative or watch it die.

Your screenshot only shows that you've failed to prevent google code from executing in your browser (evidenced by the status 200).
No. It shows what I said earlier: that Google's recaptcha is an iframe and that a successful HTTP request to Google is made. There is no "code execution" because JS is disabled completely (also visible on screenshot).

Maybe read what HTTP 200 means:

This reply only shows that you don't understand what HTTP 200 means.    As I demonstrated already with my screenshot, if you know what you're doing no connection to google.com is made on a CF "attention required" landing page.

@Eloston

I'm curious to know what the Tor devs have done; could you describe what changes they've done?

Finally I found something:
https://lists.torproject.org/pipermail/tor-access/
There are additional links inside the discussions. One of them is to a project by Cloudflare:
https://github.com/cloudflare/challenge-bypass-specification
which as per the top warning seems deprecated and recommends visiting:
https://privacypass.github.io/
which in its turn links to a paper named "Privacy Pass: Bypassing Internet Challenges Anonymously" describing the protocol:
https://www.petsymposium.org/2018/files/papers/issue3/popets-2018-0026.pdf
Just like I thought there is something done at level of cryptography. Perhaps you can understand what they are talking about.
Also the start of a related thread on tor-dev list:
https://lists.torproject.org/pipermail/tor-dev/2018-January/012776.html

@nchv That's a good find. I'll look into it more when I get a chance, but the latest post in the first link seems to suggest it doesn't actually help much...

but the latest post in the first link seems to suggest it doesn't actually help much...

It is from March 2018 and I don't think it reflects the current state of things. Of course you can always test for yourself. Personally, as I also told Cloudflare, I don't see any recaptcha when using Tor browser.

Regarding curl... have a look at the TLS version and cipher suite that Tor Browser uses when connecting to any https site, then simply replicate this in curl:
curl --http2 -A "Mozilla/5.0 (Windows NT 6.1; rv:60.0) Gecko/20100101 Firefox/60.0" -v --socks5-hostname <IP_ADDRESS>:9050 --max-redirs 0 --tls-max 1.2 --ciphers ECDHE-ECDSA-AES128-GCM-SHA256 https://www.cloudflare.com
Not a solution for UC, but does address the issue you had with the curl script. Works even without bothering to customise headers (apart from user agent).

That curl command works fine! Thank you.

Regarding curl... have a look at the TLS version and cipher suite that Tor Browser uses when connecting to any https site

Where exactly do you look that up please?

Looking at:
https://www.ssllabs.com/ssltest/viewMyClient.html
it seems the culprit in this issue may be the TLS_GREASE_AA cipher suite which is at the top of "order of preference" for Chromium and cannot be disabled through --cipher-suite-blacklist command line option.
I don't know what role this TLS_GREASE_AA plays (whether it is beneficial or not) but it doesn't appear at all in Tor Browser's cipher suite list.
Perhaps making it possible to disable that cipher (e.g. through a flag) would result in using the other ciphers only which would make it possible to browse Cloudflare sites through Tor.
@Eloston what do you say?

I'm getting various results from TLS_GREASE_2A (0x2a2a), TLS_GREASE_0A (0xa0a), TLS_GREASE_AA (0xaaaa), and so on (via opening new incognito windows) - howsmyssl.com lists e.g. TLS_GREASE_IS_THE_WORD_7A as top entry.
So I looked up what this GREASE is:

GREASE is specifically looking to find servers that don’t deal well with unexpected values. For example, when establishing a connection, the client provides the server with a list of capabilities it supports – for everything from protocol version to cipher suites. If the server does not recognize one of the values the client provides, it should be able to ignore it and successfully start a connection with the best available options.

GREASE will provide a formalized way to get wide-scale and real-world data on where clients and servers are not compatible. “Servers that do not tolerate unknown values will fail to interoperate with existing clients, revealing the mistake before it is widespread.” Remember, the goal of extensibility is to allow the TLS protocol to develop over time and add new functionality without breaking existing implementations.

Read more at: https://www.thesslstore.com/blog/google-wants-grease-chrome/
Chrome Platform Status

TLS clients offer lists of 16-bit code points (e.g. cipher suites) that servers select from. To remain extensible, servers must ignore unknown values. However, servers may have bugs and reject unknown values. These servers will interoperate with existing clients, so the mistake may spread unnoticed, breaking extensibility for the whole ecosystem. We will reserve some values to advertise at random, to prevent such mistakes before broken servers are widespread.
Status in Chromium
Blink components: Blink
Enabled by default (tracking bug) in:

Chrome for desktop release 55

Latest IETF TLS GREASE draft on github.com and on
datatracker.ietf.org: Applying GREASE to TLS Extensibility
Transport Layer Security (TLS) Parameters

So this may not be related to Cloudflare's filtering, or maybe the error is on Cloudflare's end?
I think the TLS GREASE 'feature' in Chromium is unrelated but possibly also unwanted for Ungoogled Chromium as it was intended (presumably by Google) to

get wide-scale and real-world data on where clients and servers are not compatible.

Not sure if Ungoogled Chromium should keep or remove the feature…
BTW: Cloudflare clients can whitelist TOR traffic
Also makes you wonder how privacy-friendly a joint proposal between Cloudflare and Google will turn out to be: The push to QUIC and HTTP/3 has come from two main companies - Google and Cloudflare

This is still an issue.
@Eloston is there any hope to have a flag for disabling TLS_GREASE_AA?

@emanruse Sorry for the late response.
I think a flag is the right way to go. Feel free to submit a PR.

I think a flag is the right way to go. Feel free to submit a PR.

I am not a programmer.
I was just wondering if it is in your (or other programmers') plans.

This is my successful patch to make chromium's http requests similar to tor-browser:
--- chromium-96.0.4664.110/content/public/browser/frame_accept_header.cc
+++ chromium-96.0.4664.110/content/public/browser/frame_accept_header.cc
@@ -14,25 +14,7 @@ namespace content {

 std::string FrameAcceptHeaderValue(bool allow_sxg_responses,
                                    BrowserContext* browser_context) {
-  std::string header_value = kFrameAcceptHeaderValue;
-
-#if BUILDFLAG(ENABLE_JXL_DECODER)
-  // In case the buildflag and the runtime flag are enables, we insert
-  // "image/jxl," into the header value at the correct place.
-  if (base::FeatureList::IsEnabled(blink::features::kJXL)) {
-    std::string header_value_prefix = header_value.substr(0, 54);
-    std::string header_value_suffix = header_value.substr(54);
-    header_value = header_value_prefix;
-    header_value.append("image/jxl,");
-    header_value.append(header_value_suffix);
-  }
-#endif  // BUILDFLAG(ENABLE_JXL_DECODER)
-  if (allow_sxg_responses &&
-      content::signed_exchange_utils::IsSignedExchangeHandlingEnabled(
-          browser_context)) {
-    header_value.append(kAcceptHeaderSignedExchangeSuffix);
-  }
-  return header_value;
+  return "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";
 }

 }  // namespace content
--- chromium-96.0.4664.110/net/socket/ssl_client_socket_impl.cc
+++ chromium-96.0.4664.110/net/socket/ssl_client_socket_impl.cc
@@ -302,7 +302,7 @@ class SSLClientSocketImpl::SSLContext {
     SSL_CTX_sess_set_new_cb(ssl_ctx_.get(), NewSessionCallback);
     SSL_CTX_set_timeout(ssl_ctx_.get(), 1 * 60 * 60 /* one hour */);

-    SSL_CTX_set_grease_enabled(ssl_ctx_.get(), 1);
+    SSL_CTX_set_grease_enabled(ssl_ctx_.get(), 0);

     // Deduplicate all certificates minted from the SSL_CTX in memory.
     SSL_CTX_set0_buffer_pool(ssl_ctx_.get(), x509_util::GetBufferPool());

With this patch, all is left is to start chromium with the command-line option --user-agent="Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0". The result is: as expected, cloudflare becomes less interrupting.

Is this still an issue?
@carlob415 Does your approach still work?

@PF4Public Yes, cloudflare still gives Tor-browser beneficial treatement in this way. I have been successful with the previously posted patch, I haven't experienced any hickups with clouldflare when using tor since.
This is a simple test using curl:
$ curl https://<IP_ADDRESS> -I -XGET

HTTP/2 403
date: Wed, 13 Apr 2022 22:28:50 GMT
content-type: text/html; charset=UTF-8
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=EvgZDDm26UXLY1Fzt062DVIfKVF7RA%2F43VfWA8ySQqzaMF7OCiLq0Aw4eFNk83uIdiyqF6UWbb6S1d2TaFyzFCYu1C%2FwjTXihZY5sFt7CGtOvQ4ofk%2ByKhg%3D"}],"group":"cf-nel","max_age":604800}
nel: {"report_to":"cf-nel","max_age":604800}
cf-chl-bypass: 1
permissions-policy: accelerometer=(),autoplay=(),camera=(),clipboard-read=(),clipboard-write=(),fullscreen=(),geolocation=(),gyroscope=(),hid=(),interest-cohort=(),magnetometer=(),microphone=(),payment=(),publickey-credentials-get=(),screen-wake-lock=(),serial=(),sync-xhr=(),usb=()
cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
expires: Thu, 01 Jan 1970 00:00:01 GMT
x-frame-options: SAMEORIGIN
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
set-cookie: __cf_bm=ORxBDkpNOG_xWmyiIBsz5jdkhvuBBCxNLu60XjXcpD8-1649888930-0-AatASUT2kZiv3nvPkPDVrAc5Gy6rqLfMxxJ8Ove1pi13myVcsfCv/xUu8IKWByYOSu7bCUUv2PxymzsiJrTUopE=; path=/; expires=Wed, 13-Apr-22 22:58:50 GMT; domain=.every1dns.com; HttpOnly; Secure; SameSite=None
server: cloudflare
cf-ray: 6fb7a318de7354a9-YYZ

$ curl https://<IP_ADDRESS> -I -XGET -A 'Mozilla/5.0 (Windows NT 10.0; rv:91.0) Gecko/20100101 Firefox/91.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' --tlsv1.3 --tls13-ciphers TLS_AES_128_GCM_SHA256

HTTP/2 200
date: Wed, 13 Apr 2022 22:28:29 GMT
content-type: text/html
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=GzJOpBZX3dJB0R65clcg%2B4u%2B2QjGQBpH%2BokncgqDBXcTI3EtjmmtEeZTklG5IztVba6WTSZxJa1JXME%2FOOlru9T0RJY5Pulu9l7YfPTxrSE33ncxTheYrhc%3D"}],"group":"cf-nel","max_age":604800}
nel: {"report_to":"cf-nel","max_age":604800}
last-modified: Thu, 31 Mar 2022 18:35:55 GMT
x-rgw-object-type: Normal
x-amz-request-id: tx00000ebb0210041d04de3-0062574c6a-9ef0780-default
strict-transport-security: max-age=31536000
served-in-seconds: 0.003
cache-control: public, max-age=14400
cf-cache-status: HIT
age: 547
expires: Thu, 14 Apr 2022 02:28:29 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
set-cookie: __cf_bm=DCDS6tkPUf8CFa.JOkFHAmOxfTkwk3P0gXG3beL7Dfs-1649888909-0-AfOv4s7Hgqf1iY9F/Vf7pEOYka8dPPDnXv7afsnef38C/TTia8Q4SaqbTTYHwLVq53+fiIKjgvNlG4ufCzAEAHU=; path=/; expires=Wed, 13-Apr-22 22:58:29 GMT; domain=.every1dns.com; HttpOnly; Secure; SameSite=None
server: cloudflare
cf-ray: 6fb7a2922ca03ff1-YYZ
alt-svc: h2="cflareusni3s7vwhq2f7gc4opsik7aa4t2ajedhzr42ez6uajaywh3qd.onion:443"; ma=86400; persist=1

Is it possible to rewrite this patch to make its effect be configurable by a flag?

I hope it is resolved by #1920.


Integration Test - Release - 2020/02/04 21:58:51

Auto create for integration test. IssueCommandFlow

#pri1

Just type '#pri3'

#pri2

#pri3

#assign PRMergerTest1

#assign NonAllowedUser

NonAllowedUser is not a valid GitHub ID, or is not a collaborator on this repo.

#reassign PRMergerTest2

GitHub user PRMergerTest2 has been removed from the MicrosoftDocs organization, so they were automatically removed as an assignee.

#reassign NonAllowedUser

NonAllowedUser is not a valid GitHub ID, or is not a collaborator on this repo.

#please-close


shellcode

if i have the shellcode from cobalt strike how i make like your shellcode (msf)
"\x89\xe5\x83\0c\x8b\x5b"

You can use something like CyberChef to convert the hex string into raw binary:
https://gchq.github.io/CyberChef/#recipe=From_Hex('Auto')&input=XHg4OVx4ZTVceDgzXDBjXHg4Ylx4NWI
Then save the file. I decided to remove the ability in the source code to support internal arrays to simplify the program.

You can also join https://discord.gg/ETqM4cvJtE to ask any questions, there is a channel there to support this project.


does not work for raspbian stretch today

File: engine/installation/linux/docker-ce/debian.md, CC @mstanleyjones
There appears to be no armhf stretch stable docker-ce package in your repo, so these instructions don't work.
I was able to find an rpm under stretch/pool/test on the download page and this will install once some additional dependencies have been satisfied.
The hello-world example does not work though. Just exits with no errors or logs.

I got the same issue.
root@raspberrypi:/etc/apt/sources.list.d# cat /etc/apt/sources.list.d/docker.list  deb [arch=armhf] https://download.docker.com/linux/raspbian      stretch stable root@raspberrypi:/etc/apt/sources.list.d# apt-get update Hit:1 http://archive.raspberrypi.org/debian stretch InRelease                        Hit:2 http://mirrordirector.raspbian.org/raspbian stretch InRelease                  Hit:3 https://download.docker.com/linux/raspbian stretch InRelease Reading package lists... Done root@raspberrypi:/etc/apt/sources.list.d# apt-get install docker-ce Reading package lists... Done Building dependency tree        Reading state information... Done E: Unable to locate package docker-ce root@raspberrypi:/etc/apt/sources.list.d#
Thanks in advance

Closing this ticket due to its age, and the impending refactor. If you think this is in error, feel free to reopen. Thanks!


on EL7 with systemd the init script is still added by the slurm RPM

let's disable it and have systemd start slurmd and slurmctld
when/if the slurm.spec is adjusted to not install the init script on a system with systemd
https://github.com/SchedMD/slurm/blob/master/slurm.spec#L452
/etc/init.d/slurmdbd is added by the RPM too but seems systemd takes precedence when the slurmdbd.service is present.

This task will fail when the spec file is updated. Shall we add it anyway?
Guess it doesn't hurt to keep the init service, only bad I can see is that "systemctl status" shows system state as degraded because the slurm init service fails to start.

Do you have a comment on this @jabl ?

Looks alright. And yes, I think this is a bug in the slurm spec file, having both the old init.d script and systemd service units is confusing.


Get login button back in navbar

We need login button in navbar back
Acceptance Criteria

[ ] User is able to login with github account

Tasks

[x] comment out the part for login button from codebase
[x] Test (try and actually sign up)

FIXED works if logged in from https. Further discussions in #422


Add integration test to ensure that the plugin works with the tflint we're using

Could we investigate this integration test structure and adapt? @lonegunmanb
https://github.com/terraform-linters/tflint-ruleset-aws/tree/v0.30.0/integration

@lonegunmanb please consider #10
I have implemented the same integration test framework as the aws provider.
Can we close this now?

Closing as implemented in #10


Per-root Telemetry Opt Out/In Options

I currently have the logging/telemetry turned off because I don't want to leak code from my employer. However, I would be thrilled to provide telemetry while working on my public/OSS projects.
Maybe allow this to be set with either an environment variable, or some "root file" such as a .codeiumrc with contents such as:
telemetry = true

A very close second would be to have something like:
let g:codeium_telemetry = v:true

This would completely override the setting on the website (i.e. the value on the website is ignored if this configuration is present). This would allow a user who normally opts into telemetry to opt out, and a user who normally opts out of telemetry to opt in. Orgs could also commit this file into their repo to prevent developers from accidentally leaking private code.

This is an interesting idea. Right now, we store this information on a per user basis but had not considered finer granularity. I will get back on this.


Insert Menu

Can we make the three buttons a bit more graceful please? I think they should be on the same line, even better would be a button group, or one of those drop down buttons which defaults to 'at end', but allows you to select from the other options.

What do you see? Buttons seem fine to me?

Can we please not assign the tasks, and let the owners pick it up themselves? Fay created the Insert Menu.

Sorry, I see this:

I'm looking at this at the moment. Will change it to a drop down button thing.

Can you think of better text for the labels? Should we have options for first and last as well as before and after?

Is that screenshot from Firefox? What do you see in Chrome or IE? Wondering if it's zoom or font size etc and on my machine Firefox always seems to display much larger text and images even when definitely set to 100%.
I think the current labels are fine. At end is the same as last I think so if anything At beginning would be the correct match but I suppose First and Last would be shorter labels.


Rhbrms 2487 OptaPlanner extension doesn't run on 1 processor machine

The test failures are in JBPM and have nothing to do with this PR.

jenkins retest this


[MM-59072] Add retry logic to fetching user profile

Summary
PR adds some retry logic to the API call to fetch the user profile, as a single error can cause the transcribe to fail.
Ticket Link
https://mattermost.atlassian.net/browse/MM-59072

Looks great, nice improvement! Is this the only API request transcriber makes that could be retried?

@cpoile Posting the transcription is covered. The only other API calls would be updating the job status. But if these fail, then the job fails, which at least is a clear outcome. In the case of the API call covered here, everything would look like working but the transcription for a given track would be missing at the end of the process as the job would succeed.


fix: Update FLTAppCheckProvider.m

use "alloc" instead of "new" to fix the error:
Semantic Issue (Xcode): 'new' is unavailable
Description
Replace this paragraph with a description of what this PR is doing. If you're modifying existing behavior, describe the existing behavior, how this PR is changing it, and what motivated the change.
Related Issues
Replace this paragraph with a list of issues related to this PR from the issue database. Indicate, which of these issues are resolved or fixed by this PR. Note that you'll have to prefix the issue numbers with flutter/flutter#.
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process. Updating the pubspec.yaml and changelogs is not required.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[x] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (melos run analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x] I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change.
[x] No, this is not a breaking change.

Closing since it's already fixed here: https://github.com/firebase/flutterfire/pull/10734


Incorrect description in fleet.rs CI/CD example

Describe the bug
      - name: Cache .cargo/bin
        uses: actions/cache@v3
        env:
          cache-name: cache-node-modules
        with:
          # npm cache files are stored in `~/.npm` on Linux/macOS
          path: ~/.cargo/bin
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/cargo.lock') }}
          restore-keys: |
            ${{ runner.os }}-build-${{ env.cache-name }}-
            ${{ runner.os }}-build-
            ${{ runner.os }}-

To Reproduce
Linux CI/CD
Expected behavior
We are caching for cargo, not npm node modules.
Screenshots

Will fix this later today when I get home from school.


Fix error when hidding processes

You merged the branch on itself. As it was rebased against master, there are conflicts right now.
Please, do the following steps:

copy your changes to another file;
go to master;
remove your local branch shiviz-integration, using git branch -D shiviz-integration;
get a fresh copy of this branch: git fetch && git checkout shiviz-integration
apply your changes and do a git push -f shiviz-integration


[release] update README for 0.17.0 release.

closes #1362

mind taking a look @srmadscience ?

Looks good to me!


Proper handling of exceptions

There are some situations which results to Logstash crash which should be handled by this plugin. They are: temporary timeouts in connections, connection refuses due to group rebalancing, invalid credentials. All these situations caused LS to crash even almost all (except for creds one) are just temporary.

Yes leaving group due to rebalancing leaves logstash running but kafka plugin not working which results in a "defunct" pipeline and requires manual intervention.

related to elastic/logstash#11603


Next Alarm Time in the wrong timezone

In the attribute "Next_alarms" for the next_alarm_time entity the first listed alarm has the correct time - for example "20 April 2024 at 07:20:00". However, in the next_alarm_time entity the corresponding time is incorrect - using the same example it would show "20 April 2024 at 08:20".

Have been experiencing the same issue and in my case, it was the host's timezone that was causing the problem, it uses it for calculation resulting time instead of timezone configured in HA (as I assumed at first).


Regexes appear to be significantly slower than re2 in some cases

 Issue by brson
Thursday Sep 04, 2014 at 18:15 GMT
For earlier discussion, see https://github.com/rust-lang/rust/issues/16989
This issue was labelled with: A-an-interesting-project, A-libs, I-slow in the Rust repository

My understanding is that we believe them to be competitive, but two benchmarks I've seen were not really in the ballpark. The easiest one to test is the shootout's regexdna, which you can see from the upstream shootout is drastically slower than the re2-based C++ implementation.
cc @BurntSushi

I'm going to close this in favor of #66 since this is kind of a vague ticket and #66 more directly addresses this concern.


How to build Tensorflow lib files with bazel from source code on Windows?

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Win10 pro Visual studio 2015
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: no
TensorFlow installed from (source or binary): trying to build from source
TensorFlow version: gpu-1.13.0
Python version: 3.6.7
Installed using virtualenv? pip? conda?: no
Bazel version (if compiling from source): 0.21.0
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: CUDA 10.0/ cudnn 7.4.2
GPU model and memory: Geforce 1080Ti

Describe the problem
I'm trying to build Tensorflow-gpu lib files(tensorflow.dll and .lib files) with bazel from source code on Win10, but I have encountered serveral problems.
Provide the exact sequence of commands / steps that you executed before running into the problem
After build configure
C:\tensorflow>python configure.py
WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:
nul
WARNING: Running Bazel server needs to be killed, because the startup options are different.
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
INFO: Invocation ID: 412c8d20-03eb-46e5-bf73-fd54952f1e01
You have bazel 0.21.0 installed.
Please specify the location of python. [Default is C:\Users\Administrator\AppData\Local\Programs\Python\Python36\python.exe]:

Found possible Python library paths:
  C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages
Please input the desired Python library path to use.  Default is [C:\Users\Administrator\AppData\Local\Programs\Python\Python36\lib\site-packages]

Do you wish to build TensorFlow with XLA JIT support? [y/N]:
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with ROCm support? [y/N]:
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Please specify the CUDA SDK version you want to use. [Leave empty to default to CUDA 10.0]:

Please specify the location where CUDA 10.0 toolkit is installed. Refer to README.md for more details. [Default is C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0]:

Please specify the cuDNN version you want to use. [Leave empty to default to cuDNN 7]:

Please specify the location where cuDNN 7 library is installed. Refer to README.md for more details. [Default is C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0]:

Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size. [Default is: 3.5,7.0]:

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is /arch:AVX]:

Would you like to override eigen strong inline for some C++ compilation to reduce the compilation time? [Y/n]:
Eigen strong inline overridden.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
        --config=mkl            # Build with MKL support.
        --config=monolithic     # Config for mostly static monolithic build.
        --config=gdr            # Build with GDR support.
        --config=verbs          # Build with libverbs support.
        --config=ngraph         # Build with Intel nGraph support.
        --config=dynamic_kernels        # (Experimental) Build kernels into separate shared objects.
Preconfigured Bazel build configs to DISABLE default on features:
        --config=noaws          # Disable AWS S3 filesystem support.
        --config=nogcp          # Disable GCP support.
        --config=nohdfs         # Disable HDFS support.
        --config=noignite       # Disable Apacha Ignite support.
        --config=nokafka        # Disable Apache Kafka support.
        --config=nonccl         # Disable NVIDIA NCCL support.

I ran
C:\tensorflow>bazel build --config=opt --config=cuda --define=no_tensorflow_py_deps=true //tensorflow/tools/lib_package:libtensorflow

Any other info / logs
After a long time of compiling, the build did indeed succeed but the lib file it generates is
libtensorflow.so
I expect the output lib file for windows to be tensorflow.dll and tensorflow.lib but I think this .so file is not useful on Windows, so I'd like to know:

if I can build a tensorflow.dll and tensorflow.lib file on Win10 or not?

and if not, how do I suppose to use this libtensorflow.so on Windows and deal with all the include file?

reference on this thread


Fix copyright year in api_gateway.go

Ⅰ. Describe what this PR does
Fix copyright year in api_gateway.go
Ⅱ. Does this pull request fix one issue?

NONE
Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews

/lgtm


AskScript: TypeError: undefined is not iterable (cannot read property Symbol(Symbol.iterator))

Watching https://youtu.be/Su1FBJ5r7CU?t=221 shows running the current snippet
should be possible:
find(philosophers, fun(is(at(scorePerPhilosopher, get('$0')), max(scorePerPhilosopher))))
However, here is what happens:
🦄 find(philosophers, fun(is(at(scorePerPhilosopher, get('$0')), max(scorePerPhilosopher))))
Thrown:
TypeError: undefined is not iterable (cannot read property Symbol(Symbol.iterator))
    at Resource.<anonymous> (/Users/piotrzientara/xfaang/askql/dist/askvm/resources/core/get.js:19:29)
    at Generator.next (<anonymous>)
    at /Users/piotrzientara/xfaang/askql/dist/askvm/resources/core/get.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/Users/piotrzientara/xfaang/askql/dist/askvm/resources/core/get.js:4:12)
    at Resource.compute (/Users/piotrzientara/xfaang/askql/dist/askvm/resources/core/get.js:18:16)
    at Object.<anonymous> (/Users/piotrzientara/xfaang/askql/dist/askvm/lib/run.js:37:56)
    at Generator.next (<anonymous>)

Related to #46

@Tuhaj  The outstanding issues regarding call have been fixed. I assume your code is in AskScript not AskCode, in this case the valid code for what you're doing is:
ask {
  find(philosophers, fun(name) { scorePerPhilosopher:at(name):is(max(scorePerPhilosopher)) })
}

And it produces the expected answer:
🦄 find(philosophers, fun(name) { scorePerPhilosopher:at(name):is(max(scorePerPhilosopher)) })
string ask(call(get('find'),get('philosophers'),fun(let('name',get('$0')),call(get('is'),call(get('at'),get('scorePerPhilosopher'),get('name')),call(get('max'),get('scorePerPhilosopher'))))))
'Turing'


Add system locale information

System locale are very helpful thus should be added to the device object property by default.
Android locale docs
iOS locale docs

Merged to master