Add linters and refactor code for maintainability and code reuse

Resolves: #47 #46

Adds new dependencies for dev environment to support hot reloading via nodaemon and ts-node
Extends the Probot class with Kubernetes client and updateToken method (these methods now also use app.log) Moved to separate file from the handlers.
Update handlers to use Probot app methods
Adds Prettier and Eslint config for code linting and formatting
Adds npm run lint to highlight violations
Adds npm run dev for dev env with hot reloading
Adds NVM config to match dev environment NodeJS version
Simplified logging, since Probot log uses Pino, which already contains most of the information passed to our log lines like installation ID...
Fixes a case when the org has no repository present - instead of parsing payload.repositories, we use payload.installation.account.login

Note: Pre-commit may fail if the CI runner doesn't have node installed in its runtime (Eslint hook requires it)

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please ask for approval from tumido after the PR has been reviewed.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Hey, I know this PR is huge, sorry for that. Most of it is moving the code around and letting Prettier to format it. I had to move the content from index.ts into app.ts so we can support hot-reloading via index.ts. Please review
/cc @harshad16 @mimotej @SamoKopecky @KPostOffice

@tumido: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

pre-commit
cfe8f0253af8c7e3ba0a7865241a862188ac6f0e
link
true
/test pre-commit

Full PR test history. Your PR dashboard. Please help us and open an issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@tumido: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

pre-commit
688b95c61362df104ee5194035212193f8c7256d
link
true
/test pre-commit

Full PR test history. Your PR dashboard. Please help us and open an issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: mimotej, SamoKopecky
To complete the pull request process, please ask for approval from tumido after the PR has been reviewed.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: harshad16, mimotej, SamoKopecky
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [harshad16]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Update Jesús Bossa README.md

:warning: This is a Pull Request Template :warning:

Checkout all the things you've done by placing an x within the square braces below.

Proposed Changes

[] I've read the CODE OF CONDUCT and abide to it.
[] I've read the CONTRIBUTING.md
[] I've forked the repository.
[] I've created a branch and made my changes in it.
[] I understand opening a PULL REQUEST doesn't mean it will be merged for sure.

I hope everything is right :)


Array wildcard validation is broken

From the docs, this example:
const data = {
  users: [
    {
      name: 'John',
      bio: {
        age: 28,
        education: {
          primary: 'Elementary School',
          secondary: 'Secondary School'
        }
      }
    }
  ]
}

const rules = {
  'users.*.name': 'required',
  'users.*.bio.age': 'min:18',
  'users.*.bio.education.primary': 'string',
  'users.*.bio.education.secondary': 'string'
}

const validation = new Validator(data, rules)

if (validation.fails()) {
  console.log(validation.errors.all())
}

It gives error:

The field is actually defined, but the error is given and error key is a string without array index.
Package version - 3.13.3

Hi @VladShcherbin , the wildcard validation has not been added yet. :/

@alanhr yes, I found out that although it was added more than a month ago, a new version is still not published. I linked this issue to https://github.com/skaterdav85/validatorjs/issues/204, docs are misleading now.
Thank you, hope a new version will be released soon.

@VladShcherbin I am aiming to have a new release by EOD Wednesday @alanhr

@mikeerickson hey, any news on a new release?

Fixed in 3.13.4.


Overtake behavior in Apollo

Hi Guys,
I am trying to understand the overtake behavior of the autonomous vehicle in Apollo. Kindly help me in getting clearing my doubts.
Please find the below points

Minimum speed for the obstacle to be considered as moving is 2 m/s
If I test on an obstacle with speed less than 2 m/s the autonomous car would stop and go rather than following it
If I test on an obstacle with speed greater than 2 m/s the autonomous car would follow
I am curious to know in which cases the autonomous car would overtake the moving obstacle. What are parameters (speed, type etc) should be considering to test the overtake behavior.

Thank you,
KK

Closing the issue due to no further communication. Feel free to re-open if there are any follow up questions.


Fix reversed transparentShare logic

Flips the check in Share.__str__() to correctly report whether a share added extra text or not - currently it's backwards.

again, big "how did i even do this" energy to this one - thank you so much!


Incorrect snackbar import

Codecov Report

Merging #715 into rc0.11.0 will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           rc0.11.0     #715   +/-   ##
=========================================
  Coverage     95.18%   95.18%
=========================================
  Files            73       73
  Lines          2905     2905
  Branches        441      441
=========================================
  Hits           2765     2765
  Misses           45       45
  Partials         95       95

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30e2caf...519e663. Read the comment docs.


v0.4.1

Tagged and released on crates.io.
If https://github.com/rust-windowing/softbuffer/pull/192 looks good, I'd like to release that as well.


Type error in backward pass

Hi, here's the MWE for the failing test in AdvancedVI.
using Bijectors
using DifferentiationInterface
using Distributions
using Mooncake
using Optimisers
using Random

function f(params, aux)
    (; re, samples, baseline, q_stop) = aux
    q = re(params)

    ℓq = logpdf.(Ref(q), eachcol(samples))
    ℓq_stop = logpdf.(Ref(q_stop), eachcol(samples))
    ℓπ = sum(abs2, samples, dims=1)[1,:]
    ℓπ_mean = mean(ℓπ)
    score_grad =<EMAIL_ADDRESS>ℓq * (ℓπ - baseline))
    score_grad_stop =<EMAIL_ADDRESS>ℓq_stop * (ℓπ - baseline))
    energy = ℓπ_mean + (score_grad - score_grad_stop)
    energy
end

function main()
    rng = Random.default_rng()
    q0  = MvNormal(zeros(3), ones(3))
    b   = Bijectors.Stacked(
        Bijectors.bijector.([LogNormal(0, 1), MvNormal(zeros(2), ones(2))]),
        [1:1, 2:3]
    )
    q  = Bijectors.transformed(q0, Bijectors.inverse(b), )

    params, re = Optimisers.destructure(q)

    adtype = AutoMooncake(; config=nothing)
    aux = (
        samples  = rand(rng, q, 10),
        baseline = 1.0,
        re       = re,
        q_stop   = q,
    )
    value_and_gradient(f, adtype, params, Constant(aux))
end

Alas, the emergence of v1.11 in CI has meant that I need to focus on that. I'll try to ensure that this is resolved by the end of the week.

The v0.3 release of AdvancedVI will wait until this is resolved!

Annoyingly the 1.11 transition is taking longer than I had planned, so this isn't going to get a look until next week. Apologies!

I saw all the works that you're going through. No apology needed! And thanks for all the effort.

@Red-Portal I believe that this is resolved in 0.4.18. Please let me know if it is, and close this issue if you find that it is!

edit: I meant 0.4.17

@willtebbutt The original error seems to have disappeared, but a new one took its place. Until I can package a MWE, here is the error.

Thanks for this. I'll take a look later today!

@Red-Portal this problem should be fixed on 0.4.20 . If you could please let me know, and close this issue if it is resolved.

@yebai this should work on 0.4.24, which will be available shortly. Could you re-start CI once it's released, and let me know if the problem has been resolved?


How is it ended up here?

Doesn't seem to compatible with https://modrinth.com/resourcepack/tras-fresh-player, it went up on me face instead (´・ω・｀)
PS. There was a crash before but it seem to fixed itself or not sure what config me did to fix it.

You using the latest versions of the mod and Accessories? This should be fixed already.

Ah.. there it is, guess that's why me still using old version of that.
Here's the crashlog - https://pastebin.com/NGNtFSeH

Yeah this should all be resolved now, if you're still crashing on latest versions of Accessories and Immersive Lanterns, lmk


Replicated Turn In Place Rotation Fighting

Description:
Occasionally [NOT reliable in every instance] the player in a replicated game will not turn in place as far as they should [for instance, when the character should turn 180 degrees, it may jitter slightly and only turn 90 degrees or so]
Steps To Reproduce:
1: In a fresh copy of the project, run a multiplayer game of any client size, playing as client. (You may need to repeat this step until it happens.)
2: try to turn to face a direction, either by aiming or otherwise by using the looking direction turning mode.
3: Observe a very slight jitter and notice that the character [both locally and on remotes] doesn't turn as far as it should.
Expected Results:
When facing 180 degrees of the character in aim or looking direction modes in multiplayer, the player character should turn in place to face 180 degrees as it does in single-player, non-replicated games.
Actual Results:
Occasionally, the character will jitter and only turn partway - usually around 90 degrees - during a multiplayer game. The cause is unknown, and does not reliably occur in every single instance. So far happens around 50% of instances and when it does occur, lasts the duration of replicated play until terminated.

I've noticed this as well. When playing as a listen server, I find that is consistently happens with the client player, but not really with the server player. Especially in first person mode where it uses rotate in place instead of turn in place. The client does rotate/turn far enough and will slide and the rotate/turn in place animations will need to play multiple times to catch up. I could be wrong, but it almost feels like the character is not rotating fast/far enough to keep up with the animation.

@MartianDays I managed to reproduce the problem, but I could not identify the source, if anyone wants to be willing to help I also thank you.
I need to know if this problem continues after the game build, if it continues this is a problem in the project itself, otherwise it may be a small problem in the synchronization of information from the engine itself

I've noticed as well that remote clients will show your rotation correctly - however, your own rotation will not. This tells me that the problem likely exists in how the rotation replication is being handled.
I'm not a C++ expert so I can't really look into how you're replicating the variables, but it could be worth looking at where the replication begins and ends? For whatever event that's replicating rotation, is there a "is locally controlled" check after multicast? Typically, how I replicate is Client Set Own Vars > Replicate Vars To Server, then, Server Multicasts Vars > Checks Is Locally Controlled > Only Sets Vars If Locally Controlled Is False.
Maybe that'll help?

@MartianDays I was debugging the variables and found the following

Input movement amount and function can sprint, seem not to be perfectly synchronized, error of 0.01 or something like that, but this cannot be fixed since it is a latency problem to synchronize.

The second problem I encountered was when setting the actor rotation when the movement is very sudden. This can be improved by replacing with variavies but performance can get worse.


fix(http): allow CORS in v2 compatibility endpoints

Closes #19715 ... contains cherry-picks from master1.x branch ( #18410)
Describe your proposed changes here.

[ ] CHANGELOG.md updated with a link to the PR (not the Issue)
[x] Well-formatted commit messages
[x] Rebased/mergeable
[x] Tests pass
[x] Signed CLA (if not already signed)

Is there anyone still facing the CORS issue?

The fix is already part of v1.9.3 https://github.com/influxdata/influxdb/blob/v1.9.3/services/httpd/handler.go proves it.


update(ui): change default fonts to Roboto

Close #366

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Serhii Frolov seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Python tests should allow custom requirements

I think the python tests should allow custom deps spec. For example, in this feedstock I need to set the python version of a noarch package and it is not allowed:
tests:
  - python:
      imports:
        - pptx
      requirements:
        run:
          - python =${{ python_min }}

so I had to replace it with:
tests:
  - script:
      - python -c "import pptx"
    requirements:
      run:
        - python =${{ python_min }}

What do you think is the best way to address this?

allow requirements or requirements.run?
add python_version so only the python version can be customized?

Let me know, what you think is best here and I can submit a PR if you want.

xref https://github.com/conda-forge/python-pptx-feedstock/pull/28


Unexpected cause in SslCloseCompletionEvent

I noticed one thing when trying to remove SslHandler to downgrade connection to plaintext -
There is no check on handshake state here
https://github.com/netty/netty/blob/d34212439068091bcec29a8fad4df82f0a82c638/handler/src/main/java/io/netty/handler/ssl/SslHandler.java#L700
but message says  "...before handshake completed" and sometimes it's not true - see reproducer, I can see that handshake is done but still see this message.
Expected behavior
either don't notify close promise if handshake is done by additional check like
if (!sslClosePromise.isDone() && !handshakePromise.isDone())
or keep condition but rephrase error message like "...removed before ssl close future is done" - not sure it makes sense
Actual behavior
Handshake is done but message "SslHandler removed before handshake completed" is shown (reproducer output below)
Minimal yet complete reproducer code (or URL to code)
EventLoopGroup group = new NioEventLoopGroup();
Bootstrap bootstrap = new Bootstrap()
    .group(group)
    .channel(NioSocketChannel.class)
    .handler(new ChannelInitializer<Channel>() {
        @Override
        protected void initChannel(Channel ch) throws Exception {
            SslContext build = SslContextBuilder.forClient().trustManager(InsecureTrustManagerFactory.INSTANCE).build();
            final SslHandler sslHandler = build.newHandler(ch.alloc());
            sslHandler.engine().setUseClientMode(true);
            ch.pipeline().addFirst("client-handler", sslHandler);
        }
    });

ServerBootstrap serverBootstrap = new ServerBootstrap()
    .group(group)
    .channel(NioServerSocketChannel.class)
    .childHandler(new ChannelInitializer<Channel>() {
        @Override
        protected void initChannel(Channel ch) throws Exception {
            SelfSignedCertificate ssc = new SelfSignedCertificate();
            SslContext build = SslContextBuilder.forServer(ssc.certificate(), ssc.privateKey()).build();
            final SslHandler sslHandler = build.newHandler(ch.alloc());
            sslHandler.handshakeFuture().addListener(future -> {
                if (future.isSuccess()) {
                    System.out.println("server: handshake is done");
                    ch.pipeline().remove("server-handler");
                }

            });
            sslHandler.sslCloseFuture().addListener(future -> {
                if (future.cause() != null) {
                    System.out.println("server: close future has not non null casue");
                    future.cause().printStackTrace();
                }
            });
            ch.pipeline().addFirst("server-handler", sslHandler);
        }
    });
serverBootstrap.bind("localhost", 1234).sync();
bootstrap.connect("localhost", 1234).sync();

Output:
server: handshake is done
server: close future has not non null casue
javax.net.ssl.SSLHandshakeException: SslHandler removed before handshake completed
	at io.netty.handler.ssl.SslHandler.handlerRemoved0(SslHandler.java:695)

Netty version
4.1.73.Final
JVM version (e.g. java -version)
openjdk version "15.0.2" 2021-01-19
P.S.
Removing SslHandler works in general and it looks like such SslCloseCompletionEvent doesn't hurt but there is one case where code hangs - I tried adding if (!sslClosePromise.isDone() && !handshakePromise.isDone()) to my local custom SslHandler and this error was gone but it didn't help with hanging. I decided to report anyway since this message looked confusing.

@BaurzhanSakhariev thanks for reporting! PTAL https://github.com/netty/netty/pull/12033


test data ? needs updating in 0.10.9.

ZeroDivisionError is absent from the file tested read and tested in test/test_outputfilter.py
''' The virtual module needs a valid file attribute.
If not, the Google app engine development server crashes on windows.'''
A Windows issue it seems in test/test_importhook.py
These two tests possibly need a fix.  There was one other but that is a gentoo sandbox issue that isn't an error.

    sed -e 's:ZeroDivisionError:Internal Server Error:g' -i test/test_outputfilter.py || return
    sed -e "s/        self.assertTrue/#        self.assertTrue/" -i test/test_importhook.py || return

in the ebuild sees these succeed.

Hi @idella
Are these still issues?
If so, could you provide specific steps to reproduce and any environment details you think may be relevant and I'd be happy to take a look.
Thanks in advance 😄


[Work In Progress] Delete existing tokens by scope intersection

Closes #61

TODO: Add unit tests to cover this

@shoatman Comments addressed. Feel free to have another look

@iambmelt Looks good


Downloading nodejs binaries is sometimes timing out

When installing node binaries, rtx will timeout because the nodejs servers are running very slow. We've noticed this a lot in the last couple of weeks, for different versions of node. Often, they can take 10 minutes to download. Is there any way to increase the timeout, or a better place to raise this problem?

this has been a problem for months: https://github.com/nodejs/nodejs.org/issues/4495
I don't think increasing the timeout is likely to be helpful, but I think there is an env var to do that. I think if it's not responding in 10 minutes it's not likely to ever respond. You're better off just retrying—perhaps many times.

Got it, thank you!


Removed bot 'Simulate' feature

The "Simulate" feature attempted to execute bot code in browser using eval() in a sandboxed <iframe>.  Unfortunately, as we have started adding npm dependencies to bot-layer, the capabilities of "Simulate" have diverged from reality.  This is causing too much user confusion, so we're cutting the feature entirely.
Asana: https://app.asana.com/0/1201910659736061/1202817922325631/f

Coverage decreased (-0.003%) to 94.381% when pulling 015138c07d73e771a4916f8ce7140beb1e2af070 on cody-remove-bot-simulate into 2b60896fc98e5e2b67cb262d9ada0af3ebf01b84 on main.


Ath9k : high video latency and low transfer speed

I copy here @user1321 observations :

I have some news..

After replacing firmware file:
/lib/firmware/ath9k_htc/htc_9271-1.4.0.fw
with htc_9271-1.4.0.fw from build tree video not freezing anymore.
It`s look like ImageBuilder script DONT overwrite  existing files.
I have some errors during build process, so, maybe my guess is incorrect. But I belive I am correct in my guess.
Also, I strongly recommend to take a look at /etc/fstab file.
As file system in RW state.

So, solution can be:

remove file first, or maybe add some parameter to replace if file exist.

So, to confirm, you may try to replace file at your side.
https://github.com/RespawnDespair/wifibroadcast-image-builder/tree/master/stages/04-Wifibroadcast/FILES/overlay/lib/firmware

file to replace:
/lib/firmware/ath9k_htc/htc_9271-1.4.0.fw

also, it`a good idea to check file system state. Is it ReadOnly or ReadWrite. Run: mount


Implement File Store provider

Only CRUD on users
Signed-off-by: Cédric Couralet<EMAIL_ADDRESS>

First draft for file store provider. Still needs some test, and some more implementations, but advices welcome !

All methods are not implemented but I think those could be added later.

rebase done

rebase done


Grpc port changed due to non keeping connection alive ?

Bug
What did you do?
I am proxify grpc through port 80 with path prefix.
I have a jaeger agent sending span to jaeger collector agent through proxy
Agent:
launched with :
command: --reporter.grpc.host-port=<IP_ADDRESS>:80

Server:
 labels:
      - "traefik.http.services.jaeger-collector.loadBalancer.server.scheme=h2c"
      - "traefik.http.routers.router0.service=jaeger-collector"
      - "traefik.http.routers.router0.rule=Path(`/jaeger.api_v2.CollectorService/PostSpans`)"
      - "traefik.http.services.jaeger-collector.loadbalancer.server.port=14250"
    command:  --log-level debug

What did you expect to see?
Keeping connection on port 80
What did you see instead?
Oct 23 10:16:54 localhost jaeger-agent[3352]: {"level":"info","ts":1571825814.7104955,"caller":"agent/main.go:75","msg":"Starting agent"}
Oct 23 10:16:54 localhost jaeger-agent[3352]: {"level":"info","ts":1571825814.710551,"caller":"healthcheck/handler.go:130","msg":"Health Check state change","status":"ready"}
Oct 23 10:16:54 localhost jaeger-agent[3352]: {"level":"info","ts":1571825814.710547,"caller":"base/balancer.go:76","msg":"base.baseBalancer: got new resolver state: {[{<IP_ADDRESS>:80 0  <nil>}] }","system":"grpc","grpc_log":true}
Oct 23 10:16:54 localhost jaeger-agent[3352]: {"level":"info","ts":1571825814.7106001,"caller":"base/balancer.go:130","msg":"base.baseBalancer: handle SubConn state change: 0xc00003a0b0, CONNECTING","system":"grpc","grpc_log":true}
Oct 23 10:16:54 localhost jaeger-agent[3352]: {"level":"info","ts":1571825814.7106805,"caller":"app/agent.go:69","msg":"Starting jaeger-agent HTTP server","http-port":5778}
Oct 23 10:16:54 localhost jaeger-agent[3352]: {"level":"info","ts":1571825814.7172532,"caller":"base/balancer.go:130","msg":"base.baseBalancer: handle SubConn state change: 0xc00003a0b0, READY","system":"grpc","grpc_log":true}
Oct 23 10:16:54 localhost jaeger-agent[3352]: {"level":"info","ts":1571825814.7172918,"caller":"roundrobin/roundrobin.go:50","msg":"roundrobinPicker: newPicker called with readySCs: map[{<IP_ADDRESS>:80 0  <nil>}:0xc00003a0b0]","system":"grpc","grpc_log":true}

Oct 23 10:17:01 localhost jaeger-agent[3352]: {"level":"debug","ts":1571825821.9618437,"caller":"processors/thrift_processor.go:116","msg":"Span(s) received by the agent","bytes-received":1367}
Oct 23 10:17:13 localhost jaeger-agent[3352]: {"level":"debug","ts":1571825833.9135826,"caller":"processors/thrift_processor.go:116","msg":"Span(s) received by the agent","bytes-received":427}
Oct 23 10:17:13 localhost jaeger-agent[3352]: {"level":"debug","ts":1571825833.9636493,"caller":"processors/thrift_processor.go:116","msg":"Span(s) received by the agent","bytes-received":799}
Oct 23 10:17:14 localhost jaeger-agent[3352]:
...
{"level":"debug","ts":1571825850.9625118,"caller":"processors/thrift_processor.go:116","msg":"Span(s) received by the agent","bytes-received":1366}
Oct 23 10:17:50 localhost jaeger-agent[3352]: {"level":"info","ts":1571825870.2970083,"caller":"base/balancer.go:130","msg":"base.baseBalancer: handle SubConn state change: 0xc0002508b0, CONNECTING","system":"grpc","grpc_log":true}
Oct 23 10:17:50 localhost jaeger-agent[3352]: {"level":"info","ts":1571825870.2971056,"caller":"roundrobin/roundrobin.go:50","msg":"roundrobinPicker: newPicker called with readySCs: map[]","system":"grpc","grpc_log":true}
Oct 23 10:17:50 localhost jaeger-agent[3352]: {"level":"info","ts":1571825870.2974412,"caller":"grpc/clientconn.go:1139","msg":"grpc: addrConn.createTransport failed to connect to {<IP_ADDRESS>:14250 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing dial tcp <IP_ADDRESS>:14250: connect: no route to host\". Reconnecting...","system":"grpc","grpc_log":true}
Oct 23 10:17:50 localhost jaeger-agent[3352]: {"level":"info","ts":1571825870.2974916,"caller":"base/balancer.go:130","msg":"base.baseBalancer: handle SubConn state change: 0xc0002508b0, TRANSIENT_FAILURE","system":"grpc","grpc_log":true}

Output of traefik version: (What version of Traefik are you using?)
Version:      2.0.2
Codename:     montdor
Go version:   go1.13.1
Built:        2019-10-09T19:26:05Z
OS/Arch:      linux/amd64

It s working but I wonder how grpc annonce original port and if there is missing in grpc routing with traefik.

Hi! I'm Træfiker :robot: the bot in charge of communication regulation.
Thanks for your interest in Traefik!
We dedicate the issue tracker to bug reports and feature requests only. My advanced AI has spotted that your issue might be a configuration problem or relates to something that doesn't look like a bug.
To confirm this, please join our Community Forum and reach out to us on the Traefik section.
In case I'm wrong (well, that would be embarrassing :sweat_smile:), my developers will re-open the issue and fix me!
In the meantime, you can double check Traefik's documentation.


Shared or static MAPL?

Is your feature request related to a problem? Please describe.
Question: in spack-stack-1.6.0, we had static ESMF and static MAPL. This somehow changed, and in 1.7.0 and develop (i.e. soon to be 1.8.0), ESMF is still static but MAPL is shared. What is correct? MAPL can be configured either or, but not both within one installation I think.
Describe the solution you'd like
Input from @AlexanderRichert-NOAA, @mathomp4 and @srherbener what works and what doesn't.

I believe you still should be able to build MAPL2 as static. We don't run that way so it's possible we broke something, but I think the build still supports it.
We also have used ESMF as shared for a very long time, but I can't see any reason why MAPL2 can't use it.
Now, MAPL3 is SHARED only. By its design it can no longer be used as a static library.

Decided to go with static mapl since esmf is static. This is how 1.6.0 was configured, which works for the UFS. And in the past, we had issues with double-free corruption at the end of ufs runs when esmf was static and mapl was shared.

See https://github.com/JCSDA/spack-stack/pull/1256 for the PR that makes mapl static


Package name not found

I think there is a typo or outdated name in the README:
-> @haleosde/ra-language-german
should be
-> @haleos/ra-language-german

Already a PR open #1


Paginated search retrieves incomplete documents upon multiple filters

The following code runs on each request for a specifc page number:
  const searchPaginatedData = await invoicesCollection.search(
    {
      q: '',
      searchFields: [
        "invoiceNumber",
        "customer.customerName",
        "customer.customerPhone",
      ],
      filter: {
        op: LogicalOperator.OR,
        selectorFilters: [
          { status: "Fully Paid" },
          { status: "Partial Paid" },
          { status: "Unpaid" },
        ],
      },
      hitsPerPage: 5,
      sort: { field: "createdAt", order: Order.DESC },
    },
    page
  );
  console.log("Hits count", searchPaginatedData.hits.length);
  console.log(searchPaginatedData.meta);

Following are the console outputs for 3 requests, one for each page. The first request shows correct document/hits count, but the subsequent pages are missing data:

page = 1 All good as 5 documents retrieved for a page size of 5

Hits count 5
SearchMeta {
  _found: 11,
  _totalPages: 3,
  _page: Page { _current: 1, _size: 5 },
  _matchedFields: []
}

page = 2 Only 1 document retrieved when there should have been 5

Hits count 1
SearchMeta {
  _found: 11,
  _totalPages: 3,
  _page: Page { _current: 2, _size: 5 },
  _matchedFields: []
}

page = 3 Zero documents retrieved when there should have been 1

Hits count 0
SearchMeta {
  _found: 11,
  _totalPages: 3,
  _page: Page { _current: 3, _size: 5 },
  _matchedFields: []
}

Note: The issue only arises if I have more than one filters combined via logical OR. Seems to be working fine with a single filter or no filters

Hey @adilansari , is this fix thoroughly tested? I'm still facing the bug as it retrieves fewer rows when I search with multiple OR fields. Or perhaps the fix is not released on Tigris Cloud yet?

Hey @adilansari , is this fix thoroughly tested? I'm still facing the bug as it retrieves fewer rows when I search with multiple OR fields. Or perhaps the fix is not released on Tigris Cloud yet?

Will be deployed most likely tomorrow, I shall update this thread when its ready.
Thank you for reporting it!

@binjamil, the fix was not released to Tigris Cloud yet. It has been now. Let us know if you still see the issue.


null value under KeyPath

Hi,
Awesome library! I'm trying to figure out how to handle a case when a given field may contain null value. Right now I receive
UnexpectedInputType("Unexpectedly found nil input.  KeyPath: .KeyPath(Date2) Expected: String")

which is quite understandable given it contains null value.
How should I approach it?

@kamil-tomaszewski Thanks for checking it out!
That error occurs because it's expecting to find a value of type String for your model, but unfortunately it's finding nil.
Instead, you should use type String? so that nil is an acceptable value.
Let me know if that doesn't solve your problem.
Best,
Logan

Then we have a problem :) Here's the struct:
struct Event {
    var id: Int?
    var name: String?
    var date1: NSDate?
    var date2: NSDate?
    var location: String = ""
    var city: String = ""
    var state: String = ""

    func fullLocation() -> String {
        return location + " " + city + ", " + state
    }

}

And here's the mapping:
extension Event: StandardMappable {
    init(map: Map) throws {
        try id <~ map["EventID"]
        try name <~ map["Name"]
        try location <~ map["Location"]
        try city <~ map["City"]
        try state <~ map["State"]
        try date1 <~ map["Date1"]
            .transformFromJson(NSDate.dateWithJSString)
        try date2 <~ map["Date2"]
            .transformFromJson(NSDate.dateWithJSString)
    }

    mutating func sequence(map: Map) throws {
        try id ~> map["EventID"]
        try name ~> map["Name"]
        try location ~> map["Location"]
        try city ~> map["City"]
        try state ~> map["State"]
    }
}

As you can see date2 is clearly an optional, yet it still expects a String.

Hi @kamil-tomaszewski
I see what's going on now, I didn't realize you were using a transformer.  Transformers fall under the same type restrictions as variables, so if NSDate.dateWithJSString expects a String and we receive nil in the Json, the system doesn't know how to handle it.
If you make dateWithJSString accept a String? it should be resolved.  Or you could put it in a closure:
        try date2 <~ map["Date2"]
            .transformFromJson { (input: String?) -> NSDate? in
                return NSDate.dateWithJSString(input)
            }

I've considered in the past investigating transform returns and if it's optional, skip the transformer and return the argument.  Unfortunately, this creates some situations where the user expects a failure for an unsatisfied value, and it won't exactly work.  For now, you'll have to be explicit about input types in transformers.

Thanks for your help! :) This works.
I've also found an issue with keyPath names, but I'll post it with a playground so you can test it.
Awesome library TBH, I very much like the failure driven approach.

@kamil-tomaszewski Happy to investigate the key paths issue as well, there are a few caveats to the current system, and I'm happy to help figure out what's going on.

Closing this for now, we can still chat about the keypath on this thread, and reopen if we need to.


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target/pull/731 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-target.git

Docs Build status updates of commit 831227e:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error-GitBranchDeletedOrForcePushed] Cannot sync git repo to specified commit because branch Release_Archive_master_2021-01-29-06-20-40 has been deleted or has been force pushed

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel

Docs Build status updates of commit 831227e:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error-GitBranchDeletedOrForcePushed] Cannot sync git repo to specified commit because branch Release_Archive_master_2021-01-29-06-20-40 has been deleted or has been force pushed

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching in the Docs contributor and Admin GuideSee the frequently asked questionsPost your question in the Docs support channel


Hello, I have a self-built cluster, but my kube-controller-manager is a binary program, not a container. Can pinniped be connected using the first token method?

Is your feature request related to a problem? Please describe.
Token Credential Request API: Can be run on any Kubernetes cluster where a custom pod can be executed on the same node running kube-controller-manager. This type of cluster is typically called “self-hosted” because the cluster’s control plane is running on nodes that are part of the cluster itself. Most managed Kubernetes services do not support this.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Are you considering submitting a PR for this feature?

How will this project improvement be tested?
How does this change the current architecture?
How will this change be backwards compatible?
How will this feature be documented?

Additional context
Add any other context or screenshots about the feature request here.

Hi @riven-blade,
The Pinniped Concierge has two modes of operation. If your cluster does not run the kube-controller-manager as a pod, then the Concierge will automatically detect that and run the impersonation proxy. Then the pinniped get kubeconfig command will auto-detect that, and generate the kubeconfig appropriately for that cluster. The other alternative is to use the Pinniped Supervisor without any Concierge for that cluster, by configuring the flags of the Kubernetes API server to use the Supervisor as its OIDC provider. We have documentation for that here.
I'm going to close this issue for now because there is no detail or questions in the description. Please feel free to let us know if you still have questions.


PokemonGoMap-Win for 0.3.2 installs V 0.1.7?

I downloaded the latest PokemonGoMap-Win.exe for 0.3.2 but it's installing Version 0.1.7?
Reverted from the clean version I was running.
Tried removing everything and trying again with no luck.

Version says <IP_ADDRESS> so just wondering what step I am missing?
The last Windows install worked fine?

Well, seems like I just needed to wait a little while and it updated :/


Translation readme

Pre-launch Checklist

[x] I updated/added relevant documentation (doc comments with ///).
[x] I added new tests to check the change I am making or feature I am adding, or @jonataslaw said the PR is test-exempt.
[x] All existing and new tests are passing.

a im not done yet but i opened this pull request so no one will start from scratch translating this to Arabic #1852

Hi, thank you very much for your contributions, you are very welcome and helpful to the community.
LGTM


Feature/add story book

need to do this
https://github.com/vercel/next.js/issues/8617#issuecomment-570067815


imageScrollView.zoomImageView is nil

when try to close the full image give me a error
fatal error: unexpectedly found nil while unwrapping an Optional value
file: AKMediaViewerController.swift
func uninstallZoomView() {
    let frame: CGRect = contentView!.convert(imageScrollView.zoomImageView!.frame, from: imageScrollView)
    imageScrollView.isHidden = true
    mainImageView.isHidden = false
    mainImageView.frame = frame
}

when I check
this line
imageScrollView.zoomImageView is nil
this problem happen when I disable zoom

Thanks @nelsongc , I already fixes the issue, please update to 1.0.5


Java Model Exception: Error in Java Model (code 969): common/common/src/test/java/dev/enola/common/io/resource [in _main] does not exist

@guw Hi, me again! 😈  Today tried again to use this with https://github.com/enola-dev/enola (with https://github.com/enola-dev/enola/pull/285), to try if #49 did the trick...
It seems like it's definitely much better now, and I now see e.g. Bazel Build Targets in the Explorer - yay!
However it seems like it's still not entirely working; e.g. if I open the EmptyResourceTest.java then the class name has a "red squiggle" in the editor, which says Implicit super constructor Object() is undefined for default constructor. Must define an explicit constructor - huh? (It's not finding the JDK, perhaps?)
And the Output for Language Support for Java shows this error, when opening that source file:
[Error - 3:31:55 PM] Dec 2, 2023, 3:31:55 PM common/common/src/test/java/dev/enola/common/io/resource [in _main] does not exist
common/common/src/test/java/dev/enola/common/io/resource [in _main] does not exist
Java Model Exception: Error in Java Model (code 969): common/common/src/test/java/dev/enola/common/io/resource [in _main] does not exist
	at org.eclipse.jdt.internal.core.JavaElement.newNotPresentException(JavaElement.java:550)
	at org.eclipse.jdt.internal.core.PackageFragmentRoot.getUnderlyingResource(PackageFragmentRoot.java:761)
	at org.eclipse.jdt.internal.core.PackageFragment.getUnderlyingResource(PackageFragment.java:426)
	at org.eclipse.jdt.internal.core.Openable.getUnderlyingResource(Openable.java:349)
	at org.eclipse.jdt.internal.core.CompilationUnit.getUnderlyingResource(CompilationUnit.java:987)
	at org.eclipse.jdt.ls.core.internal.handlers.BaseDiagnosticsHandler.collectNonJavaProblems(BaseDiagnosticsHandler.java:149)
	at org.eclipse.jdt.ls.core.internal.handlers.BaseDiagnosticsHandler.endReporting(BaseDiagnosticsHandler.java:135)
	at org.eclipse.jdt.internal.core.ReconcileWorkingCopyOperation.reportProblems(ReconcileWorkingCopyOperation.java:148)
	at org.eclipse.jdt.internal.core.ReconcileWorkingCopyOperation.executeOperation(ReconcileWorkingCopyOperation.java:112)
	at org.eclipse.jdt.internal.core.JavaModelOperation.run(JavaModelOperation.java:742)
	at org.eclipse.jdt.internal.core.JavaModelOperation.runOperation(JavaModelOperation.java:808)
	at org.eclipse.jdt.internal.core.CompilationUnit.reconcile(CompilationUnit.java:1326)
	at org.eclipse.jdt.ls.core.internal.handlers.BaseDocumentLifeCycleHandler.publishDiagnostics(BaseDocumentLifeCycleHandler.java:326)
	at org.eclipse.jdt.ls.core.internal.handlers.BaseDocumentLifeCycleHandler.publishDiagnostics(BaseDocumentLifeCycleHandler.java:289)
	at org.eclipse.jdt.ls.core.internal.handlers.BaseDocumentLifeCycleHandler$PublishDiagnosticJob.run(BaseDocumentLifeCycleHandler.java:751)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

So looks like some ... "mapping" (?) is still not quite right?

Thanks for continue trying it and giving feedback @vorburger!
I think we have different issues. I noticed the Java Model Exception in my logs as well but I believe this is an unrelated JDTLS issue. It could indicate a problem in the workflow, i.e. something not available yet. It could also indicate a gap in JDTLS (we are using linked resources/folders heavily).
The other issue (about EmptyResourceTest) is a feature gap. We currently do not read in individual java_test targets and set them up as source projects. I have been hesitating to do this. It would require creating one Eclipse/Java project per single test class to adhere to the Bazel principles and provide a correct classpath.
As a workaround, try using the project-per-package provisioning strategy. This is more suitable for Maven like projects at the cost of a more relaxed classpath in the IDEs in contrast to Bazel.

cat .eclipse/.bazelproject
directories:
  .

derive_targets_from_directories: true

target_provisioning_strategy: project-per-package


Fix exec/portforward minions->nodes

Exec and port forward weren't working with v1beta3 because they were
proxying minions. Change minions to nodes. This appears to work with
v1beta1 and v1beta2 as well.

@smarterclayton PTAL

LGTM


Add docs on layout functionality

Summary
I noticed we don't have any documentation around the layout functionality using gatsby-plugin-layout, this adds some bare minimum info.

👋 @LizBaker  😄


Inconsistency with fish shadow sizes

There seems to be an inconsistency with fish shadow sizes.
I'm assuming there are 6 size differences but "Medium (4)" and "Large (4)" appear in the list.
Example
Soft shelled turtle is "Medium (4)"
Koi is "Large (4)"
Is there a difference?

Hi, yes, the Koi value is incorrect in this case. This part of the data was done by hand before most data was datamined, I'll correct it on next push. Thanks for reporting.

Corrected in 520124f17b98bfb6adf95eba8d2173bbcdc48010 .


ControllerMap doesn't work with subdirectory controllers

Currently, I have a module 'base' where I have arranged my controllers under the 'admin' or 'site' folders. Below is the sample structure:
---yii2base (module)
---controllers
------admin
----------UserController.php
----------TenantController.php
I want to override the TenantController in my advanced app structure. And I have registered the controllerMap like this:

However, it doesn't work. I did some debugs and I have found that the yii/base/Module.php only check controllerMap[$id] which only return 'admin' part of the route while it should also check 'admin/tenant' route exists in controllerMap.
What do you think?

This issue is valid. My suggestion
this code https://github.com/yiisoft/yii2/blob/master/framework/base/Module.php#L499-L504
        if (strpos($route, '/') !== false) {
            list ($id, $route) = explode('/', $route, 2);
        } else {
            $id = $route;
            $route = '';
        }

can be changed with
        $id = $route;
        $route = '';
        while (($pos = strrpos($id, '/')) !== false) {
            if (isset($this->controllerMap[$id])) {
                $controller = Yii::createObject($this->controllerMap[$id], [$id, $this]);
                return [$controller, $route];
            }
            if ($route === '') {
                $route = substr($id, $pos + 1);
            } else {
                $route = substr($id, $pos + 1).'/'.$route;
            }
            $id = substr($id, 0, $pos);
        }

show url rules

^ its not about url rule.
@samdark IMO its also not a bug. Its must be features request.
FR: controllerMap support for subdirectory.

@mdmunir:

url rule must break it
need revert slash

@lynicidn

I am not sure what you mean. But, its not about url rule.
Ya, i know. @nganhtuan63 wrong when write controller id using backslash. But revert slash also not suported.

@mdmunir

i mean //
also in ur code

if ($route === '') {

$route allways be ''

Just let url rule as empty (use default setting).
Thats why i say, its enhancment or FR. I want controlerMap support for subdirectory
e.g

'controllerMap' => [
    'user/register' => [
        'class' => 'app\controllers\user\RegisterController',
        'userModel' => 'my\ext\models\User',
    ]
]

I am unsure about this issue.
Module::controllerMap serves ability to share controller defined elsewhere with the module without necessity of copying it.
Although controllerMap can be used for overriding existing controllers, this issue sounds like abuse of the original idea.

@klimov-paul
Ignore my comments above.
I just want to ask, Do you have plan to support subdirectory in controllerMap?

That is not for me to decide. At least, not for me alone.

I think it's good to have supported if it won't result in singificant performance drop.

This is not trivial. Imagine you have a URL a/b/c/d. When in module a, you need to test b, b/c, b/c/d. Note that this lies on the critical path.

@qiangxue
For now, we suport subdirectory both for application and module. will we suport for controllerMap to?
If you ask, do you using subdirectory for module. I said yes, i do.

Vote up for this issue. :+1:
A lot of times I like to ship Controller class as simple extension instead of big module.
Currently my website uses twitter like username directory, e.g. www.example.com/petrabarus.
Without supporting a slash in the controllerMap name, I have to add the controller name to the username blacklist to prevent user registering as the user. If the slash is supported, I just need to define one username blacklist and attach the Controller as the subdirectory.
But still, this is trivial issue. Might like to see this gets implemented if not dropping performance.

Won't be implemented in 2.0.


refactor: nuking encode_and_encrypt_note(...)

encode_and_encrypt_note(...) was not was not used very much as it was impractical for use when you want to produce efficient circuits (it commonly led to devs fetching the same keys twice: once for note, once inside the func). This PR nukes it and renames encode_and_encrypt_note_with_keys as encode_and_encrypt_note.

#8815  👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @benesjan and the rest of your teammates on  Graphite

Docs Preview
Hey there! 👋 You can check your preview at https://66f53033a5424eb1d76210e5--aztec-docs-dev.netlify.app

Merge activity

Sep 27, 3:19 AM EDT: @benesjan started a stack merge that includes this pull request via Graphite.


Add long path support for Windows [Approach #2]

Currently, ninja only works for paths less than MAX_PATH (255) on Windows. This change makes ninja long-path aware by manifesting the application as "longPathAware", and using the unicode version of windows file APIs.
See the below docs for more info:
https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation
https://docs.microsoft.com/en-us/dotnet/standard/io/file-path-formats#skip-normalization
https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation#enable-long-paths-in-windows-10-version-1607-and-later
This is a second attempt based off of https://github.com/ninja-build/ninja/pull/1939 , which attempts to address some of the shortcomings / difficulties with the previous approach.
Difference:
The main difference between this and the previous PR, is that previously all windows APIs were moved to the UNICODE version only. In this one, both UNICODE and non-UNICODE are supported based on the BUILD_UNICODE option set in CMakeLists.txt. This has a couple of advantages:

For non-unicode version, there are no extra copies incurred to convert paths to wstrings
Compiling for the non-UNICODE version executes the same code paths (mostly) as for non-windows platforms. One of the big difficulties I hit previously was getting the correct behavior between windows and non-windows platforms
No core logic changes. Given how deep the concept of paths as std::string is ingrained in the ninja code-base, trying to convert everything to use wstring (and still support std::string for non-windows) is a massive undertaking (And I found that I missed a ton of instances in the previous PR, especially around C file functions). By converting to wstring on-demand, there are no core logic changes with this PR. Just conversions to/from std::string/std::wstring when calling platform APIs.

Based on these performance results, I've enabled the BUILD_UNICODE option by default. If this is not desirable, I can switch it to OFF instead.

Test Name
UNICODE
ANSI

build_log_perftest
min 13ms  max 14ms  avg 13.2ms
min 14ms  max 14ms  avg 14.0ms

canon_perftest
min 189ms  max 204ms  avg 196.8ms
min 187ms  max 207ms  avg 192.2ms

clparser_perftest
Parse 16384 times in 3019ms avg 184.3us
Parse 8192 times in 2367ms avg 288.9us

depfile_parser_perftest
(Unknown what args to use)

hash_collision_bench
0 collisions after 20000000 runs
0 collisions after 20000000 runs

manifest_parser_perftest
(Failed to run python script)

Could you provide a comparison / contrast against https://github.com/ninja-build/ninja/pull/1671 ?

Could you provide a comparison / contrast against #1671 ?

Sure thing @jonesmz . I would say that the fundamental concept between the two is similar, but they have different goals.
The linked PR aims to fully support Unicode within ninja. This means it cares about string contents as well as string type, which results in changes that ripple through a large portion of the ninja code base. It also means that many code paths are different between windows and linux builds. The fact that long paths are supported with this approach is just a side-effect of using the Unicode version of windows C / Win32 APIs. It also fully removes the ability to build for windows with no UNICODE support.
In contrast, this PR enables UNICODE for the sole purpose of enabling long path support. That means that std:;string is still the main representation of a path within ninja, and std::string are converted to std::wstring on-demand only when being passed into win32 / CRT functions which take a file path. This means that no core-logic is actually changed between the unicode/non-unicode builds, at the cost of some minor overhead when copying a std::string to a std::wstring. It also maintains the ability to build without UNICODE defined, which greatly simplifies debugging issues between the windows build and linux build. The downside is that, while UNICODE is defined, this change doesn't actually make any attempt to ensure unicode string contents are handled differently than ascii (Same behavior as today)

when this will be merged and a new release will be available?

Both this PR and #1671 seem to move in the right direction, but I would suggest creating a dedicated FilePath class that holds a file path in the "native" format, irrespective of "UNICODE", with methods to wrap fopen()/unlink()/stat()/whatever, plus some helpers like FilePath::AppendUTF8(). Of course only use them when actually accessing the file system, Ninja can keep using UTF-8 strings for its internal representation of paths in the build graph, for example.
This will make the code easier to maintain, and avoids introducing yet another build configuration setting / variant for the Win32 build.
And to be clear, I'd happily introduce a PR to do that after any one of these two is merged.

Just adding a reference to the issue that this PR fixes.
Closes #1900.

@digit-google , I 100% agree. That's the path I originally started down, however it ended up being a much broader change than I felt comfortable introducing. Especially since it impacted the linux build as well, and I do not have as much experience there.

Does it work on MinGW?
If this is restricted to MSVC, then the title should be changed.

It looks like the code change should work on Mingw, only the CMakeList changes needs an update. I wonder if this needs an update in configure.py too? The latter is not used to generate release binaries for Windows, but I assume some people would wnat a Unicode-aware Windows binary generated with it too.

I would prefer it if Microsoft lifts the 260 character limit for *A functions instead.

That's unlikely for new versions of windows, and never going to happen for existing versions.
Is ninja planning to drop support for e.g. windows 7, 8, 8.1 and 10?

[...] and never going to happen for existing versions.

Microsoft has added UTF-8 support to Windows 10 via Windows Update so I don't see why this is never going to happen.

Is ninja planning to drop support for e.g. windows 7, 8, 8.1 and 10?

There is no "ninja" entity which "plans" anything.

Microsoft has added UTF-8 support to Windows 10 via Windows Update so I don't see why this is never going to happen.

You want to simultaneously support versions of Linux so old that they don't have C++11 compilers, while also drop support for versions of windows older than windows 10? That's inconsistent. Are we supporting older systems, or not?

There is no "ninja" entity which "plans" anything.

The group of people who are entitled to make decisions about what code gets merged into the codebase is what i meant by "ninja" here.
That means, effectively, you, since you're the only one out of the current list of maintainers on the Ninja-build organization on github who's done anything of substance in years.
So are you planning to drop support for older versions of windows, or not? Personally, I'm all for it, and would be happy to hear that's the plan.

I agree it is unlikely that the *A functions are updated to change behavior due to back-compat requirements. While it would be more convenient if we could just use the A functions in this code base, under the hood the A functions also widen the string and forward the calls to the W version of the APIs as well (After performing additional legacy logic related to DOS paths), so there's not actually a big difference.
From an outside contributor's perspective, I think the most helpful thing for me would have been if c++17 could be used easily throughout ninja, which would enable use of std::filesystem and std::string_view APIs . I think that would get the code closer to the kind of change @digit-google described above.

@majaeger I haven't rebased on latest ninja codebase in quite a while, life and all that.
But you're welcome to take a look at my fork of ninja, where I did add a patch to use std::filesystem, and another patch to use std::string_view.
I'd be happy to collaborate on things like this with you.
https://github.com/splinter-build/splinter/tree/16-std-filesystem
Note that that repo is organized as a series of patches, so a branch starting with N is intended to also include the commits from a branch starting with N-1, and so on.
This, of course, can make it confusing to work with, but the intention of that project has always been to get the patches applied to ninja, and as such i thought it would be easier to work in terms of patches, instead of doing a hard-fork.

well, i spent the evening rebasing all of my patches on the latest changes. Couldn't get the filesystem branch to build (see 16-filesystem-broken), but the other ones should all build, or be pretty close. Looks like there are some CI failures due to environmental issues that i haven't had time to mess with.
Regardless, the std::filesystem changes aren't terribly far from working again, so like i said, happy to collaborate if you have any interest.

While it would be more convenient if we could just use the A functions in this code base, under the hood the A functions also widen the string and forward the calls to the W version of the APIs as well (After performing additional legacy logic related to DOS paths), so there's not actually a big difference.

What's the runtime performance of this?
If the utf-8 strings are being widened every time one of these functions is called, how is that any better, in terms of runtime perf? It seems to me like, if the strings have to be widened at every call, we'd be much better off with this implementation: https://github.com/ninja-build/ninja/pull/1671

@jonesmz , the performance numbers for this change are in the initial PR description. It's basically a win in some tests and a slight loss in others. This is because the win32 API *A() functions themselves also widen the strings passed in, but do additional processing regarding legacy DOS paths. By doing the widening in the ninja code and using the *W() functions, we skip that additional processing, and the path gets passed directly to the filesystem driver.
I haven't looked through all of the changes you linked, but in general I expect performance to be about the same unless ninja uses  wide strings everywhere, thus eliminating the need for either the ninja code or the win32 A* functions to widen the strings.
I did actually see your fork before I started work on this. While I personally like the direction you've taken a lot of the code, I'm currently restricted in my corporate build environment to the main project. If my free time opens up though, I'll keep an eye out for issues in your repo that I might be able to assist with :)

If a new feature only works on newer Windows versions, that's not "dropping support for older versions". See UTF-8 or ANSI color output for example.
If a PR would result in ninja.exe not running anymore (or a regression) on Windows 7, I wouldn't merge it.

@majaeger I was trying to pull your changes into Ninja 1.10.2, but its not so easy. I think it may help if you rebased your work on Ninja master, rather than pulling Ninja master into your branch. Or maybe my git-foo is not sufficient to cherry-pick your commits.

@emmenlau , I pushed a new version which is rebased on upstream/master

Awesome, thanks! I can confirm that your changes work successfully in our build system to compile hundreds of common open source projects. The changes also resolve the following issue where Qt can not be compiled.

Can you rework this into a smaller and cleaner list of commits? The erroneous additions / deletions / last-minute compilation fixes are annoying when inspecting the history, and will make bisecting difficult in the future when we need this. Ideally each commit should compile and work properly.
Unfortunately, I am not maintainer and cannot approve running workflows to check that everything works as expected. Will try locally.

I would prefer it if Microsoft lifts the 260 character limit for *A functions instead.

[...] and never going to happen for existing versions.

Microsoft has added UTF-8 support to Windows 10 via Windows Update so I don't see why this is never going to happen.

To my limited understanding the path limit remains on the A functions for backcompat reasons (e.g. apps that pass around fixed MAX_PATH sized buffers before calling these).

@BillyONeal The change is behind a manifest setting so it wouldn't break backwards compatibility.

@jhasse, do you think this PR is something that can be considered for integration? The change helps us quite a bit building Qt on MSVC.

@emmenlau IMHO this PR is a workaround for something Microsoft needs to fix. Have you contacted them about it?

@jhasse I agree that Microsoft should fix this in their code.
But from the top of my head, I would know at least ten more things that are significantly broken in Microsoft code. When I was younger, I sometimes even went to the effort to contact them. But not a single time has this lead to anything, not even feedback on my report. So I personally would not be willing to invest into Microsoft fixing their code, I'd much rather have something working. And looking at this thread, I see a number of other responses that seem to hint at a similar feeling or stance.
If you are saying you prefer keeping ninja free of workarounds for other peoples bugs, I can understand. But it would be just nice to know if this PR here has a potential future in ninja, because currently it helps me a lot to get my work done and I'm a bit concerned that as a patch, it may not stay applicable and maintained forever.

You're probably right. I haven't decided on what to do regarding this issue. Not something for 1.11.0 though, as there's the UTF-8 change already.

IMHO this PR is a workaround for something Microsoft needs to fix. Have you contacted them about it?

@jhasse: The A apis are DOCUMENTED to be limited by MAX_PATH (260 CHARS). (see e.g. here: GetLongPathNameA)
So instead of requesting something unreasonable from MS, fix the usage of the wrong API in ninja.
From my view this is definitely a bug in ninja not reading the MS docs well enough.

@Neumann-A I know, I want Microsoft to add a way to opt-in to lift this limitation just as it was added for the W APIs.

Thanks a lot for the thumbs up, @jhasse ! I can very much relate that finding the right balance what to put into ninja and what not can be a challenge. But this PR seems to have potential to make the lives of a lot of people better, so I'm happy if it can be considered 😀.
What I can say so far is that we've been using this patch-set successfully in ninja master c5e50948 since about a month on Windows 11, Windows 10, Ubuntu 20.04 and macOS 11.5 and it resolved the MSVC Qt build problems.
Let me know if there is anything else I can do to move it forward. @majaeger I guess it would be great if it can be kept up to date against master, and maybe the commits could be squashed into a smaller set?
@jhasse, does that even make sense at this point in time, or should this PR wait for a thorough review from you or other core developers first?

To add to this:
https://github.com/microsoft/vcpkg/pull/23201
also uses this patch + a newer version of ninja. No problems observed in any ports depending on zlib or meson which are the entry points for the ninja-dependency in that PR.

@jhasse, does that even make sense at this point in time, or should this PR wait for a thorough review from you or other core developers first?

I don't have time to review this right now and as I said, I haven't decided what I think would be the best way forward. I'm very sorry :(

Depending on utf-8 support is nice if you don't care about past Windows, but otherwise you're leaving out many customers who have unicode in their paths.

I don't understand. UTF-8 is Unicode.

Depending on utf-8 support is nice if you don't care about past Windows, but otherwise you're leaving out many customers who have unicode in their paths.

I don't understand. UTF-8 is Unicode.

Past Windows, as in already shipped releases. Even if the policy gets changed to what you want it would only affect whatever the most current Win10 and Win11 releases are.

I don't understand. UTF-8 is Unicode.

UTF-8 is not Unicode - UTF-8 is a standard representation of unicode text, but there are many other representations of unicode text, including (but not limited to):

The three standard representations:

UTF-8
UTF-16
UTF-32

Many non-standard representations, including the most important - GB18030, the official encoding of the PRC

Windows and environments like it (Java, Javascript, SMS) were based originally on UCS-2 since they came before Unicode had decided to break its 65,536 character limit, and so they are now WTF-16 -- UTF-16 but allowing unpaired surrogates (although to be fair, not supporting unpaired surrogates is not a thing anyone cares about unless you're a file system implementation).
This fundamentally means that, to talk to these systems, you need to convert to UTF-16 on the boundaries.

I misunderstood your sentence about leaving out past Windows, of course I know UTF-8 is not Unicode itself but a Unicode encoding.

This fundamentally means that, to talk to these systems, you need to convert to UTF-16 on the boundaries.

You don't need to do that, otherwise Ninja wouldn't work on Windows before this PR. You do need to convert to UTF-16 to gain long path support.

Hi,
I don't understand the resistance here to add long path support.
It is thousand of hours if not more that people loose every single month because of the long path issue on Windows. And when the build fails it is very hard to detect and extremely disappointing to find that in 2022 we're stuck because of long path issue on Windows. This leads to bad practices, like building in C:\x\ to have the shortest path possible and hope that it will work.
I believe it should be a top priority. We're lucky to have people from Microsoft helping out to find the best solution in regard of Windows history and backward compatibility which you seem to be concerned about.
I hope this gets solved soon.
Regards,
Alex.

ping for merge? anything stop this?

For older OS, we can always using UNC path to handle long path.
I am not sure if this PR are fixing the issue properly, also we can using utf-8 everywhere, there is no double about

I don't have time to review this right now and as I said, I haven't decided what I think would be the best way forward. I'm very sorry :(

@jhasse do you have time to review this? If not, can you pass this off to someone else to review? This is blocking our company as well. Our paths aren't even that long -- it's just generated intermediary files which can be nearly 2x the length of the actual paths checked into source which are causing problems, and it's a bit unfortunate for something as robust as ninja to be stuck on dinosaur OS APIs that enable these problems.

My workplace just burnt 3 engineers for > 1 month working around this same issue.
Ultimately we played whack-a-mole with the various scripts that could generate long paths, shortening folder names whereever we could, to get under the limit.

I compiled the latest Ninja git master with this patch in hope to resolve this issue: https://github.com/microsoft/vcpkg/issues/23714
Sadly, it didn't help. Only this fixed the problem: https://github.com/microsoft/vcpkg/issues/23714#issuecomment-1075002693
So it seems this patch is either not complete (catching all cases?) or is buggy.

So it seems this patch is either not complete (catching all cases?) or is buggy.

This is because there are other tools being a road block.. In the qt case there are moc/uic/rcc which don't have long paths enabled (although they easily could) but the ultimativ roadblock will be the MSVC tools which heavily use MAX_PATH and are thus limited. So unless you use a full llvm toolchain which includes stuff like rc/mt you won't get full long path support but only a few extra characters.

So it seems this patch is either not complete (catching all cases?) or is buggy.

This is because there are other tools being a road block.. In the qt case there are moc/uic/rcc which don't have long paths enabled (although they easily could) but the ultimativ roadblock will be the MSVC tools which heavily use MAX_PATH and are thus limited. So unless you use a full llvm toolchain which includes stuff like rc/mt you won't get full long path support but only a few extra characters.

so there is no chance ninja + msvc will receive a patch to resolve this nasty issue?

So it seems this patch is either not complete (catching all cases?) or is buggy.

This is because there are other tools being a road block.. In the qt case there are moc/uic/rcc which don't have long paths enabled (although they easily could) but the ultimativ roadblock will be the MSVC tools which heavily use MAX_PATH and are thus limited. So unless you use a full llvm toolchain which includes stuff like rc/mt you won't get full long path support but only a few extra characters.

Ninja supporting the long path is still valuable at least for my project. The long path won't be involved in any msvc compilation but it blocked CMake + ninja.

Ninja supporting the long path is still valuable at least for my project. The long path won't be involved in any msvc compilation but it blocked CMake + ninja.

Well, if you can build ninja from source, this PR here worked nicely for us for about a year now. If you can not build from source let me know, maybe I can send a pre-built Windows executable.

When it worked for over a year, can this "just" be merged?

This branch has conflict, so the PR should be updated.

When it worked for over a year, can this "just" be merged?

Well, as was discussed before, there is still no consensus inside the ninja team if this PR is the right way forward. Kindly pinging @jhasse for suggestions where to go from here...

Why would you want to turn it off?

Why would you want to turn it off?

It seems that the ninja team was against having long path for some reasons, to not break legacy build systems etc...
So having a switch to turn it on/off solves the issue with builds that requires long path disabled.

It seems that the ninja team was against having long path for some reasons, to not break legacy build systems etc...
So having a switch to turn it on/off solves the issue with builds that requires long path disabled.

I do not think that this is the main concern. I think the main concern is that this PR adds a significant change to the codebase, and there are other ways how these changes could be implemented. Depending on the implementation, there are different pro's and con's considering the complexity of the change-set and the effort of future maintenance.
This is just my understanding. But I would be quite surprised if anybody would explicitly disable long path support when its merged. The downsides I can see are in the code complexity and maintenance.

The amount of changes in this PR can be reduced by removing #if _UNICODE logic and always using W-versions of  WinAPI.
There is no point in supporting A-versions.

I think the main concern is that this PR adds a significant change to the codebase, and there are other ways how these changes could be implemented.

Well kind of. If you mean by "other changes" changes by Microsoft and not in Ninja's code: Support for long paths in combination with UTF-8.

At https://github.com/ninja-build/ninja/pull/2225 I have a very small PR that just adds the "longPathAware" manifest, which allows usage of long paths if the proper registry key is set.
It has 0 code changes and for the users that need long path support they can have it.

@jhasse , I am very skeptical that the change in #2225 resolved all of the long path issues that this PR handles. I also can't continue to keep up this branch updated forever if it is not going to be considered though. If I perform testing on #2225 for additional cases and find it is still not sufficient, will this PR actually be considered?

IMHO all additional cases not solved by #2225 should be reported to Microsoft. This really shouldn't be app developers responsibility.

I tested this PR and it solved the long path issues of ninja on Windows. First I tried #2225 and it did not solve anything. I don't understand your reasoning why Microsoft should solve this?

ninja isn't the only Windows program. There are thousands of others. It's less work if Microsoft puts in the (probably huge? No idea, my bet would be that there code sucks BIG TIME. Still:) work to fix this in Windows so that all the thousands apps only need #2225 and not #2056, than every program doing #2056.

In the qt case there are moc/uic/rcc which don't have long paths enabled (although they easily could) but the ultimativ roadblock will be the MSVC tools which heavily use MAX_PATH and are thus limited. So unless you use a full llvm toolchain which includes stuff like rc/mt you won't get full long path support but only a few extra characters.

I took care of moc/uic/rcc in Qt 6.5 with the longPathAware manifest at https://codereview.qt-project.org/c/qt/qtbase/+/447113, similar to https://github.com/ninja-build/ninja/pull/2225 of ninja.
Soon you can on Windows use: clang-cl, Qt, and ninja to build applications for Windows using the MSVC ABI.

@jhasse

ninja isn't the only Windows program. There are thousands of others.

But it is the one used to build say xformers from source code, and now that is failing thanks to ninja still not supporting long paths after 4 years of deliberation on how to proceed on the path issue(s) reported in the repo.
I don't see what is stopping the team from fixing ths?

Using *W APIs should have happened long ago
Why not use std::filesystem::path instead of rolling your own filepath class as suggested elsewhere? It supports all relevant conversions.
Manifest will be ignored on older Windows which don't support it.
ninja can check the registry key on startup and print a warning for the user to enable it if they have issues with long paths.
on older Windows versions ninja can prepend \\?\ to the path and override MAX_PATH limitation like pretty much all apps dealing with paths on Windows were doing since like forever.

It's 2024 for God sake, issues with long paths (and/or spaces in paths) should be a thing of the past.

It was reported that long path support works with current master, please give it a try and comment there if it doesn't work for you: https://github.com/ninja-build/ninja/issues/1900#issuecomment-1817532728

It was reported that long path support works with current master, please give it a try and comment there if it doesn't work for you

The solution in master does not work for me. I'm sorry, I thought this was known? A number of people keep on having problems with the solution in master, me for one.
I've enabled the registry setting for long paths in Windows 11, then built ninja from latest master, and it does work in many cases, where it failed without the manifest fix/registry hack before.
However, there are still some cases where the build fails. I could not really figure out why, but for example, Qt fails to build for me (fully reproducible). In contrast, I've switched to the PR https://github.com/ninja-build/ninja/pull/2056 from Matt Jaeger successfully since a rather long time, and subsequently never had any long path issues ever since.

It was reported that long path support works with current master, please give it a try and comment there if it doesn't work for you: #1900 (comment)

I can do that but I don't know whether putting the ninja.exe built from master into a folder which is in the PATH is enough for pip to use that version when building a wheel provided that no ninja package is installed in current venv or machine wide?
See, that's why we need a release. Install the package and Bob's your uncle instead of having to figure out a way to override each and every build system which is now using ninja.


use laravel Authentication in pure php scripts

I am working on a laravel App. a part of that is written pure php that is a html form and form submission script that stores data to Database.
Suppose form.php page is like :
<?php
	require('config.php');
?>
<form action="process.php" method="post">
   <input type="text" name="username">
   <input type="submit" value="Send">
</form>

And process.php page is :
<?php
require ('config.php');
?>
if (isset($_POST['submit_edu'])) {
		$username = trim($_POST['username']);
        // other codes to store data to DB
}

Now I want just Authenticated Users in main laravel app can access to those pages and send their info.
To access Auth laravel Object, I added below codes to beginning of config.php file that holds connection settings and required by those two page:
require __DIR__ . '/../../bootstrap/autoload.php';
	$app = require_once __DIR__ . '/../../bootstrap/app.php';
	$app->make('Illuminate\Contracts\Http\Kernel')->handle(Illuminate\Http\Request::capture());
	$kernel   = $app->make(Illuminate\Contracts\Http\Kernel::class);
	$response = $kernel->handle(
		$request = Illuminate\Http\Request::capture()
	);

	if (!Auth::check()) {
		exit('Only authorized Users can access to this page');
	}

As you can see if an unAuthorized user want to pages Encounters an error.
Now when an Authenticated user want to open form.php page (in face via GET method) Auth::check() works correctly but when clicks on submit form button and data sent to process.php (via POST method) , it always returns false.
While if I open same process.php via GET method  Auth::check() works correctly again.
Means in this case Auth::check() can not work properly.
What is problem in your idea. can anyone help me to solve this problem?

Please use the forums to post questions, this repo is for bug reporting only :)


LaTeX equation parsing error

I use this code to upload my file:
with open("a.md", "r", encoding="utf-8") as mdFile:
    newPage = page.children.add_new(PageBlock, title="TestMarkdown Upload")
    upload(mdFile, newPage, notionPyRendererCls=addLatexExtension(NotionPyRenderer))

And my file  a.md is that:
$\phi\left(v_{i, l}\right)=\mathbf{R}_{i} v_{i, l}+t_{i}$

$$
P\left(v_{i, l}\right)=\sum_{j \neq i}^{M} \frac{1}{M^{\prime}} \mathcal{N}\left(\mathbf{R}_{i} v_{i, l}+t_{i} ; \mathbf{R}_{j} v_{j, c(j, l)}+t_{j}, \Sigma\right)
$$

It shows in typora:

But in Notion, the subscript _  seems to be parsed as * or null

Same as #31, I will look at tonight

Closing as duplicate, working on in #31


Streamed Response is freezing without explanation

I'm using Firebase Functions and the below code:
const events = await client.listChatCompletions(
  deploymentId,
  arrMsgs,
  {maxTokens: 1000});

 for await (const event of events) {
          for (const choice of event.choices) {
            const delta = choice.delta?.content;
            if (delta !== undefined) {
              buffer += delta;

             // I know it's an inefficient solution for the stream. Firebase Functions is lacking.
              admin.firestore().collection("buffers/" + uid+ "/buffer/").doc("lastMsg").set(
                  {"msg": buffer});

            }
            console.log("choice " + choice.delta?.content);
            //console.log("finish reason " + choice.finishReason);
            if (choice.finishReason === 'stop') {
              res.status(200).send(buffer);
            }
          }
        }

I ask a long answer - Some times the answer will be complete. Yet half of the times, the answer will stop streaming (tested also with console.log)
All previous choice.finishReason == null
So it just hangs, until the timeout.
It's very similar to the test code, so I fail to understand if it's an Azure side problem or not.

Hi @mikron123 thanks for filing this report. I think what you're describing is directly related to https://github.com/Azure/azure-sdk-for-js/issues/26348. The hanging happens most likely because your code needs a finishReason first which may not be returned by the API. This is a known bug in the API.

Hi @mikron123, I am going to close this for now but please feel free to share any more questions you have.


What's with the assert in the examples?

Description
I am wondering what the purpose of the assert respose['ok'] statements in the official examples is.
All official examples for using the slack WebClient have an assert statement checking that ok is true (see below for an example). But depending on how the library works it should either not be needed or it's a bad choice.
When the API returns an error the library is always raising a SlackApiError exception and the assert statement is never reached.
Or should the library not be thorough and users still need to check ok in their code than an assert statement is a bad choice, because it will usually be optimized out in production.
So can I rely on exceptions from the library for every API call - then I would not need the assert - or do I need to put a proper check in my code (e..g, if not response[ok]: raise ...) ?
Example from readme.md:
import os
    import slack

    client = slack.WebClient(token=os.environ['SLACK_API_TOKEN'])

    response = client.chat_postMessage(
        channel='#random',
        text="Hello world!")
    assert response["ok"]

What type of issue is this? (place an x in one of the [ ])

[ ] bug
[ ] enhancement (feature request)
[x] question
[ ] documentation related
[ ] testing related
[ ] discussion

Requirements (place an x in each of the [ ])

[x] I've read and understood the Contributing guidelines and have done my best effort to follow them.
[x] I've read and agree to the Code of Conduct.
[x] I've searched for any related issues and avoided creating a duplicate issue.

Thank you for taking the time to send this feedback. Also, I'm sorry about our late response here.
You're right. When the ok value in an API method response is False, the WebClient always raises an exception named slack.errors.SlackApiError. The assert snippets in README don't make sense. I will fix the snippets in README.
import os
import slack

try:
    client = slack.WebClient(token=os.environ['SLACK_API_TOKEN'])
    response = client.chat_postMessage(
        channel='#random',
        text="Hello world!")
    assert response["ok"] # never come here if the ok is False
    assert response["message"]["text"] == "Hello world!"
except Exception as e:
    print(f"Got an exception here: {e}")
    raise e

$ python app.py
Got an exception here: The request to the Slack API failed.

The server responded with: {'ok': False, 'error': 'not_in_channel'}
Traceback (most recent call last):
  File "app.py", line 17, in <module>
    raise e
  File "app.py", line 10, in <module>
    response = client.chat_postMessage(
  File "/path-to-python/site-packages/slack/web/client.py", line 559, in chat_postMessage
    return self.api_call("chat.postMessage", json=kwargs)
  File "/path-to-python/site-packages/slack/web/base_client.py", line 171, in api_call
    return self._event_loop.run_until_complete(future)
  File "/path-to-python/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/path-to-python/site-packages/slack/web/base_client.py", line 226, in _send
    return SlackResponse(**{**data, **res}).validate()
  File "/path-to-python/site-packages/slack/web/slack_response.py", line 176, in validate
    raise e.SlackApiError(message=msg, response=self)
slack.errors.SlackApiError: The request to the Slack API failed.
The server responded with: {'ok': False, 'error': 'not_in_channel'}

Fixed by #651


Enhancement request:  Support for multiple CA's for Syslog-TLS

At present syslog-tls can use a single CA cert for server authentication. Ideally each of the syslog clients would be singed by the same CA so that a single CA cert would be sufficient, but this isn't always practical. Few or no enterprises are designed perfectly, and bringing down each system to change their certificates in order to onboard them with a syslog server could be a serious inconvenience, or even break existing architecture.
Assume 1000 ESXi hosts, 100 firewalls, or what have you. If they are self signed or signed by a different central CA to that of the CA configured on SC4S that means they can't authenticate with SC4S over syslog-tls.
To address these issues, it would be very helpful if a CA cert for syslog-tls could be configured either per TCP socket on the SC4S instance, or per sourcetype.

For now, this can be worked around by using an instance of the container per TLS cert pair.

Please reopen the ticket if you still need this feature to be enhanced


Chained exceptions not fully displayed in client process

When handling an exception in Python 3, one can raise a new exception without discarding the original traceback.  (I'm referring to "exception chaining" syntax: raise new_exc from orig_exc.)
In dask, exceptions from worker threads are pickled and shown in the client process, which is nice.  But chained exceptions aren't shown correctly.
When using the single-threaded scheduler, the chained exception traceback appears correctly.  The following program produces a chained exception:
import dask
import dask.bag as db

def raise_twice(x):
    try:
        raise RuntimeError("Oops 1")
    except Exception as ex:
        raise RuntimeError("Oops 2") from ex
    return x

dask.config.set(scheduler='single-threaded')
db.from_sequence([1,2,3]).map(raise_twice).compute()

Here's the traceback:

Chained traceback
Traceback (most recent call last):
  File "test.py", line 6, in raise_twice
    raise RuntimeError("Oops 1")
RuntimeError: Oops 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 13, in <module>
    db.from_sequence([1,2,3]).map(raise_twice).compute()
  File "/prefix/lib/python3.6/site-packages/dask/base.py", line 156, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/prefix/lib/python3.6/site-packages/dask/base.py", line 397, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/prefix/lib/python3.6/site-packages/dask/local.py", line 542, in get_sync
    return get_async(apply_sync, 1, dsk, keys, **kwargs)
  File "/prefix/lib/python3.6/site-packages/dask/local.py", line 488, in get_async
    fire_task()
  File "/prefix/lib/python3.6/site-packages/dask/local.py", line 484, in fire_task
    callback=queue.put)
  File "/prefix/lib/python3.6/site-packages/dask/local.py", line 531, in apply_sync
    res = func(*args, **kwds)
  File "/prefix/lib/python3.6/site-packages/dask/local.py", line 277, in execute_task
    result = pack_exception(e, dumps)
  File "/prefix/lib/python3.6/site-packages/dask/local.py", line 272, in execute_task
    result = _execute_task(task, data)
  File "/prefix/lib/python3.6/site-packages/dask/local.py", line 253, in _execute_task
    return func(*args2)
  File "/prefix/lib/python3.6/site-packages/dask/bag/core.py", line 1562, in reify
    seq = list(seq)
  File "/prefix/lib/python3.6/site-packages/dask/bag/core.py", line 1722, in map_chunk
    yield f(*a)
  File "test.py", line 8, in raise_twice
    raise RuntimeError("Oops 2") from ex
RuntimeError: Oops 2

But when using the processes or distributed scheduler, the original traceback is omitted.  Only the newly-raised exception is shown:

Incomplete traceback
Traceback (most recent call last):
  File "test.py", line 13, in <module>
    db.from_sequence([1,2,3]).map(raise_twice).compute()
  File "/prefix/lib/python3.6/site-packages/dask/base.py", line 156, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/prefix/lib/python3.6/site-packages/dask/base.py", line 397, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/prefix/lib/python3.6/site-packages/dask/multiprocessing.py", line 192, in get
    raise_exception=reraise, **kwargs)
  File "/prefix/lib/python3.6/site-packages/dask/local.py", line 501, in get_async
    raise_exception(exc, tb)
  File "/prefix/lib/python3.6/site-packages/dask/compatibility.py", line 111, in reraise
    raise exc.with_traceback(tb)
  File "/prefix/lib/python3.6/site-packages/dask/local.py", line 272, in execute_task
    result = _execute_task(task, data)
  File "/prefix/lib/python3.6/site-packages/dask/local.py", line 253, in _execute_task
    return func(*args2)
  File "/prefix/lib/python3.6/site-packages/dask/bag/core.py", line 1562, in reify
    seq = list(seq)
  File "/prefix/lib/python3.6/site-packages/dask/bag/core.py", line 1722, in map_chunk
    yield f(*a)
  File "test.py", line 8, in raise_twice
    raise RuntimeError("Oops 2") from ex
RuntimeError: Oops 2

I haven't investigated whether or not this is a problem in dask per se, or if it's due to a problem in cloudpickle.

Thanks for the nice reproducer. Was able to reproduce it with cloudpickle alone and filed upstream as issue ( https://github.com/cloudpipe/cloudpickle/issues/248 ).

This issue remains live, but there is nothing that we can do about it on our side.

IIRC we tracked this down to CPython itself in the cloudpickle issue linked above. FWIW would be ok closing this as it is not a Dask issue. That said, I do appreciate the effort everyone put in to isolate this one.

PR ( https://github.com/dask/distributed/pull/3430 ) may be relevant.

This is fixable through tblib >= 1.6; I'd suggest reopening the ticket

I think we can leave this closed since we are already tracking this through issue ( https://github.com/dask/distributed/issues/3158 ) and a fix is already in flight (via the PR above). Though please correct me if you think the issue is different somehow. 🙂

The above PR won't fix dask.multiprocessing

Makes sense. Would you be comfortable outlining the recommended course of action then?

I can take care of it (can't promise to be fast though)

That would be great if you can. 😄
If not, just some guidance for someone to pick up later would also be good. 🙂

literally just need to copy what's been done in https://github.com/dask/distributed/pull/3430 in a slave-side wrapper function inside the multiprocessing.Pool. tblib would become an optional dependency. Probably a good idea to raise a warning if it's not installed and you would benefit from it.

This does appear to be working correctly for multiprocessing
import dask
import dask.bag as db

def raise_twice(x):
    try:
        raise RuntimeError("Oops 1")
    except Exception as ex:
        raise RuntimeError("Oops 2") from ex
    return x

dask.config.set(scheduler='multiprocessing')
db.from_sequence([1,2,3]).map(raise_twice).compute()

Exception chain
Traceback (most recent call last):
  File "<ipython-input-2-545e40ee9fda>", line 6, in raise_twice
    raise RuntimeError("Oops 1")
RuntimeError: Oops 1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/bert/.conda/envs/dask-dev/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-545e40ee9fda>", line 12, in <module>
    db.from_sequence([1,2,3]).map(raise_twice).compute()
  File "/home/bert/projects/dask/dask/base.py", line 167, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/bert/projects/dask/dask/base.py", line 447, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/bert/projects/dask/dask/multiprocessing.py", line 208, in get
    result = get_async(
  File "/home/bert/projects/dask/dask/local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "/home/bert/projects/dask/dask/local.py", line 316, in reraise
    raise exc
  File "/home/bert/projects/dask/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/home/bert/projects/dask/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/bert/projects/dask/dask/bag/core.py", line 1797, in reify
    seq = list(seq)
  File "/home/bert/projects/dask/dask/bag/core.py", line 1984, in __next__
    return self.f(*vals)
  File "<ipython-input-2-545e40ee9fda>", line 8, in raise_twice
    raise RuntimeError("Oops 2") from ex
RuntimeError: Oops 2

Not sure if this is relevant but it doesn't work for exceptions with are not from another but simply occur during handling of another exception eg.
import dask
import dask.bag as db

def raise_twice(x):
    try:
        raise RuntimeError("Oops 1")
    except Exception as ex:
        raise RuntimeError("Oops 2")
    return x

dask.config.set(scheduler='multiprocessing')
db.from_sequence([1,2,3]).map(raise_twice).compute()

Exception chain
Traceback (most recent call last):
  File "/home/bert/.conda/envs/dask-dev/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3343, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-97c5b88499b5>", line 12, in <module>
    db.from_sequence([1,2,3]).map(raise_twice).compute()
  File "/home/bert/projects/dask/dask/base.py", line 167, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File "/home/bert/projects/dask/dask/base.py", line 447, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/bert/projects/dask/dask/multiprocessing.py", line 208, in get
    result = get_async(
  File "/home/bert/projects/dask/dask/local.py", line 486, in get_async
    raise_exception(exc, tb)
  File "/home/bert/projects/dask/dask/local.py", line 316, in reraise
    raise exc
  File "/home/bert/projects/dask/dask/local.py", line 222, in execute_task
    result = _execute_task(task, data)
  File "/home/bert/projects/dask/dask/core.py", line 121, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/bert/projects/dask/dask/bag/core.py", line 1797, in reify
    seq = list(seq)
  File "/home/bert/projects/dask/dask/bag/core.py", line 1984, in __next__
    return self.f(*vals)
  File "<ipython-input-4-97c5b88499b5>", line 8, in raise_twice
    raise RuntimeError("Oops 2")
RuntimeError: Oops 2

Distributed has this same behavior, which is probably due to the fact that exceptions raised with the from syntax are stored in the __cause__ attribute however ones which are not are stored in the __context__ attribute which is not handled in [dask/distributed#3158](https://github.com/dask/distributed/issues/3158 . See PEP 3134. Either way this ticket doesn't really make any sense any more and should probably be closed.

Thanks for sharing @bernardpazio! 😄
Actually retrying this today, I find all of the scheduling modes "single-threaded", "threads", and "processes" work. 🎉
Sure let's go ahead and close. Do you mind raising any remaining tasks as new issues? 😉