[BUG] Backup is successful (as indicated on the UI) but the backup job/pod never stops running for a specific volume

Describe the bug
Very odd behavior on one of my volumes, the rest seem to back up fine and the backup job/pod finishes.
Still running:

Trying to find logs in that pod:
$ k logs pod/pvc-b58ad35f-9b80-4778-a703-315e96e9e0a9-c-m9w1pw-c-159651n6wzc -n longhorn-system
<nothing>

UI says everything is fine:

More context in Slack
https://rancher-users.slack.com/archives/CC2UQM49Y/p1596468622360500
To Reproduce
Steps to reproduce the behavior:
Hard to tell seems pretty specific...
Expected behavior
The backup job/pod to complete.
Log
Creating backups on the UI yields these errors for this volume:
Failed to backup snapshot b62520bb-be98-4dd4-afd5-20dc59f686ad with label map[KubernetesStatus:{\"pvName\":\"pvc-b58ad35f-9b80-4778-a703-315e96e9e0a9\",\"pvStatus\":\"Bound\",\"namespace\":\"default\",\"pvcName\":\"plex-config\",\"lastPVCRefAt\":\"\",\"workloadsStatus\":[{\"podName\":\"plex-64b86f76df-qw5cp\",\"podStatus\":\"Running\",\"workloadName\":\"plex-64b86f76df\",\"workloadType\":\"ReplicaSet\"}],\"lastPodRefAt\":\"\"}] for volume pvc-b58ad35f-9b80-4778-a703-315e96e9e0a9: failed to execute: /var/lib/longhorn/engine-binaries/longhornio-longhorn-engine-v1.0.1/longhorn [--url <IP_ADDRESS>:10002 backup create --dest<EMAIL_ADDRESS>--label KubernetesStatus={\"pvName\":\"pvc-b58ad35f-9b80-4778-a703-315e96e9e0a9\",\"pvStatus\":\"Bound\",\"namespace\":\"default\",\"pvcName\":\"plex-config\",\"lastPVCRefAt\":\"\",\"workloadsStatus\":[{\"podName\":\"plex-64b86f76df-qw5cp\",\"podStatus\":\"Running\",\"workloadName\":\"plex-64b86f76df\",\"workloadType\":\"ReplicaSet\"}],\"lastPodRefAt\":\"\"} b62520bb-be98-4dd4-afd5-20dc59f686ad], output , stderr, time=\"2020-08-03T18:26:36Z\" level=info msg=\"Backing up b62520bb-be98-4dd4-afd5-20dc59f686ad on tcp://<IP_ADDRESS>:10135, to<EMAIL_ADDRESS>
\"workloadsStatus\":[{\"podName\":\"plex-64b86f76df-qw5cp\",\"podStatus\":\"Running\",\"workloadName\":\"plex-64b86f76df\",\"workloadType\":\"ReplicaSet\"}],\"lastPodRefAt\":\"\"} c-m9w1pw-577f58cd], output , stderr, time=\"2020-08-03T17:42:21Z\" level=info msg=\"Backing up c-m9w1pw-577f58cd on tcp://<IP_ADDRESS>:10135, to s3://longhorn@<IP_ADDRESS>/backups\"\ntime=\"2020-08-03T17:43:21Z\" level=fatal msg=\"Error running create backup command: failed to create backup to<EMAIL_ADDRESS>for volume pvc-b58ad35f-9b80-4778-a703-315e96e9e0a9: rpc error: code = DeadlineExceeded desc = context deadline exceeded\"\n, error exit status 1"

Environment:

Longhorn version: 1.0.1
Kubernetes version: 1.18.3-k3s1
Node OS type and version: Ubuntu 20.04.1

The drives Longhorn are all using nvme and my nodes are connected via a single 10Gb link.
Additional context
I'll try to capture logs on the next schedule run.

I have the same or a similar problem, on two different clusters (rarely on my private one, where Longhorn volumes are <2G and content is pretty static, every day on a bigger one with volumes up to 500G and up to almost 200G turnover per day on a single volume). So both very small snapshots and big snapshots seem to be affected. I use MinIO for backup storage.
On both clusters, I see a log message "Cleaned up snapshot" in the pod, nothing else, only that one line.
On my private cluster, I also saw the UI reporting the backup as done.
On the big cluster, the UI does not list the backup.
When I delete the backup pod, it is recreated and then, usually it works. But sometimes it gets stuck again.

I am still encountering this issue. I have noticed this is only for large volumes as well. Another thing I noticed is my backup retention is set to 7 days, however the backups for this volume do not get deleted. Maybe there is some timeout happening on trying to remove the old backups?

I also have it. Every day, most of the time for 2 volumes.
When I delete the jobs and go to the UI, I see that Longhorn needs quite some time to merge snapshots.
It is not only about size, my guess would be that many changes over the day are more likely for larger volumes - and these many changes cause issues with cleaning up old stuff / merging old snapshots.

Everyday for me too, I get a notification from Prometheus Operator that a job is still running, so I have to manually delete it.
@yasker any thoughts on what is happening here?

@onedr0p I think you're right. Somehow the backup cleanup process or snapshot purge process cannot be completed successfully. So the backup has been recreated repeatedly. It shouldn't take extra space in the storage (S3 or NFS), but we do want to figure out what can be the reason for the cleanup to fail. Maybe for the bigger volume, backup deletion timed out.

I don't know whether this helps, but I do have the same issue as @aspettl. Multiple times a day, Longhorn CronJobs in Cluster1 get stuck in running with the the only line being
time="2020-11-05T08:20:08Z" level=debug msg="Cleaned up snapshot c-sktm7m-4dc33a35 for pvc-b10c11f2-eafc-49a1-bcb6-bc935a9d50ad"
This happens for all volumes in Cluster1. I use MinIO + Longhorn in Cluster2 to store the backups of Cluster1. Both clusters are currently running on single nodes, but don't even get close to 50% CPU or RAM (but could spike higher for seconds). Whether this is independent or separate, MinIO just as regularly stops working properly and logs the following Errors:
This one is no problem - MinIO keeps running and working
API: SYSTEM() 04/11/2020 10:33:13 Time: 09:33:13 UTC 11/04/2020 04/11/2020 10:33:13 DeploymentID: cfe660d9-34df-41d2-a3db-a78ed71d7fa0 04/11/2020 10:33:13 Error: unexpected EOF 04/11/2020 10:33:13        3: cmd/data-usage-cache.go:452:cmd.(*dataUsageCache).load() 04/11/2020 10:33:13        2: cmd/fs-v1.go:267:cmd.(*FSObjects).CrawlAndGetDataUsage() 04/11/2020 10:33:13        1: cmd/data-crawler.go:108:cmd.runDataCrawler()
These two mean problems - backups can neither be created nor read anymore
`API: PutObject(bucket=application-backups, object=backupstore/volumes/f8/e2/pvc-b10c11f2-eafc-49a1-bcb6-bc935a9d50ad/backups/backup_backup-fb4d2a32b08a43b9.cfg)
05/11/2020 09:25:45 Time: 08:25:20 UTC 11/05/2020
05/11/2020 09:25:45 DeploymentID: cfe660d9-34df-41d2-a3db-a78ed71d7fa0
05/11/2020 09:25:45 RequestID: 16448F19C9EEA207
05/11/2020 09:25:45 RemoteHost: <IP_ADDRESS>
05/11/2020 09:25:45 Host: s3-backup-1.xxx.cloud
05/11/2020 09:25:45 UserAgent: aws-sdk-go/1.25.16 (go1.13.3; linux; amd64)
05/11/2020 09:25:45 Error: write /export/.minio.sys/buckets/application-backups/backupstore/volumes/f8/e2/pvc-b10c11f2-eafc-49a1-bcb6-bc935a9d50ad/backups/backup_backup-fb4d2a32b08a43b9.cfg/fs.json: input/output error
05/11/2020 09:25:45        4: cmd/api-errors.go:1979:cmd.toAPIErrorCode()
05/11/2020 09:25:45        3: cmd/api-errors.go:2004:cmd.toAPIError()
05/11/2020 09:25:45        2: cmd/object-handlers.go:1551:cmd.objectAPIHandlers.PutObjectHandler()
05/11/2020 09:25:45        1: net/http/server.go:2042:http.HandlerFunc.ServeHTTP()
API: ListObjectsV1(bucket=application-backups)
05/11/2020 09:26:02 Time: 08:26:02 UTC 11/05/2020
05/11/2020 09:26:02 DeploymentID: cfe660d9-34df-41d2-a3db-a78ed71d7fa0
05/11/2020 09:26:02 RequestID: 16448F26830773E4
05/11/2020 09:26:02 RemoteHost: <IP_ADDRESS>
05/11/2020 09:26:02 Host: s3-backup-1.xxx.cloud
05/11/2020 09:26:02 UserAgent: aws-sdk-go/1.25.16 (go1.13.3; linux; amd64)
05/11/2020 09:26:02 Error: stat /export/application-backups: input/output error
05/11/2020 09:26:02        4: cmd/api-errors.go:1979:cmd.toAPIErrorCode()
05/11/2020 09:26:02        3: cmd/api-errors.go:2004:cmd.toAPIError()
05/11/2020 09:26:02        2: cmd/bucket-listobjects-handlers.go:407:cmd.objectAPIHandlers.ListObjectsV1Handler()
05/11/2020 09:26:02        1: net/http/server.go:2042:http.HandlerFunc.ServeHTTP()`
However, the MinIO pod does keep running all the time, no restarts, and logs these errors whenever something wants to read or write. The recurring backup problem, as well as the MinIO problem can be temporarily corrected by just deleting the corresponding pods (Longhorn CronJob pod that's stuck running and the MinIO pod).
Both clusters are running k3s v1.19.3+k3s1 on Ubuntu 20.04.1 LTS and use Longhorn 1.0.2.

@skolkmann Your issue might be a little different. Since you mentioned you're running Longhorn + MinIO on a single node cluster (and I assume Longhorn is making backup to the MinIO too), you don't need to have three replicas for the MinIO.
Also, are you using a spinning disk or SSD? It might be Longhorn experienced write latency on the physical disk level, which results in errors on the block level.

Cluster1 (VPS) runs workloads using Longhorn as StorageClass. That Longhorn instance uses Cluster2 (RS with SSD) - running MinIO + Longhorn - as backup target. Since Cluster2 is used for backups by other clusters, it does not itself store backups of the backups anywhere.

So the problem is with the Cluster2. You mentioned but could spike higher for seconds, which concerns me. The disconnection may happen when the spike happens. How many replicas you have with Cluster2 Longhorn setup?
Can you file another issue? Btw, what's RS you mentioned?

Everything is default, so Cluster2 Longhorn uses 3 replicas on the same node. By RS I mean Root Server (bare metal) and you could have already been right: it uses spinning disk (SAS) hard drive - not SSD as I said. Should have double-checked that, I'm sorry. Maybe 3 replicas on spinning disk is just too much for the system to handle? However I filed another issue #1968.

remove stale due to the incorrect exempt label

Now looking at this issue, the main issue might be caused by https://github.com/longhorn/longhorn/issues/2278 . Also https://github.com/longhorn/longhorn/issues/2507 might have a play too.
Both issues I mentioned here will be fixed in the upcoming v1.1.2 release.

Besides, there are fixes for this kind of backup creation timeout issue in #2218 as well.

I am going to go ahead and close this issue too, looks like v1.2.0 might solve these problems. I will open a new bug report if there's any other issues. Thanks all!


stuck with Spark 2.0.2. Is that a problem?

Hello and thanks for this wonderful package.
For some reasons I am stuck with Spark 2.0.2.  I wonder if this limits in any way the things I can do with rsparkling and h20. I wasnt able to figure out by reading your documentation.
Many thanks!!

Hi @randomgambit
What seems to be the issue exactly? The documentation should reference how to use Spark 2.0.2.

Will close this for now. Please reopen if you have issues.


app says "Your server is not up" when it is

I was trying to do a one-person stress test and I had the PROOF app open in 3 browser tabs. I had a proof server running and I was logged into each tab and I had uploaded a simple WDL to each in the Submit Jobs tab.
I clicked Submit Job in the first tab and it worked fine, and then clicked the same button in the other 2 tabs right away. In both tabs I got the message Error: Your server is not up! and this appeared in the app logs:
Error in : Your server is not up!
<Anonymous>
stop
stop_safe
stop_safe_loggedin_serverup
renderPrint [/srv/shiny-server/server.R#409]
func
renderFunc
output$submissionResult
runApp

But then if I go to the PROOF Server tab, in all 3 tabs, it says my server is RUNNING.
I wonder if the code is consulting a local or session variable to determine whether the server is running (when submitting a workflow), and since the server was started in another session/tab, it doesn't realize there is a running server. Maybe instead it could consult the PROOF API.
On the other hand, this is a bit of a contrived situation and I am not sure how important it is to track this down.

We're calling the rcromwell function to get the Cromwell server version running https://github.com/FredHutch/shiny-cromwell/blob/main/app/proof.R#L26 as a way to test if the server is up or not. That's within a try catch so we'd have to remove that trycatch and let the error(s) surface to see what's going on.


FIPS: Unhide flag

By unhiding the FIPS flag we effectively expose it for all customers to provision FIPS-enabled clusters.
Related: https://gitlab.cee.redhat.com/service/uhc-clusters-service/-/merge_requests/4803

/lgtm


cmd/cue: add diff support

Allow diffing between (snapshots of) previous version and current version.

Any updates on this feature?

It would be good for people to give examples of how they would like to use this functionality in a CLI.
There is an internal package for diffing CUE, which could potentially also be exposed as a Go API.

Commenting simply to add the word semantic into the mix 😄
I've taken (perhaps incorrectly) to referring to cue diff as a semantic diff, distinct from cmp and friends ("plain diff") used in unity.

Would be nice to have a diff API in cue lib. Currently I'm using Value.Decode and https://github.com/r3labs/diff to diff two cue instances in order to detect which part of the configuration has changed (based on this CI jobs are run).
However this works if the instances have only concrete values. So a similar API that can diff cue values directly  would be nice (and required in my case to be able to use the flow API)

@eonpatapon take a look at<EMAIL_ADDRESS>That is internal for now, but as we shape up the API it should be made non-internal.

A semantic diff would be really interesting. I think with diffing kubernetes files, the most awkward part is the way that the kubernetes API wants to be given a list of yaml objects but the identity of those objects is usually best defined by their name rather than their position in a list. My current idea for diffing them is to write them out into a directory tree where the name of the file is based on the name and type of the object and then using git diff. Then as long as the yaml and all of its fields and lists of named objects are sorted in some stable way, this is good enough for most kubernetes things.

Adding something of an experience report here from the world of unity.
unity tests (of cmd/cue) generally follow this rough pattern:

Ensure that evaluation of a given configuration semantically matches expectations. This is, in effect, a CUE semantic diff
Verify that output in a specific format (JSON, Yaml, etc) matches expectations. This is, in effect, a semantic diff in the output format

In the case of point one this will look like:
# CUE semantic diff
cue eval -o out.cue X
cue diff out.cue ref.cue

# JSON diff
cue eval -o out.json X
cue diff out.json ref.json

Some questions:

Is cue eval the right command here? cue eval will become cue - but does that give the intended result, as far as concreteness etc are concerned?
file.ref will need to be a complete, self-contained configuration. This might well require it to be a txtar archive?

Stepping back a bit further, we should also be able to write a semantic diff for point 2, on the basis that CUE knows about the semantics of these different formats (even to some extent the different versions of, say, Yaml, JSON).

This issue has been migrated to https://github.com/cue-lang/cue/issues/8.
For more details about CUE's migration to a new home, please see https://github.com/cue-lang/cue/issues/1078.


Tracking invalid type assertions

Before we had no-object-literal-type-assertion applied, there were a lot of suspicious casts.
These have all been labeled with comments containing the text GH#18217.

So.. I kinda hit this today on
https://github.com/microsoft/TypeScript/blob/master/src/services/services.ts#L265

I've run into an exception thrown by this line, which has a comment referencing this issue: https://github.com/microsoft/TypeScript/blob/6bcb6bb36917cf8f89d0f58f1a96ee236ca76eb4/src/services/codefixes/fixCannotFindModule.ts#L55
What's the best way to proceed? It looks like it would be relatively straightforward to add a check, but do I need to create a repro case to show it's a real issue? Should I open another issue and link to this one? Please advise.

This issue was a catch-all for eventually removing every postfix !. Please open a new issue outlining how to repro the problem you encountered. Thanks!

It looks like custom transformers (at least those hooked into the before phase) can't successfully create new get/set accessor nodes due to an invalid non-null assertion at the very top of compiler > checker > createTypeChecker > createResolver > getAllAccessorDeclarations. Reproduced in 4.8.4, but the assertion is still there on main currently.

@dannymcgee Would you mind filing a new issue for that, especially if you have a simple transformer that shows the bug? We can add those as unit tests.
(As with https://github.com/microsoft/TypeScript/issues/18217#issuecomment-744561166 this issue isn't really the right place to report isuses.)


adding custom image builder code base

Description

Release required?

[x] No release
[ ] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content

Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

/run pipeline

/run pipeline

@nupurg-ibm @bksagar I nearly missed it but you are going to need a test for this. I would put it in the other_test.go so it runs weekly. I am recommending this because not all your versions are pinned so you are running on untested versions. At least if you are running a weekly test you will be notified if something breaks. You should also consider how you plan on maintaining the versions you do have pinned in the scripts so you are not providing customers with versions that contain critical security vulnerabilities. The rest of the PR looks good to me, I won't block you from merging this but I strongly advise you implement at least one test...

@daniel-butler-irl This repo is not set up with a continuous test pipeline, so anything in other_test.go will never get executed unless its added


redo(ticdc): limit worker number in redo reader

What problem does this PR solve?

Issue Number: close #xxx
What is changed and how it works?
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note
Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

/run-all-tests

/run-engine-integration-test
/run-dm-integration-test

/merge

/run-integration-test

/run-dm-integration-test

/run-dm-integration-test

/run-dm-integration-test

/run-dm-integration-test


unnecessary  Comments were adding

When these packages are incorporated to application some unusual comments are being added to code
here is an example : https://stackoverflow.com/questions/44800866/mvcdonutcaching-how-can-i-remove-the-enddonut-comment
<!--Donut# SOME RANDOM GIBBERISH #-->Content Of partial here<!--EndDonut-->

@moonpyk  I have the same question, @HappyCoder99 already asked.
It is also 700+ bytes checked in the notepad. Is it required?
If it is then can we reduce it to SHAHash

I can seem the same stuff while I am running on my local machine with Visual Studio. Once it is on the server it does not show this any more?


Filtering may not be working 100%

If I filter by e.g. Sorting, React-Table is the first time (not react-window),  and then in the 3rd row, there is a blank React-Table? I'm assuming that the first one should be React-Window?

Hmm, I'm not able to reproduce this. I did just refactor the filtering and the code is here if it looks good to you: https://github.com/statico/jsgrids/blob/main/components/FilterBar.tsx#L150-L172

Hmm.  Sorry, I can't recreate it now either.  I was messing around in the console so perhaps I messed up the DOM.


Update UI

Current UI is working fine, but it can be better. Try to make some changes to it so that it can look better.

I Shall Suggest to add a Search View Instead of Edittext in the SearchActivity.

Nice idea, I would be happy to merge a PR of the same

Currently rank list is generated for CGPI only. Try to make it semester wise also i.e. rank list for any semester based on SGPI of that semester.

@RamolaWeb backend API and database is available here:
https://github.com/shubham12541/NITHResultWebApp/blob/master/server.js (API in node)
https://github.com/Nithmr/NithResultAppBackend (database and API in php)


chore: bdd-ghfjxy-import-1568075245 to 0.0.1

chore: Promote bdd-ghfjxy-import-1568075245 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


feat: tarball url redirect

Type: feature
The following has been addressed in the PR:

There is a related issue? verdaccio/monorepo/issues/250
Unit or Functional tests are included in the PR: Yes

Description:
Part of verdaccio/monorepo/issues/250, redirect tarball URL based on config.tarball_url_redirect. Major usage is to rewriting the redirect tarball URL to a CDN server.

@juanpicado

It majorly benefits the S3 back-end to swap the URL to use the S3's network infrastructure, (see my issue https://github.com/verdaccio/monorepo/issues/250 or other's issue https://github.com/verdaccio/verdaccio/issues/1620).
It can also use to separate the domain name for regular request (endpoints) and the static files (tarballs). i.e. example.com/endpoints... and static.example.com/tarballs... a trivial practice for web application for various reason. Like put entire static.example.com under CDN, or use a different web server to serve static files.

The first case is surely a specific (but common) scenario, but there's no way to implement without changing the verdaccio main repository. If we reject for this reason, maybe we could think about offering a alternative solution?
The second case is a common practice for web application.

If we reject for this reason, maybe we could think about offering an alternative solution?

Not necessarily a rejection, but rather start a discussion about what we are trying to solve and which other issues this creates.
As I exposed him worth to research whether a filter plugin might solve this https://github.com/verdaccio/verdaccio/issues/1620#issuecomment-569884226  (example: https://verdaccio.org/docs/en/dev-plugins#api-1)
I have the same concerns as @DanielRuf exposes here https://github.com/verdaccio/verdaccio/issues/1620#issuecomment-569441371  , once this flag is enabled you lose the ability to use the proxy (uplinks) for all packages since I see the flag tarball_url_redirect applies all request (independently what you have configured in the package access.  This assumes your CDN contains all tarballs in npmjs.
The unique way to go I see is you filter the packument then the package manager will download from the location you provide, then is not required any change at Verdaccio, you can read more here about it https://github.com/verdaccio/verdaccio/pull/1161.
"dist": {
    "shasum": "0eb570293ca87b272a0f3a62b50adb9a5f0a8f1a",
     "tarball": "https://{custom}/object-assign/-/object-assign-0.1.1.tgz"
},
``

once this flag is enabled you lose the ability to use the proxy (uplinks) for all packages

I see. the downloadStream will invoke storage.getTarball, which will download tarball from the upstream to local storage, then serve from local. Indeed the redirect flag brokes the uplink feature. Maybe I should let it finish the download, then return to CDN URL.

As I exposed him worth to research whether a filter plugin might solve this #1620 (comment) (example: https://verdaccio.org/docs/en/dev-plugins#api-1)

So filter can change the meta data. But seems filter was only invoked by _syncUplinksMetadata, so it's limited to uplink?

The unique way to go I see is you filter the packument then the package manager will download from the location you provide, then is not required any change at Verdaccio, you can read more here about it #1161.

The intention is to let user download from S3/CDN infrastructure directly.
npm fetches meta data => verdaccio returns meta data
npm fetches tarball from verdaccio URL => verdaccio returns S3 URL (HTTP 302)
npm fetches from S3 URL => S3 serving tarball

The benefit of HTTP 302 is that, verdaccio still get the access, so it can trace the DL stats (not available yet, but a popular feature request). If I change the tarball URL using filter, verdaccio will not be able to trace the access.

Update PR to handle uplinks

If the local tarball exists, return 302 based on config.tarball_url_redirect
If not, fall back to the regular download stream.

This would make my life so much easier, any chance of it going in?

This would make my life so much easier, any chance of it going in?

Your point of view is valuable, perhaps helps to move this on.

Sure, basically, we have users around the world who need to access our internal npm registry. Currently we are running multiple instances of this in different parts of the world, and have to jump through some hoops regarding auth/vpn/internal networks etc.
Ideally we'd be able to have the actual download of the tarballs happen via cloudfront signed urls. This would mean that we would only need to run a single instance of  verdaccio, while still having the tarball downloads for users in India come from the cloudfront edge cache in Mumbai, etc.
I realize this PR doesn't support signed urls for the redirect, but it's a big step in the right direction.

We use the a javascript config, so if there was also config.tarball_url_redirect_signing_func option that let you pass a function something like:
const config = {
    tarball_url_redirect: 'http://d111111abcdef8.cloudfront.net/${packageName}/${filename}',
    tarball_url_redirect_signing_func(tarballUrl) {
        const signedUrl = // use aws sdk to generate signed url
        return signedUrl;
    }
}

That would let us do what we want, as described above.

@gordlea Thanks for your comment. I didn't realize that the verdaccio configuration  supports js format? If so, it can be simplified to
const config = {
    # The tarball_url_redirect can be a template string.
    tarball_url_redirect: 'http://d111111abcdef8.cloudfront.net/${packageName}/${filename}',

   # The tarball_url_redirect can be a function, takes packageName and filename and returns the url
    tarball_url_redirect(packageName, filename) {
        const signedUrl = // use aws sdk to generate signed url
        return signedUrl;
    }
}

It's been months since my last update, I'd like to summarize the status of the PR.
Is the request useful?
Yes, it is necessary if you want to work with modern content delivery network infrastructure.
Will this PR break the uplink feature?
No. It only redirects the URL if the local tarball file exists. Otherwise, it will fall back to the default logic to query uplinks.
Can it be implemented using the filter plugin?
With a filter plugin, you can hijack the packument and modify it. However, it seems the filter plugin is only applied to the uplinks. The only method invokes filter.filter_metadata is _syncUplinksMetadata. If so, it's not useful here.
Even if a filter plugin applied to all packages, the approach of modifying the packument directly to change the tarball URL will make it hard to implement download stats feature like #1201 or #1489, which requires all download requests go through verdaccio at the first place.
route.get('/:package/-/:filename', can('access'), function(req: $RequestExtend, res: $ResponseExtend): void {
    // Update stats first
    UpdateStats(...);
    // Serve the tarball later
    downloadStreamOrRedirect(req.params.package, req.params.filename, storage, req, res, config);
  });

Any other options?
Plugin-middleware seems another option if there's a way to get the local storage instance. If we do this, then the download stats feature have to be implemented as a middleware plugin as well, since it should process before the tarball redirect.
Whether it worth doing more research here depends on how bad to put this PR into the core system.

@gordlea Thanks for your comment. I didn't realize that the verdaccio configuration  supports js format? If so, it can be simplified to
const config = {
    # The tarball_url_redirect can be a template string.
    tarball_url_redirect: 'http://d111111abcdef8.cloudfront.net/${packageName}/${filename}',

   # The tarball_url_redirect can be a function, takes packageName and filename and returns the url
    tarball_url_redirect(packageName, filename) {
        const signedUrl = // use aws sdk to generate signed url
        return signedUrl;
    }
}

Any change you might add this to the PR?

@favoyang please take a look at: https://github.com/favoyang/verdaccio/pull/1

@gordlea your patch merged, also synced with the master branch.
Next, shall we continue the discussion like https://github.com/verdaccio/verdaccio/pull/1688#issuecomment-619322689?

@gordlea your patch merged, also synced with the master branch.
Next, shall we continue the discussion like #1688 (comment)?

Great, sounds good.

Just FYI I'm going to be trying this branch out in production soon.

I'm using it on production (before your contribution) for a few months, works for me.

We've been running this in production for about 3 weeks with no issues now.

We've been running this in prod since May 2020 now, with no issues. This PR is so powerful, it basically lets you create a global CDN from a single verdaccio instance.

We've been running this in prod since May 2020 now, with no issues. This PR is so powerful, it basically lets you create a global CDN from a single verdaccio instance.

Thanks for the feedback, I added this PR to 5.x just to be able to track it. Just fyi, I haven't discarded the idea, just :) you know, lack of time.  The next major is more flexible for new ideas, please, check 5.x when you guys have time.

I changed the base branch to 5.x. I guess this would be a more stable branch to track with.


HttpInterface "data" should not include query params

According to the docs, query parameters shouldn't be included in data, only form params or the raw request body should be in there.
Logging the raw request body would probably be tricky because you'd also need some way to exclude certain requests (like a file upload), and/or limit the body size.
I'm not sure how to do it, but I image the form params can be pulled from ServletRequest without the query params

I'm also running into this. It's highly confusing to see the query string expressed in a JSON like manner as if it were the request body:
{
  argument: [
    value
  ]
}

@ColinHebert it seems the reason for this is to report the request body when a form is submitted, correct? If so, would it be possible to only use this behaviour if the request is form encoded?


GROOVY-8894: Improve performance of DGM `leftShift(OutputStream, Inpu…

…tStream)`

If you had a multi-threaded Swing app that was displaying tail of a file, wouldn't Thread.yield be relevant?

The inputstream is not thread-safe.

If it is shared by multi-threads, it should be synchronized, so it will be read by only one thread at the same time. Once it is read, the count is -1 or greater than 0;
If each thread has its own inputstream, i.e. not shared by other threads, the private inputstream will be read util end of stream reached;

To sum up, as you can see, the scenario that the count is 0 will not happen.

Your analysis seems correct. I tried to go back in the commit history and looks like it is lost why that line was added. It's over 10 years old - perhaps very old JDK versions had different behavior. In any case, I think we can ignore it and go with your change.


Flush cached realm descriptions when stream_id or description is changed

The cached rendered and text realm descriptions were not cleared when the description of a realm is changed. This PR fixes that bug.
Testing Plan:

Open the organisation login page in an incognito window, and note the description shown in the login page
Login as admin in another window, and change the organisation description
Reload the login page in the incognito window and notice that the description has changed.

Similar test can be performed for the og:description field too.

Nice, merged, thanks @punchagan!


bump json-schema-ref-parser version

Addresses #2385
Summary
Bumping the json-schema-ref-parser version brings in the changes required to better resolve all external file refs.
Checklist

The basics

[x] I tested these changes manually in my local or dev environment

Tests

[ ] Added or updated
[x] N/A

Event Tracking

[ ] I added event tracking and followed the event tracking guidelines
[x] N/A

Error Reporting

[ ] I reported errors and followed the error reporting guidelines
[x] N/A

This was released in version 5.4.0.


Do not set content type when request body is blank in old net package

This checks to see if there is body content before setting the content type to application json in the OLD net package. The new net package already handles this correctly.
This is necessary as certain load balancers reject blank requests with application json types to prevent chunked requests. On many load balancers (akamai) this is the default behavior.
How Often Will This Functionality Be Used?
Every old command that can have a blank body will benefit from this change. Specifically we encountered difficulty with the bind route service command.
Possible Drawbacks
I'm not aware of any possible drawbacks.
Why Should This Be In Core?
n/a

Related issue #1318

Hey @jberkhahn please when you folks can, test this again and let us know if the fix works. We'll pull it in once we have confirmation, thanks again.

@andrew-edgar for tracking purposes


Please remove predefined files

Every time when i create file like:
mydir/string.hpp
mydir/basic.hpp
...
etc file names which are possibly predefined by clangd
everything breaks:

And its fixed next moment after renaming file into something else:

Possibly related?
https://github.com/clangd/clangd/issues/440

Please share clangd logs for further diagnosis.


README documentation - first example - image.png needs to be image.jpg

In the first example, for file.html:

When it should be

As a beginner to understanding loaders, this slight typo really threw me off. :)

Hi, seems to be my bad here, but can you explain little more detailed what threw off please :)

From the README doc:

Notice the src attribute points to image.png. I believe it should point to image.jpg because ...
In the proceeding example:
require("html!./file.html");

// => ''

The src attribute is updated accordingly, and it has a jpg extension. That's what threw me off, because originally it had a png extension.

kk 👍  i will make the examples consistent

Hi,
I believe that only the compiled example should be changed since after this instance in the code it clearly always references png extension with the same hash (a992ca).

require("html!./file.html");

// => ''

require("html?attrs=img:data-src!./file.html");

// => ''

require("html?attrs=img:src img:data-src!./file.html");
require("html?attrs[]=img:src&attrs[]=img:data-src!./file.html");

// => ''

I've made the changes in #86 if that's ok with you @michael-ciniawsky.

Pulled it in, LGTM, let me know if any others changes.

@andreicek thx 👍


update target mean discretizer

Hi @Morgan-Sell
I did some work on the MeanDiscretiser:

renamed class
replaced inheritance
expanded docstrings
added a few more tests
expanded documentation
added class to readme

Also, rebased main to get the latest changes, that is why you see all the below irrelevant commits. If you sync your repo you probably will be able to see my changes.
There are 2 small bits that remain doing and then we are good to merge:

[ ] on the new class MeanDiscretiser, add logic to fill in the attribute binner_dict_ (see my comment in # TODO)
[ ] on the user_guide MeanDiscretiser.rst just add the output of the binner_dict once the logic is updated (also flagged with at #TODO)

Would you be able to do that when you are back from hols?

Hi @solegalli,
I'm confused I thought we weren't moving forward with this transformer. Why the change of heart?


Update writing-bots.md.

The second commit is based on confusion over fixture implementation:
https://chat.zulip.org/#narrow/stream/integrations/subject/beeminder-bot/near/482364

Looks good, merged, thanks @roberthoenig!


use RegExp in .gitignore

Now works:
.swp
.DS_Store
public/app/model/.js

Lovin it :)
Would you take a second look at this? I updated some code recently and your pull no longer merges cleanly. Thank you!

sorry, a'm new with git
is that commit right?

Sorry& i'm new with Git.
Is that commit right?

added this functionality in 0.2.2, cheers!
D

Finally! ::)
But dependency 'glob' added (
2011/11/29 David Trejo
<EMAIL_ADDRESS>
added this functionality in 0.2.2, cheers!
D

Reply to this email directly or view it on GitHub:
https://github.com/DTrejo/run.js/pull/11#issuecomment-2941029

Yea, I didn't want to re-write globs. It still works for you, correct?


fix panic if logged in account is deleted

truly impressive work in this pullreq

Argh why is this possible


Dapr GRPC proxy connection leak

In what area(s)?

/area runtime

What version of Dapr?

1.1.x

Expected Behavior
Proxying GRPC calls through dapr sidecar doesn't open many connections to destination app during load
Actual Behavior
During load, dapr sidecar is opening many connections to the destination app causing high TCP usage and memory leak in our .NET Core application. When I remove the dapr sidecar and make a direct connection between the server and client, I only have a single connection open on port 8081 (HTTP2 traffic).
dotnet-monitor

ss

Steps to Reproduce the Problem
Proxy GRPC calls through dapr sidecar with high load
Possible related issue
https://github.com/dapr/dapr/issues/4937

So to be clear, in the following call chain: App 1 --> Dapr 1 -------> Dapr 2 --> App 2
You see connections being opened and not re-used in the path of Dapr 2 --> App 2?

Yep, that is exactly right @yaron2 but there is some connection reuse between Dapr 2 --> App 2. If I am doing a single request at a time in something like Postman, connection count stays at 1. As soon as I throw JMeter with 10 threads at it, connection count grows steadily, but it doesn't increase by 1 for every request.

Does the connection count continue growing after you've finished opening the 10 threads with JMeter?

It stops growing but the connections remain open.

Ok thanks a lot of the info, will check this out. Do you have any special connection max age/timeout settings on App 2's server?

I have this configured to keep alive the HTTP2 connection so it doesn't die while idle. I tried without this logic and still saw the same behavior with dapr sidecar --> App 2
private static readonly SocketsHttpHandler _handler = new SocketsHttpHandler
{
    PooledConnectionIdleTimeout = Timeout.InfiniteTimeSpan,
    KeepAlivePingDelay = TimeSpan.FromSeconds(60),
    KeepAlivePingTimeout = TimeSpan.FromSeconds(30),
    EnableMultipleHttp2Connections = true
};

I believe i'm able to reproduce this locally.. tracking this for 1.12 @jsheetzmt.

Awesome, thanks for the update @yaron2!

@jsheetzmt I've made some progress and am able to correlate new connection creation to the number of parallel requests coming in to the app. Can you provide more details on the concurrency parameters for your JMeter test?

With the 10 threads, I was just letting it run indefinitely and would see a new connection every 100 requests or so. Each thread sends one request to the app, gets the response and then repeats the test.

And how long would it take to for 100 requests to come back?

@jsheetzmt it looks like you're seeing a new connection opening up due to this setting: https://github.com/dapr/dapr/blob/e3032618c6c350d9ee56bbf98dc17625b329ef3b/pkg/grpc/pool.go#L32.
Essentially Dapr would open a new connection if the current one holds 100 streams in a Dapr to App connection.

Interesting. Do you know why the sidecar is opening a new stream and not reusing an existing one?

When I change App 1's URL to go to App 2 directly (not dapr sidecar), there's only a single connection open on App 2.

It looks like we're treating every request as a stream when incrementing the counter. cc @ItalyPaleAle to chime in on that.

@yaron2 Would you expect connections to close after a certain period of a time? Right now, it seems like connections ramp up and never ramp down. Would always like a min connection of 1 to keep the connection warm

@yaron2 Would you expect connections to close after a certain period of a time? Right now, it seems like connections ramp up and never ramp down. Would always like a min connection of 1 to keep the connection warm

We have a max connection age of 3 minutes. If there's keep alive enabled on your app server, it might mess with that.

Okay, I might have to play around with that some more. We have keep alive on our clients but not the server. Even when I disable the client's keep alive, I am not seeing total connections drop back down.

It looks like we're treating every request as a stream when incrementing the counter. cc @ItalyPaleAle to chime in on that.

In the context of grpcMaxConcurrentStreams, "stream" refers to HTTP/2 streams, not to gRPC streaming RPC. Each gRPC RPC uses a HTTP/2 stream underlying (even unary RPCs use a stream in HTTP/2)

So the next question is why does Dapr dial a new connection to talk to the app every for 100 RPCs per connection? When a Dapr SDK talks to Dapr via gRPC, or when users open native gRPC clients, or even when users use gRPC without Dapr, this logic isn't present.

@yaron2
It's been a while so I don't recall 100% of details but from memory, and from some refreshing Google searches, it was probably related to:
First. The need to ensure better load balancing across multiple apps. Before we were closing connections right after they were used, so each RPC was creating a new connection so it was connecting to (possibly) a different sidecar. If we keep connections open, the worry was that it may not load balance. However, in hindsight I wonder if this was some sort of premature optimization and maybe the gRPC SDK does that already?
Second. From the gRPC docs

However, in hindsight I wonder if this was some sort of premature optimization and maybe the gRPC SDK does that already?

This should be done by gRPC based on our dns:// schema in combination with the max connection age, that should most probably be reduced from 3m to a much lower value.
The 2nd point of the gRPC docs is relevant, however I can confirm (and also @jsheetzmt) that if you open a client/server dial, you do not see additional TCP connection beyond 100.
My conclusion is this: if load balancing isn't affected, we should default to remove this logic by default and make it configurable to allow users to handle special cases that may suffer due to queuing.

This should be done by gRPC based on our dns:// schema in combination with the max connection age (which should most probably be reduced from 3m to a much lower value).

Yes, using dns:// causes gRPC to round-robin when establishing a connection. I am just not sure what happens if the connection is not closed every time (like it used to in the past): will it still send each request to a different app?
Some testing may help figuring out if the additional pooling we're doing is needed or not.

Some testing may help figuring out if the additional pooling we're doing is needed or not.

Indeed, but for Dapr to app channel where there's expected to be a single endpoint for the app (most of the time, anyway), this should likely skip this logic by default and be made configurable. Testing is definitely needed to see if this is also the case for Dapr to Dapr communication.

We are using Dapr in Azure Container Apps, and we will have scaling so that we spin up additional replicas during load and ideally that traffic is load balanced across the replicas.

Indeed, but for Dapr to app channel where there's expected to be a single endpoint for the app (most of the time, anyway), this should likely skip this logic by default and be made configurable.

Oh, yes. I was primarily fixated on sidecar-to-sidecar. sidecar-to-app can be different

We are using Dapr in Azure Container Apps, and we will have scaling so that we spin up additional replicas during load and ideally that traffic is load balanced across the replicas. I would expect each replica to have a connection open to Dapr sidecar, and the side car is still doing round-robin

That is a valid expectation

@yaron2 Pulled down the nightly and things are looking much, much better now. Thank you!


Task Assignee Reward Crash

I skimmed the logs and the only thing that stood out to me is log id 17061

Penalty removed

Perhaps there is a bug when the contributor has a "debt" and then the reward amount must subtract the debt.
Logs from the last 15 minutes:
supabase_iyybhhiflwbsjopsgaow_Get all results for issue 663 in the logs table (1).csv

Error: Error: call revert exception [ See: https://links.ethers.org/v5-errors-CALL_EXCEPTION ] (method="symbol()", data="0x", errorArgs=null, errorName=null, errorSignature=null, reason=null, code=CALL_EXCEPTION, version=abi/5.7.0)

Originally posted by @ubiquibot[bot] in https://github.com/ubiquity/ubiquity-dollar/issues/663#issuecomment-1696985103

nope, the bug here, is the front end is calling a method .symbol() in the contract to render like  etc but it's not available. But why is on this repo?

nope, the bug here, is the front end is calling a method .symbol() in the contract to render like etc but it's not available. But why is on this repo?


feat: add external-secrets to the bootstrapper

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context. List any dependencies that are required for this change.
What type of PR is this? (check all applicable)

[x] 🍕 Feature
[ ] 🐛 Bug Fix
[ ] 📝 Documentation Update
[ ] 🎨 Style
[ ] 🧑💻 Code Refactor
[ ] 🔥 Performance Improvements
[ ] ✅ Test
[ ] 🤖 Build
[ ] 🔁 CI
[ ] 📦 Chore (Release)
[ ] ⏩ Revert

Related Tickets & Documents

Related Issue # (issue)
Closes # (issue)
Fixes # (issue)

Remove if not applicable

Screenshots

Added tests?

[ ] 👍 yes
[ ] 🙅 no, because they aren't needed
[ ] 🙋 no, because I need help
[ ] Separate ticket for tests # (issue/pr)

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Added e2e-tests?

[ ] 👍 yes
[ ] 🙅 no, because they aren't needed
[ ] 🙋 no, because I need help
[ ] Separate ticket for e2e-tests

Added to documentation?

[ ] 📜 README.md
[ ] 🙅 no documentation needed

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules

So far so good:
Release of bootstrap component successful.

Works now!
Image:         ghcr.io/skarlso/mpas-bootstrap-external-secrets/external-secrets/external-secrets:v0.9.6

Closes #112

/bin/mpas bootstrap github --owner skarlso --repository=mpas-test-project --personal --registry ghcr.io/skarlso/mpas-bootstrap-external-secrets
Running mpas bootstrap ...
 ✓   Preparing Management repository mpas-test-project
 ✓   Fetching bootstrap component from ghcr.io/skarlso/mpas-bootstrap-external-secrets
 ✓   Installing flux with version v2.1.0
 ✓   Installing cert-manager with version v1.13.1
 ✓   Installing external-secrets-operator with version v0.9.6
 ✓   Reconciling infrastructure components
 ✓   Waiting for cert-manager to be available
 ✓   Waiting for external-secrets to be available
 ✓   Generating git-controller manifest with version v0.8.0
 ✓   Generating mpas-product-controller manifest with version v0.5.1
 ✓   Generating mpas-project-controller manifest with version v0.3.1
 ✓   Generating ocm-controller manifest with version v0.14.0
 ✓   Generating replication-controller manifest with version v0.8.0
 ✓   Generate certificate manifests
 ✓   Reconciling infrastructure components
 ✓   Waiting for components to be ready

Bootstrap completed successfully!

Needs https://github.com/open-component-model/git-controller/pull/73 to use the local http://gitea instance.

And this https://github.com/open-component-model/mpas-project-controller/pull/46

@phoban01 where what which? 🤣 the disable of the component? Actually I found a better way to do that. I'll just leave out of the list of components instead of having that huge if there. That will be a bit nicer. Or are you referring to something else?

PASS
ok      github.com/open-component-model/mpas/e2e        326.154s

DONE 40 tests in 343.800s

E2E tests after using the latest project and git controller.

With the switch:
✗ ./bin/mpas bootstrap github --owner skarlso --repository=mpas-test-project --personal --registry ghcr.io/skarlso/mpas-bootstrap-external-secrets --disable-external-secret-component
Running mpas bootstrap ...
 ✓   Preparing Management repository mpas-test-project
 ✓   Fetching bootstrap component from ghcr.io/skarlso/mpas-bootstrap-external-secrets
 ✓   Installing flux with version v2.1.0
 ✓   Installing cert-manager with version v1.13.1
 ✓   Reconciling infrastructure components
 ✓   Waiting for cert-manager to be available
 ✓   Generating git-controller manifest with version v0.8.0
 ✓   Generating mpas-product-controller manifest with version v0.5.1
 ✓   Generating mpas-project-controller manifest with version v0.3.1
 ✓   Generating ocm-controller manifest with version v0.14.0
 ✓   Generating replication-controller manifest with version v0.8.0
 ✓   Generate certificate manifests
 ✓   Reconciling infrastructure components
 ✓   Waiting for components to be ready

Bootstrap completed successfully!

Works with the new suggestion from Soule:
./bin/mpas bootstrap github --owner skarlso --repository=mpas-test-project --personal --registry ghcr.io/skarlso/mpas-bootstrap-external-secrets
Running mpas bootstrap ...
 ✓   Preparing Management repository mpas-test-project
 ✓   Fetching bootstrap component from ghcr.io/skarlso/mpas-bootstrap-external-secrets
 ✓   Installing flux with version v2.1.0
 ✓   Installing cert-manager with version v1.13.1
 ✓   Reconciling infrastructure components
 ✓   Waiting for cert-manager to be available
 ✓   Generating external-secrets-operator manifest with version v0.9.6
 ✓   Generating git-controller manifest with version v0.8.0
 ✓   Generating mpas-product-controller manifest with version v0.5.1
 ✓   Generating mpas-project-controller manifest with version v0.3.1
 ✓   Generating ocm-controller manifest with version v0.14.0
 ✓   Generating replication-controller manifest with version v0.8.0
 ✓   Generate certificate manifests
 ✓   Reconciling infrastructure components
 ✓   Waiting for components to be ready

Bootstrap completed successfully!

k get pods -A
NAMESPACE            NAME                                                READY   STATUS    RESTARTS   AGE
cert-manager         cert-manager-5c698fd7-gtjf7                         1/1     Running   0          2m19s
cert-manager         cert-manager-cainjector-dbc6bddff-xxs9w             1/1     Running   0          2m19s
cert-manager         cert-manager-webhook-5644bc9cf9-trrd8               1/1     Running   0          2m19s
default              external-secrets-59ddc788cd-j5lg6                   1/1     Running   0          96s
default              external-secrets-cert-controller-78f6d694cb-8hl8g   1/1     Running   0          96s
default              external-secrets-webhook-6c8c667647-kzhwm           1/1     Running   0          96s
flux-system          helm-controller-7d5b45865d-r6rmk                    1/1     Running   0          2m58s
flux-system          image-automation-controller-7549fcbf6f-p2nq8        1/1     Running   0          2m58s
flux-system          image-reflector-controller-6b9b5d444-xcbz8          1/1     Running   0          2m58s
flux-system          kustomize-controller-854b649f79-bn9vm               1/1     Running   0          2m58s
flux-system          notification-controller-6988665c99-2jtjp            1/1     Running   0          2m58s
flux-system          source-controller-79c4b99987-89mg4                  1/1     Running   0          2m58s
kube-system          coredns-5d78c9869d-bl5jt                            1/1     Running   0          9m41s
kube-system          coredns-5d78c9869d-hbzcg                            1/1     Running   0          9m41s
kube-system          etcd-ocm-test-control-plane                         1/1     Running   0          9m54s
kube-system          kindnet-jqxrf                                       1/1     Running   0          9m41s
kube-system          kube-apiserver-ocm-test-control-plane               1/1     Running   0          9m54s
kube-system          kube-controller-manager-ocm-test-control-plane      1/1     Running   0          9m54s
kube-system          kube-proxy-jkzvr                                    1/1     Running   0          9m41s
kube-system          kube-scheduler-ocm-test-control-plane               1/1     Running   0          9m54s
local-path-storage   local-path-provisioner-6bc4bddd6b-55rdn             1/1     Running   0          9m41s
mpas-system          mpas-product-controller-76fb65dfcc-vk4vl            1/1     Running   0          96s
mpas-system          mpas-project-controller-f64bd64cd-77v9k             1/1     Running   0          96s
ocm-system           git-controller-7f85df6857-9hwdv                     1/1     Running   0          96s
ocm-system           ocm-controller-7bfb75f946-9c6g4                     1/1     Running   0          96s
ocm-system           registry-8667fbb848-sgcdb                           1/1     Running   0          95s
ocm-system           replication-controller-675cbf8cb8-p5q6h             1/1     Running   0          95s

Now, trying to do one without the external-secrets component:


Pie Charts and Bar Graphs broken on maximizing

Screenshot
Without Maximize

With Maximize

Without Maximize (bar graph)

With Maximize (scroll till bottom)

[drag & drop image(s) here!]
Description
Pie Charts and Bar Graphs have broken on maximizing.
Design input
Pie Chart: It should not disappear from the screen.
Bar Graph: It should fit on the page.
@slayerjain

Is there a way to disable the maximize option in superset?

thanks for reporting! this is a known issue. if you already have a solution to fix either or both charts, we will make sure your PR is reviewed. 🙏 @vivekdixit2510

@junlincc is it possible to disable the maximise button in dashboards for the time being?

i dont think so, unfortunately. @slayerjain 🙏

worldmap is also broken

Resolved #14371


README.md: add installation instruction for Gentoo

Description
Add installation instructions for Gentoo to the README file.
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[X] Documentation (no code change)
[ ] Refactor (refactoring production code)
[ ] Other

Checklist:

[ ] My code follows the code style of this project.
[X] I have updated the documentation accordingly.
[ ] I have formatted the code with rustfmt.
[ ] I checked the lints with clippy.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

Nice, thanks!


Query youtube directly via mpv

It would be the best to actually query the youtube link directly through my player, using the youtube-dl library (i'm using mpsyt right now, the command being "mpsyt playurl", but livestreamer or mpv works too).
How could I achieve that?
Thanks you!

I don't undertant exactly what you want to do.
Do you want to use ivy-youtube to find a video, but instead open the url on browser you pass it to youtube-dl?
Can you give me more details?

Do you want to use ivy-youtube to find a video, but instead open the url on browser you pass it to youtube-dl?

Exactly. And by talking about youtube-dl, I was talking about tools which directly play the video in my media player (currently mpv). Tools like mpsyt, livestreamer (mpv support this too, with the syntax "mpv https://www.youtube.com/watch?v=vid").
It would be amazing.
Thanks you for the extension.

@troublew it's a good idea! I'll try to do it. If you have more ideas feel free to share.

@troublew I opened a PR here that (I think) implements what you need. If you want to test you can download the code or read the new readme in https://github.com/squiter/ivy-youtube/tree/support-process

Works perfecty for me, with mpv.

Anyway, thanks for the feature. It's very handy.

When I was thinking about the feature, my first idea was to create a variable that stores a lambda to make it more flexible... maybe I implement that in the future.
Anyway thanks for the idea.


Some errors of comment

Some errors of comment in the stm32f0xx_hal_gpio_ex.h file.
https://github.com/STMicroelectronics/stm32f0xx_hal_driver/blob/5eeaad10dc3816830abd9b3bfd74c6974453e907/Inc/stm32f0xx_hal_gpio_ex.h#L505
https://github.com/STMicroelectronics/stm32f0xx_hal_driver/blob/5eeaad10dc3816830abd9b3bfd74c6974453e907/Inc/stm32f0xx_hal_gpio_ex.h#L519-L520
https://github.com/STMicroelectronics/stm32f0xx_hal_driver/blob/5eeaad10dc3816830abd9b3bfd74c6974453e907/Inc/stm32f0xx_hal_gpio_ex.h#L589

Hi @weycen,
Thank you for this report. I see you have created a pull request #3 to fix the erroneous comments . Please allow me then to close this thread and to consider the pull request.
With regards,


Refactor code in pr_code_suggestions.py

Type
enhancement, documentation

Description

Enhanced the presentation of code suggestions in pr_code_suggestions.py by removing commented-out code and adjusting the HTML structure for better readability.
Updated Azure DevOps provider documentation by removing the use_repo_settings_file configuration option from both the installation guide and the usage guide.

Changes walkthrough
Relevant filesEnhancement

pr_code_suggestions.pyEnhance Code Suggestions Presentation

pr_agent/tools/pr_code_suggestions.py
Removed commented-out code for cleaner presentation of suggestions.  Adjusted the HTML structure for displaying suggestions to use rowspan for better readability.  Minor formatting adjustments for consistency.

+2/-6

Documentation

azure.mdUpdate Azure DevOps Provider Documentation

docs/docs/installation/azure.md
Removed the use_repo_settings_file configuration option from the Azure DevOps provider documentation.

+0/-1

automations_and_usage.mdUpdate Azure DevOps Usage Guide

docs/docs/usage-guide/automations_and_usage.md
Removed the use_repo_settings_file configuration option from the Azure DevOps provider documentation in the usage guide.

+0/-1

✨ PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

/review auto_approve

Generating PR code suggestions
Work in progress ...


The grand Lua pull request

For background, at the beginning of the year, we began experimenting with rewriting some of API Umbrella's core in Lua & OpenResty. That toying around has proved worthwhile, so we're moving forward with this plan. See this issue for some of the original thinking and more details: https://github.com/NREL/api-umbrella/issues/86
The short version is that these updates should be backwards compatible (there are a few small caveats noted below), the platform is faster, and this cleans up a few pain points.
There are still a few loose ends, but it's very close, so I wanted to start a pull request where we can start to document the changes.
Code Organization
This consolidates most of our separate repositories into this single git repo. Currently we have:

api-umbrella-gatekeeper
api-umbrella-router
api-umbrella-web
api-umbrella-static-site
api-umbrella-config
omnibus-api-umbrella
api-umbrella (this doesn't really store much as things exist now)

Under the new repository structure we will have:

api-umbrella
api-umbrella-static-site

The gatekeeper, router, and web projects have been merged into the api-umbrella repo. api-umbrella-config and omnibus-api-umbrella have been rendered defunct.
The reasoning behind this shift is for a couple reasons:

The gatekeeper and router are merged since all of that logic is combined inside OpenResty now.
We've increasingly seen how interconnected some of these core components are. While things still remain modularized (and you can run different pieces of API Umbrella on different servers), we are basically pulling all the core functionality together to make it easier to work on the project.

To be clear, we're not aiming towards a monolithic codebase with these changes, but we want to make the codebase easier to work on, which hopefully these code organization changes will do. As a quick example of some of the pain points with the previous setup, see this recent feature. It required coordinating 3 pull requests, with some duplicate config across projects, and the router changes having failing tests until the gatekeeper changes were merged and released.
Architecture
As outlined in https://github.com/NREL/api-umbrella/issues/86, the proposed architecture was essentially:
[ incoming requests ] => [ API Umbrella (nginx) ] => [ API Backends ]

That didn't quite pan out like that once we got to adding the caching layer. The plan had been to use nginx's built-in caching layer, but the major pitfall with that is that nginx's caching does not work when response streaming is enabled. Since response streaming is important to us, we can't use this approach for now (but if that issue gets resolved with nginx, we can revisit).
What we ended up landing on for the architecture looks like:
[ incoming requests ] => [ nginx ] => [ TrafficServer (caching) ] => [ nginx ] => [ API Backends ]

So while we still have the nginx sandwich we were trying to simplify, things are still dramatically better, since we remove the multiple nodejs processes, and the nginx instances contain all the relevant configuration in the same instance (rather than trying to have to sync config changes between the independent nodejs processes and the nginx backend router).
The switch to TrafficServer for caching was due to it performing a fair bit better than Varnish in this sandwich situation, and allowing us to still achieve sub-millisecond overhead (to be clear, both Varnish and TrafficServer perform very well as caches, but TrafficServer seems to have a lower proxy overhead on a per request basis). We've also been eying TrafficServer for a while for other reasons. The primary reasons (streaming) were obviated by the Varnish 4 upgrade, but there are still some nice things that TrafficServer addresses (more flexible configuration for keep-alive settings, a hybrid disk and memory cache, so we can cache larger things on disk, support for HTTP/2, and TrafficServer is pursuing a Lua approach for plugins, which nicely aligns with OpenResty).
I experimented with Ledge a Lua-based http cache that can be embedded, but the tests didn't pan out too well (there were some proxy behavior issues that didn't match our expectations for a transparent proxy, and the performance wasn't as good). Those initial experiments were much earlier this year, so it's possible things have changed since then (Ledge is under pretty active development), but for now, we'll stick with a more established proxy solution.
Backwards Compatibility
By and large, this upgrade should be backwards-compatible with previous API Umbrella releases, despite the significant underlying changes. In nearly all cases, the proxying functionality should remain unchanged, and the internal admin APIs we provide are also unchanged. However, there are a couple of small areas where we do have backwards-incompatible changes. They mostly affect some more esoteric features that I don't think were actively being used by anyone, but it's worth noting:

Logic inside dynamic headers: On an API backend, if you set custom headers to be passed to the API backend, you could use Handlebars.js variables inside to reference other variables. The functionality remains unchanged for simple variables. However, if you used Handlebars.js logic, like an {{#if}} block, the syntax has changed since we are now just using Mustache templates for parsing. Something like {{#if headers.x-missing}}{{headers.x-missing}}{{else}}default{{/if}} in Handlebars would now become {{#headers.x-missing}}{{headers.x-missing}}{{/headers.x-missing}}{{^headers.x-missing}}default{{/headers.x-missing}}.
Global rate limits disabled by default: Previously, we had some default rate limits that applied within nginx on a per-server basis to prevent high-levels of abuse. However, these were always confusing, since they operated at a server level and were completely separate from API Umbrella's managed rate limits. These global limits have been disabled by default, but you can still turn them on via config settings if you'd like.

Analytics Logging
Gathering the log data for analytics was a particular pain point in the old stack. It had grown rather unwieldy over time with multiple queues to try to aggregate data from the multiple points in our proxy stack (eg, combining analytics from the nodejs processes and nginx). The overall approach is much, much simpler, now that everything is handled inside OpenResty. But another big improvement is that we have a much simpler approach to queuing this data using heka, which is much faster and solves potentially unbounded memory growth with disk buffering: https://github.com/18F/api.data.gov/issues/233
Performance
Basically, things are much faster and efficient. Our general goal based on prototypes had been to get the local time spent in the API Umbrella proxy down below 1 millisecond. I ended up stripping some of the initial over-eager optimizations I had done to simplify the initial code, so I think we might have crept up more in the 1-2ms range (depending on hardware). However, things are still definitely faster and more efficient than the old stack in any benchmark I've run. I also think there's quite a bit of room for further optimization if we make take a more focused look at that again (which should be easier to do now that we have all the functionality out of the way).
Resiliency
The stack should be much better at handling unexpected failures, particularly with the database going down. We now cache more things, like valid API keys locally, so even in the event of a complete database outage, API Umbrella's proxying should be able to continue to serve recent, existing users for a while (new users, or uncached users will fail, but this at least reduces the general impact of a small database outage).
We've also added a health check API endpoint that can be used to better determine the overall health of the API Umbrella platform. This could be used in load balancers to proactively remove servers that might be responding, but aren't actually able to service requests.
Process Management
One area that continued to cause pain in the old stack was how we were running the various processes that make up API Umbrella. API Umbrella is composed of multiple processes, but we want to offer an easy way to start and stop all of these services together. What we had before had grown a little unwieldy and could occasionally get in a funky state where things would not stop or start properly. All of that has been much simplified and improved.
We were previously using a combination of forever.js and supervisord, which was a bit of an odd mix, since they both do similar things (essentially, supervisor was being used for managing the processes, and forever.js was responsible for starting supervisord and responding to custom signals). We've switched to just using perp for all this. It's much lighter weight and more reliable since we replace the "api-umbrella" process with the perpboot process, and then trap specific signals we need to respond to. This helps cleanup our overly complex process tree and removes the need for us to run our own persistent process to respond to signals. It should lead to faster starts/stops, and also removes some overly-fancy logic we had that could lead to API Umbrella shutting down if a single component failed to start (which isn't ideal from a resiliency perspective).
Configuration Reloading
Despite improvements made, we still had some issues where sending a "reload" signal to API Umbrella wouldn't always reload specific config settings. Thanks to the process management simplifications made, this has been greatly simplified to better guarantee that a reload will always get the completely updated config.
DNS Changes & Keep-Alive Connections
To handle DNS changes with API backend servers, we currently end up reloading nginx. This works, but leads to hyper-frequent nginx reloads if you have a lot of API backends with changing IPs. These reloads should be zero-downtime, but I think this may lead to corner-case issues with keep-alive connections. I haven't proved this, but it's my current theory that this might be responsible for some ELB proxying issues we've potted.
In the new stack, all of the DNS changes are handled inside nginx without any reloads necessary. Overall, it's much simpler (and I'm hoping might explain/fix possible keep-alive issues). In addition, there's a path towards making the code even simpler, since a future version of OpenResty will natively support managing upstream server connections (right now we rely on an nginx plugin that works, but has a few rough edges).
Packaging Improvements
The binary packages for API Umbrella are much smaller in size since we've been able to simplify our dependencies. We've also made an effort to rely more on system dependencies wherever possible, rather than packaging all of our dependencies into our own binary. In addition to a smaller binary, this also means that when there are things like OpenSSL security updates, you only have to worry about updating the copy on your system, rather than also needing to upgrade API Umbrella.
We were previously using omnibus for building our packages (which is what led to a lot of the over-eager dependency inclusion), but now we're using a simpler standalone make process combined with fpm. Docker is being used to create packages for different distros. These changes should speed up the packaging process so it will be easier for us to release packages more frequently.
We'll also be looking to publish our binaries in real Yum and Apt repos so that dependencies can be automatically fetched (this is more of an issue for apt). This should also simplify upgrading API Umbrella.
Misc
There's probably more I'm forgetting, so I'll update this if I remember anything else. There's also more we need to outline on the roadmap related to this (shifting our APIs to Lua, unifying our testing frameworks), but this is the bulk of the big revamp we want to get out of the way.
If you've made it through this long-winded list, you'll probably see that the changes really extend beyond the switch to Lua. By taking a fresh look at things more broadly, it has allowed us to make improvements in other areas we've identified over the years. I hate to do such a massive code dump all at once in this pull request, but that's unfortunately how things turned out. However, with the major portions of the revamp out of the way, it should make things much easier to improve and iterate on moving forward. I don't take these kind of big changes (including a language switch) lightly, so hopefully we won't be doing anything like this again in the foreseeable future. But the benefits of the revamp have proved worthwhile, and we're excited about where this will let us go. To the future!

Whoa.


state is erased in pullToRefresh with PagedNotifierMixin

enable pull to refresh on custom example, then set a breakpoint: https://github.com/ftognetto/riverpod_infinite_scroll/blob/5d596555f653f201da086a7d775a3607e6df239b/example/lib/custom/custom_example_provider.dart#L63
You will see that all of the members of the state object are null.

You may need to pull to refresh a few times.
However in my app, the state is "erased" after the first pullToRefresh

Hi @h-unterp sorry for the late reply I unfortunately broken my arms, and now I'm slowly coming back :)
What's the problem here? When you pull to refresh riverpod will refresh entirely your provider so it's normal that your state is empty at that point


API Add DetectLocaleMiddleware to detect and set persisted locale information

This replaces the HandleRedirectsMiddleware. I'll make a follow up pull request (or push another commit to this branch) to handle root URL redirects if they're configured.
I've removed the "persist locale" data from the FluentState because I think we can wrap it up within the "locale" data in the state instead. If we disagree it can be added back in, but should differentiate between the CMS and frontend persistence keys.

I've removed the "persist locale" data from the FluentState because I think we can wrap it up within the "locale" data in the state instead. If we disagree it can be added back in, but should differentiate between the CMS and frontend persistence keys.

I agree with this, but I'll let you answer my question above and I'll have a think about this. :)


Clarify support for FASTQ

The web uploader and README needed to advertise/plumb through FASTQ support.

I pulled #7 into this to resolve the conflict.


WIP: Build libs separately

Picking up a much cleaned-up & destilled #26 again after some new abseil builds.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


[BUG] SCSS variables does not compile

Describe the bug
I've added into _bscore_variables.scss:
$primary: red;
$primary-rgb: rgb(10,20,30);
$body-bg: #000;

and nothing happens. Its a fresh install. I've worked with bootscore a few times before and compile scss was never an issue.
Steps to reproduce

install bootscore
install bootscore-child
delete other themes
activate bootscore-child

Screenshots and Additional Info
No response
Website link
No response

Your variables work fine on my site. Did you tried to enable dev_mode to force recompile? https://bootscore.me/documentation/theme/bootstrap-css-sass/#Developer_mode


Realtime Database stays offline and doesn't reconnect after sleeping

[REQUIRED] Step 1: Describe your environment

Xcode version: 13.2.1
Firebase SDK version: 8.15.0
Installation method: CocoaPods
Firebase Component: Database
Target platform(s): macOS

[REQUIRED] Step 2: Describe the problem
Steps to reproduce: Have the mac going to sleep. After waking up, sometimes (with really low frequency), the Realtime Database client would stay offline indefinitely, even if network conditions are good.
We have an app that uses Realtime Database for implementing Presence in Firestore. https://firebase.google.com/docs/firestore/solutions/presence. If we have the app running and the mac goes to sleep, sometimes, after waking up, the Realtime Database client doesn't reconnect.
We have strong evidence that database.reference(withPath: ".info/connected").observe reported a false value while the mac was sleeping but it never reported true again, even after waking up and having good network conditions.
If we use any operation while Realtime database is offline, like statusReference.setValue to write to the database, the operation never runs the completion callback.
Firestore is able to reconnect properly without problems but Realtime Database client is not working properly.
We saw one case that recovered by itself after a some hours (12 hours after disconnected, 2 hours after the mac woke up). But as we need the Realtime Database to implement Presence, that delay is not acceptable.
Restarting the app makes it work again.

We have a similar issue since iOS15. It seems to work 99% of the time, but once in a while, the reconnection does not happen.

This keeps happening to us. It's been like 3 times I hit this myself since I reported the issue. Our users are hitting it a lot too.

Hey all, apologies for the slow response. Do you have any reliable repro steps or are you able to enable logging and share the logs when this bug occurs?

This is happening to me as well, running locally at the moment, haven't tested this issue in prod, but every time my macbook sleeps, the realtime connection is ended and I have to refresh the browser to reconnect. I am using for user presence as well so it becomes an issue when any mac users in my app encounter this bug expecting to see others online to communicate.

Hey @morganchen12, sorry about the delayed response on my side this time. I couldn't reproduce it myself but we have reports that it is still happening to users. Unfortunately we don't have any reliable repro steps. When you say enable logging is just doing Database.setLoggingEnabled(true)? I will try that. Let me know if there's anything else we can try. Thanks.

Yep, or by calling this method.

Ok, I could reproduce it and collect logs and other information!
In my experience, it is more likely to hit this issue on Monday mornings, that is, after having my Mac sleeping during the weekend with the app running. So last Friday at the end of the day I set up debug mode for Realtime Database, started recording logs left the app running attached to Xcode and put my mac to sleep.
Yesterday morning, while the mac was sleeping the app tried reconnecting to the Realtime Database. This behavior is normal as far as I know: even if the computer is "sleeping" the app would wake app every now and then for some seconds and try to reconnect to servers because it doesn't know the mac "is actually sleeping". Some seconds later, it actually goes to sleep so the app stop running.
2022-06-05 08:44:21.249940-0300 0xdc7174   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB082014] Realtime connection established
2022-06-05 08:44:21.251504-0300 0xdc7c99   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB034003] On ready
2022-06-05 08:44:21.251590-0300 0xdc7c99   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB034041] Handling timestamp:<PHONE_NUMBER>722
2022-06-05 08:44:21.300121-0300 0xdc7c99   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB034033] Calling restore state
2022-06-05 08:44:21.300225-0300 0xdc7c99   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB034035] Restoring auth
2022-06-05 08:44:21.300324-0300 0xdc7c99   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB082004] Sending data (contents hidden)
2022-06-05 08:44:21.300432-0300 0xdc7c99   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB065002] Raising value event on connected
2022-06-05 08:44:21.463667-0300 0xdc7c99   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB083006] (wsc:29) handleNewFrameCount: 1
2022-06-05 08:44:21.463730-0300 0xdc7c99   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB083007] (wsc:29) handleIncomingFrame sending complete frame: 0
2022-06-05 08:44:21.463794-0300 0xdc7c99   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB082010] Got data message: {
    [SOME STUFF]
}
2022-06-05 10:07:56.067777-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB082004] Sending data (contents hidden)
2022-06-05 10:07:56.069026-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB083014] (wsc:29) resetting keepalive, to 2022-06-05 13:08:41 +0000 ; old: 2022-06-05 11:45:06 +0000

The problem is that, during these spontaneous wake ups, there are high chances that we are not connected to the network, so connection issues might happen:
2022-06-05 10:07:56.108435-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB083010] (wsc:29) didFailWithError didFailWithError: Error Domain=NSPOSIXErrorDomain Code=54 "Connection reset by peer" UserInfo={_kCFStreamErrorCodeKey=54, _kCFStreamErrorDomainKey=1}
2022-06-05 10:07:56.108510-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB083013] Websocket is closing itself
2022-06-05 10:07:56.108573-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB082007] Realtime connection lost.
2022-06-05 10:07:56.108632-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB082002] Closing realtime connection.
2022-06-05 10:07:56.109244-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB034004] Got on disconnect due to other
2022-06-05 10:07:56.109370-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB034008] Scheduling connection attempt
2022-06-05 10:07:56.109444-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB065002] Raising value event on connected
2022-06-05 10:07:56.109504-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB038019] Running onDisconnectEvents
2022-06-05 10:07:56.109564-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB034009] Trying to fetch auth token
2022-06-05 10:07:56.109706-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB034011] Successfully fetched token, opening connection
2022-06-05 10:07:56.109764-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB083001] (wsc:30) Connecting to: wss://s-usc1f-nss-2565.firebaseio.com/.ws?somestuff
2022-06-05 10:07:56.109829-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB082001] Calling open in FConnection
2022-06-05 10:07:56.109893-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB083002] (wsc:30) FWebSocketConnection open.

As there were network problems, opening this connection timed out. And the Realtime Database client hanged here.
2022-06-05 11:47:00.281060-0300 0xdc79c6   Debug       0x0                  16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB083012] (wsc:30) Websocket timed out on connect

Today I started my workday, opened my mac and this got logged.
2022-06-06 11:17:09.814642-0300 0xdcd704   Debug       0x285b42c            16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB065002] Raising value event on connected

I checked server side state and confirmed this issue was happening: I was offline for the rest of the world but my app was running and had a good Internet connection.
Using an internal debugging feature I tried to interact with the Realtime Database, and this is what I got:
2022-06-06 11:24:46.647367-0300 0xdce6c2   Debug       0x285cb8a            16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB038003] Setting: /some_document with {
    value = 1;
} pri: <null>
2022-06-06 11:24:46.655124-0300 0xdce6c2   Debug       0x285cb8a            16384  0    MyAppName: (GoogleUtilities) 8.15.0 - [Firebase/Database][I-RDB034025] Wasn't connected or writes paused, so added to outstanding puts only. Path: /some_document

This confirmed previous observations on this issue. The Realtime Database client was "stuck". Using Xcode, I checked two Realtime Database client internal bits of state:

connection.interruptReasons: empty
connectionState: ConnectionStateConnecting

So it looks like the app hanged in this ConnectionStateConnecting state after the network timeout. I checked the client code and, from what I could see, the Realtime Database client doesn't have any retry logic in place to recover from these errors.
In the past, I had seen the app recovering itself after several hours so I'm still waiting to collect logs in case that could be helpful. I still have Xcode attached so if you want to debug it with me now, let me know. :)
I have more logs if that might be useful to you but they have some (slightly) sensitive data so I am fine with sending them to you directly instead of attaching them here publicly if you need them.
Thank you!

Thanks, it sounds like the right way to fix this is to just retry on socket connection failures, respecting the NSWorkspace state. I'll take a crack at implementing this when I have the bandwidth.

Thanks!
You might also consider the timeout case. I think the socket error in this case was 2 hours late so it's not safe to wait for it unless the SDK can force it to happen earlier somehow.
Alternatively, could the SDK discard the timed out web socket (and future notifications) and try a new one?

Maybe it'd be better to deliberately terminate the socket connection on sleep and restore it on wakeup then? @IanWyszynski what do you think?

Thanks! In my opinion, terminating the connection on sleep and restore it on wake up might reduce the chances of this from happening, but it doesn't fix the root problem. The main problem is that the SDK doesn't retry after the timeout (it just logs an error), it waits for the socket error and that might take a lot of time (if it actually happens).
I think terminating the connection on sleep and restore it on wake up might mitigate it though so it's a good workaround we can do on our side in the meantime. We are going to implement this behavior to try to reduce the chances of it from happening by invoking goOffline when the computer goes to sleep, and invoking goOnline when the computer wakes up. This way, network problems happening while the computer is sleeping shouldn't affect us.
Having said that, this won't fix the root problem since network issues might happen anyway when the computer is awake, specifically when starting the app and the SDK wants to setup the connection, so it would be great if the SDK can handle connection timeouts and retries more gracefully.

Hi, this issue is more widespread than just macos, it definitely happens on iOS as well in ionic cordova apps (environment info below).
A couple observations and research items of note that may help:

iOS suspends the app within ~25 seconds of being backgrounded
The Firebase realtime database connection survives about 60 seconds, during this period resuming the app the realtime db will keep working like nothing happened.
To observe a 'frozen' / 'dead' realtime connection, wait at least 90 seconds, I was using 4 mins to know it was dead.
Firestore will happily resume, debug level logs show it detecting inactive threads and then reconnecting
Realtime detects the app resuming but does not reconnect! No new events, no debug logs saying threads are dead.
Using 'goOnline' when the app detects a resume does not work
Using all kinds of methods of unsubscribing the old angular fire and deleting the conn and starting again on resume does not work, the realtime db still knows of the connection but it's a tunnel to no where.
- However if you use goOnline on resume AND use 'goOffline' when the app detects being backgrounded it works! Happily reconnects all prior threads/connections

It's quite a bit of effort to setup a new 'minimal' repo, ionic, angularfire and a subscription to a listval with detection events on resume/background. If someone else has that handy then go for it, I hope this post helps someone else so they don't spend days searching and experimenting!
Note: using modular angularfire.
ENVIRONMENT
Angular:
Angular CLI: 13.3.10
Node: 14.20.0
Package Manager: yarn 1.22.15
OS: darwin x64

Angular: 13.3.12
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router, service-worker

Package                         Version
---------------------------------------------------------
@angular-devkit/architect       0.1303.6
@angular-devkit/build-angular   13.3.10
@angular-devkit/core            13.3.10
@angular-devkit/schematics      13.3.10
@angular/cli                    13.3.10
@angular/fire                   7.5.0
@schematics/angular             13.3.10
rxjs                            6.6.7
typescript                      4.6.2
webpack                         5.70.0

Ionic:

   Ionic CLI                     : 6.20.3 (/Users/markterrill/.config/yarn/global/node_modules/@ionic/cli)
   Ionic Framework               : @ionic/angular 6.2.4
   @angular-devkit/build-angular : 13.3.10
   @angular-devkit/schematics    : 13.3.10
   @angular/cli                  : 13.3.10
   @ionic/angular-toolkit        : 3.1.1

Capacitor:

   Capacitor CLI      : 4.5.0
   @capacitor/android : 4.5.0
   @capacitor/core    : 4.1.0
   @capacitor/ios     : 4.5.0

Cordova:

   Cordova CLI       : not installed
   Cordova Platforms : not available
   Cordova Plugins   : not available

Utility:

   cordova-res (update available: 0.15.4) : 0.15.3
   native-run                             : 1.7.1

System:

   Android SDK Tools : 26.1.1 (/Users/markterrill/Library/Android/sdk)
   NodeJS            : v14.20.0 (/Users/markterrill/.nvm/versions/node/v14.20.0/bin/node)
   npm               : 6.14.17
   OS                : macOS Monterey
   Xcode             : Xcode 14.2 Build version 14C18

How is this still an issue? Seems to be a pretty critical bug imo...

Hello, this is still an issue on both Firebase and Firestore databases. Not only that, but, it's found in the web & Android versions, not just iOS.
Do we have to implement a watchdog or an app lifecycle event ourselves for something this critical? There have been countless times where I get a push notification while the app is on the background / sleep, I open it and the data from Firestore never arrives without restarting the app.

this cost me a lot of negative reviews on my app, I guess we need to remove rtdb. I tried @markterrill solution listening the lifecycle of the app but it doesn't work, rtdb never recovers and it doesn't show an error or something in debug window.

In my case, when the app goes offline then online the listener to ".info/connected" is never called again.  This means when the app reconnects to WiFi again, my app is not able to update its online status again in the realtime database.

Hello,
I had a similar problem while using Firebase web modular API with Capacitor on iOS.
Each time a tried a write operation on the database (with set()), and a network change occurred meanwhile, the database API got stuck forever.
My listener to ".info/connected" was never called.
I think a found a workaround:
import { Injectable } from '@angular/core';
import { getApp } from 'firebase/app';
import { getDatabase, goOffline, goOnline, onValue, ref } from 'firebase/database';
import { Network } from '@capacitor/network';

@Injectable({
  providedIn: 'root'
})
export class DatabaseService {

  public connectedToDatabase: boolean;

  private app = getApp();

  private database = getDatabase(this.app, "...firebasedatabase.app/");

  constructor() {
    // Get network status changes
    // Network status changes when switching between wifi, cellular and offline
    Network.addListener('networkStatusChange', status => {
      console.log('Network status changed', status);
      if (status.connected == true) {
        // We need to go offline first, because realtime database can get stuck when switching between wifi and cellular
        goOffline(this.database);
        goOnline(this.database);
      }
      if (status.connected == false) {
        goOffline(this.database);
      }
    });

    // Now the callback on connectedRef is triggered at each network change
    // And write operations like (set()) don't get stuck anymore
    const connectedRef = ref(this.database, ".info/connected");
    onValue(connectedRef, (snap) => {
      if (snap.val() === true) {
        this.connectedToDatabase = true;
      } else {
        this.connectedToDatabase = false;
      }
    });
  }
}

@UZ8VLqifK36Z  Thanks for sharing. I'm going to give this a try at some point with my React app. I just discovered this window event, which hopefully works the same.
// https://developer.mozilla.org/en-US/docs/Web/API/Navigator/onLine

window.addEventListener("offline", () => {});
window.addEventListener("online", () => {});

Hey there, having the same issue here, react native firebase though. I believe it's related.

I'll try downgrade my version before 10.27.0 and see if that improves things :)

@paulb777 We're on the latest version 11.6.0 and this could be potentially disastrous for us. I personally already hit this issue 3 times. Just randomly after not using the app for some time 10-30min or so and bringing the app up from background, the real time database won't connect at all. Can you please help?


Consistency in implementing SecureTransportAction class

Consistency in implementing SecureTransportAction class
Currently when one Transport Action calls another, and the called TransportAction implements SecureTransportAction for backend filtering, we generally do not need the calling class to implement SecureTransportAction.
However we can implement it and "nip it in the bud" ( For example, if backend filtering is enabled and the user does not have a backend role configured) and raise the Exception. Only TransportGetFindingsAction nips it in the bud. TransportGetAlertsAction and TransportAcknowledgeAlertsAction do not.
This issue when resolved will take a consistent approach for all such cases.
It is not a bug, just being consistent in coding.

Working on it.


What is the correct way to change the text while the loading activity is presented?

I want to show the user the progress of the time-consuming task that I'm performing. Let's say I want to first show "Performing Task 1 of 10". After the first task is completed, I want to change the text to say "Performing Task 2 of 10" and so on. And after performing all tasks, I want to hide it.
Do I just simply call show with different text multiple times?

Right now that seems to be the only way.
You could also edit the Settings parts a little so the library allows you to put a new text while its showing


[Phone] Add sample using WebAuthenticationBroker with OAuth

Add a sample that demonstrates how to use the recommended workflow for OAuth secured resources with Windows Phone 8.1 using WebAuthenticationBroker with IdentityManager in the ArcGIS Runtime SDK for .NET.
For discussion and background details see: http://blogs.msdn.com/b/thunbrynt/archive/2014/04/14/windows-phone-8-1-for-developers-web-authentication-broker.aspx

@rexhansen  @mbranscomb is this still relevant?

@anttikajanus @mbranscomb  Yes, this is still relevant. Microsoft is promoting the use of WebAuthenticationBroker with OAuth workflows.  It's built into the ArcGIS Runtime SDK for .NET - Windows Store API.  It is not in the Windows Phone API due to complexity of implementation.   For sake of completeness we should include a sample.  Note, the current OAuth samples show use of web views.   The WebAuthenticationBroker is limited by one issue, the domain in the authorize and approval url must match.  If during the OAuth login process a user chooses to change the url (eg enter an organization subdomain) the WebAuthenticationBroker will not work.  The solution is either 1) use app logic to determine the organization url before the OAuth workflow starts or 2) Use a web view.


prevent private [[sources]] from showing up in Pipfile(s)

@techalchemy I've been using pipenv for months using PIP_EXTRA_INDEX_URL to point to our own private packages in our own index.
pipenv never added this url to the Pipfile, but i've noticed it recently has started to. Not sure if this has been due to some update to pipenv or due to me switching to pip.conf files (instead of using environment variables--see below) to specify the extra-index-url.
This is not ideal, because this url contains a "secret" token and so should ideally not be on version control (it'd be nice to avoid using yet another tool, e.g. git-crypt to deal with this).
Any ideas/suggestions?
Why pip.conf?
The reason I had to switch to pip.conf was that pip/pipenv suddenly stopped "respecting" the PIP_EXTRA_INDEX_URL environtment variable. I realize this is likely outside the scope of pipenv, since I can't reproduce this myself on different setups (i.e. this suddenly happened when I switched to ubuntu 18) but would appreciate any ideas/help on how to resolve it.

thanks @uranusjr.
Here is the link to documentation on credential injection into Pipfile