Update "Writing Tests" doc to use Enzyme

Fixes #1679.
Notes:

I didn't bother updating all tests in the TODOMVC app because it's (supposedly) already being worked on, see #1471 and #1493 (and they were even removed in #1480), as well as #1481 claiming that the TODOMVC app will be "completely replaced". If you have changed your mind on that (I'm not sure where that decision came from and that was from 2 months ago), let me know and I'm happy to update the rest of the tests to use Enzyme.
I removed mention of an old issue with setState() that Enzyme automatically handles.

Looks like there are some conflicts. Can you rebase against master? Thanks!

Ha! Someone had changed a single sentence of a sentence I had also changed by removing it.
Anyway, I have rebased as requested.

Sorry, I missed the todomvc updates included here. Can you remove those? We can track that work in #1493 or another PR if that one doesn't pick back up again.

Ok, done.

Ping?

Ping @timdorr @gaearon
Dan, I'm pinging you because:

I'm not sure of Tim's status/availability
Your original "go ahead" on #1679
The quality of Redux's docs IS Redux, in my opinion, and I think yours too. Redux is powerful because it's simple. The library is small; it's explaining how to use it that matters. The quality of the doc is what made me start with Redux over Flux way back when.

Looks reasonable at first glance to me.  If no one else responds in another day or so, ping me in here and I'll accept it.

Let's do it!

I'm looking at http://redux.js.org/docs/recipes/WritingTests.html and don't see the changes. No mention of Enzyme.
I'm guessing it's due to the fact that the code isn't in the gh-pages branch? I'm sorry, perhaps that means that I should have made changes in my gh-pages branch, and if so, I'm sorry for my rookie mistake (we should add a note in CONTRIBUTING :-). Would you like me to do that and submit another pull request, or can you make the fix?

The docs have to be re-published once the Markdown files are updated.  I'll try to get to it this evening.  (I want to do so anyway, because I don't think the current published version includes the plugin I added to the Gitbook build to fix the anchor link jumping behavior.)

It's been a month. Is it worth publishing now?

@tylercollier Yeah, it's been a while (2 months!). Just pushed an updated build now.

@timdorr , do you have your Gitbook setup updated with the "fix the page jumping" plugin I added when I set up the FAQ page?  Looks like that's still not working right now.

Nope, but what's the name of the plugin exactly? I can make sure it's added into the project.

It is actually already listed - I updated book.json after I published the initial FAQ page.  It's https://github.com/sifue/gitbook-plugin-anker-enable , which basically just does a setTimeout(jumpBackToAnchor, 300).
I had it installed in my local Gitbook setup and did a publish with it, but Gitbook must not complain if you don't have it installed or something.

Gitbook auto-installs requested plugins. book.json acts like package.json, essentially:
info: No version specified, resolve plugin anker-enable
info: install plugin anker-enable from npm (gitbook-plugin-anker-enable) with version 0.0.4
<EMAIL_ADDRESS>node_modules/gitbook-plugin-anker-enable
info: >> plugin anker-enable installed with success


Attempting to Shoot Arrows at Whiterun Guards

When I shoot at a Whiterun Guard (any for this matter) with a bow, they don't receive damage, and don't become hostile.  I attempted to melee sneak attack to see if it would trigger guards to become hostile and it worked.  Not sure if this is an isolated issue with the bow or if it's an issue attacking non-hostile NPCs. Will continue to test and see if it's just Whiterun Guards affected. Only tested on guards outside of Whiterun, will update if anything changes.

This is due to Simple Offence Supression. Open the MCM menu there, and you can allow damage to friendlies. This mod is in the list because Elder Souls - Sweep Attacks made it it way too easy to accidently kill friendlies in combat. I removed that mod when Precision was released.
In practice though, Precision can also cause the mod to not work properly, and can still cause aggro and hit when you use melee.
It's also possible there's more to it than that, but change the MCM and let me know. I'll see if there's a patch available, I really have not looked since I added Precision.

It was already checked when I looked at the MCM, toggling it both on and off did not work.  Is there anything else I should try?

If that's the case, then there's definitely something else going on. I'll take a look at it, thank you!

I have narrowed the problem down the No Follower Attack Collision.
The mod has a configuration option in it's ini file, bProtectNeutralActor. This was already set to false, but the mod seemed to be protecting guards no matter what, until you attacked another NPC.
Removing this mod solves the issue. The mod really was not working with MCO anyways, friendlies were still being hit most of the time. Not a big loss. Simple Offence Suppression will remain in and on by default, but you can always disable the option in MCM to prevent accidental hits.


webRTC on surface pro 3

Hi,
I'm experiencing issues in displaying remote video in microsoft surface pro 3 on chrome browser.
Are you aware of any issues in that platform?
on the spec it should be same as a normal PC but it does not work well.

i've heard about some issues related to screensharing on a surface, but nothing else.
@marwahvikas might know more?

Does apprtc on loopback also fails?  Some logs or chrome://webrtc-internals dump would be useful. Btw someone posted on discuss forum similar issue https://groups.google.com/forum/#!searchin/discuss-webrtc/surface/discuss-webrtc/H4PL_ZIJ8Sk/kNKCnDU6faIJ

I have a SimpleWebRTC based app that works with remote video and screen sharing on a Surface Pro 3.


Replace the shell with a TCL interpreter?

I found this project: https://github.com/dragoncoder047/tinytcl, currently abandoned, but seemingly usable, and there are several others like it, some seem to need less than 10k of flash.
TCL supports the "Command arg1 arg2" syntax, but it is a real full programming language.  Perhaps it would be useful to replace the shell with TCL, to allow for real scripting?

Sorry, wasn't receiving notifications and working at other projects.
Your idea is quite good and isn't difficult to include, the question is how often would it be used in real world?
Most cases would likely involve to write the script commands to a file and run them as needed.
Right now is included the Wrench scripting language, so one can already script the functionality and get it running.
But I think you mean direct pin reading and handling from the command line?


edit all authentication files and add a test

https://github.com/Azure/azure-sdk-for-python/issues/11028

/azp run python - storage

I was removed as an author 😢

@aviramha sorry that I should have gave you a heads up that I didn't merge your pr, and thanks for the commit!
I didn't merge your PR because at that time the CI in your PR failed https://github.com/Azure/azure-sdk-for-python/pull/13346, and I wasn't sure what caused the failure, plus I added a commit to your PR and I was worried my change broke the CI, so I created a new PR to play around with. (If I use your PR to play around you will get a lot of notifications)
Finally it turned out that the changes in our dependency, and the CI failure was blocking other people's pr, so once I got the approval I didn't think too much and merged the fix.

Hmm it's a bit tricky but you could rebase and amend author for the commit
https://stackoverflow.com/questions/3042437/how-to-change-the-commit-author-for-one-specific-commit

@aviramha I've partially reverted the commit and merged it to master.
You can rebase against master and we will merge your pr later

@xiafu-msft Thanks, I really appreciate it. Sorry for being "annoying" :)

@aviramha for sure. Not at all, I should have noticed that, thanks for your understanding!


Specifying gitea server Gitea Base URL when it is normally accessed via SSH Local Port Forwarding?

On my gitea server, I normally access it via ssh Local Port Forwarding; and it isn't accessible from the outside because I bind it to the localhost address.
I'm accessing it from http://<IP_ADDRESS>:8080 via "Local Port Forwarding", and it thinks that it's being accessed from http://<IP_ADDRESS>:3000; so certain features (the preview in the wiki and the issues for instance do not work (you can however save them, just not preview them); anyway it's mis-configured, see below, I'd like to change the Gitea Base URL so that anytime the server currently writes http://<IP_ADDRESS>:3000 it will write http://<IP_ADDRESS>:8080 instead (but that said, it would be good if it still operated off of port 3000):

What line in gitea.ini changes this functionality? or is it more like a situation where I need to throw a reverse proxy in front of it that rewrites the URLs for me?

You are essentially using your port forwarding as a reverse proxy. The correct setting to change would be ROOT_URL in the [server] section.
See https://docs.gitea.io/en-us/config-cheat-sheet/#server-server
In general for configuration queries using the forums or our discord is better.


Fix superbuild install permission errors

I was trying to create a debian package but I was getting permission errors during building. After digging long enough I found out that the Superbuild is trying to install files to CMAKE_INSTALL_PREFIX but it doesn't have access at the time.
With this PR, during the build, the microxrcedds_agent is installed to a temporary directory under project's build directory. Then, during the install step, the files are installed to a proper path. The Micro-XRCE-DDS-Agent is implemented similarly.

If you need binary packages for micro-ROS agent please take a look here: file:///home/pgarrido/dev/eProsima/vulcanexus/docs/build/html/rst/installation/linux_binary_installation.html

I'm aware of the vulcanexus project. I still have reasons for distributing the package myself. I created this PR because I consider this change a bugfix.


Latest update will not install

The latest update will not install
Even removing the plugin and re-adding it fails

Due to a typo in version in manifest.json, Obsidian tries to download files for non-existent release. :(
I fixed this; it will be available once Obsidian updates plugin version (in 24 hours or less).
Sorry for the inconvenience!

Things work as intended already. Cheers!


Releasing new version of quarkus-primefaces-extensions maven repo

Hello. I am unfamiliar with the process of new releases on maven. A new version of primefaces-extensions, v13.0.12, was recently released. What is the process of releasing a new version of quarkus-primefaces-extensions to reflect the update?

i have not really been releasing older version of this quarkus plugin i always just release the new latest branch?
If you are using Quarkus why not update to 14.x?

I believe I've misunderstood your comments from here. My thinking is that this fix is only available in v13.0.12 and will be available in the next new release, 14.0.2 for example.
I updated quarkus-primefaces-extensions to 3.14.1 (which uses primefaces-extensions 14.0.1) just now, but the code fix is not there.

right i have not released 14.0.2 yet.  So your question about 13.12 threw me off.
I update this library whenever i do official PFE releases.  So Quarkus PF 3.14.2 will have this fix in it.

Okay understood. Thank you!

3.14.2 is now in Maven Central.

Thank you for the update. Much appreciated


Checking if the generated code is in The Stack shortcut not working

Hi,
I have done testing of the extension installation steps and then tried Cmd+shift+a shortcut. It didn't work. I changed system boundings for this shortcut, but it does nothing anyway.
Have I missed something?

Can you provide steps to reproduce your error?
The extension is being refactored, I've tested the attribution check in the unreleased 0.0.39 and it seems to work.

0.1.0 is released, let me know if your issue is fixed


feat(elastic): support single requests in addition to the bulk api

Disable use of the bulk elasticsearch api by setting ES_DISABLE_BULK to true.
Quite a messy PR but time is of the essence

at least once the specs pass


Enable ppc64le builds

Hi! This is the friendly automated conda-forge-webservice.
I've rerendered the recipe as instructed in #12.
Here's a checklist to do before merging.

[ ] Bump the build number if needed.

Fixes #12

@conda-forge-admin, please re-render

@conda-forge-admin, please re-render

cc @beckermr

Looking at the recipe it appears repackaged binaries are used on Linux
https://github.com/conda-forge/shellcheck-feedstock/blob/85d51813a19bd1130ca3a6f143927c14b7c679f4/recipe/meta.yaml#L12-L15
Presumably this is because building stack, also known as the Haskell Tool Stack, is a big lift
Am not seeing ppc64le packages for stack or prebuilt binaries of shellcheck for ppc64le
So think it may be a while before we can resolve this

Looks like upstream dug into building packages for ppc64le, but gave up due to difficulties encountered
https://github.com/koalaman/shellcheck/issues/2081


feat: add management permission for workspace create user

Description
Add management permission by default to workspace creator
Issues Resolved

Screenshot

Testing the changes

Check List

[ ] All tests pass

[ ] yarn test:jest
[ ] yarn test:jest_integration
[ ] yarn test:ftr

[ ] New functionality includes testing.
[ ] New functionality has been documented.
[ ] Update CHANGELOG.md
[ ] Commits are signed per the DCO using --signoff

Codecov Report

Merging #92 (0a82483) into workspace (d732833) will decrease coverage by 23.12%.
The diff coverage is n/a.

@@              Coverage Diff               @@
##           workspace      #92       +/-   ##
==============================================
- Coverage      65.76%   42.65%   -23.12%
==============================================
  Files           3334     1597     -1737
  Lines          64442    32315    -32127
  Branches       10258     6063     -4195
==============================================
- Hits           42383    13784    -28599
+ Misses         19516    17488     -2028
+ Partials        2543     1043     -1500

Flag
Coverage Δ

Linux_3
42.65% <ø> (ø)

Windows_1
?

Windows_2
?

Windows_3
?

Windows_4
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed
Coverage Δ

src/core/server/index.ts
100.00% <ø> (ø)

... and 2291 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


hugo new posts/my-first-post.md command fails with hugo modules

What version of Hugo are you using (hugo version)?

$ hugo version
hugo v0.89.2+extended linux/amd64 BuildDate=unknown

Does this issue reproduce with the latest release?
Yes

I have followed the steps of the tutorial: Hugo modules for “dummies” and it works correctly but when i want to create a new post using hugo new command I get:

$ hugo new posts/my-first-post.md -v
Error: open : no such file or directory

If I don't use hugo modules but gitmodules or download the theme in the themes folder then I have no problems.

There isn't enough information here to determine what's happening. The part about "this not working with Hugo modules" is obviously not correct, as all of my projects (in which have tested this with), works fine here.

@bep Do you require any information from me? I have tested the demo that you indicate and I cannot create the .md from the archetypes when I set the theme using hugo modules, but I can when I use git modules or copy the theme directly.

@bep The error message I get with hugo v0.89.3+extended is:
$ hugo new posts/my-first-post.md -v
Error: could not determine content directory for "posts/my-first-post.md"

The config.toml is:
baseURL = 'http://example.org/'
languageCode = 'en-us'
title = 'My New Hugo Site'
theme = ["github.com/theNewDynamic/gohugo-theme-ananke"]

And the structure of the project is:
├── archetypes
│   └── default.md
├── config.toml
├── content
├── data
├── go.mod
├── go.sum
├── layouts
├── resources
│   └── _gen
│       ├── assets
│       │   └── css
│       │       └── ananke
│       │           └── css
│       │               ├── main.css_bb5467e0521bbea6b1e66429f6ec028e.content
│       │               └── main.css_bb5467e0521bbea6b1e66429f6ec028e.json
│       └── images
├── static
└── themes

The hugo serve command works:
Start building sites …
hugo v0.89.3+extended linux/amd64 BuildDate=unknown

                   | EN
-------------------+-----
  Pages            |  7
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  1
  Processed images |  0
  Aliases          |  0
  Sitemaps         |  1
  Cleaned          |  0

Built in 123 ms
Watching for changes in /home/eyllanesc/blog/{archetypes,content,data,layouts,static}
Watching for config changes in /home/eyllanesc/blog/config.toml, /home/eyllanesc/blog/go.mod
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address <IP_ADDRESS>)
Press Ctrl+C to stop

@bep The error message I get with hugo v0.89.3+extended is:
$ hugo new posts/my-first-post.md -v
Error: could not determine content directory for "posts/my-first-post.md"

Steps to reproduce the problem:

$ hugo new site quickstart
Congratulations! Your new Hugo site is created in /home/eyllanesc/test_blog/quickstart.

Just a few more steps and you're ready to go:

1. Download a theme into the same-named folder.
   Choose a theme from https://themes.gohugo.io/ or
   create your own with the "hugo new theme " command.
2. Perhaps you want to add some content. You can add single files
   with "hugo new /.".
3. Start the built-in live server via "hugo server".

Visit https://gohugo.io/ for quickstart guide and full documentation.

$ cd quickstart
$ echo theme = \"github.com/theNewDynamic/gohugo-theme-ananke\" >> config.toml
$ hugo mod init eyllanesc/quickstart
WARN 2021/11/15 22:48:28 module "github.com/theNewDynamic/gohugo-theme-ananke" not found; either add it as a Hugo Module or store it in "/home/eyllanesc/test_blog/quickstart/themes".: module does not exist
go: creating new go.mod: module eyllanesc/quickstart
go: to add module requirements and sums:
        go mod tidy
$ hugo server
hugo: downloading modules …
go get: added github.com/theNewDynamic/gohugo-theme-ananke v0.0.0-20211001175655-869b03ea78b6
hugo: collected modules in 13637 ms
Start building sites …
hugo v0.89.3+extended linux/amd64 BuildDate=unknown

                   | EN
-------------------+-----
  Pages            |  7
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  1
  Processed images |  0
  Aliases          |  0
  Sitemaps         |  1
  Cleaned          |  0

Built in 415 ms
Watching for changes in /home//test_hugo/quickstart/{archetypes,content,data,layouts,static}
Watching for config changes in /home//test_hugo/quickstart/config.toml, /home/eyllanesc/test_blog/quickstart/go.mod
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address <IP_ADDRESS>)
Press Ctrl+C to stop
^C
$ hugo new posts/my-first-post.md
Error: could not determine content directory for "posts/my-first-post.md
$ tree
.
├── archetypes
│   └── default.md
├── config.toml
├── content
├── data
├── go.mod
├── go.sum
├── layouts
├── resources
│   └── _gen
│       ├── assets
│       │   └── css
│       │       └── ananke
│       │           └── css
│       │               ├── main.css_bb5467e0521bbea6b1e66429f6ec028e.content
│       │               └── main.css_bb5467e0521bbea6b1e66429f6ec028e.json
│       └── images
├── static
└── themes

@bep The error message I get with hugo v0.89.3+extended is:
$ hugo new posts/my-first-post.md -v
Error: could not determine content directory for "posts/my-first-post.md"

Steps to reproduce the problem:

$ hugo new site quickstart
Congratulations! Your new Hugo site is created in /home//test_hugo/quickstart.

Just a few more steps and you're ready to go:

1. Download a theme into the same-named folder.
   Choose a theme from https://themes.gohugo.io/ or
   create your own with the "hugo new theme " command.
2. Perhaps you want to add some content. You can add single files
   with "hugo new /.".
3. Start the built-in live server via "hugo server".

Visit https://gohugo.io/ for quickstart guide and full documentation.

$ cd quickstart
$ echo theme = \"github.com/theNewDynamic/gohugo-theme-ananke\" >> config.toml
$ hugo mod init eyllanesc/quickstart
WARN 2021/11/15 22:48:28 module "github.com/theNewDynamic/gohugo-theme-ananke" not found; either add it as a Hugo Module or store it in "/home//test_hugo/quickstart/themes".: module does not exist
go: creating new go.mod: module eyllanesc/quickstart
go: to add module requirements and sums:
        go mod tidy
$ hugo server
hugo: downloading modules …
go get: added github.com/theNewDynamic/gohugo-theme-ananke v0.0.0-20211001175655-869b03ea78b6
hugo: collected modules in 13637 ms
Start building sites …
hugo v0.89.3+extended linux/amd64 BuildDate=unknown

                   | EN
-------------------+-----
  Pages            |  7
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  1
  Processed images |  0
  Aliases          |  0
  Sitemaps         |  1
  Cleaned          |  0

Built in 415 ms
Watching for changes in /home//test_hugo/quickstart/{archetypes,content,data,layouts,static}
Watching for config changes in /home//test_hugo/quickstart/config.toml, /home//test_hugo/quickstart/go.mod
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address <IP_ADDRESS>)
Press Ctrl+C to stop
^C
$ hugo new posts/my-first-post.md
Error: could not determine content directory for "posts/my-first-post.md
$ tree
.
├── archetypes
│   └── default.md
├── config.toml
├── content
├── data
├── go.mod
├── go.sum
├── layouts
├── resources
│   └── _gen
│       ├── assets
│       │   └── css
│       │       └── ananke
│       │           └── css
│       │               ├── main.css_bb5467e0521bbea6b1e66429f6ec028e.content
│       │               └── main.css_bb5467e0521bbea6b1e66429f6ec028e.json
│       └── images
├── static
└── themes

@bep The error message I get with hugo v0.89.3+extended is:
$ hugo new posts/my-first-post.md -v
Error: could not determine content directory for "posts/my-first-post.md"

Steps to reproduce the problem:

$ hugo new site quickstart
Congratulations! Your new Hugo site is created in /home/user/test_hugo/quickstart.

Just a few more steps and you're ready to go:

1. Download a theme into the same-named folder.
   Choose a theme from https://themes.gohugo.io/ or
   create your own with the "hugo new theme " command.
2. Perhaps you want to add some content. You can add single files
   with "hugo new /.".
3. Start the built-in live server via "hugo server".

Visit https://gohugo.io/ for quickstart guide and full documentation.

$ cd quickstart
$ echo theme = \"github.com/theNewDynamic/gohugo-theme-ananke\" >> config.toml
$ hugo mod init eyllanesc/quickstart
WARN 2021/11/15 22:48:28 module "github.com/theNewDynamic/gohugo-theme-ananke" not found; either add it as a Hugo Module or store it in "/home/user/test_hugo/quickstart/themes".: module does not exist
go: creating new go.mod: module eyllanesc/quickstart
go: to add module requirements and sums:
        go mod tidy
$ hugo server
hugo: downloading modules …
go get: added github.com/theNewDynamic/gohugo-theme-ananke v0.0.0-20211001175655-869b03ea78b6
hugo: collected modules in 13637 ms
Start building sites …
hugo v0.89.3+extended linux/amd64 BuildDate=unknown

                   | EN
-------------------+-----
  Pages            |  7
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  1
  Processed images |  0
  Aliases          |  0
  Sitemaps         |  1
  Cleaned          |  0

Built in 415 ms
Watching for changes in /home/user/test_hugo/quickstart/{archetypes,content,data,layouts,static}
Watching for config changes in /home/user/test_hugo/quickstart/config.toml, /home/user/test_hugo/quickstart/go.mod
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind address <IP_ADDRESS>)
Press Ctrl+C to stop
^C
$ hugo new posts/my-first-post.md
Error: could not determine content directory for "posts/my-first-post.md
$ tree
.
├── archetypes
│   └── default.md
├── config.toml
├── content
├── data
├── go.mod
├── go.sum
├── layouts
├── resources
│   └── _gen
│       ├── assets
│       │   └── css
│       │       └── ananke
│       │           └── css
│       │               ├── main.css_bb5467e0521bbea6b1e66429f6ec028e.content
│       │               └── main.css_bb5467e0521bbea6b1e66429f6ec028e.json
│       └── images
├── static
└── themes

Duplicate of #9177


[Gally]: master <- dev

Automatically created by Git-Ally

:tada: This PR is included in version 6.2.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Split Hash into multiple types

Currently, many data structures are of type Hash, including:

Path
Pointer
Account Address
Transaction ID
Block ID
Chain ID
Actual SHA256

Separation of these data types is necessary so that we can have clearer logics and type can be enforced at compile time.

finished #510


[SPARK-32446][SHS] Add percentile distribution REST API of peak memory metrics for all executors

What changes were proposed in this pull request?
Add REST API for summary of executor peak memory metrics
Why are the changes needed?
Help users understand executor peak memory metrics distributions
Does this PR introduce any user-facing change?
Yes
How was this patch tested?
Added UT in HistoryServerSuite

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@gengliangwang Could you help take a look?

@gengliangwang Could you help take a look?

Add to whitelist

ok to test

Test build #130353 has started for PR 29247 at commit 1854e74.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34956/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34956/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34956/
Test PASSed.

Test build #130353 has finished for PR 29247 at commit 1854e74.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130353/
Test FAILed.

@gengliangwang Could you help retest? The failed sparkr tests should be unrelated with this change.

retest this please.

Test build #130394 has started for PR 29247 at commit 1854e74.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34997/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34997/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/34997/
Test PASSed.

Test build #130394 has finished for PR 29247 at commit 1854e74.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130394/
Test FAILed.

@gengliangwang Any ideas about sparkr build failure?

Test build #130441 has started for PR 29247 at commit 9028552.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35046/

Test build #130441 has finished for PR 29247 at commit 9028552.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130441/
Test FAILed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35046/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35046/
Test PASSed.

Test build #130549 has started for PR 29247 at commit 4ba4a15.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35148/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35148/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35148/
Test FAILed.

Test build #130549 has finished for PR 29247 at commit 4ba4a15.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130549/
Test FAILed.

Test build #130560 has started for PR 29247 at commit 71a8e50.

Test build #130560 has finished for PR 29247 at commit 71a8e50.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130560/
Test FAILed.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35160/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35160/

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35160/
Test PASSed.

Merged build finished. Test PASSed.

Test build #130571 has started for PR 29247 at commit 32a55a6.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35173/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35173/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35173/
Test PASSed.

Test build #130571 has finished for PR 29247 at commit 32a55a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130571/
Test PASSed.

Test build #130581 has started for PR 29247 at commit 5a55030.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35182/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35182/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35182/
Test FAILed.

Test build #130581 has finished for PR 29247 at commit 5a55030.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/130581/
Test PASSed.

@gengliangwang Build passed. Could you help merge this?

@HeartSaVioR Could you help take a look at this?

Can one of the admins verify this patch?

@tgravescs Could you help take a look?

ok to test

Test build #131073 has started for PR 29247 at commit 5a55030.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35677/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35677/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/35677/
Test FAILed.

Test build #131077 has started for PR 29247 at commit 479183c.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35680/

@warrenzhu25  This PR has good value.  Spark 3.1 branch-cut will happen tomorrow 12/04.  It will be great if this PR can make it to Spark 3.1.  Can you tell me what issue you are facing?

@warrenzhu25 This PR has good value. Spark 3.1 branch-cut will happen tomorrow 12/04.  It will be great if this PR can make it to Spark 3.1. Can you tell me what issue you are facing?

I just fixed conflict. There's no other issues. After build pass, could you help merge this?

Test build #132153 has started for PR 29247 at commit 355fb86.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36754/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36754/

Build finished.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36754/

Test build #132153 has finished for PR 29247 at commit 355fb86.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132153/

@tgravescs  @gengliangwang Could you help merge this?

Test build #132164 has started for PR 29247 at commit 9e712c6.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36765/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36765/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36765/

Test build #132164 has finished for PR 29247 at commit 9e712c6.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132164/

Test build #132238 has started for PR 29247 at commit 1f3afab.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36838/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36838/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36838/

Test build #132238 has finished for PR 29247 at commit 1f3afab.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132238/

@tgravescs @gengliangwang Do you have more comments for this?

Test build #132700 has started for PR 29247 at commit c12b25f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37303/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37303/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37303/

Test build #132700 has finished for PR 29247 at commit c12b25f.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132700/

Test build #133031 has started for PR 29247 at commit 7acc98f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37630/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37630/

Test build #133036 has started for PR 29247 at commit 4443f1f.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37635/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37635/

Test build #133031 has finished for PR 29247 at commit 7acc98f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133031/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37635/

Test build #133036 has finished for PR 29247 at commit 4443f1f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133036/

@tgravescs @gengliangwang Could you help merge this?

I'm not against this but this is a one-off additional API endpoint for some summary stats of metrics available elsewhere?

I'm not against this but this is a one-off additional API endpoint for some summary stats of metrics available elsewhere?

@srowen I will send another PR to add Web UI of this feature in executors page.

I'm not against this but this is a one-off additional API endpoint for some summary stats of metrics available elsewhere?

@srowen I will send another PR to add Web UI of this feature in executors page.

Many Spark users will look at the executorSummary information using Web UI.  it is a good idea to keep the feature's web UI and REST API consistent.

I'm not against this but this is a one-off additional API endpoint for some summary stats of metrics available elsewhere?

@srowen I will send another PR to add Web UI of this feature in executors page.

Many Spark users will look at the executorSummary information using Web UI.  it is a good idea to keep the feature's web UI and REST API consistent.

@warrenzhu25 Sorry for the late reply. My major concern is how this info shows in the executor page. The current summary section is more like an aggregated one.
The peak memory won't be displayed by default, it becomes tricky to add a new section.

@warrenzhu25 Sorry for the late reply. My major concern is how this info shows in the executor page. The current summary section is more like an aggregated one.
The peak memory won't be displayed by default, it becomes tricky to add a new section.

@warrenzhu25 Sorry for the late reply. My major concern is how this info shows in the executor page. The current summary section is more like an aggregated one.
The peak memory won't be displayed by default, it becomes tricky to add a new section.

We could add a new section like stage page, but this issue could be discussed further in another pr. Do you have more comments for this pr?

@warrenzhu25 Sorry for the late reply. My major concern is how this info shows in the executor page. The current summary section is more like an aggregated one.
The peak memory won't be displayed by default, it becomes tricky to add a new section.

We could add a new section like stage page, but this issue could be discussed further in another pr. Do you have more comments for this pr?

Just general comment; it'd be easier to pursue reviewers if you have some UI page the REST API is leveraged. (Doesn't need to be a code, screenshot would be sufficient.) It's not easy to see values in REST API without the actual use case.

Just general comment; it'd be easier to pursue reviewers if you have some UI page the REST API is leveraged. (Doesn't need to be a code, screenshot would be sufficient.) It's not easy to see values in REST API without the actual use case.

Any update for this PR?


updating master

Codecov Report

Merging #138 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #138   +/-   ##
=======================================
  Coverage   99.63%   99.63%
=======================================
  Files          42       42
  Lines        1089     1089
  Branches      127      127
=======================================
  Hits         1085     1085
  Misses          4        4

Impacted Files
Coverage Δ

src/redux/reducers/datatableReducer.js
99.67% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e350e64...b92d5f3. Read the comment docs.


httpcore required for PUT response to CloudFormation

When using this library with a minimal example I get the following exception in my CloudWatch Logs:
org/apache/http/ssl/SSLContexts: java.lang.NoClassDefFoundError java.lang.NoClassDefFoundError: org/apache/http/ssl/SSLContexts at
org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:966) at
com.sunrun.cfnresponse.CfnResponseSender.<init>(CfnResponseSender.java:32)

Adding this dependency to my pom.xml fixed the issue:
<dependency>
  <groupId>org.apache.httpcomponents</groupId>
  <artifactId>httpcore</artifactId>
  <version>4.4.4</version>
</dependency>

@timmattison I think there might be a version mismatch between some other httpcore and httpclient JARs on your classpath. It seems org/apache/http/ssl/SSLContexts moved packages between 4.3 and 4.4. Are you sure you don't have another dependency in your POM that is resolving httpclient to a different version?
Here is the dependency hierarchy for cfn-response-java as calculated by Eclipse... httpcore 4.4.4 should already be on the classpath.

I'd like to figure out what is causing this issue, since adding this dependency to the POM should not be necessary.

@timmattison sounds like this is a conflict with aws-java-sdk-core in your POM due to the way aws-java-sdk declares a dependency on httpclient and httpcore explicitly.
I would recommend either including the below snippet inside the <dependency> block for aws-java-sdk-core or adding the extra httpcore dependency yourself to elsewhere in your POM as it sounds like you have done. If I were you I would prefer the exclude method since then you don't need to track any minor updates to httpclient if cfn-response-java updates to a later version or another library does the same. 4.3.6 is 1.5 years old at this point so likely other libraries may provide a newer version...
      <exclusions>
        <exclusion>
          <artifactId>httpclient</artifactId>
          <groupId>org.apache.httpcomponents</groupId>
        </exclusion>
        <exclusion>
          <artifactId>httpcore</artifactId>
          <groupId>org.apache.httpcomponents</groupId>
        </exclusion>
      </exclusions>

For clarity to explain what is going on when aws-java-sdk-core 1.10.74 is included in the same project:

@timmattison sounds like this can be worked around by excluding httpcore from your aws-java-sdk-core dependency. Also it looks like AWS may update their POM in a future release of aws-java-sdk.
I'll close this issue, but re-open it if you feel there is some further action needed on my part.


Where are the Error Codes and Mitigations

Can you provide a TroubleShootingGuide.md  for this repo?

At least add
https://docs.microsoft.com/en-us/windows/desktop/Debug/system-error-codes
to the readme

Links disconnected from where an error pops out aren't incredibly useful in my experience, certainly shouldn't be handling the 99% cases. This is a perfect example. The answer for your original question was a relatively simple search away (that you shouldn't really have had to make), and I'd argue thats more accessible than trawling a readme.
I took a quick look through the code and I'm inclined that there should be a distinguished PrintErrorWithCode() that takes error code along with the rest of the printf varargs, and does the right job of formatting it out with the text for the win32 error codes. The bulk of the current PrintError fall into that category (or should … some seem to be missing code output, where they really should have).


Support non-recursive single-toolchain multi-core builds

AMP architecture's like lpcxpresso54114 require multiple images to be built, one for each core. Currently this is supported in Zephyr through recursive invocations of CMake. But recursive invocation's should be avoided whenever possible.
A limitation in CMake prevents us from achieving a multi-toolchain build, but for ARM platforms like lpcxpresso54114, one toolchain is enough. Recursive-CMake is employed today because of a limitation
inherent in the Zephyr build system, not in CMake. CMake supports building multiple application's just fine.
To avoid recursive invocations of CMake this Zephyr build system limitation should be resolved.
NB: Future work can look at multi-toolchain builds, Kitware has expressed that this is near-unachievable, so there is hope. An interesting approach would be to evaluate if one could have a single toolchain that acted as a compiler-frontend to the multiple toolchains. Thereby avoiding multiple toolchains as far as CMake is concerned.

This issue should be generalized for multi-image (i.e. single core with bootloader)

For the record I believe there are two ways to do a non-recursive build with a bootloader.
Do a completely vanilla build and treat the bootloader as any other library that the linker happens to be very particular about how is placed.
Do a multi-image (multi-app/ multi-executable whatever you want to call it) build and merge the hex files.
(Finally, you want to support merging in a fixed hex file for the use-case where you are developing for an app that is out in the wild and therefore requires the bootloader to remain unchanged.)

This will be addressed in a more generic way in the future.


[CLI] add --config flag

Summary
Motivation
We want to introduce a --config flag and deprecate the use of config as an argument.

ephemeral shells. By enabling a way to (eventually) do devbox shell --config path/to/devbox/json  nodejs-18_x, or even more simply devbox shell nodejs-18_x.
using devbox add/rm from shells started in parent directories: today, if I have my_sub_directory/devbox.json and start devbox shell my_sub_directory/devbox.json from the root of the repository, then doing devbox add or devbox rm inside this shell will fail.

A first attempt at fixing this was in #206. This PR (#225) is a generalization of that idea. I'll be stacking #206 on top of this PR, and enabling the --config flag to override the env-var.

How was it tested?
preliminary steps

i added nodejs-18_x to testdata/nodejs/nodejs-18
I (temporarily) deleted the devbox.json at the root of this repo. This is to ensure I verify that the CLI still complains if no devbox.json is found.

# giving a path
> cd testdata/nodejs
> devbox shell nodejs-18
Warning: devbox shell <path> is deprecated, use devbox shell --config <path> instead
Installing nix packages. This may take a while...done.
Starting a devbox shell...
> which node
# get nix store path
> exit

# using --config
> devbox shell -c nodejs-18
> which node
# get nix store path
> exit

# using neither path nor --config
> cd testdata/nodejs/node-18
> devbox shell
> which node
# get nix path
> exit

# using neither path nor --config, where devbox.json resides in a parent directory
> cd testdata/nodejs/nodejs-18/fake-dir
> devbox shell
which node
# get nix path
> exit

# using both path and --config
> cd testdata/nodejs
> devbox shell -c nodejs-18 nodejs-18

Error: Cannot specify devbox.json's path via both --config and the command arguments. Please use --config only.

# No devbox.json in parent dirs
> cd testdata/nodejs
> devbox shell

Error: No devbox.json found in this directory, or any parent directories. Did you run `devbox init` yet?

# giving a path where no devbox.json exists
> devbox shell nodejs-18/fake-dir
Warning: devbox shell <path> is deprecated, use devbox shell --config <path> instead

Error: No devbox.json found in nodejs-18/fake-dir. Did you run `devbox init` yet?

Current dependencies on/for this PR:

main

PR #225   👈

This comment was auto-generated by Graphite.

I'll fix the linter errors (minor), but this is ready for review.

Two more thoughts:

The warning message feels too long – it should be more succinct.  Something like WARNING: devbox shell <path> is deprecated, use devbox shell --config <path> instead
While I'm happy enough with --config, I'm not sure I love -c when seeing it in the examples. I'm not entirely sure people will be able to immediately guess that in devbox shell -c nodejs-18 the -c stands for config. Thoughts?

@savil there are a couple of options for testing findConfig.

Use t.TempDir() to create a test directory layout and give its path to findConfig.
Refactor findConfig to use an fs.FS. The CLI would pass in os.DirFS("/") and tests would pass in an fstest.MapFS. This might be a bit of work.

With either approach I think findConfig will need to be updated to take in a root path and a subpath. For example:
// or: findConfig(root fs.FS, dir string)
func findConfig(root, subdir string) (string, error) {
	root, _ := filepath.Abs(root)
	for subdir != "." {
		rooted := filepath.Join(root, cur)
		debug.Log("finding %s in dir: %s\n", configFilename, rooted)
		if plansdk.FileExists(filepath.Join(rooted, configFilename)) {
			return rooted, nil
		}
		subdir = filepath.Dir(subdir)
	}
}

// CLI
wd, _ := os.Getwd()
findConfig("/", wd)

// Tests
tmpDir := t.TempDir()
os.MkdirAll(filepath.Join(tmpDir, "a/b/c"))
os.WriteFile(filepath.Join(tmpDir, "a", "devbox.json"), []byte(`{}`), 0666)
findConfig(tmpDir, "a/b/c")

That will let you control where the search stops (cur will equal .) so the test doesn't go outside of the test directory.


Enabling gnome keyring blocks login via sssd

Describe the bug
If gnome keyring is enabled, it is not possible to login from the console or the display manager with a user managed by sssd.
To Reproduce
Steps to reproduce the behavior:

Setup the sssd identity provider with ad integration or ldap + kerberos
Enable gnome keyring with services.gnome3.gnome-keyring.enable = true
Try logging in from the console or display manager with a user managed by sssd (this fails)
Try logging in using ssh with a user managed by sssd (this succeeds)

Expected behavior
The sssd managed user should be able to login to both the text based and graphical consoles.
Additional context
When gnome keyring is enabled the security.pam module adds the line auth required pam_unix.so  likeauth try_first_pass to /etc/pam.d/login. This requires the presence of a local user account to be able to successfully authenticate. ssh login succeeds, since this line is not present in /etc/pam.d/sshd.
Instead of requiring pam_unix when gnome keyring is enabled, it would be better to implement the authentication flow similar to the below snippet. The only downside is that if additional authentication modules are added, the success=n parameters need to be updated for the flow to jump to the correct module.
auth [success=2 default ignore] pam_unix.so nullok_secure
auth [success=1 default ignore] pam_sss.so use_first_pass
auth requisite pam_deny.so
auth optional pam_gnome_keyring.so
auth optional pam_permit.so

Alternatively the solution utilizing the substack parameter described in https://wiki.gnome.org/Projects/GnomeKeyring/Pam could be implemented.
This issue is also present when enabling other pam modules:
          (optionalString (cfg.unixAuth &&
          (config.security.pam.enableEcryptfs
            || cfg.pamMount
            || cfg.enableKwallet
            || cfg.enableGnomeKeyring
            || cfg.googleAuthenticator.enable
            || cfg.duoSecurity.enable)) ''
              auth required pam_unix.so ${optionalString cfg.allowNullPassword "nullok"} ${optionalString cfg.nodelay "nodelay"} likeauth
              ${optionalString config.security.pam.enableEcryptfs
                "auth optional ${pkgs.ecryptfs}/lib/security/pam_ecryptfs.so unwrap"}
              ${optionalString cfg.pamMount
                "auth optional ${pkgs.pam_mount}/lib/security/pam_mount.so"}
              ${optionalString cfg.enableKwallet
                ("auth optional ${pkgs.plasma5.kwallet-pam}/lib/security/pam_kwallet5.so" +
                 " kwalletd=${pkgs.libsForQt5.kwallet.bin}/bin/kwalletd5")}
              ${optionalString cfg.enableGnomeKeyring
                "auth optional ${pkgs.gnome3.gnome-keyring}/lib/security/pam_gnome_keyring.so"}
              ${optionalString cfg.googleAuthenticator.enable
                "auth required ${pkgs.googleAuthenticator}/lib/security/pam_google_authenticator.so no_increment_hotp"}
              ${optionalString cfg.duoSecurity.enable
                "auth required ${pkgs.duo-unix}/lib/security/pam_duo.so"}

Notify maintainers

@edolstra
@Mic92
@ixmatus
@gnidorah
@flokli
Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
Maintainer information:
# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:
 - security.pam

The problem here appears to be that pam_unix.so is required when for example cfg.enableGnomeKeyring is true. That means if the authentication with pam_unix fails, the pam_sss authentication wail fail, too.
In my opinion the line
              auth required pam_unix.so ${optionalString cfg.allowNullPassword "nullok"} ${optionalString cfg.nodelay "nodelay"} likeauth

needs to be removed entirely.
The case pam_unix is already correclty covered a few lines below:
auth sufficient pam_unix.so  likeauth try_first_pass

related to https://github.com/NixOS/nixpkgs/issues/128116


Suggestion for method 'from'

I try to create the following query:
  => UPDATE users SET password = 'new password'
  -> FROM contacts
  -> WHERE users.contactId = contact.id AND users.username = 'username'
  -> RETURNING contacts.something
  // e.g. knex.table('users').update({password: 'new password'}).from('contacts')...

The value in 'from' method will always replace the value in 'table' method. By now, I can only try to use knex.raw method to rewrite the whole query. Is there any better way to achieve this?

Duplicate e.g. https://github.com/tgriesser/knex/issues/557


[QUESTION] asyncpg slower than psycopg2?

Sorry for a somewhat unrelated query, but I am unable to figure this out. Please close this issue if this is out of scope. Thanks.
Ref: https://github.com/MagicStack/asyncpg/issues/322
I am building a simple FastAPI app, without using an ORM, but using psycopg2 and asyncpg directly.
I am finding that same responses from psycopg2 are upto 30% faster than asyncpg. Queries are simple selects.
Does this mean that for simple webapps using psycopg2 is better?

Thanks for the help here @dmontagu ! :cake: :bowing_man:
Thanks @sagrawal-idrc for reporting back and closing the issue :+1:


TTS Support

I plan to add a TTS command that uses Google's API for it, I'm just leaving a list of the resources I had open for later.
I'm currently blocked in terms of testing the code so far because a phone number is required to sign up, which I can solve later.

https://pkg.go.dev/cloud.google.com/go/texttospeech/apiv1
https://pkg.go.dev/google.golang.org/genproto/googleapis/cloud/texttospeech/v1#SynthesizeSpeechResponse
https://cloud.google.com/text-to-speech/#section-2
https://github.com/bwmarrin/dgvoice/blob/master/examples/play/main.go
https://github.com/diamondburned/arikawa/tree/v3/voice
https://discord.com/developers/docs/topics/voice-connections#speaking
https://github.com/GoogleCloudPlatform/golang-samples/blob/main/texttospeech/list_voices/list_voices.go
https://cloud.google.com/docs/authentication/production

Update on this: The Discord lib I'm using now has updated it's voice examples since I posted this issue, which removed the TODOs in the code and simplified it.


view page

Page to view one product and it's description

Function built, page created but not filled out with commit  b7f0f2ea4e2bbab8bdb896b67a4b7182861f5946


 Amazon Cloud Drive has its own RESTful API.

Just thought the same thing.
I may look at this once #21 is in place - uploading without compression seems like a waste of bandwidth.

That depends on your use case ;)

There’s some proof of concept code I found: http://sprunge.us/fdQF — it requires that an oauth token is in /tmp/token.json, but seems to work for me.
Motivated people could turn that into a clean backend for ACD :).

+1 from me :)
I compile the backend @stapelberg found, and it indeed appears to work and use code from rclone.

+1 too
I'm trying to use Restic + acd_cli (FUSE python client for ACD), but it's very unreliable for now : some operations do not behave as expected (file truncate, rename) and Restic randomly panics as a result.
A working ACD storage backend would do wonder.

I'm currently reworking the interface to the backends, this includes a radical simplification. This is basically done, but not yet merged. For the plan, see #383, the PR is #395.
Afterwards it will be much easier to implement new backends.
Before implementing many new backends, I'd like to have a list of rules that services we write backends for must fulfill, this may include that a test instance of the service must be available that we can run the integration tests against.
Do you by chance know whether there is a test service for ACD we can use for tests?

As far as I know, there is no test instance for ACD. No mention of such a thing here : https://developer.amazon.com/public/apis/experience/cloud-drive/content/restful-api
But the https://github.com/ncw/rclone project already did an ACD backend in go. It seems to be fairly reusable as demonstrated by the proof of concept shared by @stapelberg .

Actually, looking at the revised interface, it would be reasonably easy to do a full wrapper for rclone filesystems. Maybe that way separate implementations isn't needed?

I don't know what @fd0 vision for Restic future is, but it would seem logical to focus the project on the backup intelligence instead of re-implementing a ton of remote filesystems one by one. Besides both project licenses are compatible.
It would also solve the worries about how to test all those backends.
@klauspost was your idea to create a wrapper around rclone/fs/fs.go ? Is it doable without being tightly coupled with the internal logic of rclone ?

Each backend implements the fs.Fs interface.
It should be fairly easy to create a restic backend that uses an rclone filessystem+folder, provided it is already set up in the rclone configuration.

Hm, interesting idea, I have to think about it. Not having to implement all the backends by ourselves looks like a good idea, on the other side (at least at the moment) I must admit that I don't like the thought of a tight coupling between restic and rcclone, as this introduces a dependency that we can't control...

I pledged a 5$ bounty for this feature.
Some thoughts:

Amazon Cloud Drive is using AWS S3 / CloudFront as its backend. The GET requests are always redirected with a Location header to Cloudfront. So you could use the Range header to request only a portion of a pack file as it is required by the new backend API.
You could start with integrating go-acd. That's the library which is used by rclone.

in case the priority of this FR depends on the popular vote: +1

👍

👍

Yes please 👍

How about adding some more bounties to this feature?
See: https://www.bountysource.com/issues/23684796-storage-backend-amazon-cloud-drive

The bounty is now 35 USD: https://www.bountysource.com/issues/23684796-storage-backend-amazon-cloud-drive

Hey, thanks for your interest in restic in general and this backend in particular.
Just to give you a heads-up what's the blocker here: I'm not sure how to handle third-party web services. For the local and sftp backends we have extensive tests in place that are ran for every push/PR as part of the CI tests on Travis. This is also true for the s3 backend, there we're using a local Minio s3 server instance, due to this I've found several bugs in the minio client library we're using in the s3 backend.
How can we run CI tests for backend implementations that require a third-party service? Is there e.g. a test service for ACD we could use? Or maybe just take well-tested code from other projects such as rclone?

One solution might be to register an account with Amazon, whitelist it for the Cloud Drive API and then use that for the CI tests? The downside is that such a test depends on Cloud Drive being available, but I guess we can wait for an hour or so occasionally before merging a PR? :)

That's the only solution I can imaging right now that allows us to run the tests against a live service (and that's desirable in my opinion).
When we add more backends for other services the following will happen:

The number of dependent services for running the CI tests grows
For each backend we will need a test account to use in the tests

Did I forget anything?

Your list looks good. There are of course more effects, but I’m not sure whether they are in scope for the question you’re trying to answer:

More backends means changes that touch the backends API become more involved (need to update more code, test more code).
People who want to run the tests locally need to create test accounts or use their own account.
More backends make restic more appealing to more people :).

This test account must allow parallel connections, as the tests (e.g. for a PR) are run in parallel

I think a simple way to take care of this requirement is to use different directories for each test invocation. Sending requests in parallel is usually not an issue with these services, and the different directories make sure the tests don’t clash.

I think a simple way to take care of this requirement is to use different directories for each test invocation. Sending requests in parallel is usually not an issue with these services, and the different directories make sure the tests don’t clash.

What I meant was more of a question how many parallel connections a service accepts. For most web-based services this won't be limited (at least concerning the number of connections we require), but this may not be the case for other, more obscure services.

When a service limits the number of connections so aggressively that our testing is impacted, we could ask the service owner for an exception or rate-limit on our end as well. As a last resort, we could disable the tests for the backend in question or remove that backend altogether.
But, I suggest we cross that bridge when we get there :).

I tried to use ACD with restic (through Fuse) and the system is still reliable (same errors as kisscool). I tried rClone to check their backend and there isn't any problem.
But I do not like rClone (snapshots...).
Conclusion: +1 for a native ACD backend!

+1 for native ACD backend because the acd fuse driver does not seem to work at all with restic.
acd mount under /mnt/restic
restic 0.3.0 (v0.3.0-29-g190673b)
compiled at 2016-11-15 21:58:36 with go1.7.3 on linux/386
restic init gives me this error.
create key in backend at /mnt/restic failed: Calibrate: scrypt.Calibrate: scrypt: the parameters provided are invalid

Is there a problem with trying to write this? The issue seems to imply it's basically done, but delayed due to other architecture goals. This was a while ago, though, so... should I write this backend from scratch and pull request, or is it still being done internally?

Writing the backend is the not so hard part, figuring out a good user interface for configuring access to the service is what's still missing here. What's the workflow for acquiring an oauth token for ACD?

rclone does this using a local webserver, and the backend for duplicity provided a link to a solution hosted by the developer. The former is cumbersome for headless, and the latter makes this into a service that you'd want to continue to provide for free, although the resources would cost quite a bit. So... I'm not sure. That makes the former the only viable option.
Are you opposed to a webserver, in the style of rclone, to accomplish this? Perhaps restic could ask you what interface to bind on, or give you the opportunity to use a different copy of the same program (this is what rclone does) on a non-headless machine.
I'll have to check how rclone automates this process for testing, if it does that at all.

Yeah, testing is another story. How do we run the CI tests for these backends? Please don't get me wrong, I'd love to add cloud-based backends, but we need a clear strategy for this.
For configuration, I can also imagine having a CLI-based process, where restic prints instructions to the user.
What do you think?

###Idea #1
What about simply integrating with rclone? Kinda a "unix philosophy" type of idea, you continue to be great at doing backups, they continue to be great at cloud copy, and users get best of both worlds.
I'd have to see what their API affords, if it exists, but since we'd be leaving configuration solely to rclone (users would use rclone to add their cloud accounts, then would configure restic to "use rclone") you'd add far less complexity to restic.
###Idea #2
I'll preface this by saying I've not read the docs for ACD, but I have implemented some basic oauth stuff in the past.
If we implemented a webserver, we could simply test the following thing:

Does the webserver respond on the right ports/addresses
When I GET the / route, does it contain what I'd expect it to (a redirect, the text we put there, etc)
If I mock the response that would come from amazon (more thought needed on how to do this correctly) how does it behave?
Still needs a lot more thought: static "testing" credentials of some sort, to test the API. I'll have to look at how you test other services first.

and a few others, but the point is, we mock what we have to, and test the rest. I don't know if that'd test everything but it'd test as much as we possibly could. I've not written extensive tests for my projects in the past, so please correct me if this isn't testing the right things haha

Sorry for glossing over your reply, was typing mine a bit before I saw yours!

For configuration, I can also imagine having a CLI-based process, where restic prints instructions to the user.

I imagine the process would probably be something like this, and this... again, is just rclone. I only reference it over and over because it's the only program that does exactly what we're talking about quite well.
To start:

Then, setup ACD:

Using auto config (non-headless)
It, very quickly, opened a local webserver in my browser which immediately redirected to Amazon for login, and once I signed in, it communicated with the CLI app producing:
This, in my browser:

and this, in my command line
(60% of the credential is missing, I don't think this is a security risk, haha)

Then I save it, and it's now usable in the program.

Thanks for describing the process in such great detail, that is already very similar to what I had in mind.
I'm wondering: Why is the webserver needed at all? This process works for a "workstation" type of machine, but not on a server (where there is not browser). The workflow used by rclone is described here: http://rclone.org/remote_setup/
I don't know why we need a webserver for this, but I haven't implemented an oauth-based login workflow yet.
We'll also need a config file to store the token configured for the remote in, that's also not yet done.

From my understanding, which is limited, the oauth data is provided to the user using the GET data in a redirect.
Have a look at the URL in my screenshot of the browser. That was put there by Amazon. After I hit sign-in on my amazon cloud drive, it redirected me, immediately, to that <IP_ADDRESS> URL.
Perhaps, and this is likely since they (rclone) implemented a webserver instead of picking another simpler solution, that is the only way to get this data. When I implemented oauth before, this seemed to be the implication. If I am correct, then it follows that you must run your own webserver to provide a page to redirect to amazon, and a page to handle the redirect from amazon to do this.

That sounds plausible. Let me think about a strategy here, this may take some time.
We'll need to:

have a config file to store the authentication tokens
have "instances" of backends, e.g. something called my_amazon_account which is a ACD backend configured with a login token, so users can run restic --repo my_amazon_account:/foo/bar/dir ...
have a workflow to create these login tokens

Anything else I'm missing here?

I think you got the big stuff outlined there.
Would you want to move all current backends into a single abstraction that supports this, or would this whole system become a "cloud" backend in the current sense of a backend (which itself is configured through special restic commands)?

Have a workflow to refresh auth tokens (if they expire and the provider supplies a refresh token, which should be stored next to the auth token in your point 1)

Here's some background in regards to embedding a client secret in open source applications: http://stackoverflow.com/a/28109307

Go std library has this: https://godoc.org/github.com/ncw/rclone/fs#MustObscure
and it's what rclone uses: https://github.com/ncw/rclone/blob/f3365dd2516709c8eb6d9b55496344b853ec89c7/googlecloudstorage/googlecloudstorage.go#L53

There is no real solution, it is a broken concept to assume that any client can keep a secret.
However, if you consider what the client secret contains, it is not that important. The only real thing it allows is for Amazon (and others) to be able to identify a specific client, nothing more. It does not grant any special access - your tokens are used for that.
Sure a publicly available "client secret" can make other application identify themselves as restic, but other than risk that "restic" will be banned (or more likely rate limited) as a client, there is not much risk at exposing the client "secret". It will never put any user data in jeopardy.

The problem here is that somebody needs to register the clientID, for example me. If I'm using my normal Amazon account (just to name an example service), and "violate" the TOS for the service by publishing the client secret, they can terminate my account. That's not something I'm going to risk.
Another problem is that once the client secret changes (or is revoked), we're stuck with older versions of restic e.g. in Debian stable which are unable to communicate with the service. This is the case even if access to the service is restored shortly after, but the client secret is changed.
I've thought about possible solutions and found only two:

Live with the risk and just put the client secret into the source
Build restic in a way that users need to register their own client ID and client secret, via a nice UI that minimizes the hassle

Currently, I'm in favor of the second option, we need a UI for the oauth token thing anyway. What do you think?

If I'm using my normal Amazon account [...]

I know that Nick has had some correspondence with Amazon, since rclone was being rate limited due to many users. It is however my impression (from memory) that they were quite forthcoming and encouraged OS development, and have made exemptions for his client. So I guess my advise would be to contact them and see how things go from there. In the overall picture I don't think they would mind the business coming from restic users.

Interesting idea, do you have any hint on who to contact at Amazon?
For Microsoft OneDrive he said that he did not contact anyone: https://github.com/ncw/rclone/issues/372

I know that @breunigs had bad luck with his amazon cloud drive duplicity backend — they wouldn’t give him any rate limit exemptions AFAIK.

I have only read the last few comments, so please forgive me if this info is not needed:

rclone implements the web server on top of it offering remote setup where you copy the URL. Having a local webserver is just more convenient
if you want to whitelist any redirect target in Amazon, it has to be on a https machine – linking to http URLs is not okay. Only exception is localhost. So, for remote setup you can either redirect the user to a blank page and hope they realize what they have to do, or host some page with instructions. I added https://breunig.xyz/duplicity/copy.html for duplicity, since it doesn't have https infrastructure yet. Amazon will add all details in the query string, so you can get away with making this a static page
You need an Amazon Developer account. You can use your existing credentials to log in I believe, but you can also create a new one
There is a process where you register your app and then at some later stage you can create a security profile for said app. This process is very confusing, because of horrible UX, but it should work without human interaction from Amazon.
 It's clear there are multiple stages: per user, per API endpoint, per credentials
If you want production limits, you send an email with "details" to<EMAIL_ADDRESS>Use a big player mail server, or they will tag you as spam and it takes a month or two until some poor soul went through all their spam.

Also, a final word of advice: read through rclone's workarounds for Amazon Drive. The API contains a lot of undocumented "eventual consistency" gotchas. It even goes out of its way to cache an outdated response it gave you, so that you need to wait even longer if you were too hasty to begin with. This is on top of it reporting errors when there are none, one just needs to wait.
HTH,
Stefan

Thanks for the information!

Just throwing something out there:
What is we remove all (but local and REST) backends from restic and stick them into restic/rest-server?
This allows restic to focus on doing backups properly and filesystem implementations are done in the rest-server.
This also leaves restic with just 1 backends API to maintain.
This doesn't solve the testing problem, but will certainly help keep the restic source clean/focussed and it is easier to make API changes inside restic.

Thanks for the suggestion. Unfortunately I don't like it at all, in my opinion this approach will lead to even more errors. The backend API interface was stable for a long time, then changed recently, and will be stable again. The interface is already rather small.
We should try to get backends into restic (including proper CI tests) as soon as possible, that's IMHO the only way to make sure they work.
In case of the Amazon ACD backend, we need to answer the outstanding questions first.

The Amazon Developer Guide for Amazon Drive (what's it called these days) states that:

What Not To Build
We understand that developers spend a lot of time creating apps that customers love to use. We want to give you some guidelines for apps not to build on the Amazon Drive platform to save you time in building something that will not be approved during the App Review process. While this isn’t an all-inclusive list, focus your energy on developing apps that solve customer problems, which don’t violate our Terms of Use, Brand Guidelines and the App Distribution and Services Agreement.
[...]

Don’t build apps that encrypt customer data

I feel that Amazon Drive is not the right platform for storing encrypted backups.

Interesting. This must be a new addition, as it definitely was not the case when ACD support was added to Arq.
Seems ACD is not a real storage option after all.

Indeed an addition within the last year, wasn't listed one year ago: http://web.archive.org/web/20160322034250/https://developer.amazon.com/public/apis/experience/cloud-drive/content/developer-guide

Amazon has since clarified this in https://forums.developer.amazon.com/questions/54909/impact-of-dont-encrypt-customer-data-part-of-drive.html:

What if the customer choses to encrypt their data?
They can do that, and that is fine.

So, restic and other apps should be good.

I think their intention is to protect the users having their data encrypted without a way to recover it.
Steffen

One other motivation which I find plausible is to increase interoperability — if each application encrypts their files, the user’s ability to switch between applications is severely hampered.

I asked Arq Backup support. They encrypt everything, and said that their app had been approved by Amazon, and to not worry.
I'm not sure what Amazon is trying to say. But seems that are now evaluating each case as they come in.

Not sure if anybody is aware of the recent ACD drama with acd_cli and rclone, but a TL;DR of the situation is that they have had their ACD API access revoked due to TOS violations.  Their efforts to regain API access are apparently being hampered by the fact that Amazon has stopped accepting new third-party apps for ACD.  I assume this latter revelation stops any Restic ACD support in its tracks, unless the project had already obtained ACD API access.

acd_cli API access was revoked due to a security issue with their ouath app, not a TOS violation.  The problem has been fixed and Amazon re-instated their key.  Although this is off topic from this project.
New ACD API access is currently closed.

Thanks for posting this here, I wasn't aware of it. I had reservations implementing ACD, and it seems that Amazon indeed did not like secrets in the code of an Open Source program: rclone was banned for it: https://forum.rclone.org/t/rclone-has-been-banned-from-amazon-drive/2314
On the other hand, acd_cli implemented an OAUTH auth service (not sure what the correct nomenclature here is). This handles authorization for all users, and there apparently was a bug that allowed people to access/modify other people's files.
Since Amazon isn't accepting new clients anyway I'm closing this issue for now. Thanks!


Increment version for extensions releases

Increment package version after release of Azure.Extensions.AspNetCore.Configuration.Secrets

API change check
API changes are not detected in this pull request.


Test with external project before releasing beta version

As we will be releasing the first beta version in two days, we must test it with an external project before doing it.
https://github.com/HQ20/contracts/milestone/2
Who can I assign it to?

@obernardovieira, what does testing it with an external project mean?

$ mkdir some-project
$ cd some-project
$ npm init -y
$ yarn add truffle
$ npx truffle init
$ yarn add HQ20/contracts#dev

and write some contracts using the contracts from this repository.

Is this tested?

Should it have appeared here?

Trying with npm install @hq20/contracts#dev I get this:

see https://docs.npmjs.com/files/package.json#github-urls
so you should have used yarn add HQ20/contracts#dev instead.
I just tested this with a local repository.
Deployed ok, and also some methods calls are also ok
truffle(development)> mycontract = await MyContract.deployed()
undefined
truffle(development)> root = await mycontract.ROOT_ROLE()
undefined
truffle(development)> await mycontract.roleExists(root)
true
truffle(development)> web3.utils.toUtf8(root)
'ROOT'
truffle(development)> await mycontract.hasRole('0xe8a5e64E6EBb88F7DdCE7C91723e4d92b73B8FFc', root)
true
truffle(development)> await mycontract.hasRole('0xB1d9253c58E263242D8B8a790D486D20e64610DC', root)
false
truffle(development)>

https://www.npmjs.com/package/@hq20/contracts

Close this issue when you see it :+1:

I did use yarn add HQ20/contracts#dev first, but I was expecting the contracts to appear in node_modules:

Ok, I see it now!


Timezone issue in calendar navigation

The date range used to fetch a user's personal calendar has a remaining timezone issue.

Reassigning to @Coenego for review.


[ZEPPELIN-1109] Remove bootstrap dialog fade-in/out animation

What is this PR for?
This PR will fix ZEPPELIN-1109.
I'm not sure this approach can be the best way for fixing this issue since I would prefer to have the fade-in/out animation for bootstrap dialog. So if anyone has better idea for this, please let me know.
What type of PR is it?
Bug Fix
What is the Jira issue?
ZEPPELIN-1109
How should this be tested?

Press Enter button right after clicking any bootstrap dialog (i.e. trash can icon in the notebook)
-> Bootstrap dialog will be shown up multiple times
Apply this patch and build zeppelin-web
Try number 1 again -> you can see there is no fade animation so you don't have time interval to generate multi dialogs

Screenshots (if appropriate)

Before

After

Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? no

Thanks @AhyoungRyu for the fix.
LGTM

Merge into master if there're no more discussions


fix fix dead loop in _handle_mem_exceed_limit function when reduce memory for load

Proposed changes
Issue Number: close #xxx

W0712 03:25:36.371089 3393743 thread_mem_tracker_mgr.h:178] MemHook alloc large memory:<PHONE_NUMBER>0, stacktrace:
0x5607a64e0b04  malloc
0x5607afb6c3d8  operator new()
0x5607a64d7f0a  std::vector<>::_M_realloc_insert<>()
0x5607a64d2421  doris::LoadChannelMgr::_handle_mem_exceed_limit()
0x5607a64d1b86  doris::LoadChannelMgr::add_batch()
0x5607a65cff16  std::_Function_handler<>::_M_invoke()
0x5607a5b54422  doris::PriorityThreadPool::work_thread()
0x5607afbedde0  execute_native_thread_routine
0x7fbee6f96609  start_thread
0x7fbee7225133  clone
       for (auto item : writers_mem_snap) {
                // multimap is empty
                if (item.second.empty()) {
                    continue;
                }
                all_writers_mem.emplace_back(kv.second, item.first, std::move(item.second));
                size_t pos = all_writers_mem.size() - 1;
                tablets_mem_heap.emplace(std::get<2>(all_writers_mem[pos]).begin(),
                                         std::get<2>(all_writers_mem[pos]).end(), pos);
            }

The Iterator of std::get<2>(all_writers_mem[pos]).begin() or  std::get<2>(all_writers_mem[pos]).end() may lose effectiveness when all_writers_mem is expanded.  An invalid Iterator is used, which causes an endless loop in some scenarios.
Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.92 seconds
stream load tsv:          504 seconds loaded<PHONE_NUMBER>9 Bytes, about 141 MB/s
stream load json:         19 seconds loaded<PHONE_NUMBER> Bytes, about 118 MB/s
stream load orc:          65 seconds loaded<PHONE_NUMBER> Bytes, about 16 MB/s
stream load parquet:          31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select:          29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size:<PHONE_NUMBER>1 Bytes


Add AWS ECR support (round two)

Closes https://github.com/jupyterhub/binderhub/issues/705 and follow-on/alternative to https://github.com/jupyterhub/binderhub/pull/920.
Includes updated user documentation on how to configure AWS IAM and BinderHub to use AWS ECR as the Docker registry. Modifications are additive, in that existing configurations should not be affected, and have been tested with ECR.

Cc @scotthq

Thanks for this PR.
Has someone been running this code on AWS?
Having different registry classes is a good idea. Could also be used for GitLab like  registries (xref #986)

Yes we are running this code on AWS with ECR and no further modifications.

Hi guys,
Any news about this PR? This is the missing point to me to adopt binderhub as my private notebooks provisioning pipeline.
Can I help in anything?
Best regards,
Clovis Chedid

Hi @clovischedid! Thanks for your interest and offer. I went ahead and responded to all the conversations and applied the naming change requested. I have also re-requested a review from @betatim. Hopefully it's up to snuff now. Otherwise I will try to be quicker on the iteration this time around. :)

Status update: addressed all unresolved review comments. Awaiting re-review. :)

Nice work!
@clovischedid do you have a moment to give this a spin on your deployment? Our tests in this area aren't mega so the knowledge of "someone actually used this feature and it worked" would be great to have.

Hello guys,
I'm sorry for my delay.
@betatim I will do my best, but currently, I do not have a production ready deployment. At this time, I'm focusing myself on make binderhub running with AWS managed Kubernets service EKS. I'm facing some network problems as newer version o EKS do not make bridge network as default.
By the way, I will try to make AWS ECR as my container registry, but, could you help me? I can't realize how to make helm deploy binderhub from a github branch instead of published version on helm repo.
Regards,
Clovis Chedid

Nice work!
@clovischedid do you have a moment to give this a spin on your deployment? Our tests in this area aren't mega so the knowledge of "someone actually used this feature and it worked" would be great to have.

Hello guys,
I'm sorry for my delay.
@betatim I will do my best, but currently, I do not have a production ready deployment. At this time, I'm focusing myself on make binderhub running with AWS managed Kubernets service EKS. I'm facing some network problems as newer version o EKS do not make bridge network as default.
By the way, I will try to make AWS ECR as my container registry, but, could you help me? I can't realize how to make helm deploy binderhub from a github branch instead of published version on helm repo.
Regards,
Clovis Chedid

Rebuild the BinderHub image from source and set the image to it with helm.
I might be able to test this PR in a couple of weeks.
Thanks @ivan-gomes for the work!

@clovischedid @chicocvenancio just checking in if you had a chance to test yet, no pressure.

@betatim do you think it would be a good idea to merge so it would be easier for those interested like @clovischedid to try it out without having to build from source? For what it's worth, I have it deployed as documented with ECR and this addition doesn't affect default behavior. Hoping to avoid merge conflicts with master again.

I'd be happy to test and report back as well!

It would be great to have someone test drive these instructions. Though given how long they've already been "in flight" it might be easier to merge them and then update them later as we discover things. With it being merged we might get more eyeballs on it.

@betatim I totally agree, especially considering we just resolved another round of merge conflicts in documentation.

I'll be testing this out over the next few days - will report back

@TomasBeuzen you can submit a PR to https://github.com/ivan-gomes/binderhub/tree/aws-ecr-support and I can merge it in. If it's minor and easier for you, you can also describe it here and I would be happy to incorporate the feedback into this PR.

Though given how long they've already been "in flight" it might be easier to merge them and then update them later as we discover things. With it being merged we might get more eyeballs on it.

@betatim I think we should go for it.

Apologies for delays on this, I got this up and running. I just made a PR for a few potentially helpful clarifications to the docs but otherwise thanks for all your work on this!

That's great news @TomasBeuzen! I merged in your helpful changes. Thanks!

@betatim just wanted to put this back on your radar. Hope you don't mind. :)

Pinging this as it would be nice to have it merged 🙂  Happy to help in any way I can!

Hi folks, would love to see this functionality if it's still in the roadmap!

I would be happy to resolve conflicts again once this PR has been tentatively / conditionally approved. :)

I think one issue to consider is how to test this. In the short term we can probably ask a few AWS users to test it for real and sign off on it. In the long term how do we ensure we don't inadvertently break it? It looks like the AWS registry works in a completely different way to a standard docker registry, so it's not going to be regularly tested. Do you have any ideas?

Happy to test and feed back @manics

A few thoughts after quickly going through this and related issues:

The Registry class has been made configurable since this PR was written so this part will require some minor changes
It looks like boto3 is only used in the registry class. Although it's usually fine to install I've found it can occasionally cause conflicts with other dependencies. Not an issue at the moment, but might be a problem if we have to add other optional libraries in future. I'm therefore inclined to put the import inside the class instead.
I'm uneasy about increasing the RBAC privileges for all deployments, maybe make it conditional? Could you also document the _patch_docker_config_secret method to include an explanation of what it does and why it's necessary? Or if you can think of a nicer way of doing this even better!
Is there any way of getting some mock tests in, e.g. with http://docs.getmoto.org/en/latest/docs/services/ecr.html (it may not be possible). Otherwise there's a risk we'll accidentally break it, and you won't notice until someone deploys it in production.

This PR sparked a lot of interest in the past, and it would be really neat to allow Binder to run with AWS ECR.
I updated the PR by rebasing and making the changes that @manics and others requested. Those include:

moving the import of boto3 within the registry class;
adding some explanation on the patching of secrets;
making the RBAC increase conditional (with associated docs in setting up Binder);
referencing the secret name instead of hardcoding it.

As for the mock-testing, I fully agree that this would certainly be nice to have. However the Moto library has not implemented all the calls that would be necessary to test here. I looked for other solutions but could not come up with much.
Re-iterating here, but my organization has been using this fork for a while and it has been working perfectly. Maybe @teticio or @TomasBeuzen can confirm. I will be sure to test again with this updated version and report back.
Would love to hear some feedback!

Coming back to this after a while to take a fresh look, and after seeing https://github.com/jupyterhub/binderhub/issues/1506#issuecomment-1220103944, I was wondering if we could either move the ECR token request to the Builder and pass in the credentials as an environment variable of volume instead of overwriting the Kubernetes secret?
It's a bigger change, but I've just done some work to refactor the build classes https://github.com/jupyterhub/binderhub/pull/1518 and I think further changes to KubernetesBuildExecutor to support custom registries or passing tokens is reasonable, especially if the outcome is cleaner code. A few ideas:

Split the submit() method to make it easier to override parts of the method, or add a hook that can be called https://github.com/jupyterhub/binderhub/blob/c4af713bb20ed90606e9795834d160bd8a332578/binderhub/build.py#L348-L379 and fetch the ECR token, then either:
a. Convert the ECR token into login credentials, pass them in as an environment variable to the build pod instead of passing a secret. The build pod would need to accept a script/hook to run docker login or equivalent. We already use GIT_CREDENTIAL_ENV for private git credentials
https://github.com/jupyterhub/binderhub/blob/c4af713bb20ed90606e9795834d160bd8a332578/binderhub/build.py#L78-L87
so registry login credentials sounds reasonable?
b. Use an service account (IRSA for AWS) as suggested in https://github.com/jupyterhub/binderhub/issues/1506 and handle the entire ECR login process in the build pod. This has the advantage of being cleaner, but means the repo2docker build pod needs to includes the AWS dependencies.

@yuvipanda what do you think? I'm leaning towards option (a), and modifying repo2docker to optionally login to the registry before pushing, but I'm very keen to hear other ideas.

This is the sort of repo2docker change I had in mind if we go down the route of passing the credentials into the build pod:
https://github.com/jupyterhub/repo2docker/compare/main...manics:repo2docker:registry_login_kwargs?expand=1

any updates on this PR? we would really love to have that support!

@omri-shilton I'm working on an alternative approach:
https://github.com/jupyterhub/binderhub/issues/1623


ip pool option for EPG

IP pool for endpoint group:
This feature enables users to specify an optional range of ip address
for EPG from the network subnet.
fixes https://github.com/contiv/netplugin/issues/710

@jojimt  @gkvijay @abhinandanpb PTAL


Metric list from Filecoin Status & Priorities (Challenges, Opportunities, Roadmap)

There are a few interesting metrics in FIL Dev Summit FDS-5 | Filecoin Status & Priorities (Challenges, Opportunities, Roadmap).
Metrics Overview

~Storage Onboarding~
~Capacity, Power, Utilization~
~Data Onboarding: Focusing on deals (DDO and non DDO)~

~How much data is coming from each region?~
~How much data is coming from each use case?~
~How much data is coming from each onramp?~

Should we make this public?

~Filecoin Plus~

https://github.com/davidgasquez/filecoin-data-portal/issues/62

Economy

~How many tx are happening within the economy (in FIL)?~
What’s the GDP of the Filecoin Network (internal tx)?
How much are we selling (exports)?
How much are we selling to each market?
How much are we buying (imports / OpEx)?
What are the network’s largest costs (OpEx)?

~User Dev Activity~

https://github.com/opensource-observer/oso/issues/2475

Made some visualizations for Data Onboarding by Client Region and Industry.
https://numbers.filecoindataportal.xyz/data-onboarding

Filecoin In Numbers now has many of the visualizations in the initial list!
https://numbers.filecoindataportal.xyz/
Closing this issue as the trickier ones are now on their own issue.


Closes #349 - Extend demo scenarios by EUM server

The following Pull request: https://github.com/openapm/spring-petclinic-microservices/pull/1 should be taken into account.
I would push the updated images of api-gateway and grafana, after the pull request is removed.

This change is

@tangerstein @JonasKunz  I'm closing this PR for now, because it has to be checked whether these changes are still valid with the latest EUM server and its configuration.
Please open a new PR or reopen this one, when the changes have been updated.


Any major differences to manga-py?

Is there any major difference vs manga-py? I exclude the folder naming or organisation of files or the way app displays information at download. I think more like better quality of images, or additional content like soundtracks.

Not sure, I haven't really tried it :). I'm just trying to focus on downloading webtoons as fast as possible while still making it portable in that sense.


POST not allowed in basic auth ?

Is there a strong decision for not to allow POST in basic auth requests?
I have this use case.

POST is absolutely allowed.
Can you please fill out the issue template that you had to delete to file this? What are you trying to do? What happened? What did you expect to happen? What code did you use? What version of Requests?

+1. I'd expect POST to be supported in all auth examples.


[BUG] ios onRecvNewMessages fail

OpenIM Server Version
3.8.0
Operating System and CPU Architecture
Linux (AMD)
Deployment Method
Docker Deployment
Bug Description and Steps to Reproduce
The message reception works fine on Android, but iOS is unable to receive messages.
Screenshots Link
No response

Please update the sdk and server to the latest. If the problem still exists, please collect detailed logs so that we can better locate the problem.

SDK version "open-im-sdk-rn": "3.8.1-rc.1",

onRecvNewMessages is not triggered when a message is received, but is it possible to query messages via other apis?

i don't understand object-c development, can not fix it
please fix it

When I use onRecvNewMessage on IOS, I can successfully receive new messages

Thank you. According to your description, we should already know where the problem is and we will fix it as soon as possible

When I use onRecvNewMessage on IOS, I can successfully receive new messages

Thank you. According to your description, we should already know where the problem is and we will fix it as soon as possible

You can try using the content of this pr to confirm if the fix has been completed. However, it is expected that onRecvNewMessages will also be deprecated in the next version, please note