App rejected for Mac Catalyst due to opening external browser during Sign-In

Describe the bug
A clear and concise description of what the bug is.
My app was rejected and I received the following message from Apple:
<<
The user is taken to the default web browser to sign in or register for an account, which provides a poor user experience.
We advise to update to the latest Google SDK and use “SFSafariViewController” instead of “UIWebView”.
This will ensure that the app correctly implements an account authentication mechanism that does not link users out to Safari.

I am using the latest version as of March 31st 2024.
Even after using the example app inside GoogleSignIn-iOS, this is still opening an external browser.
This seems to only happen in Mac Catalyst.
To Reproduce
Steps to reproduce the behavior:

Get the latest version of GoogleSignIn-iOS repository on a Mac
Run "open GoogleSignIn-iOS/Samples/ObjC/SignInSample/SignInSample.xcodeproj" and build/run for Mac catalyst
Click on the Sign-In button
It opens an external browser and not an in-app browser.

Expected behavior
An in-app browser should be opened instead so that Apple doesn't reject apps.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment

Device: [ MacBook Pro ]
OS: MacOS 14 (likely earlier versions also)
Browser: [ your default browser: Safari, Brave, Firefox, ... ]

Additional context
Add any other context about the problem here.

The quote from the app store review above is confusing to me. ASWebAuthenticationSession is used by AppAuth (and thus GoogleSignIn-iOS - what I call 'GSI' below) when running on Mac Catalyst, and is specifically made for authentication via 3rd party services. I'm not aware of any changes in Apple policy on this subject.
Moreover, neither GSI nor AppAuth use UIWebView. Are you perhaps creating your own UIWebView somewhere?
Can you confirm the versions of GSI and AppAuth you see in your app? One thing I noticed is that the SignInSample app doesn't declare a Mac Catalyst run destination. Can you verify that you have one for your app? I modified the SignInSample app and you should see something like the below:

Another thing: can you set a breakpoint in OIDAuthorizationService+IOS.m at this location? I want to make sure that the correct user agent is being created for your Mac Catalyst run destination. This is important because ASWebAuthenticationSession is used for the auth flow when using Mac Catalyst (the agent created should eventually call through to here).

+1. The same here. I've been rejected with this reason too.

Apple: The user is taken to the default web browser to sign in or register for an account, which provides a poor user experience.

They report seeing this pop-up window.

Can you confirm the versions of GSI and AppAuth you see in your app? One thing I noticed is that the SignInSample app doesn't declare a Mac Catalyst run destination. Can you verify that you have one for your app? I modified the SignInSample app and you should see something like the below...

I don't think it's Catalyst related. It's macOS related at all. There is the same popup on the Mac destination in the Sample app.

@ostatnicky are you able to inquire with the reviewer to understand what they want the macOS app to do instead? GSI (via AppAuth) uses ASWebAuthenticationSession, which will use the default browser on macOS for the auth flow (see the quote from the documentation below). This is out of our control and is by Apple's design. Additionally, ASWebAuthenticationSession is Apple's recommended solution for authentication; so, I'm pretty confused by the rejection per the terms mentioned above.
"In macOS, the system opens the user’s default browser if it supports web authentication sessions, or Safari otherwise."
https://developer.apple.com/documentation/authenticationservices/aswebauthenticationsession

@mdmathias Ok, I’ll try to ask them.

The quote from the app store review above is confusing to me. ASWebAuthenticationSession is used by AppAuth (and thus GoogleSignIn-iOS - what I call 'GSI' below) when running on Mac Catalyst, and is specifically made for authentication via 3rd party services. I'm not aware of any changes in Apple policy on this subject.
Moreover, neither GSI nor AppAuth use UIWebView. Are you perhaps creating your own UIWebView somewhere?
Can you confirm the versions of GSI and AppAuth you see in your app? One thing I noticed is that the SignInSample app doesn't declare a Mac Catalyst run destination. Can you verify that you have one for your app? I modified the SignInSample app and you should see something like the below:

Another thing: can you set a breakpoint in [`OIDAuthorizationService+IOS.m` at this location](https://github.com/openid/AppAuth-iOS/blob/master/Sources/AppAuth/iOS/OIDAuthorizationService%2BIOS.m#L36)? I want to make sure that the correct user agent is being created for your Mac Catalyst run destination. This is important because `ASWebAuthenticationSession` is used for the auth flow when using Mac Catalyst ([the agent created should eventually call through to here](https://github.com/openid/AppAuth-iOS/blob/master/Sources/AppAuth/iOS/OIDExternalUserAgentCatalyst.m#L81)).

The quote from the app store review above is confusing to me. ASWebAuthenticationSession is used by AppAuth (and thus GoogleSignIn-iOS - what I call 'GSI' below) when running on Mac Catalyst, and is specifically made for authentication via 3rd party services. I'm not aware of any changes in Apple policy on this subject.
Moreover, neither GSI nor AppAuth use UIWebView. Are you perhaps creating your own UIWebView somewhere?
Can you confirm the versions of GSI and AppAuth you see in your app? One thing I noticed is that the SignInSample app doesn't declare a Mac Catalyst run destination. Can you verify that you have one for your app? I modified the SignInSample app and you should see something like the below:

Another thing: can you set a breakpoint in [`OIDAuthorizationService+IOS.m` at this location](https://github.com/openid/AppAuth-iOS/blob/master/Sources/AppAuth/iOS/OIDAuthorizationService%2BIOS.m#L36)? I want to make sure that the correct user agent is being created for your Mac Catalyst run destination. This is important because `ASWebAuthenticationSession` is used for the auth flow when using Mac Catalyst ([the agent created should eventually call through to here](https://github.com/openid/AppAuth-iOS/blob/master/Sources/AppAuth/iOS/OIDExternalUserAgentCatalyst.m#L81)).

@mdmathias I am not creating my own UIWebView.  I am just calling GIDSignIn.sharedInstance.signIn(withPresenting: self)  from my view controller.  I was able to reproduce with the embedded example.   It appears that Apple may have changed their policy in that regard because in the past, my app was accepted (I never had this issue before).  It appears that they don't want a browser to be opened externally but within the app.
I tried this with GSI version 7.0.0 and 7.1.0: both builds were rejected.
Regarding the SignInSample app, it does not have Mac Catalyst as a target (I had to configure one).  I also noticed the same behavior if I use the Mac target (designed for iPad) [which was available without adding any new target].
Here is a screenshot of what I have after putting a breakpoint at the location you provided.  Please let me know if there is any particular field you are interested in? (It is painfully slow / I can try to give you more information tomorrow if that works)

One more thing, I saw that another user posted the screenshot that Apple provided showing the reason why the app was rejected: I had the exact same screenshot (with a different app name though).

@startechm thanks for following up. It does look like you're getting the ASWebAuthenticationSession that you're supposed to. If you look above, you will see my comment that this is how ASWebAuthenticationSession works on macOS. It will open the default browser and start the auth flow. I'm confused by the rejection because it runs counter to how ASWebAuthenticationSession works, and that type is what Apple tells us to use for auth flows. A policy change to use a web flow within a macOS app is essentially the same as saying, "Don't use ASWebAuthenticationSession," which hasn't been announced to my knowledge and would be a kind of deprecation of that type.
I suggest reaching out to the reviewer to ask why your app is getting rejected on macOS for using ASWebAuthenticationSession, which is what Apple says to use.
Link to the class for your convenience:
https://developer.apple.com/documentation/authenticationservices/aswebauthenticationsession

@mdmathias Yes, I'm waiting for a response from Apple reviewer...

@mdmathias Ok, I've got the response from the Apple reviewer:
Hello,

Thank you for your reply regarding the external login and Google SDK, and your attempted compliance with the 4.0 guidelines.

We advise to update to the latest Google SDK and use “SFSafariViewController” instead of “UIWebView”.

This will ensure that the app correctly implements an account authentication mechanism that does not link users out to Safari.

We look forward to reviewing your revised binary.

Best regards,

App Review

But from what I understand, you're using SFSafariViewController there, aren't you?

I tried easy example how to present SFSafariViewController on macOS and it appears in "poor" way. So, not sure what's is the solution.
func openInSafari(url: URL) {
    let config = SFSafariViewController.Configuration()
    config.entersReaderIfAvailable = true

    let vc = SFSafariViewController(url: url, configuration: config)
    present(vc, animated: true)
}

@mdmathias When I read the documentation it looks that init(url:callbackURLScheme:completionHandler:) is going to be deprecated in iOS and Mac Catalyst 17.4. Maybe it's causing the whole issue here. And AppAuth is using it in OIDExternalUserAgentCatalyst.m on line 82.
https://developer.apple.com/documentation/authenticationservices/aswebauthenticationsession
https://developer.apple.com/documentation/authenticationservices/aswebauthenticationsession/2990952-init

Thanks for the reply @ostatnicky. I appreciate it, and I'm sure that this is frustrating for you.
The answer you received seems to repeat the same information. I still don't understand it, as it implies that ASWebAuthenticationSession shouldn't be used on macOS, which doesn't make sense to me.
Good find regarding the deprecation of -[ASWebAuthenticationSession initWithURL:callbackURLScheme:completionHandler:]. I will point out that OIDExternalUserAgentIOS also uses this method, and it appears that iOS apps are not getting rejected for this reason. This leads me to think the deprecation is not the cause of this rejection, and gives me reason to suspect that the rejection is an error.
I'm not sure what I can do to help on my side, but I will reach out to colleagues and see if they have any ideas. On your side, perhaps you can ask for clarification on whether or not the reviewer is saying that ASWebAuthenticationSession shouldn't be used on macOS or Mac Catalyst?
I will follow up here if I learn anything.

@mdmathias I've asked the reviewer about if ASWebAuthenticationSession shouldn't be used on macOS or Mac Catalyst and it looks that there are non-technical people. It makes sense. @mdmathias Are you able to connect with Apple to resolve this issue?
Hello,

Thank you for your response.

Regarding Guideline 4.0, should you require more assistance with resolving this issue, [Apple Developer Technical Support](https://developer.apple.com/support/technical/) is available to provide direct one-on-one support for discrete code-level questions.

Best regards,

App Review

It's funny, I have a screen in my app that use WKWebView wrapped to UIViewController and just simply presented and the result looks much better with nice modal screen. But when you try to use SFSafariViewController all started to be mess on MacOS :D

@startechm Did you resolve it somehow?

@ostatnicky Unfortunately, I do not personally have the ability to directly connect with anyone at Apple on this. I have talked with colleagues internally and everyone is just as perplexed as we are. I still believe this rejection is a mistake.
That said, perhaps your next step is to schedule a meeting with Apple Developer Technical Support as the reviewer suggests. I imagine that person will look at the situation as we do.

@startechm Did you resolve it somehow?
I have had multiple interactions with Apple but every time they send me about the same answer.  I don't think they are technical also.

I was not able the resolve the issue on my end so I asked for an exemption (because I told them that the previous version worked like this and I will try to fix it in the next version) and they granted it to me but they said that I cannot submit a future version until this is fixed.
It appears that the issue is the fact that it should open a ViewController (so it looks like it is running within the app) instead of opening an external app. For iOS/iPhone it seems to be working as they expect.  Would it be hard to fix the behavior so it opens a ViewController instead of an external app?
If you try to use for example "Firefox" as your default browser, it becomes more obvious that it is opening an external app (browser).
Not sure what can be done at this point.  It is very difficult to contact Apple and get a good answer.

@mdmathias @startechm I’ve sent a request for a technical meeting. We will see if they can give us some advice.

We've received the same rejection reason for the last couple of versions. Our macOS app, not a Catalyst one, doesn't allow control over where the Google SDK redirects users for authentication.
Any advice on convincing the Apple review team? No success yet.

Thanks @mattisssa for the note. I will share this will my colleagues internally.
For you and others on this thread, I'm doing what I can to get clarification and resolution. I will be sure to follow up here when I have something to share.

Hi @ostatnicky @startechm @mattisssa. We are engaging with Apple to try to better understand this issue. Are you able to share your app name(s)? Please share them here if so. If you're not comfortable sharing on this thread, then please let me know if you'd be able to share if we found a more private mechanism. I can figure something out there.

@mdmathias we have the same issue for macOS "Reflection Journal & Prompts".
Here is a screenshot, if they will need the submission ID:

We are engaging with Apple to try to better understand this issue. Are you able to share your app name(s)? Please share them here if so. If you're not comfortable sharing on this thread, then please let me know if you'd be able to share if we found a more private mechanism. I can figure something out there.

@mdmathias our app name is "logdt - get things done & more".  We have a version on the Mac App Store that has this issue (version 1.7.0 which was initially rejected but we got an exemption assuming we can fix it in the next version).

@mdmathias Sure, our app is Done! – Happy Productivity. The current version in the AppStore already has the "poor experience flow" in the Sign In/Up, so you can try it.
It weird that there is no issue similar like this for the Facebook iOS SDK. But they have exactly the same flow.

@mdmathias Unfortunately, no one has yet replied to my request for the technical support from Apple.

Thanks @startechm @ostatnicky and @isaacadariku. I will share these names with my colleagues and let you know what I hear.

@mdmathias  Oops, I missed the discussion.
It's Spellar AI
We passed the review a few times, compelling Apple to approve the submission due to critical bug fixes.

@mdmathias Our app (Done!) was approved accidentaly without any further interaction. If this was due to you, thank you very much!

This issue is critical. All mac catalyst apps are rejected now.
I know external browser is the behaviour of API.
Are there any workaround or solution?

@tigernghk I'm doing what I can by raising this issue with colleagues internally who are able to reach out to Apple. Otherwise, @ostatnicky posted here that engaging with Apple Developer Technical Support helped to resolve the rejection.
I will be sure to post here if I receive any update.

My mac catalyst app update was just rejected for this same reason, but I am given the opportunity to have the released approved this time due to bug fixes.  I definitely agree this is critical if Apple is really going to reject any future updates that use Google Sign In calling the default browser.  I do plan to write to the review team to explain that the issue needs to be taken up directly with Apple and Google, and not every developer that uses Google Sign In for a macCatalyst app....we'll see how that goes.

I do plan to write to the review team to explain that the issue needs to be taken up directly with Apple and Google, and not every developer that uses Google Sign In for a macCatalyst app....we'll see how that goes.

@ghb101, we've gone through 5-7 releases with the same rejection, and each time, we've tried to explain the issue to Apple. However, this has been utterly futile, as they've insisted that we "fix" the problem.
Passing the review every time by citing "critical bug fixes" is incredibly frustrating and time-consuming.

We are doing our best to consult with counterparts at Apple. In the meantime, I think this message from ostatnicky indicates that working with Apple Developer Technical Support (ADTS) will result in an approved submission.
Perhaps if more apps reach out to ADTS, then app store review process will be updated more quickly?

Google Sign In SDK depends on AppAuth SDK. I have changed code in AppAuth SDK to use WKWebView as embedded browser instead of ASWebAuthenticationSession for quick and dirty way to solve the problem. Specifically, I only have changed the file "OIDExternalUserAgentCatalyst.m". My app is approved eventually.

@tigernghk Could you share the "dirty" code with us?

FYI. Here is my modified code in OIDExternalUserAgentCatalyst.m:
https://github.com/tigernghk/AppAuth_WKWebView/blob/main/OIDExternalUserAgentCatalyst.m

@tigernghk You shouldn't be using a WKWebView for authorization flows. Webviews are disallowed user agents for most identity providers. This is because the developers implementing the webview can inject JavaScript, read cookies, etc. In short, they are not secure for authorization.

Thanks. I hope you can find better solution.

@tigernghk Unfortunately, this is not a problem we can solve in GSI or AppAuth. I am of the position that these rejections are an error in app submission review process. The "fix" will come from there.
In any case, using a web view for authorization will likely lead to a 403 - disallowed_useragent error when using Sign in with Google. I highly recommend that you change using a web view for sign ins.

My Mac game Open Space was also rejected on Testflight. I don't understand why Apple cares. With password managers I prefer external safari anyway. I don't think it makes it any better to handle links in an embedded browser.

I sent this to<EMAIL_ADDRESS>- If anyone agrees please email them
https://github.com/google/GoogleSignIn-iOS/issues/388
Apple rejecting mac apps that use Google Sign in external browser sign in. I think its anti trust valid cause they're coming out with their own password manager

omg, this issue is still open?..

This should not be occurring any longer. Please let me know if you have an app that was recently rejected for this reason. I will reopen the issue.

do we need to update pod files? or resubmit? my mac build was last rejected on Apr 12, 2024 I just sent another version in for approval.


Dark mode on pages not working correctly (Casper theme)

Issue Summary
Pages looks great on light mode but not on dark mode (see GIF below)

To Reproduce
You can check live website at https://betzerra.com/about/
Technical details:

Ghost Version: 3.0.2
Node Version:
Browser/OS: Safari & Chrome / macOS
Database: mySQL

I see this issue too in Firefox Developer Edition 71.0b6

I just noticed this too, the white background and the fact that the title gets lost makes the dark mode when using pages somewhat unusable

This is fixed by this commit: https://github.com/TryGhost/Casper/commit/a66beb4e1044210f452cc0bda685631b500435f4
It's just an additional definition in screen.css at roughly line 2849 so the block becomes:
    .site-main,
    .post-template .site-main,
    .page-template .site-main {
        background: var(--darkmode);
    }

Thank you @docherty !


Allow CDN path or prefix as direct config option

Minify version 2.1.3

Feature request: CDN path as a config option

http://code.google.com/p/minify/wiki/UriRewriting#Manual_Rewriting shows how to
add a prefix with relative paths (../../images/123.jpg => /cdn/images/123.jpg),
but the same style won't work if you want to prepend a domain instead of a path
(you end up with paths like cdn.example.com/../../images/123.jpg).

It would be wonderful to have a 'cdnPath' option to add a prefix to all CSS
images when they get minified, as it stands now you have to something like:

function minify_post_process_css_cdn($content, $type) {
  if ($type === Minify::TYPE_CSS) {
    return str_replace('url(/', 'url(' . $CDN_PATH, $content);
  }
  return $content;
}
$min_serveOptions['postprocessor'] = 'minify_post_process_css_cdn';

This will properly rewrite "../../images/123.jpg" to
"http://cdn.example.com/images/123.jpg" if $CDN_PATH is set appropriately.

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 14 Dec 2010 at 3:01

A simple solution would be for the user to prepend something like
"http://cdn.example.org/a/a/" and then Minify to automatically remove parent
traversals where possible.
../../images/123.jpg => (prepend)
http://cdn.example.org/a/a/../../images/123.jpg => (remove traversal)
http://cdn.example.org/a/../images/123.jpg => (remove traversal)
http://cdn.example.org/images/123.jpg

Thoughts?

Original comment by<EMAIL_ADDRESS>on 14 Dec 2010 at 4:27

Changed state: Accepted
Added labels: Type-Enhancement, Release-2.1.3
Removed labels: Type-Defect

In config.php you would have code like:

if (/* is production environment */) {
  $min_serveOptions['rewriteCssUris'] = false;
  $min_serveOptions['minifierOptions']['text/css']['prependRelativePath'] = 'http://cdn.example.org/a/a/';
}

The result is that, on development Minify rewrites as normal, but on production
the rewrited URLs point to the CDN (with "/a/a/../../" replaced with "/").

Original comment by<EMAIL_ADDRESS>on 16 Dec 2010 at 7:09

That would work, but it doesn't address the use case I had in mind.

For development, user creates CSS and assets with relative paths.

For deployment, minify is used to combine, compress and rewrite the CSS to
appropriate paths. This case is exactly how it is used and works today.

The enhancement would be that I don't want to point at my production server
(CSS-relative path), I want to point to an absolute path for assets.

Original comment by<EMAIL_ADDRESS>on 15 Dec 2010 at 7:11

R436 adds "../" removal to prependRlativePath operations. Just replace
UriRewriter.php from the new version. With my suggestion in comment 3
(branching config code per environment) you should be able to get the desired
behavior.

Original comment by<EMAIL_ADDRESS>on 19 Dec 2010 at 9:07

Changed state: Fixed


Hide the add to project dropdown if no project

I stumbled on this for a second, probably indicating that it could be clearer. Usually if there isn't a choice we make a dropdown into a heading.
I have no projects, should I add to project or create a new? Both? Having no projects (but not realizing it atm since I create & delete them all the time) I found this odd:

Hmm, no options, so why is it a drop down?

Kind of clear, but is still a bit abnormal.

I'm open to disagreement here, just pointing it out as we usually don't show a drop-down menu if there isn't actually an item to select.
@spadgett @jeff-phillips-18 @rhamilto

What should happen if a user has no projects and doesn't have rights to create a project?
Should we display an appropriate error message under "Add to Project".  In this scenario, the user wouldn't be able to order a service, so the [Next] wizard button should never become enabled.
@benjaminapetersen, @spadgett

Yeah I would think we should preempt that as well as we can.  They should probably get that message on login, rather than waiting until they click a button that won't let them do anything anyway.  Thoughts?
@ncameronbritt

Hi, when user can only create a project, I have:

Feels like the 'create project' fields should be indented, correct?

Honestly I would prefer if this was a different step in the wizard, would clear things up a good bit.  Deal with project select/create in a step, then deal with the app being added in the next.  Might just be my opinion, however.

I'm not sure this is better. It looks like the whole form is for creating a project :/

@openshift/team-ux-review thoughts?

One option is to disable the dropdown when the only option is to create. Possibly with a small note stating why the dropdown is disabled. That way we don't have to re-organize the page for the different situations, but we also don't waste anyone's time

One option is to disable the dropdown when the only option is to create a new project, possibly with a short note describing why. That way, we don't need to reorganize the page for a single situation and we also don't waste anyone's time

I don't think that having it a separate step in the wizard makes sense.
For the case when the user has ability to Create a Project, but there are no existing projects, how about something like this? (hide the drop down and show an instruction)

For the case where there ARE existing projects & they can create a new ... you'd still have this

But then the question is ... when there is only one project allowed, and it has already been created, do we display the dropdown, with that project selected, but disable the dropdown?

@dtaylor113 @benjaminapetersen @spadgett @ncameronbritt @beanh66 @cshinn @jeff-phillips-18 how bout a 👍 or 👎  on each of the 3 items above?

Hi @serenamarie125, I think the issue with Case 1 is that the Create Project title seems to also be including the OTHER FIELDS.

Here's a screenshot with a little left-hand padding to offset the 'create project' form:
Can only Create Project:

Choosing to Create Project:

Selecting a Project:

@benjaminapetersen I just realized we have the same problem with the version dropdown. I have selected a few catalog items that only have a single version, but this option still appears in a dropdown where there are no other versions available.
I agree with @cshinn! My preference for these cases would be to show the default selection (Create Project) and disable the dropdown with a hover message explaining why.

I agree with @cshinn as well, but maybe "Create Project (no existing projects)" or some such thing.

We probably also should deal with "There are no projects and you can't create projects".

Seems like we're liking @cshinn suggestion of disabling the dropdown when there is only 'Create Project'.  I'll work on that.

Can only create project

Only one project and cannot create

No existing projects and cannot create projects

Not sure this is the best treatment/message here, but it's a start.  Kinda want to add a 'Please contact your cluster administrator' to the message.

@dtaylor113 I'm good with the first 2 scenarios.
As for the No existing projects & cannot create ... can that actually happen?  Hoping NOT :D

@serenamarie125 It can happen :/ Usually in organizations where they want you to request projects through some process outside of OpenShift. You can disable create project in the UI.

That seems like a situation where it would be good to let the admin put a url or email address or something in the message

That seems like a situation where it would be good to let the admin put a url or email address or something in the message

Hi, @spadgett, @cshinn suggestion seems like a good idea, but how often does this occur? -thanks

That seems like a situation where it would be good to let the admin put a url or email address or something in the message

We support custom messages and add it to the projects sidebar. We could show it here as well.

Created #494

I think its a little confusing yet for "Create Project" to appear in the disabled input.  Maybe "No existing projects" should be the text, though then we need a "create project" heading, which brings back the issue @spadgett mentioned.   I have no strong feelings on the tooltip.

Also odd that the disabled input is a required field.

What about making Add to Project and Configuration both headers:

And I think we should block a user from going past the info screen if there are no projects and they don't have permission to create one. We couldn't do that before because the info and configuration were the same step. We should be able to now.

:+1:  sounds good.  I believe currently the [Next] button is disabled until they have a valid project specified.

Note that we use this widget in a bunch of different dialogs now. We should check how it looks in them all.

@dtaylor113 can you wait on this for a bit?  I'd like to have consensus from the Design side before you continue to change things

@dtaylor113 @spadgett @benjaminapetersen the consensus from designers is the following:

Essentially we are saying (1) projects section should always be at the top of the form (moving version down below project stuff) (2) projects section should always have a divider line under it (3) when “Create Project” is the only option, the dropdown should disappear and the subheader “Create Project” should replace it
Note: this styling of the project section would also apply to the Deploy Image and Import YAML wizards as well.

@beanh66, looks good.  Where does the "contact your cluster administrator" link goto?  Are we saying that should be a custom/extension message?

@dtaylor113 It should be the request project message that the admin configures. See
https://github.com/openshift/origin-web-catalog/blob/master/src/components/projects-summary/projects-summary.html#L23
https://github.com/openshift/origin-web-catalog/blob/master/src/components/projects-summary/projects-summary.controller.ts#L86-L109

@dtaylor113 Please update all of these dialogs when you make this change.
origin-web-console/app/views/directives/deploy-image.html
2:  <select-project ng-if="!project" selected-project="input.selectedProject" name-taken="projectNameTaken"></select-project>

origin-web-console/app/views/directives/process-template.html
4:      <select-project ng-if="!$ctrl.project" on-project-selected="$ctrl.onProjectSelected" available-projects="$ctrl.availableProjects" selected-project="$ctrl.selectedProject" name-taken="$ctrl.projectNameTaken"></select-project>

origin-web-console/app/views/directives/from-file.html
1:<select-project ng-if="!project" selected-project="input.selectedProject" name-taken="projectNameTaken"></select-project>

origin-web-catalog/src/components/order-service/order-service-configure.html
4:      <select-project selected-project="$ctrl.selectedProject" name-taken="$ctrl.nameTaken"></select-project>

origin-web-catalog/src/components/create-from-builder/create-from-builder-configure.html
18:      <select-project selected-project="$ctrl.selectedProject" name-taken="$ctrl.projectNameTaken"></select-project>

Thanks @spadgett, so it looks like $ctrl.newProjectMessage' is set after a failed ProjectsService.canCreate().  Does the project request message come from the returned data.details.causes', and/or is it set via some Constants like the other custom messages/HTML?

It's in data.details.causes

Thanks @spadgett :+1:

Hi, just want to verify that this is how we want to display when user has existing projects and can create projects, and chooses to create a project:

Also, is there a way to taper the start and end of the bottom border like the mockups suggest?  If so, pls send the css -thanks

@dtaylor113 I believe it already does that.

Hi @cshinn, @beanh66, @spadgett,
Latest screenshots located here: #493


Docker workshop

Workshop 類型的，移到 ganhuaking/ganhuaking.github.io#20 待處理
Project 參考 https://github.com/orgs/ganhuaking/projects/1/views/1


Required for PlayStore deployment

For the Google Play Store deployment, it's required to have these entries in the AndroidManifest.xml

See: https://github.com/android/uamp/issues/396


Game cover images not visible

Describe the bug
The game cover images are not longer visible, although all images are present in the media tab and are visible with other themes.
To Reproduce
Steps to reproduce the behavior:

Go to Details View

Screenshots

Problem was caused by the extension BackbroundChanger. Disabling the extension fixed the issue. I tried replicating the issue but could not, the extension works as intended (performance will be fixed in the near future) perhaps a wrong file or wrong config. Will close this issue.

Ive finally been able to reproduce this problem while trying to fix performance issue with plugin BackgroundChanger, the problem was in the theme and can be fixed by enabling covers function in options of BackgroundChanger or disable said plugin.
This issue is fixed in version 3.75, the update will be released later today or tomorrow!


[Structured list] React Wrapper: enhancement and new variant

User Story

As a [user role below]:
Carbon for ibm.com developer

I need to:
create/change the Structured list

so that I can:
provide the ibm.com adopter developers components they can use to build ibm.com web pages

Additional information

Story within Storybook with corresponding knobs
Utilize Carbon
Create with Shadow DOM and Custom Elements standards
The React version is created as a wrapper to the Web Components version
See the Epic (#8385) for the Design and Functional specs information
Visual QA testing issue (#8386)
Prod QA testing issue (#8387)

1. Structured list enhancement

Ability to include paragraph breaks
Ability to add checkmark icon in row items
Ability to add tooltip

2. Structured list with row headers (vertical)
New variant of current structured list

Ability for Rows to have heading style (no column headers)

3. 8 columns
Not sure if this is an enhancement or new, I think structured list can already span 8 columns?

Additional Information

Design specs
Carbon web components structures list
Carbon React structured list

Acceptance criteria

[ ] Include README for the web component and corresponding styles
[ ] Create Web Components styles in styles package
[ ] Create React wrapper
[ ] Do not create knobs in Storybook that include JSON objects
[ ] Break out Storybook stories into multiple variation stories, if applicable
[ ] Create codesandbox example under /packages/web-components/examples/codesandbox and include in README
[ ] Minimum 80% unit test coverage
[ ] add the trigger to create the React Wrapper
[ ] create the React storybook stories and related documentation
[ ] Update the Carbon for ibm.com website component data file to be sure the web site Component Status and Storybook links are correct
[ ] Use the Visual QA checklist to verify design quality
[ ] Update the e2e test scripts, if needed.
[ ] Confirm e2e test results have no major errors.
[ ] A comment is posted in the Prod QA issue (#8387) when development is finished.
[ ] The dev complete label is added to the Prod QA issue (#8387).

@kennylam @jeffchew I defer to you both on when you'd like to schedule the work for implementing the React Wrapper version of this component. In the meantime, our Innovation team is focused on contributing the work needed on the Web Component package side (https://github.com/carbon-design-system/carbon-for-ibm-dotcom/issues/8388) for AEM consumption.

Hey team! Please add your planning poker estimate with ZenHub @annawen1 @ariellalgilmore @emyarod @IgnacioBecerra @kennylam


rosbag2: 0.1.8-1 in 'dashing/distribution.yaml' [bloom]

Increasing version of package(s) in repository rosbag2 to 0.1.8-1:

upstream repository: https://github.com/ros2/rosbag2.git
release repository: https://github.com/ros2-gbp/rosbag2-release.git
distro file: dashing/distribution.yaml
bloom version: 0.8.0
previous version for package: 0.1.7-1

ros2bag
* install resource marker file for package (#177 <https://github.com/ros2/rosbag2/issues/177>)

rosbag2

No changes

rosbag2_converter_default_plugins

No changes

rosbag2_storage

No changes

rosbag2_storage_default_plugins

No changes

rosbag2_test_common

No changes

rosbag2_tests

No changes

rosbag2_transport

No changes

shared_queues_vendor

No changes

sqlite3_vendor

No changes

@nuclearsandwich Please merge whenever it is convenient for the buildfarm.


SMTP Email configuration generate error

Attempted Debugging

[X] I have read the debugging page

Searched GitHub Issues

[X] I have searched GitHub for the issue.

Describe the Scenario
I configured the SMTP email service like the docs but, when trying to send an email from the web gui, I always get the same error and no logs in laravel.log file (and yes I set the right permission to those folder as asked in docs)
Here is the error message

Here is my config in .env file :
MAIL_DRIVER=smtp
MAIL_HOST=ssl0.ovh.net
MAIL_PORT=465
<EMAIL_ADDRESS>MAIL_PASSWORD=mypassword
MAIL_ENCRYPTION=tls
MAIL_VERIFY_SSL=true (test with and without and with true and false, changged nothing)
I also got a nextcloud app on my network that use those same credentials and I could get email from it.

Exact BookStack Version
24.02.3
Log Content
No log generated..
Hosting Environment
Ubuntu server 22.04.4_amd64

Hi @Thomas-Lecuppre,

Does your MAIL_PASSWORD contain any special characters?
Can you share the output of running ls -al bootstrap/cache from your BookStack install directory?

Hi @ssddanbrown , thank you for your reply.
Of course my password contains specials characters (since it hasn't been mentioned in the docs that it shouldn't).
When I run your command it say there is no such file or directory.
I tried to set a password with no special character and it's seems to work well.
Maybe it can be mentioned in the wiki on this page :
https://www.bookstackapp.com/docs/admin/email-webhooks/
Thank you for helping.

@Thomas-Lecuppre Do the special characters used include a space and/or hash?

@ssddanbrown @, # and *

@Thomas-Lecuppre Okay, then it was likely the hash causing trouble.
This is something mentioned at the top of the env file: https://github.com/BookStackApp/BookStack/blob/493d8027cd99656cdbeb92c01da99b5baa485a09/.env.example#L6-L7
Since you've managed to find and address the issue here I'll close this off.


Enviroment variables

How to use enviroment variables?

https://linuxize.com/post/how-to-set-and-list-environment-variables-in-linux/
I'll just leave it at that ;)


coprocessor/dag/expr: add some un-pushed builtin UDFs(Length, BitLength)

What have you changed? (mandatory)
implemented ScalarFuncSig::Length for #3275
What are the type of the changes? (mandatory)
Improvement
How has this PR been tested? (mandatory)
unittest
Does this PR affect documentation (docs/docs-cn) update? (mandatory)
No
Does this PR affect tidb-ansible update? (mandatory)
No
Refer to a related PR or issue link (optional)
#3275
Benchmark result if necessary (optional)
Add a few positive/negative examples (optional)

LGTM

/run-integration-tests


API items bugfix

Closes #114.

Codecov Report

Merging #115 (a0bf796) into main (62184a2) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #115   +/-   ##
=======================================
  Coverage   83.66%   83.66%
=======================================
  Files          37       37
  Lines        1322     1322
=======================================
  Hits         1106     1106
  Misses        216      216

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62184a2...a0bf796. Read the comment docs.


Move should_use_cudnn and should_use_cudnn_tensor_core to chainer.cuda

Moving these functions to chainer.cuda to keep chainer.__init__.py uncluttered.

Jenkins test this please

LGTM!


Feature: toggle to filter privacy-sensitive info from logs

Hey just a sidenote:
2022-04-21 01:28:28.731 +02:00 [INF] [HASS_API] System connected with [domain]
The logs show external domains. Is it possible to hide that with a switch?
People will most likely be posting lots of logs and this could be a security issue if forgotten to remove by the user.

Very good point, will do, thanks!
hassagent-48

I'm closing most feature requests from github's issue list. They're not ignored, just moved to the HASS.Agent Issue Tracker (you can find your link in a comment above).
That tracker allows me to organise better through what's pending, being done, being released, etc. - instead of one long list, which is beginning to stress me out :p
I will however post a reply here when I've implemented your request, and feel free to still post here when you have any questions/additions.


relatedImages contains incorrect image path

Background
While testing catalog mirroring found the common services catalog contains incorrect reference to operand image in relatedImages.  Xinchun Liu indicated the image belongs to install operator and to open issue here :
- image: icr.io/cpopen/multi-instance-conversion@sha256:a9ed199056ae20a28c4949e1fe4b5e851bba1f50446897b069382cb7e5156619
  name: CS_MULTI_INSTANCE_CONVERSION_IMAGE

should be :
- image: icr.io/cpopen/cpfs/multi-instance-conversion@sha256:a9ed199056ae20a28c4949e1fe4b5e851bba1f50446897b069382cb7e5156619
  name: CS_MULTI_INSTANCE_CONVERSION_IMAGE

Thank you @morstad for finding this out!
We will fix it ASAP. Open a GHE issue to track the progress


swat port for t/* stuff

Hi @Manwar!
swat is a test automation framework.
I found it quite interesting to port some of t/* stuff to swat DSL. On my laptop output looks like this:
vagrant@Debian-jessie-amd64-netboot:~/projects/Map-Tube-London$ carton exec swat swat-test/
/home/vagrant/.swat/.cache/8803/prove/route/00.GET.t .........
ok 1 - response is already set
# response saved to /home/vagrant/.swat/.cache/8803/prove/S3gLorL1z0
ok 2 - GET /route returns Route 1 Baker Street => Farringdon : Baker Street (Circle, Hammersmith & City, Metropolitan), Great Portland Street (Circle, Hammersmith & City, Metropolitan), Euston Square (Circle, Hammersmith & City, Metropolitan), King's Cross St. Pancras (Circle, Hammersmith & City, Metropolitan), Farringdon (Circle, Hammersmith & City, Metropolitan)
ok 3 - GET /route returns Route 2 Bank => Monument : Bank (Tunnel), Monument (Tunnel)
ok 4 - GET /route returns Route 3 Euston => King's Cross St. Pancras : Euston (Northern, Street, Victoria), King's Cross St. Pancras (Northern, Street, Victoria)
1..4
ok
/home/vagrant/.swat/.cache/8803/prove/test-case-01/00.GET.t ..
ok 1 - response is already set
# response saved to /home/vagrant/.swat/.cache/8803/prove/2izmIDf9GG
ok 2 - GET /test-case-01 returns | Route 1 found: Wembley Central (Overground, Bakerloo), Stonebridge Park (Overground, Bakerloo), Harlesden (Overground, Bakerloo), Willesdon Junction (Overground, Bakerloo), Shepherd's Bush (Central, Overground), Holland Park (Central), Notting Hill Gate (Central, Circle, District), Queensway (Central), Lancaster Gate (Central), Marble Arch (Central), Bond Street (Central, Jubilee)
ok 3 - GET /test-case-01 returns | Route 1 expected: Wembley Central (Overground, Bakerloo), Stonebridge Park (Overground, Bakerloo), Harlesden (Overground, Bakerloo), Willesdon Junction (Overground, Bakerloo), Shepherd's Bush (Central, Overground), Holland Park (Central), Notting Hill Gate (Central, Circle, District), Queensway (Central), Lancaster Gate (Central), Marble Arch (Central), Bond Street (Central, Jubilee)
ok 4 - GET /test-case-01 returns | Route 2 found: Bond Street (Central, Jubilee), Oxford Circus (Central, Victoria, Bakerloo), Warren Street (Northern, Victoria), Euston (Northern, Overground, Victoria, Street)
ok 5 - GET /test-case-01 returns | Route 2 expected: Bond Street (Central, Jubilee), Oxford Circus (Central, Victoria, Bakerloo), Warren Street (Northern, Victoria), Euston (Northern, Overground, Victoria, Street)
ok 6 - GET /test-case-01 returns | Route 3 found: White City (Central), Shepherd's Bush (Central, Overground), Kensington (Olympia) (District, Overground), Earl's Court (District, Piccadilly), Gloucester Road (Circle, District, Piccadilly), South Kensington (Circle, District, Piccadilly), Sloane Square (Circle, District), Victoria (Circle, District, Victoria)
ok 7 - GET /test-case-01 returns | Route 3 expected: White City (Central), Shepherd's Bush (Central, Overground), Kensington (Olympia) (District, Overground), Earl's Court (District, Piccadilly), Gloucester Road (Circle, District, Piccadilly), South Kensington (Circle, District, Piccadilly), Sloane Square (Circle, District), Victoria (Circle, District, Victoria)
ok 8 - GET /test-case-01 returns | Route 4 found: Temple (Circle, District), Embankment (Circle, District, Northern, Bakerloo), Waterloo (Northern, Bakerloo, Jubilee, Waterloo & City), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Moorgate (Circle, Northern, Hammersmith & City, Metropolitan), Barbican (Circle, Hammersmith & City, Metropolitan), Farringdon (Circle, Hammersmith & City, Metropolitan)
ok 9 - GET /test-case-01 returns | Route 4 expected: Temple (Circle, District), Embankment (Circle, District, Northern, Bakerloo), Waterloo (Northern, Bakerloo, Jubilee, Waterloo & City), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Moorgate (Circle, Northern, Hammersmith & City, Metropolitan), Barbican (Circle, Hammersmith & City, Metropolitan), Farringdon (Circle, Hammersmith & City, Metropolitan)
ok 10 - GET /test-case-01 returns | Route 5 found: Turnham Green (District, Piccadilly), Hammersmith (Circle, District, Hammersmith & City, Piccadilly), Barons Court (District, Piccadilly), Earl's Court (District, Piccadilly), Gloucester Road (Circle, District, Piccadilly), South Kensington (Circle, District, Piccadilly), Knightsbridge (Piccadilly), Hyde Park Corner (Piccadilly), Green Park (Victoria, Jubilee, Piccadilly), Westminster (Circle, District, Jubilee), Waterloo (Northern, Bakerloo, Jubilee, Waterloo & City), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Shadwell (DLR, Overground), Whitechapel (District, Hammersmith & City, Overground)
ok 11 - GET /test-case-01 returns | Route 5 expected: Turnham Green (District, Piccadilly), Hammersmith (Circle, District, Hammersmith & City, Piccadilly), Barons Court (District, Piccadilly), Earl's Court (District, Piccadilly), Gloucester Road (Circle, District, Piccadilly), South Kensington (Circle, District, Piccadilly), Knightsbridge (Piccadilly), Hyde Park Corner (Piccadilly), Green Park (Victoria, Jubilee, Piccadilly), Westminster (Circle, District, Jubilee), Waterloo (Northern, Bakerloo, Jubilee, Waterloo & City), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Shadwell (DLR, Overground), Whitechapel (District, Hammersmith & City, Overground)
ok 12 - GET /test-case-01 returns | Route 6 found: Goldhawk Road (Circle, Hammersmith & City), Shepherd's Bush Market (Circle, Hammersmith & City), Wood Lane (Circle, Hammersmith & City), White City (Central), Shepherd's Bush (Central, Overground), Willesdon Junction (Overground, Bakerloo), Harlesden (Overground, Bakerloo), Stonebridge Park (Overground, Bakerloo), Wembley Central (Overground, Bakerloo)
ok 13 - GET /test-case-01 returns | Route 6 expected: Goldhawk Road (Circle, Hammersmith & City), Shepherd's Bush Market (Circle, Hammersmith & City), Wood Lane (Circle, Hammersmith & City), White City (Central), Shepherd's Bush (Central, Overground), Willesdon Junction (Overground, Bakerloo), Harlesden (Overground, Bakerloo), Stonebridge Park (Overground, Bakerloo), Wembley Central (Overground, Bakerloo)
ok 14 - GET /test-case-01 returns | Route 7 found: Wembley Central (Overground, Bakerloo), Stonebridge Park (Overground, Bakerloo), Harlesden (Overground, Bakerloo), Willesdon Junction (Overground, Bakerloo), Shepherd's Bush (Central, Overground), Holland Park (Central), Notting Hill Gate (Central, Circle, District), Bayswater (Circle, District), Paddington (District, Circle, Hammersmith & City, Bakerloo), Edgware Road (Circle, District, Hammersmith & City, Bakerloo), Marylebone (Bakerloo)
ok 15 - GET /test-case-01 returns | Route 7 expected: Wembley Central (Overground, Bakerloo), Stonebridge Park (Overground, Bakerloo), Harlesden (Overground, Bakerloo), Willesdon Junction (Overground, Bakerloo), Shepherd's Bush (Central, Overground), Holland Park (Central), Notting Hill Gate (Central, Circle, District), Bayswater (Circle, District), Paddington (District, Circle, Hammersmith & City, Bakerloo), Edgware Road (Circle, District, Hammersmith & City, Bakerloo), Marylebone (Bakerloo)
ok 16 - GET /test-case-01 returns | Route 8 found: Baker Street (Circle, Hammersmith & City, Bakerloo, Metropolitan, Jubilee), Finchley Road (Metropolitan, Jubilee), Wembley Park (Metropolitan, Jubilee), Neasden (Jubilee)
ok 17 - GET /test-case-01 returns | Route 8 expected: Baker Street (Circle, Hammersmith & City, Bakerloo, Metropolitan, Jubilee), Finchley Road (Metropolitan, Jubilee), Wembley Park (Metropolitan, Jubilee), Neasden (Jubilee)
ok 18 - GET /test-case-01 returns | Route 9 found: Baker Street (Circle, Hammersmith & City, Bakerloo, Metropolitan, Jubilee), Finchley Road (Metropolitan, Jubilee), Wembley Park (Metropolitan, Jubilee), Preston Road (Metropolitan)
ok 19 - GET /test-case-01 returns | Route 9 expected: Baker Street (Circle, Hammersmith & City, Bakerloo, Metropolitan, Jubilee), Finchley Road (Metropolitan, Jubilee), Wembley Park (Metropolitan, Jubilee), Preston Road (Metropolitan)
ok 20 - GET /test-case-01 returns | Route 10 found: Oval (Northern), Kennington (Northern), Waterloo (Northern, Bakerloo, Jubilee, Waterloo & City), Westminster (Circle, District, Jubilee), Green Park (Victoria, Jubilee, Piccadilly), Oxford Circus (Central, Victoria, Bakerloo), Warren Street (Northern, Victoria), Euston (Northern, Overground, Victoria, Street)
ok 21 - GET /test-case-01 returns | Route 10 expected: Oval (Northern), Kennington (Northern), Waterloo (Northern, Bakerloo, Jubilee, Waterloo & City), Westminster (Circle, District, Jubilee), Green Park (Victoria, Jubilee, Piccadilly), Oxford Circus (Central, Victoria, Bakerloo), Warren Street (Northern, Victoria), Euston (Northern, Overground, Victoria, Street)
ok 22 - GET /test-case-01 returns | Route 11 found: South Ealing (Piccadilly), Acton Town (District, Piccadilly), Ealing Common (District, Piccadilly), North Ealing (Piccadilly), Park Royal (Piccadilly), Alperton (Piccadilly)
ok 23 - GET /test-case-01 returns | Route 11 expected: South Ealing (Piccadilly), Acton Town (District, Piccadilly), Ealing Common (District, Piccadilly), North Ealing (Piccadilly), Park Royal (Piccadilly), Alperton (Piccadilly)
ok 24 - GET /test-case-01 returns | Route 12 found: Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Waterloo (Northern, Bakerloo, Jubilee, Waterloo & City), Westminster (Circle, District, Jubilee)
ok 25 - GET /test-case-01 returns | Route 12 expected: Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Waterloo (Northern, Bakerloo, Jubilee, Waterloo & City), Westminster (Circle, District, Jubilee)
ok 26 - GET /test-case-01 returns | Route 13 found: Hoxton (Overground), Haggerston (Overground), Dalston Junction (Overground), Dalston Kingsland (Overground), Canonbury (Overground), Highbury & Islington (Overground, Victoria), Caledonian Road & Barnsbury (Overground), Camden Road (Overground), Kentish Town West (Overground), Gospel Oak (Overground)
ok 27 - GET /test-case-01 returns | Route 13 expected: Hoxton (Overground), Haggerston (Overground), Dalston Junction (Overground), Dalston Kingsland (Overground), Canonbury (Overground), Highbury & Islington (Overground, Victoria), Caledonian Road & Barnsbury (Overground), Camden Road (Overground), Kentish Town West (Overground), Gospel Oak (Overground)
ok 28 - GET /test-case-01 returns | Route 14 found: Baker Street (Circle, Hammersmith & City, Bakerloo, Metropolitan, Jubilee), Finchley Road (Metropolitan, Jubilee), Harrow-on-the-Hill (Metropolitan), North Harrow (Metropolitan)
ok 29 - GET /test-case-01 returns | Route 14 expected: Baker Street (Circle, Hammersmith & City, Bakerloo, Metropolitan, Jubilee), Finchley Road (Metropolitan, Jubilee), Harrow-on-the-Hill (Metropolitan), North Harrow (Metropolitan)
ok 30 - GET /test-case-01 returns | Route 15 found: Baker Street (Circle, Hammersmith & City, Bakerloo, Metropolitan, Jubilee), Finchley Road (Metropolitan, Jubilee), Harrow-on-the-Hill (Metropolitan), Moor Park (Metropolitan), Croxley (Metropolitan)
ok 31 - GET /test-case-01 returns | Route 15 expected: Baker Street (Circle, Hammersmith & City, Bakerloo, Metropolitan, Jubilee), Finchley Road (Metropolitan, Jubilee), Harrow-on-the-Hill (Metropolitan), Moor Park (Metropolitan), Croxley (Metropolitan)
ok 32 - GET /test-case-01 returns | Route 16 found: Sloane Square (Circle, District), Victoria (Circle, District, Victoria), St. James's Park (Circle, District), Westminster (Circle, District, Jubilee)
ok 33 - GET /test-case-01 returns | Route 16 expected: Sloane Square (Circle, District), Victoria (Circle, District, Victoria), St. James's Park (Circle, District), Westminster (Circle, District, Jubilee)
ok 34 - GET /test-case-01 returns | Route 17 found: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 35 - GET /test-case-01 returns | Route 17 expected: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 36 - GET /test-case-01 returns | Route 18 found: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 37 - GET /test-case-01 returns | Route 18 expected: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 38 - GET /test-case-01 returns | Route 19 found: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 39 - GET /test-case-01 returns | Route 19 expected: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 40 - GET /test-case-01 returns | Route 20 found: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 41 - GET /test-case-01 returns | Route 20 expected: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 42 - GET /test-case-01 returns | Route 21 found: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 43 - GET /test-case-01 returns | Route 21 expected: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 44 - GET /test-case-01 returns | Route 22 found: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 45 - GET /test-case-01 returns | Route 22 expected: Westferry (DLR), Limehouse (DLR), Shadwell (DLR, Overground), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel), Cannon Street (Circle, District)
ok 46 - GET /test-case-01 returns | Route 23 found: Tower Gateway (DLR, Street), Tower Hill (Circle, District, Street), Aldgate (Circle, Metropolitan)
ok 47 - GET /test-case-01 returns | Route 23 expected: Tower Gateway (DLR, Street), Tower Hill (Circle, District, Street), Aldgate (Circle, Metropolitan)
ok 48 - GET /test-case-01 returns | Route 24 found: Liverpool Street (Central, Circle, Hammersmith & City, Metropolitan), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel)
ok 49 - GET /test-case-01 returns | Route 24 expected: Liverpool Street (Central, Circle, Hammersmith & City, Metropolitan), Bank (Central, DLR, Northern, Waterloo & City, Tunnel), Monument (Circle, District, Tunnel)
1..49
ok
/home/vagrant/.swat/.cache/8803/prove/test-case-00/00.GET.t ..
ok 1 - response is already set
# response saved to /home/vagrant/.swat/.cache/8803/prove/DOCTZmp8kg
ok 2 - GET /test-case-00 returns | get_shortest_route() : Map::Tube::get_shortest_route(): ERROR: Either FROM/TO node is undefined
ok 3 - GET /test-case-00 returns | get_shortest_route('Bond Street') : Map::Tube::get_shortest_route(): ERROR: Either FROM/TO node is undefined
ok 4 - GET /test-case-00 returns | get_shortest_route('XYZ','Bond Street') : Map::Tube::get_node_by_name(): ERROR: Invalid station name
ok 5 - GET /test-case-00 returns | get_shortest_route('Bond Street', 'XYZ') : Map::Tube::get_node_by_name(): ERROR: Invalid station name
1..5
ok
All tests successful.
Files=3, Tests=58,  2 wallclock secs ( 0.03 usr  0.01 sys +  1.18 cusr  0.02 csys =  1.24 CPU)
Result: PASS

If that sounds interesting for you we could talk about further co-working.
The main idea is to instead of using is_deep from Test::More I would suggest a swat DSL to verify various text output.
At least because it is much easier to implement and less fragile , as long as one holds a dedicated text format emitted to  swat , which is not a big deal. Also it might be consider as black box testing as this approach less copulated with inner object structures.

and, yes to run a tests just say: carton exec swat swat-test/

Hi @melezhik,
Thanks for your contribution, much appreciated.
I must confess that I have never heard of 'swat' before. This is my first introduction to the testing tool. I like the philosophy behind it. However I find it difficult to follow. It could be just me. I am being honest with you.
I like your offer of working together. I am always ready for it. I have a suggestion for you, though. I like to keep test code and source code separate, to keep it light weight. If you have noticed the package Map::Tube::London is very thin wrapper around the package Map::Tube. So adding 'swat' is going to be quite heavy, in my humble opinion. Don't get me wrong, I am not against it.
Instead I suggest, we (both) can add 'swat' to the package Test::Map::Tube, which currently expose one method ok_map(). It validates the map data. I would suggest add new method ok_map_features() to this package that would use 'swat' to do all kind of unit testing.
This way other users of Map::Tube can also benefit without any overhead. What do you think?
Please share your view.
Many Thanks.
Best Regards,
Mohammad S Anwar

Hi Mohammad! Thanks alot for your feedback. I understand your concerns
about dependencies overheads. Indeed swat is standalone tool and not to be
used as Perl module, it acts as Perl prove util though and produces TAP
output as well. There are probably two variants we choose between.
First try to ship swat test together with map-tube*modules and run swat
tests after unit tests , but this requires modification of existing
makefile.pl or build.pl and hacking into distribution installation process.
Second way is to keep swat tests separately from source code and run them
separately from unit tests, this means only developers not end users could
run such a tests and they won't be run during cpan installation process.
Both ways have pros and cons, whatever way we choose I want to emphasize an
idea behind swat and map-tube.
I consider map tube system as black box one, the way I verify it works
correctly is a verifying of text output provided by map tube methods, like
shortest_route one - It is quite  convenient to map subway routes into
a chains of subway stations names represented by a regular line of text:
"foo, bar, baz, etc." and then "grep" inside this line , and this is where
swat fits the best, as it is text oriented test tool ( of course swat
could do more ... )
So the thing I guess we really need to focus in is real life use cases
could be tested in terms of swat, and this is where you definitely knows
much more than me.
PS btw , I am just curios - what is practical meaning of all theses
map-tube* modules ? Who use it ?

Hi Mohammad ! Any news here? Just want to hear what do you think. Thanks,

Hi Alexey,
Sorry for late response, got distracted in other projects. I am not sure if I understood everything you said. I would like you to take a lead role in this new project of incorporating swat with 'Map::Tube'. The only thing I am concerned that it should be optional for end user. I am not sure how this can be achieved but I will leave that to your expertise. I am happy to answer any questions you have in the process. In the meantime, I will try to get myself familiarize with 'swat' tool.
Do I make any sense?
Many Thanks.
Best Regards,
Mohammad S Anwar

Hi Mohammad ! Thanks for your answer. After some mediation I decided to postpone further activity on this , I want to create a brand new test framework - outhentic  which is inspired by swat but pure generic ( while swat is web testing one ) and fits much better for map-tube-* testing, when I have done I will let you know. Thanks

Hi Alexey,
That's great news !!!
Is there anything I can be of any help, please let me know.
Best Regards,
Mohammad S Anwar

Hi!

Is there anything I can be of any help, please let me know.

Once I asked, "I am just curios - what is practical meaning of all theses map-tube* modules ? Who use it ?"
Thanks (

Hi Alexey,
Nearly 5 years, 2010-10-25, to be precise, I uploaded my first draft of Map::Tube to CPAN. It started as a fun project and to put something in public domain. It has been through many fold of design changes since then, thanks to many people.
It was meant to deal with London tube map, which is the life line of many of us, here in London. I had no previous experience of AI, this was a big task for me. Currently there are four of us, who is working on this project, namely Michal Špaček (SKIM), Gisbert W. Selke (GWS) and Slaven Rezić (SREZIC).
There is no real purpose of this project at the moment. However I am planning to create REST API to expose the functionalities and host it on my website www.manwar.org. And even create a page to let the online user play with it. All for fun.
Looking at the GitHub/MetaCPAN, it appears few do like the project, which is encouraging.
Best Regards,
Mohammad S Anwar

Hi Mohammad! I have just released https://github.com/melezhik/outthentic to CPAN. I guess it might be useful in map tube tests. I think we could close this one. I am going to rewrite my initial merge request using new framework and then I will MR to you again.

Hi Alexey,
I will get back to you once I look through the details.
All the best.
Best Regards,
Mohammad S Anwar

Sure , Mohammad.

Hi Mohammad!
I have created a simple test suite for map-tube family modules -  https://github.com/melezhik/map-tube-testers .  You may play with it. It based on outthentic - a new test framework with the DSL very like to swat one. Eventually if it sounds interesting a new tests / maps may popup. Please take a look and let me know what do you think. Thanks.

Hi Alexey,
Thanks for the update.
I find no documentation whatsoever. It would really encourage, if you had explained the design of your tool. Also if you could throw some lights on how it is better than traditional approach. Although you have plenty of examples, I find it hard to follow.
Best Regards,
Mohammad S Anwar

Hi Mohammad. Outthentic documentaion is here - https://github.com/melezhik/outthentic ,  the link I sent you ( https://github.com/melezhik/map-tube-testers ) is an example how o

is an example how outtentic might be used in map tube project. To understand the difference against a common test frame works , it is better read outthentic documentaion first.
Regards, Akeksei .

Hi Alexey,
Thanks for the link to the documentation.
I will get back to you once I have absorbed the details.
Best Regards,
Mohammad S Anwar

Hi Alexey,
I have a proposal for you.
Can you please upgrade it to cover all the maps available on CPAN?
Once done then release the under than namespace Map::Tube::Tester to CPAN, so that other can benefit from your hard work and also submit PR if they want.
Many Thanks.
Best Regards,
Mohammad S Anwar

Hi Mohammad!

Once done then release it under than namespace Map::Tube::Tester to CPAN, so that other can benefit from your hard work and also submit PR if they want.

How do you see a use case for such Map::Tube::Tester cpan module? Who is going to use it?
Could you please provides some cases for it? Before go into realization I wish we could understand some practical meaning of it. There are some possible variants:

trigger travis jobs which hit map-tube-testers tests against a brand new Map::Tube::* module code

let other Map::Tube::* developers to git clone  map-tube-testers and then somehow run tests against their current Map::Tube::* code

making a real Map::Tube::Tester module which encapsulate all the testing logic, but how one could use it? Another drawback behind having tests packed into cpan modules is PAUSE index gets updated quite slow, in case we need to change something in test logic quickly we'd better look towards keeping tests on github and running them over git clone && carton exec ( previous two points ). Tough packaging tests into cpan module has profits as well. I just can't see how we gonna use it, still, may be you have better ideas ...

something else ?

Hi Michal,
Apology for late response.
I am going to be the first user of Map::Tube::Testers and hopefull others, who have map code would also join the group as well.
Ideally what I am looking for is, the centralised point of testing all maps. For example, everytime I make changes to core module i.e. Map::Tube, it would be great if I can test the change against all the maps dependent on Map::Tube before making the changes public.
In the past, there have been instances where I made changes and it broke other users map but not mine.
To kick start the project, I would suggest collect all existing test cases from every maps available and put in format that your tool expects. Then the user can run all or just a particular map depending on his requirement.
This would also make the map lighter as well since all map testing code is in now centralised.
Have I made myself clear?
Please give me a shout if you have any queries. Please bear with me if I don't reply promptly.
Many Thanks.
Best Regards,
Mohammad S Anwar

Hi Mohammad. Let me think about. As far as I know there are a plenty of existed maps, not sure if I am able to cover all the stuff shortly ...

further discussion moved there - https://github.com/melezhik/map-tube-testers , has emailed you.


New Warning on Django 1.10 after commit c3e504a0bf77deb9a79bd841e9cacc62c8127901

On commit c3e504a0bf77deb9a79bd841e9cacc62c8127901 a new function install_middleware() is added to raven/contrib/django/models.py .
This function seems to be trying to install the Sentry middleware if they are not manually added to settings, but seems to be ignore the new middleware style introduced in Django 1.10.
As this function only checks settings.MIDDLEWARE_CLASSES the middleware is always added to that setting when using the new MIDDLEWARE setting, which causes a warning when running the check management command from Django.

Bumping this issue. Getting the same warning:
?: (1_10.W001) The MIDDLEWARE_CLASSES setting is deprecated in Django 1.10 and the MIDDLEWARE setting takes precedence. Since you've set MIDDLEWARE, the value of MIDDLEWARE_CLASSES is ignored.

For reference, please check out django-debug-toolbar perspective on automatic setup.
They removed automatic setup in their latest version (1.6).
http://django-debug-toolbar.readthedocs.io/en/1.6/changes.html#removed-features

Support for automatic setup has been removed as it was frequently problematic. Installation now requires explicit setup.

Previous version (1.5) of their documentation elaborated on this.
http://django-debug-toolbar.readthedocs.io/en/1.5/installation.html#automatic-setup

The automatic setup is known to interfere with the start-up sequence of some projects and to prevent them from loading or functioning properly.
The explicit setup described below is recommended for all but the most trivial projects. The automatic setup is kept for backwards-compatibility.

https://github.com/getsentry/raven-python/pull/883

This will be fixed in next release.


SST-2 performance in ROBERTA paper

The SST-2 BERT-Large performance listed in ROBERTA paper is much lower then the performance listed in BERT original paper.
93.7 vs 94.9
Any explanation?

94.9 is on test set from ensemble. 93.7 is on dev. (Table 6 https://arxiv.org/pdf/1810.04805.pdf)


Postgres backup verification error - The server failed to start

First time executing the backup was this morning and I cannot get the verification to work. We do not have a public schema in our database, hopefully it is not required as it was not for the backup. I have attempted the verification several times. Any help in steps that I can take to investigate the cause of this issue is appreciated.
Namespace = 30245e-dev.

sh-4.2$ ./backup.sh -s -v all

Verifying backup ...

Settings:
- Database: postgres=patroni-master-dev/mals
- Backup file: /backups/daily/2021-07-27/patroni-master-dev-mals_2021-07-27_11-12-00.sql.gz

Already clean ...

waiting for server to start.....................................................................................................................
The server failed to start within 0h:2m:0s.

The server started in 0h:2m:0s.

waiting for server to shut down.... done
server stopped
Cleaning up ...

[!!ERROR!!] - Backup verification failed: /backups/daily/2021-07-27/patroni-master-dev-mals_2021-07-27_11-12-00.sql.gz

Elapsed time: 0h:2m:1s - Status Code: 1

In the run above, the verification is failing because the local server instance (used for validation) did not start within the timeout period set for DATABASE_SERVER_TIMEOUT; in this case 2 minutes.  You can increase the time by setting DATABASE_SERVER_TIMEOUT to a higher value, for example;

There are other factors that can cause the server to start slowly such as CPU and Memory resource allocation.  The backup-container performs best when allowed to use "best-effort" resource allocation, which is the default specified in the example template; https://github.com/BCDevOps/backup-container/blob/master/openshift/templates/backup/backup-deploy.yaml#L428-L447

If your database does not use a "public" schema, you can specify the name of your database's schema via the TABLE_SCHEMA variable.  This will tell the backup-container which schema to backup and verify.
https://github.com/BCDevOps/backup-container/blob/master/openshift/templates/backup/backup-deploy.yaml#L293-L297

Thanks Wade. Those four parameter values are set to the defaults and the table_schema parameter is set to the appropriate schema. I added the timeout parameter with 300 seconds and the verification now times out in 5 minutes.
I have updated the Dockerfile to pull Postgres v13 as our app database is v13. Hopefully that isn't the cause of the issue.
FROM centos/postgresql-13-centos7

That could be the issue.  We haven't tested the backup-container with that version.  It could be that some startup parameters have changed.  What's the version of the database you're backing up?

Full version:
 PostgreSQL 13.3 (Debian 13.3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit

I will redo the backup build using v12 for to see if it is compatible.

It's typically preferable to have the backup-container use the same or newer version as the database you are backing up.  Would you be up for troubleshooting the startup issues and contributing the upgrade back to the project?  I can provide you with some guidance on how to go about troubleshooting.

Certainly. Keep in mind that this is my first and only experience with OpenShift and I am still learning how everything works/interacts.

Open a terminal on a running version of your backup-container.  The you'll run the following command after replacing the <DatabaseName>, <DatabaseUser> and <DatabasePassword> with some real values:
POSTGRESQL_DATABASE=<DatabaseName> \
POSTGRESQL_USER=<DatabaseUser> \
POSTGRESQL_PASSWORD=<DatabasePassword> \
run-postgresql

For the purpose of troubleshooting it does not really matter what the values are for <DatabaseName>, <DatabaseUser> and <DatabasePassword>.  Running the command will start start the server and you'll be able to see the log output and hopefully find out where it's failing.

I'm interested in this as well, using Postgis extension of Postgres 13, and just starting to look at using backup container.

Below is the output;
sh-4.2$ POSTGRESQL_DATABASE=mals \

POSTGRESQL_USER=postgres
POSTGRESQL_PASSWORD=#####
run-postgresql
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/pgsql/data/userdata ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... PST8PDT
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
pg_ctl -D /var/lib/pgsql/data/userdata -l logfile start

waiting for server to start....2021-07-28 13:25:13.746 PDT [38077] LOG:  redirecting log output to logging collector process
2021-07-28 13:25:13.746 PDT [38077] HINT:  Future log output will appear in directory "log".
done
server started
/var/run/postgresql:5432 - accepting connections
createuser: error: creation of new role failed: ERROR:  role "postgres" already exists

I'm interested in this as well, using Postgis extension of Postgres 13, and just starting to look at using backup container.

Once the upgrade is complete, the Postgis extension would need to be installed on the backup-container for your use case.  It should be a simple update to the dockerfile.

@MScallion, Other then the createuser: error: creation of new role failed: ERROR:  role "postgres" already exists, it looks like it's running.  So you're issue may be due to the way the container is configured.

Settings:
- Run mode: cron

- Backup strategy: rolling
- Current backup type: daily
- Backups to retain:
  - Daily: 1
  - Weekly: 1
  - Monthly: 1
- Current backup folder: /backups/daily/2021-07-28/
- Time Zone: PDT -0700

- Schedule:
  - 0 1 * * * default ./backup.sh -s
  - 0 4 * * * default ./backup.sh -s -v all

- Container Type: postgres
- Databases (filtered by container type):
  - postgres=patroni-master-dev/mals

- FTP server: not configured
- Webhook Endpoint: https://parcsystems.webhook.office.com/webhookb2/14b3e8f2-8035-4b3b-bf09-82f0b981616e@0d413f05-310c-4aad-8859-8943f3ded1f0/IncomingWebhook/8ac18ec5da754f5dbd7701dbb173c63e/9d0970f2-adfc-4c59-b638-053c71980087
- Environment Friendly Name: MALS
- Environment Name (Id): dev

Should the Environment Name be the namespace?

Should the Environment Name be the namespace?

Yes, but that shouldn't effect the verification.

I accepted all of the default values for the build. Below is our param file for the deployment.
Side note, the webhook did not produce notifications in Teams, as desired.
APP_NAME                              =mals-bkup
BACKUP_VOLUME_NAME                    =backup-mals
DAILY_BACKUPS                         ="1"
DATABASE_DEPLOYMENT_NAME              =patroni-dev
DATABASE_NAME                         ="mals"
DATABASE_PASSWORD_KEY_NAME            =app-db-owner-password
DATABASE_USER_KEY_NAME                =app-db-owner-username
ENVIRONMENT_NAME                      ="dev"
ENVIRONMENT_FRIENDLY_NAME             =""
IMAGE_NAMESPACE                       =30245e-tools
MONTHLY_BACKUPS                       ="1"
NAMESPACE_NAME                        ="30245e"
SOURCE_IMAGE_NAME                     =mals-bkup
TABLE_SCHEMA                          =mals_app
TAG_NAME                              =v1
WEBHOOK_URL                           ="https://parcsystems.webhook.office.com/webhookb2/14b3e8f2-8035-4b3b-bf09-82f0b981616e@0d413f05-310c-4aad-8859-8943f3ded1f0/IncomingWebhook/8ac18ec5da754f5dbd7701dbb173c63e/9d0970f2-adfc-4c59-b638-053c71980087"
WEEKLY_BACKUPS                        ="1"
#
#
#
# Parameter Defaults
#
#
# BACKUP_DIR                            =/backups/
# BACKUP_PERIOD                         =""
# BACKUP_STRATEGY                       =rolling
# BACKUP_VOLUME_CLASS                   =netapp-file-backup
# BACKUP_VOLUME_SIZE                    =5Gi
# CONFIG_FILE_NAME                      =backup.conf
# CONFIG_MAP_NAME                       =backup-conf
# CONFIG_MOUNT_PATH                     =/
# CPU_LIMIT                             ="0"
# CPU_REQUEST                           ="0"
# DATABASE_SERVICE_NAME                 =""
# FTP_PASSWORD                          =""
# FTP_SECRET_KEY                        =ftp-secret
# FTP_URL                               =""
# FTP_URL_HOST                          =
# FTP_USER                              =""
# MEMORY_LIMIT                          =0Mi
# MEMORY_REQUEST                        =0Mi
# MONGODB_AUTHENTICATION_DATABASE       =""
# MSSQL_SA_PASSWORD                     =
# NAME                                  =backup-postgres
# NUM_BACKUPS                           =""#
# ROLE                                  =backup-container#
# VERIFICATION_VOLUME_CLASS             =netapp-file-standard
# VERIFICATION_VOLUME_MOUNT_PATH        =/var/lib/pgsql/data
# VERIFICATION_VOLUME_NAME              =backup-verification
# VERIFICATION_VOLUME_SIZE              =1Gi#
# WEBHOOK_URL_HOST                      =

When I updated the app database to v13 I did not contribute the changes because I split the database user into a DB owner and an application proxy user. I understood that this could not be used because it was inconsistent with previous templates.
Hopefully this is not an issue for the verification as the backup appears to have accepted the user/password that I provided.

@MScallion, You are missing environment variables for PATRONI_MASTER_DEV_USER and PATRONI_MASTER_DEV_PASSWORD, as defined here; https://github.com/BCDevOps/backup-container#backupconf

Examples:
DC - https://github.com/bcgov/orgbook-configurations/blob/master/openshift/templates/backup/backup-deploy.yaml#L128-L147
backup.conf - https://github.com/bcgov/orgbook-configurations/blob/master/openshift/templates/backup/config/backup.conf#L39-L40

That would be causing these lookups to return blank strings and cause the startup to fail; https://github.com/BCDevOps/backup-container/blob/master/docker/backup.postgres.plugin#L119-L120

When I updated the app database to v13 I did not contribute the changes because I split the database user into a DB owner and an application proxy user. I understood that this could not be used because it was inconsistent with previous templates.
Hopefully this is not an issue for the verification as the backup appears to have accepted the user/password that I provided.

Are you saying you've modified the code, or just configured the container in a very specific way?

Thanks Wade, I don't see those in the backup-container repository, which is the repo that I cloned. I added the four parameters to the deployment template.
I am not sure how to update the config file. Currently I have defined the database as;
postgres=patroni-master-dev/mals

Should I add the two lines as-is? ie are these values to be set the same for all projects or should I update them to reflect project specific components?
 db-bc:5432/aries-vcr
 wallet-bc:5432/agent_bc_wallet

Those values are a customization that need to be done for each application of the container when you use the backup.conf file to define the backups and schedule.  I was just providing examples of the customization done for another project.
The examples were to show:
DB_BC_USER, and DB_BC_PASSWORD environment variables are associated with the db-bc:5432/aries-vcr entry in the backup.conf file, and WALLET_BC_USER and WALLET_BC_PASSWORD environment variables are associated with the wallet-bc:5432/agent_bc_wallet entry in the backup.conf file.  They map the credentials for the databases, via naming convention for the scripts to pick up.

Are you saying you've modified the code, or just configured the container in a very specific way?

I replaced the APP_USER/APP_PASSWORD secret values with app-db-owner-username/app-db-owner-password and app-proxy-username/app-proxy-password in order to configure a super user, DB owner and an application user, in an attempt to be in compliance with the principle of least privileges. I then administered the necessary role and grants for the app user.
I also updated the Dockerfile to references:
FROM postgres:13
ENV PATRONI_VERSION=2.0.1

and to include grants that helped to resolve an error early on, possibly as a result of the changes to the users;
RUN chmod a+x /usr/share/scripts/patroni/health_check.sh
RUN chmod a+x /usr/share/scripts/patroni/post_init.sh

So for our application database we have used the following parameters in the deployment template;
            - name: DATABASE_USER
              valueFrom:
                secretKeyRef:
                  name: ${DATABASE_DEPLOYMENT_NAME}
                  key: ${DATABASE_USER_KEY_NAME}
            - name: DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: ${DATABASE_DEPLOYMENT_NAME}
                  key: ${DATABASE_PASSWORD_KEY_NAME}

For which I have provided the following values;
DATABASE_DEPLOYMENT_NAME              =patroni-dev
DATABASE_NAME                         ="mals"
DATABASE_PASSWORD_KEY_NAME            =app-db-owner-password
DATABASE_USER_KEY_NAME                =app-db-owner-username

So for our application database we have used the following parameters in the deployment template;
            - name: DATABASE_USER
              valueFrom:
                secretKeyRef:
                  name: ${DATABASE_DEPLOYMENT_NAME}
                  key: ${DATABASE_USER_KEY_NAME}
            - name: DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: ${DATABASE_DEPLOYMENT_NAME}
                  key: ${DATABASE_PASSWORD_KEY_NAME}

For which I have provided the following values;
DATABASE_DEPLOYMENT_NAME              =patroni-dev (Name of the Secret)
DATABASE_NAME                         ="mals"
DATABASE_PASSWORD_KEY_NAME            =app-db-owner-password
DATABASE_USER_KEY_NAME                =app-db-owner-username

Those are the default ones which are used for the legacy mode of operation only.  When using the newer cron mode of operation with the backup.conf file you need to update your deployment configuration with the additional "credential" variables as laid out in the documentation here, https://github.com/BCDevOps/backup-container#backupconf.  The examples I provided follow those conventions.

Understood, I will work to add those then post my results.
I had assumed that because the backups execute successfully that the credentials might not  be the root cause of the issue. The backups successfully complete in ~2 seconds, condensing the 66mb DB storage into 13mb in the backup file. I just updated the secret username and password pair to use the postgres user and the issue persisted,

The readme states that the parameter names are prefixed with the service name. Does it have to be exact? If so then I will likely need to alter our service names as they currently include the environment, ie patroni-master-dev.
I am hoping that I can use PATRONI_MASTER_USER, PATRONI_MASTER_PASSWORD for all environments, without having to modify the existing service names.

The verification again timed out in 5m after I added those parameters to the deployment template and redeployed;
PATRONI_MASTER_DEV_USER
PATRONI_MASTER_DEV_PASSWORD

@WadeBarnes, I have cloned/updated the orgbook-containers repo and I am able to run both the backup and validation. I am now receiving a validation error stating that the application role, to which I grant all object privileges, does not exist. The grants are required because I have split the user into a DB owner and application proxy user. I am investigating alternate dumpfile options, ie pg_dumpall and/or the -x option, to find a best fit for my current configuration and restore capabilities.

The readme states that the parameter names are prefixed with the service name. Does it have to be exact? If so then I will likely need to alter our service names as they currently include the environment, ie patroni-master-dev.
I am hoping that I can use PATRONI_MASTER_USER, PATRONI_MASTER_PASSWORD for all environments, without having to modify the existing service names.

No, it has to be exact.  It's best practice to keep environment designations (i.e. dev, test, prod) out of your resource names where ever possible.  Ideally you want your resource names to be exactly the same in each and every environment, except for image tag used to deploy the images into a given namespace.

@WadeBarnes, I have cloned/updated the orgbook-containers repo and I am able to run both the backup and validation. I am now receiving a validation error stating that the application role, to which I grant all object privileges, does not exist. The grants are required because I have split the user into a DB owner and application proxy user. I am investigating alternate dumpfile options, ie pg_dumpall and/or the -x option, to find a best fit for my current configuration and restore capabilities.

Would you be able to share the error log?

Thanks for the tip Wade. One limitation that I see is that we have test and UAT in the same namespace. We will need to review our configuration and naming standards.
I have confirmed that the role does not exist in the generated SQL. The documentation states that pg_dumpall can include the roles and I will work with that to determine the best settings for our requirement.
[!!ERROR!!] - Backup verification failed: /backups/daily/2021-07-31/patroni-master-dev-mals_2021-07-31_01-00-00.sql.gz

The following issues were encountered during backup verification;
Restoring '/backups/daily/2021-07-31/patroni-master-dev-mals_2021-07-31_01-00-00.sql.gz' to '<IP_ADDRESS>/mals' ...

ERROR:  role "mals_app_role" does not exist

Restore failed.

FYI, I tried running backup verification with my postgis 13:3-.1 image, within a OpenShift CronJob, and got the following error:
find: ‘/var/lib/pgsql/data’: No such file or directory
Cleaning up ...
waiting for server to start.................................................................................................
The server failed to start within 0h:2m:0s.

The server started in 0h:2m:0s.

pg_ctl: directory "/var/lib/pgsql/data/userdata" does not exist
find: ‘/var/lib/pgsql/data’: No such file or directory
Cleaning up ...
Checking the image, /var/lib/postgresql/data exists instead.
The backups seems to run properly (from a CronJob), I just seem to have issues running backup verification.

@basilv, Do you have a volume mounted to /var/lib/pgsql/data in order to hold the data being restored during the verification?

Oops, yup would help to mount the verification PVC.

Here's the latest result after adding the verification PVC and setting the database startup timeout to 300 seconds
waiting for server to start.......................................................................................................................................................................................................................................................
The server failed to start within 0h:5m:0s.

The server started in 0h:5m:0s.

pg_ctl: directory "/var/lib/pgsql/data/userdata" does not exist
Already clean ...
[!!ERROR!!] - Backup verification failed: /backups/daily/2021-08-04/fom-db-dev-86-fom_2021-08-04_14-46-14.sql.gz
Note that I'm not using separate roles/accounts like MScallion, so that shouldn't be the issue. Presumably I'm either missing some required setting, or there's a Postgres 13 compatibility issue. I do have the output from /backup.sh -c, provided below:
Listing configuration settings ...
Settings:

Run mode: legacy

Backup strategy: rolling

Current backup type: daily

Backups to retain:

Daily: 7
Weekly: 4
Monthly: 1

Current backup folder: /backups/daily/2021-08-04/

Schedule:

1d

Container Type: postgres

Databases (filtered by container type):

fom-db-dev-86:5432/fom

FTP server: not configured

Webhook Endpoint: not configured

Environment Friendly Name: not configured

Environment Name (Id): not configured

@basilv, Did the trouble you were having here get resolved over here, https://github.com/BCDevOps/backup-container/issues/85?

@MScallion, have you been able to resolve the issues you were having?

@basilv, Did the trouble you were having here get resolved over here, #85?

No, as per #85 I figured out how to run the backup verification from my cron job, but it still fails as described above. I've manually tested the database restore process and confirmed the backups are working properly, so its just something in the verification script. Not sure if it is a Postgres 13 issue, or a PostGIS issue (e.g. maybe the verification script is skipping initialization needed for PostGIS extension).

It's failing the start postgres during the verification, which is typically due to the database credentials not being wired up correctly.  You could troubleshoot by running the onStartServer scripts manually without the >/dev/null 2>&1 bit so you can see the logs; https://github.com/BCDevOps/backup-container/blob/master/docker/backup.postgres.plugin#L104-L123

Would be nice to have an option to show the logs without having to build a completely different container to remove the >/dev/null 2>&1 bit. Looking at the onStartServer function defined in backup.postgres.plugin, it seems to execute 'run-postgresql', but I don't see that defined anywhere (and would have expected pg_ctl to be used, like it is in onStopServer).
I tried running pg_ctl start/init, here's the results:
$ pg_ctl start
pg_ctl: directory "/var/lib/postgresql/data" is not a database cluster directory
(this is after overriding VERIFICATION_VOLUME_MOUNT_PATH to be this location instead of /var/lib/pgsql/data)
$ pg_ctl init
initdb: could not look up effective user ID<PHONE_NUMBER>: user does not exist
(my postgres container runs as user postgres, but the backup container Dockerfile uses user 26, maybe that's the issue?)
Also, I've had a hard time telling from the documentation how to supply the credentials. I'm using Openshift CronJob approach. For the cronjob container, I'm supplying env variables DATABASE_USER and DATABASE_PASSWORD as well as POSTGRESQL_USER and POSTGRESQL_PASSWORD (all values pulled from secret). I'm pretty sure I just need DATABASE_USER & DATABASE_PASSWORD though.
(FYI, I've updated my database to use patroni (with postgres 13 and PostGIS). This hasn't seemed to impact the backup procedure at all - still works fine, so at least that's positive.)

@MScallion, have you been able to resolve the issues you were having?

No. I would need to update the backup.postgres.plugin file in the BCDevOps/backup-container repo to use pg_dumpall, which I have verified locally, in place of pg_dump -Fc,. I am also not able to pull from my fork of that repo due to security settings. In the short-term I was able to backup and successfully restore the database manually by 1) executing my deployment script to create the database, users and roles then 2) restoring the SQL from the backup. Fortunately I had separated the create database from the create DB objects in my deployment scripts.
I am not able to successfully validate the backup files using the automation provided in the backup-container scripts, due to the grants to the roles, which I have added to the source templates and which are not included in the pg_dump function.

Hi! I'm a security researcher @TruffleSecurity. I found a valid API key leaked in a comment in this Issue. I highly recommend rotating the API key immediately (not just deleting the comment). Here's a blog post I wrote on remediating these types of vulnerabilities: https://trufflesecurity.com/blog/remediate-leaked-api-keys-with-key-rotation.If you can't find the leaked API key, you can use the following TruffleHog command to automatically scan for it:trufflehog github --repo https://github.com/<your_username>/<your_reponame> --issue-comments --only-verifiedIf it's not clear where/what the security issue is, please let me know. I'd be happy to help. Please note that the API key might be in the past edit of a comment.(cc @MScallion)

Thanks @joeleonjr , I checked all my prior comments but see no API keys. (I didn't check old edits of comments, no idea how to do that.)

Thanks @joeleonjr , I checked all my prior comments but see no API keys. (I didn't check old edits of comments, no idea how to do that.)

MScallion's post from July 28, 2021 has a WEBHOOK_URL value exposing the secret. And a separate comment from that day as a Webhook Endpoint value exposing the secret. Please let me know once that's addressed and I'll delete this comment.

Thanks! @MScallion is on a separate team, so they'll need to respond.


Fixes NPE When Using JdbcIO.PoolableDataSourceProvider

This PR attempts to address a null pointer exception caused by the PoolableDataSourceProvider.
When using a simple PoolableDataSourceProvider in the Dataflow Runner I get a null pointer exception at runtime.
JdbcIO.<~>write()
  .withDataSourceProviderFn(
    JdbcIO.PoolableDataSourceProvider.of(
      JdbcIO.DataSourceConfiguration
        .create("org.postgresql.Driver", jdbcUrl)
          .withUsername(jdbcUsername)
          .withPassword(jdbcPassword)
    )
  )

Other users seem to have a similar issue: https://issues.apache.org/jira/browse/BEAM-7230?focusedCommentId=16845769&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16845769.

R: @lukecwik or @kennknowles

Thanks! Can you add a test that reproduces the NPE?

It would actually be great to either tag this with BEAM-7230 if that is appropriate, or file a new issue just about the NPE. That will make the bug show up in automatically generated release notes, and gather the info for anyone looking into the problem.

Just as a minor comment on this, the reason why the impl went into the static approach (that now I see has clear issues) was because of the opposite case the original reported issue happened because JdbcIO was building a pool of connections per DoFn instantiation which was reported to be overwhelming the target db on max number of connections on streaming pipelines.

This is superseded by #9927


Added default country check to en-PAK unit tests

Added unit test which checks that default country includes substring "Pakistan" in en-PAK locale.
bundle exec rake executed - all green.
Thank you.

Looks like the assert_include failed even if it was green on local. Replaced this with regex.