IOU - IOU is duplicated after request errors

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Send request money from user B to A(MAIN)
Turn off Internet in device A pay the request
Cancel IOU in user B
Turn internet on in Device A -red dot error message occurs in request
Request the same amount from user B to A

Expected Result:
IOU should not be duplicated after request.
Actual Result:
Previous IOU is duplicated after request over "Error" request
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[x] Android / native
[x] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

Version Number: <IP_ADDRESS>
Reproducible in staging?: Yes
Reproducible in production?: Yes
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
https://user-images.githubusercontent.com/93399543/220142072-9a768eaa-709a-4133-b52d-2a8f9e3e1825.mp4

Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
View all open jobs on GitHub

Triggered auto assignment to @abekkala (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

Bug0 Triage Checklist (Main S/O)

[ ] This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
[ ] This bug is not a duplicate report (check E/App issues and #expensify-bugs)

If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead

[ ] This bug is reproducible using the reproduction steps in the OP. S/O

If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
If the reproduction steps aren't clear and you determine the correct steps, please update the OP.

[ ] This issue is filled out as thoroughly and clearly as possible

Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.

[ ] I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

@PauloGasparSv would the fix you're working on here: https://github.com/Expensify/App/issues/14982
Resolve this?

Hey @abekkala, will get back to you on that (probably) tomorrow, still not sure!

Ok thanks @PauloGasparSv!

still waiting to see if fix from other will resolve this as well

Hey @abekkala, sorry for the delay but I only had time for the issue today!
I've been working on https://github.com/Expensify/App/issues/14982 this morning and so far, I think this is a different error that is also related to Onyx.
Tested here  and user A ends up with an extra reportAction. I think this is different from https://github.com/Expensify/App/issues/14982 because this extra reportAction has a valid reportActionID (instead of a sequenceNumber in it's place) Here are some screenshots:
USER B:

USER A:

The duplicated message starts to show after we send back each participant notification from their perspective here, but this needs further investigation : )

@abekkala I think this will require it's own investigation but will have to be fixed internally, so maybe no to the Upwork job and we just assign an engineer. IMO it's your call, I think we can also add a Upwork job and let a contributor confirm if this is indeed internal

@aldo-expensify I see you created this yesterday https://github.com/Expensify/App/issues/15584
Do you want to close yours and I can make an external job for this?
unless you prefer Internal?

@aldo-expensify I see you created this yesterday #15584 Do you want to close yours and I can make an external job for this? unless you prefer Internal?

I think it needs to be worked on the backend. I'll try to investigate today or tomorrow.

sounds good, thanks @aldo-expensify
I'll wait for the outcome of that before we decide on this one

@abekkala this issue was created 2 weeks ago. Are we close to a solution? Let's make sure we're treating this as a top priority. Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

PD: There is some chance that this is related to https://github.com/Expensify/App/issues/14982 and https://github.com/Expensify/Expensify/issues/263900. I was planning on investigating those today/tomorrow.

@aldo-expensify did your investigation help determine if this could be fixed by the other one. Or should we proceed with it's own Upwork job here?

@abekkala this issue was created 2 weeks ago. Are we close to a solution? Let's make sure we're treating this as a top priority. Don't hesitate to create a thread in #expensify-open-source to align faster in real time. Thanks!

@abekkala I investigated this more and I think there is frontend and backend stuff that needs fixing:
Front end: Around here https://github.com/Expensify/App/blob/6413a798a64f179541eee79778102563d0c0a74d/src/libs/actions/IOU.js#L797-L867 we create the optimisticData and failureData. I'm guessing that the modifications we did optimistically are not rolled back properly when the user clears the error.
I'm also not 100% about how this should look from a user perspective. For example:
After a failure, should the "Settled up elsewhere" line in the IOU details remain there?

This is how a failure looks when we add a comment (noticed the opacity change):

I'm thinking that the approach should be the same. The "Settled up elsewhere" should look a bit transparent on error and should disappear when the user clicks the X to dismiss the error.
Back end: The error coming from the backend seems to be set in the wrong position:

The errors should be placed inside a reportAction, not outside like it appears now.

I think can be worked on simultaneously. It's a valid outstanding bug that shouldn't conflicts too much with our changes.

Thanks @Julesssss!

Job added to Upwork: https://www.upwork.com/jobs/~01fd0d26525bf69427

Current assignee @abekkala is eligible for the External assigner, not assigning anyone new.

Triggered auto assignment to Contributor-plus team member for initial proposal review - @aimane-chnaif (External)

Current assignee @aldo-expensify is eligible for the External assigner, not assigning anyone new.

Proposal
Please re-state the problem that we are trying to solve in this issue.
When money is requested by another user and there are previous failed sends to that user, then the money request is duplicated in the report.
It is also shown at places where previous sends failed. Here's an example:

What is the root cause of that problem?
This is due to an incorrect check in src-> components-> ReportActionItem-> IOUAction.js-> IOUAction():

const shouldShowIOUPreview = (
props.isMostRecentIOUReportAction
&& Boolean(props.action.originalMessage.IOUReportID)
&& props.chatReport.hasOutstandingIOU) || props.action.originalMessage.type === 'pay';

The OR condition and brackets are not placed correctly.
What changes do you think we should make in order to solve the problem?
We should check if the IOU report is the most recent even when the originalMessage.type === 'pay';
Therefore the above condition should be corrected to:

const shouldShowIOUPreview = props.isMostRecentIOUReportAction
&& Boolean(props.action.originalMessage.IOUReportID)
&& (props.chatReport.hasOutstandingIOU || props.action.originalMessage.type === 'pay');

This fixes the issue as shown:

Proposal
Please re-state the problem that we are trying to solve in this issue.
Request money message is duplicated on main device after request error
This is the problem from both be and fe side.
As @aldo-expensify mentioned here

I'm thinking that the approach should be the same. The "Settled up elsewhere" should look a bit transparent on error and should disappear when the user clicks the X to dismiss the error.

We should transparent on error and disappear when user clicks the X button
What is the root cause of that problem?
In https://github.com/Expensify/App/blob/657b439007004a5a128fc76df00327cb14203915/src/components/OfflineWithFeedback.js#L88
we don't make opacity for the iou error message
and in ReportActionItem we also don't have the logic to delete settled up elsewhere
What changes do you think we should make in order to solve the problem?
we should have the new logic to transparent the iou error message in OfflineWithFeedback,js
    const isIUOError = hasErrors && props.actionName===CONST.BETAS.IOU
    const needsOpacity = (isOfflinePendingAction && !isUpdateOrDeleteError) || isAddError || isIUOError;

In ReportActionItem.js
we should pass actionName and deleteOptimisticReportAction if actionName===IOU
onClose={() => {
    if (this.props.action.pendingAction === CONST.RED_BRICK_ROAD_PENDING_ACTION.ADD || this.props.action.actionName === CONST.BETAS.IOU) {
    ReportActions.deleteOptimisticReportAction(this.props.report.reportID, this.props.action.reportActionID);
    } else {
    ReportActions.clearReportActionErrors(this.props.report.reportID, this.props.action.reportActionID);
  }
}}
pendingAction={this.props.draftMessage ? null : this.props.action.pendingAction}
errors={this.props.action.errors}
errorRowStyles={[styles.ml10, styles.mr2]}
actionName={this.props.action.actionName}```

In https://github.com/Expensify/App/blob/main/src/pages/iou/IOUDetailsModal.js
we have to get `pendingAction` from `reportActions` in `findPendingAction` function. It may not related to this issue but we should do this to prevent unexpected error.

Result
https://user-images.githubusercontent.com/113963320/224228792-c2c44493-d559-4fc0-883a-d9c02b7e875b.mp4

not overdue, melvin! Proposals are being viewed!

@abekkala @aldo-expensify @aimane-chnaif this issue is now 3 weeks old. There is one more week left before this issue breaks WAQ and will need to go internal. What needs to happen to get a PR in review this week? Please create a thread in #expensify-open-source to discuss. Thanks!

@aimane-chnaif any thoughts on the proposals? I'll have a look after lunch

@aimane-chnaif my take is that the expected results should be:
RHN, IOU details with failed "Settled up elsewhere":

Notice the opacity change in the entry "Settled up elsewhere" and the alignment of the error. Don't worry about the error message text context for now, I think that has to be fixed in the server.
When you dismiss the error the semi-transparent "Settled up elsewhere" entry should be removed too
Chat

The failed "Settled up elsewhere" in the main chat should have less opacity and the error should be below it.
Dismissing the error should clear the "Settled up elsewhere" entry and return to what you initially had:

The failed "Settled up elsewhere" entries are not really stored in the server, if you log out and log in, they are not there anymore. This is why it makes sense for me that clearing the error should clear them too.
@shawnborton can you check if the above expected behaviour looks correct ^ 🙏

I think that looks correct to me, but once again, calling on my offline pattern experts @trjExpensify and @JmillsExpensify for a second opinion here.

Oh, interesting! Couple of questions..

why are the error messages above different? I would have thought they should be the same.
why does the error message show in the RHP view as well as the chat view?

On the second Q, there should only ever really be one "end" of the red brick road, which is where the error message to dismiss lives. So if the end of the road is where the reportAction is displayed in the report detailed view (RHP), then there should only be a red dot in the chat view.

why are the error messages above different? I would have thought they should be the same.

Don't mind the error message content, that I something I have to fix in the backend I believe and they are different just because I edited them manually to take the screenshot.

why does the error message show in the RHP view as well as the chat view?
On the second Q, there should only ever really be one "end" of the red brick road, which is where the error message to dismiss lives. So if the end of the road is where the reportAction is displayed in the report detailed view (RHP), then there should only be a red dot in the chat view.

Interesting point, I think I agree, @shawnborton ?

Updated Proposal

Notice the opacity change in the entry "Settled up elsewhere"

As @aldo-expensify mentioned above, I want to update my proposal to transparent the "Settled up elsewhere" and remove it when user dismiss the error.
What changes do you think we should make in order to solve the problem?
we also need to pass actionName and deleteOptimisticReportAction if actionName===IOU in ReportTransaction.js

<OfflineWithFeedback
    onClose={() => {
        if (this.props.action.pendingAction === CONST.RED_BRICK_ROAD_PENDING_ACTION.ADD || this.props.action.actionName === CONST.BETAS.IOU){
            ReportActions.deleteOptimisticReportAction(this.props.report.reportID, this.props.action.reportActionID);
        } else {
            ReportActions.clearReportActionErrors(this.props.report.reportID, this.props.action.reportActionID);
        }
    }}
    pendingAction={this.props.draftMessage ? null : this.props.action.pendingAction}
    errors={this.props.action.errors}
    errorRowStyles={[styles.ml10, styles.mr2]}
    actionName={this.props.action.actionName}
>

Result
https://user-images.githubusercontent.com/113963320/225502150-1d8ea21e-17ef-4580-a937-eebdeca34c52.mov

Interesting point, I think I agree, @shawnborton ?

Yup, I agree.

Thanks @shawnborton, can you help me with how it should look the red dot next to the failed payment without the error message in the chat? I'm having a hard time imagining that.

I think the red dot would go next to the right caret? (Presumably it's clickable to access the report where you can see the failed reportAction?

@abekkala @aldo-expensify @aimane-chnaif this issue is now 4 weeks old and preventing us from maintaining WAQ, can you:

Decide whether any proposals currently meet our guidelines and can be approved as-is today
If no proposals meet that standard, please take this issue internal and treat it as one of your highest priorities
If you have any questions, don't hesitate to start a discussion in #expensify-open-source

Thanks!

Current assignee @aimane-chnaif is eligible for the Internal assigner, not assigning anyone new.

We still need to confirm 2 things here:

After hiding dismissible error in chat page, how design look like? Where's red dot's position?
I think we've not discussed yet clearly whether to show or hide this view on both pages.

On your second point about the box with "You payed X", I think it should stay there with less opacity because that is how pattern B feedback work for things that failed to be added. We don't remove them automatically when we get the error.

We decided to continue with a design of mine for now, and if later we decided to adjust, we can do it in a follow up PR. I'll update the "Expected results" in the issue's body.

Triggered auto assignment to @bfitzexpensify (External), see https://stackoverflow.com/c/expensify/questions/8582 for more details.

Current assignee @aimane-chnaif is eligible for the External assigner, not assigning anyone new.

Current assignee @aldo-expensify is eligible for the External assigner, not assigning anyone new.

Updated the expected position of the red dot in the chat again.

Upwork job price has been updated to $2000

@aimane-chnaif we are ready to start reviewing proposals again. The expected results are described in the GH issue's body, let me know if you find something unclear.

Updated Proposal
As new expectations for the offline pattern is updated, here is my updated proposal.
Please re-state the problem that we are trying to solve in this issue.
Request money message is duplicated on main device after request error.
What is the root cause of that problem?

We don't add opacity in case of IOU error

In https://github.com/Expensify/App/blob/657b439007004a5a128fc76df00327cb14203915/src/components/OfflineWithFeedback.js#L88

The ReportActionItem currently use OfflineWithFeedback to show error messages. Therefore the expectation to only use red dot for IOU error is not fulfilled.

What changes do you think we should make in order to solve the problem?

To fix 1, we check if this is an IOU error and add transparent condition for that in OfflineWithFeedback,js:

    const needsOpacity = (isOfflinePendingAction && !isUpdateOrDeleteError) || isAddError || isIUOError;

To fix 2, set shouldShowErrorMessages in ReportActionItem to false in case this is an IOU; and use dot instead, the same as in OptionRowLHN:

https://github.com/Expensify/App/blob/b0b31d1b9f424a443a5a475a37ba2029c31a7dbf/src/components/LHNOptionsList/OptionRowLHN.js#L204-L213

@bfitzexpensify  I'm reassigning as I'm ooo until Tues April 04 and this one will see some movement in that time.
Current Status:

Upwork Job posted
Proposals are being reviewed

When its time to make payments:
Issue reported by: Applause - Internal Team (no payment needed)
Fix: TBD - a proposal hasn't been officially accepted, assigned to this GH yet
PR Review: will be done by @aimane-chnaif

Looks like we're still seeking proposals here

@aimane-chnaif @bfitzexpensify As you've reminded, this is my final proposal.

@aimane-chnaif checking in here, what do you think of the proposal? Otherwise we could get Callstack to help with this one

Upwork job price has been updated to $2000

My concern is 2. is expected behavior or bug to be fixed.

@aimane-chnaif did you check the updated "Expected Result" in the GH issue description? I feel like the case you are asking about is contained there, unless I'm missing something.
According to the "Expected results", this is indeed a bug:

The widget

should have not disappeared, but it should have lower opacity, and the error text should no appear in the chat.

@tienifr do you have some thoughts on @aimane-chnaif 's comments? https://github.com/Expensify/App/issues/15297#issuecomment-1488701103
Will the proposal meet what the "Expected results" described in the GH issue?

From @aimane-chnaif's comment, there are two points:

if we re-use needsOpacity in OfflineWithFeedback, it will be applied to red dot as well. Red dot should have 100% opacity

I have updated my proposal to match this, also uploaded the demo video.

IOUPreview still says "You owe..." with "Pay" button. It should be "You paid..." without "Pay" button to be aligned with "Settled up elsewhere"

I believe there had been changes from the back-end where they resolved the IOU amount. The IOU amount is set to 0 after being settled up or canceled. This makes the IOU preview hidden:
https://github.com/Expensify/App/blob/df10eeba7106a2b7477208a8b4325d5b219d26af/src/components/ReportActionItem/IOUPreview.js#L130-L132
Is it okay if I removed this logic? Or else we need to find another soution.
Result

Result
This is the result after applying my second suggestion above:

Just straight removing the check: https://github.com/Expensify/App/blob/df10eeba7106a2b7477208a8b4325d5b219d26af/src/components/ReportActionItem/IOUPreview.js#L130-L132
will probably cause some problem, have you tested that? Also, I'm guessing it will show "0 Payed", which would be wrong as just not showing the amount.
Is the old amount still stored somewhere? Or it gets completely lost? if it is stored somewhere, maybe we can still use it if props.iouReport.total === 0 and there are errors found.

Proposal
Please re-state the problem that we are trying to solve in this issue.
IOU Preview with Pay button is duplicated after the 2nd request if the first request is paid but with an error because get cancelled by the requestor.
What is the root cause of that problem?
I have explained a little bit on how iou request works here https://github.com/Expensify/App/issues/16775#issuecomment-1498587658. In short, each IOU request create a new report action and is attached to an instance of IOU report. Each IOU report will be closed when the payment is settled. In our case, this is what happens:
Let say this is a fresh IOU request, which means iou report does not exist yet

User A request X amount of money from user B (iou report and report action will be generated and the total amount is attached to the iou report while the request amount is saved to the report action (which can be seen here https://github.com/Expensify/App/issues/15297#issuecomment-1499828964))
User B turn offline and pay
User A cancel the request which means, the iou report total now will be 0.
User B turn online and see that the pay request is failed because user A canceled it. The IOU preview is now hidden because the iou report total is 0. Remember, all these request belong to the same iou report.
User A request again X amount of money, which means the total is X now, thus the IOU preview is shown on the failed/error request.
(If we add more requests, the total shown for both IOU preview will be increased)

The issue with the opacity has the same root cause as my proposal here https://github.com/Expensify/App/issues/16775#issuecomment-1497928130.
What changes do you think we should make in order to solve the problem?

For the opacity issue, we can apply the solution from my proposal here https://github.com/Expensify/App/issues/16775#issuecomment-1497928130.
Now, for the new design of the red dot, I honestly don't think it's feasible and worth to do.

Currently, OfflineWithFeedback wraps the whole red box. Setting the opacity of the red dot to 1 does not work. If we disable it, then we need to wrap each of the other component (except the red dot) with OfflineWithFeedback (ignore the green box). Then, it won't be consistent with the other red dot design and user now need to open the IOU details to clear the error.

The widget
should have not disappeared,

For the last issue, I think we are good to hide the IOU preview when there is an error. Yes, we can use the amount from the report action as shown here https://github.com/Expensify/App/issues/15297#issuecomment-1499828964
However, IOU preview does not only show the amount, but also show the "You owe .."/"You paid ..." text, the green checkmark, etc and all of the information (hasOutstandingIOU, stateNum), but those don't present in the report action data, that's why this could happen

IOUPreview still says "You owe..." with "Pay" button. It should be "You paid..." without "Pay" button to be aligned with "Settled up elsewhere"

This is how we can hide if there are errors:
https://github.com/Expensify/App/blob/64f4a7814e87bd7c4d085776abf5cb8a67b9e8bc/src/components/ReportActionItem/IOUAction.js#L73-L76
Change the condition to !props.action.errors && ...currentCondition. This will hide the IOU preview for all IOU action that show an IOU preview (pay & request).

If we really want to keep the IOU preview when pay request failed, then we need to apply these changes:
Currently, the amount, hasOutstandingIOU boolean, etc. is retrieved from the IOU report props, for example.
https://github.com/Expensify/App/blob/64f4a7814e87bd7c4d085776abf5cb8a67b9e8bc/src/components/ReportActionItem/IOUPreview.js#L221-L225
Because we want to show the IOU preview on failed pay request, we can't use all the data (total, hasOutstandingIOU, & stateNum) comes from IOU report, so we need to get the data from the report action originalMessage. For the iouReport.total, we will replace it by props.action.originalMessage.IOUDetails.amount, hasOutstandingIOU will always be false (because we paid it, even though fails), and stateNum to CONST.REPORT.STATE_NUM.SUBMITTED.
let total = props.iouReport.total;
let hasOutstandingIOU = props.iouReport.hasOutstandingIOU;
let stateNum = props.iouReport.stateNum;

if (props.action && !_.isEmpty(props.action.errors) && props.action.originalMessage.type === CONST.IOU.REPORT_ACTION_TYPE.PAY) {
    total = props.action.originalMessage.IOUDetails.amount;
    hasOutstandingIOU = false;
    stateNum = CONST.REPORT.STATE_NUM.SUBMITTED;
}

(but I still prefer to hide it)
https://user-images.githubusercontent.com/50919443/230591181-5d441de8-dd5f-4849-9eaf-f22176e7c4e0.mov

@aldo-expensify Thanks for pointing that out, pass !props.action.errors in App/src/components/ReportActionItem/IOUPreview.js is good idea. And I updated my proposal a bit. Please help take a look

@bfitzexpensify, @aldo-expensify, @aimane-chnaif Whoops! This issue is 2 days overdue. Let's get this updated quick!

@bfitzexpensify, @aldo-expensify, @aimane-chnaif Whoops! This issue is 2 days overdue. Let's get this updated quick!

Update: I asked in slack if callstack could help us with getting this done faster.

Hi I'm Bartek from Callstack - expert contributor group - I'll take care of this issue

@aldo-expensify Just want to know if there is any problem with our proposals.

@tienifr @bernhardoj I requested Callstack's help to move with this faster. If your solutions are used in the final implementation you will be compensated accordingly.

📣 @burczu You have been assigned to this job by @aldo-expensify!
Please apply to this job in Upwork and leave a comment on the Github issue letting us know when we can expect a PR to be ready for review 🧑💻
Keep in mind: Code of Conduct | Contributing 📖

Daily update: I've read the whole thread above and now I'm checking the proposals of both @tienifr and @bernhardoj if we can go with one of them (fully or partially) - I'll most likely post some insights about it tomorrow.

@burczu, @bfitzexpensify, @aldo-expensify, @aimane-chnaif Whoops! This issue is 2 days overdue. Let's get this updated quick!

Proposal
Please re-state the problem that we are trying to solve in this issue.
IOU Preview messages with the Pay button are duplicated after getting back from offline if the request was cancelled meanwhile and the new payment was requested.
What is the root cause of that problem?

IOU Preview with the Pay massages duplication

In the IOU Preview component we rely on the IOU report action (represented by the props.iouReport object in the component) which, in case of failed request, is an optimistic one and is filled by the data from the newly requested payment. The check that is responsible for showing the Pay button in the IOU Preview component looks like below:
https://github.com/Expensify/App/blob/f54a8bd23018644d33114d1820f7e428f18c74f9/src/components/ReportActionItem/IOUPreview.js#L247-L260
In the above condition, this check is responsible for duplicating Pay IOU Preview messages:
props.iouReport.stateNum === CONST.REPORT.STATE_NUM.PROCESSING

In case of failed request, we stay with an optimistic report action with its iouReport object filled with the new payment request data. That's why the iouReport.stateNume has value 1 and the above condition is met.

IOU Preview not staying visible in case of errors

If we fix the previous issue explained above, we encounter another one - for failed payment requests in the chat, the IOU Preview won't show up. The reason is explained below:
The IOU Preview component is rendered when the below condition is met (which is correct):
https://github.com/Expensify/App/blob/f54a8bd23018644d33114d1820f7e428f18c74f9/src/components/ReportActionItem/IOUAction.js#L73-L76
Additionally, inside the IOU Preview we check for the value of total and prevent rendering the component if its value equals 0:
https://github.com/Expensify/App/blob/f54a8bd23018644d33114d1820f7e428f18c74f9/src/components/ReportActionItem/IOUPreview.js#L136-L138
The second condition blocks the preview to be rendered because, the props.iouReport.total equals 0 for optimistic report actions with errors.

The whole IOU Action does does not disappear when the Dismiss (X) button is clicked - only error message is removed.

Additional change requests: lowered opacity on the IOU Preview with errors, with the red dot moved to the top right corner and without the text message in the chat. Dismissing error should be done in the details modal only.

This functionality is just not implemented and needs to be added.
What changes do you think we should make in order to solve the problem?

To solve the issue with duplicated IOU Preview messages with the Pay button we can apply the change proposed by @bernhardoj: for pay reports that has errors we need to override some values in the IOUPreview component.

let total = props.iouReport.total;
let hasOutstandingIOU = props.iouReport.hasOutstandingIOU;
let stateNum = props.iouReport.stateNum;

if (props.action && !_.isEmpty(props.action.errors) && props.action.originalMessage.type === CONST.IOU.REPORT_ACTION_TYPE.PAY) {
    total = props.action.originalMessage.IOUDetails.amount;
    hasOutstandingIOU = false;
    stateNum = CONST.REPORT.STATE_NUM.SUBMITTED;
}

where actually this line: stateNum = CONST.REPORT.STATE_NUM.SUBMITTED; fixes the duplication issue. Other changes will be useful to show correct information on the failed IOU Preview.

Now that we have no duplicated Pay button preview messages anymore, we can observe that the IOU Preview does not show up at all for the payment requests with errors.

It can be fixed by additional check for errors in the condition inside the IOU Preview component, so we need to change this:
https://github.com/Expensify/App/blob/f54a8bd23018644d33114d1820f7e428f18c74f9/src/components/ReportActionItem/IOUPreview.js#L136-L138
to this:
if (props.iouReport.total === 0 && !(props.action || {}).errors) {
    return null;
}

To completely remove the optimistic report action when clicking on the Dismiss button we need to change the ReportTransaction component.

https://github.com/Expensify/App/blob/c144663fcad41a69344311a27dd6a147a28e7191/src/components/ReportTransaction.js#L59-L68
should be changed like below:
onClose={() => {
    if (this.props.action.actionName === CONST.REPORT.ACTIONS.TYPE.IOU) {
        ReportActions.clearSendMoneyErrors(this.props.chatReportID, this.props.action.reportActionID);
        ReportActions.deleteOptimisticReportAction(this.props.chatReportID, this.props.action.reportActionID);
    }
    if (this.props.action.pendingAction === CONST.RED_BRICK_ROAD_PENDING_ACTION.ADD) {
        ReportActions.deleteOptimisticReportAction(this.props.chatReportID, this.props.action.reportActionID);
    } else {
        ReportActions.clearReportActionErrors(this.props.chatReportID, this.props.action.reportActionID);
    }
}}

We need to change it only here, because after we implement the next point we will be able to dismiss the error only in the details modal.

To apply opacity to the IOU Preview message that has errors (in the chat and in the details modal) and change the way how it's presented (red dot, no error message) we need to slightly change the implementation of the OfflineWithFeedback component.

First, we need to add two new optional props:
const propTypes = {
    /** The name of the action */
    actionName: PropTypes.string,

    ...

    /** Whether we should show only the error indicator (overrides the `shouldShowErrorMessages`) */
    shouldShowErrorIndicatorOnly: PropTypes.bool,
 }

 const defaultProps = {
    actionName: null,
    ...
    shouldShowErrorIndicatorOnly: false,
};

Knowing the actionName we can find out if the error is related to the IOU Report or not, and use it to set the needsOpacity:
const isIOUError = props.actionName === CONST.REPORT.ACTIONS.TYPE.IOU && !!props.errors;
const needsOpacity = (isOfflinePendingAction && !isUpdateOrDeleteError) || isAddError || isIOUError;

To show the red dot and remove error message we need to change the JSX based on newly passed prop shouldShowErrorIndicatorOnly (wrap it with additional View, show error message with red dot only when this prop is not true and if it is true show separate red dot, that is outside the current view to not apply opacity on it):
<View>
    <View style={props.style}>
        {!hideChildren && (
            ...
        )}
        {(!props.shouldShowErrorIndicatorOnly && props.shouldShowErrorMessages && hasErrors) && (
            ... // currently existing code that shows error message
        )}
    </View>
    {props.shouldShowErrorIndicatorOnly && hasErrors && (
        <View style={{position: 'absolute', top: 30, right: 40}}>
            <Icon src={Expensicons.DotIndicator} fill={colors.red} height={variables.iconSizeSmall} width={variables.iconSizeSmall} />
        </View>
    )}
</View>

Now, we need to set the actionName prop of the OfflineWithFeedback components used in both ReportTransaction and ReportActionItem components.
actionName={this.props.action.actionName}

Additionally, we need to set the shouldShowErrorIndicatorOnly prop of the OfflineWithFeedback component used in ReportActionItem component.
Screen recording of how it works after the above changes:
https://user-images.githubusercontent.com/7771498/233284749-1cd74db5-2814-465a-95da-75310cd15b57.mp4
Reminder: Please use plain English, be brief and avoid jargon. Feel free to use images, charts or pseudo-code if necessary. Do not post large multi-line diffs or write walls of text. Do not create PRs unless you have been hired for this job.

@burczu Just a little suggestion

For the opacity issue, we can apply the solution from my proposal here https://github.com/Expensify/App/issues/16775#issuecomment-1497928130.

taken from the link above

We shouldn't clear the pending action immediately when the request fails. (remove the pendingAction: null from failureData)

this will make the behavior consistent with other add action explained in 2nd part of my comment here https://github.com/Expensify/App/issues/16775#issuecomment-1498587658
With that solution, we can completely remove your 3rd solution and isIOUError from the 4th solution.

Thanks @burczu for your very detailed explanation and solution. While we continue discussing @bernhardoj idea above, here're some of my wrap up, this issues has two problems to tackle:

The IOU Preview is duplicated
The error UI is not correct (i.e. doesn't match with the latest expected behavior)

The first problem was resolved in the first point of @burczu proposal, taken from @bernhardoj's idea.
On the other hand, the second problem was resolved in the 2, 3, 4 points, also specified in mine: https://github.com/Expensify/App/issues/15297#issuecomment-1482424281.

Thanks @burczu for the detailed proposal :)
@aimane-chnaif can I get your opinion here: https://github.com/Expensify/App/issues/15297#issuecomment-1515806608 🙏

Sounds good to me @aimane-chnaif , thanks.
@burczu please start working on the PR.

I think we should hold this issue on manual requests since this view will be completely redone.

@burczu there is some overlap with the issue this PR is solving.
Also, the details view we are seeing is getting redone by "manual request". I agree with @luacmartins about putting a HOLD until that is implemented.

@aldo-expensify thanks for the information - please keep me posted when it's possible to get back to this issue.

Moving this to Weekly since it is on HOLD

Still on HOLD

Still on HOLD

@aldo-expensify can you elaborate on why this is still held? The PR referenced has been deployed.

True, the Manual Request project was redoing some interfaces and a lot of the code related to IOUs, including cancelling/deleting.
I think we should revisit now this and see if it is still an issue we can reproduce.

Removing the HOLD to investigate where are we at now.

Sounds good! :)

OK, I'm trying to reproduce this using these steps:

Send request money from user B to A(MAIN)
Turn off Internet in device A pay the request
Cancel IOU in user B
Turn internet on in Device A -red dot error message occurs in request
Request the same amount from user B to A

After paying the IOU in account A while offline in step 2, I'm not able to Delete this request from account B:
https://github.com/Expensify/App/assets/38236554/7e8cf876-706f-4e39-8bb8-eb0af1cc3537
@kbecciv can you confirm you see the same behaviour?

@bfitzexpensify There is KI https://github.com/Expensify/App/issues/19066 that user can not delete request amount

Thanks @luacmartins , so back to HOLD I guess?

Yup

Still held on https://github.com/Expensify/App/issues/19066, but nice to see we've identified a solution there so things are moving

Still held

Still held

https://github.com/Expensify/App/issues/19066 still chugging along, and this is still held on that

Same ^

All PRs needed for https://github.com/Expensify/App/issues/19066 are in review, but not finalized yet - but we're closer to being able to revisit this

https://github.com/Expensify/App/issues/19066 still not ready

Still held on #19066

OK, now when I try to pay when offline, it's not possible:

Given the issue here is duplication after paying while offline, I think we can go ahead and close this since that flow isn't possible anymore


ISS-177 postgres failed with segmentation fault on pg_crc32c_sse42.c

postgres failed with segmentation fault on pg_crc32c_sse42.c
2024-08-24 10:57:18.823 CEST [621555] DEBUG:  transaction ID wrap limit is<PHONE_NUMBER>, limited by database with OID 1
====read ptr->value : 536870912
2024-08-24 10:57:18.824 CEST [621555] DEBUG:  MultiXactId wrap limit is<PHONE_NUMBER>, limited by database with OID 1
====read ptr->value : 536870912
2024-08-24 10:57:18.824 CEST [621555] DEBUG:  creating and filling new WAL file
====read ptr->value : 536870912
2024-08-24 10:57:18.841 CEST [621555] DEBUG:  done creating and filling new WAL file
Thread 9.1 "postgres" received signal SIGSEGV, Segmentation fault.
0x0000000001c5eea4 in pg_comp_crc32c_sse42 () at pg_crc32c_sse42.c:38
38                      crc = (uint32) _mm_crc32_u64(crc, *((const uint64 *) p));

will be partially fix by 1.0.23 but need to finish the vector management (for now it's a temp solution only to be able to compile postgres)


Jest failed to parse a file

Describe the bug
When running a basic jest test on the component that  import { FCM } from '@capacitor-community/fcm'; I get the following error
Jest encountered an unexpected token

    Jest failed to parse a file. This happens e.g. when your code or its dependencies use non-standard JavaScript syntax, or when Jest is not configured to support such syntax.

    Out of the box Jest supports Babel, which will be used to transform your files into valid JS based on your Babel configuration.

    By default "node_modules" folder is ignored by transformers.

    Here's what you can do:
     • If you are trying to use ECMAScript Modules, see https://jestjs.io/docs/ecmascript-modules for how to enable it.
     • To have some of your "node_modules" files transformed, you can specify a custom "transformIgnorePatterns" in your config.
     • If you need a custom transformation specify a "transform" option in your config.
     • If you simply want to mock your non-JS modules (e.g. binary assets) you can stub them out with the "moduleNameMapper" config option.

    You'll find more details and examples of these config options in the docs:
    https://jestjs.io/docs/configuration
    For information about custom transformations, see:
    https://jestjs.io/docs/code-transformation

    Details:

    /home/paddy/sites/app/client/node_modules/@capacitor-community/fcm/dist/esm/index.js:1
    ({"Object.<anonymous>":function(module,exports,require,__dirname,__filename,jest){import { registerPlugin } from "@capacitor/core";
                                                                                ^^^^^^

    SyntaxError: Cannot use import statement outside a module

Additional context
Ionic:
Ionic Core : 5.6.11
Ionic Framework : @ionic/react 5.6.11
Capacitor:
Capacitor CLI : 3.1.1
@capacitor/core : 3.1.1
System:
NodeJS : v14.18.1
npm : 6.14.15
I can see a similar issue related to ESM file parsing here https://github.com/capacitor-community/http/issues/29
But the suggestion https://github.com/capacitor-community/http/issues/29#issuecomment-654934290
to update the transformIgnorePatterns doesn't seem to be working here
jest.config
module.exports = {
...
  transformIgnorePatterns: ['node_modules/(?!capacitor-community).+\\.js$'],
...
};

Same here
Any workarounds?


Issue in uploading files

Versions are as follows
meteor 1.8.3
<EMAIL_ADDRESS>If i use " transport: 'http' " network tab response return "JavaScript is required to use this app" and if use " transport: 'ddp' " no network tab response but console error "Error invoking Method '_FilesCollectionAbort_Images'", anyone please help
--------------------------server code----------------------
import { Meteor }   from 'meteor/meteor';
import { FilesCollection } from 'meteor/ostrio:files';
const Images = new FilesCollection({
debug: true,
collectionName: 'Images',
storagePath : '/public',
allowClientCode: false, // Disallow remove files from Client
onBeforeUpload: function (file) {
console.log("fileRef before",  file)
return true;
},
onAfterUpload(fileRef) {
}
});
export default Images;
------------------------------------server -ends------------------------------
--------------client ----------------------------------
var logo = event.target.files[0];
const upload = Images.insert({
  file: logo,
  streams: 'dynamic',
  chunkSize: 'dynamic',
  transport: 'http',
}, false);

upload.on('start', function() {
console.log('Upload started');
});
upload.on('end', function (error, fileObj) {
if (error) {
console.log('Error during upload: ' + error);
} else {
console.log('File "' + fileObj.name + '" successfully uploaded');
}
});
upload.start();

Images not getting upload at all, removing the line  '''storagePath : '/public',''' still same issue as stated above.

Hello @eramitpatel ,

You need to initiate const Images = new FilesCollection({/*...*/}) on the Client as well
What browser you're on? Version?
What OS you're on? Version?
You have debug enabled, any debug logs to share?
Message about JavaScript is required to use this app do you have JS enabled in a browser?

solved with your given instructions

@eramitpatel I'm glad it's solved so quickly
Please support this project with:

Star this package on GitHub;
Like at Facebook;
Follow on Twitter;
Support via patreon;
Support via PayPal;
Check and star our other open source projects on GitHub.


New patch release of Golang: v1.20.10

Update EKS Go Patch Version: v1.20.10-2
SPEC FILE STILL NEEDS THE '%changelog' UPDATED
PLEASE UPDATE WITH THE FOLLOWING FORMAT
* Wed Sep 06 2023 Cameron Rozean<EMAIL_ADDRESS>- 1.20.8-1
- Bump tracking patch version to 1.20.8 from 1.20.7

/lgtm

/approve


hipconfig doesn't work out of the box. missing some package dependency

# /opt/rocm-3.5.0/hip/bin/hipconfig
HIP version  : 3.5.20214-a2917cd

== hipconfig
HIP_PATH     : /opt/rocm-3.5.0/hip
ROCM_PATH    : /opt/rocm
HIP_COMPILER : hcc
HIP_PLATFORM : hcc
HIP_RUNTIME  : HCC
CPP_CONFIG   :  -D__HIP_PLATFORM_HCC__=  -I/opt/rocm-3.5.0/hip/include -I/opt/rocm/hcc/include -I/opt/rocm/hsa/include

== hcc
HSA_PATH     : /opt/rocm/hsa
HCC_HOME     : /opt/rocm/hcc
Can't exec "/opt/rocm/hcc/bin/hcc": No such file or directory at /opt/rocm-3.5.0/hip/bin/hipconfig line 216.
Can't exec "/opt/rocm/hcc/bin/llc": No such file or directory at /opt/rocm-3.5.0/hip/bin/hipconfig line 217.
HCC-cxxflags : Can't exec "/opt/rocm/hcc/bin/hcc-config": No such file or directory at /opt/rocm-3.5.0/hip/bin/hipconfig line 219.

HCC-ldflags  : Can't exec "/opt/rocm/hcc/bin/hcc-config": No such file or directory at /opt/rocm-3.5.0/hip/bin/hipconfig line 222.

It also incorrectly reports some paths with /opt/rocm/. Such directory doesn't exist.

Still the same issue in 3.7

Maybe you should try remove rocm 3.5.0 and reinstall rocm 3.7.0.
I can run hipconfig successly on rocm-3.7.0.
$ /opt/rocm/bin/hipconfig
HIP version  : 3.7.20315-077bcfa0

== hipconfig
HIP_PATH     : /opt/rocm-3.7.0/hip
ROCM_PATH    : /opt/rocm
HIP_COMPILER : clang
HIP_PLATFORM : hcc
HIP_RUNTIME  : ROCclr
CPP_CONFIG   :  -D__HIP_PLATFORM_HCC__=  -I/opt/rocm-3.7.0/hip/include -I/opt/rocm/llvm/bin/../lib/clang/11.0.0 -I/opt/rocm/hsa/include -D__HIP_ROCclr__

== hip-clang
HSA_PATH         : /opt/rocm/hsa
HIP_CLANG_PATH   : /opt/rocm/llvm/bin
clang version 11.0.0 (/src/external/llvm-project/clang ee4e4ebbadcc8ea14ce99e34ed31ab31e94827ac)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm/llvm/bin
LLVM (http://llvm.org/):
  LLVM version 11.0.0git
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: haswell

  Registered Targets:
    amdgcn - AMD GCN GPUs
    r600   - AMD GPUs HD2XXX-HD6XXX
    x86    - 32-bit X86: Pentium-Pro and above
    x86-64 - 64-bit X86: EM64T and AMD64
hip-clang-cxxflags : -D__HIP_ROCclr__ -std=c++11 -isystem /opt/rocm-3.7.0/llvm/lib/clang/11.0.0/include/.. -isystem /opt/rocm/hsa/include -D__HIP_ROCclr__ -isystem /opt/rocm-3.7.0/hip/include -D__HIP_ARCH_GFX803__=1  -O3
hip-clang-ldflags  :

 -L/opt/rocm-3.7.0/hip/lib -O3 -lgcc_s -lgcc -lpthread -lm

=== Environment Variables
PATH=/home/work/anaconda3/bin:/home/work/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

== Linux Kernel
Hostname     : 31d2ed49d9f3
Linux 31d2ed49d9f3 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

I never had 3.5 installed on this machine before. I never had any rocm / hip stuff on this machine. It is freshly installed OS.

The issue is because hipconfig has wrong dependencies. The package it is in should depend on llvm-amdgpu, but it isn't.

Still broken in ROCm 3.8:
root@debian:~# /opt/rocm-3.8.0/bin/hipconfig
Can't exec "/opt/rocm/llvm/bin/clang++": No such file or directory at /opt/rocm-3.8.0/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-3.8.0/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/bin/hipconfig line 145.
HIP version  : 3.8.20371-d1886b0b

== hipconfig
HIP_PATH     : /opt/rocm-3.8.0/hip
ROCM_PATH    : /opt/rocm
HIP_COMPILER : clang
HIP_PLATFORM : hcc
HIP_RUNTIME  : ROCclr
CPP_CONFIG   :  -D__HIP_PLATFORM_HCC__=  -I/opt/rocm-3.8.0/hip/include -I/opt/rocm/llvm/bin/../lib/clang/ -I/opt/rocm/hsa/include -D__HIP_ROCclr__

== hip-clang
HSA_PATH         : /opt/rocm/hsa
HIP_CLANG_PATH   : /opt/rocm/llvm/bin
Can't exec "/opt/rocm/llvm/bin/clang++": No such file or directory at /opt/rocm-3.8.0/bin/hipconfig line 236.
Can't exec "/opt/rocm/llvm/bin/llc": No such file or directory at /opt/rocm-3.8.0/bin/hipconfig line 237.
hip-clang-cxxflags : Can't exec "/opt/rocm/llvm/bin/clang++": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-3.8.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm/llvm/bin/clang++": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-3.8.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm/llvm/bin/clang++": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-3.8.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm/llvm/bin/clang++": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-3.8.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm/llvm/bin/clang": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipcc line 203.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-3.8.0/hip/bin/hipcc line 204.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipcc line 208.
Use of uninitialized value $HIP_CLANG_INCLUDE_PATH in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipcc line 233.
Use of uninitialized value $HIP_CLANG_INCLUDE_PATH in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipcc line 234.
Can't exec "/opt/rocm/bin/rocm_agent_enumerator": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipcc line 724.
Use of uninitialized value $targetsStr in substitution (s///) at /opt/rocm-3.8.0/hip/bin/hipcc line 725.
Use of uninitialized value $targetsStr in split at /opt/rocm-3.8.0/hip/bin/hipcc line 731.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipcc line 848.
-D__HIP_ROCclr__ -std=c++11 -isystem /.. -isystem /opt/rocm/hsa/include -D__HIP_ROCclr__ -isystem /opt/rocm-3.8.0/hip/include -O3
hip-clang-ldflags  : Can't exec "/opt/rocm/llvm/bin/clang++": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-3.8.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm/llvm/bin/clang++": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-3.8.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm/llvm/bin/clang++": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-3.8.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm/llvm/bin/clang++": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-3.8.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm/llvm/bin/clang": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipcc line 203.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-3.8.0/hip/bin/hipcc line 204.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipcc line 208.
Use of uninitialized value $HIP_CLANG_INCLUDE_PATH in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipcc line 233.
Use of uninitialized value $HIP_CLANG_INCLUDE_PATH in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipcc line 234.
Can't exec "/opt/rocm/bin/rocm_agent_enumerator": No such file or directory at /opt/rocm-3.8.0/hip/bin/hipcc line 724.
Use of uninitialized value $targetsStr in substitution (s///) at /opt/rocm-3.8.0/hip/bin/hipcc line 725.
Use of uninitialized value $targetsStr in split at /opt/rocm-3.8.0/hip/bin/hipcc line 731.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-3.8.0/hip/bin/hipcc line 848.
--driver-mode=g++ -L/opt/rocm-3.8.0/hip/lib -O3 -lgcc_s -lgcc -lpthread -lm

=== Environment Variables
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

== Linux Kernel
Hostname     : debian
Linux debian 5.7.0-1-amd64 #1 SMP Debian 5.7.6-1 (2020-06-24) x86_64 GNU/Linux
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	unstable
Codename:	sid

rocminfo, clinfo do all work just fine.

Same issue here.

Hi @baryluk and others, thanks for reaching out.
Mentioned issue is no more observed with ROCm 4.0.
taccuser@taccuser-X399-DESIGNARE-EX:/opt/rocm-4.0.0$ /opt/rocm-4.0.0/hip/bin/hipconfig
HIP version  : 4.0.20496-4f163c68
== hipconfig
HIP_PATH     : /opt/rocm-4.0.0/hip
ROCM_PATH    : /opt/rocm-4.0.0
HIP_COMPILER : clang
HIP_PLATFORM : hcc
HIP_RUNTIME  : ROCclr
CPP_CONFIG   :  -D__HIP_PLATFORM_HCC__=  -I/opt/rocm-4.0.0/hip/include -I/opt/rocm-4.0.0/llvm/bin/../lib/clang/12.0.0 -I/opt/rocm-4.0.0/hsa/include -D__HIP_ROCclr__
== hip-clang
HSA_PATH         : /opt/rocm-4.0.0/hsa
HIP_CLANG_PATH   : /opt/rocm-4.0.0/llvm/bin
clang version 12.0.0 (/src/external/llvm-project/clang dac2bfceaa8d4a90257dc8a6d58f268e172ce00e)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-4.0.0/llvm/bin
LLVM (http://llvm.org/):
LLVM version 12.0.0git
Optimized build.
Default target: x86_64-unknown-linux-gnu
Host CPU: znver1
Registered Targets:
amdgcn - AMD GCN GPUs
r600   - AMD GPUs HD2XXX-HD6XXX
x86    - 32-bit X86: Pentium-Pro and above
x86-64 - 64-bit X86: EM64T and AMD64
hip-clang-cxxflags : -D__HIP_ROCclr__ -std=c++11 -isystem /opt/rocm-4.0.0/llvm/lib/clang/12.0.0/include/.. -isystem /opt/rocm-4.0.0/hsa/include -D__HIP_ROCclr__ -isystem /opt/rocm-4.0.0/hip/include -O3
hip-clang-ldflags  :  -L/opt/rocm-4.0.0/hip/lib -O3 -lgcc_s -lgcc -lpthread -lm
=== Environment Variables
PATH=/home/taccuser/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
== Linux Kernel
Hostname     : taccuser-X399-DESIGNARE-EX
Linux taccuser-X399-DESIGNARE-EX 5.4.0-59-generic #65-Ubuntu SMP Thu Dec 10 12:01:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal
Request to open issues, for any new.
Thank you.

Hi @baryluk and others, thanks for reaching out.
Mentioned issue is no more observed with ROCm 4.0.
taccuser@taccuser-X399-DESIGNARE-EX:/opt/rocm-4.0.0$ /opt/rocm-4.0.0/hip/bin/hipconfig
HIP version  : 4.0.20496-4f163c68
== hipconfig
HIP_PATH     : /opt/rocm-4.0.0/hip
ROCM_PATH    : /opt/rocm-4.0.0
HIP_COMPILER : clang
HIP_PLATFORM : hcc
HIP_RUNTIME  : ROCclr
CPP_CONFIG   :  -D__HIP_PLATFORM_HCC__=  -I/opt/rocm-4.0.0/hip/include -I/opt/rocm-4.0.0/llvm/bin/../lib/clang/12.0.0 -I/opt/rocm-4.0.0/hsa/include -D__HIP_ROCclr__
== hip-clang
HSA_PATH         : /opt/rocm-4.0.0/hsa
HIP_CLANG_PATH   : /opt/rocm-4.0.0/llvm/bin
clang version 12.0.0 (/src/external/llvm-project/clang dac2bfceaa8d4a90257dc8a6d58f268e172ce00e)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/rocm-4.0.0/llvm/bin
LLVM (http://llvm.org/):
LLVM version 12.0.0git
Optimized build.
Default target: x86_64-unknown-linux-gnu
Host CPU: znver1
Registered Targets:
amdgcn - AMD GCN GPUs
r600   - AMD GPUs HD2XXX-HD6XXX
x86    - 32-bit X86: Pentium-Pro and above
x86-64 - 64-bit X86: EM64T and AMD64
hip-clang-cxxflags : -D__HIP_ROCclr__ -std=c++11 -isystem /opt/rocm-4.0.0/llvm/lib/clang/12.0.0/include/.. -isystem /opt/rocm-4.0.0/hsa/include -D__HIP_ROCclr__ -isystem /opt/rocm-4.0.0/hip/include -O3
hip-clang-ldflags  :  -L/opt/rocm-4.0.0/hip/lib -O3 -lgcc_s -lgcc -lpthread -lm
=== Environment Variables
PATH=/home/taccuser/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
== Linux Kernel
Hostname     : taccuser-X399-DESIGNARE-EX
Linux taccuser-X399-DESIGNARE-EX 5.4.0-59-generic #65-Ubuntu SMP Thu Dec 10 12:01:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal
Request to open issues, for any new.
Thank you.

@ROCmSupport Issue is still in 4.0.
Can't exec "/opt/rocm-4.0.0/hcc/bin/llc": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 223.
HCC-cxxflags : Can't exec "/opt/rocm-4.0.0/hcc/bin/hcc-config": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 225.

HCC-ldflags  : Can't exec "/opt/rocm-4.0.0/hcc/bin/hcc-config": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 228.

=== Environment Variables
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

== Linux Kernel
Hostname     : debian
Linux debian 5.9.0-4-amd64 #1 SMP Debian 5.9.11-1 (2020-11-27) x86_64 GNU/Linux
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	unstable
Codename:	sid

@ROCmSupport Issue is still in 4.0.
Can't exec "/opt/rocm-4.0.0/hcc/bin/llc": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 223.
HCC-cxxflags : Can't exec "/opt/rocm-4.0.0/hcc/bin/hcc-config": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 225.

HCC-ldflags  : Can't exec "/opt/rocm-4.0.0/hcc/bin/hcc-config": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 228.

=== Environment Variables
PATH=/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

== Linux Kernel
Hostname     : debian
Linux debian 5.9.0-4-amd64 #1 SMP Debian 5.9.11-1 (2020-11-27) x86_64 GNU/Linux
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	unstable
Codename:	sid

Hi @baryluk
In your machine, HIP_COMPILER is showing as hcc but it should be clang instead.
I tried on a clean system and it always says clang. So I request you to try on a clean or a different system.
No need to worry about HIP_PLATFORM=hcc in both of our cases as no problem with this for now, but slowly it will also be changes as per the information shared by HIP/compiler team.
Thank you.

Hi @baryluk
In your machine, HIP_COMPILER is showing as hcc but it should be clang instead.
I tried on a clean system and it always says clang. So I request you to try on a clean or a different system.
No need to worry about HIP_PLATFORM=hcc in both of our cases as no problem with this for now, but slowly it will also be changes as per the information shared by HIP/compiler team.
Thank you.

@ROCmSupport That test was on a clean computer, with no ROCm ever installed.
Here is an example on fresh system:
# apt install hip-base4.0.0
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  hip-base4.0.0
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 250 kB of archives.
After this operation, 2,239 kB of additional disk space will be used.
Get:1 https://repo.radeon.com/rocm/apt/debian xenial/main amd64 hip-base4.0.0 amd64 4.0.20496.5685.40000-23 [250 kB]
Fetched 250 kB in 1s (321 kB/s)
Selecting previously unselected package hip-base4.0.0.
(Reading database ... 753192 files and directories currently installed.)
Preparing to unpack .../hip-base4.0.0_4.0.20496.5685.40000-23_amd64.deb ...
Unpacking hip-base4.0.0 (4.0.20496.5685.40000-23) ...
Setting up hip-base4.0.0 (4.0.20496.5685.40000-23) ...

# /opt/rocm-4.0.0/bin/hipconfig
HIP version  : 4.0.20496-4f163c68

== hipconfig
HIP_PATH     : /opt/rocm-4.0.0/hip
ROCM_PATH    : /opt/rocm
HIP_COMPILER : hcc
HIP_PLATFORM : hcc
HIP_RUNTIME  : HCC
CPP_CONFIG   :  -D__HIP_PLATFORM_HCC__=  -I/opt/rocm-4.0.0/hip/include -I/opt/rocm/hcc/include -I/opt/rocm/hsa/include

== hcc
HSA_PATH     : /opt/rocm/hsa
HCC_HOME     : /opt/rocm/hcc
Can't exec "/opt/rocm/hcc/bin/hcc": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 222.
Can't exec "/opt/rocm/hcc/bin/llc": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 223.
HCC-cxxflags : Can't exec "/opt/rocm/hcc/bin/hcc-config": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 225.

HCC-ldflags  : Can't exec "/opt/rocm/hcc/bin/hcc-config": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 228.

=== Environment Variables
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

== Linux Kernel
Hostname     : debian
Linux debian 5.10.0-1-amd64 #1 SMP Debian 5.10.4-1 (2020-12-31) x86_64 GNU/Linux
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	unstable
Codename:	sid
#

The dependencies are just wrong / missing on hip-base.

@ROCmSupport That test was on a clean computer, with no ROCm ever installed.
Here is an example on fresh system:
# apt install hip-base4.0.0
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  hip-base4.0.0
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 250 kB of archives.
After this operation, 2,239 kB of additional disk space will be used.
Get:1 https://repo.radeon.com/rocm/apt/debian xenial/main amd64 hip-base4.0.0 amd64 4.0.20496.5685.40000-23 [250 kB]
Fetched 250 kB in 1s (321 kB/s)
Selecting previously unselected package hip-base4.0.0.
(Reading database ... 753192 files and directories currently installed.)
Preparing to unpack .../hip-base4.0.0_4.0.20496.5685.40000-23_amd64.deb ...
Unpacking hip-base4.0.0 (4.0.20496.5685.40000-23) ...
Setting up hip-base4.0.0 (4.0.20496.5685.40000-23) ...

# /opt/rocm-4.0.0/bin/hipconfig
HIP version  : 4.0.20496-4f163c68

== hipconfig
HIP_PATH     : /opt/rocm-4.0.0/hip
ROCM_PATH    : /opt/rocm
HIP_COMPILER : hcc
HIP_PLATFORM : hcc
HIP_RUNTIME  : HCC
CPP_CONFIG   :  -D__HIP_PLATFORM_HCC__=  -I/opt/rocm-4.0.0/hip/include -I/opt/rocm/hcc/include -I/opt/rocm/hsa/include

== hcc
HSA_PATH     : /opt/rocm/hsa
HCC_HOME     : /opt/rocm/hcc
Can't exec "/opt/rocm/hcc/bin/hcc": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 222.
Can't exec "/opt/rocm/hcc/bin/llc": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 223.
HCC-cxxflags : Can't exec "/opt/rocm/hcc/bin/hcc-config": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 225.

HCC-ldflags  : Can't exec "/opt/rocm/hcc/bin/hcc-config": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 228.

=== Environment Variables
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

== Linux Kernel
Hostname     : debian
Linux debian 5.10.0-1-amd64 #1 SMP Debian 5.10.4-1 (2020-12-31) x86_64 GNU/Linux
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux bullseye/sid
Release:	unstable
Codename:	sid
#

The dependencies are just wrong / missing on hip-base.

`  lambda-tensorflow-benchmark git:(master) ✗ /opt/rocm-4.0.0/bin/hipconfig
Can't exec "/opt/rocm-4.0.0/llvm/bin/clang++": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-4.0.0/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/bin/hipconfig line 145.
HIP version  : 4.0.20496-4f163c68
== hipconfig
HIP_PATH     : /opt/rocm-4.0.0/hip
ROCM_PATH    : /opt/rocm-4.0.0
HIP_COMPILER : clang
HIP_PLATFORM : hcc
HIP_RUNTIME  : ROCclr
CPP_CONFIG   :  -D__HIP_PLATFORM_HCC__=  -I/opt/rocm-4.0.0/hip/include -I/opt/rocm-4.0.0/llvm/bin/../lib/clang/ -I/opt/rocm-4.0.0/hsa/include -D__HIP_ROCclr__
== hip-clang
HSA_PATH         : /opt/rocm-4.0.0/hsa
HIP_CLANG_PATH   : /opt/rocm-4.0.0/llvm/bin
Can't exec "/opt/rocm-4.0.0/llvm/bin/clang++": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 236.
Can't exec "/opt/rocm-4.0.0/llvm/bin/llc": No such file or directory at /opt/rocm-4.0.0/bin/hipconfig line 237.
hip-clang-cxxflags : Can't exec "/opt/rocm-4.0.0/llvm/bin/clang++": No such file or directory at /opt/rocm-4.0.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-4.0.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm-4.0.0/llvm/bin/clang++": No such file or directory at /opt/rocm-4.0.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-4.0.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm-4.0.0/llvm/bin/clang++": No such file or directory at /opt/rocm-4.0.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-4.0.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm-4.0.0/llvm/bin/clang++": No such file or directory at /opt/rocm-4.0.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-4.0.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm-4.0.0/llvm/bin/clang": No such file or directory at /opt/rocm-4.0.0/hip/bin/hipcc line 203.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-4.0.0/hip/bin/hipcc line 204.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipcc line 208.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipcc line 846.
-D__HIP_ROCclr__ -std=c++11 -isystem /opt/rocm-4.0.0/llvm/lib/clang/include/.. -isystem /opt/rocm-4.0.0/hsa/include -D__HIP_ROCclr__ -isystem /opt/rocm-4.0.0/hip/include -O3
hip-clang-ldflags  : Can't exec "/opt/rocm-4.0.0/llvm/bin/clang++": No such file or directory at /opt/rocm-4.0.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-4.0.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm-4.0.0/llvm/bin/clang++": No such file or directory at /opt/rocm-4.0.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-4.0.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm-4.0.0/llvm/bin/clang++": No such file or directory at /opt/rocm-4.0.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-4.0.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm-4.0.0/llvm/bin/clang++": No such file or directory at /opt/rocm-4.0.0/hip/bin/hipconfig line 141.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-4.0.0/hip/bin/hipconfig line 142.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipconfig line 145.
Can't exec "/opt/rocm-4.0.0/llvm/bin/clang": No such file or directory at /opt/rocm-4.0.0/hip/bin/hipcc line 203.
Use of uninitialized value $HIP_CLANG_VERSION in pattern match (m//) at /opt/rocm-4.0.0/hip/bin/hipcc line 204.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipcc line 208.
Use of uninitialized value $HIP_CLANG_VERSION in concatenation (.) or string at /opt/rocm-4.0.0/hip/bin/hipcc line 846.
--driver-mode=g++ -L/opt/rocm-4.0.0/hip/lib -O3 -lgcc_s -lgcc -lpthread -lm
=== Environment Variables
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/rocm/bin:/opt/rocm/rocprofiler/bin:/opt/rocm/opencl/bin
== Linux Kernel
Hostname     : pc
Linux pc 5.4.0-65-generic #73-Ubuntu SMP Mon Jan 18 17:25:17 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal
`
The dependencies are just wrong

Hi all, as the issue is already in closed state, request to open a new issue for better progress.
Thank you.

@ROCmSupport Just reopen this issue please.

Hi @baryluk
This issue does not occur when we install complete rocm as rocm-dkms or rocm-dev(versioned/unversioned).
This issue occurs only when you install hip-base package alone, which is not a recommend way of running hipconfig from there.
Because hip-base needs hip-rocclr as dependency so its not a real issue.
Though its not a real issue, I can ask for possible fix, if any, so request you to log a separate ticket to track this.
Reg this ticket, as the initial issue is taken care, we have closed this ticket.

Filled new issue: https://github.com/RadeonOpenCompute/ROCm/issues/1382

Thanks @baryluk
I will take it forward, thank you.


feat: add Batcher#close(timeout) and Batcher#cancelOutstanding

There have been reports of batcher.close() hanging every once in awhile. Currently it is impossible to debug because we dont expose any internal state to analyze.
This PR adds 2 additional methods that should help in diagnosing issues:

close(timeout) will try to close the batcher, but if any of the underlying batch operations fail, the exception message will contain a wealth of information describing the underlying state of operations as provided by #3140
cancelOutstanding this allows for remediation for close(timeout) throwing an exception.

The intended usecase is dataflow connector's FinishBundle:
try {
  batcher.close(Duration.ofMinutes(1));
} catch(BatchingException e) {
   batcher.cancelOutstanding();
  batcher.close(Duration.ofMinutes(1));
}

Example exception message:

Exception in thread "main" com.google.api.gax.batching.BatchingException: Timed out trying to close batcher after PT1S. Batch request prototype: com.google.cloud.bigtable.data.v2.models.BulkMutation@2bac9ba. Outstanding batches: Batch{operation=CallbackChainRetryingFuture{super=null, latestCompletedAttemptResult=ImmediateFailedFuture@6a9d5dff[status=FAILURE, cause=[com.google.cloud.bigtable.data.v2.models.MutateRowsException: Some mutations failed to apply]], attemptResult=null, attemptSettings=TimedAttemptSettings{globalSettings=RetrySettings{totalTimeout=PT10M, initialRetryDelay=PT0.01S, retryDelayMultiplier=2.0, maxRetryDelay=PT1M, maxAttempts=0, jittered=true, initialRpcTimeout=PT1M, rpcTimeoutMultiplier=1.0, maxRpcTimeout=PT1M}, retryDelay=PT1.28S, rpcTimeout=PT1M, randomizedRetryDelay=PT0.877S, attemptCount=8, overallAttemptCount=8, firstAttemptStartTimeNanos=646922035424541}}, elements=com.google.cloud.bigtable.data.v2.models.RowMutationEntry@7a344b65}

Thank you for opening a Pull Request! Before submitting your PR, please read our contributing guidelines.
There are a few things you can do to make sure it goes smoothly:

[ ] Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
[ ] Ensure the tests and linter pass
[ ] Code coverage does not decrease (if any source code was changed)
[ ] Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕

api compatibility is failing with:

Run REPOS_UNDER_TEST="java-datastore" ./.kokoro/presubmit/downstream-compatibility.sh
Setup maven mirror
Installing this repo's modules to local maven.
Error:  7012: com.google.api.gax.batching.Batcher: Method 'public void cancelOutstanding()' has been added to an interface
Error:  7012: com.google.api.gax.batching.Batcher: Method 'public void close(org.threeten.bp.Duration)' has been added to an interface
Error:  Failed to execute goal org.codehaus.mojo:clirr-maven-plugin:2.8:check (default) on project gax: There were 2 errors. -> [Help 1]

But it is wrong because the Batcher interface is marked as @InternalExtensionOnly so adding new methods should be allowed. I tried ignoring it using a clirr ignore, but that doesnt seem to work. Whats the correct way to get it to start passing?


Highlight problems with records

Adds a new feature to identify and highlight problems with records.
Currently manifests as:

New tile on homepage (non functional)
New badge Problem
New filter for Has problems
Banner shown on record

 How should we be flagging issues in trainee records?


Need to exclude build directory from sphinx build

Add this line in Makefile: EXCLUDENB     ?= -D exclude_patterns="notebooks/*","build","_build","**.ipynb_checkpoints"
Add this line in conf.py: exclude_patterns = ['build', '_build', '**.ipynb_checkpoints']
Otherwise we end up with the nested build directory thing again.

Actually, what is written above is incorrect. We need to make a modification to the pre-commit hook.
Currently it looks like this:
  - repo: local
    hooks:
      - id: sphinx-build
        name: Build documentation with Sphinx
        entry: sphinx-build
        language: system
        always_run: true
        exclude_types: [file, symlink]
        args:
          [
            "-M", # Run sphinx in make mode, so we can use -D flag later
                  # Note: -M requires next 3 args to be builder, source, output
            "html", # Specify builder
            "./docs", # Source directory of documents
            "./docs/build/html", # Output directory for rendered documents
            "-T", # Show full trace back on exception
            "-E", # Don't use saved env; always read all files
            "-d", # Flag for cached environment and doctrees
            "./docs/build/doctrees", # Directory
            "-D", # Flag to override settings in conf.py
            "exclude_patterns=notebooks/*", # Exclude our notebooks from pre-commit
          ]

We need to make it look like this. Note the _build:
  - repo: local
    hooks:
      - id: sphinx-build
        name: Build documentation with Sphinx
        entry: sphinx-build
        language: system
        always_run: true
        exclude_types: [file, symlink]
        args:
          [
            "-M", # Run sphinx in make mode, so we can use -D flag later
                  # Note: -M requires next 3 args to be builder, source, output
            "html", # Specify builder
            "./docs", # Source directory of documents
            "./docs/_build/html", # Output directory for rendered documents
            "-T", # Show full trace back on exception
            "-E", # Don't use saved env; always read all files
            "-d", # Flag for cached environment and doctrees
            "./docs/_build/doctrees", # Directory
            "-D", # Flag to override settings in conf.py
            "exclude_patterns=notebooks/*", # Exclude our notebooks from pre-commit
          ]


[PT Run] Observe clipboard & offer to open http[s]-URLs

📝 Provide a description of the new feature
What is the expected behavior of the proposed feature?  What is the scenario this would be used?
I find myself frequently juggling with URLs which get pasted to the clipboard and then inserted into some browser-window, so (just as a proof-of-concept) I wrote a little tool (independ of PT) which observes the clipboard and if it finds one, suggests to open - which I can then accept of neglect. I was unsure whether this would be practical of if I would get bombarded with prompts that I don't use. But after a few wks of using this, I still like the idea and would like to take it further - and that is beyond my level of knowledge, beyond the capabilities of the environment I worked on etc., and it also feels like PT Run would be a more obvious home for such functionality. Which is why I opened this issue.
I suggest that PT Run should observe the clipboard and if it finds a new entry commencing with http[s]:, it should display a dialog that offers a choice of browsers (either as "found on the system" or configured by the user) which could open this URL. The dialog also has a Cancel-Button. Ideally there should be a configurable timeout after which the dialog will close.
As a milder form (if the observer is disabled through a config-option), it would be good if when entering such URLs into PTs Search-Field there was a choice of browser to open the URL with.
If you'd like to see this feature implemented, add a 👍 reaction to this post.

@mbaas2
in what scenario do you copy a URL and then open it in the browser?
Personally I found that is very unlikely that a user will copy a URL and then open a browser and copy that URL in the browser.
Why going through all that when in 99% of the cases you just click or cntrl-click to open it?
When copying a URL is usually to share it through an app (email, chat, messaging etc.)

Hmm, I may want to open it in a different browser and  not change the default browser setting. Or it may come from an environment where Ctrl+Click doesn't open URLs. Or it has been opened in one browser but I want to look at it using another browser.

@mbaas2
it seems a very limited scenario to justify a pop up that will be interfering every time the URL is copied for other reasons.

Ok, maybe it's too specific - but as a developer, I need that several times a day.

@enricogior
maybe I'm wrong and it's more common that I think.

I hope some others will jump in to rescue the idea ;)

Like sometimes you need to copy an url (from a mail for example) and edit it before opening. So I like the idea

@Aaron-Junker
your scenario is not a match for this feature request, actually it's the opposite.

"observe clipboard" sounds like a privacy problem to me,if I'm sceptical

"observe clipboard" sounds like a privacy problem to me,if I'm sceptical

If it would be so a big privacy problem, MSFT would not allow to access the clipboard so easely.

Thanks for all the feedback. I was able to implement that idea in another tool, so I'll close this issue.


feat(sbom)!: overhaul SBOM generation logic

This is substantial overhaul of how Melange produces SPDX SBOMs for the APK packages it builds.
What's changing
Forging a path to better "build-time" SBOMs
The main change here is that the concept of an "SBOM" has been promoted to an explicitly managed entity, starting at the beginning of the build process and lasting until the end. Before, some SBOM-related data was tracked, but the connection to the ultimate SBOM wasn't apparent until the last minute.
This means we can now add more logic to the build process over time that contributes software composition data to the SBOM during any pipeline steps, such as to describe language ecosystem packages and relate them to our APKs using first-hand knowledge about the build.
Using distinct packages for distinct software entities
This also addresses an issue where we had multiple PURLs associated with a single package. The PURL values were helpful, but a more idiomatic SPDX representation tracks distinct concepts like the upstream source and the build configuration as their own packages. This lets us be more explicit about how each PURL maps to one of these concepts. We then add relationships to the SBOM to describe how these concepts interact with one another to produce the APK artifact.
Better API for SBOMs
There are now explicit types for packages and SBOM documents in the sbom package, which let consumers create and iterate on an SBOM throughout the build.
The build package has a new SBOMGroup type, whose purpose is to centralize interactions with the SBOM package to minimize the SBOM-related work in the build logic itself. This lets us create SBOMs tailored to the origin package or any subpackage, where some packages are specific to one SBOM, while other packages should be added to all SBOMs in the whole build group.
More explicit supply chain information
Rather than detect things like the current git commit as a side effect deep into the build steps, this information has been pushed out to the CLI entrypoint, where it can still be detected OR be explicitly specified using CLI flags. Then, the build itself is going off of explicit information and can operate more deterministically.
One such change is that we add a --license string flag, so that we can know which license to attach to the build configuration package — e.g. for Wolfi this would be Apache-2.0. I'm interested in folks' thoughts on the best way to wire this up for our builds. We could specify it in the distro's Makefile, attempt to autodetect it from the local filesystem, etc. Left unset, it defaults to NOASSERTION.
General cleanup
This PR also cleans up existing code, including to fix https://github.com/chainguard-dev/melange/issues/1460.
This PR also fixes an issue where the SPDX document namespace uses a different hash now, because the previous code didn't factor in epoch (which was wrong). This means that the SPDX document namespace will show a different hash with this PR merged than it did before, even with no changes to the package itself.
Follow-up work

Tests: While some SBOM tests exist already in GitHub Actions workflows (using YAML and bash), it'd be ideal to write tests in Go to ensure we're getting the expected data in our SBOMs.

Full example of SBOMs before and after

Click here!
Before
{
  "SPDXID": "SPDXRef-DOCUMENT",
  "name": "apk-crane-0.20.2-r0",
  "spdxVersion": "SPDX-2.3",
  "creationInfo": {
    "created": "2024-08-06T17:00:47Z",
    "creators": [
      "Tool: melange (devel)",
      "Organization: Chainguard, Inc"
    ],
    "licenseListVersion": "3.22"
  },
  "dataLicense": "CC0-1.0",
  "documentNamespace": "https://spdx.org/spdxdocs/chainguard/melange/3fe50e511d389d87aa83cfe4aff488dcff156e09",
  "documentDescribes": [
    "SPDXRef-Package-crane-0.20.2-r0"
  ],
  "packages": [
    {
      "SPDXID": "SPDXRef-Package-crane-0.20.2-r0",
      "name": "crane",
      "versionInfo": "0.20.2-r0",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "Apache-2.0",
      "downloadLocation": "NOASSERTION",
      "originator": "Organization: Wolfi",
      "supplier": "Organization: Wolfi",
      "copyrightText": "\n",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:apk/wolfi/crane@0.20.2-r0?arch=aarch64",
          "referenceType": "purl"
        },
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator"<EMAIL_ADDRESS>          "referenceType": "purl"
        },
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:github/google/go-containerregistry@c195f151efe3369874c72662cd69ad43ee485128",
          "referenceType": "purl"
        }
      ]
    }
  ],
  "relationships": []
}

After
{
  "SPDXID": "SPDXRef-DOCUMENT",
  "name": "apk-crane-0.20.2-r0",
  "spdxVersion": "SPDX-2.3",
  "creationInfo": {
    "created": "2024-08-06T17:00:47Z",
    "creators": [
      "Tool: melange (devel)",
      "Organization: Chainguard, Inc"
    ],
    "licenseListVersion": "3.22"
  },
  "dataLicense": "CC0-1.0",
  "documentNamespace": "https://spdx.org/spdxdocs/chainguard/melange/6bc62744084d5c2695c9e44af7abbc83b386da6d",
  "documentDescribes": [
    "SPDXRef-Package-crane-0.20.2-r0"
  ],
  "packages": [
    {
      "SPDXID": "SPDXRef-Package-crane-0.20.2-r0",
      "name": "crane",
      "versionInfo": "0.20.2-r0",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "Apache-2.0",
      "downloadLocation": "NOASSERTION",
      "originator": "Organization: Wolfi",
      "supplier": "Organization: Wolfi",
      "copyrightText": "\n",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:apk/wolfi/crane@0.20.2-r0?arch=aarch64",
          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-crane.yaml-1ac0df6651232eec66d1bba8f95f495b61b52507",
      "name": "crane.yaml",
      "versionInfo": "1ac0df6651232eec66d1bba8f95f495b61b52507",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "NOASSERTION",
      "downloadLocation": "NOASSERTION",
      "originator": "Organization: Wolfi",
      "supplier": "Organization: Wolfi",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator"<EMAIL_ADDRESS>          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-go-containerregistry-v0.20.2",
      "name": "go-containerregistry",
      "versionInfo": "v0.20.2",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "Apache-2.0",
      "downloadLocation": "NOASSERTION",
      "originator": "Organization: Google",
      "supplier": "Organization: Google",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator"<EMAIL_ADDRESS>          "referenceType": "purl"
        }
      ]
    }
  ],
  "relationships": [
    {
      "spdxElementId": "SPDXRef-Package-crane-0.20.2-r0",
      "relationshipType": "DESCRIBED_BY",
      "relatedSpdxElement": "SPDXRef-Package-crane.yaml-1ac0df6651232eec66d1bba8f95f495b61b52507"
    },
    {
      "spdxElementId": "SPDXRef-Package-crane-0.20.2-r0",
      "relationshipType": "GENERATED_FROM",
      "relatedSpdxElement": "SPDXRef-Package-go-containerregistry-v0.20.2"
    }
  ]
}

</details

[ ] Need to fix: the upstream source package claims "Wolfi" as the originator/supplier

      "originator": "Organization: Wolfi",
      "supplier": "Organization: Wolfi",

Nitpicks:
    {
      "SPDXID": "SPDXRef-Package-go-containerregistry-v0.20.2",
      "name": "go-containerregistry",
      "versionInfo": "v0.20.2",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "Apache-2.0",
      "downloadLocation": "NOASSERTION",
      "originator": "Organization: Google",
      "supplier": "Organization: Google",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator"<EMAIL_ADDRESS>          "referenceType": "purl"
        }
      ]
    }

Previously multiple PURLs were generated for pkg:github style references, because tag is not fixed in time; and lots of upstreams delete tags or rewrite them. Given we have versionInfo as a top level information on this spdx package, can we please switch externalref to using git commit hash, rather than version string. As that would be more precise.
Separately, I actually originally wanted to simply use downloadLocation - but couldn't, as I only had a single downloadLocation and needed to encode multiple locations. Now that there is 1:1 downloadLocation, i wonder if we can shove a git hurl into that, and not even emit externalRefs.
 I ponder if we can omit filesAnalyzed, licenseConcluded copyrightText keys, as we never fill them in, so they have no value. I guess that's unrelated to the above reorg and can be submitted stand alone.
Have you attempted an image build with these packages? are there changes needed in apko which "merges" things?

Expanding on the above, given we now have packages for things, I do wonder if we should remove externalref PURL alltogether and instead fill in:
https://spdx.github.io/spdx-spec/v2.3/package-information/#77-package-download-location-field with direct url to plain yaml file; tarball; git+https url with commit encoded
And for tarball downloads also emit https://spdx.github.io/spdx-spec/v2.3/package-information/#710-package-checksum-field for the expected hash. Possibly for git checkouts too.
Because purls are ugly, and difficult to decode, and are not "click to download" experience.

Awesome feedback ❤ — talked more w/ @xnox about some of the nuanced points. Some of these beyond the intended scope of this PR, but we'll go ahead and tackle any that are easy enough to include now, and we'll open an issue for anything left.
Capturing a list of TODOs:

[ ] For upstream source packages, attach a PURL only for github-based git-checkout pipelines. (The pkg:generic/... PURL doesn't seem useful for now. We'll continue to use tags over commits when available, to make PURL-based vulnerability matching easier.)
[ ] Don't attach PURLs to the build config (YAML) package.
[ ] Use downloadLocation for upstream source package. We can use https URLs for fetch, and git+https for git-checkout, and here we can prefer commits over tags.
[ ] Also use downloadLocation for the build config package, using git+https with a commit and subpath for the config file.
[ ] Omit non-required fields where we have no usable information, like with licenseConcluded. (Be careful omitting filesAnalyzed, because the assumed default is true.)
[ ] Test how these new SBOMs interact with apko's SBOM aggregration process.

I also dream a dream, to be able to encode/include 'libfoo-static' as a package in sbom, when it is used in build-depends, or some other field declaring that.
I.e. such that boringssl-fips-static does make it into SBOMs of like envoy-fips package.

Overall thoughts — these are all great comments, and there's a few I should address ASAP.
Also this PR is pretty huge already, so what I'm thinking is we should focus on getting tests in place to assert the SBOMs we expect to produce, and then keep iterating on this as fast follows. Thoughts?

Yes that. Plan was for me to build a few things. And then see how they look. And yeah instrument small thing that is a useful demonstrator of all the things expected to still work.  Cause we need good small test cases of all the things.

FYI — I plan to return to this next week after my vacation to finish this up.

The new melange integration tests are working great. It's already been so nice to have debugging as an option when things go unexpectedly. 🎉
I've also tried to add an integration test that uses apko to build images from melange's packages (and their SBOMs), but I'm hitting enough frictions that I'm going to stop and come back to that later.
In the meantime, here's an example of an apko SBOM using these new changes. I've built crane locally, which pulls in ca-certificates-bundle from the repo since it's a runtime dependency of crane.

`sbom-aarch64.spdx.json`
{
  "SPDXID": "SPDXRef-DOCUMENT",
  "name": "sbom-sha256:0ba74df91115b4c378c31d29d449ad1fef18f5c2b29fbc806d0183fc202a276e",
  "spdxVersion": "SPDX-2.3",
  "creationInfo": {
    "created": "2024-10-10T21:49:59Z",
    "creators": [
      "Tool: apko (devel)",
      "Organization: Chainguard, Inc"
    ],
    "licenseListVersion": "3.16"
  },
  "dataLicense": "CC0-1.0",
  "documentNamespace": "https://spdx.org/spdxdocs/apko/",
  "documentDescribes": [
    "SPDXRef-Package-sha256-248efa08bf88cba44c471bc969160798fb70ddfc852ad528cd4382f952d3cc5d"
  ],
  "packages": [
    {
      "SPDXID": "SPDXRef-Package-sha256-248efa08bf88cba44c471bc969160798fb70ddfc852ad528cd4382f952d3cc5d",
      "name": "sha256:248efa08bf88cba44c471bc969160798fb70ddfc852ad528cd4382f952d3cc5d",
      "versionInfo": "sha256:248efa08bf88cba44c471bc969160798fb70ddfc852ad528cd4382f952d3cc5d",
      "filesAnalyzed": false,
      "description": "apko container image",
      "downloadLocation": "NOASSERTION",
      "supplier": "Organization: apko-generated image",
      "primaryPackagePurpose": "CONTAINER",
      "checksums": [
        {
          "algorithm": "SHA256",
          "checksumValue": "248efa08bf88cba44c471bc969160798fb70ddfc852ad528cd4382f952d3cc5d"
        }
      ],
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:oci/test@sha256%3A248efa08bf88cba44c471bc969160798fb70ddfc852ad528cd4382f952d3cc5d?arch=arm64\u0026mediaType=application%2Fvnd.oci.image.manifest.v1%2Bjson\u0026os=linux",
          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-sha256-0ba74df91115b4c378c31d29d449ad1fef18f5c2b29fbc806d0183fc202a276e",
      "name": "sha256:0ba74df91115b4c378c31d29d449ad1fef18f5c2b29fbc806d0183fc202a276e",
      "versionInfo": "unknown",
      "filesAnalyzed": false,
      "description": "apko operating system layer",
      "downloadLocation": "NOASSERTION",
      "supplier": "Organization: apko-generated image",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:oci/test@sha256%3A0ba74df91115b4c378c31d29d449ad1fef18f5c2b29fbc806d0183fc202a276e?arch=arm64\u0026mediaType=application%2Fvnd.oci.image.layer.v1.tar%2Bgzip\u0026os=linux",
          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-ca-certificates-bundle-20241010-r0",
      "name": "ca-certificates-bundle",
      "versionInfo": "20241010-r0",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "MPL-2.0 AND MIT",
      "downloadLocation": "NOASSERTION",
      "originator": "Organization: Wolfi",
      "supplier": "Organization: Wolfi",
      "copyrightText": "\n",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:apk/wolfi/ca-certificates-bundle@20241010-r0?arch=aarch64",
          "referenceType": "purl"
        },
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:generic/ca-certificates@20241010?vcs_url=git%2Bhttps%3A%2F%2Fgitlab.alpinelinux.org%2Falpine%2Fca-certificates%40153107a3beeff73331fd6ac8fa40312f3e92f4f0",
          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-crane-0.20.2-r1",
      "name": "crane",
      "versionInfo": "0.20.2-r1",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "Apache-2.0",
      "downloadLocation": "NOASSERTION",
      "originator": "Organization: Wolfi",
      "supplier": "Organization: Wolfi",
      "copyrightText": "\n",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator": "pkg:apk/wolfi/crane@0.20.2-r1?arch=aarch64",
          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-crane.yaml-f90fa1593a6b8e20cc29047659e5a6e4c95ed94a",
      "name": "crane.yaml",
      "versionInfo": "f90fa1593a6b8e20cc29047659e5a6e4c95ed94a",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "NOASSERTION",
      "downloadLocation": "NOASSERTION",
      "originator": "Organization: Wolfi",
      "supplier": "Organization: Wolfi",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator"<EMAIL_ADDRESS>          "referenceType": "purl"
        }
      ]
    },
    {
      "SPDXID": "SPDXRef-Package-github.com-google-go-containerregistry-v0.20.2-c195f151efe3369874c72662cd69ad43ee485128",
      "name": "go-containerregistry",
      "versionInfo": "v0.20.2",
      "filesAnalyzed": false,
      "licenseConcluded": "NOASSERTION",
      "licenseDeclared": "Apache-2.0",
      "downloadLocation": "NOASSERTION",
      "originator": "Organization: Google",
      "supplier": "Organization: Google",
      "externalRefs": [
        {
          "referenceCategory": "PACKAGE-MANAGER",
          "referenceLocator"<EMAIL_ADDRESS>          "referenceType": "purl"
        }
      ]
    }
  ],
  "relationships": [
    {
      "spdxElementId": "SPDXRef-Package-sha256-248efa08bf88cba44c471bc969160798fb70ddfc852ad528cd4382f952d3cc5d",
      "relationshipType": "CONTAINS",
      "relatedSpdxElement": "SPDXRef-Package-sha256-0ba74df91115b4c378c31d29d449ad1fef18f5c2b29fbc806d0183fc202a276e"
    },
    {
      "spdxElementId": "SPDXRef-Package-crane-0.20.2-r1",
      "relationshipType": "DESCRIBED_BY",
      "relatedSpdxElement": "SPDXRef-Package-crane.yaml-f90fa1593a6b8e20cc29047659e5a6e4c95ed94a"
    },
    {
      "spdxElementId": "SPDXRef-Package-crane-0.20.2-r1",
      "relationshipType": "GENERATED_FROM",
      "relatedSpdxElement": "SPDXRef-Package-github.com-google-go-containerregistry-v0.20.2-c195f151efe3369874c72662cd69ad43ee485128"
    }
  ]
}

(For later, a branch with additional WIP work to add coverage for apko builds: https://github.com/chainguard-dev/melange/compare/main...luhring:melange:spdx-rewiring-apko-int-test)


How to make database instance persistent?

Hi!
Is there any documentation, how I can separate the database installation in docker container from the database instance data in a persistent volume?
Is this possible with this images?
Kind regards
Torsten

Hi @tkleiber,
At this point in time this isn't possible with those images but this is something that we are investigating.
I'll leave this issue open to act as a tracker for this.
Thanks,

This guy builds a oracleDB image and uses persistence storage.
https://fritshoogland.wordpress.com/2015/08/11/installing-the-oracle-database-in-docker/
He volumes the folder "/u01/app/oracle/data" inside the container. No clue wich one will be in this installation.

@sigaELwebeo you tryed the solution using volumes? I tryed creating a volume for the folder "/u01/app/oracle/oradata/XE" and "/u01/app/oracle/",but don't work, the oracle instance could not be mounted.  I tryed with this: docker create --name oracle  -v /Users/ramon/Development/docker/database/oracle:/u01/app/oracle --shm-size=1g -p 1521:1521 -p 8080:8080 -p 1234:22 oracle/database:<IP_ADDRESS>-xe .

Y did persistent data but in an awful way!!!
1.- Create a container1 with volume:
/docker/volumes/oradb:/opt/oracle/oradata
and it was empty.
2.- Create a container2 with volume /docker/volumes/oradb:/data
Docker exec on it and copy the entire ORCLCDB folder to /data
3.- Docker exec into container 1, chown and chmod on ORCLCDB
Copy and replace file /opt/oracle/oradata/ORCLCDB/control01.ctl to /opt/oracle/fast_recovery_area/ORCLCDB/control02.
4.- Delete container2.
I'm waiting to have more availability to modify Dockerfile to build oradata folder and expose as volume.

With this image, https://github.com/MaksymBilenko/docker-oracle-xe-11g , I can mount a volume with the way I want, but I really want is use the official image.  @ratroncoso I will try your solution,thank you.

@ratroncoso Your solution is for oracle 12c, right? I'm asking because the folders hierarchy are different on Oracle 11 XE.

Oracle Database Enterprise Edition 12C

@ratroncoso did you try with  Oracle 11g XE or just Oracle 12c?

@ramonPires only on Oracle12c

@gvenzl any update about this issue? I found other solution, http://marceloochoa.blogspot.com.br/2016/06/is-time-for-oracle-xe-on-docker.html ,but I don't  test yet.

With the latest commit we now have containers that provide persistence for all database editions.

When starting the xe docker container in vagrant box via
docker run  ... --restart unless-stopped
and calling
vagrant halt
vagrant up
the docker container comes up after this, as I can see via docker ps, but database is not available.
Have to do to make the database available again via
docker stop container
docker start container

@tkleiber can you provide logs of what's happening? really hard to figure out what's going on.
We'd also appreciate if you could post a new issue with these logs.

How can I find this logs?

$ docker logs should be a good start

@brunoborges I have some problems with oracle https://github.com/oracle/docker-images/issues/1480