Support Fee Bump Transactions (CAP 15)

The Stellar Core protocol 13 release will introduce a new type of transaction called Fee Bump transactions. Here is a summary of the changes (see https://github.com/stellar/stellar-protocol/blob/master/core/cap-0015.md for the full set of details):

The XDR representation of a transaction envelope has changed to be a union of legacy (pre protocol 13) transactions, normal transactions where the source account pays for their own transactions, and fee bump transactions where another account elects to pay for someone else's transaction

A fee bump transaction wraps a normal transaction with two extra pieces of information: the account which will be paying for the fee and the new fee amount the account is willing to pay. Consequently, the transaction hash of the fee bump transaction will differ from the hash of the fee bump's inner transaction.

A fee bump transaction envelope will contain the signature of the fee source account.

The fee bump transaction envelope will also contain the full contents of the inner transaction including the original source account, signatures, and fee of the inner transaction.

To support Fee Bump transactions we will need to implement the following changes in Horizon and the SDKs:

[ ] Update XDR definitions and auto generated code derived from XDR definitions in Horizon

[ ] Update XDR definitions and auto generated code derived from XDR definitions in all the SDKs

[ ] Update horizon schema to support fee bump transactions. I propose the following schema changes:

add inner_transaction_hash, inner_signatures,  and fee_account columns to the history_transactions table
add index on inner_transaction_hash
if a transaction is not a fee bump transaction, inner_transaction_hash, inner_signatures, and fee_account will be NULL
if a transaction is a fee bump transaction, inner_transaction_hash will be the hash of the inner transaction wrapped by the fee bump, inner_signatures will be the set of signatures for the inner transaction, and fee_account will be the account paying the fee on behalf of the inner transaction
if a transaction is a fee bump transaction, transaction_hash will be the hash of the fee bump transaction, max_fee will be the fee set by the fee bump transaction, fee_charged will be the amount paid by the fee source account, and all remaining fields in the history_transactions row will be populated using fields from the inner transaction.

[ ] Update horizon ingestion to support fee bump transactions.

We can punt on adding any new types of effects for fee bump transactions because Horizon effects are already missing cases like "Account Debited" for transaction fees (see https://github.com/stellar/go/issues/2175)
We will still need to modify all ledger transaction processors because the way we extract operation results from the transaction results XDR object has changed
If a fee bump transaction succeeds, we will need to examine the results field from the TransactionResult XDR object.
If a fee bump transaction fails, we will need to examine the innerResult field from the TransactionResult XDR object.
When ingesting transactions we extract the transaction hash from the TransactionResultPair XDR object in the TransactionResultSet. In the case of a fee bump transaction, I assume the transaction hash in TransactionResultPair will correspond to the fee bump transaction, not the inner transaction. How should we obtain the transaction hash of the inner transaction? Should we calculate the inner transaction hash during ingestion? Or should TransactionResultPair be amended to include the inner transaction hash? @jonjove any thoughts on this topic?

[ ] Update the horizon protocol transaction response struct so that it includes the new fields added to the  history_transactions table

[ ] Update SDKs so that they are compatible with the new horizon transaction response structure

[ ] Update horizon GET /transactions/{hash} handler so that hash can match either the transaction_hash or inner_transaction_hash column from the history_transactions table. In other words, we should be able to look up a transaction by either its hash or its inner hash (in the case the transaction is a fee bump)

[ ] Update txsub package in horizon so that a request to submit a normal transaction will respond with the transaction result from its corresponding fee bump transaction if the normal transaction was fee bumped.

[ ] Updates SDKs with functionality to create and submit fee bump transactions. Note that fee bump transactions have a different fee rate calculation compared to normal transactions, see https://github.com/stellar/stellar-protocol/blob/master/core/cap-0015.md#fee-rate .

[ ] Consider SEP 10 implications of fee bump transactions. We probably want to add the restriction that a fee bump transaction cannot be a valid SEP 10 challenge transaction.

@tamirms Unfortunately TransactionResultPair cannot be changed directly. It doesn't have an extension point, and it is hashed into the LedgerHeader via TransactionResultSet. If the actual TransactionEnvelope is available to Horizon when it is processing results, then I think computing the inner hash is a fine approach. I note, however, that the TransactionResultPair was not designed like that. We could modify the TransactionResult to reveal the inner hash for fee-bump transactions, rather than the TransactionResultPair.
One thing that I don’t see addressed in this issue is that there won’t be any true TransactionResult for the inner transaction. But the “GET /transactions/{hash}” endpoint returns a field “result_xdr”. How do you plan to resolve this issue? I have discussed this with @ire-and-curses, so he may have some thoughts on this as well. It is possible to modify the TransactionResult to make this easier for you, we just need to agree on what is appropriate.
Below is a conceptual prototype of what the new TransactionResult XDR could be in order to resolve both of these issues. Be aware that this XDR is simplified, and definitely will not compile with xdrc.
enum TransactionResultCode
{
    txFEE_BUMP_INNER_SUCCESS = 1, // all operations succeeded for fee bump
    txSUCCESS = 0,                                    // all operations succeeded

    txFAILED = -1, // one of the operations failed (none were applied)

    txTOO_EARLY = -2,         // ledger closeTime before minTime
    txTOO_LATE = -3,          // ledger closeTime after maxTime
    txMISSING_OPERATION = -4, // no operation was specified
    txBAD_SEQ = -5,           // sequence number does not match source account

    txBAD_AUTH = -6,             // too few valid signatures / wrong network
    txINSUFFICIENT_BALANCE = -7, // fee would bring account below reserve
    txNO_ACCOUNT = -8,           // source account not found
    txINSUFFICIENT_FEE = -9,     // fee is too small
    txBAD_AUTH_EXTRA = -10,      // unused signatures attached to transaction
    txINTERNAL_ERROR = -11       // an unknown error occured

    txFEE_BUMP_INNER_FAILED = -12 // the inner transaction failed
};

struct TransactionResult
{
    int64 feeCharged; // actual fee charged for the transaction

    union switch (TransactionResultCode code)
    {
    case txFEE_BUMP_INNER_SUCCESS:
    case txFEE_BUMP_INNER_FAILED:
        TransactionResultPair innerResult;
    case txSUCCESS:
    case txFAILED:
        OperationResult results<>;
    default:
        void;
    }
    result;

    // reserved for future use
    union switch (int v)
    {
    case 0:
        void;
    }
    ext;
};

With XDR along these lines, fee-bump transactions will never return a TransactionResult with code txSUCCESS or txFAILED, but all other results are possible. If code is txFEE_BUMP_INNER_SUCCESS or txFEE_BUMP_INNER_FAILED then the inner code will never be txFEE_BUMP_INNER_SUCCESS or txFEE_BUMP_INNER_FAILED. As always, negative codes will indicate failure while non-negative codes will indicate success.
Would a change like this be helpful? I’m happy to optimize the XDR further if possible.

@jonjove and I discussed https://github.com/stellar/go/issues/2360#issuecomment-595100969 on slack. Following our conversation, here is my new proposal for Horizon responses:
We want to structure the Horizon responses so that the hash in the response always matches the hash in the  GET /transactions/{hash} request. Additionally, we want it to be very easy to disambiguate normal transaction responses from inner and fee bump transaction responses. Finally, we want to be more explicit regarding which fields in the response are derived from the inner tx vs the outer fee bump tx.

If a GET /transactions/{hash} request queries for a normal (non fee bump) transaction the response from horizon will resemble the current format . The only difference is that there will be an extra string field called fee_account. In this case, fee_account will always equal source_account. Because the source account of the transaction is paying for the fee.

If a GET /transactions/{hash} request looks up a transaction using the fee bump tx hash (as opposed to the inner tx hash) the response from horizon will have the following form:

Attribute
Type

id
string
This attribute has not changed from the current format.

paging_token
string
This attribute has not changed from the current format.

successful
bool
This attribute has not changed from the current format.

hash
string
The hash is equal to the fee bump tx hash, not the inner tx hash.

ledger
number
This attribute has not changed from the current format.

created_at
ISO8601 string
This attribute has not changed from the current format.

fee_account
string
This is a new field. It is set to account which is paying for the fee

source_account
string
This is the source account of the inner tx

source_account_sequence
string
This is the sequence number of the source account of the inner tx

max_fee
number
The the maximum fee the fee account was willing to pay for the fee bump tx.

fee_charged
number
The fee paid by the  fee account  when the transaction was applied to the ledger.

operation_count
number
The number of operations that are contained within the inner tx.

envelope_xdr
string
The XDR string for the fee bump tx envelope

result_xdr
string
The XDR string for the fee bump TransactionResult. Note that this XDR object also contains results from operations in the inner tx.

result_meta_xdr
string
The XDR string for TransactionMeta (note that the XDR definition hasn't changed for this field and that there is no concept of TransactionMeta applying to the inner transaction separately from the fee bump transaction).

fee_meta_xdr
string
The XDR string for LedgerEntryChanges xdr struct produced by taking fees for this transaction (note that the XDR definition hasn't changed for this field and that there is no reason to separate LedgerEntryChanges into inner tx changes and fee bump tx changes)

memo_type
string
The memo type for the inner tx.

memo
string
The memo for the inner tx.

signatures
string[]
The list of signatures used to sign the fee bump tx

valid_after
RFC3339 date-time string
Derived from the inner tx time bounds.

valid_before
RFC3339 date-time string
Derived from the inner tx time bounds.

inner_transaction
Object with the following fields: max_fee, hash, signatures
This object contains the max fee from the inner transaction, the hash of the inner transaction, and a list of signatures from the inner transaction

If a GET /transactions/{hash} request looks up a transaction using the inner tx hash (as opposed to the fee bump tx hash) the response from horizon will have the following form:

Attribute
Type

id
string
This attribute has not changed from the current format.

paging_token
string
This attribute has not changed from the current format.

successful
bool
This attribute has not changed from the current format.

hash
string
The hash is equal to the fee bump tx hash, not the inner tx hash.

ledger
number
This attribute has not changed from the current format.

created_at
ISO8601 string
This attribute has not changed from the current format.

fee_account
string
This is a new field. It is set to account which is paying for the fee

source_account
string
This is the source account of the inner tx

source_account_sequence
string
This is the sequence number of the source account of the inner tx

max_fee
number
The the maximum fee the source account was willing to pay for the inner tx.

fee_charged
number
This will always be 0 because the source account does not pay for fees in a fee bump tx.

operation_count
number
The number of operations that are contained within the inner tx.

envelope_xdr
string
This will always be the empty string. The XDR strings for the fee bump transaction can be found  in the fee_bump_transaction object.

result_xdr
string
This will always be the empty string. The XDR strings for the fee bump transaction can be found  in the fee_bump_transaction object.

result_meta_xdr
string
This will always be the empty string. The XDR strings for the fee bump transaction can be found  in the fee_bump_transaction object.

fee_meta_xdr
string
This will always be the empty string. The XDR strings for the fee bump transaction can be found  in the fee_bump_transaction object.

memo_type
string
The memo type for the inner tx.

memo
string
The memo for the inner tx.

signatures
string[]
The list of signatures used to sign the inner tx

valid_after
RFC3339 date-time string
Derived from the inner tx time bounds.

valid_before
RFC3339 date-time string
Derived from the inner tx time bounds.

fee_bump_transaction
Object with the following fields: max_fee, fee_charged, hash, signatures, envelope_xdr, result_xdr, result_meta_xdr, fee_meta_xdr
All the fields in this object are derived from the fee bump transaction

Note that all the links in the horizon response will be based off the fee bump transaction hash.

I found a case where the Java SDK parses the transaction envelope XDR while deserializing the Horizon JSON response. This means that we will need to release the protocol 13 support in the SDKs before it is deployed in Horizon and Stellar Core. Otherwise, clients which use the protocol 12 version of the Java SDK will fail to parse Horizon responses corresponding to fee bump transactions.

Regarding how we obtain the hash of the inner transaction, I think it's best if Horizon computes the hash manually during ingestion using the transaction envelop. @jonjove mentions that it should be relatively easy for Stellar Core to provide the inner transaction hash somewhere in TransactionResult. But, I think it's best to lean on the side of caution by avoiding Stellar Core / XDR changes which are not absolutely necessary.

@tamirms when loading an inner transaction I wonder if we should generalize the fee_bump_transaction  to parent_transaction  or outer_transaction  --  @jonjove do you think there could be scenarios where we end up with other transactions that have an innerTransaction field?
Also shall we add links to the fee_bump_transaction and inner_transaction representation (mostly for consistency with the rest of the API)?

@abuiles It's definitely possible that we one day might have other "nesting" transaction types. I had an idea for one a few months ago, although I don't currently think we will end up developing that idea.

@jonjove thanks!

We'll be pausing further design work on this until CAP 15 hits accepted state. Active discussion ongoing here.

Ignore previous comment. I was talking about CAP 23. This one is good to go.

Seems like this has been fixed in the recent versions of Horizon.


Unexpected exception encountered while provisioning agent

We have some issue when Jenkins wants to create slaves.
It is not always the case, but it happens frequently.
The behavior is the following:

VM are actually created in openstack but the floating IP is not associated to the VM and we have to manually delete these VMs to release the ressources
We have floating IP in openstack that are actually reserved but not active (e.g. kind of orphan floating IP)
The Jenkins logs shows a lot of error "Unexpected exception encountered while provisioning agent" with "jenkins.plugins.openstack.compute.internal.Openstack$ActionFailed: timeout Allocating for" (See Attached trace)

Note that we have scripted some openstack commands to "simulate" the plugin jenkins behavior (It creates a VM, creates a floating IP and associates the floating IP to the VM). We have looped on such scenario, but we never reproduced the issue.
Of course, I guess that this kind of scenario does not use the same command as openstack4j does.
Anyway, I'm wondering whether it was possible to make some change on timeout values at:
'''public @Nonnull FloatingIP assignFloatingIp(@Nonnull Server server, @CheckForNull String poolName) throws ActionFailed {
    debug("Allocating floating IP for " + server.getName());
    ComputeFloatingIPService fips = clientProvider.get().compute().floatingIps(); // This throws when user is not authorized to manipulate FIPs
    FloatingIP ip;
    try {
        ip = fips.allocateIP(poolName);
    } catch (ResponseException ex) {
        // TODO Grab some still IPs from JCloudsCleanupThread
        throw new ActionFailed(ex.getMessage() + " Allocating for " + server.getName(), ex);
    }'''

If you have some suggestion of code modification, I may recompile the plugin and make some tests.
jenkins.log

From the log it is apparent you are running into this situation during both server boot and FIP allocation. I am not convinced this is in any way related to the FIPs your tenant has allocated as the cause here is socket read timeouts. As this does not appear to be configured by openstack4j nor this plugin so I believe you are running out of 10 second per openstack request[1].
Here[2] is a patched 2.42 with the 20s timeout hardcoded. If you can get that deployed and you experience improvement, we will have the cause confirmed. Then we can proceed with making this configurable for cases like this.
[1] https://github.com/square/okhttp/blob/cd0c8188a2f7aed6ecaeea8edafb2770981c63c1/okhttp/src/main/java/okhttp3/OkHttpClient.java#L476-L478
[2] https://github.com/olivergondza/openstack-cloud-plugin/commit/8d81d2d993564601f307e977004380b8a7145699

Hi @olivergondza,
I have seen the same problem a lot in the past weeks and deployed your olivergondza@8d81d2d patch. This helps here to avoid timeouts. I would really appreciate if you for now can merge the hardcoded increase and later can make that configurable somewhere.
The only thing which is left over as error in the stats page is now this but thats probably not related to the discussed timeout:
java.util.NoSuchElementException: No such server running: e8095c97-14d6-4526-8e01-e871a3bc7f1d 	at jenkins.plugins.openstack.compute.internal.Openstack.getServerById(Openstack.java:480) 	at jenkins.plugins.openstack.compute.JCloudsSlave.getPublicAddressIpv4(JCloudsSlave.java:290) 	at jenkins.plugins.openstack.compute.slaveopts.LauncherFactory$SSH.isWaitingFor(LauncherFactory.java:176) Caused: jenkins.plugins.openstack.compute.JCloudsCloud$ProvisioningFailedException: No such server running: e8095c97-14d6-4526-8e01-e871a3bc7f1d 	at jenkins.plugins.openstack.compute.slaveopts.LauncherFactory$SSH.isWaitingFor(LauncherFactory.java:178) 	at jenkins.plugins.openstack.compute.JCloudsCloud.slaveIsWaitingFor(JCloudsCloud.java:319) 	at jenkins.plugins.openstack.compute.JCloudsSlaveTemplate.provisionSlave(JCloudsSlaveTemplate.java:215) 	at jenkins.plugins.openstack.compute.JCloudsCloud$NodeCallable.call(JCloudsCloud.java:310)
Thanks for your great work here!

@inka, thanks for confirmation. The outstanding exception is likely caused by the Server not being found during agent launching for which I have no explanation. As a followup investigation, consult the server log for the VM id and name.
@tgatinea, Can you confirm?

I was busy these last days.
I'll come back to this issue next week.
Anyway, thanks both of you for your answers.

I've tested the patch 10s to 20s:

3 clouds with new image created in glance
112 slaves that were deleted and re-created using the new image

Note that when the image is new in glance, it takes some time (e.g. Spawning) to be replicated in hosts and it increases the issue.
I've checked that the delete was correct (Before, it often happens that agents were deleted from Jenkins point of view, while it was remaining alive in Openstack).
=> The patch seems to improve that
During the creation of new slaves I had few issues. Less that before, but still some issues.
And if I set the retention time to 0, I have seen on one cloud the same issue as before (e.g. VM without floating IP)
So I've increased the timeout to 40s and I've checked on another Jenkins. It seems better, but I will have to redo the test to confirm the fix. I'll try to do that tomorrow or in this evening.
Anyway, I guess that our cloud/network is not in a good health and I suppose that the fix is more a work around.

Any luck with the further stages of testing and merging? We are also really counting seconds until this goes live :+1:

@bmoqimi, waiting for a second user to confirm this is working - @tgatinea appears busy.  Once we know we have nailed it, I will merge and release.

Hi @olivergondza
I have deployed the snapshot over the weekend and so far it has been working really well. We even see less timeouts on those jenkins instances :+1:


Allow isolated workspaces per test suite

It currently only seems possible to configure a single workspace path in wdio.conf.ts under wdio:vscodeOptions -> workspacePath. This works great for some scenarios, but it can be problematic in cases where test suites are run in parallel. Namely, changing vscode settings as part of tests is not isolated to the current test suite but bleeds into other concurrently running suites. It would be helpful to provide a mechanism for isolating workspace locations per test suite. One thought that comes to mind would be to support a callback function as a value for workspaceSettings that could dynamically return a file path when a test suite runs.

I think this is a great idea! I am personally working at a lot of different things on WebdriverIO and probably won't have time to contribute to the VS Code service part anytime but I would appreciated any contributions on this side. Note that we pay out such contributions thanks for our project sponsors.

@christian-bromann I’d love to hear more details about how you pay out for contributions

You can read more about it here: https://bromann.dev/post/automated-contributor-expense-process/. Essentially everyone who reviews and merges PRs on behalf of the WebdriverIO project decides how much money should be expensed for the work. You will be able to expense the amount through OpenCollective.


Investigate the Master System entries missing from a standard DAT file for SMS

TIL Master System is supported with Libraries, assuming via way of using an SMS > GG adapter with the Game Gear adapter for the Pocket.
Images can be built today, but like the other systems, there are a number of image names missing from the DAT file. Go through specially-noted-titles.md for SMS.
If this is anything like Game Gear, these entries will mostly (if not all be) duplicate images with incorrect titles that already have a correctly named image and can be omitted from special processing.
Any romhacks or other non-cartridge titles (e.g. [BIOS] flagged images) should be explicitly ignored or detected in the file exclusion filter (latter preferred).

I recently picked up a SMS > GG adapter and games are properly showing up in Library as SMS and not GG.  Could you add the pocket image library for SMS to the release page?  I tried generating myself, but didn't have much luck on mac. Thanks!


e2e tests failing for atuin

Expected Behaviour

make e2e should report all tests passed when building arkade

Current Behaviour

make e2e fails on "Test_CheckTools/Download_of_atuin"

Possible Solution

Correct the URL location for the atuin tool

Steps to Reproduce (for bugs)

clone arkade
make e2e

Context

This issue stops all PRs from pasisng, as global tests are failing

If requesting a CLI for "arkade get"
How many downloads does this tool have? Check at:
https://somsubhra.github.io/github-release-stats/
If approved, are you willing to submit a pull request for this?
Your Environment

What Kubernetes distribution are you using?

kubectl version

Operating System and version (e.g. Linux, Windows, MacOS):

uname -a

cat /etc/os-release

What arkade version is this?

arkade version

failing at git commit 6f5d90ec728cbf9a34249cedb0009d30021266f4

Thanks for raising this. And by the way Atuin seems to have arm binaries now if you wanted to send a PR to add them?

Ah awesome! I'll do that tomorrow and get a PR in. Btw, I didn't get your email last week (not at this address at least).Glad I could help! I'll raise two PRs, one for the ARM binaries and another for kubestr tomorrow On 31 Jul 2023 19:12, Alex Ellis @.***> wrote:
Thanks for raising this. And by the way Atuin seems to have arm binaries now if you wanted to send a PR to add them?
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

I'll get a PR submitted!

Is this still something you're taking care of @mkimberley ?


Update existing API routes

Pull request #100 implements database schema rework that breaks the backend API. That pull request was merged into a development branch 'schema_upgrade' until the backend API has been adjusted to. This is the goal of this task.
Changes should be made the backend API functions so that the existing backend API unit test pass.

#104 #105 #106 #107 #128


Resolving DNS servers

I think we should be reading the list of DNS servers from /etc/resolv.conf by default. WDYT? I'm thinking that it is painful to have to declare the DNS servers for each service.

Just to let you know that I'm working on a PR for this right now.

I've thought about it, but I intended this library to be used only for special cases and not as a replacement for default mechanism. If you think this might be a good idea, then we should definitely read resolv.conf. But we also need to read /etc/hosts to mimic default behavior.
Glad to hear you actually need this library:)

I've added a PR - see what you think: https://github.com/ilya-epifanov/akka-dns/pull/5


Total page counts are reported strangely for IPP jobs

When a print job is sent via socket://, CUPS reports printed pages straightforwardly:
printer user job# [date] page_number copies ...
However, when a print job is sent via IPP, in addition to processed pages CUPS reports back intermittent number of pages printed as reported by the printer:
printer user job# [date] total 1 ...
printer user job# [date] total 3 ...
printer user job# [date] total 8 ...

These numbers are more confusing than helpful in the log. If anything, it should report final "total" number, or not report it at all. This information looks to be more pertaining to the debug log, rather than page log.
Or maybe there is a configuration option to suppress these totals in the log?
Please change it to report just final total number of pages, or none whatsoever.
Typical page log attached, and job_log attached.

This change is part of CUPS 2.2.10 and 2.3 (Issue #5439) - we now only log pages when the job is completed.

Quick question: is it expected that the number of copies is no longer reported in 2.2.11?
If I print 2 pages with two copies, see "total 4 -" in the log, with
PageLogFormat being the default ""%p %u %j %T %P %C %{job-billing} %{job-originating-host-name} %{job-name} %{media} %{sides}""

@akorobkin Yes, in order to fix the accounting values we've dropped the per-page reporting in favor of getting accurate totals. You can always add the copies attribute to the PageLogFormat...