Multitenancy and Endorsement

Hi all,
I have some questions regarding the setup of my Multitenancy Node with a separate Endorser Node.
I feel as though I am going in circles and am stuck in a configuration nightmare so to speak and think I have just confused myself to the point that I need to ask for some clarifications on the setup.
I feel I am missing something as I can't get my Sub wallets (hereafter referred to simply as Tenants) to create the Schemas and associated Credential Definitions.
I have already read the following:

#1657
#983
#1460

My takeaway from the above is to use a separate instance of ACA-py to run as an Endorser and do the Endorsement for transactions for my Tenants.
I am using the following version of ACA-py, denoted by this docker image, I am also using askar as the wallet type.

ghcr.io/hyperledger/aries-cloudagent-python:py3.12-1.0.0

I have also read the documentation here:

https://aca-py.org/main/features/Multitenancy/

I will detail below what I have done and where I am currently at.
I am using the von-network for testing, I have it running on a VM in GCP. I created a DID and Wallet for it following this guide:

https://github.com/bcgov/von-network/blob/main/docs/Indy-CLI.md

This gives me a Public DID, Verkey, and Seed. I then registered, via the UI, the DID as an Endorser on the von-network.

Now, I have a bootstrap script, this is used to:

Create a new Tenant
Create a DID for the Tenant
Write the Tenant DID to the Ledger (I call the Endorser to do this)
Create a static connection between the Tenant and the Endorser
Set the Tenant DID as Public

This can be seen below (Please note, I have summarized my code into a small single function to convey the steps I am following)
async def create_subwallet_with_setup(
    multitenant_client: AcaPyClient,
    endorser_client: AcaPyClient,
    wallet_name: str,
    endorser_info: Dict[str, Any]
) -> Tuple[str, str, str]:
    try:
        # Step 1: Create a new subwallet
        response = await multitenant_client.multitenancy.create_wallet(
            body=CreateWalletRequest(
                wallet_name=wallet_name,
                wallet_key=f"{wallet_name}_key",
                wallet_type="askar",
                label=wallet_name,
                wallet_webhook_urls=["<endorser-url>"]
            )
        )
        token = response.token

        # Step 2: Create a client for the new subwallet
        async with managed_sub_client(
            base_url=multitenant_client.configuration.host,
            api_key=multitenant_client.api_key,
            token=token
        ) as subwallet_client:
            # Step 3: Create a new DID for the subwallet
            create_did_response = await subwallet_client.wallet.create_did(DIDCreate())
            did_info = create_did_response.result

            # Step 4: Register the DID on the ledger using the endorser
            await endorser_client.ledger.register_nym(
                did=did_info.did,
                verkey=did_info.verkey,
                alias=None
            )

            # Step 5: Establish a static connection with the endorser
            static_conn_request = ConnectionStaticRequest(
                alias=f"{wallet_name}-to-endorser",
                their_did=endorser_info['did'],
                their_endpoint=endorser_info['endpoint'],
                their_verkey=endorser_info['verkey']
            )
            static_conn_result = await subwallet_client.connection.create_static_connection(
                body=static_conn_request
            )
            connection_id = static_conn_result.record.connection_id

            # Step 6: Set the Endorser Role
            await endorser_client.endorse_transaction.set_endorser_role(
                conn_id=connection_id,
                transaction_my_job="TRANSACTION_AUTHOR"
            )

            # Step 7: Set the Endorser Info via metadata
            endorser_info_metadata = {
                "endorser_did": endorser_info['did'],
                "verkey": endorser_info['verkey'],
                "endpoint": endorser_info['endpoint'],
                "name": "Endorser"
            }
            metadata_request = ConnectionMetadataSetRequest(metadata={"endorser_info": endorser_info_metadata})
            await subwallet_client.connection.set_metadata(connection_id, body=metadata_request)

            # Step 8: Set the DID as public for this subwallet
            await subwallet_client.wallet.set_public_did(did_info.did)

        return token, did_info.did, connection_id
    except Exception as e:
        logger.error(f"Error creating subwallet '{wallet_name}': {str(e)}")
        raise

Questions:

I am unsure about points 5, 6, and 6. I am not sure if I am doing this correctly.
I am unsure if I am supposed to be creating this connection via the static connection endpoint.

When I run the code, it successfully creates the Tenant, DID, Writes it to the Ledger, creates the connection, sets the metadata and sets the DID as public as be seen from my logs:
2024-09-22 14:23:50,505 - INFO - Created subwallet 'Test' with token: <token>
2024-09-22 14:23:51,271 - INFO - Created new DID in subwallet: <did>
2024-09-22 14:23:52,435 - INFO - Registered DID <did> on the ledger using endorser client
2024-09-22 14:23:53,016 - INFO - Established static connection. Connection ID: 0611db15-9488-47df-8af1-64228c6b99c6
2024-09-22 14:23:53,237 - INFO - Connection 0611db15-9488-47df-8af1-64228c6b99c6 is active.

But when I try to perform actions against my Subwallet I am running into issues, I shall just show the one method I use to create a Schema, I have a feeling this approach is completely wrong:
async def create_and_endorse_schema(
    subwallet_client: AcaPyClient,
    endorser_client: AcaPyClient,
    schema_name: str,
    schema_version: str,
    attributes: List[str],
    connection_id: str
) -> SchemaSendResult:
    try:
        # Step 1: Publish the schema, requesting transaction creation for the endorser
        schema_request = SchemaSendRequest(
            schema_name=schema_name,
            schema_version=schema_version,
            attributes=attributes
        )
        result = await subwallet_client.schema.publish_schema(
            conn_id=connection_id,
            create_transaction_for_endorser=True,
            body=schema_request
        )

        if result.txn is not None:
            transaction_id = result.txn.transaction_id

            # Step 3: Endorse the transaction using the endorser client
            endorsed_txn = await endorser_client.endorse_transaction.endorse_transaction(
                tran_id=transaction_id
            )

            # Step 4: Write the endorsed transaction to the ledger
            written_txn = await endorser_client.endorse_transaction.write_transaction(
                tran_id=endorsed_txn.transaction_id
            )

            # Step 5: Fetch and return the schema details
            schema = await subwallet_client.schema.get_schema(schema_id=written_txn.schema_id)
            return schema
        else:
            raise Exception("Unexpected result: Transaction data not returned.")
    except Exception as e:
        logger.error(f"Error creating schema '{schema_name}': {str(e)}")
        raise

This always resulted in a Transaction Not Found error.
NB: Please also note, the Multitenant Agent and Endorser Agent are using separate databases, I am not sure if this should be the case.
I then started looking at the Endorser CLI arguments and amended my Endorser Agent to include:
--endorser-protocol-role endorser
--endorser-endorse-with-did <endorser-public-did>
--auto-endorse-transactions

and my Multitenant Agent to include:
--endorser-protocol-role author
--endorser-public-did <endorser-public-did>
--auto-request-endorsement
--auto-write-transactions
--auto-create-revocation-transactions
--auto-promote-author-did

Now when I run my script, it fails with the following error:
HTTP response body: 400: No endorser connection found

I would like some guidance if anyone has some time as I feel I took a wrong turn and wound up in an abandoned mine full of confusion and despair.
I will need to document this entire setup at work and would be very willing to contribute it back to the ACA-py documentation once done.
Please let me know if anything is unclear or if I have simply misunderstood something basic.

@esune @WadeBarnes — could one of help out (or find someone on the BC team) to help out with this question. It sounds like the answer is very close, and we’ve definitely had this going for some time.

Hi @esune and @swcurran thank you for the reply! I'll tackle this tomorrow and provide feedback!

Hi @esune and @swcurran sorry for the delay, this is part of my RnD and other tickets came in that required my attention.
This is where I have gotten today.
I am using this docker-compose file:

version: '3'

services:
  postgres:
    image: postgres:16
    environment:
      - POSTGRES_USER=admin
      - POSTGRES_PASSWORD=admin
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  endorser_agent:
    image: ghcr.io/hyperledger/aries-cloudagent-python:py3.12-1.0.0
    depends_on:
      postgres:
        condition: service_started
    ports:
      - "8000:8000"
      - "11000:11000"
    environment:
      - ACAPY_WALLET_STORAGE_CONFIG={"url":"postgres:5432","max_connections":5}
      - ACAPY_WALLET_STORAGE_CREDS={"account":"admin","password":"admin","admin_account":"admin","admin_password":"admin"}
    command: start
      --inbound-transport http <IP_ADDRESS> 8000
      --outbound-transport http
      --admin <IP_ADDRESS> 11000
      --wallet-name <wallet-name>
      --wallet-key <wallet-key>
      --seed <wallet-seed>
      --admin-api-key <admin-api-key>
      --webhook-url <webhook-url>
      --auto-respond-credential-proposal
      --auto-store-credential
      --auto-respond-presentation-request
      --auto-respond-presentation-proposal
      --auto-respond-credential-request
      --auto-respond-credential-offer
      --auto-accept-requests
      --auto-accept-invites
      --auto-respond-messages
      --debug-connections
      --debug-credentials
      --endpoint http://endorser_agent:8000
      --genesis-url <genesis-url>
      --accept-taa on_file 1.0
      --log-level error
      --emit-new-didcomm-mime-type
      --emit-new-didcomm-prefix
      --exch-use-unencrypted-tags
      --tails-server-base-url <revocation-url>
      --auto-provision
      --wallet-type askar
      --wallet-storage-type postgres_storage
      --endorser-protocol-role endorser
      --endorser-endorse-with-did <endorser-did>
      --auto-endorse-transactions

  multitenant_agent:
    image: ghcr.io/hyperledger/aries-cloudagent-python:py3.12-1.0.0
    depends_on:
      postgres:
        condition: service_started
    ports:
      - "8001:8000"
      - "11001:11000"
    environment:
      - ACAPY_WALLET_STORAGE_CONFIG={"url":"postgres:5432","max_connections":5}
      - ACAPY_WALLET_STORAGE_CREDS={"account":"admin","password":"admin","admin_account":"admin","admin_password":"admin"}
    command: start
      --inbound-transport http <IP_ADDRESS> 8000
      --outbound-transport http
      --admin <IP_ADDRESS> 11000
      --seed <wallet-seed>
      --admin-api-key <admin-api-key>
      --wallet-name <wallet-name>
      --wallet-key <wallet-key>
      --webhook-url <webhook-url>
      --auto-respond-credential-proposal
      --auto-store-credential
      --auto-respond-presentation-request
      --auto-respond-presentation-proposal
      --auto-respond-credential-request
      --auto-respond-credential-offer
      --auto-accept-requests
      --auto-accept-invites
      --auto-respond-messages
      --debug-connections
      --debug-credentials
      --endpoint http://multitenant_agent:8000
      --genesis-url <genesis-url>
      --accept-taa on_file 1.0
      --log-level error
      --emit-new-didcomm-mime-type
      --emit-new-didcomm-prefix
      --exch-use-unencrypted-tags
      --tails-server-base-url <revocation-url>
      --auto-provision
      --wallet-type askar
      --wallet-storage-type postgres_storage
      --multitenant
      --multitenant-admin
      --jwt-secret <secret>
      --endorser-protocol-role author
      --endorser-public-did <endorser-did>
      --auto-request-endorsement
      --auto-write-transactions
      --auto-create-revocation-transactions
      --auto-promote-author-did

volumes:
  postgres_data:

I think it's better to just show where I am at the moment:
I create a connection like this now:
async def create_endorser_connection(subwallet_client: AcaPyClient, endorser_client: AcaPyClient, wallet_name: str) -> str:
    """Create a DID Exchange invitation with custom roles and establish a connection."""

    try:
        # Step 1: Create the OOB invitation using the endorser_client
        oob_invitation = await endorser_client.out_of_band.create_invitation(
            auto_accept=True,
            body=InvitationCreateRequest(
                accept=[
                    "didcomm/aip1",
                    "didcomm/aip2;env=rfc19"
                ],
                alias=f"{wallet_name}-to-endorser",
                handshake_protocols=[
                    "https://didcomm.org/didexchange/1.0"
                ]
            )
        )

        logger.info(f"OOB Invitation URL: {oob_invitation.invitation_url}")

        # Step 2: Subwallet accepts the invitation
        logger.info("Subwallet is accepting the invitation...")
        accept_response = await subwallet_client.out_of_band.receive_invitation(
            body=oob_invitation.invitation,
            alias=f"{wallet_name}-from-multitenant",
            auto_accept=True  # Ensure the endorser agent auto-accepts the invitation

        )

        # Step 3: Update connection metadata from subwallet's perspective
        await subwallet_client.connection.set_metadata(
            conn_id=accept_response.connection_id,
            body=ConnectionMetadataSetRequest(
                metadata={
                    "my_role": "author",
                    "their_role": "endorser",
                    "connection_purpose": "endorsement",
                    "endorser_info": {
                        "label": "Endorser",
                        "endorser_did": "<endorser-did>",
                        "endpoint": "http://endorser_agent:8000"
                    },
                    "transaction_jobs": {
                        "transaction_my_job": "TRANSACTION_AUTHOR",
                        "transaction_their_job": "TRANSACTION_ENDORSER"
                    }
                }
            )
        )

        logger.info(f"Connection established successfully. Connection ID: {accept_response.connection_id}")

        return accept_response.connection_id

    except Exception as e:
        logger.error(f"Error in create_did_exchange_invitation: {str(e)}")
        raise

Then I try to Create my DID and write it to the ledger:
async def create_and_register_subwallet_did(
        subwallet_client: AcaPyClient,
        endorser_client: AcaPyClient,
        connection_id: str,
        name: str
) -> DID:
    try:
        # Create a new DID in the subwallet
        create_did_response = await subwallet_client.wallet.create_did(DIDCreate())
        did_info = create_did_response.result
        logger.info(f"Created new DID in subwallet: {did_info.did}")

        # Register the DID on the ledger using the endorser
        register_response = await subwallet_client.ledger.register_nym(
            create_transaction_for_endorser=True,
            conn_id=connection_id,
            did=did_info.did,
            verkey=did_info.verkey,
            alias=name
        )

        logger.info(f"Transaction initiated. Response: {register_response}")

        # Wait for the transaction to be endorsed by the endorser
        max_attempts = 10
        attempt = 0

        while attempt < max_attempts:
            transaction_record = await subwallet_client.endorse_transaction.get_transaction(
                register_response.txn.transaction_id
            )
            logger.info(f"Current transaction state: {transaction_record.state}")

            if transaction_record.state == "transaction_acked":
                await subwallet_client.endorse_transaction.write_transaction(
                    tran_id=register_response.txn.transaction_id)
            if transaction_record.state == "transaction_endorsed":
                logger.info("Transaction has been endorsed successfully.")
                break

            await asyncio.sleep(2)  # Wait before checking again
            attempt += 1

        if transaction_record.state != "transaction_endorsed":
            raise Exception("Transaction endorsement failed or timed out.")

        return did_info

    except Exception as e:
        logger.error(f"Error in create_and_register_subwallet_did: {str(e)}")
        raise

I am creating a wallet in this way:
async def create_new_wallet(multitenant_client: AcaPyClient, wallet_name: str) -> Tuple[str, str]:
    """Create a new wallet and return its token and wallet ID."""
    response = await multitenant_client.multitenancy.create_wallet(
        body=CreateWalletRequest(
            wallet_name=wallet_name,
            wallet_key=f"{wallet_name}_key",
            wallet_type="askar",
            label=wallet_name,
            wallet_webhook_urls=[
                <webhook-url>
            ]
        ),
    )
    logger.info(f"Created subwallet '{wallet_name}' with token: {response.token}")
    return response.token, response.wallet_id

These are my logs:
2024-09-30 17:07:14,907 - INFO - Wallet 'Test16' already exists.
2024-09-30 17:07:14,952 - INFO - OOB Invitation URL: <invitation>
2024-09-30 17:07:14,952 - INFO - Subwallet is accepting the invitation...
2024-09-30 17:07:15,048 - INFO - Connection established successfully. Connection ID: 83d21958-8171-48da-b723-64cbe7926308
2024-09-30 17:07:15,113 - INFO - Created new DID in subwallet: 24mRMrwnLH868jtSpSRxNR
2024-09-30 17:07:16,247 - INFO - Transaction initiated. Response: success=False txn=TransactionRecord(type='https://didcomm.org/sign-attachment/1.0/signature-request', connection_id='83d21958-8171-48da-b723-64cbe7926308', created_at='2024-09-30T15:07:16.230785Z', endorser_write_txn=False, formats=[{'attach_id': 'f95e7cba-b644-426e-9bb6-5e94e9bb266d', 'format'<EMAIL_ADDRESS>messages_attach=[{'@id': 'f95e7cba-b644-426e-9bb6-5e94e9bb266d', 'mime-type': 'application/json', 'data': {'json': '{"did": "did", "verkey": "verkey", "alias": "Test16", "role": null}'}}], meta_data={'did': '24mRMrwnLH868jtSpSRxNR', 'verkey': 'verkey', 'alias': 'Test16', 'role': None}, signature_request=[{'context': 'did:sov', 'method': 'add-signature', 'signature_type': 'default', 'signer_goal_code': 'aries.transaction.ledger.write_did', 'author_goal_code': 'aries.transaction.register_public_did'}], signature_response=[], state='request_sent', thread_id=None, timing={'expires_time': None}, trace=False, transaction_id='b636bd86-3656-47bf-b6e5-e807e260d1d6', updated_at='2024-09-30T15:07:16.235174Z')
2024-09-30 17:07:16,255 - INFO - Current transaction state: request_sent
2024-09-30 17:07:18,269 - INFO - Current transaction state: transaction_acked
2024-09-30 17:07:20,281 - INFO - Current transaction state: transaction_acked
2024-09-30 17:07:22,292 - INFO - Current transaction state: transaction_acked

These are the ACA-py logs:
endorser_agent-1     | Created new invitation
multitenant_agent-1  | Created new connection record from invitation
multitenant_agent-1  | Created connection request
multitenant_agent-1  | Sent connection request
endorser_agent-1     | Receiving connection request
endorser_agent-1     | Received connection request from invitation
endorser_agent-1     | Creating connection response
endorser_agent-1     | Created connection response
endorser_agent-1     | Sent connection response
multitenant_agent-1  | Accepted connection response
multitenant_agent-1  | Sent connection complete
endorser_agent-1     | Connection promoted to active
endorser_agent-1     | Received connection complete

Then it logs this:
onse status 400, caused by: Bad Request'), <traceback object at 0x7f3ca0463480>); Re-queue failed message ...
multitenant_agent-1  | 2024-09-30 15:07:18,136 aries_cloudagent.wallet.routes ERROR Error promoting to public DID: No endorser connection found
multitenant_agent-1  | Traceback (most recent call last):
multitenant_agent-1  |   File "/home/aries/.local/lib/python3.12/site-packages/aries_cloudagent/wallet/routes.py", line 1365, in on_register_nym_event
multitenant_agent-1  |     _info, attrib_def = await promote_wallet_public_did(
multitenant_agent-1  |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
multitenant_agent-1  |   File "/home/aries/.local/lib/python3.12/site-packages/aries_cloudagent/wallet/routes.py", line 861, in promote_wallet_public_did
multitenant_agent-1  |     raise web.HTTPBadRequest(reason="No endorser connection found")

I still think I am just missing something small as I am staring at logs all day.

@Executioner1939 it looks like you may need to set an endorser-alias in the startup parameters for the agent to be able to pick-up the right connection to the endorser (see here.

@esune @swcurran thank you for the help, it seems I finally have it working.
I disabled  --auto-promote-author-did on the Multitenant Agent.
I also enabled  --requests-through-public-did on the Endorser Agent.
Then I do the following (below code is very messy):

Create the Connection to the Endorser and when it's active I set the Endorser Role and Information.

async def create_endorser_connection(subwallet_client: AcaPyClient, wallet_name: str) -> str:
    """Create a DID Exchange invitation with custom roles and establish a connection."""

    try:
        request = await subwallet_client.did_exchange.create_request(
            their_public_did="NF8U1Dp9xrRikXisTJuaEs",
            alias=f"{wallet_name}-from-multitenant",
            my_endpoint="http://multitenant_agent:8000"

        )

        max_attempts = 10
        attempt = 0

        while attempt < max_attempts:
            conn = await subwallet_client.connection.get_connection(conn_id=request.connection_id)
            logger.info(f"Current connection state: {conn.state}")

            if conn.state == "active":
                await subwallet_client.endorse_transaction.set_endorser_role(conn_id=conn.connection_id, transaction_my_job="TRANSACTION_AUTHOR")
                await subwallet_client.endorse_transaction.set_endorser_info(conn_id=conn.connection_id, endorser_did="<endorser-did>")
                return conn.connection_id

        await asyncio.sleep(2)
        attempt += 1

        logger.info(f"Connection established successfully. Connection ID: {conn.connection_id}")
        return conn.connection_id

    except Exception as e:
        logger.error(f"Error in create_did_exchange_invitation: {str(e)}")
        raise

Then once it's created properly, I create the DID and Write it to the Ledger and set it to public using the Endorser

async def create_and_register_subwallet_did(
        subwallet_client: AcaPyClient,
        connection_id: str,
        name: str
) -> DID:
    try:
        # Create a new DID in the subwallet
        create_did_response = await subwallet_client.wallet.create_did(DIDCreate())
        did_info = create_did_response.result
        logger.info(f"Created new DID in subwallet: {did_info.did}")

        # Register the DID on the ledger using the endorser
        register_response = await subwallet_client.ledger.register_nym(
            create_transaction_for_endorser=True,
            conn_id=connection_id,
            did=did_info.did,
            verkey=did_info.verkey,
            alias=name
        )
        logger.info(f"Transaction initiated. Response: {register_response}")

        max_attempts = 10
        attempt = 0

        while attempt < max_attempts:
            transaction_record = await subwallet_client.endorse_transaction.get_transaction(
                register_response.txn.transaction_id
            )
            logger.info(f"Current transaction state: {transaction_record.state}")

            if transaction_record.state == "transaction_acked":
                await subwallet_client.wallet.set_public_did(did=did_info.did, conn_id=connection_id, create_transaction_for_endorser=True)
                return did_info

            await asyncio.sleep(2)
            attempt += 1

        return did_info

    except Exception as e:
        logger.error(f"Error in create_and_register_subwallet_did: {str(e)}")
        raise

Then using the connection I am able to create Schemas and Credential Definitions.

That’s great. We need to get an update to the Endorser and/or Multitenancy documentation to capture what is in here.

Sorry I'm late to the party.  Here are a number of references that would likely have helped:

hyperledger/aries-endorser-service - Endorser Configuration Documentation
hyperledger/aries-cloudagent-python - Connecting to an Endorser Service
bcgov/dts-endorser-service Repo

OpenShift Templates
./manage registerAuthor script for automating the author-endorser connection.

I think @loneil might be able to point at some examples of how the connections are established in Traction too.

@WadeBarnes thanks for the additional information, I actually stumbled across the second link you posted which helped.
A few clarifying questions:

Once I have created this connection, I will need to use it for all transactions for this wallet going forward, including Credential Issuance and so on?
Are you using that registerAuthor script in k8s? If were to do the following, with an initContainer would I only need to ensure a single connection is created? Or should it be for each Tenant?

Once I have created this connection, I will need to use it for all transactions for this wallet going forward, including Credential Issuance and so on?

Yes, it's a dedicated (named) connection between the author and endorser.

Are you using that registerAuthor script in k8s? If were to do the following, with an initContainer would I only need to ensure a single connection is created? Or should it be for each Tenant?

Each tenant requires a connection, I believe.  We were using the script initially to setup standalone agents.  @loneil would have details of how that's configured and managed in Traction.
You might find it difficult to configure an endorser connection with and initContainer since it requires interaction between the author and endorser including some final configuration on both sides once the connection is established.  If you were to do it with an initContainer you'd need to ensure only the single connection is made.  If you look at the script is goes out of it's way to ensure there is a single connection and if anything goes wrong, that same connection is detected before the process continues.

@WadeBarnes thanks for the information, that makes sense.
I think we can mark this issue as solved. The information in this document served perfectly: https://github.com/hyperledger/aries-cloudagent-python/tree/main/demo/docker-agent#connecting-to-an-endorser-service


Fix: PADV-646 - Default values for first access and last access.

Description
This PR is a fix of the implementation made in https://agile-jira.pearson.com/browse/PADV-646.
Changes made

[x] Add validation in first_access and last_access columns.

Screenshots
The error is when the endpoint returns a null value for first access and last_access. The method Date(row.values.first_access).toUTCString() returns a default Date. For example, as you can see, there are rows with values from the year 1970.

The desired behavior is that if those values are not there, it does not return anything as shown in the following screenshot:

How to test

Clone the Institution Portal MFE
Run npm install.
Run npm run start

@anfbermudezme could you please apply a default value for created column?

@anfbermudezme please mention Created column in Changes made section.


fix(TDI-40005): fix connection issue, fix integration tests.

What is the current behavior? (You can also link to an open issue here)
https://jira.talendforge.org/browse/TDI-40005
What is the new behavior?
Please check if the PR fulfills these requirements

[x] The commit message follows Talend standard
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features) ?
[ ] The code coverage on new code >75%
[ ] The new code does not introduce new technical issues (sonar / eslint)

What kind of change does this PR introduce?

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build / CI related changes
[ ] Other... Please describe:

Does this PR introduce a breaking change?

[ ] Yes
[x] No

If this PR contains a breaking change, please describe the impact and migration path for existing applications: ...
Other information:

Pull request test result: success :star:
Sonar results


How to set a specific GPU device to use when running multiple thread?

I am implementing A3C algorithm with CARLA, but by default, it always use GPU0.
How set a specific GPU device to use?

@mvpcom
Thanks, It's working to me also, I would like to use 3 GPUs and 8 GPUs for a single docker container, Is it possible to implement in Turicreate and Tensorflow?
Thanks,

Hi @felipecode @nsubiron is there any specific reason for Carla server to have extremely low fps on a remote gpu server with multiple GPU but works fine on a laptop with single GPU and lower configuration in comparison to the GPU server


Add a hwdb format option

This new subcommand generates an entry in the iocost hardware database
format for the given merged result. It is used by the CI pipeline to
generate the hwdb file that will be used by systemd.
Signed-off-by: Gustavo Noronha Silva<EMAIL_ADDRESS>

Not 100% happy with having to look at the props to make opts decisions, but I can't think of any way other than that. Should we add a second format subcommand that is undecorated, for instance?
Another thing I'd like your input on is on the default solutions. The reason I did that is as we've seen on some of the results some solutions are missing, so I thought we should have a list of preferred solutions and get the best one we find. I expect this will not be a big issue once we have enough results, as it'll be rare that a merged result is missing a solution, but still good to have it handled properly.

re. undecoreated, I think what you did is fine. Eventually, what we need to do is separating out the format option parse phase into a separate callback so that the benchmark has a say in whether a given format command should be decorated or not in a non-hacky way but this doesn't concern the user, so this can and should be handled internally.
So, I'm not sure whether having default makes sense at hwdb generation step. It really should contain all the solutions generated for the device. In terms of work-conersvancy (if the aim is not losing any work that the ssd can do), the order would be no control -> naive -> bandwidth -> isolated-bandwidth -> isolation. If the aim is isolation, the order is reversed. It makes sense to present the ordering but I'm not sure whether whether we can pick one solution as "the default" at hwdb generation time.

haha I was just writing a message to suggest we land this version =D thank you @htejun!


test_wal_acceptor_async.py::test_race_conditions is flaky

Example failed runs:
https://github.com/neondatabase/neon/runs/7739496574?check_suite_focus=true
https://github.com/neondatabase/neon/runs/7758969477?check_suite_focus=true

Looked at the pageserver logs in the first run, seems similar to walreceiver issues which can be fixed with #2253.
In the logs pageserver spends a minute without an active walreceiver connection:
2022-08-09T05:21:45.095713Z  INFO walreceiver_connection{id=6141cd2f21fa4662a3720426c4cc99de/a7f8176ed947d05cd806ea3c12aa734d}: IdentifySystem { systemid:<PHONE_NUMBER>689875055, timeline: 1, xlogpos: 0/16D5DB0, dbname: None }
2022-08-09T05:21:45.095847Z  INFO walreceiver_connection{id=6141cd2f21fa4662a3720426c4cc99de/a7f8176ed947d05cd806ea3c12aa734d}: last_record_lsn 0/16D1CD8 starting replication from 0/16D1CD8, server is at 0/16D5DB0...
2022-08-09T05:21:45.096708Z  INFO walreceiver_connection{id=6141cd2f21fa4662a3720426c4cc99de/a7f8176ed947d05cd806ea3c12aa734d}: caught up at LSN 0/16D5DB0
2022-08-09T05:21:45.229354Z  INFO walreceiver_connection{id=6141cd2f21fa4662a3720426c4cc99de/a7f8176ed947d05cd806ea3c12aa734d}:safekeeper_handle_db: Connection closed regularly: connection closed
2022-08-09T05:21:45.229486Z  WARN wal_connection_manager{id=6141cd2f21fa4662a3720426c4cc99de/a7f8176ed947d05cd806ea3c12aa734d}: WAL receiving task failed: walreceiver connection handling failure: connection closed
2022-08-09T05:22:46.033198Z  INFO wal_connection_manager{id=6141cd2f21fa4662a3720426c4cc99de/a7f8176ed947d05cd806ea3c12aa734d}: Switching to new connection candidate: NewWalConnectionCandidate { safekeeper_id: NodeId(3), wal_source_connstr: "host=<IP_ADDRESS> port=18165 options='-c ztimelineid=a7f8176ed947d05cd806ea3c12aa734d ztenantid=6141cd2f21fa4662a3720426c4cc99de'", reason: NoExistingConnection }
2022-08-09T05:22:46.033256Z  INFO walreceiver_connection{id=6141cd2f21fa4662a3720426c4cc99de/a7f8176ed947d05cd806ea3c12aa734d}: connecting to host=<IP_ADDRESS> port=18165 options='-c ztimelineid=a7f8176ed947d05cd806ea3c12aa734d ztenantid=6141cd2f21fa4662a3720426c4cc99de'
2022-08-09T05:22:46.033764Z  INFO walreceiver_connection{id=6141cd2f21fa4662a3720426c4cc99de/a7f8176ed947d05cd806ea3c12aa734d}:safekeeper_handle_db: connected!

Shouldn't repeat after latest walreceiver patches.


Tiny refactor of pattern search form

This is just to pull a couple minor changes out of the "Needs ID" PR.
Does not alter the front end in any material way. (Alters nesting of an icon and a text field.)

Coverage: 93.549%. Remained the same when pulling e52671e0bade47e048495fbebe49ac42f05c134b on nimmo-tiny-search-form-refactor into 567a29de19e17826001ed2c2e38a2d752787ea2f on main.


Implement beam search for CopyNet

Coverage decreased (-0.7%) to 97.424% when pulling 6c5eff1d313880a0c8517a7734245804d1ccfc91 on ISSUE-9 into 16b64395c6888a73e76f7822acf219c64f62e995 on master.

Codecov Report

Merging #10 into master will decrease coverage by 2.34%.
The diff coverage is 91.66%.

@@            Coverage Diff             @@
##           master      #10      +/-   ##
==========================================
- Coverage   98.07%   95.73%   -2.35%
==========================================
  Files          10        7       -3
  Lines         312      258      -54
==========================================
- Hits          306      247      -59
- Misses          6       11       +5

Impacted Files
Coverage Δ

modules/nn/beam_search.py
91.66% <91.66%> (ø)

modules/tests/data/dataset_readers/copynet_test.py

...odules/tests/data/tokenizers/word_splitter_test.py

modules/tests/models/copynet_test.py

modules/tests/data/dataset_readers/nl2bash_test.py

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 102bbd1...441f16b. Read the comment docs.


Proxy: Excessive EADDRNOTAVAIL errors when connect()ing to upstreams

Is there an existing issue for this?

[x] I have searched the existing issues

Version
equal or higher than v1.16.4 and lower than v1.17.0
What happened?
tl;DR: cilium-envoy is intermittently returning 503s due to ephemeral port collision.
In our production environment:

EKS 1.29
Bottlerocket 1.27

Linux 6.1.115

Kong Gateway Enterprise v<IP_ADDRESS>
Cilium v1.16.2

requests going through our Kong dataplane pods are returning 503 responses with the header x-envoy-response-code-details: upstream_reset_before_response_started{remote_connection_failure,immediate_connect_error:_Cannot_assign_requested_address}.
Using retsnoop, we've narrowed the connect failures down to __inet_check_established propagating an EADDRNOTAVAIL error (ostensibly because of ephemeral port collision):
# /retsnoop -T -e  '__sys_connect' -x EADDRNOTAVAIL -a '*connect*' -a '*sock*' -a '*inet*'

18:15:49.710872 -> 18:15:49.710987 TID/PID 6424/2869 (wrk:worker_11/cilium-envoy):

FUNCTION CALLS                                               RESULT             DURATION
----------------------------------------------------------   ----------------  ---------
→ __sys_connect
    → __sys_connect_file
        → security_socket_connect
            → selinux_socket_connect
                → selinux_socket_connect_helper.isra.0
                    ↔ sock_has_perm                          [0]                 2.654us
                ← selinux_socket_connect_helper.isra.0       [0]                11.877us
                → selinux_netlbl_socket_connect
                    ↔ lock_sock_nested                       [void]              0.466us
                    ↔ selinux_netlbl_socket_connect_locked   [0]                 4.497us
                    ↔ release_sock                           [void]              4.318us
                ← selinux_netlbl_socket_connect              [0]                23.669us
            ← selinux_socket_connect                         [0]                41.968us
            ↔ bpf_lsm_socket_connect                         [0]                 2.304us
        ← security_socket_connect                            [0]                55.632us
        → inet_stream_connect
            ↔ lock_sock_nested                               [void]              0.436us
            → __inet_stream_connect
                → tcp_v4_connect
                    ↔ inet_sk_state_store                    [void]              4.633us
                    → inet_hash_connect
                        → __inet_hash_connect
                            → __inet_check_established
                                ↔ inet_ehashfn               [0x569ed867]        0.517us
                            ← __inet_check_established       [-EADDRNOTAVAIL]    2.103us
                        ← __inet_hash_connect                [-EADDRNOTAVAIL]    9.944us
                    ← inet_hash_connect                      [-EADDRNOTAVAIL]   12.648us
                    ↔ inet_unhash                            [void]              0.767us
                    ↔ inet_sk_state_store                    [void]              0.321us
                    ↔ inet_bhash2_reset_saddr                [void]              2.318us
                ← tcp_v4_connect                             [-EADDRNOTAVAIL]   34.722us
            ← __inet_stream_connect                          [-EADDRNOTAVAIL]   37.360us
            ↔ release_sock                                   [void]              0.422us
        ← inet_stream_connect                                [-EADDRNOTAVAIL]   41.439us
    ← __sys_connect_file                                     [-EADDRNOTAVAIL]  102.672us
← __sys_connect                                              [-EADDRNOTAVAIL]  111.272us

We're fairly sure that the involvement of Kong is correlation and not causation; we think it's Envoy load-related and Kong is just the thing generating said load.
We've got retries enabled on the relevant CECs, but we experience bursts of collisions that exceed the retry limits and we're hesitant to dial up the retry count further to avoid causing a thundering herd scenario.
How can we reproduce the issue?

Install Cilium with our config (attached)
Create a service with a CEC
Run a large number of requests through cilium-envoy via L7 CECs

Cilium Version
v1.16.2
Kernel Version
6.1.115
Kubernetes Version
1.29
Regression
No response
Sysdump
cilium-sysdump-20241206-101641.zip
Relevant log output

Anything else?
api-external.svc.txt
api.cec.txt
api.svc.txt
I've got a ~100MB trace log file as well that I likely cannot upload here; I can provide that via Slack if needed.
Cilium Users Document

[ ] Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

[x] I agree to follow this project's Code of Conduct

Do you have this problem when using "ordinary" L7 filters, or only with the CiliumEnvoyConfig?

Hmm - define "ordinary L7 filters"? If you can provide some alternate config to try, I can see if I can reproduce in test.
In doing some more investigation, we think this issue might be related to https://github.com/cilium/cilium/issues/30406 in that making multiple concurrent requests to a ClusterIP service with a small number of backend pods seems to be the primary trigger - but we're not exhausting ports (since we never go above ~2k total sockets on the host), so there's likely a 5-tuple collision happening.
We've been flipping some toggles back and forth on the CECs, notably connectionPoolPerDownstreamConnection which alleviates the issue most of the time, but we still get random 5-8 minute stretches of intermittent Cannot assign requested address errors (we've nicknamed this particular error CARA) on the nodes running Kong when we have that enabled.
We noticed when running tcpconnect on a system where EADDRNOTAVAIL was being thrown, cilium-envoy was reusing the source address and port from the downstream connection:
189.031  185530  reactor-http 4  <IP_ADDRESS>   44052  <IP_ADDRESS>   80
189.034  3715    wrk:worker_3 4  <IP_ADDRESS>   44052  <IP_ADDRESS>   8888
189.038  185530  reactor-http 4  <IP_ADDRESS>   49828  <IP_ADDRESS>     80
189.040  3715    wrk:worker_2 4  <IP_ADDRESS>   49828  <IP_ADDRESS>   8888
189.042  185530  reactor-http 4  <IP_ADDRESS>   51312  <IP_ADDRESS>   80
189.044  3715    wrk:worker_3 4  <IP_ADDRESS>   51312  <IP_ADDRESS>    8888
189.156  185530  reactor-http 4  <IP_ADDRESS>   51324  <IP_ADDRESS>   80
189.156  185530  reactor-http 4  <IP_ADDRESS>   51326  <IP_ADDRESS>   80
189.156  185530  reactor-http 4  <IP_ADDRESS>   51328  <IP_ADDRESS>   80
189.157  185530  reactor-http 4  <IP_ADDRESS>   51334  <IP_ADDRESS>   80
189.157  185530  reactor-http 4  <IP_ADDRESS>   51336  <IP_ADDRESS>   80
189.157  185530  reactor-http 4  <IP_ADDRESS>   51350  <IP_ADDRESS>   80
189.157  185530  reactor-http 4  <IP_ADDRESS>   51356  <IP_ADDRESS>   80
189.157  185530  reactor-http 4  <IP_ADDRESS>   51370  <IP_ADDRESS>   80
189.157  185530  reactor-http 4  <IP_ADDRESS>   51378  <IP_ADDRESS>   80
189.158  185530  reactor-http 4  <IP_ADDRESS>   51382  <IP_ADDRESS>   80
189.158  185530  reactor-http 4  <IP_ADDRESS>   51396  <IP_ADDRESS>   80
189.159  185530  reactor-http 4  <IP_ADDRESS>   51404  <IP_ADDRESS>   80
189.159  185530  reactor-http 4  <IP_ADDRESS>   51410  <IP_ADDRESS>   80
189.160  185530  reactor-http 4  <IP_ADDRESS>   51414  <IP_ADDRESS>   80
189.160  185530  reactor-http 4  <IP_ADDRESS>   51426  <IP_ADDRESS>   80
189.160  185530  reactor-http 4  <IP_ADDRESS>   51436  <IP_ADDRESS>   80
189.164  3715    wrk:worker_3 4  <IP_ADDRESS>   51324  <IP_ADDRESS>    8888
189.165  3715    wrk:worker_2 4  <IP_ADDRESS>   51334  <IP_ADDRESS>    8888
189.165  3715    wrk:worker_3 4  <IP_ADDRESS>   51326  <IP_ADDRESS>    8888
189.166  3715    wrk:worker_3 4  <IP_ADDRESS>   51328  <IP_ADDRESS>    8888
189.166  3715    wrk:worker_0 4  <IP_ADDRESS>   51336  <IP_ADDRESS>    8888
189.168  3715    wrk:worker_3 4  <IP_ADDRESS>   51350  <IP_ADDRESS>    8888
189.168  3715    wrk:worker_2 4  <IP_ADDRESS>   51370  <IP_ADDRESS>    8888
189.169  3715    wrk:worker_0 4  <IP_ADDRESS>   51378  <IP_ADDRESS>    8888
189.170  3715    wrk:worker_3 4  <IP_ADDRESS>   51356  <IP_ADDRESS>    8888
189.172  3715    wrk:worker_0 4  <IP_ADDRESS>   51410  <IP_ADDRESS>    8888
189.172  3715    wrk:worker_3 4  <IP_ADDRESS>   51382  <IP_ADDRESS>    8888
189.173  3715    wrk:worker_1 4  <IP_ADDRESS>   51414  <IP_ADDRESS>    8888
189.173  3715    wrk:worker_3 4  <IP_ADDRESS>   51396  <IP_ADDRESS>    8888
189.174  3715    wrk:worker_3 4  <IP_ADDRESS>   51404  <IP_ADDRESS>    8888
189.174  3715    wrk:worker_0 4  <IP_ADDRESS>   51426  <IP_ADDRESS>    8888
189.175  3715    wrk:worker_1 4  <IP_ADDRESS>   51436  <IP_ADDRESS>    8888
189.210  3715    wrk:worker_3 4  <IP_ADDRESS>   51312  <IP_ADDRESS>    8888```

so we thought that the 5-tuple collision was potentially from multiple different envoy threads/connpools getting multiple requests over the same http/1.1 downstream connection and colliding that way? But even when setting `--concurrency=1`, if we don't set `connectionPoolPerDownstreamConnection`, we still see CARAs.

Looks like this might be a subset of https://github.com/cilium/cilium/issues/25613.

With our original setup, if I run fortio load -allow-initial-errors -abort-on 599 -qps 0 -c 20 -n 10000 http://destination-svc; I get ~15% 503 errors with EADDRNOTAVAIL return values from connect.
So if I enable a CiliumNetworkPolicy as such:
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: http
  namespace: destination-service-ns
spec:
  endpointSelector: {}
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: source-service
            io.kubernetes.pod.namespace: source-service-ns
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
            - port: "8888"
              protocol: TCP
            - port: "80"
              protocol: TCP
          rules:
            http: [{}]

and remove the CiliumNetworkPolicy, the fortio command runs to completion.

I need to double-check after re-baselining the configuration in my test cluster, but it appears that the major difference is that when requests go through the normal envoy listener, none of the sockets connecting to upstreams bind, so the collision scenario no longer exists.

Resolved by setting cilium.io/use-original-source-address: false on our CECs to prevent cilium-envoy from binding to to original source/port of the upstream request. Frustrating that this flag/behavior was undocumented, but happy to have a fix.


First code example doesn't work

There's no NavigationStrategy property on this class:
    var options = new FindNextElementOptions ()
    {
        SearchRoot = TicTacToeGrid,
        NavigationStrategy = NavigationStrategyMode.Heuristic
    };

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: b8acece5-c4d8-8247-49ad-93317d0de2d5
Version Independent ID: ebe3f26a-5bda-296f-237d-477145eab671
Content: Programmatic focus navigation with keyboard, gamepad, and accessibility tools - Windows UWP applications
Content Source: windows-apps-src/design/input/focus-navigation-programmatic.md
Product: uwp
GitHub Login: @Karl-Bridge-Microsoft
Microsoft Alias: kbridge

Thanks @michael-hawker - snippet has been fixed.


Feature/add need status

Add need.status regarding task:
https://trello.com/c/tVy0goVC/374-13-blocked-need-status
Please also check filters, export and need navs (Assigned to me and Unmet needs).
Please take a note we have fallback need label "To do" when the new status column is null. We can not filter by it.
https://github.com/wearefuturegov/beacon/pull/161/commits/947306db484c68e444657307d9381d9430714398 is a rake task. Rake tasks are one time task running in Rails project environment. In our case it means we dont care about database configurations because will come from Rials env. To run this rake task:
i-have-i-need$ bundle exec rake needs_status:update_needs_statuses

Hmm, seems to work for me

The unmet needs filter seems to be broken.
To recreate click unmet needs on the left - I get no matches
Or update the filters manually.

Please take a note we have fallback need label "To do" when the new status column is null. We can not filter by it. Is that the case?

 13. Blocked “Need Status”

Need to improve the rake before merge to master.

Fix to ensure if we run rake task twice:
https://github.com/wearefuturegov/beacon/pull/161/commits/be211966a6e66400fa7a4ad9f8051ac6921a34b3


Replaced HexView with Data Editor

Removed instances of the HexView and replaced with the Data Editor.
Added revised DFDLDebugger <--> Data Editor Extension/UI message structure.

Closes #790
Closes #905

@stricklandrbls Can you also add that this closes https://github.com/apache/daffodil-vscode/issues/905? Also I am not able to open the data editor I am getting an error for creating the session and viewport.

Before I had ${workspaceFolder}, so I figured maybe that variable wasn't being replaced with actual workspace folder. I am not sure if before you are updating that before trying to open that file but either way using ${workspaceFolder} or giving it the full path to my file it gives the same error as pictured in the screenshot.

@stricklandrbls So I don't think it does with that because I have nothing running, I had another service on 9000 that I killed. But now when I try to run it says its logging to /Users/sdell/Library/Application\ Support/omega_edit/dataEditor-9010.log and when I lookup the omega-edit process using ps -ef | grep "omega" I see it is being ran with --port 9010. However, the connection is trying to connect to port 9000 instead of 9010. So, there is a disconnect somewhere on the port.
In the launch config to have this snippet:
...
            "dataEditor": {
                "port": 9001,
                "logging": {
                    "file": "${workspaceFolder}/dataEditor-${omegaEditPort}.log",
                    "level": "info"
                }
            }
...

It will still say /Users/sdell/Library/Application\ Support/omega_edit/dataEditor-9010.log, the omega-edit process using is still being ran with --port 9010and the server still tries hitting 9000 not 9010. So, this configuration is not being honored. If I changeDEFAULT_OMEGA_EDIT_PORTinside ofsrc/dataEditor/dataEditorClient.ts` I can open the data editor.

So I found what my issue was, if you configure the dataEditor item multiple times it takes the last instance that shows up. So I had this in my launch.json:
            ...
            "dataEditor": {
                "port": 9001,
                "logging": {
                    "file": "${workspaceFolder}/dataEditor-${omegaEditPort}.log",
                    "level": "info"
                }
            ...
            "dataEditor": {
                "port": 9010,
                "logFile": "${workspaceFolder}/dataEditor-${omegaEditPort}.log",
                "logLevel": "info"
            }
           ...

So that is why it was always trying to use 9010. If I remove that and try again everything is working. However, one thing is that if I change that port I need to reload the extension before it will actually pick it up for some reason.

Okay so found the issue is me using ${workspaceFolder}. So if I do ${workspaceFolder}/dfdl-pcap/src/test/resources/com/tresys/pcap/data/dns.cap it fails but if I remove ${workspaceFolder} and give it the full path it works fine. So, this PR needs updated to check for ${workspaceFolder} in the path and then replace it with the actual path. There are other places in the code where this done so shouldn't be too bad to add.

@shanedell

Okay so found the issue is me using ${workspaceFolder}. So if I do ${workspaceFolder}/dfdl-pcap/src/test/resources/com/tresys/pcap/data/dns.cap it fails but if I remove ${workspaceFolder} and give it the full path it works fine.

I'm not sure if that's what causing your issues because my launch.json is also using ${workspaceFolder} but my only issue is a lingering Omega Edit server when closing the window.

@shanedell

Okay so found the issue is me using ${workspaceFolder}. So if I do ${workspaceFolder}/dfdl-pcap/src/test/resources/com/tresys/pcap/data/dns.cap it fails but if I remove ${workspaceFolder} and give it the full path it works fine.

I'm not sure if that's what causing your issues because my launch.json is also using ${workspaceFolder} but my only issue is a lingering Omega Edit server when closing the window.
I think we may need to sync up and run through this because I tried all the changes you had made for resolving the failed viewport but it's works for me. We've got to locate the discrepancy.

I am not sure because whenever I used ${workspaceFolder} I get an error trying to create the session and viewport after a couple of seconds but when I remove it everything works fine.

@shanedell I wonder if somehow the URI is not properly being parsed and sent to the Data Editor / Omega Edit because there's a difference between how ${workspaceFolder} is resolved in Windows vs Linux/Mac?
I've been unable to test anything further because of my build issues (https://github.com/apache/daffodil-vscode/pull/926#discussion_r1442126852).

Yeah I am not sure why it doesn't work on Windows at all it seems but on Mac, which is what I am testing on, it does open up the Data Editor it just fails to fully create the session and viewport.

@stricklandrbls I was able to resolve by issue by updating src/dataEditor/dataEditorClient.ts to before doing
const dataEditorView = new DataEditorClient(
  ctx,
  'dataEditor',
  'Data Editor',
  fileToEdit
)

do
// Make sure to remove ${workspaceFolder} from fileToEdit
let rootPath = vscode.workspace.workspaceFolders
  ? vscode.workspace.workspaceFolders[0].uri.fsPath
  : vscode.Uri.parse('').fsPath
fileToEdit = fileToEdit.includes('${workspaceFolder}')
  ? fileToEdit.replace('${workspaceFolder}', rootPath)
  : fileToEdit

Then everything worked fine for me. Can you add this to the PR? I will approve it once that is added as I am not sure how yours is working on Linux if you have ${workspaceFolder} in the path for the data file as without adding the above snippet the ${workspaceFolder} is never removed from the path.

@shanedell

Can you add this to the PR?

Can do.
daffodil-debugger.bat Issue Resolved

Currently, I'm unable to actually get anything running because of this error I'm receiving while attempting to run the DFDL Debugger.

Apparently, during an installation of the Eclipse IDE, there was a proprietary_ish_ JRE installed named Eclipse Temurin JDK with HotSpot which was causing the java runtime error I had been seeing. I removed it, installed JDK 21 and the extension would still not completely initialize when starting the DFDL Debugger. Finally, I switch to JDK 17 and everything works just fine.
Current Windows Issue
I'm currently working on an issue on the Windows side while auto-starting the Data Editor on debug start. Manually opening the Data Editor from the VSCode command palette will work fine. I need to pinpoint why this is happening. Hopefully the fix @shanedell  mentioned will fix this as well.

@shanedell
The changes you requested have been made.
@lrbarber

Testing on Windows...
"openDataEditor=true" did not work for me.

I resolved my Windows build environment and only saw this issue on my initial DFDL debug session. My preceding attempts were all able to auto-open the data editor w/ the corresponding data file. Try pulling my latest changes and see if this resolves your issue.

@shanedell @scholarsmate @lrbarber
Could you all test this and see if you're seeing the same issue? I'm currently seeing an error with auto-starting the data editor where it will fail to create the viewport. It's odd because this log output mentions that a session is already started but these logs are from the first time starting the DFDL Debugger after booting Windows.
I've only seen this issue on Windows, as well.
Data Editor Log Output
[16:32:13.489] INFO (17896): logger set
    fn: "setLogger"
[16:32:13.495] WARN (17896):
    fn: "startServer"
    host: "<IP_ADDRESS>"
    port: 9420
    pidFile: "C:\Users\Orion\AppData\Roaming\xdg.data\omega_edit\serv-9420.pid"
    logConf: "C:\Users\Orion\AppData\Roaming\xdg.data\omega_edit\serv-9420.logconf.xml"
    err: {
      "msg": "pidFile already exists",
      "pid": 4608
    }
[16:32:39.101] ERROR (17896):
    fn: "createSession"
    err: {
      "msg": "13 INTERNAL: ",
      "details": "",
      "code": 13,
      "stack":
          Error: 13 INTERNAL:
                at t.callErrorFromStatus (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:5335)
                at Object.onReceiveStatus (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:30617)
                at Object.onReceiveStatus (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:26506)
                at Object.onReceiveStatus (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:25951)
                at c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:113039
                at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
          for call at
                at s.makeUnaryRequest (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:30158)
                at s.createSession (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:97633)
                at c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:625864
                at new Promise (<anonymous>)
                at c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:625841
                at Generator.next (<anonymous>)
                at s (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:624000)
                at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    }
[16:32:39.102] ERROR (17896):
    err: {
      "msg": "Failed to create session for c:\\Users\\Orion\\Documents\\GitHub\\PNG/basi0g01.png",
      "stack":
          Error:
                at x.<anonymous> (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:991929)
                at Generator.throw (<anonymous>)
                at a (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:983164)
                at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    }
[16:32:39.118] ERROR (17896):
    fn: "createViewport"
    err: {
      "msg": "5 NOT_FOUND: session not found",
      "details": "session not found",
      "code": 5,
      "stack":
          Error: 5 NOT_FOUND: session not found
                at t.callErrorFromStatus (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:5335)
                at Object.onReceiveStatus (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:30617)
                at Object.onReceiveStatus (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:26506)
                at Object.onReceiveStatus (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:25951)
                at c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:113039
                at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
          for call at
                at s.makeUnaryRequest (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:30158)
                at s.createViewport (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:97633)
                at c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:637185
                at new Promise (<anonymous>)
                at c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:637162
                at Generator.next (<anonymous>)
                at s (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:636177)
                at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    }
[16:32:39.118] ERROR (17896):
    err: {
      "msg": "Failed to create viewport for c:\\Users\\Orion\\Documents\\GitHub\\PNG/basi0g01.png",
      "stack":
          Error:
                at x.<anonymous> (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:992848)
                at Generator.throw (<anonymous>)
                at a (c:\Users\Orion\Documents\GitHub\daffodil-vscode\dist\ext\extension.js:2:983164)
                at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    }

@stricklandrbls I have had this happened yes, but it was only happening to me if omega-edit was already running in the background

Pulled latest version of code and still wasn't getting DataEditor to auto start.
Discovered that package.json actually has 4 places that set openDataEditor... I changed 3 of the 4 to true -- the one I left set to false is the one in initialConfigurations. With these changes made, the data editor does open as another tab when debugging starts.

Now I'm wondering if there might be a way to get it to open in split screen mode by default -- like this:

@lrbarber

With these changes made, the data editor does open as another tab when debugging starts.

Did you run the launch wizard prior to starting the DFDL Debugger? I'd think that those openDataEditor config variables would get overridden when configuring the launch.json within whichever DFDL repo the debugger is being ran from.
Having the data editor not automatically open by default was what I assumed to be expected behavior. Given that those config options were just text replacements from the openHexView.
Try debugging from a fresh DFDL repo after running the launch wizard and setting the Open Data Editor checkbox.

Now I'm wondering if there might be a way to get it to open in split screen mode by default

I've been meaning to look into this. The HexView did this but the Data Editor replacement does not. I have just been preoccupied with trying to debug the Windows issues.

@shanedell I just pushed those changes. Thanks for that! Saved me having to use time to pinpoint it.

It has been intermittently working in windows environment. Every time I let the extension completely compile and package before starting the Extension debug launch command, it always works first few times, data editor opens up and debug workflow works correctly.  When I restart the debug session, Data Editor is unable to open OmegaEdit session and viewport and I am getting following errors.

Same behavior occurs in main branch as well with the Data Editor. It seems like this issue is pre-existing in Windows environment  even before this PR was created. The intended feature works consistently in Linux and Mac environment.

Like @hdalsania mentiones I believe that code in here is good. I honestly think there maybe is a bug on the omega-edit side that is causing the issues. Mostly an issue of it not killing the server in some instances for kind of unknown reasons. So in my option is that this gets merged in once @stricklandrbls gets another binding +1 and then we can take a better look at the omega-edit side and see if we can figure out whats going wrong with killing the server.
I believe the killing of the server has always been a headache especially in windows because I think the way its done in windows may not always kill it. But then in Mac and Linux I think it does pretty good for the most part other than if you stop debugging the extension before exiting the Data Editor I think the server doesn't get killed.


Work Flows separated static analysis.

@Jean85 can you review this change?
And how do you look to add more test on Symfony versions? Сurrently missing on version Symfony 5.

@Jean85 can you review this change?
And how do you look to add more test on Symfony versions? Сurrently missing on version Symfony 5.

Sure, please add a 5.x job

@Jean85 in this pr https://github.com/getsentry/sentry-symfony/pull/404 - update check in more Symfony version.

@ste93cry In php-cs has problem hi not support run in php 8.0 - https://github.com/Yozhef/PHP-CS-Fixer/blob/2.17/php-cs-fixer#L26
I'll create a fix for them but I'll leave 7.4 here for now

https://github.com/FriendsOfPHP/PHP-CS-Fixer/pull/5394

In php-cs has problem hi not support run in php 8.0

It it supported to run the tool on PHP 8 if a certain environment variable is set, what is not supported is running the tool on code written using PHP 8 features. Anyway, it's fine for us to run the tool on PHP 7.4 👍


Change the default version used by kube-inject to be the full SHA

What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:
Release note:

None

/retest

@chxchx: The following test failed, say /retest to rerun them all:

Test name
Commit
Details
Rerun command

prow/pilot-e2e-smoketest.sh
3eb117debddca35484902478e1ce6a6ecad4a002
link
/test pilot-e2e-smoketest

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest
~shriram

wonder how we test this but... merging!

just realized it's still wrong:
        image: docker.io/istio/proxy_debug:330dd286541d1b84c5ac1f4fc504556796c070af

should be gcr.io/testing - or we should push to docker the postsubmit...


drivers: update minizip to zlib:v1.3.1

Updates the vendored minizip code to v1.3.1 from madler/zlib.
Also includes one change to the defaults in the Dockerfile, so that I could build a container image and run the selfcheck in it.  I don't know whether that proves much about the upgrade; I did find the magics-test regression tests but haven't been able to run those yet. (reverted, to avoid mixing unrelated changes)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hi @iainrussell - apologies for the direct mention; you appeared when I checked the list of recent commits in this repository and I thought it might be worth tagging you on this PR.
This was part of a series of pull requests in that I opened in various projects after learning about CVE-2023-45853 -- a problem related to very long filenames or comments in zip files -- in minizip, as distributed with the zlib sources.
I'm working through some old/stale pull requests to determine whether they can be closed, and this is one of them.
I wasn't and am not super familiar with magics (I'm not a user of it), but on inspection I notice that minizip is used during opening of KML and GeoJSON files -- and uses fixed, short filenames, without any use of comments in those sections of code -- so I believe that this codebase is unaffected by the problem.
Even so, I'd personally generally recommend staying up-to-date with external dependency updates, because it can make future upgrades to them easier -- but I don't think there's any necessity to do so in this case.  I'll likely close this pull request in a week or so's time, unless the project feels it is worthwhile to merge.

Closing this pull request; please feel free to recover the commits (git fetch origin pull/102/head:pr-102-recovered, or similar) and apply/compare against those if they'd be useful in future.


Move survey condition degrader to rail_astro_tools

I would like to move the survey condition degrader obs_condition in the creation module from rail_attic to here, because it depends on healpy and PhotErr which will be included here (depending on progress in #3).

Ah, I realise it is already moved here! Sorry!


Only Mono output when playing WebRTC

System Details

OS:Ubuntu 22.04
Thorium Version 114.0.5735.134

Problem
When playing a WebRTC stream, the audio is only played in mono, even it shows 2 channels in webrtc internals and the offer from server side includes stereo=1. On Firefox it works normally in Stereo (sdp answer is correct). Thorium and also Chromium  defaults to Mono playback (sdp answer unsufficient).
Screenshots

Additional Notes
SDP log:
offer on server side:
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 cbr=1;maxaveragebitrate=131072;minptime=10;sprop-stereo=1;stereo=1;useinbandfec=1
answer on browser side:
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 minptime=10;useinbandfec=1 <<< here i should see stereo=1 ... then stereo is put out. Chromium always answers like this, so only mono output.

If you have the same problem on chromium/chrome, it may be a bug or the corresponding code in the upstream is not fully written, you can try to report the problem you encounter to Google on crbug.
@zapf2000


new place_and_take_perp_order flag

Currently place_and_take_perp_order tx will succeed whether or not there is a fill.
I suggest retro'ing the unused _maker_order_id into a flag that specifies when the tx should fail:

there is no fill
there is only a partial fill

With these new flags, it will be easier for devs to write client code and may lead to less tx actually needed to land onchain (you can skip sending the tx if it fails sim).

yes please!

May I ask if the fulfill_perp_order_ with amm function means to fill everything? Do I need to rewrite a function to implement this feature?

One question I have is assuming that in the case of a partial order that can be fulfilled, the program is in the stage of filling the order and the data of the order and position has actually changed. If I subsequently cancel this order. Will the data that has been changed before be restored? In other words is this whole program atomic?

When I tried to implement this new functionality in the handle_place_and_take_perp_order function, my thought process was to run a simulation of the fill_perp_order function first to get the number of underlying assets that were expected to be completed. But I ran into a problem where running the real fill_perp_order function after a successful simulation run reported a stack overflow.

Here is the code I tried. Feel free to discuss.
#[access_control(
    fill_not_paused(&ctx.accounts.state)
)]
pub fn handle_place_and_take_perp_order<'c: 'info, 'info>(
    ctx: Context<'_, '_, 'c, 'info, PlaceAndTake<'info>>,
    params: OrderParams,
    should_fail_on_partial_or_no_fill: Option<bool>,
) -> Result<()> {
    let clock = Clock::get()?;
    let state = &ctx.accounts.state;

    let remaining_accounts_iter = &mut ctx.remaining_accounts.iter().peekable();
    let AccountMaps {
        perp_market_map,
        spot_market_map,
        mut oracle_map,
    } = load_maps(
        remaining_accounts_iter,
        &get_writable_perp_market_set(params.market_index),
        &MarketSet::new(),
        clock.slot,
        Some(state.oracle_guard_rails),
    )?;

    if params.post_only != PostOnlyParam::None {
        msg!("post_only cant be used in place_and_take");
        return Err(print_error!(ErrorCode::InvalidOrderPostOnly)().into());
    }

    let (makers_and_referrer, makers_and_referrer_stats) =
        load_user_maps(remaining_accounts_iter, true)?;

    let is_immediate_or_cancel = params.immediate_or_cancel;

    controller::repeg::update_amm(
        params.market_index,
        &perp_market_map,
        &mut oracle_map,
        &ctx.accounts.state,
        &Clock::get()?,
    )?;

    let user_key = ctx.accounts.user.key();
    let mut user = load_mut!(ctx.accounts.user)?;

    controller::orders::place_perp_order(
        &ctx.accounts.state,
        &mut user,
        user_key,
        &perp_market_map,
        &spot_market_map,
        &mut oracle_map,
        &Clock::get()?,
        params,
        PlaceOrderOptions::default(),
    )?;

    drop(user);

    let user = &mut ctx.accounts.user;
    let order_id = load!(user)?.get_last_order_id();
    // let user_clone = &user.clone();

    msg!("=== handle_place_and_take_perp_order === clone test");

    // Simulation test of the function fill_perp_order with input core structure parameters all cloned
    let(test_base_asset_amount, test_quote_asset_amount) = controller::orders::fill_perp_order(
        order_id,
        &ctx.accounts.state.clone(),
        &user.clone(),
        &ctx.accounts.user_stats.clone(),
        &spot_market_map.clone(),
        &perp_market_map.clone(),
        &mut oracle_map.clone(),
        &user.clone(),
        &ctx.accounts.user_stats.clone(),
        &makers_and_referrer.clone(),
        &makers_and_referrer_stats.clone(),
        None,
        &Clock::get()?,
        FillMode::PlaceAndTake,
    )?;

    msg!("test base_asset_amount: {}", test_base_asset_amount);
    msg!("test quote_asset_amount: {}", test_quote_asset_amount);

    let(base_asset_amount, quote_asset_amount) = controller::orders::fill_perp_order(
        order_id,
        &ctx.accounts.state,
        user,
        &ctx.accounts.user_stats,
        &spot_market_map,
        &perp_market_map,
        &mut oracle_map,
        &user.clone(),
        &ctx.accounts.user_stats.clone(),
        &makers_and_referrer,
        &makers_and_referrer_stats,
        None,
        &Clock::get()?,
        FillMode::PlaceAndTake,
        &maker_orders_info,
    )?;

    msg!("real_base_asset_amount: {}", base_asset_amount);
    msg!("real_asset_amount: {}", quote_asset_amount);

    let order_exists = load!(ctx.accounts.user)?
        .orders
        .iter()
        .any(|order| order.order_id == order_id);

    if is_immediate_or_cancel && order_exists {
        controller::orders::cancel_order_by_order_id(
            order_id,
            &ctx.accounts.user,
            &perp_market_map,
            &spot_market_map,
            &mut oracle_map,
            &Clock::get()?,
        )?;
    }

    Ok(())
}

i dont think you need to run fill_perp_order twice, the base_asset_amount returned is the amount filled, so if base_asset_amount != 0, then a fill (full or partial) has happened.
You probably also want to use an enum instead of bool to allow for the ix to allow user to choose when to revert (partial fill, no fill, or dont revert)

One question I have is assuming that in the case of a partial order that can be fulfilled, the program is in the stage of filling the order and the data of the order and position has actually changed. If I subsequently cancel this order. Will the data that has been changed before be restored? In other words is this whole program atomic?

If there is a partial fill and the order is canceled, the user state does not revert if the transaction is successful (aka. immediate-or-cancel). This feature will allow the transaction to error in the event of partial or no fill - which will result in no change to user account (also known as fill-or-kill)

i dont think you need to run fill_perp_order twice, the base_asset_amount returned is the amount filled, so if base_asset_amount != 0, then a fill (full or partial) has happened.我认为您不需要运行 fill_perp_order 两次，返回的 base_asset_amount 是填充的数量，因此如果 base_asset_amount != 0，则发生填充（全部或部分）。
You probably also want to use an enum instead of bool to allow for the ix to allow user to choose when to revert (partial fill, no fill, or dont revert)您可能还想使用枚举而不是布尔值来允许 ix 允许用户选择何时恢复（部分填充、不填充或不恢复）

I'll say what I understand, because I looked at the code, and I found that it cancels the order if it is partially filled successfully, but the successful part is not withdrawn, or returned to the state it was in before it all started. So, my understanding is to simulate walking through the process first, and if it doesn't fill completely it doesn't perform the later steps.

One question I have is assuming that in the case of a partial order that can be fulfilled, the program is in the stage of filling the order and the data of the order and position has actually changed. If I subsequently cancel this order. Will the data that has been changed before be restored? In other words is this whole program atomic?

If there is a partial fill and the order is canceled, the user state does not revert if the transaction is successful (aka. immediate-or-cancel). This feature will allow the transaction to error in the event of partial or no fill - which will result in no change to user account (also known as fill-or-kill)

So, what do you think are the steps to correctly implement this feature?

You don't need to simulate because if you throw an error in the tx, the whole state is reverted

prototyped this here: https://github.com/drift-labs/protocol-v2/pull/1218/files


Add --editor option to kubectl get

What would you like to be added:
An --editor option to kubectl get that will open the manifest in an editor just as kubectl edit does it.
Why is this needed:
An editor usually offers syntax highlighting and makes it easier to nagivate a manifest. I curently use kubectl edit for this, even in situation where I do not want to edit anything. This has some issues:

It can result in accidentally applying a change
kubectl edit doesn't have all the functionality that kubectl get has, for example it is currently unable to show managed fields (despite having an option for that)

The idea was originally suggested by @henvic in https://github.com/kubernetes/kubectl/issues/1322#issuecomment-1317351825

/assign

@ardaguclu,
Doesn't --output control only the output format and filtering with json_path, or is there anything else that I'm not aware of? If there isn't, it doesn't conflict.
If it conflicts, maybe we can make it exclusive so you can only use --output or --editor at once or (a little bit riskier) make it smart enough to fail when it's not possible to use both.
What do you think? I won't be able to attend the sig-cli meeting today, unfortunately. I'd appreciate it if this could be clarified there.

Wait, I'm confused, kubectl edit doesn't open in your editor? Have you properly set & exported your $EDITOR shell variable? Because AFAIK, it's supposed to use that and do the right thing.

I think we can close this issue, since there seems to be a consensus about the solution in here https://github.com/kubernetes/kubectl/issues/1322
Feel free if I'm missing something
/close


[Task]: Line at the bottom does not match the box

Task description
The bottom line on the landing page doesn't match the box width. They should be of the same size.

Requirements
The width of line should match the box width.
Acceptance criteria
N/A
Creation checklist

[ ] I have assigned this task to the correct people
[X] I have added the most appropriate labels
[X] I have linked the correct milestone and/or project

Couldn't be reproduced on latest build.


Added 'verify_ssl' configuration flag.

Also suppress InsecureRequestWarning urllib3 warnings when verify_ssl == False. This PR solves  #26.

Hi @lukesneeringer,
This PR is related to cf37c28, but it suppress only InsecureRequestWarning. Can you take a look at this PR?

this config feature would be helpful for me as well (cc @lukesneeringer @wybczu )

Can you rebase it so an admin can merge this please ? it'd be great :-)

:+1:. Annoying/scary to others that our SSL isn't passing, because we have an internal CA which our CI/CD doesn't recognize.

Also it is possible to specify custom CA bundle via verify_ssl to verify SSL certificate.

On second thought this warning should be displayed. Not verifying ssl certificate is a big security issue, it makes MITM attacks easy :) I'm closing this pull request and I will create another one.

@wybczu What made you change your mind? What's wrong with turning off ssl warnings? If it's not default, then I'd like to be able to turn if off. What if the particular tower instance is just a test instance and we don't want to take the time to deploy proper certificates? Or what if it was a situation where I don't manage the service?

@ccollicutt as a user I would like to know when SSL verification is disabled (warnings should be displayed).
If you don't want to deploy proper cert you can always download tower's certificate (bash oneliner) and pass path to it e.g. via ca_path option. What do you think about such solution?

Hi, was this merged?
I am trying to disable the verification it but no success.


Switch felix from debug to info level logging

Fixes #783

@djosborne
Noticed this was true for 2.1 docs for Etcdless Hosted Install too:
Not sure if we want to take care of that in this PR.
http://docs.projectcalico.org/v2.1/getting-started/kubernetes/installation/hosted/k8s-backend/calico.yaml

each of these manifest also has debug logging listed for cni_network_config section too. Shouldn't that be listed "info" as well?

The problem this was trying to solve was over-logging. Is the CNI-plugin generating too many logs?

Noticed this was true for 2.1 docs for Etcdless Hosted Install too:

We avoid updating already released manifests unless absolutely necessary, so let's not make this change to any 2.1 manifests.


Using async on flask (Working outside of application context.)

I'm using the new async function but with a flask app.
You will hit an error if you do anything with the app_context
[1490788127700] Working outside of application context.

This typically means that you attempted to use functionality that needed
to interface with the current application object in a way.  To solve

To solve this i wrapped my function in
def do_something():
     with app.app_context():
         db.query.do_whatever..

I'm not sure if we should add native support for this vs having to wrap the functions?

I actually wasn't able to reproduce this. Can you post a small example Flask app that causes this error?

I'm getting the same thing. If I try to use session it throws the same error

getting the came error while calling a function in the same module recursively and i think its making my lambda call to the flask endpoint rerun 8 separate times. pinning up 15 lambdas when only 3 should be needed.


[stable/mariadb] clusterIp provided IP is already allocated

master-svc.yaml and slave-svc.yaml is use the same clusterIP from the values-production.yaml (or values.yaml) So finally it say
Error: release maria failed: Service "maria-mariadb-slave" is invalid: spec.clusterIP: Invalid value: "xx.xx.xx.xx": provided IP is already allocated

cc @carrodher Hello sir, Is this has some special settings for production deployments?

Nope. Difference is described here

Thanks.

Hi @cronolio ,
I think that PR will fix the problem you are facing. Could you confirm please?

hm. possible problem is that production values has slave.replicas: 2


Problem with nuxt imports: Vitest Coverage stays at 100% for components regardless of the existence of any component test because of nuxt imports

Environment

Operating System: Darwin
Node Version:     v18.16.0
Nuxt Version:     3.11.2
CLI Version:      3.11.1
Nitro Version:    2.9.6
Package Manager<EMAIL_ADDRESS>Builder:          -
User Config:      devtools, typescript, extends, components, ssr, runtimeConfig, app, css, build, modules, experimental, i18n, proxy, vite, public, nitro, hooks
Runtime Modules<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Build Modules:    -

Reproduction
https://stackblitz.com/edit/nuxt-starter-gvfsue?file=nuxt.config.ts,components%2Fnested%2FTestNestedUntested.vue
vitest --coverage is not working in StackBlitz. Please download the project and try it locally.
Describe the bug
Description:
When the components option in the Nuxt configuration is set to true (enabling automatic global registration of components), the test coverage report shows 100% coverage for all components, regardless of whether tests actually exist for them. Something is importing the untested TestNestedUntested. V8 is reporting coverage for it. Maybe Nuxt imports all those files.
Actual Behavior:
The coverage report incorrectly shows 100% coverage for all components, including those without any tests like TestNestedUntested.vue in component directory. Seems to be a problem caused by nuxt importing all files.
Expected Behavior:
The coverage report should accurately reflect the coverage for each component. Only TestNested.vue should show 100% coverage, while TestNestedUnested.vue should show 0% coverage or appropriate coverage based on the absence of tests.
Additional context
No response
Logs
No response

This is unlikely something to fix in Nuxt. It might possibly be a vitest bug or something to be resolved in nuxt/test-utils. Have you tried ensuring that .nuxt is added to your coverage exclude patterns?

I first raised the issue at vitest but they think its a nuxt problem:
https://github.com/vitest-dev/vitest/issues/6029#issuecomment-2210914420

It seems like the issue is the one Ari linked - it's a Vue sourcemap bug: https://github.com/vitest-dev/vitest/issues/3607#issuecomment-1608954728.

This report contains two issues:

Something (nuxt ?) is importing files that are not being tested. Node's V8 coverage includes every file that is loaded via Node. The repro is testing TestNested.vue, but is also unexpectedly seeing coverage for UntestNested.vue. Vitest is not importing UntestNested.vue.
Coverage is incorrectly 100%. This is caused by Vue's sourcemaps being off. We've seen so many of these issues that I think I'll need to raise an issue on Vue some day.

@AriPerkkio, @danielroe: Thank you both for your help so far.
The issue with the coverage report has evolved after updating vitest to version 2 (see comment in vitest issue, Stackblitz demo). Previously, the coverage incorrectly showed 100% for all components, but now it shows 0% coverage for files that do have tests.
To summarize the current status:

With the istanbul provider, the coverage report collects only the coverage for the utils directory
With the v8 provider, the coverage report now shows 0% coverage for files that have tests

Given these changes, could we clarify who will be investigating this further? Is this something that should be addressed by the Nuxt team, or should we continue the discussion with the Vitest team?


[GFI] Does AGE follow PG's C-code practices?

Read the Writing Code seciton. Verify that Apache AGE codebase follow the recommendations. If not, make a list of violations and report here.

Hello i have a question regarding this task that i wanted to clear up, i am relatively new to contributing to age hence please guide me regarding this matter. Should i also report instances of code where after memory is allocated to an object using palloc0 it is not deallocated using pfree? I have seen few instances of this while going through the code and the writing code section of PostegreSQL mentions that palloc0 automatically frees memory but its not clearly specified wether pfree should still be used. I would appreciate the help since its my first time contributing to open source

There are multiple violations for PostgreSQL coding conventions. I will try to fixes them and create multiple PRs,  as single PR/Commit will create confusion.

Added first PR #1908 for master branch


Major Flickering

Hello, I just followed the tutorial for building your own input device using an arduino uno, however I am getting some major flickering in NintendoSpy, even with blink reduction turned on. Any ideas?

I added some extra anti-flickering features, expect it tomorrow on my branch and as pull request.


Update saxon to latest version

The version of Saxon used by the build should be updated to the following:
<dependency>
    <groupId>net.sf.saxon</groupId>
    <artifactId>Saxon-HE</artifactId>
    <version>9.6.0-3</version>
</dependency>

See: http://search.maven.org/#artifactdetails|net.sf.saxon|Saxon-HE|9.6.0-3|jar

Was there a reason to use the older version?

Yes, you're right, it was an old misprint which embedded the jar. fix it

The previous version was an osgi bundle whereas what you suggest it's a library, so I embedded it into the importer bundle as a private package

Yes, embedding the lib into the bundle is fine.


Text output

I am back to testing ctransformers, now with the quantized mpt-30b-instruct model and I have a really basic problem. Does ctransformers handle output in a different way to transformers?
I have been trying for a while to get it to print a block of text, but when using the native huggingface tokenizer I either get a streaming output from the for loop where each decoded token is placed in new line, or I get a list:
python mpt-30b_transformers_test.py
['Blue', ' can', ' be', ' associated', ' with', ' many', ' things', ',', ' such', ' as', ' sadness', ' or', ' depression', ' due', ' to', ' its', ' association', ' in', ' mood', 's', ' and', ' feelings', ';', ' however', ' it', '’', 's', ' also', ' been', ' proven', ' through', ' research', ' studies', ' conducted', ' by', ' psychologists', ' at', ' Harvard', ' University', ' that', ' looking', '/', 'sur', 'round', 'ing', ' yourself', ' within', ' this', ' shade', ' will', ' reduce', ' your', ' stress', ' levels', '.', '  ', 'It', ' is', ' a', ' very', ' cal', 'ming', ' colour', ' which', ' makes', ' sense', ' when', ' you', ' think', ' about', ' the', ' sky', ',', ' sea', ' etc', ' where', ' blue', ' dominates', ' as', ' these', ' are', ' places', ' we', ' often', ' go', ' to', ' relax', ' and', ' switch', ' off', ' from', ' our', ' busy', ' lives', '!']

I can wrangle this into text, but it is an extra step and a bit of a pita as not every token is a word or a free syntax symbol so I am wondering if there is a better way to get plain text output when using ctransformers with a HF tokenizer.

Can you please share the code you are using.

I do not actually have a working example as I neglected to save revisions, but I narrowed down the issue to me not knowing what I am doing, and ctransformers.generate() outputting a list that is not in the same format as transformers.generate(). I therefore cannot just do this for example:
python mpt-30b_transformers_test.py
Traceback (most recent call last):
  File "/home/*****/transformers/mpt-30b_transformers_test.py", line 35, in <module>
    print(tokenizer.decode(output_text[0]))
TypeError: 'generator' object is not subscriptable

The only way that I can get the output I can use with minimal effort is if I do not use the HF tokenizer. The main reason why I find this an issue is because I cannot just drop in ctransformers in place of transformers and carry on with the other parts of the code. without having to make any other changes.

A generator object is an efficient way to create a list. You can convert generator object to list using list():
from ctransformers import AutoModelForCausalLM
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("marella/gpt-2-ggml")

inputs = tokenizer.encode(prompt)
outputs = model.generate(inputs)
outputs = list(outputs) # convert generator object to list
text = tokenizer.decode(outputs)

If you have a list of decoded tokens as you mentioned in the initial post, you can just do "".join(decoded_tokens)


letskencrypt: /var/empty: chroot: No such file or directory

Now I'm getting the following:
$ sudo LD_LIBRARY_PATH=/usr/local/lib ./letskencrypt -vNn foo.com www.foo.com
: letskencrypt: /var/empty: chroot: No such file or directory
/var/empty: chroot: No such file or directory
/var/empty: chroot: No such file or directory
letskencrypt: /var/empty: chroot: No such file or directory
/var/empty: chroot: No such file or directory
I'm on Ubuntu.

Ubuntu doesn't have /var/empty?  What does it use instead for privilege-dropped daemons?

You can always just mkdir /var/empty as root in the meantime.

Did that, now it is the following:
$ sudo LD_LIBRARY_PATH=/usr/local/lib ./letskencrypt-portable/letskencrypt -vNn foo.com www.foo.com
letskencrypt: -k file must not exist when using -N
letskencrypt: -f file must not exist when using -n

It's pretty self-explanatory.  You used -N when the domain key already exists and -n when the account key already exists.  The default locations are privkey.pem in /etc/ssl/letsencrypt/private for the former and /etc/letsencrypt for the latter.  If you already made them, drop the -N and -n.

Now with just -v it is the following:
$ sudo LD_LIBRARY_PATH=/usr/local/lib ./letskencrypt-portable/letskencrypt -v
letskencrypt: letskencrypt: /etc/letsencrypt/privkey.pem: PEM_read_PrivateKey
139701815826080:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: ANY PRIVATE KEY
/etc/ssl/letsencrypt/private/privkey.pem: PEM_read_PrivateKey
139701815826080:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: ANY PRIVATE KEY
letskencrypt: bad exit code: keyproc(9240)
letskencrypt: bad exit code: acctproc(9241)

@tfarina, I really can't help you with so little information.  Please clear all the directories the system has used (the manpage will tell you what they are), read the manpage, and re-run the program.  If you can demonstrate that your keys are valid and letskencrypt isn't able to read them, then that's a problem.  Obviously the keys in the given locations aren't really private keys: you've either put something else there or the files are empty or something.  I'm closing this issue until you have a bug to report.

ubuntu@tfarina:~$ sudo rm -rf /etc/ssl/letsencrypt                              ubuntu@tfarina:~$ sudo rm -rf /etc/letsencrypt                                  ubuntu@tfarina:~$ sudo rm -rf /var/www/letsencrypt
$ sudo rm -rf /var/empty
$ sudo mkdir /var/www/letsencrypt
$ sudo mkdir /etc/ssl/letsencrypt
$ sudo mkdir /etc/ssl/letsencrypt/private /etc/letsencrypt
$ sudo chmod 0700 /etc/ssl/letsencrypt/private /etc/letsencrypt
$ sudo mkdir /var/empty
$ sudo LD_LIBRARY_PATH=/usr/local/lib ./letskencrypt-portable/letskencrypt -vNn foo.com www.foo.com
letskencrypt: /etc/ssl/letsencrypt/private/privkey.pem: generated RSA domain key
letskencrypt: /etc/letsencrypt/privkey.pem: generated RSA account key
letskencrypt: https://acme-v01.api.letsencrypt.org/directory: directories
letskencrypt: acme-v01.api.letsencrypt.org: DNS: <IP_ADDRESS>
letskencrypt: tls_init: No such file or directory
letskencrypt: https://acme-v01.api.letsencrypt.org/directory: bad comm
letskencrypt: bad exit: netproc(10461): 1
What more information do you need to make it work out of the box than the command lines I'm running?

Ok, much better.  It looks like tls_init() can't be run within a chroot.  (It calls through to tls_config_new(), which requires /etc/ssl/cert.pem.)  I'll put up a fix in a few.

Actually, that doesn't seem to be it.  What version of libressl are you running?  Can you run ldd on the letskencrypt binary?

$ dpkg -l | grep ssl
ii  libgnutls-openssl27:amd64            2.12.23-12ubuntu2.5              amd64        GNU TLS library - OpenSSL wrapper
rc  libpolarssl5                         1.3.4-1                          amd64        lightweight crypto and SSL/TLS library
ii  libssl-dev:amd64                     1.0.1f-1ubuntu2.19               amd64        Secure Sockets Layer toolkit - development files
ii  libssl1.0.0:amd64                    1.0.1f-1ubuntu2.19               amd64        Secure Sockets Layer toolkit - shared libraries
ii  openssl                              1.0.1f-1ubuntu2.19               amd64        Secure Sockets Layer toolkit - cryptographic utility
ii  python-openssl                       0.13-2ubuntu6                    amd64        Python 2 wrapper around the OpenSSL library
ii  ssl-cert                             1.0.33                           all          simple debconf wrapper for OpenSSL
$ ldd ./letskencrypt
linux-vdso.so.1 =>  (0x00007fff79192000)
libtls.so.11 => /lib/x86_64-linux-gnu/libtls.so.11 (0x00007ff50f6bc000)
libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007ff50f45d000)
libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007ff50f080000)
libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007ff50ee71000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff50eaac000)
libssl.so.39 => /lib/x86_64-linux-gnu/libssl.so.39 (0x00007ff50e84b000)
libcrypto.so.38 => /lib/x86_64-linux-gnu/libcrypto.so.38 (0x00007ff50e442000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff50e23e000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff50f8d0000)

Do you have libressl installed?

As this will be up to a downstream packager (creating /var directories isn't right for third-party software to do!), I'm closing this out.  I have added a value to [config.h], PATH_VAR_EMPTY, which can be set to any directory by downstream.


question about testing

https://github.com/zju3dv/SMAP/blob/9fbcbe4ee2cd6f6d6d6a4e5a05a637629cb5b2e1/exps/stage3_root2/config.py#L70
Did convert the original format of the moputs dataset to your own format? Could you please release the covert code. I could't find the "M3E_gt.json" in the mopiuts dataset I have downloaded and I can't run the test code, using your model released. Thanks a lot for your time and reading.

We have provide the format in 'lib/preprocess'. For MuCo, refer to [https://github.com/zju3dv/SMAP/issues/3#issuecomment-692488374](this issue)

I have tried to convert the format of MuPoTS dataset to your formate following what you did for MuCo, but I found that there are still some problem in corverting. Especially, it seems to be several cameras corresponding to the same frame of a image. So which camera did you choose?
And could you please release the coverting code if it's convenient?

We have provide the format in 'lib/preprocess'. For MuCo, refer to this issue, which is similar to the conversion of mupots.

I have tried to convert the format of MuPoTS dataset to your formate following what you did for MuCo, but I found that there are still some problem in corverting. Especially, it seems to be several cameras corresponding to the same frame of a image. So which camera did you choose?
And could you please release the coverting code if it's convenient?