POST {resource_type}/_search returns 415 (Unsupported Media Type) if a query parameter is duplicated in URL and request body

Describe the bug
When using the POST {resource_type}/_search endpoint to query resources, the server returns 415 Unsupported Media Type when repeating a query parameter in the URL query string and the request body.
The FHIR search specification states

Note that in the POST variant, parameters may appear in both the URL and the body. Parameters have the same meaning in either place. Since parameters can repeat, putting them in both places is the same as repeating them (which is valid for some parameters and not for others).

It is expected that when a query parameter is repeated in the URL and the body:

The server does not reject the request (assuming it is otherwise well-formed) and returns a Bundle as per the normal search specification
All query parameters from both the URL and the body are used as inputs to the the search operation

FHIR Version?
R4
Data provider?
CosmosDB
Workaround
When using the POST /_search endpoint, clients must send all query parameters in the request body and leave the URL query parameters empty.
To Reproduce

Create a patient resource to show successful results in later examples.

POST /Patient HTTP/1.1
Host: <removed>.azurehealthcareapis.com
Content-Type: application/fhir+json
Authorization: <removed>

{
    "resourceType": "Patient",
    "identifier": [
        {
            "system": "http://example.com/fhir/post_search_bug",
            "value": "id_value_1"
        },
        {
            "system": "http://example.com/fhir/post_search_bug",
            "value": "id_value_2"
        }
    ],
    "active": true,
    "birthDate": "2001-02-03",
    "gender": "female"
}

Receive 201 Created with the following body
{
    "resourceType": "Patient",
    "id": "d5f35698-4981-4aca-af0e-2819df79c633",
    "meta": {
        "versionId": "1",
        "lastUpdated": "2023-03-03T15:35:59.78+00:00"
    },
    "identifier": [
        {
            "system": "http://example.com/fhir/post_search_bug",
            "value": "id_value_1"
        },
        {
            "system": "http://example.com/fhir/post_search_bug",
            "value": "id_value_2"
        }
    ],
    "active": true,
    "gender": "female",
    "birthDate": "2001-02-03"
}

(❌ Unexpected Behaviour) Demonstrate example search with POST /_search endpoint, with overlapping query parameters in both the URL and the body. More successful examples are provided below to show that the key condition appears to be duplicating a query parameter in the URL and the body.

POST /Patient/_search?active=true&gender=female&birthdate=2001-02-03&identifier=http://example.com/fhir/post_search_bug|id_value_1 HTTP/1.1
Host: <removed>.azurehealthcareapis.com
Content-Type: application/x-www-form-urlencoded
Authorization: <removed>
Content-Length: 61

identifier=http://example.com/fhir/post_search_bug|id_value_2

Receive 415 Unsupported Media Type with the following OperationOutcome. The error message states that the Content-Type header is not supported, but the content-type (application/x-www-form-urlencoded) is exactly the same as in the later successful requests to POST /_search.
{
    "resourceType": "OperationOutcome",
    "id": "fca8e1ee698d5664933a0063efa8cb1f",
    "meta": {
        "lastUpdated": "2023-03-03T15:59:37.6032741+00:00"
    },
    "issue": [
        {
            "severity": "error",
            "code": "not-supported",
            "diagnostics": "Value supplied for the \"Content-Type\" header is not supported."
        }
    ]
}

(✔ Correct Behaviour) Demonstrate example search operation with GET /Patient endpoint (using URL only)

GET /Patient?active=true&gender=female&birthdate=2001-02-03&identifier=http://example.com/fhir/post_search_bug|id_value_1&identifier=http://example.com/fhir/post_search_bug|id_value_2 HTTP/1.1
Host: <removed>.azurehealthcareapis.com
Content-Type: application/fhir+json
Authorization: <removed>

Receive 200 OK with the following response Bundle
{
    "resourceType": "Bundle",
    "id": "1c2d5ff1e631403a36dd3e6dbb0de164",
    "meta": {
        "lastUpdated": "2023-03-03T15:48:25.5211147+00:00"
    },
    "type": "searchset",
    "link": [
        {
            "relation": "self",
            "url": "https://<removed>.azurehealthcareapis.com/Patient?active=true&gender=female&birthdate=2001-02-03&identifier=http%3A%2F%2Fexample.com%2Ffhir%2Fpost_search_bug%7Cid_value_1&identifier=http%3A%2F%2Fexample.com%2Ffhir%2Fpost_search_bug%7Cid_value_2"
        }
    ],
    "entry": [
        {
            "fullUrl": "https://<removed>.azurehealthcareapis.com/Patient/d5f35698-4981-4aca-af0e-2819df79c633",
            "resource": {
                "resourceType": "Patient",
                "id": "d5f35698-4981-4aca-af0e-2819df79c633",
                "meta": {
                    "versionId": "1",
                    "lastUpdated": "2023-03-03T15:35:59.78+00:00"
                },
                "identifier": [
                    {
                        "system": "http://example.com/fhir/post_search_bug",
                        "value": "id_value_1"
                    },
                    {
                        "system": "http://example.com/fhir/post_search_bug",
                        "value": "id_value_2"
                    }
                ],
                "active": true,
                "gender": "female",
                "birthDate": "2001-02-03"
            },
            "search": {
                "mode": "match"
            }
        }
    ]
}

(✔ Correct Behaviour) Demonstrate example search with POST /_search endpoint, using body only

POST /Patient/_search HTTP/1.1
Host: <removed>.azurehealthcareapis.com
Content-Type: application/x-www-form-urlencoded
Authorization: <removed>
Content-Length: 170

active=true&gender=female&birthdate=2001-02-03&identifier=http://example.com/fhir/post_search_bug|id_value_1&identifier=http://example.com/fhir/post_search_bug|id_value_2

Receive 200 OK with the following response Bundle
{
    "resourceType": "Bundle",
    "id": "26af93b6dba4596c41fc145f365d2b90",
    "meta": {
        "lastUpdated": "2023-03-03T15:55:54.23725+00:00"
    },
    "type": "searchset",
    "link": [
        {
            "relation": "self",
            "url": "https://<removed>.azurehealthcareapis.com/Patient?active=true&gender=female&birthdate=2001-02-03&identifier=http%3A%2F%2Fexample.com%2Ffhir%2Fpost_search_bug%7Cid_value_1&identifier=http%3A%2F%2Fexample.com%2Ffhir%2Fpost_search_bug%7Cid_value_2"
        }
    ],
    "entry": [
        {
            "fullUrl": "https://<removed>.azurehealthcareapis.com/Patient/d5f35698-4981-4aca-af0e-2819df79c633",
            "resource": {
                "resourceType": "Patient",
                "id": "d5f35698-4981-4aca-af0e-2819df79c633",
                "meta": {
                    "versionId": "1",
                    "lastUpdated": "2023-03-03T15:35:59.78+00:00"
                },
                "identifier": [
                    {
                        "system": "http://example.com/fhir/post_search_bug",
                        "value": "id_value_1"
                    },
                    {
                        "system": "http://example.com/fhir/post_search_bug",
                        "value": "id_value_2"
                    }
                ],
                "active": true,
                "gender": "female",
                "birthDate": "2001-02-03"
            },
            "search": {
                "mode": "match"
            }
        }
    ]
}

(✔ Correct Behaviour) Demonstrate example search with POST /_search endpoint, with non-overlapping query parameters in both the URL and the body.

POST /Patient/_search?active=true&gender=female&birthdate=2001-02-03 HTTP/1.1
Host: <removed>.azurehealthcareapis.com
Content-Type: application/x-www-form-urlencoded
Authorization: <removed>
Content-Length: 123

identifier=http://example.com/fhir/post_search_bug|id_value_1&identifier=http://example.com/fhir/post_search_bug|id_value_2

Receive 200 OK with the following response Bundle
{
    "resourceType": "Bundle",
    "id": "7c9bf9813cb77d170863729465db00f1",
    "meta": {
        "lastUpdated": "2023-03-03T15:57:32.200619+00:00"
    },
    "type": "searchset",
    "link": [
        {
            "relation": "self",
            "url": "https://<removed>.azurehealthcareapis.com/Patient?active=true&gender=female&birthdate=2001-02-03&identifier=http%3A%2F%2Fexample.com%2Ffhir%2Fpost_search_bug%7Cid_value_1&identifier=http%3A%2F%2Fexample.com%2Ffhir%2Fpost_search_bug%7Cid_value_2"
        }
    ],
    "entry": [
        {
            "fullUrl": "https://<removed>.azurehealthcareapis.com/Patient/d5f35698-4981-4aca-af0e-2819df79c633",
            "resource": {
                "resourceType": "Patient",
                "id": "d5f35698-4981-4aca-af0e-2819df79c633",
                "meta": {
                    "versionId": "1",
                    "lastUpdated": "2023-03-03T15:35:59.78+00:00"
                },
                "identifier": [
                    {
                        "system": "http://example.com/fhir/post_search_bug",
                        "value": "id_value_1"
                    },
                    {
                        "system": "http://example.com/fhir/post_search_bug",
                        "value": "id_value_2"
                    }
                ],
                "active": true,
                "gender": "female",
                "birthDate": "2001-02-03"
            },
            "search": {
                "mode": "match"
            }
        }
    ]
}

Expected behavior
As per the FHIR Search specification, all search parameters from the URL query string and the request body should be used as inputs to the search operation when using the POST {resource_type}/_search endpoint.
Actual behavior
FHIR server returns 415 Unsupported Media Type when a query parameter is duplicated in the URL and the body. The search appears to work as expected when query parameters do not collide between the URL query string and the POST body.
AB#101268

Hi, while discussing this with my team and experimenting with the current implementation, we realised the current implementation has several "search" scenarios that are probably unintended. It appears all POST /{x}/_search requests are being rewritten as GET /{x} internally, which leads to the following examples.
Workaround
Explicitly block these additional requests through url filtering/pattern matching in our api gateway. (For us, only POST /_history/_search represented any potential security risk in multi-tenanted environments)
Examples
(taken from https://hl7.org/fhir/http.html#summary)

original request
rewritten as
intended?
risk?

POST /{type}/_search?a=b
GET /{type}?a=b
✔
-

POST /_search
GET /
❌

POST /metadata/_search
GET /metadata
❌

POST /_history/_search
GET /_history
❌
query parameters are not applied to the history search, so can return/leak data unexpectedly!

POST /_history//{type}/_search
GET /_history/{type}
❌
query parameters are not applied to the history search, so can return/leak data unexpectedly!

POST /_history//{type}/{id}/_search
GET /_history/{type}/{id}
❌
query parameters are not applied to the history search, so can return/leak data unexpectedly! However, clients must know the resource id already.

@moloneyd-draeger - this issue is resolved with PR https://github.com/microsoft/fhir-server/pull/3232. Let us know if you have any additional questions.


Add hardknott to LAYERSERIES_COMPAT

Yocto master has changed the distro name from gatesgarth to hardknott, please add the latter to  LAYERSERIES_COMPAT_meta-scipy in layer.conf to keep compatibility.

There's a little more than that that needs to be done with a new series.
First, we need to bump the SciPy version to whatever the latest is around the time the new series is released (1.6.2 is most recent release as of this writing) and then of course we need to make sure that it build successfully in the new layer.
If you want to see this done sooner, then please take a stab at doing that yourself and then making a PR if everything works. It will take me a bit longer to do it myself.

I think the two issues are orthogonal.
I was testing your layer with Yocto master recently (then still named "gatesgarth" but way ahead of the actual Gatesgarth version) with an ImageAI recipe that runtime-depends on python3-scipy.

I added hardknott on master.


No build for ubuntu 15.10

As title say, Ubutnu 15.10 Wily Werewolf do not have a built release in the ppa repo.

Check https://launchpad.net/~ultradvorka/+archive/ubuntu/ppa/+packages for 15.10 build. Thank you!

@dvorka Thank you very much


js beta shopping cart - step 35 code checking too strict for braces

Describe the Issue
in the js beta shopping cart project , step 36 does not allow the code below to pass
[...addToCartBtns].forEach(
  (btn) => btn.addEventListener("click", (event) => {})
);

and instead needs braces to be added in order to allow the code to pass as below:
[...addToCartBtns].forEach(
  (btn) => {btn.addEventListener("click", (event) => {})}
);

Affected Page
https://www.freecodecamp.org/learn/javascript-algorithms-and-data-structures-v8/learn-basic-oop-by-building-a-shopping-cart/step-35
Your code
[...addToCartBtns].forEach(
  (btn) => btn.addEventListener("click", (event) => {})
);

Expected behavior
I would expect the code without the extra braces to pass.
Screenshots
No response
System
N/A
Additional context
forum post that triggered this https://forum.freecodecamp.org/t/learn-basic-oop-by-building-a-shopping-cart-step-35/716369

The issue is that without braces, the arrow function implicitly returns the result of the single expression, which doesn't allow for complex logic in the event handler, so thence by Adding braces clarifies the function body, enabling the event listener to be defined correctly.
Please assign me the project.

I don't know if there was a pedagogical reason... even if the tests are changed to be accepting of the implicit return one I don't think the seed need to be changed, considering that the returned value doesn't matter, having a body in the function is in line with that.
Considering that there is no explicit mention of wanting the function either way, I agree on making the tests accept also the version without braces

I am opening for contributions, the expected change is: change the tests in the step to accept both versions, with and without braces

Hello! I would like to propose a solution to the issue regarding the strict code checking for braces in step 35 of the JavaScript beta shopping cart project.
Current Situation
As described, the current implementation requires the use of braces for the arrow function in the following code:
[...addToCartBtns].forEach(
(btn) => {btn.addEventListener("click", (event) => {})}
);
However, the following version without braces is also valid:
[...addToCartBtns].forEach(
(btn) => btn.addEventListener("click", (event) => {})
);
Suggested Change
I propose that the test conditions for this step be modified to accept both versions of the arrow function. This would allow for a more flexible approach to writing concise code while still being pedagogically sound.
Rationale
JavaScript Flexibility: JavaScript allows for implicit returns in arrow functions without braces, which can lead to cleaner and more readable code.
User Experience: Allowing both styles accommodates different coding preferences and encourages learners to explore the language's flexibility.
Please let me know if you need any further details or if there's a specific format you'd prefer for this change. I'm happy to help!
Thank you!

@prashu0705 that's what I said, the issue is open for contributions, the expected change is to have the tests accept both versions

/assign

Please make sure you read our guidelines for contributing, we prioritize contributors following the instructions in our guides. Join us in our chat room or our forum if you need help contributing; our moderators will guide you through this.
Sometimes we may get more than one pull request. We typically accept the most quality contribution followed by the one that is made first.
Happy contributing.


[zh] sync /using-api/server-side-apply.md

Sync with en #39144. See preview.
content/zh-cn/docs/reference/using-api/server-side-apply.md

Thanks.
/lgtm
/approve


What is numbers supposed to get?

Using this with a vue.js app
I did the import
import { registerWebPlugin } from "@capacitor/core"; import { SmsManager } from "@byteowls/capacitor-sms";
I registered the plugin when component is created
created() {     registerWebPlugin(SmsManager); }
I call this method
  Plugins.SmsManager.send({         numbers: ["749093246"],         text: "test test"       })         .then(() => {           // SMS app was opened         })         .catch(error => {           // see error codes below           if (error.message !== "SEND_CANCELLED") {             // show toast with error message             // eslint-disable-next-line             console.log(error.message);           }         });
Getting error in xcode
SmsManager    send    72688974    ERROR MESSAGE:     {"message":"ERR_NO_NUMBERS","errorMessage":""}    ⚡  [log] - ERR_NO_NUMBERS
How is numbers formatted? I've tried all 3 ways
[749093246],  or ["749093246"], "749093246" ??
This plugin only opens sms right? User still has to send the sms?

This plugin only opens sms right? User still has to send the sms?
yes and yes.

Are you sure that you use a valid number?

I'm closing this as the author did not answer my followup question. Please create a new issue if the problem persists.


Added IP restriction option

Hi!
As discussed in https://forum.indiebits.io/t/issues-with-basic-authentication/256/4, I'd like to get an honest first opinion on this. I've commented the PR to help you navigate through it.
One blocking I've encountered is that for client tls terminated tunnels, the feature blocks the requests made from let's encrypt to validate the domain. Unfortunately, let's encrypt does not publish a list of IPs that we could systematically add so the only option I see would be to use dns validation, or move the cert generation/renewal to the server and transfer it to the clients but it would be a lot more work. Even DNS validation would be complicated since dns providers are plentiful and all require a different pluggable golang module. Any idea here?
This is what the UI looks like :

When checking the box :

Disclaimer, I'm not a golang expert and I've tried following the code style of the existing codebase.

@jebeaudet Does your code work? I just checked your branch and saw that you added geoblocking but not the ability to restrict based on a list of IP. This feature would greatly improve the security of my setup so please tell me. Thanks,

@rburettt Yes it does work with the caveats mentioned in the original post. The proper option would be to go for dns validation instead of http challenge, I tried a bit with it with my dns provider but somehow couldn't make it to work.
IP restriction like this was also not ideal for my setup as some of my users don't have a static IP. A proper setup would be some kind of challenge with an email code or username/password that would detect your IP server side and then add that IP to the allowlist but it's a but more work to put it all together.
I then decided to go for a fail2ban setup alongside basic geoblocking to get the easy random noise blocked.

@jebeaudet Just set it up this morning and indeed it worked! In the end I used your latest branch with the geoblocking and only encountered an issue with the fact that you hardcoded the path to the dblocation BIN file but apart from that it seems to be working quite well so thanks!

@rburettt Indeed that's one nice hardcoded path! Thanks for circling back and I'm glad it worked out for you. Let me know if you do any improvements, I might be interested.

If anyone ends up here, I managed to get around the ACME requirements by simply always allow requests with the protocol acme-tls/1. This is quite easy to do by inspecting the tls.ClientHelloInfo struct. It's not foolproof as it lets those requests through to the upstream servers but the blast radius is limited to what people can do with that.


Make sure that Acceptance tests (androidTests):

Description
This PR aims to detach completely the tests from an running edX instance by mocking some of the basic APIs and making sure that all existing acceptance tests are still running correctly. This might be used as a partial solution to MA-2082 (https://openedx.atlassian.net/browse/MA-2082).
Changes are the following:
Gradle Updates:

Add a dependency needed for acceptance tests

Main application (OpenEdXMobile/src/main/java/org/edx/mobile/base/MainApplication.java):

Make sure that we get the right injector for the tests (so we can mock the base services)

Other:

Add a mock class to mock LoginAPI and CourseAPI and mock data
Modify some tests so they work according to the new UI layout (tests were likely to be oudated)

Notes
I have noticed that some of the work of Mocking the API was already done in the unit tests. Did not find any sure way of merging the two approaches as use case are different but it could be the next step.
Testing:

Using android studio: run test in the the org.edx.mobile package.

Thanks for the pull request, @laurentdavid! I've created OSPR-1605 to keep track of it in JIRA. JIRA is a place for product owners to prioritize feature reviews by the engineering development teams.
Feel free to add as much of the following information to the ticket:

supporting documentation
edx-code email threads
timeline information ("this must be merged by XX date", and why that is)
partner information ("this is a course on edx.org")
any other information that can help Product understand the context for the PR

All technical communication about the code itself will still be done via the GitHub pull request interface. As a reminder, our process documentation is here.
If you like, you can add yourself to the AUTHORS      file for this repo, though that isn't required.     Please see the CONTRIBUTING     file for more information.

Current coverage is 18.97% (diff: 0.00%)

Merging #885 into master will increase coverage by <.01%

@@             master       #885   diff @@
==========================================
  Files           470        470
  Lines         19978      19978
  Methods           0          0
  Messages          0          0
  Branches       2922       2922
==========================================
+ Hits           3789       3790     +1
+ Misses        15738      15737     -1
  Partials        451        451

Powered by Codecov. Last update b8d9bd8...699d09d

@laurentdavid Attempting to run on CI https://github.com/edx/edx-app-android/pull/895

@miankhalid @1zaman I'd like to merge this. We can't turn it on on Travis yet because of some travis issues but this PR opens up some flexibility to our E2E testing.

@BenjiLee Have we had a chance to resolve the Travis issues so we can move forward with this PR?

@gsong Travis issue is separate of this PR. Waiting on one more review, by may merge before then.

Hello @laurentdavid: We are unable to continue with review of your submission at this time. Please see the associated JIRA ticket for more explanation.

@BenjiLee Any idea when the next review can happen?

@miankhalid @farhan-arbisoft poke

@BenjiLee Any idea when the next review can happen?

Sorry for the delay, I'll review it today.

@laurentdavid the PR looks solid! Lets get the tests to pass and merge this soon 👍

@laurentdavid Do you have intentions of revisiting this PR, it would be great to get it across the finish line.
@miankhalid If @laurentdavid doesn't pick this back up, is this something we're interested in continuing with?

@gsong /@miankhalid : I will do the ammends over the next two weeks. I noticed that there was some changes in the API on the master, so it might complicate the rebase a bit, but that should not be a major problem.

I will do the ammends over the next two weeks. I noticed that there was some changes in the API on the master, so it might complicate the rebase a bit, but that should not be a major problem.

Great! I'll review the amends asap this time.

I have changed the formatting accordingly and rebased from current master. That should be fine now.
Let me know if anything.

@laurentdavid I'll look @ this on monday, got hooked up in some of my own tasks today. Sorry!

All the tests are failing locally, is there any specific configuration that I need to do ?

Column 1
Column 2

Also, @BenjiLee this PR aims to mock the APIs that our E2E tests call, don't we want our E2E to query our live servers ?

Also, @BenjiLee this PR aims to mock the APIs that our E2E tests call, don't we want our E2E tests to query our live servers ?

This is true. As this PR is now, it that's the E out of E2E and makes it more of an integration test, this PR does open up the possibly of using mocked responses https://github.com/edx/edx-app-android/pull/885/files#diff-b5bcbfaa5913ba27827f156e377f59f4R25 with this line so we get some flexibility.
My thoughts were along the lines of, we currently do not runs these tests on CI and CI per PR probably shouldn't be running live E2E tests. If we can run these tests on CI without a network dependency they are less likely to get stale and we are more likely to be able to mock data for other tests that require it. Perhaps on the release build it should run live E2E tests but those also have some set up that still needs to be done. Thoughts?

My thoughts were along the lines of, we currently do not runs these tests on CI and CI per PR probably shouldn't be running live E2E tests. If we can run these tests on CI without a network dependency they are less likely to get stale and we are more likely to be able to mock data for other tests that require it. Perhaps on the release build it should run live E2E tests but those also have some set up that still needs to be done.

Agreed.

Hi,
Following up your conversation. true I was thinking along the line of the integration tests as it difficult to replicate the exact environment on the edx platform side. My first thought when I started with the app, that it was really hard to test without having a local install of the same infrastructure as you would have for a live E2E test (edx-platform + services + setup...). So this is aiming for an intermediate (in between releases where API might change for example) local / development testing/checks.
Laurent

@laurentdavid any comments on this: https://github.com/edx/edx-app-android/pull/885#issuecomment-284367852 ?

@miankhalid : Sorry for not answering your previous query. No special configuration. However I have slightly changed the mock to suit some API changes here: (ce78214157e58688d197c8fe8567c96764f967c4)
I was thinking that the pull request was automatically updated with the latest version of my branch (https://github.com/laurentdavid/edx-app-android/commits/laurentdavid/mock-android-tests), but for some reason it is not. Do I need to do something on my side ?

@miankhalid : sorry for the delay. Yes, the test are running fine on my side.
I have only corrected the tests in org.edx.mobile.test.feature so they are the ones that are passing. However I have noticed that if you run all tests in org.edx.mobile, some are failing (as they were before).

@miankhalid : sorry for the delay. Yes, the test are running fine on my side.
I have only corrected the tests in org.edx.mobile.test.feature so they are the ones that are passing. However I have noticed that if you run all tests in org.edx.mobile, some are failing (as they were before).

No problem. The tests finally ran successfully @ my end. What I had to do was to set the config flag NEW_LOGISTRATION_ENABLED to false.

@miankhalid : Hi! Just made the changes.

These points got missed:

https://github.com/edx/edx-app-android/pull/885#discussion_r104385369
https://github.com/edx/edx-app-android/pull/885#discussion_r104385834
https://github.com/edx/edx-app-android/pull/885#discussion_r104385992

And the commits aren't squashed to one commit.
Bonus Point: Can you make the commit message brief, maybe something like this https://github.com/edx/edx-app-android/pull/909/commits/8bebc4df3327c5cde01a631b1bfd8eb13407a4b2 commit's message?

poke @laurentdavid

@miankhalid : Sorry, did see your message, and told myself that I needed to get a bit of dedicated time to do it correctly. Did not have a chance since... But will do over the week!

Cool. Thanks!

Like you said in your notes, these tests are for a different use case; These are meant to be end-to-end acceptance tests with a live server, they shouldn’t use mock data, otherwise you aren't testing the integration.
If you want to test that a particular class in the app processes certain mock data correctly, a unit test is what you are after.

@miankhalid @laurentdavid Where are we on this PR? I noticed some changes were made, but not sure if all the comments have been addressed.

@miankhalid @brianguertin @BenjiLee
Where are we on this PR? do you guys think we actually need this PR after Brian's last comment?

@miankhalid @BenjiLee @brianguertin Can you please chime in on whether we want to continue with this PR or if it should be closed out?

Hi all,
At this stage I would understand that we drop the PR as as time has probably made it obsolete (and code has definitely changed).
I also understand @brianguertin 's comment regarding the use of unit test. However, I think there is still something missing in term of end to end testing on a local install.
I would say that my patch here was aiming to solve two different issues. And that's maybe what is confusing and make it awkward to integrate as it is:

first issue being that the current acceptance tests were outdated
the second one that, as far as I am aware, we cannot run the end to end tests without at least a properly configured local edX install (https://openedx.atlassian.net/browse/MA-2082). I was thinking that was a bit of a huge prerequisite, just to run the tests for the mobile app. as it does make it easy for someone "entering" in the project.

I would be happy to split that in two later on if it is deemed necessary, but I would probably need a bit of guidance on what the plans are (or existing practice) on both points (existing live server for testing and what's new is planned for end to end testing).
Laurent

Hello @laurentdavid: We are unable to continue with review of your submission at this time. Please see the associated JIRA ticket for more explanation.


Ignore connext dynamic package while support is paused

otherwise you get build errors

This makes it impossible to "enable" rmw_connext_dynamic_cpp on the build farm. Therefore I would suggest to create that file locally for now. Even if that is more effort.

ok we can add it to the source build instructions. alternatively it could just be present in the "beta" tag

Adding it just for the tagging and removing it directly afterwards sounds good me.

Instead of temporary ignoring this package #211 simply makes it compile again.

I think it would still be good to disable it in the beta branch to reduce the compile time for people building from source (the additionally generated message stuff and libraries/executables adds to the build time considerably).

Ok, we will add the ignore file just before beta and remove it immediately after tagging.


npm install fails in crouton

name: precise
encrypted: no
Entering /mnt/stateful_partition/crouton/chroots/precise...
crouton: version 1-20160715134041~master:0dc76a22
release: precise
architecture: i386
targets: core,cli-extra
host: version 7520.67.0 (Official Build) stable-channel x86-zgb_he
kernel: Linux localhost 3.8.11 #1 SMP Wed Jan 6 22:08:47 PST 2016 i686 i686 i386 GNU/Linux
freon: yes
Not unmounting /mnt/stateful_partition/crouton/chroots/precise as another instance is using it.

Please describe your issue:
Installing npm via apt-get works fine, but then when I attempt to run an npm install from a crouton chroot fails with a log like the following. I confirmed that all of the urls are all correct, and also tried adding --no-bin-links this issue from another repo, but it made no difference.
Here's what gets dumped to npm's debug log...
verbose cli [ 'node',
verbose cli   '/usr/bin/npm',
verbose cli   'install',
verbose cli   'bower',
verbose cli   'ember-cli',
verbose cli   '-g',
verbose cli   '--no-bin-links' ]
info using<EMAIL_ADDRESS>info using<EMAIL_ADDRESS>verbose config file /home/brad/.npmrc
verbose config file /usr/etc/npmrc
verbose config file /usr/share/npm/npmrc
silly exec /usr/bin/node "/usr/share/npm/bin/npm-get-uid-gid.js" "nobody" 1000
silly spawning [ '/usr/bin/node',
silly spawning   [ '/usr/share/npm/bin/npm-get-uid-gid.js', 'nobody', 1000 ],
silly spawning   null ]
silly output from getuid/gid {"uid":65534,"gid":1000}
silly output from getuid/gid
verbose cache add [ 'bower', null ]
silly cache add: name, spec, args [ undefined, 'bower', [ 'bower', null ] ]
verbose parsed url { pathname: 'bower', path: 'bower', href: 'bower' }
verbose cache add [ 'ember-cli', null ]
silly cache add: name, spec, args [ undefined, 'ember-cli', [ 'ember-cli', null ] ]
verbose parsed url { pathname: 'ember-cli', path: 'ember-cli', href: 'ember-cli' }
verbose addNamed [ 'bower', '' ]
verbose addNamed [ null, '' ]
silly name, range, hasData [ 'bower', '', false ]
verbose addNamed [ 'ember-cli', '' ]
verbose addNamed [ null, '' ]
silly name, range, hasData [ 'ember-cli', '', false ]
verbose raw, before any munging bower
verbose url resolving [ 'https://registry.npmjs.org/', './bower' ]
verbose url resolved https://registry.npmjs.org/bower
http GET https://registry.npmjs.org/bower
verbose raw, before any munging ember-cli
verbose url resolving [ 'https://registry.npmjs.org/', './ember-cli' ]
verbose url resolved https://registry.npmjs.org/ember-cli
http GET https://registry.npmjs.org/ember-cli
ERR! Error: failed to fetch from registry: ember-cli
ERR!     at /usr/share/npm/lib/utils/npm-registry-client/get.js:139:12
ERR!     at cb (/usr/share/npm/lib/utils/npm-registry-client/request.js:31:9)
ERR!     at Request._callback (/usr/share/npm/lib/utils/npm-registry-client/request.js:136:18)
ERR!     at Request.callback (/usr/lib/nodejs/request/main.js:119:22)
ERR!     at Request.<anonymous> (/usr/lib/nodejs/request/main.js:212:58)
ERR!     at Request.emit (events.js:88:20)
ERR!     at ClientRequest.<anonymous> (/usr/lib/nodejs/request/main.js:412:12)
ERR!     at ClientRequest.emit (events.js:67:17)
ERR!     at HTTPParser.onIncoming (http.js:1261:11)
ERR!     at HTTPParser.onHeadersComplete (http.js:102:31)
ERR! You may report this log at:
ERR!     <http://bugs.debian.org/npm>
ERR! or use
ERR!     reportbug --attach /home/brad/npm-debug.log npm
ERR!
ERR! System Linux 3.8.11
ERR! command "node" "/usr/bin/npm" "install" "bower" "ember-cli" "-g" "--no-bin-links"
ERR! cwd /home/brad
ERR! node -v v0.6.12
ERR! npm -v 1.1.4
ERR! message failed to fetch from registry: ember-cli

The registry URL does work, and it seems like the package downloads, but then something goes wrong after that. Same command works fine in a proper Linux install, so I don't think it's an issue with npm. Any ideas?

I'm really not sure, have you tried asking on some npm support forum?


Bump up go and pkgs version

Concerns
Should we forgo upgrading github.com/antonmedv/expr to v1.10.0 or higher because of the impact on the runn that uses this package?

I thought that the version of expr could be bumped up, as long as the runn side specifies the appropriate version of expand.

@k2tzumi Thank you!!


Stdio kernel client

New Class StdioKernelClient

StdioKernelClient uses ProcessStartInfo
StdioKernelClient.StartAsync() waits forKernelReady events
StdioKernelClient starts and manages the process

This can be combined with a test stdio client that we created for integration tests:
https://github.com/dotnet/interactive/blob/main/src/dotnet-interactive.IntegrationTests/TestStdioClient.cs


TFlite model signature lost after populating with metadata

1. System information

Mac OS 14.1.2
TensorFlow mac 2.13.0
Tflite support version '0.1.0a1'

2. Code
def write_metadata(model_path, run_name):
    model_meta = _metadata_fb.ModelMetadataT()
    model_meta.name = "test model"
    model_meta.description = run_name
    model_meta.version = datetime.now().strftime("%Y.%m.%d")
    model_meta.author = "Test"
    model_meta.license = f"test"\
        "All rights reserved - test"

    input_meta_image = _metadata_fb.TensorMetadataT()
    input_meta_image.description = "Input image for which to score doneness."
    input_meta_image.content = _metadata_fb.ContentT()
    input_meta_image.content.contentProperties = _metadata_fb.ImagePropertiesT()
    input_meta_image.content.contentProperties.colorSpace = _metadata_fb.ColorSpaceType.RGB
    input_meta_image.content.contentPropertiesType = _metadata_fb.ContentProperties.ImageProperties

    input_meta_state = _metadata_fb.TensorMetadataT()
    input_meta_state.description = "Input state."
    input_meta_state.name = "Input state."

    # Creates output info.
    output_meta_doneness = _metadata_fb.TensorMetadataT()
    output_meta_doneness.description = "Output doneness score between 0 and 1."
    output_meta_doneness.name = "Output Doneness"

    output_meta_state = _metadata_fb.TensorMetadataT()
    output_meta_state.description = "Output state."
    output_meta_state.name = "Output state."

    subgraph = _metadata_fb.SubGraphMetadataT()

    interpreter = tf.lite.Interpreter(model_path)
    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()
    state_first_input = input_details[0]['shape'][-1] != 3
    state_first_output = output_details[0]['shape'][-1] != 1

    input_meta = [input_meta_state, input_meta_image]
    output_meta = [output_meta_state, output_meta_doneness]

    subgraph.inputTensorMetadata = input_meta if state_first_input else input_meta[::-1]
    subgraph.outputTensorMetadata = output_meta if state_first_output else output_meta[::-1]

    model_meta.subgraphMetadata = [subgraph]
    b = flatbuffers.Builder(0)
    b.Finish(
        model_meta.Pack(b),
        _metadata.MetadataPopulator.METADATA_FILE_IDENTIFIER)
    metadata_buf = b.Output()

    populator = _metadata.MetadataPopulator.with_model_file(model_path)
    populator.load_metadata_buffer(metadata_buf)
    populator.populate()

I am using above function to populate the metadata. without running above function getting signature output like
interpreter.get_signature_list()
{'serving_default': {'inputs': ['input_image', 'input_state'], 'outputs': ['output_1', 'output_2']}}

after running metadata function
interpreter.get_signature_list()
{}

Is there any solution/comment on this ?

Hi @ pkgoogle,
Please look into the issue.
Thank You

Hi @pucha48, can you either upload your model file or show us the code that produces the model? Feel free to simplify or reduce the model so that it just shows the issue. Thanks for your help.

@pkgoogle I have added a sample code in zip, showing the sample issue. Please have a look.
Archive.zip

I was able to reproduce in this gist.
If I try with tf-nightly, it crashes in the 2nd cell
If I try with tflite-support (without the specific version) there is a name/package collision when importing flatbuffers
It appears to remove the signatures and writes to the file during this call:
populator = _metadata.MetadataPopulator.with_model_file(model_path)

Hi @miaout17, can you please take a look? Thanks.

Hi @pucha48 , if you are able to access a linux system you may be able to resolve your issue by using AI-Edge-Torch, you can find more information here: googleblog.
I have actually created a simple script for converting your model here , you can try adding metadata after converting your model:
import torch
import torch.nn as nn
import ai_edge_torch

class SimpleModel(nn.Module):
    def __init__(self):
        super(SimpleModel, self).__init__()
        self.fc1 = nn.Linear(4, 2)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(2, 1)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        x = self.relu(x)
        return x

model = SimpleModel()

example_input = torch.randn(1, 4)

edge_model = ai_edge_torch.convert(model.eval(), (example_input,))

edge_model.export('simple_model.tflite')

If you want to, you can actually try visualizing the result in model-explorer as well.
Please try them out and let us know if this resolves your issue. If you still need further help, feel free to open a new issue at the respective repo.


cts: Fix divide by zero error

See https://github.com/The-OpenROAD-Project/OpenROAD/pull/2823#issuecomment-1407308584 for details

The failing tests need attention before I review.

@maliberty this one should be ready to go


Feature/patcher

Codecov Report

Merging #8 (8a76255) into master (a7fb5dc) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master       #8   +/-   ##
=======================================
  Coverage   59.06%   59.06%
=======================================
  Files          12       12
  Lines         491      491
=======================================
  Hits          290      290
  Misses        121      121
  Partials       80       80

Flag
Coverage Δ

unittests
59.06% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7fb5dc...8a76255. Read the comment docs.


Change Order

The website must allow customers to easily cancel or modify their orders before shipping.

Use case : #001-0001

CON-1: The website must provide an easy-to-use GUI for users to cancel or edit their orders before shipping.
CON-2: The website should allow the user to change the shipping/billing address with the user before shipping by contacting customer service.
CON-3: The website has to send a confirmation email to customers when they cancel or modify their orders.


Add config define line for linux aarch64.

Testing: with this change I was able to get the JIT workloads in the  dependent project google/xls to work on a Raspberry Pi 4.
Please do let me know if there's any other testing that you'd like to see in order to accept this PR.

Good eye, thanks Chandler! Sorry I cribbed that from the previous llvm_host_triple value I was using in https://github.com/google/xls/issues/286#issuecomment-782557517 -- let me check this still works with this modification and I'll let you know, RPi takes a little while to build and test...

Looks good with it that way as well!
Read up, just out of curiosity, to compare to what was working before llvm_bazel and this also seems more correct! At first wondered if the triple was defined by a left-leaning three-way split, so that you could only have dashes in the last piece of the triple, but realized it's the env suffix, as documented on llvm::Triple ( https://github.com/llvm/llvm-project/blob/ed956554f96c0a37cc1f1291dbad303d3f1d495c/llvm/include/llvm/ADT/Triple.h#L24 )  -- and for why it was working before, parseOs does a startswith so whether the environment "quadruple" was provided or not probably ended up mostly the same (whether or not env was GNU). The more you know! (Random refs for archaeologists: https://github.com/llvm/llvm-project/blob/ed956554f96c0a37cc1f1291dbad303d3f1d495c/llvm/lib/Support/Triple.cpp#L555 https://github.com/llvm/llvm-project/blob/ed956554f96c0a37cc1f1291dbad303d3f1d495c/llvm/lib/Support/Triple.cpp#L513 )

FYI, we're in the process of moving this build configuration upstream (I should add that to the README). See https://groups.google.com/g/llvm-dev/c/dcis1ebRw28/. This change will be ported over, but just wanted to give you a heads up that our aim is to move these files upstream and then turn down this repo (pending me writing down how we're going to do that :grin:)

heads up that our aim is to move these files upstream and then turn down this repo

Ack, thanks for the heads up!


Fixing typing_extensions dependency

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@2901db0). Click here to learn what that means.
 Consider uploading reports for the commit fb668ef to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #650   +/-   ##
=======================================
  Coverage        ?   75.31%
=======================================
  Files           ?       26
  Lines           ?     1839
  Branches        ?        0
=======================================
  Hits            ?     1385
  Misses          ?      454
  Partials        ?        0

Flag
Coverage Δ

python
75.31% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Shap values for XGBoost Classifier as probabilities

Hi, I was wondering if there is a way to obtain shap values for XGBoost Classifier expressed as probabilities? It looks like they are displayed in different units by default since the values go beyond [0, 1] range.

Sorry for the delay in responding! I think this should be possible. Let me check into it and report back.

@Jurgita-DS, I have released auto-shap 0.2.1. In this version, boosting classifiers will automatically have the SHAP values in probabilities rather than log odds. I hope this helps! Let me know if you run into any issues.
https://pypi.org/project/auto-shap/

@micahmelling that's great, thank you! I will test it in the next couple of weeks.


[FEATURE] Non hive partition discovery and update.

Is your feature request related to a problem?

need manually update partition. after create table, user need to run MSCK statement to refresh table partition. if new partition added, user need to refresh partition of table.
spark data source table does not support non hive partition.

What solution would you like?

user don't need to run MSCK or ALTER to refresh partition.
support non hive partition.

What alternatives have you considered?
not yet.
Do you have any additional context?
n/a

Glue crawler doesn't support non hive partition. https://medium.com/@jjwl91/aws-glue-catalog-with-non-hive-partitions-ca7fc2dbf63f


[API] Product should have category and sub category

Add a Category object to the product as required field
Given User authorized
And User creates a product
Then Category field is required during the creation
And Sub category field is required during the creation

created product contains category and sub-category if this data provider in POST api/v1/products

Was done in another user story. I close it


Negative rounding and chamfers

cuboid() and cyl(), can do edge rounding and chamfering, but it would be useful to be able to make "negative" roundings and chamfers to make flanges at the top and bottom to make masks for rounded or chamfered holes.

Implemented in bf6cfd1d65e39f6b95faa27c7254dc880994a55c


Failed to run migrate.exe with error "ERROR: Value cannot be null. Parameter name: String"

Describe what is not working as expected.
If you are seeing an exception, include the full exceptions details (message and stack trace).
When running cmd ‘migrate.exe XXX.dll /connectionString="Server=tcp:xxx.database.chinacloudapi.cn,1433;Initial Catalog=xxx_prerelease;Persist Security Info=False;User ID=xxx;Password=xxx;MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30" /connectionProviderName="System.Data.SqlClient" /force  ’
Exception message:
ERROR: Value cannot be null.
Parameter name: String
Stack trace:

Steps to reproduce
run migrate.exe command under bin\Release folder

### Further technical details
EF version: (EntityFramework.6.2.0)
Operating system: (e.g. Windows 7)

When I use "Update-database" in package management console, it is ok.

But I need to do some automation work, so I have to use cmd for upgrading.

EF Team Triage: This issue is lacking enough information for us to be able to effectively triage it. In particular, it is missing the following information requested in the new issue template. Can you please provide this information?

Steps to reproduce

Ideally include a complete code listing that we can run to reproduce the issue.
Alternatively, you can provide a project/solution that we can run.
BTW we're not just doing this to be mean :smile:... we get a lot traffic on this project and it takes time to attempt to reproduce an issue based on fragments of information. In addition, our attempt is often unsuccessful as the exact conditions required to hit the issue are often not explicitly included in the code provided. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we ask that folks give us a self-contained way to reproduce an issue.
For a guide on submitting good bug reports, read Painless Bug Tracking.
BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.

EF Team Triage: Closing this issue as the requested additional details have not been provided and we have been unable to reproduce it.
BTW this is a canned response and may have info or details that do not directly apply to this particular issue. While we'd like to spend the time to uniquely address every incoming issue, we get a lot traffic on the EF projects and that is not practical. To ensure we maximize the time we have to work on fixing bugs, implementing new features, etc. we use canned responses for common triage decisions.


Create dashboard for monitoring

TBD: What should the dashboard show?

number of requests
size of data transferred
avg. request time
Error rate
IP geolocation of request origin

Create a Google Sheet of custom field mappings to CloudWatch logs.

Still blocked

It seems the Cloud Metrics team is working on optimizing their architecture and won't be able to get to Hydrocron log ingestion in the short term. In the short term, the API Gateway offers this dashboard:

@frankinspace and @torimcd - Does this give us enough info to describe usage patterns? Otherwise we could identify a few key logs that produce metrics we want to capture and create metrics from them. I have explored this too much but this documentation might be helpful if we go this route.


Postfixed keys different between computers

Please verify all of the below before submitting:

you have picked the right category for the issue in the previous screen.
You are on the latest release of Zotero
in the Zotero addons screen you can see that you have the latest release of BBT (https://github.com/retorquere/zotero-better-bibtex/releases/latest)
you are posting a single bug or feature request.
the issue has a subject that succinctly describes the problem or question.
you are available for follow-up questions and testing.
you have included an support log ID here generated by reproducing the problem, selecting the problematic reference(s), right-clicking, and submitting an BBT support log from that popup menu, not from the Help menu.
you have included the actual output you got from exporting the items you sent in the debug log, not simplified fictional output or a description of the output, and a sample (not a description) of how you want it to look instead, again based on the actual items you selected in the previous step.

Picking the right issue category is really important. Each category (Export, General error, Import, Key generation, Question) has different instructions for gathering the data necessary required to resolve the issue you are experiencing
The support log is important; it gives me your current BBT settings and a copy of the problematic reference as a test case so I can best replicate your problem. Without it, I'm effectively blind.
Support log ID:]
Computer 1: RMKW5HAY-apse
Computer 2: YULW4P2S-apse
Expected behavior:
Citekeys that are postfixed retain the same keys across computers after sync.
Actual behavior:
Two entries with postfixed citekeys (e.g., heyns2021 for Entry 1 and keyns2021a for Entry 2) have their citekeys switched after syncing Zotero (such that Computer 2 has heyns2021a for Entry 1 and keyns2021 for Entry 2).

There isn't much I can do about this (until Zotero creates a format citekey field), as Zotero does not tell me whether an item appears because it was synced in, or because it was just created by you. The closest you can get, I think, is:
On computer 1, set autoPin to whatever you deem reasonable. This will make the citekey all items that are created/modified from that moment on pinned. This includes items that are synced in. These pinned citekeys will sync back out, so eventually, all installations that sync with the account will get pinned keys get synced back.
On the other computers (in this case, just computer 2), set the conflict policy to "postfixed".
In the above scenario, both systems would generate the citekey heyns2021, but it would be pinned on computer 1, and when that syncs to computer 2, the existing heyns2021 should move out of the way... which I just now see is broken... I'll get to fixing that, but do you think you can live with the scenario I described?

Thanks! That makes sense. Is there a reason why Computer 1 couldn't also have a "postfixed" conflict policy? Seems to work either way for me.
However, more important for me is fixing existing entries. The only way to fix this automatically would be to pin all my entries and then sync, right? At the moment none of my entries are pinned. I supposed I imagined that it's more flexible this way in case I end up wanting to change the key scheme in the future (but this is unlikely). What are the downsides of pinning all the entries and then setting autoPin as you said?

Thanks! That makes sense. Is there a reason why Computer 1 couldn't also have a "postfixed" conflict policy? Seems to work either way for me.

For this scenario it doesn't matter what the key conflict policy is on computer 1. But I need to fix the keyconflict policy feature  before it'll work. I'll probably have something to test later today.

However, more important for me is fixing existing entries. The only way to fix this automatically would be to pin all my entries and then sync, right?

Correct -- once I fix the conflict policy.

At the moment none of my entries are pinned. I supposed I imagined that it's more flexible this way in case I end up wanting to change the key scheme in the future (but this is unlikely). What are the downsides of pinning all the entries and then setting autoPin as you said?

Just that they won't update as you change the entry. At some point you generally don't want that to happen anyway -- when you start using the keys in articles, the keys being stable is generally more important than that they differ in minor ways from the article they refer to.

For this scenario it doesn't matter what the key conflict policy is on computer 1. But I need to fix the keyconflict policy feature before it'll work. I'll probably have something to test later today.

Happy to help test whenever it's ready.

Just that they won't update as you change the entry. At some point you generally don't want that to happen anyway -- when you start using the keys in articles, the keys being stable is generally more important than that they differ in minor ways from the article they refer to.

Yes, I agree. Thanks!

That took a lot longer than I had expected, but build 676 should do it.

Thanks! That makes sense. Is there a reason why Computer 1 couldn't also have a "postfixed" conflict policy? Seems to work either way for me.

In this scenario, the keys on computer 1 will end up pinned, so the conflict policy won't actually do anything in the long run. Pinned keys don't move.

Great, thanks! I will only be able to test with the two computers on Monday, so I'll check back in then.
To clarify, this isn't mean to "properly" fix the issue, but should allow me to now do what you suggested above? That is, pin all entries on one computer, enable autoPin, enable the "postfixed" conflict policy (can be on both computers), and then sync.

This will be hard to structurally test for you, and I've added a test case to replay an approximation of your scenario, so you don't have to put in extra effort to validate it works -- knowing you have a workable solution would just give me closure. I'm going to roll this out in a new release anyhow, since this functionality was simply missing.

To clarify, this isn't mean to "properly" fix the issue, but should allow me to now do what you suggested above? That is, pin all entries on one computer, enable autoPin, enable the "postfixed" conflict policy (can be on both computers), and then sync.

Correct. This is as proper as I'm able to fix it -- a proper fix would mean the databases would have to know about each other, and the only realistic way to achieve this is to use the Zotero sync mechanism. When Zotero gains a citekey field, which is said to be Real Soon Now, but that's been the status for a few years, for fairly good reasons, it will be fixed structurally, as you'll just get a sync conflict and the UI that Zotero already has for this will allow you to resolve the clash.

Pinning all the entries and enabling autoPin is a workable solution for me since it means that postfixed citekeys stay consistent across computers after syncing. Thanks for the help!


rt-scope along with rt-repeat fails

The following template

{item}

produces

Uncaught ReferenceError: z is not defined

The problem seems to be rt-scope and rt-repeat attributes appearing on the same tag.
Anyway by looking at the source code, this seems something that has been worked on and that presumably worked in the past:

// Order matters. We need to add the item and itemIndex to context.boundParams before
// the rt-scope directive is processed, lest they are not passed to the child scopes

and also:

// Order matters here. Each rt-repeat iteration wraps over the rt-scope, so
// the scope variables are evaluated in context of the current iteration.

As a fix to this problem, I propose to change the way injected functions are generated. Instead of having them all at the top level, we can nest functions the same way they appear in the template, thus making scope/naming automatic.
Example of how would be rendered the template shown above:
var templateRT = function () {
    function scope1() {
        function repeat2(item, itemIndex) {
            return React.createElement('div', {}, React.createElement('div', {}, item));
        }
        var z = [1, 2, 3];
        return _.map(z, repeat2.bind(this));
    }
    return scope1.apply(this);
};

This change would be beneficial also for #156

ok, got it wrong, actually the current behavior is adherent to the docs:

When used with rt-repeat, the rt-scope is evaluated for every iteration, so that iteration's item and itemIndex are in scope.


Notify only on state change

Hi,
My team is using sensu to replace nagios. We are newcomers to sensu. It took us many days of tedious work to get it working, but now it works, we get alerts by mail and Slack... Well, too many of them actually. We are getting alerts every 30 minutes for the very same failing checks.
Is there a way to get a sane configuration, that is, to get notified only once, when the check begins to fail, and then not until it is resolved? Nagios does this out of the box. That's the obvious default.
The documentation says "A Sensu Event is created when a check result indicates a change in state." and, with respect to the observable behavior, this looks like a complete lie.
I've read this which seems to contradict the official documentation but gets it right:

Event: An event is generated every time a check is done. It’s the result of the check, positive or negative, there’s always going to be one.

I've read this blog post and frankly I don't really get it. It seems to focus on performance, whereas I'm simply looking for reasonable behavior.
So... As of today, is there a (simple and clean) way to get notifications only on actual state-changes?

Hi @hchargois,
Thank you for raising this issue. You're absolutely correct; the documentation is lying. We'll fix that today.
I think the following filter would provide the behavior you are describing:
{
  "filters": {
    "state_change_only": {
      "attributes": {
        "occurrences": "eval: value == 1 || :::action::: == :resolve"
      }
    }
  }
}

This filter works because the value of event occurrences is reset on state change, except when the event action is :resolve, e.g. as it is when returning status 0 after a previous non-zero status. You can find instructions for creating filter definitions and associating them with handlers here: https://sensuapp.org/docs/latest/reference/filters.html
Will you please try applying this filter and let me know if it provides the behavior you expect?

I've updated the documentation to better describe how events are created. Thanks again for bringing this discrepancy to our attention.

Thanks a lot for your speedy answer!
The documentation is much clearer now.
Your filter's logic is very straightforward, that's great. I tried it verbatim and it doesn't work, I have this in the server logs:
{
  "timestamp": "2016-08-05T17:25:29.904855+0200",
  "level": "error",
  "message": "filter attribute eval error",
  "raw_eval_string": "eval: value == 1 || :::action::: == :resolve",
  "error": "undefined local variable or method `create' for Kernel:Module"
  [...]
}

I guess that's because :::action::: gets replaced before evaluation and stands there without quotes or anything, so the ruby thinks it must be an identifier but doesn't know of it.
I tried:
"eval: value == 1 || \":::action:::\" == \"resolve\""

And it works!
Thanks again!

Hi @hchargois, thanks for reporting your success with the above and your feedback on the documentation change.
Interesting that you needed to change the filter. In my test case the value of the event's action attribute comes through as a symbol, which is why I didn't need to quote the strings.
FWIW I tested the filter I described on Sensu 0.25.6, so if you are on an older version the filter may need to change when you upgrade.
I'd like to add this filter to our documentation as an example; can you help me by sharing which version of Sensu you are using with your modified filter?

I'd like to add this filter to our documentation as an example

👍

quick doc feedback:  (worried I won't get back to  doing a PR or providing fuller response. )
the new docs currently say:

Sensu events are created to acknowledge that something potentially noteworthy has occurred

as well as:

A Sensu Event is created every time a check result is processed by the Sensu server, regardless of the status indicated by the check result

this sounds contradictory , or that all check results are noteworthy.     and doesn't make it clear that green check results are generally filtered from the handlers (unless special conditions are involved such as the result being a metric )   so by default a handler for normal events with no configured filters sees failing results with status > 0 (:create, :flapping),  and the 1 successful resolution event with status 0  (:resolve)
hope that helps.

I'd like to add this filter to our documentation as an example; can you help me by sharing which version of Sensu you are using with your modified filter?

Weird... I'm also using 0.25.6 :confused:

I've tried both (just like your example @cwjohnston and placed in /etc/sensu/conf.d/filters.json)
"eval: value == 1 || :::action::: == :resolve"

and
"eval: value == 1 || \":::action:::\" == \"resolve\""

but still getting alerts every 30 minutes. Could there be something else I'm missing?
I'm using 0.25.6

@scosist have you explicitly configured one or more handlers to use the filter?

Ah, I glossed over that part (How do Sensu filters work?) of the filters reference. Thanks for the hint!
"filter": "state_change_only",

Looks like it's working now and I'm using the original example you provided--the eval without the extra escaped quotes. One thing to note, the filter(s) attribute must be defined on the handler and not the handler set.

Closing this as I've opened issues to track needed documentation improvements. Thanks for the feedback all!

I thought I should mention I experienced the same error that @hchargois did.
"error":"undefined local variable or method `create' for Kernel:Module"

This occurred after I switched from using the handler filter attribute to the filters attribute with more than one filter. Perhaps this has something to do with the check result containing "action":"create"? Also, once I started using more than one filter I stopped getting any alerts at all.
Filters:
    "state_change_only": {
      "negate": false,
      "attributes": {
        "occurrences": "eval: value == 1 || :::action::: == :resolve"
      }
    },
    "state_change_interval_60_daily": {
      "negate": false,
      "attributes": {
        "check": {
          "interval": 60
        },
        "occurrences": "eval: value == 1 || value % 1440 == 0 || :::action::: == :resolve"
      }
    }

Handler filters attribute:
      "filters": [
        "state_change_only",
        "state_change_interval_60_daily"
      ],

When I use @hchargois' workaround with the escaped double quotes, I no longer get the error but I'm still not getting any alerts.
Forgive me if I shouldn't comment on a closed issue; I believe this best applies here but if not, let me know and I'll create a new issue.

After further examination it seems I didn't test the original eval thoroughly enough (in the case of using one filter). I remember testing it to make sure it silenced subsequent handler alerts on the same state but I didn't test it for sending an OK alert on the :resolve state change. Instead it logs an error:
"error":"undefined local variable or method `resolve' for Kernel:Module"

Although sometimes instead of method 'resolve' the error is on method 'create'. Also, sometimes it does send the OK alert but it doesn't seem to be reliable.
I found that using the following eval (with the action encased in single quotes) is sufficient (and more readable than escaping double quotes) for alerting on all state changes including OK and eliminates the logged error:
"eval: value == 1 || ':::action:::' == :resolve"

Unfortunately, this discovery does not resolve my issue of receiving zero alerts when using multiple filters.

@scosist @hchargois in my testing I've learned that the only reason "eval: value == 1 || :::action::: == :resolve" (unquoted token and symbol comparison) appears to work is that the first half of the logic (value == 1) sometimes evaluates to true, which causes the rest of the condition to be skipped.
For any event where occurrences is not == 1, the filter will cause events to be dropped because of the undefined local variable or method 'resolve' for Kernel:Module error encountered when evaluating that second condition.
For future reference here is the working version using single quotes as described above:
{
  "filters": {
    "state_change_only": {
      "attributes": {
        "occurrences": "eval: value == 1 || ':::action:::' == 'resolve'"
      }
    }
  }
}

This doesn't seem to alert on warning and critical states for checks with something other than the default of 1 for occurrences, for example 2. I've tried a number of variations including:
"eval: value == :::check.occurrences|1::: || ':::action:::' == 'resolve'"

What would allow this filter to handle checks on state changes regardless of their occurrences value? Or is this not supported yet?

I still don't know whether the issue with the above example is a problem with the check.occurrences token or something else. I'm assuming that when a check definition has "occurrences": 2 and the check result occurrences have reached 2 and an event is thus ready to be handled for the check, that the 'value' of value in the eval will also be 2 and thus return true for the value == :::check.occurrences|1::: statement, but if this were true I would expect it to alert on all state changes. Is it possible there is some confusion with the check definition occurrences attribute value and the check result occurrences value when using that token or is the documentation  correct?
Either way, although I like it less, I think I've found a workaround. As long as I keep any of my occurrences check definitions to 5 or less the following filter seems to work for alerting on all state changes:
{
  "filters": {
    "state_change_only": {
      "negate": true,
      "attributes": {
        "occurrences": "eval: value > 5 && ':::action:::' != 'resolve'"
      }
    }
  }
}

If anyone has any insights as to the problem above, please share.
@cwjohnston any chance we could reopen this? It seems the most recent example you provided only works entirely as expected when using the default value of 1 for occurrences on all check definitions.

HI,
I'm trying to use the following filter:
"occurrences": "eval: value == 1 || ':::action:::' == 'resolve'"
which seems to filter some of the resolved events.
and advice what can be done in order to get notifications only on state change properly?
Thanks
Tal

@ctal80 You did not provide any details...just generic statement.

following is an example where we can see that the event was filtered although action":"resolve:
{"timestamp":"2017-03-21T12:51:25.237159+0200","level":"info","message":"event was filtered","handler":{"type":"pipe","command":"/opt/sensu/embedded/bin/ruby /etc/sensu/plugins/handler-slack-custom.rb","filter":"state_change_only","name":"slack"},"event":{"id":"cf86d95e-2a65-48da-b7cd-7bfc0116943d","client":{"name":"P4_proxy_AMR","address":"unknown","subscriptions":[],"keepalives":false,"version":"0.20.3"},"check":{"command":"powershell.exe -ExecutionPolicy ByPass -f /opt/sensu/plugins/test.ps1 p4 " ","interval":3600,"subscribers":["roundrobin:web_check"],"handlers":["mailer","slack"],"source":"P4_proxy_AMR","name":"P4_Proxy_AMR_Check","issued":1490093482,"executed":1490093482,"duration":2.804,"output":"837:45:02\r\nCheck OK: \n","status":0,"history":["1","1","1","1","1","1","1","1","0","0","1","0","1","1","0","1","1","1","1","1","0"],"total_state_change":37},"occurrences":5,"action":"resolve","timestamp":1490093485}}


♻ Big refactor

✨ Add new converter structure
🧩 Split out a lot of logic to be tested and reused more easily
🧪 Lots of unit tests
🔬 Unit test coverage report
🕸 Add export for using the lib from CDN
👀 Example of how to use the lib

I made the changes @Lenni009 , shall we ship it? 😁

Let's Gamble, Try Merging :D


Cant copy lxd container in ceph due to fs mishap

Distribution: ubuntu 18

LXD version: 4.0.6
Storage backend in use: ceph

This is not with all containers just few, it seems that container have stored fs type somewhere wrongly. Container might have been converted by copying from one pool to another from ext4 to xfs because fstab did have ext4( somehow still booted).
lxc copy unimall-dev/snap0 unimall-devcopy --target neutron-star
Error: Create instance from copy: Failed to mount "/dev/rbd3" on "/var/snap/lxd/common/lxd/storage-pools/ceph-lxd/containers/unimall-devcopy" using "ext4": invalid argument
` lxc storage volume show ceph-lxd container/unimall-dev
config:
block.filesystem: xfs
block.mount_options: discard
size: 35GB
description: ""
name: unimall-dev
type: container
used_by:

/1.0/instances/unimall-dev
location: ""
content_type: filesystem
`

lxc copy unimall-dev unimall-devcopy --target neutron-star t=2021-06-28T05:29:08+0000 lvl=dbug msg="New event listener: 60a54907-79ca-4401-ac65-cd6c15f98976" t=2021-06-28T05:29:08+0000 lvl=dbug msg=Handling ip=@ method=POST protocol=unix url="/1.0/instances?target=neutron-star" username=uni058 t=2021-06-28T05:29:08+0000 lvl=dbug msg="\n\t{\n\t\t\"architecture\": \"x86_64\",\n\t\t\"config\": {\n\t\t\t\"boot.autostart\": \"false\",\n\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\"image.description\": \"ubuntu 18.04 LTS amd64 (release) (20200519.1)\",\n\t\t\t\"image.label\": \"release\",\n\t\t\t\"image.os\": \"ubuntu\",\n\t\t\t\"image.release\": \"bionic\",\n\t\t\t\"image.serial\": \"20200519.1\",\n\t\t\t\"image.version\": \"18.04\",\n\t\t\t\"volatile.base_image\": \"70d3dcaabcffb1aa1644d0ce866efcb141742179e94ad72aefb8d3502338a71f\"\n\t\t},\n\t\t\"devices\": {\n\t\t\t\"root\": {\n\t\t\t\t\"path\": \"/\",\n\t\t\t\t\"pool\": \"ceph-lxd\",\n\t\t\t\t\"size\": \"36GB\",\n\t\t\t\t\"type\": \"disk\"\n\t\t\t}\n\t\t},\n\t\t\"ephemeral\": false,\n\t\t\"profiles\": [\n\t\t\t\"ceph-ssd\"\n\t\t],\n\t\t\"stateful\": false,\n\t\t\"description\": \"\",\n\t\t\"name\": \"unimall-devcopy\",\n\t\t\"source\": {\n\t\t\t\"type\": \"copy\",\n\t\t\t\"certificate\": \"\",\n\t\t\t\"base-image\": \"70d3dcaabcffb1aa1644d0ce866efcb141742179e94ad72aefb8d3502338a71f\",\n\t\t\t\"source\": \"unimall-dev\"\n\t\t},\n\t\t\"instance_type\": \"\",\n\t\t\"type\": \"container\"\n\t}" t=2021-06-28T05:29:08+0000 lvl=dbug msg="Responding to instance create" t=2021-06-28T05:29:08+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.last_state.power project=default sourceInstance=unimall-dev sourceProject=default t=2021-06-28T05:29:08+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.eth0.hwaddr project=default sourceInstance=unimall-dev sourceProject=default t=2021-06-28T05:29:08+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.idmap.current project=default sourceInstance=unimall-dev sourceProject=default t=2021-06-28T05:29:08+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.idmap.base project=default sourceInstance=unimall-dev sourceProject=default t=2021-06-28T05:29:08+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.uuid project=default sourceInstance=unimall-dev sourceProject=default t=2021-06-28T05:29:08+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.idmap.next project=default sourceInstance=unimall-dev sourceProject=default t=2021-06-28T05:29:08+0000 lvl=dbug msg="New task Operation: bbef4c47-4847-43a2-8dfa-1aa68b103390" t=2021-06-28T05:29:08+0000 lvl=dbug msg="Started task operation: bbef4c47-4847-43a2-8dfa-1aa68b103390" t=2021-06-28T05:29:08+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/bbef4c47-4847-43a2-8dfa-1aa68b103390\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"bbef4c47-4847-43a2-8dfa-1aa68b103390\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Creating instance\",\n\t\t\t\"created_at\": \"2021-06-28T05:29:08.450697507Z\",\n\t\t\t\"updated_at\": \"2021-06-28T05:29:08.450697507Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/containers/unimall-devcopy\",\n\t\t\t\t\t\"/1.0/containers/unimall-dev\"\n\t\t\t\t],\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/unimall-devcopy\",\n\t\t\t\t\t\"/1.0/instances/unimall-dev\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"neutron-star\"\n\t\t}\n\t}" t=2021-06-28T05:29:08+0000 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url="/1.0/operations/bbef4c47-4847-43a2-8dfa-1aa68b103390?target=neutron-star" username=uni058 t=2021-06-28T05:29:08+0000 lvl=info msg="Creating container" ephemeral=false instance=unimall-devcopy instanceType=container project=default t=2021-06-28T05:29:08+0000 lvl=dbug msg="FillInstanceConfig started" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default t=2021-06-28T05:29:08+0000 lvl=dbug msg="FillInstanceConfig finished" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default t=2021-06-28T05:29:08+0000 lvl=info msg="Created container" ephemeral=false instance=unimall-devcopy instanceType=container project=default t=2021-06-28T05:29:08+0000 lvl=dbug msg="CreateInstanceFromCopy started" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default snapshots=true src=unimall-dev t=2021-06-28T05:29:08+0000 lvl=dbug msg="CreateInstanceFromCopy same-pool mode detected" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default snapshots=true src=unimall-dev t=2021-06-28T05:29:09+0000 lvl=dbug msg="Found cert" name=0 t=2021-06-28T05:29:09+0000 lvl=dbug msg="Replace current raft nodes with [{ID:8 Address:<IP_ADDRESS>:8443 Role:voter} {ID:9 Address:<IP_ADDRESS>:8443 Role:stand-by} {ID:10 Address:<IP_ADDRESS>:8443 Role:stand-by} {ID:11 Address:<IP_ADDRESS>:8443 Role:voter} {ID:6 Address:<IP_ADDRESS>:8443 Role:voter}]" t=2021-06-28T05:29:09+0000 lvl=dbug msg="DeleteInstance started" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default t=2021-06-28T05:29:09+0000 lvl=dbug msg="Deleting instance volume" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default volName=unimall-devcopy t=2021-06-28T05:29:09+0000 lvl=dbug msg="DeleteInstance finished" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default t=2021-06-28T05:29:09+0000 lvl=dbug msg="CreateInstanceFromCopy finished" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default snapshots=true src=unimall-dev t=2021-06-28T05:29:09+0000 lvl=info msg="Deleting container" created=2021-06-28T05:29:08+0000 ephemeral=false instance=unimall-devcopy instanceType=container project=default used=1970-01-01T00:00:00+0000 t=2021-06-28T05:29:09+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"No such object\"}" t=2021-06-28T05:29:09+0000 lvl=info msg="Deleted container" created=2021-06-28T05:29:08+0000 ephemeral=false instance=unimall-devcopy instanceType=container project=default used=1970-01-01T00:00:00+0000 t=2021-06-28T05:29:09+0000 lvl=dbug msg="Failure for task operation: bbef4c47-4847-43a2-8dfa-1aa68b103390: Create instance from copy: exit status 2" t=2021-06-28T05:29:09+0000 lvl=dbug msg="Event listener finished: 60a54907-79ca-4401-ac65-cd6c15f98976" t=2021-06-28T05:29:09+0000 lvl=dbug msg="Disconnected event listener: 60a54907-79ca-4401-ac65-cd6c15f98976"
lxc copy unimall-dev/snap0 unimall-devcopy --target neutron-star
dmesg EXT4-fs (rbd3): VFS: Can't find ext4 filesystem
t=2021-06-28T05:34:22+0000 lvl=dbug msg=Handling ip=@ method=POST protocol=unix url="/1.0/instances?target=neutron-star" username=uni058
t=2021-06-28T05:34:22+0000 lvl=dbug msg="\n\t{\n\t\t\"architecture\": \"x86_64\",\n\t\t\"config\": {\n\t\t\t\"boot.autostart\": \"false\",\n\t\t\t\"image.architecture\": \"amd64\",\n\t\t\t\"image.description\": \"ubuntu 18.04 LTS amd64 (release) (20200519.1)\",\n\t\t\t\"image.label\": \"release\",\n\t\t\t\"image.os\": \"ubuntu\",\n\t\t\t\"image.release\": \"bionic\",\n\t\t\t\"image.serial\": \"20200519.1\",\n\t\t\t\"image.version\": \"18.04\",\n\t\t\t\"volatile.base_image\": \"70d3dcaabcffb1aa1644d0ce866efcb141742179e94ad72aefb8d3502338a71f\"\n\t\t},\n\t\t\"devices\": {\n\t\t\t\"root\": {\n\t\t\t\t\"path\": \"/\",\n\t\t\t\t\"pool\": \"ceph-lxd\",\n\t\t\t\t\"size\": \"36GB\",\n\t\t\t\t\"type\": \"disk\"\n\t\t\t}\n\t\t},\n\t\t\"ephemeral\": false,\n\t\t\"profiles\": [\n\t\t\t\"ceph-ssd\"\n\t\t],\n\t\t\"stateful\": false,\n\t\t\"description\": \"\",\n\t\t\"name\": \"unimall-devcopy\",\n\t\t\"source\": {\n\t\t\t\"type\": \"copy\",\n\t\t\t\"certificate\": \"\",\n\t\t\t\"base-image\": \"70d3dcaabcffb1aa1644d0ce866efcb141742179e94ad72aefb8d3502338a71f\",\n\t\t\t\"source\": \"unimall-dev/snap0\"\n\t\t},\n\t\t\"instance_type\":
\"\",\n\t\t\"type\": \"\"\n\t}"
t=2021-06-28T05:34:22+0000 lvl=dbug msg="Responding to instance create"
t=2021-06-28T05:34:22+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.idmap.next project=default sourceInstance=unimall-dev/snap0 sourceProject=default
t=2021-06-28T05:34:22+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.eth0.hwaddr project=default sourceInstance=unimall-dev/snap0 sourceProject=default
t=2021-06-28T05:34:22+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.uuid project=default sourceInstance=unimall-dev/snap0 sourceProject=default
t=2021-06-28T05:34:22+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.idmap.base project=default sourceInstance=unimall-dev/snap0 sourceProject=default
t=2021-06-28T05:34:22+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.idmap.current project=default sourceInstance=unimall-dev/snap0 sourceProject=default
t=2021-06-28T05:34:22+0000 lvl=dbug msg="Skipping key from copy source" instance=unimall-devcopy key=volatile.last_state.power project=default sourceInstance=unimall-dev/snap0 sourceProject=default
t=2021-06-28T05:34:22+0000 lvl=dbug msg="New task Operation: 00786f4c-6741-4a70-855b-8f3bf6a29f1c"
t=2021-06-28T05:34:22+0000 lvl=dbug msg="Started task operation: 00786f4c-6741-4a70-855b-8f3bf6a29f1c"
t=2021-06-28T05:34:22+0000 lvl=dbug msg="\n\t{\n\t\t\"type\": \"async\",\n\t\t\"status\": \"Operation created\",\n\t\t\"status_code\": 100,\n\t\t\"operation\": \"/1.0/operations/00786f4c-6741-4a70-855b-8f3bf6a29f1c\",\n\t\t\"error_code\": 0,\n\t\t\"error\": \"\",\n\t\t\"metadata\": {\n\t\t\t\"id\": \"00786f4c-6741-4a70-855b-8f3bf6a29f1c\",\n\t\t\t\"class\": \"task\",\n\t\t\t\"description\": \"Creating instance\",\n\t\t\t\"created_at\": \"2021-06-28T05:34:22.508386577Z\",\n\t\t\t\"updated_at\": \"2021-06-28T05:34:22.508386577Z\",\n\t\t\t\"status\": \"Running\",\n\t\t\t\"status_code\": 103,\n\t\t\t\"resources\": {\n\t\t\t\t\"containers\": [\n\t\t\t\t\t\"/1.0/containers/unimall-devcopy\",\n\t\t\t\t\t\"/1.0/containers/unimall-dev/snap0\"\n\t\t\t\t],\n\t\t\t\t\"instances\": [\n\t\t\t\t\t\"/1.0/instances/unimall-devcopy\",\n\t\t\t\t\t\"/1.0/instances/unimall-dev/snap0\"\n\t\t\t\t]\n\t\t\t},\n\t\t\t\"metadata\": null,\n\t\t\t\"may_cancel\": false,\n\t\t\t\"err\": \"\",\n\t\t\t\"location\": \"neutron-star\"\n\t\t}\n\t}"
t=2021-06-28T05:34:22+0000 lvl=dbug msg=Handling ip=@ method=GET protocol=unix url="/1.0/operations/00786f4c-6741-4a70-855b-8f3bf6a29f1c?target=neutron-star" username=uni058
t=2021-06-28T05:34:22+0000 lvl=info msg="Creating container" ephemeral=false instance=unimall-devcopy instanceType=container project=default
t=2021-06-28T05:34:22+0000 lvl=dbug msg="FillInstanceConfig started" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default
t=2021-06-28T05:34:22+0000 lvl=dbug msg="FillInstanceConfig finished" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default
t=2021-06-28T05:34:22+0000 lvl=info msg="Created container" ephemeral=false instance=unimall-devcopy instanceType=container project=default
t=2021-06-28T05:34:22+0000 lvl=dbug msg="CreateInstanceFromCopy started" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default snapshots=true src=unimall-dev/snap0
t=2021-06-28T05:34:22+0000 lvl=dbug msg="CreateInstanceFromCopy same-pool mode detected" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default snapshots=true src=unimall-dev/snap0
t=2021-06-28T05:34:23+0000 lvl=dbug msg="Activated RBD volume" dev=/dev/rbd3 driver=ceph pool=ceph-lxd vol=container_unimall-devcopy
t=2021-06-28T05:34:30+0000 lvl=dbug msg="Found cert" name=0
t=2021-06-28T05:34:30+0000 lvl=dbug msg="Replace current raft nodes with [{ID:10 Address:<IP_ADDRESS>:8443 Role:stand-by} {ID:11 Address:<IP_ADDRESS>:8443 Role:voter} {ID:6 Address:<IP_ADDRESS>:8443 Role:voter} {ID:8 Address:<IP_ADDRESS>:8443 Role:voter} {ID:9 Address:<IP_ADDRESS>:8443 Role:stand-by}]"
t=2021-06-28T05:34:34+0000 lvl=dbug msg="Deactivated RBD volume" driver=ceph pool=ceph-lxd vol=container_unimall-devcopy
t=2021-06-28T05:34:42+0000 lvl=dbug msg="Found cert" name=0
t=2021-06-28T05:34:42+0000 lvl=dbug msg="Replace current raft nodes with [{ID:6 Address:<IP_ADDRESS>:8443 Role:voter} {ID:8 Address:<IP_ADDRESS>:8443 Role:voter} {ID:9 Address:<IP_ADDRESS>:8443 Role:stand-by} {ID:10 Address:<IP_ADDRESS>:8443 Role:stand-by} {ID:11 Address:<IP_ADDRESS>:8443 Role:voter}]"
t=2021-06-28T05:34:45+0000 lvl=dbug msg="DeleteInstance started" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default
t=2021-06-28T05:34:45+0000 lvl=dbug msg="Deleting instance volume" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default volName=unimall-devcopy
t=2021-06-28T05:34:45+0000 lvl=dbug msg="DeleteInstance finished" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default
t=2021-06-28T05:34:45+0000 lvl=dbug msg="CreateInstanceFromCopy finished" driver=ceph instance=unimall-devcopy pool=ceph-lxd project=default snapshots=true src=unimall-dev/snap0
t=2021-06-28T05:34:45+0000 lvl=info msg="Deleting container" created=2021-06-28T05:34:22+0000 ephemeral=false instance=unimall-devcopy instanceType=container project=default used=1970-01-01T00:00:00+0000
t=2021-06-28T05:34:45+0000 lvl=dbug msg="Database error: &errors.errorString{s:\"No such object\"}"
t=2021-06-28T05:34:45+0000 lvl=info msg="Deleted container" created=2021-06-28T05:34:22+0000 ephemeral=false instance=unimall-devcopy instanceType=container project=default used=1970-01-01T00:00:00+0000
t=2021-06-28T05:34:45+0000 lvl=dbug msg="Failure for task operation: 00786f4c-6741-4a70-855b-8f3bf6a29f1c: Create instance from copy: Failed to mount \"/dev/rbd3\" on \"/var/snap/lxd/common/lxd/storage-pools/ceph-lxd/containers/unimall-devcopy\" using \"ext4\": invalid argument"
t=2021-06-28T05:34:45+0000 lvl=dbug msg="Event listener finished: 0dba4df1-59c8-4319-9f98-b51db862d1cb"
t=2021-06-28T05:34:45+0000 lvl=dbug msg="Disconnected event listener: 0dba4df1-59c8-4319-9f98-b51db862d1cb"

Okay, so some kind of filesystem mismatch. Reading this is a bit confusing though, to clarify, could you show:

lxc storage show ceph-lxd
lxc volume show ceph-lxd container/unimall-dev
lxc volume show ceph-lxd container/unimall-dev/snap0

Sorry for confusion this is old installation and at some point I have changed block_filesystem (not manually) and few container may be converted via copy from ceph-hdd to ceph-ssd. Probably thats why this container in fstab still had ext4 in fstab.
lxc storage show ceph-lxd
config:
  ceph.cluster_name: ceph
  ceph.osd.pg_num: "128"
  ceph.osd.pool_name: ceph-lxd
  ceph.user.name: lxd
  volatile.pool.pristine: "true"
  volume.size: 35GB
description: ""
name: ceph-lxd
driver: ceph
used_by:

lxc storage volume show ceph-lxd container/unimall-dev
config:
  block.filesystem: xfs
  block.mount_options: discard
  size: 35GB
description: ""
name: unimall-dev
type: container
used_by:
- /1.0/instances/unimall-dev
location: ""
content_type: filesystem

lxc storage volume show ceph-lxd container/unimall-dev/snap0
description: ""
expires_at: 0001-01-01T01:41:16+01:41
name: snap0
config:
  block.filesystem: xfs
  block.mount_options: discard
  size: 35GB
content_type: ""

Thanks, that should be enough for us to reproduce it. Looks like LXD is not considering the volume config in this case which is wrong.

This sounds very similar to https://github.com/lxc/lxd/issues/8301

Got it reproduced here:
root@vm01:~# lxc storage set ceph volume.block.filesystem=xfs volume.block.mount_options=discard
root@vm01:~# lxc launch images:alpine/edge a1
Starting a1
root@vm01:~# lxc snapshot a1
root@vm01:~# lxc storage unset ceph volume.block.filesystem
root@vm01:~# lxc storage unset ceph volume.block.mount_options
root@vm01:~# lxc storage show ceph
config:
  ceph.cluster_name: ceph
  ceph.osd.pg_num: "32"
  ceph.osd.pool_name: persist-vm01
  ceph.user.name: admin
  source: persist-vm01
  volatile.initial_source: persist-vm01
  volatile.pool.pristine: "true"
description: ""
name: ceph
driver: ceph
used_by:
- /1.0/images/3cc5f40527ea7055beccefcedd8a89b7b8b488037a288078e7681e815543b576
- /1.0/instances/a1
- /1.0/profiles/default
status: Created
locations:
- none
root@vm01:~# lxc storage volume show ceph container/a1
config:
  block.filesystem: xfs
  block.mount_options: discard
description: ""
name: a1
type: container
used_by:
- /1.0/instances/a1
location: ""
content_type: filesystem
root@vm01:~# lxc storage volume show ceph container/a1/snap0
description: ""
expires_at: 0001-01-01T00:00:00Z
name: snap0
config:
  block.filesystem: xfs
  block.mount_options: discard
content_type: filesystem
root@vm01:~# lxc stop a1
root@vm01:~# lxc copy a1/snap0 a2
Error: Create instance from copy: Failed to mount "/dev/rbd0" on "/var/snap/lxd/common/lxd/storage-pools/ceph/containers/a2" using "ext4": invalid argument
root@vm01:~#

@tomponline so if I'm reading things right, the issue is that we're just passing a rootDiskConf as the config of the volume rather than pulling the current volume's config and using rootDiskConf as an override to apply on top.
As a result, config options like volume.* which the source may have, end up being discarded.

@tomponline sent a fix for it. Basically the issue is that the instance code which creates the DB entries has no idea it's dealing with a copy and has no access to the original volume config.
As a result, it creates a new volume or volume snapshot with a blank config and moves on. This then gets passed to the storage driver which fails when trying to actually mount the thing.
I don't think we've found the right spot for the storage volume DB logic quite yet. It's definitely quite awkward to have the instance driver handle the DB record creation in this case. Ideally, having the storage backend handle the DB bits would be great and would make such things easier (as we'd deal with volumes, have source and target and could sync up the configs) but short of that, maybe moving that logic over to instance.go would already make things a bit simpler.
In any case, to solve this bug, I've added logic which will now update the volume config immediately after its creation by the instance driver, syncing it up with the source so long as they reside on the same storage pool (as cross pool will most likely involve rsync which can handle with a change of volume config). This is done prior to snapshots being handled so new snapshots just inherit this copied config and all is well.
I've confirmed that both lxc copy a1/snap0 a2 and lxc copy a1 a2 works properly. Resulting volumes are all properly configured to use xfs while any new instance will use ext4 as expected.

@stgraber

I don't think we've found the right spot for the storage volume DB logic quite yet. It's definitely quite awkward to have the instance driver handle the DB record creation in this case. Ideally, having the storage backend handle the DB bits would be great and would make such things easier (as we'd deal with volumes, have source and target and could sync up the configs) but short of that, maybe moving that logic over to instance.go would already make things a bit simpler.

I would whole heartedly support moving the instance storage volume DB record creation/update/deletion into the storage layer, this is the logical place to do it, and also brings it inline with what we do for custom volumes.
I didn't do this at the time of the storage package creation because the instance DB record creation logic was already quite interwoven with the instance storage DB record creation, and as it was already a large change, I didn't feel confident tackling that at the time. However now the storage package is well bedded in and I am a lot more familiar now with the instance/instance storage volume relationship, especially as recently making the revert logging more nuanced, I think this would be a good idea to tackle. Shall I add to the ideas board?

Yep, that'd be good.