[RFC] Integrating SQL/PPL query languages into DSL via the _search API

Is your feature request related to a problem? Please describe
Today, OpenSearch offers support for SQL and PPL query languages through the plugin endpoints _plugins/sql and _plugins/ppl. However, clients using OpenSearch client libraries face limitations, as these libraries do not accommodate plugin endpoints. To increase adoption with minimal disruption, our proposal introduces new SQL and PPL clauses directly into the DSL. This approach aims to facilitate the use of these languages through the Search API, streamlining access and integration for users.
Describe the solution you'd like
In this approach, we aim to extend the existing SearchPlugin interface by incorporating a new function that introduces the QueryEngineSpec construct. Through this construct, plugins that implement the SearchPlugin interface can introduce their own query engines. These engines will take over the search request process to produce the appropriate SearchResponse.
The QueryEngineSpec serves a crucial role by defining the name of the specification. This name acts as a key within the DSL, under which all parameters specific to the query engine's request are grouped. When OpenSearch-core receives a search request featuring a clause with a key that matches a QueryEngineSpec, it initiates the corresponding Query Engine. The core system then forwards the request to the plugin through this Query Engine Object.

Sample SearchRequest for ppl query:
localhost:9200/_search
{
   "ppl" : {
        "query" : "source = accounts"
    }
}

Sample SearchResponse Structure.
{
 "took": 0,
"timed_out": false,
"_shards": {
        "total": 0,
        "successful": 0,
        "skipped": 0,
        "failed": 0
},
"hits": {
   "dummy"
},
"ppl": {
    "schema": [...],
    "datarows" : [....]
    }
}

Open Questions.

Can we leverage SearchExtBuilder(ext block in SearchRequest and SearchResponse) instead of creating a new ppl and sql clause in SearchRequest and SearchResponse.{
       "total" : "",
       "_shards" : {},
       "hits" : {},
       "ext" : {
          "sql" : {
              "schema" : [...],
              "datarows" : [....]
           }
        }
 }

Describe alternatives you've considered
Approach 2: Leverage Search Pipelines and introduce new SQL/PPL processors.
In the current state, search pipelines offer the following types of processors:

Request Processors → Transform SearchRequest.
Response Processors → Transform SearchResponse.
Search Phase Results Processors → Transform SearchResults between the query and fetch phases.

The Flow framework proposes to include a new type of search processor in Pipelines. This search processor would take a SearchRequest and produce a SearchResponse. We can leverage this new Search Processor type to introduce SQLSearchProcessor and PPLSearchProcessor, which take over the Search Request whenever there is an SQL and PPL block in the search request, respectively. Since an SQL request is a blocking operation, we would build something like processResponseAsync, which takes in a SearchResponse Listener. For including SQL/PPL-related request and response bodies, we could leverage the already existing ext clauses feature in SearchResponse and SearchRequest.

Request Flow

User sends a _search request with sql/ppl block in ext block.
SearchSourceBuilder would parse this block and will put the extBuilders in SearchRequest.
[Need further deep dive] SearchPipelineService identifies these extBuilders and include SQLSearchProcessors and PPLSearchProcessors respectively in the Pipeline. Should we include the SQL and PPL search processors in default pipeline ?
SQL or PPL SearchProcessor would respond back with the SearchResponse through listener passed from the CORE.

Approach 3: Route in rest layer to SQL/PPL Transport Actions.
In this approach, within the REST layer, the search API is rerouted to leverage TransportSQLAction and TransportPPLAction instead of using TransportSearch action when a SQL or PPL block is detected in the search request body. However, this method has a drawback for clients, especially plugins utilizing NodeClients, that depend on transport actions for search requests. Thus, it's more advantageous to develop a solution at the level below the TransportAction layer.
Open Questions and Edge Case Scenarios.

We will be supporting sql and ppl in _search API but not /_search  API as index name would already be present in the SQL and PPL queries.
How do we handle scenarios where a search request body contains a SQL/PPL block along with search DSL clauses or DSL related query parameters?

[Option 1] Throw Validation Error.
[Option 2] Ignore the unknown parameters and treat them as a sql/ppl request.

How do we support explain APIs in PPL and SQL API?

We can leverage existing explain query param and send the explain response back in ppl block and sql block.

For SQL/PPL, can we exclude unrelated parameters that a usual SearchResponse contains, for example: took, _shards, hits, etc.?

The current idea is to fill them with -1 and null, hinting to the customer that those fields are invalid and that the entire response will be in either the ext block or ppl block.

POC Video:

Why not modify the client so that it access the plugins/sql endpoint?

What is the usecase for this approach?

@vamsi-amazon have you thought introducing the ppl/sql as a another query clause rather than coming with a new concept Query engine?
Currently in Opensearch you can define a query type along with how to parse and convert the query into apt Lucene query clause. Wondering if we have explored that option and what is the reason for not choosing that option and rather than building a new concept all together?

[Triage - attendees 1 2 3 4 5]
@vamsi-amazon Thanks for filing this RFC looking forward to seeing where this topic lands.

However, clients using OpenSearch client libraries face limitations, as these libraries do not accommodate plugin endpoints.

Where is this problem explored?  If client library support was improved it would benefit the SQL plugin and all other plugins for OpenSearch.

Why not modify the client so that it access the plugins/sql endpoint?

We already support driver in JDBC / ODBC and dbapi. other opensearch clients would need to support the jdbc/odbc spec and enable access via SQL / PPL; this can be done in the fullness of time. Our goals are not just client user access / but also developer access without introducing inter-plugin dependencies. Many of our users still use the dsl and hand craft DSL queries.
The proposal is not to move code but maintain code modularity by adding a QueryEngineSpec.

@vamsi-amazon have you thought introducing the ppl/sql as a another query clause rather than coming with a new concept Query engine?
Currently in Opensearch you can define a query type along with how to parse and convert the query into apt Lucene query clause. Wondering if we have explored that option and what is the reason for not choosing that option and rather than building a new concept all together?
This will have many advantages:

User can fit this new ppl/sql query clause with any other complex query.
You will get out box support for various features of search like concurrent segment search etc.
Aggregations and other features can be directly supported with this.

SQL is an independent high-level query language hence 1 doesn't apply ; 2,3 can still be used

How does approach 1 works with OpenSearch client library? Do we plan to upgrade OpenSearch client library to support new SQL/PPL query type?

no, it doesn't need to work out of the box. SQL drivers JDBC/ODBC/DBAPI will continue to work for the developers and users.

What is the usecase for this approach?
Integrating SQL/PPL into OpenSearch as standard languages enhances its utility and accessibility. For users, it promises compatibility with JDBC/ODBC and DBAPI clients, opening up OpenSearch to a wider audience. All features, including dashboards, will eventually support SQL/PPL by default, increasing usability. For developers, incorporating these features into the core simplifies development, avoids plugin dependencies while ensures backward compatibility, making OpenSearch a more unified platform for querying. This move positions OpenSearch as a leading relevancy-focused SQL engine with advanced capabilities like highlighting and full-text queries.

PPL reference manual
https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/index.rst
SQL reference manual
https://github.com/opensearch-project/sql/blob/main/docs/user/index.rst
developer docs
https://github.com/opensearch-project/sql/blob/main/docs/dev/index.md
Getting started developer guide
https://github.com/opensearch-project/sql/blob/main/DEVELOPER_GUIDE.rst
Drivers
https://opensearch.org/downloads.html
this approach will/

Streamlines access to SQL and PPL through the standard Search API, enhancing usability.
Encourages broader adoption by making SQL and PPL features more accessible to users unfamiliar with plugin-specific endpoints.
Supports extensibility through the QueryEngineSpec, allowing for custom query engine implementations.
Improves system architecture by leveraging existing interfaces and patterns, promoting a more unified and coherent platform design.
Enhances the flexibility of OpenSearch by accommodating various query languages within a unified framework.
Addresses current limitations and gaps in functionality with respect to SQL and PPL usage in OpenSearch plugins/ecosystem.
Aims to maintain backward compatibility and minimize disruption to existing workflows and applications.
enable opensearch plugin like alerting to support SQL and PPL based alerts

Opensearch client don't need to support SQL / PPL by default- they are supported by the jdbc/odbc spec'ed drivers and dbapi. Since this is an optional clause clients can ignore it. Search pipelines is not a low level feature to implement a fundamental query language.

Streamlines access to SQL and PPL through the standard Search API, enhancing usability.

This is the part that bugs me. It's not using the standard search API.
We want to access SQL/PPL with JDBC/ODBC clients. Sure. The requests are not _search API requests (i.e. SearchRequest). The responses are not _search API responses (i.e. SearchResponse).
Given that we have no interest in SearchRequest and SearchResponse,  what does this have to do with the _search API?
For example, could I add a QueryEngineSpec called math, where I send a _search request, like:
localhost:9200/_search
{
   "math" : {
        "query" : "5 * 10 + 3"
    }
}

Then I get back a response like:
{
 "took": 0,
"timed_out": false,
"_shards": {
        "total": 0,
        "successful": 0,
        "skipped": 0,
        "failed": 0
},
"hits": {
   "dummy"
},
  "math": {
    "answer": 53
  }
}

Is that something we want to support? What things go into the _search API versus their own APIs? Does it make sense to read cluster settings from _search APIs?
There's nothing stopping me from adding a /_math endpoint via a plugin that can support its own API directly:
// Request:
localhost:9200/_math
{
  "expression" : "5 * 10 + 3"
}

// Response:
{
  "answer": 53
}

Is that something we want to support? What things go into the _search API versus their own APIs?

This is the right question, for which I think we need some tenets.
To me, search is defined by 1) parses a query written in some language, 2) evaluates every stored document against that query, 3) matches or doesn't match the document, 4) produces a score for all documents that match, then 5) sorts results my score and 6) returns them.
Is this an acceptable definition @msfroh?
If so, in the case of the math example or settings you're missing 2), 3), 4) and 5), so it doesn't fit under search. In the case of SQL I think it fits that definition.

We want to access SQL/PPL with JDBC/ODBC clients. Sure. The requests are not _search API requests (i.e. SearchRequest). The responses are not _search API responses (i.e. SearchResponse).

This is confusing to me. I suppose I don't understand internals. I think there should be a SearchRequest and SearchResponse independent of the transport API, aka we need RestSearchRequest < SearchRequest, ODBCSearchRequest < SearchRequest, etc.

This is confusing to me. I suppose I don't understand internals. I think there should be a SearchRequest and SearchResponse independent of the transport API, aka we need RestSearchRequest < SearchRequest, ODBCSearchRequest < SearchRequest, etc.

Aha! I like this.
I think this gets into some of the question of "What is the input used to perform an internal operation versus what is the representation sent over the wire?" that touches on the challenge that @VachaShah has encountered on her Protobuf work, made difficult by the fact that business objects have historically defined their own wire format.
I believe the approach in this proposal is "How can we embed a SQL/PPL representation of a search request inside the existing REST _search API?" Maybe instead, it should be "How can the _search API accommodate different representations of a search request?"
This almost feels like we want to support a different Content-Type for the API (albeit with a more significant interpretation of Content-Type versus the existing XContent framework from which the "business objects == serialized objects" evil arises.) Of course, dispatching a request to a /_search endpoint and forking the logic by Content-Type isn't fundamentally different from just hitting a different endpoint.
Of course, once we're on the cluster, we're forking down completely different paths. The existing SearchRequest class is married to query DSL and is remarkably low-level in its specificity. I gather that the SQL/PPL logic goes and does very different "stuff" that may eventually trigger DSL queries of its own. Ultimately, I don't think we could reasonably say that a SQL/PPL request "extends" a SearchRequest without moving, well, everything into the separate RestSearchRequest -- once you've moved the DSL-specific stuff out, there's not much left.

In the case of SQL I think it fits that definition where the language to express the query is different.

@dblock I think by your definition the took, _shards, hits, etc. fields in the response would be relevant for any search request, but this proposal explicitly calls them "unrelated parameters". If we truly need to exclude those fields then it feels like we're shoehorning something into the API similar to @froh's contrived math example.

I think I see a way that we can handle this, albeit in two steps:

On the response side, we add support for the schema/datarows concept as an OpenSearch core feature, where anyone can request it (even if they're doing a DSL query). This is arguably a better response format for a lot of use-cases and it's a good feature. (While Lucene and therefore OpenSearch supports a flexible schema where every doc can have its own set of fields, in practice you tend to return a bunch of docs with the same fields -- otherwise it's really hard to use.)
On the request side, we use something like this QueryEngine proposal to process a SearchRequest (including different syntax) and get back a SearchResponse. In this case, from an architecture standpoint, I would perhaps suggest (as in @dblock's message above) we consider it as "different" from a regular REST search request, and we fork off at the REST layer.

That way, we preserve the "SearchResponse returns documents" part that my contrived math example doesn't (though it could send an answer back in a document 😄 ). From a code standpoint, we could split off before trying to parse into a SearchSourceBuilder.

In the case of SQL I think it fits that definition where the language to express the query is different.

@dblock I think by your definition the took, _shards, hits, etc. fields in the response would be relevant for any search request, but this proposal explicitly calls them "unrelated parameters". If we truly need to exclude those fields then it feels like we're shoehorning something into the API similar to @froh's contrived math example.
While I agree with @dblock and @msfroh , lets do this lets to try and standardize;
-> but, if we see the DSL structure today; there is to spec or structure.

Opensearch response today is divided in to roughly 2 parts

response metadata ( shards, hits etc. )
response data

we should be able to fill all the response metadata fields; but the response data format I propose be jdbc spec'ed. that is easy/simple to use and understand no matter what aggregation is used.

eg.
{
 "took": 23.1,
"timed_out": false,
"_shards": {
        "total": 2,
        "successful": 2,
        "skipped": 0,
        "failed": 0
},
"hits": {
   "324"
},
  "ppl": {
    "schema": [...],
    "datarows" : [....]
    }
}

@dblock wrt how you are defining tenets/
i agree, we should have search first experience, not math for example.
SQL /PPL support almost-all opensearch relevancy features in an easy to use high level language
https://github.com/opensearch-project/sql/blob/main/docs/user/beyond/fulltext.rst
this would be the only / most powerful SQL dialect that support all relevancy features in a SQL / Piped language which is on a search engine
agree with @msfroh on the final comment here/
https://github.com/opensearch-project/OpenSearch/issues/12434#issuecomment-2000281781

@vamsi-amazon Thoughts on updating the proposal above with the information discussed?

agree with @msfroh on the final comment of support for the schema/datarows concept
we should also align on the scope and launch criteria of support SQL in _search endpoint

SQL does not support all the DSL query and aggregation, for instance, SQL does not support geo and shape queries. In case mixing SQL and native DSL queries in a single endpoint, any concern?
search on index (index00001/_search) does not align with sql from syntax, user can select any table in sql statement
scroll does not align with sql pagination syntax.
scroll / pit search does not align with sql pagination syntax
not all search URI parameters can be supported. for instance, suggest_field
not all search body can be used along with SQL, for instance, docvalue_fields

we should also

Performance benchmark, SQL latency / resource usage should similar to DSL
we also need to align on that new feature introduced in DSL should has correspond implementation in SQL.
SQL support existing DSL features. It is not P0 feature, we can keep improvement.

@dblock  Got busy with 2.13 release for last couple of days. Will update the proposal with the information discussed.

Maybe instead, it should be "How can the _search API accommodate different representations of a search request?"

This is truly awesome question and problem statement we should take deeper look if we want to support different query engine.
But here are some thoughts I have after reading the conversation:

If we have different representation of what a SearchRequest can be basically translating the RestSearchRequest to bunch of engine specific Search Request aren't we just using OpenSearch as _search as a proxy layer to call different engine(say SQL hosted on a remote endpoint).
RestSearchRequest can handle more than 1 type of engine, BulkRequest or IndexRequest should also support methods to index/put data into underline storage of Query engine.
If do support lets say both Index and Search request to different engine, what we built out is Opensearch as a thin distributed system. So to avoid getting into this, we need to tie in Opensearch Index some how with Index and Search Request. Otherwise what is point of having a data-node and shard etc.? because all you need is a machine that is just converting one type of payload to engine specific payload aka Query.

@penghuo you have good examples of how SQL doesn't align well with other queries
@navneet1v Does your question boil down to the fact that search needs to be aware of what kind of index it is?
I think both comments are just generic questions of what's constant and what's variable (OO abstractions). For example scroll / pit search does not align with sql pagination syntax - "scrolling" is a common feature over data, aka there should be a set of interfaces that together implement scrolling, but inputs may differ between SQL and DSL, and engines can have different implementations and effectively perform scrolling functions differently, all while data fetching or distributing requests to shards are common.

Summarizing the recent discussions above and evaluation regarding the integration of SQL into the OpenSearch core.
We explored various strategies listed in the updated description/ for incorporating SQL into the OpenSearch core, ultimately recommending against integration in favor of enhancing client capabilities.
The initial approaches considered involve directly integrating SQL into the _search endpoint or as a new endpoint within the core, each presenting distinct advantages. However, these strategies also face significant drawbacks, including limited parity in DSL query support, non-uniform response structures, increased complexity in core repo, potential build system integration challenges and significantly increased cost in upfront development with no extra customer value over the current approach.
The preferred approach advocates for maintaining the current setup with added transport clients for SQL, presenting a minimal change strategy that ensures compatibility with other plugins.
(Preferred): No Core Integration; Enhance Client Capabilities and Add Cohesion in core dashboards features.
The preferred approach is aiming to introduce a transport API for SQL and a new transport client library. This method ensures SQL compatibility with other plugins while maintaining the current system's integrity and minimizing changes.
“
The most compelling reasons I would see for merging the SQL code into core is if we think that long term the SQL query engine might want to integrate directly into the low-level Lucene query engine and there might benefits to having the query DSL implementation living side-by-side with the SQL implementation. We'd probably want to tease the entire current query DSL engine out of the server module into its own thing, which could then also contain the SQL implementation. However, I don't really see this happening or a need for this anytime soon. SQL depends on and uses the DSL and custom scripting via DSL; It inherits all DSL advances that the core project will make.
If the plan is to keep the SQL implementation as either a plugin or module component in the core repository, then moving the code from one repository to another seems more like an administrative decision than a software architecture one. “/paraphrasing @andrross /

@dblock https://github.com/opensearch-project/OpenSearch-Dashboards/issues/7081 if you have some time to check this one out too?


raidboss: update Ala Mhigo for 6.4

Timeline changes, primarily to the second boss.

Are there any oopsy changes, here or in the other 6.4 updates?

I haven't been checking for oopsy changes, but from what I saw of the timelines, the action ids are the same, just the timing changed for some things or some actions were removed (e.g., no more claws chasing you to your body in Ala Mhigo 2nd boss).
Biggest impact to oopsy would be the second boss of Castrum Abania since it's an all new encounter.

Ok ok! If you have any time to look while you have logs and are looking through it, I'd definitely appreciate the oopsy end of things too, since you also have the freshest knowledge on what exists or not.  It doesn't have to be perfect, but even just "these things don't exist anymore" -> remove and "here's some new ability ids" -> add (or a TODO) would be good.


Cassandra 3.x Plugin

Please answer these questions before submitting pull request

Why submit this pull request?
[ ] Bug fix
[x] New feature provided
[ ] Improve performance

New feature or improvement
http://cassandra.apache.org/

Please fix CI.

The test case is required here, https://github.com/SkyAPMTest/agent-auto-integration-testcases

The test case is required here, https://github.com/SkyAPMTest/agent-auto-integration-testcases

yes, i am doing that.

Link to your test case pull request at here, and post your local test result please. And you need to update Supported-list.md.

Link to your test case pull request at here, and post your local test result please. And you need to update Supported-list.md.

https://github.com/SkyAPMTest/agent-auto-integration-testcases/pull/84
[INFO] ---------------------------Here is the test report context -------------------------------------
Test Report - 2019-08-28 11:20

tests  : 4 passed. 0 failed
branch name : features/cassandra-3.x-plugin
commit id : a2715a94e103cff29e5b776c2692fddc3a48d890
cases branch: master
cases commit id: e0f5b4dab588ce307d889167627bbbfe28e2c6e0

Cases List

Case description
Status
Cases

Cassandra
4 passed. 0 failed
click me

Cassandra

Version
Passed
Failed

3.6.0
:heavy_check_mark:

3.7.2
:heavy_check_mark:

3.7.0
:heavy_check_mark:

3.7.1
:heavy_check_mark:

[INFO] -----------------------------------end the report-----------------------------------------------

@zhaoyuguang Please run the tests in our INFRA.

@stone-wlg The result is 2 passed, 2 failure. Could you recheck what happen?

@stone-wlg The result is 2 passed, 2 failure. Could you recheck what happen?
this is test results that i did one minutes ago
[INFO] ---------------------------Here is the test report context -------------------------------------

Test Report - 2019-08-29 10:05

tests  : 4 passed. 0 failed
branch name : features/cassandra-3.x-plugin
commit id : e9c4e1c3da38d50d2dcc21a23623764581c39cab
cases branch: master
cases commit id: bb1a8b32a66ea7695706a0922c22f924169ee206

Cases List

Case description
Status
Cases

Cassandra
4 passed. 0 failed
click me

Cassandra

Version
Passed
Failed

3.6.0
:heavy_check_mark:

3.7.2
:heavy_check_mark:

3.7.0
:heavy_check_mark:

3.7.1
:heavy_check_mark:

[INFO] -----------------------------------end the report-----------------------------------------------

@stone-wlg The result is 2 passed, 2 failure. Could you recheck what happen?
Again, i clear container and image manually, and test it again.
[INFO] ---------------------------Here is the test report context -------------------------------------

Test Report - 2019-08-29 10:31

tests  : 4 passed. 0 failed
branch name : features/cassandra-3.x-plugin
commit id : e9c4e1c3da38d50d2dcc21a23623764581c39cab
cases branch: master
cases commit id: bb1a8b32a66ea7695706a0922c22f924169ee206

Cases List

Case description
Status
Cases

Cassandra
4 passed. 0 failed
click me

Cassandra

Version
Passed
Failed

3.6.0
:heavy_check_mark:

3.7.2
:heavy_check_mark:

3.7.0
:heavy_check_mark:

3.7.1
:heavy_check_mark:

[INFO] -----------------------------------end the report-----------------------------------------------

Could the check the test log? There is a link there.

Could the check the test log? There is a link there.

ok, i have updated testcase code. please re-run it.

@stone-wlg Still 2 fails.

@stone-wlg Still 2 fails.

i have checked test report 1 and test report 2, it seems like env issue. :) Could you re-run it several times, i suppose that will give us different results.

i have checked test report 1 and test report 2, it seems like env issue. :) Could you re-run it several times, i suppose that will give us different results.

What kinds of env issue?

i have checked test report 1 and test report 2, it seems like env issue. :) Could you re-run it several times, i suppose that will give us different results.

What kinds of env issue?

i pick some words from test report,
"value": "java.lang.IllegalArgumentException: Failed to add contact point: cassandra-server\nat com.datastax.driver.core.Cluster$Builder.addContactPoint(Cluster.java:922)\nat test.apache.skywalking.testcase.cassandra.controller.CaseController
i suppose that cassandra server is not ready or it is starting, while test case application access it.

Do you finish your health check API in a real way? That is how you make sure you sever is standing by.

Do you finish your health check API in a real way? That is how you make sure you sever is standing by.

is there a sample that is health check API to check cassandra database (not server) is ready or not ? or i need to check in my code, ex: loop sleep(1s)

There is an API required in your test case, in there, you could use codes to check the target server.

There is an API required in your test case, in there, you could use codes to check the target server.
ok

There is an API required in your test case, in there, you could use codes to check the target server.

i update codes, please re-run it again. thanks

There is an API required in your test case, in there, you could use codes to check the target server.

i update codes, please re-run it again. thanks

wait a moment

Here is the test report and validate logs

Hi, i update config, this time i will test 3.7.2 and 3.7.1 at first, later i will test 3.7.0 and 3.6.0. Thanks

Please fix conflict, we have merged a new plugin.

Please fix conflict, we have merged a new plugin.

done

/run ci

/run ci
./mvnw -P"agent,backend,ui,dist,CI-with-IT" org.jacoco:jacoco-maven-plugin:0.8.3:prepare-agent clean install org.jacoco:jacoco-maven-plugin:0.8.3:report

[INFO] server-configuration ............................... SUCCESS [  0.079 s]
[INFO] configuration-api .................................. SUCCESS [  1.596 s]
[INFO] library-client ..................................... FAILURE [  2.084 s]
[INFO] library-server ..................................... SKIPPED
[INFO] server-testing ..................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  03:25 min
[INFO] Finished at: 2019-08-30T14:08:47+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.30.0:start (prepare-elasticsearch) on project library-client: Execution prepare-elasticsearch of goal io.fabric8:docker-maven-plugin:0.30.0:start failed: No  given, no DOCKER_HOST environment variable, no read/writable '/var/run/docker.sock' or '//./pipe/docker_engine' and no external provider like Docker machine configured -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :library-client

/run ci
sudo ./mvnw -P"agent,backend,ui,dist,CI-with-IT" org.jacoco:jacoco-maven-plugin:0.8.3:prepare-agent clean install org.jacoco:jacoco-maven-plugin:0.8.3:report

[INFO] apm ................................................ SUCCESS [ 11.310 s]
[INFO] apm-commons ........................................ SUCCESS [  0.220 s]
[INFO] apm-util ........................................... SUCCESS [  3.575 s]
[INFO] apm-datacarrier .................................... SUCCESS [ 17.681 s]
[INFO] apm-protocol ....................................... SUCCESS [  0.105 s]
[INFO] apm-network ........................................ SUCCESS [  4.700 s]
[INFO] apm-sniffer ........................................ SUCCESS [  0.257 s]
[INFO] apm-agent-core ..................................... SUCCESS [ 21.717 s]
[INFO] apm-agent .......................................... SUCCESS [  3.720 s]
[INFO] apm-test-tools ..................................... SUCCESS [  1.190 s]
[INFO] apm-sdk-plugin ..................................... SUCCESS [  0.257 s]
[INFO] dubbo-plugin ....................................... SUCCESS [  2.200 s]
[INFO] jdbc-commons ....................................... SUCCESS [  5.732 s]
[INFO] httpclient-4.x-plugin .............................. SUCCESS [  2.257 s]
[INFO] jedis-2.x-plugin ................................... SUCCESS [  2.306 s]
[INFO] redisson-3.x-plugin ................................ SUCCESS [  1.954 s]
[INFO] tomcat-7.x-8.x-plugin .............................. SUCCESS [  2.189 s]
[INFO] motan-plugin ....................................... SUCCESS [  2.871 s]
[INFO] mongodb-plugin ..................................... SUCCESS [  2.104 s]
[INFO] apm-feign-default-http-9.x-plugin .................. SUCCESS [  2.141 s]
[INFO] okhttp-3.x-plugin .................................. SUCCESS [  2.283 s]
[INFO] apm-sdk-plugin ..................................... SUCCESS [  0.108 s]
[INFO] async-annotation-plugin ............................ SUCCESS [  0.240 s]
[INFO] spring-commons ..................................... SUCCESS [  0.180 s]
[INFO] concurrent-util-4.x-plugin ......................... SUCCESS [  0.276 s]
[INFO] resttemplate-4.3.x-plugin .......................... SUCCESS [  0.272 s]
[INFO] mvc-annotation-commons ............................. SUCCESS [  0.240 s]
[INFO] mvc-annotation-4.x-plugin .......................... SUCCESS [  3.100 s]
[INFO] spring-cloud ....................................... SUCCESS [  0.066 s]
[INFO] netflix-plugins .................................... SUCCESS [  0.065 s]
[INFO] spring-cloud-feign-1.x-plugin ...................... SUCCESS [  0.533 s]
[INFO] mvc-annotation-3.x-plugin .......................... SUCCESS [  1.481 s]
[INFO] core-patch ......................................... SUCCESS [  1.196 s]
[INFO] mvc-annotation-5.x-plugin .......................... SUCCESS [  0.164 s]
[INFO] webflux-5.x-plugin ................................. SUCCESS [  0.156 s]
[INFO] struts2-2.x-plugin ................................. SUCCESS [  2.117 s]
[INFO] apm-sdk-plugin ..................................... SUCCESS [  0.080 s]
[INFO] http-1.x-plugin .................................... SUCCESS [  1.968 s]
[INFO] mvc-annotation-1.x-plugin .......................... SUCCESS [  0.148 s]
[INFO] jetty-plugin ....................................... SUCCESS [  0.088 s]
[INFO] jetty-client-9.x-plugin ............................ SUCCESS [  2.006 s]
[INFO] jetty-server-9.x-plugin ............................ SUCCESS [  2.097 s]
[INFO] jetty-client-9.0-plugin ............................ SUCCESS [  2.037 s]
[INFO] spymemcached-2.x-plugin ............................ SUCCESS [  1.985 s]
[INFO] sharding-jdbc-1.5.x-plugin ......................... SUCCESS [  2.104 s]
[INFO] sharding-sphere-3.x-plugin ......................... SUCCESS [  1.931 s]
[INFO] sharding-sphere-4.x-plugin ......................... SUCCESS [  1.983 s]
[INFO] xmemcached-2.x-plugin .............................. SUCCESS [  2.096 s]
[INFO] grpc-1.x-plugin .................................... SUCCESS [  0.344 s]
[INFO] apm-mysql-commons .................................. SUCCESS [  2.291 s]
[INFO] mysql-5.x-plugin ................................... SUCCESS [  1.272 s]
[INFO] mysql-6.x-plugin ................................... SUCCESS [  0.274 s]
[INFO] mysql-8.x-plugin ................................... SUCCESS [  1.174 s]
[INFO] h2-1.x-plugin ...................................... SUCCESS [  0.188 s]
[INFO] postgresql-8.x-plugin .............................. SUCCESS [  0.266 s]
[INFO] rocketMQ-3.x-plugin ................................ SUCCESS [  2.734 s]
[INFO] rocketMQ-4.x-plugin ................................ SUCCESS [  2.926 s]
[INFO] elastic-job-2.x-plugin ............................. SUCCESS [  1.780 s]
[INFO] apm-mongodb-2.x-plugin ............................. SUCCESS [  1.790 s]
[INFO] httpasyncclient-4.x-plugin ......................... SUCCESS [  2.138 s]
[INFO] apm-kafka-v1-plugin ................................ SUCCESS [  3.607 s]
[INFO] apm-sdk-plugin ..................................... SUCCESS [  0.093 s]
[INFO] servicecomb-java-chassis-0.x-plugin ................ SUCCESS [  3.934 s]
[INFO] servicecomb-java-chassis-1.x-plugin ................ SUCCESS [  4.193 s]
[INFO] apm-hystrix-1.x-plugin ............................. SUCCESS [  1.101 s]
[INFO] sofarpc-plugin ..................................... SUCCESS [  3.032 s]
[INFO] activemq-5.x-plugin ................................ SUCCESS [  2.509 s]
[INFO] elasticsearch-5.x-plugin ........................... SUCCESS [  0.602 s]
[INFO] undertow-plugins ................................... SUCCESS [  0.082 s]
[INFO] undertow-2.x-plugin ................................ SUCCESS [  3.375 s]
[INFO] rabbitmq-5.x-plugin ................................ SUCCESS [  2.261 s]
[INFO] apm-dubbo-conflict-path ............................ SUCCESS [  0.226 s]
[INFO] canal-1.x-plugin ................................... SUCCESS [  1.952 s]
[INFO] dubbo-2.7.x-plugin ................................. SUCCESS [  2.356 s]
[INFO] apm-dubbo-2.7.x-conflict-path ...................... SUCCESS [  0.207 s]
[INFO] vertx-plugins ...................................... SUCCESS [  0.073 s]
[INFO] vertx-core-3.x-plugin .............................. SUCCESS [  0.378 s]
[INFO] resteasy-plugin .................................... SUCCESS [  0.099 s]
[INFO] apm-resteasy-server-3.x-plugin ..................... SUCCESS [  2.179 s]
[INFO] solrj-7.x-plugin ................................... SUCCESS [  2.877 s]
[INFO] apm-cassandra-3.x-plugin ........................... SUCCESS [  3.210 s]
[INFO] apm-application-toolkit ............................ SUCCESS [  0.088 s]
[INFO] apm-toolkit-log4j-1.x .............................. SUCCESS [  0.123 s]
[INFO] apm-toolkit-activation ............................. SUCCESS [  0.102 s]
[INFO] apm-toolkit-log4j-1.x-activation ................... SUCCESS [  0.203 s]
[INFO] apm-toolkit-log4j-2.x-activation ................... SUCCESS [  0.167 s]
[INFO] apm-toolkit-logback-1.x-activation ................. SUCCESS [  0.283 s]
[INFO] apm-toolkit-opentracing ............................ SUCCESS [  0.209 s]
[INFO] apm-toolkit-opentracing-activation ................. SUCCESS [  3.235 s]
[INFO] apm-toolkit-trace .................................. SUCCESS [  0.116 s]
[INFO] apm-toolkit-trace-activation ....................... SUCCESS [  3.661 s]
[INFO] bootstrap-plugins .................................. SUCCESS [  0.084 s]
[INFO] apm-jdk-http-plugin ................................ SUCCESS [  0.155 s]
[INFO] optional-plugins ................................... SUCCESS [  0.079 s]
[INFO] optional-spring-plugins ............................ SUCCESS [  0.066 s]
[INFO] apm-spring-annotation-plugin ....................... SUCCESS [  0.161 s]
[INFO] optional-spring-cloud .............................. SUCCESS [  0.093 s]
[INFO] gateway-2.x-plugin ................................. SUCCESS [  0.667 s]
[INFO] apm-trace-ignore-plugin ............................ SUCCESS [  1.179 s]
[INFO] gson-2.8.x-plugin .................................. SUCCESS [  1.802 s]
[INFO] lettuce-5.x-plugin ................................. SUCCESS [  2.200 s]
[INFO] zookeeper-3.4.x-plugin ............................. SUCCESS [  2.093 s]
[INFO] customize-enhance-plugin ........................... SUCCESS [  1.327 s]
[INFO] play-2.x-plugin .................................... SUCCESS [  4.426 s]
[INFO] apm-toolkit-log4j-2.x .............................. SUCCESS [  0.157 s]
[INFO] apm-toolkit-logback-1.x ............................ SUCCESS [  0.172 s]
[INFO] oap-server ......................................... SUCCESS [  0.178 s]
[INFO] server-library ..................................... SUCCESS [  0.089 s]
[INFO] library-util ....................................... SUCCESS [  1.335 s]
[INFO] library-module ..................................... SUCCESS [  1.694 s]
[INFO] server-telemetry ................................... SUCCESS [  0.078 s]
[INFO] telemetry-api ...................................... SUCCESS [  0.351 s]
[INFO] server-configuration ............................... SUCCESS [  0.091 s]
[INFO] configuration-api .................................. SUCCESS [  1.531 s]
[INFO] library-client ..................................... SUCCESS [01:59 min]
[INFO] library-server ..................................... SUCCESS [  0.520 s]
[INFO] server-testing ..................................... SUCCESS [  1.056 s]
[INFO] server-core ........................................ SUCCESS [  9.478 s]
[INFO] library-buffer ..................................... SUCCESS [  1.463 s]
[INFO] server-receiver-plugin ............................. SUCCESS [  0.459 s]
[INFO] skywalking-sharing-server-plugin ................... SUCCESS [  0.610 s]
[INFO] skywalking-trace-receiver-plugin ................... SUCCESS [  2.319 s]
[INFO] server-storage-plugin .............................. SUCCESS [  0.069 s]
[INFO] storage-elasticsearch-plugin ....................... SUCCESS [  2.157 s]
[INFO] storage-zipkin-plugin .............................. SUCCESS [  0.922 s]
[INFO] skywalking-register-receiver-plugin ................ SUCCESS [  2.181 s]
[INFO] zipkin-receiver-plugin ............................. SUCCESS [  2.785 s]
[INFO] skywalking-mesh-receiver-plugin .................... SUCCESS [  0.802 s]
[INFO] receiver-proto ..................................... SUCCESS [  6.568 s]
[INFO] skywalking-istio-telemetry-receiver-plugin ......... SUCCESS [  1.523 s]
[INFO] skywalking-jvm-receiver-plugin ..................... SUCCESS [  1.502 s]
[INFO] envoy-metrics-receiver-plugin ...................... SUCCESS [  3.029 s]
[INFO] skywalking-clr-receiver-plugin ..................... SUCCESS [  0.575 s]
[INFO] storage-jaeger-plugin .............................. SUCCESS [  0.736 s]
[INFO] jaeger-receiver-plugin ............................. SUCCESS [  0.691 s]
[INFO] skywalking-so11y-receiver-plugin ................... SUCCESS [  0.701 s]
[INFO] server-cluster-plugin .............................. SUCCESS [  0.075 s]
[INFO] cluster-zookeeper-plugin ........................... SUCCESS [ 11.048 s]
[INFO] cluster-standalone-plugin .......................... SUCCESS [  1.491 s]
[INFO] cluster-kubernetes-plugin .......................... SUCCESS [  2.557 s]
[INFO] cluster-consul-plugin .............................. SUCCESS [ 22.273 s]
[INFO] cluster-nacos-plugin ............................... SUCCESS [01:31 min]
[INFO] cluster-etcd-plugin ................................ SUCCESS [01:21 min]
[INFO] storage-jdbc-hikaricp-plugin ....................... SUCCESS [  2.123 s]
[INFO] oal-grammar ........................................ SUCCESS [  0.717 s]
[INFO] oal-rt ............................................. SUCCESS [  3.912 s]
[INFO] server-query-plugin ................................ SUCCESS [  0.097 s]
[INFO] query-graphql-plugin ............................... SUCCESS [  1.308 s]
[INFO] server-alarm-plugin ................................ SUCCESS [  4.968 s]
[INFO] telemetry-prometheus ............................... SUCCESS [  0.349 s]
[INFO] telemetry-so11y .................................... SUCCESS [  0.293 s]
[INFO] exporter ........................................... SUCCESS [  3.819 s]
[INFO] grpc-configuration-sync ............................ SUCCESS [  0.894 s]
[INFO] configuration-apollo ............................... SUCCESS [  1.623 s]
[INFO] configuration-nacos ................................ SUCCESS [ 24.884 s]
[INFO] configuration-zookeeper ............................ SUCCESS [  7.861 s]
[INFO] configuration-etcd ................................. SUCCESS [ 15.772 s]
[INFO] server-starter ..................................... SUCCESS [  3.716 s]
[INFO] apm-webapp ......................................... SUCCESS [ 29.528 s]
[INFO] apache-skywalking-apm .............................. SUCCESS [ 18.822 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  11:35 min
[INFO] Finished at: 2019-08-30T14:27:01+08:00
[INFO] ------------------------------------------------------------------------

@stone-wlg CI improvement is in #3308. Don't worry about that. Have you figured out why integration test breaks?

improvement

Ok, i have add healthcheck for cassandra-server in docker-compose.yml file. let's try.

improvement

Ok, i have add healthcheck for cassandra-server in docker-compose.yml file. let's try.

Ok

improvement

Ok, i have add healthcheck for cassandra-server in docker-compose.yml file. let's try.

Ok

@wu-sheng
is there enough resources to do integration test? the error in the test report seems that app could not connect cassandra. the integration test is always passed on my local env. or shall we test one by one version, first test 3.7.2, then 3.7.0 ....

That VM is very powerful. 16U32G on GCP. How many resources do you need? We expect as little as possible, because every night, we run 700+ tests, which have already taken 6 hours.

That VM is very powerful. 16U32G on GCP. How many resources do you need? We expect as little as possible, because every night, we run 700+ tests, which have already taken 6 hours.

My PC is 6U24G.
by the way, the command and screenshot as below:

sudo ./deploy-test.sh --scenario cassandra-3.x-scenario https://github.com/stone-wlg/skywalking.git features/cassandra-3.x-plugin

That VM is very powerful. 16U32G on GCP. How many resources do you need? We expect as little as possible, because every night, we run 700+ tests, which have already taken 6 hours.

i have an idea, could i remove other supported version, we only support 3.7.2 version ?

I think the key is running all cases once because that is the case they facing after merged.

I think the key is running all cases once because that is the case they facing after merged.

is there possible to run all cases one by one right now?

I think we have already rumning in this way. Right? @zhaoyuguang

+1 for @kezhenxu94 's review.

I'm not familiar with Cassandra at all, but according to the API, you just intercept the method SessionManager#executeAsync and nothing more, which returns a ResultSetFuture, if I didn't misunderstand this, the request to Cassandra server did not complete here until the method ResultSetFuture#getUninterruptibly(or ResultSetFuture#cancel) is called, and, if the getUninterruptibly  throws (say TimeoutException), the request actually failed but your plugin would record it as success, doesn't it? And if the client calls ResultSetFuture#cancel, the status should be recorded (perhaps as TAG, WDYT @wu-sheng ). @stone-wlg Please correct me if I'm wrong

actually there are several functions for executing sql, as below:
ResultSet execute(String var1);
ResultSet execute(String var1, Object... var2);
ResultSet execute(String var1, Map<String, Object> var2);
ResultSet execute(Statement var1);
ResultSetFuture executeAsync(String var1);
ResultSetFuture executeAsync(String var1, Object... var2);
ResultSetFuture executeAsync(String var1, Map<String, Object> var2);
ResultSetFuture executeAsync(Statement var1);
but finally other execute will call executeAsync(Statement var1), so i only intercept this method.

I'm not familiar with Cassandra at all, but according to the API, you just intercept the method SessionManager#executeAsync and nothing more, which returns a ResultSetFuture, if I didn't misunderstand this, the request to Cassandra server did not complete here until the method ResultSetFuture#getUninterruptibly(or ResultSetFuture#cancel) is called, and, if the getUninterruptibly  throws (say TimeoutException), the request actually failed but your plugin would record it as success, doesn't it? And if the client calls ResultSetFuture#cancel, the status should be recorded (perhaps as TAG, WDYT @wu-sheng ). @stone-wlg Please correct me if I'm wrong

i don't get your point, do you mean when exception occurred or user cancel what should happened? if yes, what is the right way to do in skywalking ?

but finally other execute will call executeAsync(Statement var1), so i only intercept this method.

@stone-wlg It is absolutely OK to intercept just the executeAsync, what I meant is that you should take care of the final state of the request, while executeAsync doesn't indicate that the request is completed, in case that it is canceled or timed out, and this should be recorded in the span somehow

but finally other execute will call executeAsync(Statement var1), so i only intercept this method.

@stone-wlg It is absolutely OK to intercept just the executeAsync, what I meant is that you should take care of the final state of the request, while executeAsync doesn't indicate that the request is completed, in case that it is canceled or timed out, and this should be recorded in the span somehow

ok, what is the approach usually in skywalking? is there sample that i can follow?

but finally other execute will call executeAsync(Statement var1), so i only intercept this method.

@stone-wlg It is absolutely OK to intercept just the executeAsync, what I meant is that you should take care of the final state of the request, while executeAsync doesn't indicate that the request is completed, in case that it is canceled or timed out, and this should be recorded in the span somehow

ok, what is the approach usually in skywalking? is there sample that i can follow?
sorry, i am not familiar with skywalking. if i use tag to record it. ex. TAG.DB_EXECUTE_RESULT = true. then what is happened in skywalking ? or there is a placement that will show these tags ?

@stone-wlg For async, in order to make sure exit span duration accuration. Besides intercepting callback and creating local span there, you should take a look at AsyncSpan APIs, finish the exit span in async mode.

@stone-wlg For async, in order to make sure exit span duration accuration. Besides intercepting callback and creating local span there, you should take a look at AsyncSpan APIs, finish the exit span in async mode.

@wu-sheng
is there limitation with enhance class, ex: public or private class, abstract class, interface?

You can't enhance abstract method or interface method. You could enhance subclass or implementor of interface.

You can't enhance abstract method or interface method. You could enhance subclass or implementor of interface.

private or public class ?

You can't enhance abstract method or interface method. You could enhance subclass or implementor of interface.

ok, thanks

No limit about private or public

move to #3410

@stone-wlg if it's not absolutely necessary, just push to the original branch and you don't need to open pull requests over and over again, learn how to merge from upstream repository and update a pull request

@stone-wlg if it's not absolutely necessary, just push to the original branch and you don't need to open pull requests over and over again, learn how to merge from upstream repository and update a pull request

ok

ok, but i do not want to leave unnecessary log in git log.

Check git rebase, you could do that then force push.

ok, but i do not want to leave unnecessary log in git log.

Check git rebase, you could do that then force push.

thanks


<EMAIL_ADDRESS>Shim creation fail

Cannot shim 'Lunacy.exe': File does not exist

For some reason, innounp is extracting two copies of each file, but some of them seem to have different file sizes:

I made a fix for this, then realised that Excavator had reverted to an older version (3.1) - so I've fixed the checkver and put it back to the latest version (8.7.2). Thanks for reporting!


Webpack Externals function not saved by save-webpack-config

Hello, HUGE thanks from me as well for this massive SPFX serve booster!
I believe I struck a very very specific edge case that you can choose to not address. The gist is, I'm using a Webpack Externals Function to transform "office-ui-fabric-react/*" imports and revert them to their parent module so that instead of including tree-shaken component scripts I can externalize them using a hosted bundle of ui-fabric scripts.
My function looks like this:
function (context, request, callback) {
  // Bundle tree-shakeable imports from office-ui-fabric-react/* into its root module
  if (typeof request === "string" && /^office-ui-fabric-react/i.test(request)) {
    return callback(null, "office-ui-fabric-react");
  }

  callback();
}

However, when I run npm run serve I get an error that the webpack externals configuration is incorrect because my function has been NULL-ed during save-webpack-config.
A potential solution is to upgrade spfx-fast-serve-helpers/src/tasks/saveConfigTask.ts to serialize functions with JSON.stringify. The best approach I've found after a few minutes of research looks to be this answer under How can I serialize a function in JavaScript?. These modifications to the Function prototype will also need to apply to spfx-fast-serve-helpers/src/webpack/helpers.ts>getJSONFile where instead of require-ing the serialized webpack config in the temp folder it's read as a file using fs and then JSON.parse-d.
For now, my workaround is to leverage the fast-serve/webpack.extend.js file and filter the NULL which gets serialized instead of my function from the initialWebpackConfig and re-add it, for example:
const webpackConfig = {
  externals: [
    function (context, request, callback) {
      // Bundle tree-shakeable imports from office-ui-fabric-react/* into its root module
      if (typeof request === "string" && /^office-ui-fabric-react/i.test(request)) {
        return callback(null, "office-ui-fabric-react");
      }

      callback();
    }
  ]
}

const transformConfig = function (initialWebpackConfig) {
  const { externals } = initialWebpackConfig;
  if (Array.isArray(externals)) {
    // filter unserialized function entry, JSON.stringify([function () {}]) => "[null]"
    initialWebpackConfig.externals = externals.filter(Boolean);
  }

  return initialWebpackConfig;
}

module.exports = {
  webpackConfig,
  transformConfig
}

Thanks for taking your time to review this minor grievance! I'm only raising this because I hate having copy/pasted code, even if it's related to build-time operations. As an added benefit by fixing this any other webpack configuration that can be modified using functions will be passed over to the fast-serve runtime.

Hi,
I don't like the idea of serializing function, because it's not a 100% reliable solution. Also, given that webpack.extend.js was created to cover special cases for webpack modifications, it's perfectly fine to use it here. I'll close it a completed since it'a more an edge case and can be addressed with webpack.extend.js.


feat: introduce basic ffi implementation to write simple pact contract

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@e9f9341). Click here to learn what that means.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main       #1   +/-   ##
=======================================
  Coverage        ?   82.97%
=======================================
  Files           ?        7
  Lines           ?      141
  Branches        ?        0
=======================================
  Hits            ?      117
  Misses          ?       24
  Partials        ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e9f9341...ab3028e. Read the comment docs.


Implement DataChannels

This is the big covering ticket, will include links to RFC/design docs as needed.
As far as pion-WebRTC is concerned we need to start with a SCTP library to parse/create packets

And we landed! https://github.com/pions/webrtc/pull/31 was merged
Things aren't perfect yet, but it works so merged and we can keep iterating on it.


extend probe initial delays

Coverage remained the same at 50.538% when pulling 595e687937e5ec26183142c3032c12d3a3b7dbea on rastislavszabo:master into 25c45536ff86ada21def023ff59d93adee832941 on contiv:master.


Any .snmprec files repository?

Hi,
I'm wondering if there is any snmp rec repository available with different type of device rec ?
Best regards

Not that I know of, unfortunately. I think the problem here is that SNMP data could potentially leak undesirable information. Keep in mind that sensitive things may also occur in OIDs, not just in values. So it's not just the matter of values obfuscation.
On the other hand, if one could factory reset a device and snmpwalk it, that might be reasonable harmless, I hope.
At any rate, I'd be happy to host SNMP snapshots perhaps at snmplabs.com, should they be contributed.

On the other hand, one could use the mib2dev tool to build agent model out of MIBs. That's not the same as having a snapshot from a real agent, but still might serve the purpose...

https://github.com/librenms/librenms/tree/master/tests/snmpsim


chore: update commit message in auto-upgrade

:tada: This PR is included in version 1.4.17 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


fix for broken CTRL-R in zsh

fixes #241 and the followup of #203
When looking for the bug I found that whenever CTRL-R is broken, it never entered the if selected=...
This change fixes that and covers all cases I could think of.

Thanks, but can you explain how this slight adjustment of the code fixes the issue? I don't see anything wrong with the current code (and I can't reproduce the problem in the first place). Is this a known issue of zsh? By the way, you have to change [ -n $selected ] to [ -n "$selected" ] or it will always evaluate to true even when it's empty.
/cc @Tranquility

Which version of zsh are you running? Do you have any zsh plugin that might affect the behavior of fzf?

I have a non-trivial zsh setup and didn't have time to test this bug in more detail.
Sourcing the original keybinding didn't fix the bug. Sourcing the modified one did.
The patch corresponds to the change that came out of 5 mins playing around with the zsh-function bound to CTRL-R. I overlooked the missing ", so if your right, the fix is equivalent to removing the outer if. The if for num should be enough anyway.
I'll let you know if the bug shows up again when I have more time to spend investigating it.

The outer if allows you to early-exit the code when fzf is terminated with ctrl-c or esc, but as you mentioned the end result is the same with or without it. But anyway I don't think the code is flawed in itself, and if it somehow causes the problem, then it is likely an issue of zsh (or tmux) and that's why I asked you about the zsh/tmux version you have. I'm not against having workarounds in the code for external/environmental issues, but in this case we seem to have little knowledge of the exact situation.

I had the same problem and found it was due to git_prompt_short_sha() in my prompt, defined in oh-my-zsh/lib/git.zsh. It somehow breaks fzf whenever the pwd is not in a git repo. Simply defining a new function
function git_prompt_short_sha() {
  echo $(command git rev-parse --short HEAD 2> /dev/null)
}

fixed it for me.
Further, using set -A selected ... in place of selected=( ... ) also fixed it independently.

@lyeoh thanks for the suggestion
I also have git_prompt_short_sha defined via oh-my-zsh, so this seems plausible. My issue is, that I'm not able to get the issue reproducibly, inside or outside of repositories. Nevertheless it appeared less often recently and if your suggestion fixes it for you it should be included in my opinion.

This fix seems to only select the second change in the list? No matter what item I select in the list I always get the same entry unless I change
    num=$selected[1]

to
    num=$selected[5]

or some other number.
I have a similar situation where I have a function that adds some git information to my prompt and when I remove it the tmux script works however I tried putting command in front as suggested by @lyeoh and it still seems to fail.


Count and about page

Added a call to get the pv count for a given query in api.js. Display the count on the search page and on the services page.
Added calls for info for different services in api.js. Display the results on new Services page.
Added components for the Services page (Services.js) and individual services themselves (Service.js). Can change the name of one of these if they are too similar

looks great!


Fixing default top value for queries to pinot

Was seeing exception:
com.linkedin.thirdeye.client.cache.ResultSetGroupCacheLoader: Error when running pql:SELECT COUNT(*) FROM ollection WHERE  Date >= 408788 AND Date < 408955 GROUP BY dimension TOP<PHONE_NUMBER>
! com.linkedin.pinot.client.PinotClientException: Query had processing exceptions:
! [{"errorCode":700,"message":"java.lang.RuntimeException: Value for 'TOP'<PHONE_NUMBER> exceeded maximum allowed value of 300000

Lgtm

LGTM

lgtm


Integration tests fail when CPATH is not set

/Users/ubaid/Desktop/OpenSource/lc/integration_tests/array_09.cpp:2:10: fatal error: 'xtensor/xtensor.hpp' file not found
#include <xtensor/xtensor.hpp>
         ^~~~~~~~~~~~~~~~~~~~~
LCOMPILERS_ASSERT failed: /Users/ubaid/Desktop/OpenSource/lc/src/libasr/../libasr/asr.h
function down_cast(), line number 40 at
f != nullptr
/Users/ubaid/Desktop/OpenSource/lc/integration_tests/array_05.cpp:2:10: fatal error: 'xtensor/xfixed.hpp' file not found
#include <xtensor/xfixed.hpp>
         ^~~~~~~~~~~~~~~~~~~~
1 error generated.
Error while processing /Users/ubaid/Desktop/OpenSource/lc/integration_tests/array_05.cpp.
make[2]: *** [CMakeFiles/array_05.cpp.dir/build.make:74: array_05.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:312: CMakeFiles/array_05.cpp.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

After setting the CPATH as follows and running tests
$ CPATH=$CONDA_PREFIX/include:$CPATH ./integration_tests/run_tests.py -b llvm

The above issue works (it finds xtensor).

I can reproduce this issue, it happens for me also.


Update msbuild version info to use property expected by repotoolset

msbuild was updated to use a newer version of repotoolset.  It now expects VersionPrefix instead of VersionBase.

Built source-build and smoke-tests passed.  Building the tarball now.


Clear typeahead input in timeout

Because typeahead will set the value of the input, sometimes after we've cleared it, which leaves the input with text we don't want.
This solution just wraps the clearing of the input in a setTimeout so it is performed later.

I'm closing this PR, as it appears this repo is no longer maintained, and I want to keep my PR list clean.


'Cannot start .NET Core language services because dotnet was not found.', followed by 'Not in a F# project (still loading....)'

Describe the bug
When I open an F# project in WSL 2, the Ionide plugin is not loaded. Instead, I see the following error:

Cannot start .NET Core language services because dotnet was not found.
Consider:

setting the FSharp.dotnetRoot settings key to a directory with a dotnet binary,
including dotnet in your PATH, or
installing .NET Core into one of the default locations.

When I set theFSharp.dotnetRoot settings key and restart VS Code, I see a second error, Not in a F# project (still loading....) in the status bar as soon as I open a *.fs file.
Steps to reproduce

In WSL, create a new F# project in a folder of your choice: dotnet new classlib -lang F#
Open VS Code in that folder: code .
When the error Cannot start .NET Core language services because dotnet was not found. appears, set the FSharp.dotnetRoot settings key and restart VS Code.
Now, the second error appears.

Expected behaviour
The Ionide plugin should load as soon as VS Code opens in an F# project.
Machine info

I am using the most recent versions of VS Code and the Ionide plugin.
WSL 2 on Windows 11 with the following dotnet installation:.NET SDK (reflecting any global.json):
 Version:   6.0.300
 Commit:    8473146e7d

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  20.04
 OS Platform: Linux
 RID:         ubuntu.20.04-x64
 Base Path:   /usr/share/dotnet/sdk/6.0.300/

Host (useful for support):
  Version: 6.0.5
  Commit:  70ae3df4a6

.NET SDKs installed:
  6.0.300 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

I had the same problem on normal Ubuntu Linux.
After manually setting FSharp.dotnetRoot to /usr/bin it started working for me.
Previously FSharp.dotnetRoot was empty.

Having this problem with
gina@SurfaceBaby:~$ dotnet --info
.NET SDK (reflecting any global.json):
 Version:   6.0.301
 Commit:    43f9b18481

Runtime Environment:
 OS Name:     debian
 OS Version:  11
 OS Platform: Linux
 RID:         debian.11-x64
 Base Path:   /usr/share/dotnet/sdk/6.0.301/

Host (useful for support):
  Version: 6.0.6
  Commit:  7cca709db2

.NET SDKs installed:
  6.0.301 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs:
  https://aka.ms/dotnet-download

and latest VS Code with an FSharp project. I'm probably doing something dumb, but figured I'd ask here.

No, you're right! I somehow lost track of doing and releasing this in an ionide update, but it's in the upcoming MEGA UPDATE that includes the update to vscode-languageclient@8.


Obx processing changes in nested tree-structure

Is your feature request related to a problem? Please describe.
I would like Obx to listen to a deeply nested tree-structure and trigger a complete rebuild if a tree-leaf (or any intermediary node) is added/deleted/modified.
I am using a model similar to the one depicted below:
A Book with Chapters, and Chapters having Scenes.
In reality the tree-structure is way deeper, this is a simplified example.
The idea is to create two DropdownButton: one for Chapters, one for Scenes.
When a Scene (or Chapter) is added or deleted, the respective DropdownButton has to reflect these changes.
Both DropdownButton are visible all the time on the screen.
Controller
MyController
  final book = Book.obs

Model
Book
-- Chapter 1
--- Scene 1
--- Scene 2

At the moment Obx gets triggered by changes of book attribute, only.
Describe the solution you'd like
I would like Obx to get triggered when (for example) a new Scene to Book is added.
Describe alternatives you've considered
I want to avoid referencing each level of the tree (Book, Chapters, Scenes) in specific attributes and make Obx listening to these individually. In reality the tree-structure is way deeper, this is a simplified example.

@xErik you could try the following
Keep your single Book -level observable (probably in a BookController or something similar) and then ensure that you call book.refresh() anytime a attributes of the book are mutated
Ex:
   void onAddNewScene({Scene scene, int chapterIndex}) {
      book.value.chapters[chapterIndex].add(scene);
      book.refresh(); // this will trigger a redraw in the `Obx(() => )` widget
   }

Will do, thanks!


Still no Answer: Add more options to Utility API

I made some changes to Bootstrap v5 Utility API.
What I have added?
remove responsive option and merge it inside option called variants.
what variants option do ?
it generate pseudo classes such as hover, focus... in single utility if need it.
The problem that I face is :
when I try to make the single Utility responsive it make responsive as pseudo classe. and that's its wrong I want it to allow media breakpoints for the utility only nothing more or less.
Here is an example of wrong compiling CSS :
.float-left {
    float: left;
}

// this it shouldn't be here... the responsive key only for enabling the
// breakpoints for single utility if need it...

.responsive\:float-left:responsive {
    float: left;
}

My Full code :
$enable-important-utilities:   true  !default;
$enable-pseudo-class-variants:  true  !default;
$breakpoint-inset-namespace:   "\\:" !default;
$variant-namespace:            "\\:" !default;

@mixin _assert-ascending($map, $map-name) {
  $prev-key: null;
  $prev-num: null;
  @each $key, $num in $map {
    @if $prev-num == null or unit($num) == "%" or unit($prev-num) == "%" {
    } @else if not comparable($prev-num, $num) {
      @warn "Potentially invalid value for #{$map-name}: This map must be in ascending order, but key '#{$key}' has value #{$num} whose unit makes it incomparable to #{$prev-num}, the value of the previous key '#{$prev-key}' !";
    } @else if $prev-num >= $num {
      @warn "Invalid value for #{$map-name}: This map must be in ascending order, but key '#{$key}' has value #{$num} which isn't greater than #{$prev-num}, the value of the previous key '#{$prev-key}' !";
    }
    $prev-key: $key;
    $prev-num: $num;
  }
}

@mixin _assert-starts-at-zero($map, $map-name: "$grid-breakpoints") {
  @if length($map) > 0 {
    $values: map-values($map);
    $first-value: nth($values, 1);
    @if $first-value != 0 {
      @warn "First breakpoint in #{$map-name} must start at 0, but starts at #{$first-value}.";
    }
  }
}

@function breakpoint-min($name, $breakpoints: $grid-breakpoints) {
  $min: map-get($breakpoints, $name);
  @return if($min != 0, $min, null);
}

@function breakpoint-infix($name, $breakpoints: $grid-breakpoints) {
    @return if(breakpoint-min($name, $breakpoints) == null, "", "#{$name}#{$breakpoint-inset-namespace}");
}

@mixin media-breakpoint-up($name, $breakpoints: $grid-breakpoints) {
  $min: breakpoint-min($name, $breakpoints);
  @if $min {
    @media (min-width: $min) {
      @content;
    }
  } @else {
    @content;
  }
}

$grid-breakpoints: (
  xs: 0,
  sm: 576px,
  md: 768px,
  lg: 992px,
  xl: 1200px,
  xxl: 1400px
) !default;

@include _assert-ascending($grid-breakpoints, "$grid-breakpoints");
@include _assert-starts-at-zero($grid-breakpoints, "$grid-breakpoints");

$utilities: () !default;
$utilities: map-merge(
  (
    "float": (
        variants: (responsive, hover),
        property: float,
        values: left
    ),
  ),
  $utilities
);

@mixin generate-utility($utility, $infix) {
    $values: map-get($utility, values);

    @if type-of($values) == "string" or type-of(nth($values, 1)) != "list" {
        $values: zip($values, $values);
    }

    @each $key, $value in $values {

        $variants: if(
            map-has-key($utility, variants),
            map-get($utility, variants),
            null
        );

        @if type-of($variants) == "string" {
            $variants: append((), $variants);
        }

        $properties: map-get($utility, property);

        @if type-of($properties) == "string" {
            $properties: append((), $properties);
        }

        $property-class: if(
            map-has-key($utility, class),
            map-get($utility, class),
            nth($properties, 1)
        );

        $property-class: if($property-class == null, "", $property-class);

        $infix: if(
            $property-class == "" and str-slice($infix, 1, 1) == "-",
            str-slice($infix, 2),
            $infix
        );

        $property-class-modifier: if(
            $key,
            if($property-class == "" and $infix == "", "", "-") + $key,
            ""
        );

        @if $value != null {
            .#{$infix + $property-class + $property-class-modifier} {
                @each $property in $properties {
                    #{$property}: $value if($enable-important-utilities, !important, null);
                }
            }
            @if $enable-pseudo-class-variants {
                @if $variants {
                    @each $pseudo in $variants {
                        .#{$infix + $pseudo + $variant-namespace + $property-class + $property-class-modifier}:#{$pseudo} {
                            @each $property in $properties {
                                #{$property}: $value if($enable-important-utilities, !important, null);
                            }
                        }
                    }
                }
            }
        }
    }
}

@each $breakpoint in map-keys($grid-breakpoints) {

  @include media-breakpoint-up($breakpoint) {
    $infix: breakpoint-infix($breakpoint, $grid-breakpoints);

    @each $key, $utility in $utilities {
      @if type-of($utility) == "map" and (map-get($utility, variants) or $infix == "") {
        @include generate-utility($utility, $infix);
      }
    }
  }
}

Try to use it in sassmeister fisrt to know what I want to achieve.

Hi
Why is this option used?
$utilities: map-merge(
  (
    "float": (
        variants: (responsive, hover),
        property: float,
        values: left
    ),
  ),
  $utilities
);

The API talks about such a solution
$utilities: map-merge(
  (
    "float": (
        responsive: true,
        state: hover,
        property: float,
        values: left
    ),
  ),
  $utilities
);

Responsive isn't the same as a pseudo-class, so they should remain separate.


Update dream.py

In the case of this "point", the Warp Terminal cannot be clicked directly to trigger the browser to open, and Chrome is a blank page. It should open properly once you remove it

Ouch, I'm getting merge conflicts because the PR was originally based against main rather than development. I'm going to go ahead and directly fix the punctuation mark from the command line, and I'll add you to CONTRIBUTORS if you're not already there.


Asura scan null pointer

Source information
Asura Scan
Source language
English
Steps to reproduce
1.) Open app
2.) Open asura scan
3.) Open comic
4.) Refresh
5.)null pointer
Expected behavior
1.) Open app
2.) Open asura scan
3.) Open comic
4.) Refresh
5.)new chapter come
Actual behavior
It is showing error nul pointer
Mihon/Tachiyomi version
Tachiyomi stable0.15.3
Android version
Android 11
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] I have updated the app to version 0.15.3.
[X] I have updated all installed extensions.
[X] I have tried the troubleshooting guide.
[X] If this is an issue with the app itself, I should be opening an issue in the app repository.
[X] I will fill out all of the requested information in this form.

Duplicate of #5198
Search the existing issues before opening a new one, and acknowledging that you did when you clearly didn't


Updated README.md

Following the Composer best practices and simplified the installation step(s).

I have processed both feedback points.

Hi @dszeto, i'm running over my (outstanding) pull request and so I came across this one. Are there open points why this pull request can not be merged? Can I help with something? Do you expect changes and/or feedback processing from me?

Sorry for the delay. This looks good to me now. Thanks @cafferata !


Hanging scan

A few times, the scan just "hang". For some file in the scan, all probes are marked as done, and others are marked as pending. But the scan will never evolve.
This as been raised by a stress test, such as #130 .

Latest version ? cause a fix for similar case was shipped in 1.5.1 c3e2d3eb

Thanks for pointing this commit out. According to its commit log, its seems that it could fix #130, but not this issue.
Also, this might be related to our deployment. If you can run a test with:

4 parallels scan attempt (through the REST API)
Each scan contains 3 random files of 3MB (you might want to pre-gen them)
At least 2 probes
Each second during 5 minutes

If this test ends correctly on your test infrastructure, then it may not be related to IRMA (and I'm sorry for your time loss).

First try with jmeter, i need a better frontend as i just flooded uwsgi. Still trying to reproduce. To be continued.

It may be related to #132


Cosmos Reademe issue

Section Link:

Suggestion:
For 1: Text change to  Java Development Kit (JDK) with version 8 or above and add link https://docs.microsoft.com/java/azure/jdk/
For 2: Add link http://www.slf4j.org/
For 3: Add link https://maven.apache.org/
@jongio , for notification

@Melanie-Ma - would you be willing to update this doc ?


switch from micro/cli/v2 to urfave/cli/v2

Description
switch from micro/cli/v2 to urfave/cli/v2 because go-micro also has done it in https://github.com/asim/go-micro/pull/2224
Related Issue

like https://github.com/asim/go-micro/pull/2224

Motivation and Context

How Has This Been Tested?

not at all

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Technical debt
[ ] Tests only (no source changes)

Checklist:

[ ] Code changes
[ ] Unit tests added
[ ] Acceptance tests added
[ ] Documentation ticket raised:

@refs I saw that go-micro did this step... Do you think we also can do this?


Bridge-Removal: Tests file cleanup with necessary profile and resources

Background: There are too many test files change, trying to separate there changes as this separate PR.

Remove the quarkus-common test scope dependency, provided with necessary profile and resources.
Add NoGlobalTestResourceProfile and IntegrationUtils
remove CustomValueSerializers, as it is not used anywhere. Remove CustomValueSerializersTest at the same time.

Checklist

[x] Changes manually tested
[x] Automated Tests added/updated
[x] Documentation added/updated
[x] CLA Signed: DataStax CLA

Good idea! Will review.


Sign docker images using cosign

cosign uses the GitHub action ID token to retrieve an ephemeral code signing certificate from Fulcio, and store the signature in the Rekor transparency log.
Once an image has been successfully signed, you should be able to verify the signature with:
cosign verify ghcr.io/astral-sh/uv:latest --certificate-identity-regexp='.*' --certificate-oidc-issuer-regexp='.*'

Closes #8670

A word of warning: I am not 100% certain I fully understand everything that the build-docker.yml pipeline does. I'd appreciate some feedback on how the digest of the final docker-republish image manifest is shared with the signing step, for example.

cc @samypr100 (as always, no obligation)

@mjpieters

Note, I'm running a release job on my fork with these changes (9b162d) as I'd like to see it working E2E
https://github.com/samypr100/uv/actions/runs/11593512252

@mjpieters signed images are here https://github.com/samypr100/uv/pkgs/container/uv in case you'd like to verify the signatures are working as expected

@mjpieters

Note, I'm running a release job on my fork with these changes (9b162d) as I'd like to see it working E2E
https://github.com/samypr100/uv/actions/runs/11593512252

@mjpieters signed images are here https://github.com/samypr100/uv/pkgs/container/uv in case you'd like to verify the signatures are working as expected

Excellent! I was working my way through understanding cargo dist in my own fork to try this out but hadn't gotten as far as actually seeing the workflow complete.
The first thing I notice is that the the .sig files are now obscuring the actual docker images on that page, which is a bummer.
The signatures are valid however!
% cosign verify ghcr.io/samypr100/uv:latest --certificate-identity-regexp='.*' --certificate-oidc-issuer-regexp='.*' --output-file /tmp/samypr100-uv-verify.json

Verification for ghcr.io/samypr100/uv:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates

% jq -r '.[].optional.Subject' /tmp/samypr100-uv-verify.json
https://github.com/samypr100/uv/.github/workflows/build-docker.yml@refs/heads/docker-test
https://github.com/samypr100/uv/.github/workflows/build-docker.yml@refs/heads/docker-test
https://github.com/samypr100/uv/.github/workflows/build-docker.yml@refs/heads/docker-test

I'll have to see what can be done about those signature files in the registry.

So cosign pushes the signature as a new 'container' to the registry, tagged with sha256-$DIGEST.sig, with the signature contained in the manifest (see https://hackmd.io/@maelvls/how-cosign-uses-registries-to-store-signatures for an overview). It's these tags that are shown at the top of the page by ghcr.io, because it only ever shows the latest tags that have been pushed. The GH packages API and UI (which include the ghcr container registry) is very very sparse and doesn't offer any control over this.
I've looked at a number of other projects that use cosign to sign their containers (of which there are a very large number), and all the ones I looked at so far show these tags at the top. E.g. the Dependabot update bundler (a Github project!) has signature tags at the top. You see this on other registries too, e.g. the boxyhq jackson container on the docker hub.
I see but a few options here:

Accept that the signature tags are going to be pushed last and end up on top.
Experiment with the Github packages API to delete and then restore specific tags, to see if that puts them back at the top again.
Tag the containers after signing. Build and push the images, store the digests, sign the digests, then handle the version and 'latest' tags afterwards.
Decline this PR and forgo the security of having signed containers (which would be a huge pity, security is important, especially in supply-chain tools like uv).

We're already pushing the latest tag separately for this reason, can we push the digest before that final image tag?

We're already pushing the latest tag separately for this reason, can we push the signature before that final image tag?

You push not only a tag, but a manifest as well. You'd have to separate the two steps; push the manifest without tags and then tag the digest of the manifest afterwards. It could be that you can retag the same digest; I'll try this out in my own fork registry.

We're already pushing the latest tag separately for this reason, can we push the signature before that final image tag?

You push not only a tag, but a manifest as well. You'd have to separate the two steps; push the manifest without tags and then tag the digest of the manifest afterwards. It could be that you can retag the same digest; I'll try this out in my own fork registry.

I think the main issue here is how GH ranks packages and gives no control over the ordering to orgs/users of what's displayed, so we have to rely on these ordering hacks related to publishing.

I think the main issue here is how GH ranks packages and gives no control over the ordering to orgs/users of what's displayed, so we have to rely on these ordering hacks related to publishing.

and from my experiments it is clear that it is order that the manifest were pushed to the registry that is used to list them. I tried deleting then restoring, and retagging (moving tags to another manifest then back again to the right manifest), and nothing shifts the signature from the top.
So the only thing that would work is to build and  not push, sign (and let cosign push the signature manifest), and only then push the actual image manifest.

I've tried to use the load: true option for docker/build-push-action to delay pushing, but that trips over docker/buildx#59. I don't quite see how to create a multi-arch container image and not push this to the registry, my docker fu is failing me at the moment, sorry, I don't fully understand what a multi-arch build does and how it could be executed in separate steps. I'll have to drop investigating this, I fear.

If I find time to pick this up again, there is an excellent overview of why you can't use the docker/build-push-action action to separate building and pushing at https://github.com/NASA-PDS/devops/issues/78#issuecomment-2430132011, which points to containerd as the way forward, but... that's not yet an official option.

I've tried to use the load: true option for docker/build-push-action to delay pushing, but that trips over docker/buildx#59. I don't quite see how to create a multi-arch container image and not push this to the registry, my docker fu is failing me at the moment, sorry, I don't fully understand what a multi-arch build does and how it could be executed in separate steps. I'll have to drop investigating this, I fear.

Agreed, I believe the tooling is limited for these types of scenarios.

To be clear: this PR is ready in every other way. If not being able to pin the current base release container to the top is an issue, then please close this as I don't think we can practically do that and provide signed containers.

I'm not sure how to balance the trade-offs with the regression, it seems pretty confusing to users to not see the actual image tag they'd use.

@mjpieters Is it possible to push the signatures elsewhere, e.g. different package? Maybe I missed if this was discussed.

@mjpieters Is it possible to push the signatures (.sig files) elsewhere, e.g. different package? Maybe I missed if this was discussed.
Given the GH limitations, that's probably the next step to explore?

Yes, it is technically possible to push them to a different registry, see the cosign documentation. However I don't recommend doing this without first verifying that you can still verify the signature. I fear that the cosign verify tool won't be able to find the signature data when you do this and I don't know if or how you can then verify container integrity and provenance.
I do not currently have the time to test this option; you'd have to pull my PR into a fork repository, add the COSIGN_REPOSITORY environment variable to the signing steps pointing to a registry the github runner can access (this may require that the step is given a personal access token with container write permissions, although it may be possible to grant write permissions across repos in the same organisation), and then trying to verify the resulting container with cosign verify ghcr.io/$FORK_OWNER/uv:latest --certificate-identity-regexp='.*' --certificate-oidc-issuer-regexp='.*'. Perhaps that succeeds anyway, which would be very welcome news indeed.

I also wonder, are there other projects that you know of that use ghcr.io + cosign? I'd like to see if this is a common issue and if there's other known workarounds that we may be missing when using cosign. I'll take a look as well.

I looked, but all those that I found put their signatures straight into the GitHub container registry by their containers (and don't try and control what's that top listed entry).


After Uploading Image Not Display

Laravel version :5.6
Package version :1.8

everything work fine in local , but after i uploaded my laravel project on host  , lfm not working properly. my images uploaded without any problem but its not display on explorer section and i found out , this problem happen,because add lfm prefix to image url.
`/*
|--------------------------------------------------------------------------
| Routing
|--------------------------------------------------------------------------
*/
// Include to pre-defined routes from package or not. Middlewares
'use_package_routes' => true,

// Middlewares which should be applied to all package routes.
// For laravel 5.1 and before, remove 'web' from the array.
'middlewares' => ['web','auth:admin'],

// The url to this package. Change it if necessary.
'prefix' => 'admin/laravel-filemanager',

/*
|--------------------------------------------------------------------------
| Multi-User Mode
|--------------------------------------------------------------------------
*/

// If true, private folders will be created for each signed-in user.
'allow_multi_user' => true,
// If true, share folder will be created when allow_multi_user is true.
'allow_share_folder' => false,

// Flexibla way to customize client folders accessibility
// If you want to customize client folders, publish tag="lfm_handler"
// Then you can rewrite userField function in App\Handler\ConfigHander class
// And set 'user_field' to App\Handler\ConfigHander::class
// Ex: The private folder of user will be named as the user id.
'user_field' => Unisharp\Laravelfilemanager\Handlers\ConfigHandler::class,

/*
|--------------------------------------------------------------------------
| Working Directory
|--------------------------------------------------------------------------
*/

// Which folder to store files in project, fill in 'public', 'resources', 'storage' and so on.
// You should create routes to serve images if it is not set to public.
'base_directory' => 'public_html',

'images_folder_name' => 'images/news',
'files_folder_name'  => 'files',

'shared_folder_name' => 'shares',
'thumb_folder_name'  => 'thumbs',`

as you can see , my images store in public_html/images/news folder but src of image in explorer is "admin/laravel-filemanager/images/news".
Really appreciated if someone give any clue to fix this.

Use the script like this  in your blade:

@farbodkain there is a similar issue here: #506.