[Bug 🐞]: Warnings from Module Manager for kerbalism.cfg

Brief description of your issue
I just installed Mini Sample Return Capsule and have Kerbalism installed, and while loading KSP, Module Manager reported that there were 4 warnings. I checked its MMPatch.log and found the warnings were coming from Mini Sample Return Capsule's kerbalism.cfg:
[WRN 11:20:15.087] more than one pass specifier detected, ignoring all but the first: MiniSampleReturnCapsule/Compatibility/Kerbalism/@KERBALISM_HDD_SIZES:BEFORE[KerbalismDefault]:NEEDS[MiniSampleReturnCapsule,FeatureScience]:FOR[MiniSampleReturnCapsule]
[WRN 11:20:15.088] more than one pass specifier detected, ignoring all but the first: MiniSampleReturnCapsule/Compatibility/Kerbalism/@part[msrc-cargobay]:NEEDS[MiniSampleReturnCapsule,FeatureScience]:AFTER[KerbalismDefault]:FOR[MiniSampleReturnCapsule]
[WRN 11:20:15.088] more than one pass specifier detected, ignoring all but the first: MiniSampleReturnCapsule/Compatibility/Kerbalism/@part[msrc-parachute]:NEEDS[MiniSampleReturnCapsule,FeatureScience]:AFTER[KerbalismDefault]:FOR[MiniSampleReturnCapsule]
[WRN 11:20:15.088] more than one pass specifier detected, ignoring all but the first: MiniSampleReturnCapsule/Compatibility/Kerbalism/@part[msrc-parachute]:NEEDS[MiniSampleReturnCapsule]:FOR[KerbalismDefault]:FOR[MiniSampleReturnCapsule]

In the first three warnings, the FOR specifier looks superfluous.
In the fourth warning, there's two FOR specifiers and one refers to KerbalismDefault. I assume the fix for this line is removing the FOR[KerbalismDefault], since you're only supposed to use FOR specifiers for the mod the cfg file is in.
Steps to reproduce

Install Mini Sample Return Capsule.
Install Kerbalism? I don't know if it's necessary, since MM is complaining about syntax errors.
Start KSP, see if there's a message about warnings. If so, search your KSP folder\Logs\MMPatch.log for '[WRN'.

Expected behavior
No warnings from Module Manager
Actual behavior
Warnings from Module Manager
Environment
Mod: 1.4.2, according to the KSP Add-on Version Checker. CKAN thinks it is <IP_ADDRESS> for some reason, but according to the changelogs, there was no kerbalism.cfg in <IP_ADDRESS>.
KSP: <IP_ADDRESS>73
Download/Installation: CKAN
KSP.log: The thing to drop a KSP.log in appears to be missing from this form.

How did you download and install this?
CKAN (auto install with dependencies)

thank you @AmanitaVerna !
new release pending.

@AmanitaVerna
if you want - here should be the fixed file (GameData/MiniSampleReturnCapsule/Compatibility/Kerbalism.cfg)
Kerbalism.zip
Fixed

[Kerbalism.cfg] v<IP_ADDRESS>

Thank you [AmanitaVerna]
'more than one pass specifier detected'
removed :FOR[MiniSampleReturnCapsule] and kept :BEFORE/AFTER[KerbalismDefault]
concerned that calling KerbalismDefault will tell MM that it is installed.
fixes #71 - [Bug 🐞]: Warnings from Module Manager for kerbalism.cfg


new Cesium.CallbackProperty

When I use this function to draw a polygon（new Cesium.CallbackProperty（callback,false)），When the drawing is over,callback Still executing。Although this did not affect me, this form will consume performance, can you provide a removal function。

@yeshihua please ask questions like this on our community forum


Bump elasticsearch-py to 7.17.0 + remove ES 6 support

Description
A big diff that I would like as much attention as possible on!
The key underlying change is bumping elasticsearch-py to the latest 7.* version, 7.17.0.
This will - hopefully in a future diff - allow us to enable the ES 8 compatibility mode.
As a consequence of the version bump, ES 6 support is removed as discussed in https://github.com/jertel/elastalert2/issues/92#issuecomment-929706985. I took the opportunity to remove a lot of the ES-6 related code from the codebase.
I have tried to update as many unit tests as possible - as far as I'm aware I've only removed unit tests where they are no longer relevant because we don't support ES 5 or 6.
That said, this has involved some extensive refactoring. I have tested locally, both using the unit tests and an Elasticsearch instance, and things seem to be working, but I would appreciate at least one other person giving this a whirl before landing.
Checklist

[x] I have reviewed the contributing guidelines.
[x] I have included unit tests for my changes or additions.
[x] I have successfully run make test-docker with my changes.
[x] I have manually tested all relevant modes of the change in this PR.
[ ] I have updated the documentation.
[ ] I have updated the changelog.

Questions or Comments

Does changing elasticsearch-py to 7.17.0 mean to stop supporting opensearch? .. The connection of opensearch will be repelled from elasticsearch-py7.14.0.

doc_type is obsolete in elasticserch 8, do you take that into consideration?

Does the elastalert-test-rule work fine with elasticsearch 8 without modification?

Update setup.py as well as requirements.txt.

Users seem to want es8 support rather than refactoring. As a developer, I know you want to get rid of old code that you want to clean.

I'm not sure that the es5 code is squeezed but the es_mapping 5 folders and files are still there. Forgot to erase it?

Thanks for the review @nsano-rururu!
To address your points:

I haven't updated the documentation yet, I will do so soon
I forgot to remove the ES 5 mappings! Will do this as well 👍
About Elasticsearch 8 compatibility in general - elasticsearch-py 7.17.0 has an Elasticsearch 8 compatibility mode that can be activated by using an environment variable. My understanding is that if this environment variable is set, the library will handle all the doc_type conversion for now. So, I think it should be easy to enable ES 8 compatibility once this PR is landed.
I don't know what is best to do about maintaining Opensearch support. If we don't upgrade elasticsearch-py, it will become quite hard to add Elasticsearch 8 compatibility.

@ferozsalam
https://github.com/jertel/elastalert2/issues/92#issuecomment-927819135

Abstracting the calls to the Elastic API away from the general ElastAlert 2 source code and into a new search.py class would give us the ability to put all the logic in that new class for choosing whether to use the opensearch-py library or the elasticsearch-py library. This might be easier said than done, but it would help isolate all of this complexity into one place.
If @ferozsalam can prove that ES8 compatibility can be had with the removal of doc_type and without switching to the new library then let's proceed with getting ES8 support into master without changing the Python library and without breaking ES7 or OpenSearch compatibility.

https://github.com/jertel/elastalert2/issues/92#issuecomment-928217013

Mapping types and anything removed in 8.0 are still removed even when using "compatibility mode". However in client versions pre-7.16 will likely need to be more hands-on with the compatibility mode by settings HTTP headers yourself. In 7.16 I'm working towards getting the mode to be much easier to use.

OpenSearch client compatibility
https://opensearch.org/docs/clients/index/

Python Elasticsearch client   7.13.4

I don't know what is best to do about maintaining Opensearch support. If we don't upgrade elasticsearch-py, it will become quite hard to add Elasticsearch 8 compatibility.

Do not upgrade the version of elasticsearch-py.
Give up compatibility mode.
In case of elasticsearch 8, doc_type is deleted

That seems to be the opposite of what was agreed here some months ago.

If you can handle all the complaints from users, you can only say that you can merge them. When you say so.

I don't know what is best to do about maintaining Opensearch support. If we don't upgrade elasticsearch-py, it will become quite hard to add Elasticsearch 8 compatibility.

Then why are you asking a question? I don't know what it means.

I think your idea is that opensearch support isn't considered from the beginning.

With the current pull request content, I think I have no choice but to announce that opensearch should continue to use the past version as well as es6.

If you merge as it is, the effect will be huge and you will get a lot of inquiries even if you write it in the changelog or put it on the FAQ, so it is realistic to add the correspondence table of elastalert2 version, elasticseach version, opensearch at the top of the readme. It may be. Do you hate that too?

@ferozsalam
I installed it in the dockerfile of elastalert-server and tried it.

COPY config/elastalert.yaml /opt/elastalert/config.yaml
COPY config/config.json config/config.json
COPY rule_templates/ /opt/elastalert/rule_templates
COPY elastalert_modules/ /opt/elastalert/elastalert_modules

# Add default rules directory
# Set permission as unpriviledged user (1000:1000), compatible with Kubernetes
RUN mkdir -p /opt/elastalert/rules/ /opt/elastalert/server_data/tests/ \
    && chown -R node:node /opt

USER node

EXPOSE 3030

WORKDIR /opt/elastalert-server

ENTRYPOINT ["npm", "start"]

rule file
alert:
  - "slack"
alert_subject: "alert_subject: \"Test {} 123 aa☃\""
alert_subject_args:
  - "message"
alert_text: "Test {}  123 bb☃"
alert_text_args:
  - "message"
filter:
  - query:
      query_string:
        query: "message:Quit"
index: "mariadblog-*"
is_enabled: true
name: "a"
num_events: 2
realert:
  minutes: 5
slack_webhook_url: 'https://hooks.slack.com/services/xxxx'
slack_channel_override: "#abc"
slack_emoji_override: ":ghost:"
slack_msg_color: "danger"
slack_parse_override: "none"
slack_timeout: 10
slack_title_link: "http://<IP_ADDRESS>:8080/rules/a"
slack_username_override: "Praeco"
terms_size: 50
timeframe:
  minutes: 5
timestamp_field: "@timestamp"
timestamp_type: "iso"
type: "frequency"
use_strftime_index: false

docker log
10:58:24.377Z  INFO elastalert-server: Config:  No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
10:58:24.398Z  INFO elastalert-server: Config:  Proceeding to look for normal config file.
10:58:24.399Z  INFO elastalert-server: Config:  A config file was found in /opt/elastalert-server/config/config.json. Using that config.
10:58:24.500Z  INFO elastalert-server: Router:  Listening for GET request on /.
10:58:24.502Z  INFO elastalert-server: Router:  Listening for GET request on /status.
10:58:24.514Z  INFO elastalert-server: Router:  Listening for GET request on /status/errors.
10:58:24.516Z  INFO elastalert-server: Router:  Listening for GET request on /rules.
10:58:24.527Z  INFO elastalert-server: Router:  Listening for GET request on /rules/:id*.
10:58:24.535Z  INFO elastalert-server: Router:  Listening for POST request on /rules/:id*.
10:58:24.542Z  INFO elastalert-server: Router:  Listening for DELETE request on /rules/:id*.
10:58:24.549Z  INFO elastalert-server: Router:  Listening for GET request on /templates.
10:58:24.557Z  INFO elastalert-server: Router:  Listening for GET request on /templates/:id*.
10:58:24.562Z  INFO elastalert-server: Router:  Listening for POST request on /templates/:id*.
10:58:24.573Z  INFO elastalert-server: Router:  Listening for DELETE request on /templates/:id*.
10:58:24.579Z  INFO elastalert-server: Router:  Listening for PUT request on /folders/:type/:path*.
10:58:24.583Z  INFO elastalert-server: Router:  Listening for DELETE request on /folders/:type/:path*.
10:58:24.620Z  INFO elastalert-server: Router:  Listening for POST request on /test.
10:58:24.628Z  INFO elastalert-server: Router:  Listening for POST request on /silence/:path*.
10:58:24.633Z  INFO elastalert-server: Router:  Listening for GET request on /config.
10:58:24.635Z  INFO elastalert-server: Router:  Listening for POST request on /config.
10:58:24.653Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert.
10:58:24.654Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_status.
10:58:24.655Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/silence.
10:58:24.657Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/elastalert_error.
10:58:24.658Z  INFO elastalert-server: Router:  Listening for GET request on /metadata/past_elastalert.
10:58:24.658Z  INFO elastalert-server: Router:  Listening for GET request on /indices.
10:58:24.662Z  INFO elastalert-server: Router:  Listening for GET request on /mapping/:index.
10:58:24.663Z  INFO elastalert-server: Router:  Listening for POST request on /search/:index.
10:58:24.663Z  INFO elastalert-server: Router:  Listening for GET request on /config.
10:58:24.684Z  INFO elastalert-server: ProcessController:  Starting ElastAlert
10:58:24.685Z  INFO elastalert-server: ProcessController:  Creating index
10:58:34.746Z  INFO elastalert-server:
    ProcessController:  Reading Elastic 7 index mappings:
    Reading index mapping 'es_mappings/7/silence.json'
    Reading index mapping 'es_mappings/7/elastalert_status.json'
    Reading index mapping 'es_mappings/7/elastalert.json'
    Reading index mapping 'es_mappings/7/past_elastalert.json'
    Reading index mapping 'es_mappings/7/elastalert_error.json'
    New index praeco_elastalert_status created
    Done!

10:58:34.746Z ERROR elastalert-server:
    ProcessController:  /opt/elastalert/elastalert/create_index.py:39: DeprecationWarning: Using positional arguments for APIs is deprecated and will be disabled in 8.0.0. Instead use only keyword arguments for all APIs. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      if es_index.exists(ea_index):
    /opt/elastalert/elastalert/create_index.py:57: DeprecationWarning: Using positional arguments for APIs is deprecated and will be disabled in 8.0.0. Instead use only keyword arguments for all APIs. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      if es_index.exists(index_name):
    /opt/elastalert/elastalert/create_index.py:64: DeprecationWarning: Using positional arguments for APIs is deprecated and will be disabled in 8.0.0. Instead use only keyword arguments for all APIs. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      es_index.create(index_name)
    /home/node/.local/lib/python3.9/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: Camel case format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead.
      warnings.warn(message, category=ElasticsearchWarning)
    /home/node/.local/lib/python3.9/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: [types removal] Using include_type_name in put mapping requests is deprecated. The parameter will be removed in the next major version.
      warnings.warn(message, category=ElasticsearchWarning)

10:58:34.746Z  INFO elastalert-server: ProcessController:  Index create exited with code 0
10:58:34.754Z  INFO elastalert-server: ProcessController:  Starting elastalert with arguments [none]
10:58:34.775Z  INFO elastalert-server: ProcessController:  Started Elastalert (PID: 377)
10:58:34.782Z  INFO elastalert-server: Server:  Server listening on port 3030
10:58:34.792Z  INFO elastalert-server: Server:  Websocket listening on port 3333
10:58:34.799Z  INFO elastalert-server: Server:  Server started
10:58:40.541Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:59:10.961Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:59:41.364Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
10:59:46.058Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1611: DeprecationWarning: The 'body' parameter is deprecated for the 'index' API and will be removed in a future version. Instead use the 'document' parameter. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.index(index=index, body=body)

10:59:46.169Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/home/node/.local/lib/python3.9/site-packages/elasticsearch/connection/base.py:200: ElasticsearchWarning: Camel case format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead.
      warnings.warn(message, category=ElasticsearchWarning)

11:00:12.052Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:00:38.115Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1631: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)

11:00:42.388Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:00:45.837Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:371: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.thread_data.current_es.search(

11:01:12.789Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:01:43.113Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:02:13.411Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:02:43.720Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:02:51.701Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1876: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.search(index=index, size=1, body=query,

11:02:53.359Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
11:02:53.372Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:02:53.390Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/config'.
11:02:53.397Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/config'.
11:02:53.424Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules'.
11:03:14.289Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:03:30.577Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/rules/:id'.
11:03:44.919Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:04:15.349Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:04:46.035Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:04:57.605Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1611: DeprecationWarning: The 'body' parameter is deprecated for the 'index' API and will be removed in a future version. Instead use the 'document' parameter. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.index(index=index, body=body)

11:05:16.366Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:05:38.115Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1631: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)

11:05:46.658Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:05:58.472Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:371: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.thread_data.current_es.search(

11:05:59.070Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1876: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.search(index=index, size=1, body=query,

11:06:16.984Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:06:47.293Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:07:17.611Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:07:47.939Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:08:18.345Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:08:48.573Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:09:18.855Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:09:49.112Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.
11:10:09.330Z ERROR elastalert-server:
    ProcessController:  WARNING:py.warnings:/opt/elastalert/elastalert/elastalert.py:1611: DeprecationWarning: The 'body' parameter is deprecated for the 'index' API and will be removed in a future version. Instead use the 'document' parameter. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
      res = self.writeback_es.index(index=index, body=body)

11:10:19.513Z  INFO elastalert-server: Routes:  Successfully handled GET request for '/'.

elastalert-create-index
(py39) [CORP\sano@a-ngft53r34ong elastalert2]$ elastalert-create-index
Reading Elastic 7 index mappings:
Reading index mapping 'es_mappings/7/silence.json'
Reading index mapping 'es_mappings/7/elastalert_status.json'
Reading index mapping 'es_mappings/7/elastalert.json'
Reading index mapping 'es_mappings/7/past_elastalert.json'
Reading index mapping 'es_mappings/7/elastalert_error.json'
Deleting index elastalert_status_status.
Deleting index elastalert_status_error.
/home/sano/py39/lib/python3.9/site-packages/elasticsearch-7.17.0-py3.9.egg/elasticsearch/connection/base.py:200: ElasticsearchWarning: Camel case format name dateOptionalTime is deprecated and will be removed in a future version. Use snake case name date_optional_time instead.
  warnings.warn(message, category=ElasticsearchWarning)
/home/sano/py39/lib/python3.9/site-packages/elasticsearch-7.17.0-py3.9.egg/elasticsearch/connection/base.py:200: ElasticsearchWarning: [types removal] Using include_type_name in put mapping requests is deprecated. The parameter will be removed in the next major version.
  warnings.warn(message, category=ElasticsearchWarning)
New index elastalert_status created
Done!
(py39) [CORP\sano@a-ngft53r34ong elastalert2]$

elastalert-test-rule
Error in elastalert-test-rule
(py39) [CORP\sano@a-ngft53r34ong elastalert2]$ elastalert-test-rule rules/a.yaml
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
            To send them but remain verbose, use --verbose instead.
Error connecting to ElasticSearch:
AttributeError("type object 'ElastAlerter' has no attribute 'modify_rule_for_ES5'")
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
                To send them but remain verbose, use --verbose instead.
1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO:elastalert:Queried rule a from 2022-03-01 20:48 JST to 2022-03-01 20:53 JST: 0 / 0 hits

Would have written the following documents to writeback index (default is elastalert_status):

elastalert_status - {'rule_name': 'a', 'endtime': datetime.datetime(2022, 3, 1, 11, 53, 55, 380438, tzinfo=tzutc()), 'starttime': datetime.datetime(2022, 3, 1, 11, 48, 52, 380438, tzinfo=tzutc()), 'matches': 0, 'hits': 0, '@timestamp': datetime.datetime(2022, 3, 1, 11, 53, 55, 575557, tzinfo=tzutc()), 'time_taken': 0.014517068862915039}

(py39) [CORP\sano@a-ngft53r34ong elastalert2]$

Running ElastAlert 2
(py39) [CORP\sano@a-ngft53r34ong elastalert2]$ python -m elastalert.elastalert --verbose --rule rules/a.yaml
1 rules loaded
INFO:elastalert:Starting up
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.99984 seconds
WARNING:elasticsearch:POST http://localhost:9200/elastalert_status_status/_search?_source_includes=endtime%2Crule_name&size=1 [status:404 request:0.016s]
ERROR:elastalert:Error querying for last run: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status_status]', elastalert_status_status, index_or_alias)
INFO:elastalert:Queried rule a from 2022-03-01 20:45 JST to 2022-03-01 21:00 JST: 3 / 3 hits
WARNING:elasticsearch:POST http://localhost:9200/elastalert_status_silence/_search?_source_includes=until%2Cexponent&size=1 [status:404 request:0.007s]
ERROR:elastalert:Error while querying for alert silence status: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status_silence]', elastalert_status_silence, index_or_alias)
WARNING:elasticsearch:POST http://localhost:9200/elastalert_status_silence/_search?_source_includes=until%2Cexponent&size=1 [status:404 request:0.008s]
ERROR:elastalert:Error while querying for alert silence status: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status_silence]', elastalert_status_silence, index_or_alias)
INFO:elastalert:Alert 'a' sent to Slack
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:1611: DeprecationWarning: The 'body' parameter is deprecated for the 'index' API and will be removed in a future version. Instead use the 'document' parameter. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.writeback_es.index(index=index, body=body)

INFO:elastalert:Ran a from 2022-03-01 20:45 JST to 2022-03-01 21:00 JST: 3 query hits (0 already seen), 1 matches, 1 alerts sent
INFO:elastalert:a range 900
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:1631: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)

INFO:elastalert:Background configuration change check run at 2022-03-01 21:00 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.997979 seconds
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:00 JST
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:371: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.thread_data.current_es.search(

INFO:elastalert:Background configuration change check run at 2022-03-01 21:06 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:06 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999773 seconds
INFO:elastalert:Queried rule a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 / 12 hits
INFO:elastalert:Alert 'a' sent to Slack
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:1611: DeprecationWarning: The 'body' parameter is deprecated for the 'index' API and will be removed in a future version. Instead use the 'document' parameter. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.writeback_es.index(index=index, body=body)

WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:1876: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.writeback_es.search(index=index, size=1, body=query,

INFO:elastalert:Ignoring match for silenced rule a
INFO:elastalert:Ran a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 query hits (9 already seen), 2 matches, 1 alerts sent
INFO:elastalert:a range 358
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:1631: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)

INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999797 seconds
WARNING:py.warnings:/home/sano/dkwork3/wk2/elastalert2/elastalert/elastalert.py:371: DeprecationWarning: The 'body' parameter is deprecated for the 'search' API and will be removed in a future version. Instead use API parameters directly. See https://github.com/elastic/elasticsearch-py/issues/1698 for more information
  res = self.thread_data.current_es.search(

INFO:elastalert:Queried rule a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 / 12 hits
INFO:elastalert:Ran a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 query hits (12 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 370
INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999757 seconds
INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999779 seconds
INFO:elastalert:Queried rule a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 / 12 hits
INFO:elastalert:Ran a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 query hits (12 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 383
INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999748 seconds
INFO:elastalert:Queried rule a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 / 12 hits
INFO:elastalert:Ran a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 query hits (12 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 396
INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999725 seconds
INFO:elastalert:Queried rule a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 / 12 hits
INFO:elastalert:Ran a from 2022-03-01 21:01 JST to 2022-03-01 21:07 JST: 12 query hits (12 already seen), 0 matches, 0 alerts sent
INFO:elastalert:a range 409
INFO:elastalert:Background configuration change check run at 2022-03-01 21:07 JST
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2022-03-01 21:07 JST
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 9.999805 seconds

make test-docker has no test failures.

@jertel
Is there anything else you care about?

Thanks for the thorough review @nsano-rururu. I'm planning to spend some time on this throughout the week to see where we are with this change, how it will affect OpenSearch users, how it will affect ES 7 users, etc. There's a lot to review!

Might not be worth just not using the native client in order to support OpenSearch.  It's just rest APIs after all.  Elastic is being ridiculous.

@ferozsalam

I don't know what is best to do about maintaining Opensearch support. If we don't upgrade elasticsearch-py, it will become quite hard to add Elasticsearch 8 compatibility.

For reference only
Elasticseach-py 7.0.0 (= opensearch also works). If you want to support es8 while leaving the code for es5 and es6.
https://github.com/nsano-rururu/elastalert2/commit/405018a1797f2ae884cf60aadd4d92f436879973

If we proceed with this PR then I'm wondering why not upgrade to elasticsearch-py8? Do we gain anything by sticking with elasticsearch-py7 (7.17.0)? Neither that version nor py8 will support OpenSearch, so that OpenSearch compatibility issue will need to be dealt with regardless.

If we proceed with this PR then I'm wondering why not upgrade to elasticsearch-py8? Do we gain anything by sticking with elasticsearch-py7 (7.17.0)? Neither that version nor py8 will support OpenSearch, so that OpenSearch compatibility issue will need to be dealt with regardless.

I was under the impression that there would be further code changes required to get things working if we moved to elasticsearch-py 8 - I can give it a go and report back though. 👍
The other option I have been thinking about is seeing if the ES 8 compatibility mode is enabled in a version that supports OpenSearch, I don't actually know in which version of elasticsearch-py it was introduced.

@ferozsalam When you get a chance will you please merge master into your branch. It's gotten out of date due to a recent PR. Also, I'm thinking that in the create-index.py it should probably abort (e.g. exit(1)) if it detects an incompatible version. It looks like right now it will log the incompatibility detection, but then keep going and log another line stating that the indexes were created.

elasticsearech-py 8.0.1
make test-docker fails
ImportError while loading conftest '/home/elastalert/tests/conftest.py'.
conftest.py:9: in <module>
    import elastalert.elastalert
../elastalert/__init__.py:6: in <module>
    from elasticsearch import RequestsHttpConnection
E   ImportError: cannot import name 'RequestsHttpConnection' from 'elasticsearch' (/home/elastalert/tests/.tox/py310/lib/python3.10/site-packages/elasticsearch/__init__.py)
ERROR: InvocationError for command /home/elastalert/tests/.tox/py310/bin/pytest --cov=../elastalert --cov-report=term-missing --cov-branch --strict-markers . -n 4 (exited with code 4)

[8.0] Allow using http_auth with RequestsHttpNode for custom authentication
https://github.com/elastic/elasticsearch-py/issues/1778

Transport-layer classes aren't supplied via the elasticsearch package any more, the elastic_transport package is used for multiple packages including elasticsearch. You can find RequestsHttpNode in that package.

https://www.elastic.co/guide/en/elasticsearch/client/python-api/8.0/index.html
I feel that the following two measures are necessary when using elasticsearech-py 8.

Strict client configuration
Previously the client would use scheme="http", host="localhost", and port=9200 defaults when specifying which node(s) to connect to. Starting in 8.0 these defaults have been removed and instead require explicit configuration of scheme, host, and port or to be configured using cloud_id to avoid confusion about which Elasticsearch instance is being connected to.

This choice was made because starting 8.0.0 Elasticsearch enables HTTPS is by default, so it’s no longer a good assumption that http://localhost:9200 is the locally running cluster.

Start using .options() for transport parameters
Previously some per-request options like api_key and ignore were allowed within client API methods. Starting in 8.0 this is deprecated for all APIs and for a small number of APIs may break in unexpected ways if not changed.

The parameters headers, api_key, http_auth, opaque_id, request_timeout, and ignore are effected:

@ferozsalam When you get a chance will you please merge master into your branch. It's gotten out of date due to a recent PR. Also, I'm thinking that in the create-index.py it should probably abort (e.g. exit(1)) if it detects an incompatible version. It looks like right now it will log the incompatibility detection, but then keep going and log another line stating that the indexes were created.

Disregard - I've pushed the suggested change/merge this morning.

Cheers, I'm slammed at work until at least Friday, will have time to do further updates based on the feedback which has come in.

Here's where I'm at:

This PR branch has been updated to remove doc_type params and the now obsolete deprecated_search() function.
Updated docs to remove doc_type references
Tested a simple rule successfully on both ES 7.17.1 and 8.0.1.

Todo:

Test additional rules, like aggregations
Test Kibana dashboard
Find a solution for OpenSearch

I haven't used the Kibana dashboard feature. Has anyone else on this thread used it?
Sorry for hijacking your PR @ferozsalam. You got us started though so I am trying to keep up the momentum.

@jertel  @ferozsalam
I haven't used the following functions, but are they still working? Since there is no inquiry from the user, I thought that it could be deleted if it is not working.
use_kibana_dashboard
use_kibana4_dashboard
kibana4_start_timedelta
kibana4_end_timedelta

Sorry for hijacking your PR @ferozsalam. You got us started though so I am trying to keep up the momentum.

No worries, there's so much cruft to clean up I need all the help I can get 👍

I removed the v5 mappings dir and updated setup.py to use es 7.17.0 lib.

I removed the v5 mappings dir and updated setup.py to use es 7.17.0 lib.

For whoever ends up testing against ES 8 - you will also need to enable the compatibility mode via an environment variable first.

I removed the v5 mappings dir and updated setup.py to use es 7.17.0 lib.

For whoever ends up testing against ES 8 - you will also need to enable the compatibility mode via an environment variable first.

This is interesting, because I did not specify a new env variable when I did my testing against an 8.0.1 cluster earlier this morning. Yet my alerts came through and no errors in the ElastAlert 2 logs.

I removed the v5 mappings dir and updated setup.py to use es 7.17.0 lib.

For whoever ends up testing against ES 8 - you will also need to enable the compatibility mode via an environment variable first.

This is interesting, because I did not specify a new env variable when I did my testing against an 8.0.1 cluster earlier this morning. Yet my alerts came through and no errors in the ElastAlert 2 logs.

Curious! I guess we can leave it off for now and see what happens 🤔

This discussion (and @nsano-rururu's beta1 branch) got me thinking and so I went back to the python client version and downgraded to 7.10.1. With that version, all three appear to work fine:

ES 7.17.1
ES 8.0.1
OpenSearch 1.1

I think we should proceed with 7.10.1 for this PR and potentially release this as the next 2.4.0 version, noting the preliminary ES 8.0 support. Upgrading existing ES clusters to 8.x is still going to be a manual process via deletion, but we at least can get a release out to the community for feedback and further improvements.
Also, if anyone has a few spare cycles to spin up this version of ElastAlert2 against their clusters to confirm everything it still functioning as expected it would be appreciated. I have pushed a special Docker image up to Docker Hub with this PR: jertel/elastalert2:pr744
Let me know what you all think about this plan.

Sounds ideal! Thanks for the help 👍

Has the following error been resolved?
elastalert-test-rule
Error in elastalert-test-rule
(py39) [CORP\sano@a-ngft53r34ong elastalert2]$ elastalert-test-rule rules/a.yaml
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
            To send them but remain verbose, use --verbose instead.
Error connecting to ElasticSearch:
AttributeError("type object 'ElastAlerter' has no attribute 'modify_rule_for_ES5'")
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
                To send them but remain verbose, use --verbose instead.
1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO:elastalert:Queried rule a from 2022-03-01 20:48 JST to 2022-03-01 20:53 JST: 0 / 0 hits

Would have written the following documents to writeback index (default is elastalert_status):

elastalert_status - {'rule_name': 'a', 'endtime': datetime.datetime(2022, 3, 1, 11, 53, 55, 380438, tzinfo=tzutc()), 'starttime': datetime.datetime(2022, 3, 1, 11, 48, 52, 380438, tzinfo=tzutc()), 'matches': 0, 'hits': 0, '@timestamp': datetime.datetime(2022, 3, 1, 11, 53, 55, 575557, tzinfo=tzutc()), 'time_taken': 0.014517068862915039}

(py39) [CORP\sano@a-ngft53r34ong elastalert2]$

Has the following error been resolved?
elastalert-test-rule
Error in elastalert-test-rule

I just now pushed that fix. Thank you for pointing it out. An updated jertel/elastalert2:pr744 Docker image has also been pushed.

Forgot kibana 8.0 support for kibana discover?

@jertel

Pushed. Are you familiar with this configuration and able to validate it works with 8.0?

I'm currently working, so I'll check the operation after business hours.

Removed 5.6-6.8 in the document. 8.0 needs to be added
https://github.com/jertel/elastalert2/blob/master/docs/source/ruletypes.rst#kibana_discover_version
current
5.6
6.0, 6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8
7.0, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9, 7.10, 7.11, 7.12, 7.13, 7.14, 7.15, 7.16, 7.17

fix
7.0, 7.1, 7.2, 7.3, 7.4, 7.5, 7.6, 7.7, 7.8, 7.9, 7.10, 7.11, 7.12, 7.13, 7.14, 7.15, 7.16, 7.17, 8.0

schema.yaml also needs modification
https://github.com/jertel/elastalert2/blob/master/elastalert/schema.yaml#L267
current
kibana_discover_version: {type: string, enum: ['7.17', '7.16', '7.15', '7.14', '7.13', '7.12', '7.11', '7.10', '7.9', '7.8', '7.7', '7.6', '7.5', '7.4', '7.3', '7.2', '7.1', '7.0', '6.8', '6.7', '6.6', '6.5', '6.4', '6.3', '6.2', '6.1', '6.0', '5.6']}

fix
kibana_discover_version: {type: string, enum: ['8.0', '7.17', '7.16', '7.15', '7.14', '7.13', '7.12', '7.11', '7.10', '7.9', '7.8', '7.7', '7.6', '7.5', '7.4', '7.3', '7.2', '7.1', '7.0']}

Forgot to update changelog

schema.yaml also needs modification https://github.com/jertel/elastalert2/blob/master/elastalert

👍 – have updated this

Forgot to remove doc_type from the example file
/examples/rules/example_single_metric_agg.yaml
L15
doc_type: metricsets

/examples/rules/example_percentage_match.yaml
L18
doc_type: http_request

/examples/rules/example_spike_single_metric_agg.yaml
L18
doc_type: metricsets

/examples/rules/example_opsgenie_frequency.yaml
L74
#doc_type: "golog"

Isn't it okay to remove the description of elasticsearch2 below?
docs/source/recipes/writing_filters.rst
For Elasticsearch 2.X, any of the filters can be embedded in ``not``, ``and``, and ``or``::

    filter:
    - or:
        - term:
            field: "value"
        - wildcard:
            field: "foo*bar"
        - and:
            - not:
                term:
                  field: "value"
            - not:
                term:
                  _type: "something"

I've updated the upgrade notes with a warning that about deleting ElastAlert indices before starting the 8.x upgrade process. More information can be found on this Elastic 8 upgrade bug here: https://github.com/elastic/elasticsearch/issues/84199

kibana discover will not support kibana 8.0 with its current implementation. It seems that it needs to be fixed.
Error message
Cannot retrieve search results

[1:478] unknown field [query]

By the way, it seems that from kibana 8.0 it has changed to data view instead of the name index pattern. The setting screen is also data views. Therefore, it is necessary to add that the description of the index pattern ID of the document should also specify the data view id in kibana 8.0.

I just tested Discover URLs with ES 8 + Kibana 8 and it worked without any changes. I don't have any ElastAlert 2 error logs. Can you give me more information about the error? Was it printed in the ElastAlert 2 output? Or in Kibana error log?

By the way, it seems that from kibana 8.0 it has changed to data view instead of the name index pattern. The setting screen is also data views. Therefore, it is necessary to add that the description of the index pattern ID of the document should also specify the data view id in kibana 8.0.

This has been pushed. Thanks for reporting this.

you give me more information about the error? Was it printed in the ElastAlert 2 output? Or in Kibana error log?

There are no elastalert2 errors. I get an error when I display the url generated by elastalert2 in a browser.
By the way, shorten_kibana_discover_url is not enabled in my environment.
alert:
  - "slack"
alert_subject: "alert_subject: \"Test {} 123 aa☃\""
alert_subject_args:
  - "message"
alert_text: "Test {}  123 bb☃"
alert_text_args:
  - "message"
filter:
  - query:
      query_string:
        query: "message:Quit"
generate_kibana_discover_url: true
import: "BaseRule.config"
index: "mariadblog-*"
is_enabled: true
kibana_discover_app_url: "http://localhost:5601/app/discover#/"
kibana_discover_from_timedelta:
  minutes: 10
kibana_discover_index_pattern_id: "mariadblog"
kibana_discover_to_timedelta:
  minutes: 10
kibana_discover_version: "8.0"
match_enhancements: []
name: "a"
num_events: 2
realert:
  minutes: 5
slack_attach_kibana_discover_url: true
slack_channel_override: "#abc"
slack_emoji_override: ":ghost:"
slack_kibana_discover_color: "#ec4b98"
slack_kibana_discover_title: "Discover in Kibana"
slack_msg_color: "danger"
slack_parse_override: "none"
slack_timeout: 10
slack_title_link: "http://<IP_ADDRESS>:8080/rules/a"
slack_username_override: "Praeco"
slack_webhook_url: 'https://hooks.slack.com/services/xxxxxxxxxxx'
terms_size: 50
timeframe:
  minutes: 5
timestamp_field: "@timestamp"
timestamp_type: "iso"
type: "frequency"
use_strftime_index: false

Ok, I see it now. Since I didn't have an actual index pattern ID defined it defaulted to an existing view which allowed it to return results. When I specify a real pattern ID it will fail as you mentioned.

The Kibana 8.0 discover URL fix is now pushed. A new Docker image is also pushed for tag pr744.
Since we've dropped 6.x and earlier support I suggest in a future PR that the kibana_version logic and related schema property be removed. There's no point in spending time updating that list when there's no differences between the versions.

Since we've dropped 6.x and earlier support I suggest in a future PR that the kibana_version logic and related schema property be removed. There's no point in spending time updating that list when there's no differences between the versions.

I should have judged by looking at kibana_version about the shortened URL. If the api specifications do not change when es7 support is completely discontinued, it can be deleted, but it cannot be said that it will definitely change. I think that it is a parameter prepared to clarify which version is supported, so I think that it is better not to delete it.

look
https://github.com/Yelp/elastalert/pull/2474#discussion_r326744538

Since we've dropped 6.x and earlier support I suggest in a future PR that the kibana_version logic and related schema property be removed. There's no point in spending time updating that list when there's no differences between the versions.

I should have judged by looking at kibana_version about the shortened URL. If the api specifications do not change when es7 support is completely discontinued, it can be deleted, but it cannot be said that it will definitely change. I think that it is a parameter prepared to clarify which version is supported, so I think that it is better not to delete it.

That's fine with me to keep it. I know you are the one that typically does the version update, so if you don't mind that maintenance work let's continue with it as-is.

Is the fix that kibana Discover supports kibana 8 without affecting the behavior of kibana 7.x?

I tested the fix against both 7 and 8, both worked fine.

generate_kibana_link
generate_kibana_link doesn't seem to work for quite some time, so you can remove it. There was a bug report in yelp / elastalert issue at kibana 6. The developer has replied that it doesn't support kibana 6.
https://github.com/Yelp/elastalert/issues/437#issuecomment-409660250

This feature is only for Kibana 3, https://elastalert.readthedocs.io/en/latest/ruletypes.html?highlight=generate_kibana_link#generate-kibana-link, so it hasn't received much attention from me recently.
I see there is a slight bug, when using Elasticsearch 5 or 6 with this (would that even work with Kibana 3?), it will break if you are using query string filters.

The documentation says it's for kibana 3 only.
https://github.com/jertel/elastalert2/blob/master/docs/source/ruletypes.rst#generate_kibana_link

generate_kibana_link: This option is for Kibana 3 only. If true, ElastAlert 2 will generate a temporary Kibana dashboard and include a link to it in alerts. The dashboard consists of an events over time graph and a table with include fields selected in the table. If the rule uses query_key, the dashboard will also contain a filter for the query_key of the alert. The dashboard schema will be uploaded to the kibana-int index as a temporary dashboard. (Optional, boolean, default False)

use_kibana_dashboard  and use_kibana4_dashboard
use_kibana_dashboard also replies that the developer doesn't support kibana 6.
https://github.com/Yelp/elastalert/issues/1840#issuecomment-409660901

There's no support or integration with the newer versions of Kibana. Triggered alerts are written into elasticsearch, with the (default) type/index elastalert_status, as are other events (each query made, errors, pending aggregated alerts, silence stashes, etc). So, you should be able to view them in Kibana, but you have to set up your dashboard manually.

The documentation states that use_kibana_dashboard is for kibana 3 and use_kibana4_dashboard is for kibana 4.
https://github.com/jertel/elastalert2/blob/master/docs/source/ruletypes.rst#use_kibana_dashboard

use_kibana_dashboard: The name of a Kibana 3 dashboard to link to. Instead of generating a dashboard from a template, ElastAlert 2 can use an existing dashboard. It will set the time range on the dashboard to around the match time, upload it as a temporary dashboard, add a filter to the query_key of the alert if applicable, and put the url to the dashboard in the alert. (Optional, string, no default)

https://github.com/jertel/elastalert2/blob/master/docs/source/ruletypes.rst#use_kibana4_dashboard

use_kibana4_dashboard: A link to a Kibana 4 dashboard. For example, "https://kibana.example.com/#/dashboard/My-Dashboard". This will set the time setting on the dashboard from the match time minus the timeframe, to 10 minutes after the match time. Note that this does not support filtering by query_key like Kibana 3. This value can use $VAR and ${VAR} references to expand environment variables.

Thanks @nsano-rururu. All references to Kibana dashboard integration are now removed. Docker image with tag pr744 is pushed with these changes.

https://github.com/jertel/elastalert2/blob/master/chart/elastalert2/values.yaml#L82
  # generate_kibana_link: true


Errors trapped in window.onerror lacks stacktrace

I tested this in at least Firefox and Chrome, and the stack trace is completely uninformative if called from window.onerror. I guess this makes sense, since by the time the error has reached the onerror handler, the stack has already been unrolled and discarded. Since there's no error object that can be used, printStackTrace creates it's own error object. Since this is now the new point where the error has been thrown, and since no previous stack trace is available anyway, the stack trace is now filled with only stack frames from within the stacktrace.js function.
If I'm doing something something wrong and don't know about it, please let me know.
My code:
window.onerror = function() {
  alert(printStackTrace({guess:true}).join("\n");
}

function a() {
  b();
}

function b() {
  c();
}

a(); // Throws an error since c is undefined

Output in Firefox:
createException()@http://localhost:4000/stacktrace.js:42
run(null)@http://localhost:4000/stacktrace.js:27
printStackTrace([object Object])@http://localhost:4000/stacktrace.js:18
onerror("c is not defined","http://localhost:4000/test.js",6)@http://localhost:4000/test.html:5

Note that the last line in the trace above seems to be not from the stack, but from the arguments of the window.onerror handler, which I could have got anyway without printStackTrace by inspecting the arguments object in the handler. There's no mention of a or b that lead up to c.
Output in Chrome:
Object.createException (http://localhost:4000/stacktrace.js:42:18)
Object.run (http://localhost:4000/stacktrace.js:27:25)
printStackTrace (http://localhost:4000/stacktrace.js:18:62)
onerror()@http://localhost:4000/test.html:5:11

I tried moving a, and b to a separate js file included with a <script> tag, to see if {guess:true} will be able to help by reading the source, but that didn't help.
Am I doing something wrong?

The homepage (http://stacktracejs.org/) specifically states that you can get the stack trace of a thrown error by hooking into window.onerror. It looks like this is not the case. At least you can't get a meaningful stack trace this way. You should probably remove this example from the homepage since it is misleading.

@jpoehls You're right, it is misleading as it doesn't really give you a meaningful trace. I have a solution in the works for this, but for now I'm removing it from all my examples.

Could you share your solution? I would like to see it, even if it's not finished.

@pawel: Basically having stacktrace.js register an onerror handler itself that adds the given line and file passed into it to the top of the stack and working backwards from there.

Note that the window.onerror handler is still on the homapge.
Also, I'd be interested to find out how you plan to "work backwards" given the line and file.

subscribe


Prince George’s County, Maryland

Feature request
Prince George’s County Maryland is the wealthiest black county in America. However police brutality runs rampant here. Our schools are underserved and over policed. The time for change is now.
Angela Alsobrooks<EMAIL_ADDRESS>Todd Turner<EMAIL_ADDRESS>Rodney Streeter<EMAIL_ADDRESS>Mel Franklin<EMAIL_ADDRESS>Calvin Hawkins<EMAIL_ADDRESS>Thomas Dernoga<EMAIL_ADDRESS>Deni Taveras<EMAIL_ADDRESS>Dannielle Glaros<EMAIL_ADDRESS>Jolene Ivey<EMAIL_ADDRESS>Derrick Davis<EMAIL_ADDRESS>Monique Anderson-Walker<EMAIL_ADDRESS>Sydney Harrison<EMAIL_ADDRESS>Subject Line: Stop Choosing Police Over People
Body Text:
Prince George's County is the wealthiest black county in the country. For decades we have pursued education and economic status with the hopes that we could improve our conditions.
We were wrong. Even as we obtain degrees and own homes, Prince George's County Police continue to brutalize us as if we were subhuman.
Maryland has the highest number of black men incarcerated per capita, and Prince George's County Police plays a role in this statistic.
We have seen numerous videos of police brutality within our county. Why should the police be funded given their history of violence within our community?
In our schools, school resource officers contribute to the school to prison pipeline and continue to make our children feel unsafe.
Your job as elected officials is to take action when your constituents request. Today, your constituents are demanding that you defund the Prince George's County Police and spend those dollars in other areas that matter.
Remember we are watching your decisions, and should you choose to not take action, your consequence will come at the ballot box.
Thank You,

Hi @rehanawc , we can definitely work on adding Prince George's County to our list but we will need your help composing the message and getting the right recicpients. Please provide:

Subject line
Recipients
CC addresses
Body text

You can refer to this issue #699 for an example template. Thanks!

@ahansrisuk Thank you. I have completed the requested information.


[CSL-619] Delete broken 'MVar' functions

These functions are bad, because they don't take asynchronous exceptions
into account. Read carefully and you'll see it.
Also they are not used at all.
If someone wants these functions, it's usually better to use STM instead.

Unicode handling by git on Windows = 🤦♂


Fix #2443: Use JavaConverters instead of JavaConversions in javalib/.

Review by @nicolasstucki

Refer to this link for build results (access rights to CI server needed):
https://scala-webapps.epfl.ch/jenkins/job/scalajs-pr/3178/
https://scala-webapps.epfl.ch/jenkins/job/scalajs-matrix-build/4123/
Test PASSed.

LGTM


If we're interested in desktop users, design larger background canvas

could serve multiple svgs based on resolution

So lou did an interesting thing a little while back and put the app in a phone-looking-asset so even desktop users could have the experience.  Does that suit the use case described by this issue or do we want a desktop native feel as well?
A native desktop design would probably be larger than just this single issue / we may want to just tweak the issue title to be more about "exploring and documenting" that idea.


Carry --comment in child processes

I added the comment (if one was given) to the child processes as well, because I need it to get all the running processes of my application and this is an easy way for filtering. It would be nice if this could get integrated into the main repo.
An other note: Currently there is no way of stopping the spawned instances with something like 'spark stop' - is this planned?
Thanks!

Was it pland? come on now it me pple! Im the one in 6th grade worried bout history repeting too much...like in 1990s for reals.


Create a "second-chance" modal

When a user performs a potentially dangerous action, pop the modal and say "are you sure"? User can click yes or cancel. Cancel closes the modal and yes completes the user's intended action.

See my comment in #121


Allow ExDTLS to run in a NIF

This is pretty quick and dirty, but I wanted to see if I could run this as a NIF rather than a CNode. That would help not only if someone wanted to run the membrane_rtc_engine on an IPv6 network, but with -proto_dist inet_tls (which I've been playing around with in my free time).
I used the changes to ex_libnice that @mickel8 did a few months ago as a basis, with the caveat that I don't totally understand what is happening in unifex. In some cases, I had to change the interface to the C code—with the upside that it seems a little more internally consistent.
The tests pass for me, though it seems like the tests are not exhaustive—I ran this in a personal app using the rtc engine until I found all the crashes that I could trigger.

@mickel8 If you have time to review this, that would be great! I did my best with my limited understanding of unifex, but I would not be surprised if I missed something.


Change "Choose Template" for "Get started"

https://github.com/CartoDB/cartodb/commit/c78f731624886b590210683a404b0a2c5346b47b


Add an option to integrate with micrometer out of the box

if I didn't do sth wrong then currently we need to pass quite a few setup for micrometer and resilience4j integration. It would great to have an option to allow adding a customization to any circuitbreaker (regardless of the id).
This is how I worked this around ATM:
pom.xml

		<dependency>
			<groupId>io.micrometer</groupId>
			<artifactId>micrometer-registry-prometheus</artifactId>
			<version>${micrometer-registry-prometheus.version}</version>
		</dependency>
<!-- I had to manually pass the version of resilience4j -->
		<dependency>
			<groupId>io.github.resilience4j</groupId>
			<artifactId>resilience4j-micrometer</artifactId>
			<version>${resilience4j.version}</version>
		</dependency>

Configuration class
@Bean
	Customizer<Resilience4JCircuitBreakerFactory> defaultCustomizer(CircuitBreakerRegistry circuitBreakerRegistry, MeterRegistry meterRegistry) {
		return factory -> {
			factory.configureDefault(id -> new Resilience4JConfigBuilder(id)
					.timeLimiterConfig(TimeLimiterConfig.custom()
							.timeoutDuration(Duration.ofSeconds(2))
							.build())
					.circuitBreakerConfig(CircuitBreakerConfig.ofDefaults())
					.build());
			// for metrics
			factory.configureCircuitBreakerRegistry(circuitBreakerRegistry);
			// we need to allow adding those customizers regardless of the id
			factory.addCircuitBreakerCustomizer(circuitBreaker -> {
				CircuitBreakerMetrics
						.ofCircuitBreakerRegistry(circuitBreakerRegistry)
						.bindTo(meterRegistry);
			}, "verifyNewUser");
		};
	}

	@Bean
	CircuitBreakerRegistry resilience4JCircuitBreakerRegistry() {
		return CircuitBreakerRegistry.ofDefaults();
	}

Circuit breaker
circuitBreakerFactory.create("verifyNewUser")
					.run(() -> restTemplate.getForObject(uriComponentsBuilder.toUriString(),
							VerificationResult.class), throwable -> userRejected(userUuid, userAge));

I’ve ever also digged into monitoring resillience4j with spring cloud circuit breaker. The next two points were concerns.

addCircuitBreakerCustomizer requires all CircuitBreaker's ids. CircuitBreakerRegistry.getAllCircuitBreaker is the method to get all CBs. However, it’s difficult to retrieve all at the initializing phase. getAllCircuitBreaker will return empty list at the init phase as spring cloud circuit breaker creates and register a circuit breaker  in the registry at the actual execution phase. So we need to put all known ids to addCircuitBreakerCustomizer in the configuration in advance...
TaggedCircuitBreakerMetrics.bindTo(MeterRegistry) calls CircuitBreakerRegistry.getAllCircuitBreaker above and addMetrics to each CB. If we call TaggedCircuitBreakerMetrics.bindTo in the Customizer<CircuitBreaker>, which is the only customizable point as far as I know, every time when a CB is created, addMetrics for all CBs will be called. It’s quite inefficient. I’m not sure it really works.

With these restrictions, I came down to the implementation like following at that time,
	@Bean
	public Customizer<ReactiveResilience4JCircuitBreakerFactory> resilience4jCustomizer(
			MeterRegistry registry, BlogProperties props) {
		ConcurrentMap<String, Boolean> registered = new ConcurrentHashMap<>();
		return factory -> {
			factory.addCircuitBreakerCustomizer(circuitBreaker -> {
				this.configureMetrics(registry, circuitBreaker, registered);
				this.configureEventPublisher(circuitBreaker.getEventPublisher());
			}, "myCb1", "myCb2", "myCb3", "myCb4", "myCb5" /* ... */);
			// ommited
		};
	}

	private void configureMetrics(MeterRegistry registry, CircuitBreaker circuitBreaker,
			ConcurrentMap<String, Boolean> registered) {
		TaggedCircuitBreakerMetrics.MetricNames names = TaggedCircuitBreakerMetrics.MetricNames
				.ofDefaults();
		String circuitBreakerName = circuitBreaker.getName();
		if (!registered.containsKey(circuitBreakerName)) {
			registered.putIfAbsent(circuitBreakerName, true);
			log.info("Register metric for {}", circuitBreaker);
			// Copy from AbstractCircuitBreakerMetrics
			Gauge.builder(names.getStateMetricName(), circuitBreaker,
					(cb) -> cb.getState().getOrder())
					.tag(TagNames.NAME, circuitBreakerName).register(registry);
			Gauge.builder(names.getCallsMetricName(), circuitBreaker,
					(cb) -> cb.getMetrics().getNumberOfFailedCalls())
					.tag(TagNames.NAME, circuitBreakerName).tag(TagNames.KIND, "failed")
					.register(registry);
			Gauge.builder(names.getCallsMetricName(), circuitBreaker,
					(cb) -> cb.getMetrics().getNumberOfNotPermittedCalls())
					.tag(TagNames.NAME, circuitBreakerName)
					.tag(TagNames.KIND, "not_permitted").register(registry);
			Gauge.builder(names.getCallsMetricName(), circuitBreaker,
					(cb) -> cb.getMetrics().getNumberOfSuccessfulCalls())
					.tag(TagNames.NAME, circuitBreakerName)
					.tag(TagNames.KIND, "successful").register(registry);
			Gauge.builder(names.getBufferedCallsMetricName(), circuitBreaker,
					(cb) -> cb.getMetrics().getNumberOfBufferedCalls())
					.tag(TagNames.NAME, circuitBreakerName).register(registry);
			Gauge.builder(names.getMaxBufferedCallsMetricName(), circuitBreaker,
					(cb) -> cb.getMetrics().getMaxNumberOfBufferedCalls())
					.tag(TagNames.NAME, circuitBreakerName).register(registry);
		}
	}

When using resillience4j directly without spring cloud circuit breaker, it’s usual to create a CircuitBreaker at the initializing phase. Then we can achieve register all CBs to MeterRegistry as follows:
	@Bean
	public InitializingBean init(MeterRegistry meterRegistry, CircuitBreakerRegistry circuitBreakerRegistry) {
		return () -> TaggedCircuitBreakerMetrics.ofCircuitBreakerRegistry(circuitBreakerRegistry).bindTo(meterRegistry);
	}

I've created a simple project
Here is a project that uses resillience4j without spring cloud circuit breaker
https://github.com/making/demo-sccb-gh47/tree/master/demo-webflux-r4j
It looks easier to give the out of the box support for micrometer
To get the same experience with spring cloud circuit breaker, a project looks like
https://github.com/making/demo-sccb-gh47/tree/master/demo-webflux-sccb-r4j
Seems to need to change something in spring cloud circuit breaker

@making @marcingrzejszczak
I could see making this simpler, but I am really curious about @making's experience
I could see metrics for my CB just by doing the following
@Bean
	public Customizer<Resilience4JCircuitBreakerFactory> defaultCustomizer(MeterRegistry meterRegistry) {
		CircuitBreakerRegistry cbr = CircuitBreakerRegistry.ofDefaults();
		TaggedCircuitBreakerMetrics.ofCircuitBreakerRegistry(cbr).bindTo(meterRegistry);
		return factory -> {
			factory.configureDefault(id -> new Resilience4JConfigBuilder(id)
					.timeLimiterConfig(TimeLimiterConfig.custom().timeoutDuration(Duration.ofSeconds(3)).build())
					.circuitBreakerConfig(CircuitBreakerConfig.ofDefaults())
					.build());
			factory.configureCircuitBreakerRegistry(cbr);
		};
	}

This was in a non-reactive app, is the experience with a reactive app different?

This was in a non-reactive app, is the experience with a reactive app different?

I've confirmed my sample works after applying same changes in #51 👍
What I misunderstood is that all circuit breakers must be registered when TaggedCircuitBreakerMetrics is created but It doesn't.
https://github.com/resilience4j/resilience4j/blob/master/resilience4j-micrometer/src/main/java/io/github/resilience4j/micrometer/tagged/TaggedCircuitBreakerMetrics.java#L66-L69

@making thanks for confirming!

If you have questions in the future about Resilience4j, don't hesitate to ping us.
Did you test if your auto configuration works together with Spring Cloud Config? We added some new classes to support it, like https://github.com/resilience4j/resilience4j/blob/master/resilience4j-micrometer/src/main/java/io/github/resilience4j/micrometer/tagged/TaggedCircuitBreakerMetricsPublisher.java


Move network-txpool.rs example to its own folder

Describe the feature
The network-txpool.rs example should be in its own folder with its own Cargo.toml like the other examples.
Remember to update the README in examples/README.md to link to the new folder
Additional context
No response

Will be done


Check for redirects

To subdomain or path...
https://x.com/tunetheweb/status/1756229370752942099

Can I have a bash at this? Any guidance? Should it be done in utils?

we could do this as a preliminary step in the server handlers - basically just fetch the domain, check if a redirect is set or anything but a 200 is sent back.
we then need to handle the redirect (= redirect to the new domain on page-speed.dev) and also handle non-200 responses (= show a user-friendly error message)

Hi @danielroe,

shouldnt it be done once for both run and crux - same domain (unless i misunderstood you)?
was thinking to keep it simple one of two options

oftenn redirect is just going to be https://www.example.com instead of https://example.com - which i dont think we should chnage the url for rather instead just put a little message undernetaht the field stating "Final url tested was finalUrl" like pagespeed.dev does? we could do a chceck to see if base domain name has changed and then redirect? else mostly the url that a person enters does correlate to the site they want to test and thus just for everything put a message of which domain was tested instead of redirecting.

What are your thoughts?

@danielroe any thought on the linked PR?


Support for cssbundling-rails?

I'm currently trying to use mission_control-jobs with a setup utilising cssbundling-rails.
Whilst Sprockets seems to compile Mission Control's assets perfectly fine they aren't provided as stylesheet links to the application layout by default. I've tried adding stylesheet links to my layout manually but that seems to only partially fix the problem.
application.html.erb
      <%= stylesheet_link_tag "mission_control/jobs/application", "data-turbo-track": "reload" %>
      <%= stylesheet_link_tag "mission_control/jobs/forms", "data-turbo-track": "reload" %>
      <%= stylesheet_link_tag "mission_control/jobs/jobs", "data-turbo-track": "reload" %>

Somewhat adding to my confusion is the fact that the view renders only the queues table not the tabs or navbar:

Is there an existing way for things to play nicely with cssbundling-rails?

Hmm, I have to ask what your goal is? Why are you including anything Mission Control related in your main application layout?
Mission Control is a Rails Engine that is supposed to have full control and management of it's own assets and layout.
Generally, you install the Gem, and then when you mount it into your Routes, you use that path and everything is taken care of for you! No need to modify anything in your application.

Hi @HamptonMakes, thanks for your response.
My goal is to simply display Mission Control at my application's /jobs path.
I started going down the path of experimenting with assets as the default behaviour of Mission Control (with my css-bundling-rails setup) seems to not work...
I assume it's something to do with css-bundling-rails as Mission Control works fine on another project of mine with a no-build css setup.
Any pointers or theories greatly appreciated.
Cheers!

Hey @olinelson, sorry for the delay! I also don't quite understand very well what's happening here 😕 What happens if you simply mount Mission Control on /jobs in your routes? What kind of error do you get?

If you have content_security_policy enabled in your app.
You will need to add the css 'bulma', used by mission_control to the style_src ,
otherwise it will not work.
Rails.application.configure do
  config.content_security_policy do |policy|
      policy.style_src       :self<EMAIL_ADDRESS> end
end

It should be documented.
Or added by default when loading mission_control.

@kivanio good point. I'm going to vendor<EMAIL_ADDRESS>instead, so that you don't need to set your CSP to allow it.

https://github.com/rails/mission_control-jobs/pull/206

I released 0.6.0 with that change and other changes I had pending. @olinelson do you think the CSP was your problem?

ohh that was fast 🚀
good work @rosa 👍🏻
I confirm it working now

Hi all, after lots of digging I've found the issue and indeed it has NOTHING to do with mission_control-jobs.
I was extending the render method in ApplicationController and this was essentially causing mission control jobs to render but without it's required layout.
Sorry for the faff!
Onwards!


Added senario for Discovery not returning a result

As per #75

Codecov Report

Merging #92 into master will decrease coverage by 0.27%.
The diff coverage is 0%.

@@             Coverage Diff              @@
##             master      #92      +/-   ##
============================================
- Coverage     12.86%   12.58%   -0.28%
  Complexity       12       12
============================================
  Files            10       10
  Lines           272      278       +6
  Branches         32       33       +1
============================================
  Hits             35       35
- Misses          230      236       +6
  Partials          7        7

Impacted Files
Coverage Δ
Complexity Δ

...ation_with_discovery/discovery/DiscoveryQuery.java
0% <ø> (ø)
0 <0> (ø)
:arrow_down:

...tion_with_discovery/discovery/DiscoveryClient.java
5.55% <0%> (-0.7%)
2 <0> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5c509fa...49bf742. Read the comment docs.


Nuitka on MacOS with tensorflow

This is not an issue per se, just documenting my experience using Nuitka on MacOS for the first time in case someone else encounters the same issues.
OSX version: 10.14.5
Python version: miniconda - didnt work; brew installed python 3.7.3 worked OK
Nuitka version: latest dev Nuitka-0.6.4rc5
First had the
Error, call to 'install_name_tool' to fix Python library path failed.
problem as referenced in https://github.com/Nuitka/Nuitka/issues/94
I have solved this by removing miniconda installation of python, and installing python with brew as suggested in the issue.
Next had the following issue:
Scons command: /usr/local/Cellar/python/3.7.3/bin/python3.7 -W ignore /usr/local/lib/python3.7/site-packages/nuitka/build/inline_copy/bin/scons.py -f /usr/local/lib/python3.7/site-packages/nuitka/build/SingleExe.scons --jobs 20 --warn=no-deprecated --no-site-dir --debug=explain name=test_model result_name=test_model source_dir=test_model.build debug_mode=false python_debug=false unstripped_mode=false module_mode=false full_compat=false experimental= trace_mode=false python_version=3.7 target_arch=x86_64 python_prefix=/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7 nuitka_src=/usr/local/lib/python3.7/site-packages/nuitka/build nuitka_cache=/Users/prelovac/Library/Caches/Nuitka module_count=2 result_exe=test_model.bin show_scons=true abiflags=m
scons: Reading SConscript files ...
SCONS:  Tools= ['default']
Scons CC 'gcc' version check:
Scons CC 'gcc' version check gives '4.2.1' from '4.2.1'
Scons CC 'cc' version check:
Scons CC 'cc' version check gives '10.0.1' from 'Apple LLVM version 10.0.1 (clang-1<IP_ADDRESS>)'
Scons CC 'g++' version check:
Scons CC 'g++' version check gives '4.2.1' from '4.2.1'
Initial CC: gcc
Initial CCVERSION: 4.2.1
Initial CXX: g++
Initial CXXVERSION: 4.2.1
SCONS:  Picked compiler: clang
SCONS:  Picked name: clang
Scons compiler: Using /usr/bin/clang
Scons: Using resource mode: code
scons: Told to run compilation on 20 CPUs.
Scons: Launching target: ['test_model.bin']
scons: done reading SConscript files.
scons: Building targets ...
....

scons: building `test_model.bin' because it doesn't exist
clang -o test_model.bin -z noexecstack -Wno-deprecated-declarations<EMAIL_ADDRESS>-L/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib -lm -lpython3.7m
ld: unknown option: -z
clang: error: linker command failed with exit code 1 (use -v to see invocation)
scons: *** [test_model.bin] Error 1
scons: building terminated because of errors.

with
clang -v
Apple LLVM version 10.0.1 (clang-1<IP_ADDRESS>)
Target: x86_64-apple-darwin18.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

ld -v
@(#)PROGRAM:ld  PROJECT:ld64-450.3
BUILD 18:45:16 Apr  4 2019
configured to support archs: armv6 armv7 armv7s arm64 arm64e arm64_32 i386 x86_64 x86_64h armv6m armv7k armv7m armv7em
LTO support using: LLVM version 10.0.1, (clang-1<IP_ADDRESS>) (static support for 22, runtime is 22)
TAPI support using: Apple TAPI version 10.0.1 (tapi-1<IP_ADDRESS>)

Since this was in the last step and the cause was clear (-z noexecstack was not supported by the linker) I manually ran the last command excluding the '-z noexecstack' and it worked.
Did not investigate this further but this seemed unsupported by the linker (man ld | grep execstack returned nothing).
--
All in all I was checking if Nuitka could speed up a tensorflow model running on CPU. Compilation with --folow-imports took about 35 minutes (had to traverse the monster that tensorflow code is) and created a 28MB executable without any problems!
Here are the results:
Python 3.7.3: execution time 1m47s
Python 3.7.3 + Nuitka: execution time 1m42s
I did expect to see better results, but I guess tensorflow python libs were already heavily optimized.

This was a bad issue, no reproducer, wrong advice and is generally obsolete.
The package tensorflow should work now and CPython should be used.


python manage.py migrate has no sample data

I run the python.manage.py migrate for migrating database structure + sample data but it just has table without sample data
…
Steps to reproduce the problem

I run python.manage.py migrate
Database has alot of table but there are no sample data (Product,..... ) like Demo

Screenshots and logs

System information
Saleor version:

[ ] 2.11

Operating system:

[ ] Windows

i forget to run python manage.py populatedb
close


fix: Bumping version to 0.5.13 and making a note about version deprecations

#37  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @tazarov and the rest of your teammates on  Graphite


Have roles defined in CRO instead of in AEON?

The roles defined in AEON that are needed for representing the realization of academic events and event series should better be modeled in CRO .

for batch term requests to CRO see: https://github.com/data2health/contributor-role-ontology/issues/120

#144 was all about importing as many external roles as possible, to not dublicate any roles and to subsume the roles needed in AEON under CRO's contributor role.
For the remaining and very context dependent contributor roles, I think it's best to bring AEON to a stage where it can be submitted to OBO first, before actually requesting these to be rather defined in CRO. This way the OBO reviewers or CRO developers can properly inspect AEON's roles and decide/sugeest, if it better fits the scope of CRO. Until then they will be marked with the status ''requires discussion''.


TypeError: '<' not supported between instances of 'WhileBodyFuncGraph' and 'FuncGraph'

Hello,
I have ran tensorflow 2.1 and python 3.8 on a tensorflow project. I got the fp;;

Start training loop... 0%|          | 0/16153 [00:00<?, ?it/s]2021-11-23 17:42:25.980547: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2021-11-23 17:42:33.055750: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2021-11-23 17:42:33.387709: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll Traceback (most recent call last): File "C:\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 350, in _flatten_module leaves = nest.flatten_with_tuple_paths( File "C:\Anaconda3\lib\site-packages\tensorflow\python\util\nest.py", line 1425, in flatten_with_tuple_paths flatten(structure, expand_composites=expand_composites))) File "C:\Anaconda3\lib\site-packages\tensorflow\python\util\nest.py", line 341, in flatten return _pywrap_utils.Flatten(structure, expand_composites) TypeError: '<' not supported between instances of 'WhileBodyFuncGraph' and 'FuncGraph'
The above exception was the direct cause of the following exception: Traceback (most recent call last): File "code2seq.py", line 29, in  model.train() File "C:\modelrunner.py", line 129, in train gradients = tape.gradient(loss, self.model.trainable_variables) File "C:\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 175, in trainable_variables return tuple( File "C:\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 390, in _flatten_module for subvalue in subvalues: File "C:\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 390, in _flatten_module for subvalue in subvalues: File "C:\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 390, in _flatten_module for subvalue in subvalues: File "C:\Anaconda3\lib\site-packages\tensorflow\python\module\module.py", line 353, in _flatten_module six.raise_from( File "", line 3, in raise_from ValueError: Error processing property '_dropout_mask_cache' of <ContextValueCache at 0x2675729b820> 0%|          | 0/16153 [00:10<?, ?it/s]

Hi! @Avra2 !
Could you please update the template too as it helps us analyse the issue [tf version, steps followed before you ran into this error or stand alone code/colab gist to reproduce the issue faced].Thanks!


(chore) remove meta-keyword and meta-string in favor of nesting

Resolves #3163.
Checklist

[x] Added markup tests, or they don't apply here because...
[ ] Updated the changelog at CHANGES.md

Oh it was because my search and replace didn't care about the quoting. I see now. :)


Instructions for creating a new database are slightly misleading

Citus requires nodes to be up and have the Citus extension installed at master_add_node time such that reference tables, types and roles can be replicated during master_add_node and tables can be created immediately after. That requirement is now enforced even if the cluster is empty.
The current instructions for CREATE DATABASE seem to suggest that you can do the CREATE EXTENSION citus on the workers after adding the nodes, which is causing some confusion.
http://docs.citusdata.com/en/v9.4/admin_guide/cluster_management.html#creating-a-new-database
It might better to simply state that on all workers you run:
CREATE DATABASE newbie;
\c newbie
CREATE EXTENSION citus;

and on the coordinator you run:
CREATE DATABASE newbie;
\c newbie
CREATE EXTENSION citus;
SELECT * from master_add_node('node-name', 5432);
SELECT * from master_add_node('node-name2', 5432);

This was raised by @ivyazmitinov in https://github.com/citusdata/citus/issues/3879

I'd also add that setting citus.enable_alter_role_propagation = off is required when a password of user differs on the coordinator and workers nodes. It is a common use-case for a security based on .pgpass file, as described here.

Thanks for reporting. We'll get it prioritized


What kind of filter do we want on the videos?

Let's not mess with this now, maybe could add the filter later. Looks good with hyperlapse stabillization


PHP Fatal error:  Uncaught Hitrov\Exception\ApiCallException:

i already set all params to .env file, but when run php index.php it warning:
PHP 7.4.16
PHP Fatal error:  Uncaught Hitrov\Exception\ApiCallException: {
"code": "NotAuthenticated",
"message": "The required information to complete authentication was not provided or was incorrect."
} in /tmp/oci-arm-host-capacity/src/HttpClient.php:42
Stack trace:
#0 /tmp/oci-arm-host-capacity/src/OciApi.php(210): Hitrov\HttpClient::getResponse()
#1 /tmp/oci-arm-host-capacity/src/OciApi.php(102): Hitrov\OciApi->call()
#2 /tmp/oci-arm-host-capacity/index.php(66): Hitrov\OciApi->getInstances()
#3 {main}
thrown in /tmp/oci-arm-host-capacity/src/HttpClient.php on line 42

it is a simple task to get these pieces of information, but i dont know why that is incorrect @@

Hard to say. Try using double quotes for all of them. Double check this part https://github.com/hitrov/oci-arm-host-capacity#private-key-issues make sure you’re able to cat /home/user/privatekey.pem

i already chmod 777 to the pem file and tried to wrap values with single quotes and double quotes, but still got same warning. it is so weird @@.

can you verify that the script can run with your API key information? @hitrov

I re-run it just now. All good https://github.com/hitrov/oci-arm-host-capacity/runs/6607805332?check_suite_focus=true

Hello, I have ran the script for around a week and this exact same error occured. It used to run flawlessly and now it just happened out of nowhere. I have not made changes to the script whatsoever, so are there any solutions to this?


401: Unauthorized

Bot worked before. Had to make some minor changes Discord side. Fixed the Token as needed. Now I am getting a 401 error. Discord API guys say this:

This is the error the command prompt is displaying:

Tokens are all corrected wihin the credentials file. I was experiencing connection issues to the Discord API webtool before, and was being forced to redo the captcha repeatedly, so I was questioning if that could be the issue but they disagree.
I am lost.

Possible silly mistake, just checking: you are sure you used the token and not the secret, right?

401 unauthorized means you either didn't put your correct bot TOKEN (not secret) or your token got reset, so you need to generate a new one.


Hard dependency on aws-lambda

This is probably for @AlexStansfield
The Table.d.ts has a hard dependency on aws-lambda
import {DynamoDBRecord} from 'aws-lambda'

This is only used for:
stream(records: DynamoDBRecord[], params?: OneParams): StreamEntityGroup

Can we get this type definition elsewhere without such a hard dep.
Failing that, can we revert to
    stream(records: any[], params?: OneParams): StreamEntityGroup

Let me know Alex when you can look into this. We've had to work around and temporarily disable.


feat: Added configurable timeouts for fetchSegments and dispatch events ODP calls

Summary
Added configurable timeouts for fetchSegments and dispatch events ODP calls. The default timeout is 10s.
Test plan

Manually tested thoroughly
All test pass

Issues
FSSDK-8689

Looks good! Will be better with unit tests.

Added unit tests


Error with video generation

ERROR:root:Error loading prompt file: 'charmap' codec can't decode byte 0x9d in position 420: character maps to
ERROR:root:Error in video generation: 'charmap' codec can't decode byte 0x9d in position 420: character maps to
Locally ran in a Windows machine.

Solved after editing reddit_thread.yaml and storytelling_script.yaml


Deploying a Next.js project with custom build requires now-build

Steps to Reproduce

First off, create a new Next.js project:

// package.json
{
  "scripts": {
    "build": "next build"
  },
  "dependencies": {
    "next": "latest",
    "react": "latest",
    "react-dom": "latest"
  }
}

// pages/index.js
export default () => test;

Next, try to deploy it:

now

Works just fine! However, now try to do something custom before the build happens:

// package.json
{
  "scripts": {
    "build": "echo 'test' && next build"
  },
  ...
}

You will notice that nothing gets printed, even though you added an echo. This is because the Next.js Builder only invokes now-build.

We need to fix this and allow for (just like with all the other frameworks) build to be used.

Closed in favor of https://zeit.atlassian.net/browse/PRODUCT-106?atlOrigin=eyJpIjoiM2U1ZWQ0ZDFlMjFhNGFmZDk4YzUwYmM5MWRjNzg5ZTgiLCJwIjoiaiJ9


feat: cover 100% of crypto with tests

Covers a 100% of the crypto package with tests.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>2
Details

83 of 97 (85.57%)  changed or added relevant lines in 8 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.7%) to 65.436%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

internal/api/hooks.go
7
21
33.33%

Totals

Change from base Build<PHONE_NUMBER>5:
0.7%

Covered Lines:
9820

Relevant Lines:
15007

💛  - Coveralls


Split bill preview in app shows only 3 candidates, causing confusion when more than 3 are selected

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Click the plus sign at the bottom of the screen.
Select "Split Bill" and add desired number.
Choose more than 5 people to split with and click "Next". Observe that the app only includes 3 of them in the preview and does not indicate that others have been left out, causing confusion.

Expected Result:
App should display all selected individuals in preview or indicate that some have not been included with three dots.
Actual Result:
App displays only three candidates in preview when more than 5 or so are selected, causing confusion by not indicating that some have been left out.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[x] MacOS / Desktop

Version Number: <IP_ADDRESS>
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
screen-capture (91) (1).webm
https://github.com/Expensify/App/assets/93399543/aeaed34e-24af-4949-8dbc-cd29a7296ea4
Expensify/Expensify Issue URL:
Issue reported by: @tewodrosGirmaA
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1686648290378409
View all open jobs on GitHub

The list is scrollable

@bernhardoj - I have a proposal to show a indicator arrow at bottom. Like i have done here. This might help at many places to identify scrolls.
How it works:

Indicator arrow is shown only when there is a scroll available.
If the scroll is at the bottom, the indicator arrow is not shown.

https://github.com/Expensify/App/assets/78416198/12d24542-4af5-4dbd-8661-039b0f029641

I agree that when the window size cuts off the list exactly between two people, that it's not clear that it's a scrollable list.
It looks fine in this example:

But in this example, when I change the window size just slightly, it's not great:

@Expensify/design I'd love your thoughts on how best to fix this. Do we have an existing pattern for scrollable lists that does a better job indicating when there is more below the fold?

Hm good question. I don't think we do, so part of me thinks we could just close this?
We could consider some kind of fade out or something, but I don't feel strongly. Maybe bring this to the open source room for more discussion, or bump the thread?

@shawnborton @spcheema - Please ping here whenever you get an update.

Thanks @shawnborton, I'll bring up for discussion in open-source and post here with the decision. I think you're probably right that we should close for now, since it's not necessarily a bug, just a design pattern that could be improved.

Actually, we do have a similar design pattern - the LHN. There's no indicator that there are more chats below the fold. Given that we're following the existing design pattern, I'm just going to close this for now.

@sakluger @shawnborton - Please do assign me this task if you think to implement something in future.


Use type unicode for argument location

Short Description:
Fix for issue with invalid  value for location argument, e.g.: invalid  value: u'Pra\xe7a' when it contains special characters like "ç".
Fixes:

issue with invalid  value for location argument with special chars, e.g. "ç"

You didn't test using both command line and JSON location.

@douglascamata Thanks for spotting! Should be working with both command line and JSON file now.

Should fix issue #1466.

@vieira my code does the same thing as yours...

Please rename the function to something more clear: parse_utf8_str or parse_unicode_str.... dunno which one fits best.

The problem is easy to reproduce, just set initial location to e.g., "Praça do Comércio, Lisboa" and you will see the difference.

@douglascamata Ok, will do.

👍


Was this amazing project's code based on https://hackaday.io/project/189520-laser-range-finder ?

Hi!
Thanks for this project, that's an amazing learning resource!
I've noticed that the code here https://hackaday.io/project/189520-laser-range-finder in the image looks almost the same, is the project based on/inspired by that?

Hello.  My project started in 2017, so Chinese project is newer.


Return scrollbars

It feels really weird to have a scrollable area without scrollbars. Even when you don’t use scrollbars for actual scrolling, they are useful as an indicator where you are in the document (position of the scrollbar) and how it is long (height of the scrollbar).

One of the people I follow just starred it: malte-wessel/react-custom-scrollbars. It may be useful, but I haven’t looked into the code.

I know of this. Scrollbars are implemented and working as charm. That was one of the things I loved about Ace :-)
They are currently hidden. I did want to have this configurable, because it might be a matter of preference. I understand your point.
Firefox has cool subtle scrollbars.

Currently I am working on #42 . It might change my mind about that when I see it more complete form and eventially admit the need to hide scrollbars is pointless.
Leaving opened until resolved as configurable or as always visible scrollbars.

#51


Sonata v1.0 SPI block driver

The main changes:

Chip select (CS) lines are now controlled by a register in the SPI controller
itself.
FIFO depths are now available in the 'info' register, and the FIFOs
are reduced to 16 entries apiece.
Ensure that zero-byte transfer requests do not result in a deadlock
between software and hardware; the controller will attempt to start
but cannot perform a zero-byte transfer.

Zero-byte blocking_write/read can be used instead to synchronise
with the controller core.

Interrupts for the SPI block added.
Added internal loopback. Internal loopback within the SPI controller
is trivial and very useful for testing; simultaneous transmit and
receive shall result in all transmitted bytes being replicated
exactly in the receive FIFO; we can test permutations of polarity and
phase settings.
Added software reset. Software reset is required to recover in the
event of an error or indeed just a re-download of the software
without an intervening IP block reset presently. Clear the TX FIFO,
reset the controller core and then clear the RX FIFO.
Adapted ethernet driver to use of the builtin SPI chip select

It looks like the coding conventions check is failing due to a name collision between the old SPI driver v0.2/platform-spi.hh and the new platform-spi.hh.


Question about Equation (4)

In Equation 4 in your paper, you specify an equation for the spatial-aware attention module which averages across the levels of the rescaled spatial pyramid. It appears this equation would map an element of R^(LxSxC) to R^(SxC). Is this intended? How do you apply multiple DyHead modules after this stage, if the L axis is collapsed?

I was working on impelementing dynamic head these days. I found that the spatial attention is just a deformable convolution v2 (Deformable ConvNets v2: More Deformable, Better Results). I think there are L different deconvs applied on L  different feature map, and then get the mean of deconv outputs on L axis.

I was working on impelementing dynamic head these days. I found that the spatial attention is just a deformable convolution v2 (Deformable ConvNets v2: More Deformable, Better Results). I think there are L different deconvs applied on L different feature map, and then get the mean of deconv outputs on L axis.

Hi, can you share with me the dynamic head code you implemented? The original code is too complicated, thanks!