Elasticsearch 7.0.1 - Trying to create too many scroll contexts. Must be less than or equal to: [500]

Good afternoon,
I upgraded my stack to 7.0.1 and I am using elastalert>=0.2.0b. When I run Elasticsearch and elastalert, it alls seems to be working fine. However, the moment I ingest data, and the rules start hitting the indices, I get the following messages:
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.016s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.007s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-sysmon-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.007s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.005s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.010s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')

I updated the max_open_scroll_context to 5000 and I still get the same errors.
curl -X PUT localhost:9200/_cluster/settings -H 'Content-Type: application/json' -d'{
    "persistent" : {
        "search.max_open_scroll_context": 5000
    },
    "transient": {
        "search.max_open_scroll_context": 5000
    }
}
'

I haven't seen this error before, though I haven't used 7 yet. I just merged in https://github.com/Yelp/elastalert/pull/2230 which may solve this problem, though, I haven't released a new package version yet.
There may be some workarounds, like switching to use_count_query or use_terms_query if applicable, or by reducing buffer_time and increasing max_query_size so that it doesn't need to scroll. This completely depends on the amount of data and what your rule looks like though.
You said this happens immediately after you start getting data? Could there be something else with open scrolls other than elastalert? I wouldn't think you'd hit 500 open ones quickly, since elastalert should only be opening one per rule per query cycle.

Hey @Qmando , thank you for the quick response. testing that now 👍

Same error messages:
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [5000]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.012s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [5000]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.004s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [5000]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.008s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [5000]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.015s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [5000]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-sysmon-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.006s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [5000]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.012s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [5000]. This limit can be set by changing the [search.max_open_scroll_context] setting.')

I installed latest like this (Part of a Dockerfile):
 # ********* Install Elastalert **************
  && git clone https://github.com/Yelp/elastalert.git ${ESALERT_HOME} \
  && bash -c 'mkdir -pv /etc/elastalert/rules' \
  && cd ${ESALERT_HOME} \
  && python -m pip install --upgrade pip \
  && pip install urllib3==1.24.3 \
  && pip install -r requirements.txt \
  && python setup.py install \

I am using official Docker image docker.elastic.co/elasticsearch/elasticsearch:7.0.1 for this.

One of the queries failing is very basic:
alert:
- debug
description: Detects potential psexec via powershell module
filter:
- query:
    query_string:
      query: registry_key_value:(COMSPEC OR powershell.exe) OR service_image_path:(COMSPEC OR powershell.exe)
index: logs-endpoint-winevent-*
name: Windows-Psexec-Powershell-Module_0
priority: 2
realert:
  minutes: 0
type: any

I get the following in the status_error index:

This is my main config.yaml:
rules_folder: rules
run_every:
    seconds: 30
buffer_time:
    seconds: 45
es_host: helk-elasticsearch
es_port: 9200
alert_time_limit:
    days: 1
writeback_index: elastalert_status
alert_text: "Index: {0} \nEvent_Timestamp: {1} \nBeat_Name: {2} \nUser_Name: {3} \nHost_Name: {4} \nLog_Name: {5} \nOriginal_Message: \n\n{6}"
alert_text_type: alert_text_only
alert_text_args: ["_index","@timestamp","beat.name","user_name","host_name","log_name","z_original_message"]

There are a total of 20k records in my database but the data has timestamps from March/April. Therefore, my query is not even supposed to find anything in the past 1 day.

Hmm...
One thing you should check is that your version of the elasticsearch Python library is > 7. Maybe this is caused by some API change.
You can see from GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 that it's setting each scroll to a 30 second timeout. 7.0 changed a bunch of APIs so maybe that assumption is no longer right.
I really don't have much expertise here but I'd probably check to see whether there are other APIs you can use to see what scrolls are active, then you could manually clear them or verify that they are coming from elastalert and not something else.

Regarding library being newer than 7, you are right, 7.0.1 was released a week ago: https://github.com/elastic/elasticsearch-py/releases/tag/7.0.1 👍 . I will test that first.

Oh wait I am actually running the Elasticsearch python library 7.0.1
Successfully installed PyJWT-1.7.1 PyStaticConfiguration-0.10.4 PyYAML-5.1 asn1crypto-0.24.0 attrs-19.1.0 aws-requests-auth-0.4.2 blist-1.3.6 boto3-1.9.151 botocore-1.12.151 certifi-2019.3.9 cffi-1.12.3 chardet-3.0.4 configparser-3.7.4 croniter-0.3.30 cryptography-2.6.1 defusedxml-0.6.0 docopt-0.6.2 docutils-0.14 elasticsearch-7.0.1 enum34-1.1.6 envparse-0.2.0 exotel-0.1.5 funcsigs-1.0.2 functools32-3.2.3.post2 future-0.17.1 futures-3.2.0 idna-2.8 ipaddress-1.0.22 jira-1.0.14 jmespath-0.9.4 jsonschema-3.0.1 mock-3.0.5 oauthlib-3.0.1 ordereddict-1.1 pbr-5.2.0 pyOpenSSL-19.0.0 pycparser-2.19 pyrsistent-0.15.2 python-dateutil-2.6.1 python-magic-0.4.15 pytz-2019.1 requests-2.22.0 requests-oauthlib-1.2.0 requests-toolbelt-0.9.1 s3transfer-0.2.0 six-1.12.0 stomp.py-4.1.22 texttable-1.6.1 thehive4py-1.6.0 twilio-6.0.0 urllib3-1.25.2

I will check the search contexts open

I get the following:
helk@ubuntu:~/HELK/docker$ sudo docker exec -ti helk-elastalert curl -X GET helk-elasticsearch:9200/_nodes/stats/indices/search?pretty
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "helk-cluster",
  "nodes" : {
    "TioiZU7KQ6-7ibvmdohLEA" : {
      "timestamp" :<PHONE_NUMBER>964,
      "name" : "helk-1",
      "transport_address" : "<IP_ADDRESS>:9300",
      "host" : "<IP_ADDRESS>",
      "ip" : "<IP_ADDRESS>:9300",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "attributes" : {
        "ml.machine_memory" : "13549408256",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20"
      },
      "indices" : {
        "search" : {
          "open_contexts" : 3705,
          "query_total" : 261321,
          "query_time_in_millis" : 11645,
          "query_current" : 0,
          "fetch_total" : 5726,
          "fetch_time_in_millis" : 19415,
          "fetch_current" : 0,
          "scroll_total" : 246750,
          "scroll_time_in_millis" :<PHONE_NUMBER>8,
          "scroll_current" : 3705,
          "suggest_total" : 0,
          "suggest_time_in_millis" : 0,
          "suggest_current" : 0
        }
      }
    }
  }
}

To be honest I'm not entirely sure how to interpret
          "scroll_total" : 246750,
          "scroll_time_in_millis" :<PHONE_NUMBER>8,
          "scroll_current" : 3705,

I wish I had a better answer for you.
If you want to try and debug further, you can use --es_debug_trace es.log parameter to dump the full requests being made to Elasticsearch. If it's an ElastAlert issue, I'll try to fix asap, but I'm out of my expertise here.

Thank you for the details and extra steps to try. yeah 5 months is crazy. Let me create a fresh ES server and fresh elastalert without any data in it. Does the --es_debug_trace es.log gos next to the --verbose command when starting Elastalert?

Next to --verbose is fine (order doesn't matter). You should see a bunch of stuff in there after running elastalert for a short time. Just note it prints out things with curl localhost:9200 despite whatever settings you have, a "feature" of the elasticsearch library. ;)

Ok I have a fresh ES server:
helk@ubuntu:~/HELK/docker$ sudo docker exec -ti helk-elasticsearch curl -X GET "localhost:9200/_nodes/stats/indices/search?pretty"
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "helk-cluster",
  "nodes" : {
    "Y9alb0wgTwO1Dj_8yEJwYw" : {
      "timestamp" :<PHONE_NUMBER>174,
      "name" : "helk-1",
      "transport_address" : "<IP_ADDRESS>:9300",
      "host" : "<IP_ADDRESS>",
      "ip" : "<IP_ADDRESS>:9300",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "attributes" : {
        "ml.machine_memory" : "13549408256",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20"
      },
      "indices" : {
        "search" : {
          "open_contexts" : 0,
          "query_total" : 0,
          "query_time_in_millis" : 0,
          "query_current" : 0,
          "fetch_total" : 0,
          "fetch_time_in_millis" : 0,
          "fetch_current" : 0,
          "scroll_total" : 0,
          "scroll_time_in_millis" : 0,
          "scroll_current" : 0,
          "suggest_total" : 0,
          "suggest_time_in_millis" : 0,
          "suggest_current" : 0
        }
      }
    }
  }
}

Then I have only a few rules running without any data in the database:
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:50 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:50 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:50 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:50 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:50 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:50 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 29.955974 seconds
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:51 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:51 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:51 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:51 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:51 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:50 UTC to 2019-05-17 22:51 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 29.913904 seconds
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:51 UTC to 2019-05-17 22:51 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:51 UTC to 2019-05-17 22:51 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:51 UTC to 2019-05-17 22:51 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:51 UTC to 2019-05-17 22:51 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:51 UTC to 2019-05-17 22:51 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:51 UTC to 2019-05-17 22:51 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 29.960197 seconds

Im also tailing the es.log file:
curl -XGET 'http://localhost:9200/?pretty' -d ''
curl -H 'Content-Type: application/json' -XGET 'http://localhost:9200/logs-endpoint-winevent-*/_search?pretty&ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000' -d '{
  "query": {
    "bool": {
      "filter": {
        "bool": {
          "must": [
            {
              "range": {
                "@timestamp": {
                  "gt": "2019-05-17T22:51:36.602383Z",
                  "lte": "2019-05-17T22:52:21.602383Z"
                }
              }
            },
            {
              "query_string": {
                "query": "registry_key_value:(COMSPEC OR powershell.exe) OR service_image_path:(COMSPEC OR powershell.exe)"
              }
            }
          ]
        }
      }
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "asc"
      }
    }
  ]
}'
curl -H 'Content-Type: application/json' -XPOST 'http://localhost:9200/elastalert_status_status/_doc?pretty' -d '{
  "@timestamp": "2019-05-17T22:52:21.614659Z",
  "endtime": "2019-05-17T22:52:21.602383Z",
  "hits": 0,
  "matches": 0,
  "rule_name": "Windows-Psexec-Powershell-Module_0",
  "starttime": "2019-05-17T22:51:36.602383Z",
  "time_taken": 0.012163877487182617
}'

After a few minutes I checked the search context open and these are the results. Without data in the database:
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "helk-cluster",
  "nodes" : {
    "Y9alb0wgTwO1Dj_8yEJwYw" : {
      "timestamp" :<PHONE_NUMBER>144,
      "name" : "helk-1",
      "transport_address" : "<IP_ADDRESS>:9300",
      "host" : "<IP_ADDRESS>",
      "ip" : "<IP_ADDRESS>:9300",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "attributes" : {
        "ml.machine_memory" : "13549408256",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20"
      },
      "indices" : {
        "search" : {
          "open_contexts" : 0,
          "query_total" : 1022,
          "query_time_in_millis" : 653,
          "query_current" : 0,
          "fetch_total" : 1022,
          "fetch_time_in_millis" : 401,
          "fetch_current" : 0,
          "scroll_total" : 0,
          "scroll_time_in_millis" : 0,
          "scroll_current" : 0,
          "suggest_total" : 0,
          "suggest_time_in_millis" : 0,
          "suggest_current" : 0
        }
      }
    }
  }
}

I now sent some data to the database:
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "helk-cluster",
  "nodes" : {
    "Y9alb0wgTwO1Dj_8yEJwYw" : {
      "timestamp" :<PHONE_NUMBER>977,
      "name" : "helk-1",
      "transport_address" : "<IP_ADDRESS>:9300",
      "host" : "<IP_ADDRESS>",
      "ip" : "<IP_ADDRESS>:9300",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "attributes" : {
        "ml.machine_memory" : "13549408256",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20"
      },
      "indices" : {
        "search" : {
          "open_contexts" : 51,
          "query_total" : 1266,
          "query_time_in_millis" : 1291,
          "query_current" : 0,
          "fetch_total" : 1196,
          "fetch_time_in_millis" : 1738,
          "fetch_current" : 0,
          "scroll_total" : 17,
          "scroll_time_in_millis" : 716175,
          "scroll_current" : 51,
          "suggest_total" : 0,
          "suggest_time_in_millis" : 0,
          "suggest_current" : 0
        }
      }
    }
  }
}

I dont see any of the error messages from before:
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:54 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:54 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:54 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:54 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:54 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:54 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 29.922236 seconds
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:55 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:55 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:55 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:55 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:55 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:54 UTC to 2019-05-17 22:55 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 29.953053 seconds
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:55 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:55 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:55 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:55 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:55 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:55 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 29.95311 seconds
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:56 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:56 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:56 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:56 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:56 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-17 22:55 UTC to 2019-05-17 22:56 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 29.941673 seconds

I checked the es.log:
curl -H 'Content-Type: application/json' -XGET 'http://localhost:9200/logs-endpoint-winevent-*/_search?pretty&ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000' -d '{
  "query": {
    "bool": {
      "filter": {
        "bool": {
          "must": [
            {
              "range": {
                "@timestamp": {
                  "gt": "2019-05-17T22:56:06.759775Z",
                  "lte": "2019-05-17T22:56:51.759775Z"
                }
              }
            },
            {
              "query_string": {
                "query": "registry_key_value:(COMSPEC OR powershell.exe) OR service_image_path:(COMSPEC OR powershell.exe)"
              }
            }
          ]
        }
      }
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "asc"
      }
    }
  ]
}'
curl -H 'Content-Type: application/json' -XPOST 'http://localhost:9200/elastalert_status_status/_doc?pretty' -d '{
  "@timestamp": "2019-05-17T22:56:51.770938Z",
  "endtime": "2019-05-17T22:56:51.759775Z",
  "hits": 0,
  "matches": 0,
  "rule_name": "Windows-Psexec-Powershell-Module_0",
  "starttime": "2019-05-17T22:56:06.759775Z",
  "time_taken": 0.011156082153320312
}'

Everything looks normal at the moment.

The only difference is the number of rules I am using. I am only running 3 rules at the moment. I believe some rules might be triggering that behavior. I have around 210 rules besides the 3 I used for this test.

I let my ELK stack and Elastalert running the whole night and I got the same scroll errors with only 3 rules:
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:48 UTC to 2019-05-18 01:48 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:48 UTC to 2019-05-18 01:49 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:49 UTC to 2019-05-18 01:50 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:50 UTC to 2019-05-18 01:51 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:51 UTC to 2019-05-18 01:51 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:51 UTC to 2019-05-18 01:52 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:52 UTC to 2019-05-18 01:53 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:53 UTC to 2019-05-18 01:54 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:54 UTC to 2019-05-18 01:54 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:54 UTC to 2019-05-18 01:55 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:55 UTC to 2019-05-18 01:56 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:56 UTC to 2019-05-18 01:57 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:57 UTC to 2019-05-18 01:57 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:57 UTC to 2019-05-18 01:58 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:58 UTC to 2019-05-18 01:59 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:59 UTC to 2019-05-18 02:00 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:00 UTC to 2019-05-18 02:00 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:00 UTC to 2019-05-18 02:01 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:01 UTC to 2019-05-18 02:02 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:02 UTC to 2019-05-18 02:03 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:03 UTC to 2019-05-18 02:03 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:03 UTC to 2019-05-18 02:04 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:04 UTC to 2019-05-18 02:05 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:05 UTC to 2019-05-18 02:06 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:06 UTC to 2019-05-18 02:06 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:06 UTC to 2019-05-18 02:07 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:07 UTC to 2019-05-18 02:08 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:08 UTC to 2019-05-18 02:09 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:09 UTC to 2019-05-18 02:09 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:09 UTC to 2019-05-18 02:10 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:10 UTC to 2019-05-18 02:11 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:11 UTC to 2019-05-18 02:12 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:12 UTC to 2019-05-18 02:12 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:12 UTC to 2019-05-18 02:13 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:13 UTC to 2019-05-18 02:14 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:14 UTC to 2019-05-18 02:15 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:15 UTC to 2019-05-18 02:15 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:15 UTC to 2019-05-18 02:16 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:16 UTC to 2019-05-18 02:17 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:17 UTC to 2019-05-18 02:18 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:18 UTC to 2019-05-18 02:18 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:18 UTC to 2019-05-18 02:19 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:19 UTC to 2019-05-18 02:20 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:20 UTC to 2019-05-18 02:21 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:21 UTC to 2019-05-18 02:21 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:21 UTC to 2019-05-18 02:22 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:22 UTC to 2019-05-18 02:23 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:23 UTC to 2019-05-18 02:24 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:24 UTC to 2019-05-18 02:24 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:24 UTC to 2019-05-18 02:25 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:25 UTC to 2019-05-18 02:26 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:26 UTC to 2019-05-18 02:27 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:27 UTC to 2019-05-18 02:27 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:27 UTC to 2019-05-18 02:28 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:28 UTC to 2019-05-18 02:29 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:29 UTC to 2019-05-18 02:30 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:30 UTC to 2019-05-18 02:30 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:30 UTC to 2019-05-18 02:31 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:31 UTC to 2019-05-18 02:32 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:32 UTC to 2019-05-18 02:33 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:33 UTC to 2019-05-18 02:33 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:33 UTC to 2019-05-18 02:34 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:34 UTC to 2019-05-18 02:35 UTC: 0 / 0 hits
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.012s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-18 01:48 UTC to 2019-05-18 13:53 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-sysmon-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.005s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-18 01:48 UTC to 2019-05-18 13:53 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=30s&size=10000 [status:500 request:0.015s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-18 01:48 UTC to 2019-05-18 13:53 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 29.569125 seconds

This is what I got from the es.log
curl -H 'Content-Type: application/json' -XPOST 'http://localhost:9200/elastalert_status_error/_doc?pretty' -d '{
  "@timestamp": "2019-05-18T13:54:53.171471Z",
  "data": {
    "query": {
      "query": {
        "bool": {
          "filter": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gt": "2019-05-18T01:48:01.123530Z",
                      "lte": "2019-05-18T01:48:46.123530Z"
                    }
                  }
                },
                {
                  "query_string": {
                    "query": "registry_key_value:(COMSPEC OR powershell.exe) OR service_image_path:(COMSPEC OR powershell.exe)"
                  }
                }
              ]
            }
          }
        }
      },
      "sort": [
        {
          "@timestamp": {
            "order": "asc"
          }
        }
      ]
    },
    "rule": "Windows-Psexec-Powershell-Module_0"
  },
  "message": "Error running query: TransportError(500, u\u0027search_phase_execution_exception\u0027, u\u0027Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\u0027)",
  "traceback": [
    "Traceback (most recent call last):",
    "  File \"/usr/local/lib/python2.7/dist-packages/elastalert-0.1.39-py2.7.egg/elastalert/elastalert.py\", line 352, in get_hits",
    "    **extra_args",
    "  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/utils.py\", line 84, in _wrapped",
    "    return func(*args, params=params, **kwargs)",
    "  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/client/__init__.py\", line 818, in search",
    "    \"GET\", _make_path(index, \"_search\"), params=params, body=body",
    "  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/transport.py\", line 353, in perform_request",
    "    timeout=timeout,",
    "  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_requests.py\", line 143, in perform_request",
    "    self._raise_error(response.status_code, raw_data)",
    "  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py\", line 168, in _raise_error",
    "    status_code, error_message, additional_info",
    "TransportError: TransportError(500, u\u0027search_phase_execution_exception\u0027, u\u0027Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.\u0027)"
  ]
}'

This is what I got also from the scroll context:
helk@ubuntu:~$ sudo docker exec -ti helk-elasticsearch curl -X GET "localhost:9200/_nodes/stats/indices/search?pretty"
[sudo] password for helk:
Sorry, try again.
[sudo] password for helk:
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "helk-cluster",
  "nodes" : {
    "Y9alb0wgTwO1Dj_8yEJwYw" : {
      "timestamp" :<PHONE_NUMBER>265,
      "name" : "helk-1",
      "transport_address" : "<IP_ADDRESS>:9300",
      "host" : "<IP_ADDRESS>",
      "ip" : "<IP_ADDRESS>:9300",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "attributes" : {
        "ml.machine_memory" : "13549408256",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20"
      },
      "indices" : {
        "search" : {
          "open_contexts" : 500,
          "query_total" : 19616,
          "query_time_in_millis" : 2208,
          "query_current" : 0,
          "fetch_total" : 6052,
          "fetch_time_in_millis" : 2648,
          "fetch_current" : 0,
          "scroll_total" : 13402,
          "scroll_time_in_millis" : 729151894,
          "scroll_current" : 500,
          "suggest_total" : 0,
          "suggest_time_in_millis" : 0,
          "suggest_current" : 0
        }
      }
    }
  }
}

I only have around 1k records but from March

Is there anything else that you would recommend to test to provide information? @Qmando

Something that I noticed was the time widows used for the queries:
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:32 UTC to 2019-05-18 02:33 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:33 UTC to 2019-05-18 02:33 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:33 UTC to 2019-05-18 02:34 UTC: 0 / 0 hits
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 02:34 UTC to 2019-05-18 02:35 UTC: 0 / 0 hits

I see from 2H 34 mins to 2H 35 mins. I am still learning Elastalert so I wasnt sure if that is the most ideal. This is part of my Elastalert config:
rules_folder: rules
run_every:
    seconds: 30
buffer_time:
    seconds: 45
es_host: helk-elasticsearch
es_port: 9200
alert_time_limit:
    days: 1

Is the 1 mins time window because of the buffer_time?

This is what the config.yaml.example look like:
run_every:
  minutes: 1

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 15

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

I will change it to the defaults for now and let it run

Ok I updated the following:
scroll_keepalive: The maximum time (formatted in Time Units) the scrolling context should be kept alive. Avoid using high values as it abuses resources in Elasticsearch, but be mindful to allow sufficient time to finish processing all the results.

the examples provided by Elastic docs: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html
That changed has stopped so far the error messages:
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 14:33 UTC to 2019-05-18 14:48 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-18 14:33 UTC to 2019-05-18 14:48 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-18 14:33 UTC to 2019-05-18 14:48 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-18 14:33 UTC to 2019-05-18 14:48 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-18 14:33 UTC to 2019-05-18 14:48 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-18 14:33 UTC to 2019-05-18 14:48 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.953273 seconds
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 14:34 UTC to 2019-05-18 14:49 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-18 14:34 UTC to 2019-05-18 14:49 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-18 14:34 UTC to 2019-05-18 14:49 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-18 14:34 UTC to 2019-05-18 14:49 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-18 14:34 UTC to 2019-05-18 14:49 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-18 14:34 UTC to 2019-05-18 14:49 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.933993 seconds
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 14:35 UTC to 2019-05-18 14:50 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-18 14:35 UTC to 2019-05-18 14:50 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-18 14:35 UTC to 2019-05-18 14:50 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-18 14:35 UTC to 2019-05-18 14:50 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-18 14:35 UTC to 2019-05-18 14:50 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-18 14:35 UTC to 2019-05-18 14:50 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.929616 seconds
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 14:36 UTC to 2019-05-18 14:51 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-18 14:36 UTC to 2019-05-18 14:51 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-18 14:36 UTC to 2019-05-18 14:51 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-18 14:36 UTC to 2019-05-18 14:51 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-18 14:36 UTC to 2019-05-18 14:51 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-18 14:36 UTC to 2019-05-18 14:51 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 59.952816 seconds

I added all the rules I use (around 210 rules) to it and I started getting all the errors about scrolling again. Something that I noticed on several rules that were potentially causing issues is that the rules were being applied to multiple indices at once. This is what I mean. i have the following indices:

As you can see, there is one logs-* that points to all of them. 119 rules have specific indices such as
logs-endpoint-winevent-security-* or logs-endpoint-winevent-sysmon-* . However, the rest point to logs-*. I took all the elastalert rules with index logs-* and now it seems to be stable:
INFO:elastalert:Queried rule PowerShell-Downgrade-Attack_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran PowerShell-Downgrade-Attack_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule PowerShell-Network-Connections_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran PowerShell-Network-Connections_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Registry-Persistence-Mechanisms_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Registry-Persistence-Mechanisms_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule RDP-over-Reverse-SSH-Tunnel-WFP_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran RDP-over-Reverse-SSH-Tunnel-WFP_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Addition-of-SID-History-to-Active-Directory-Object_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Addition-of-SID-History-to-Active-Directory-Object_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule UAC-Bypass-via-sdclt_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran UAC-Bypass-via-sdclt_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule WMI-Event-Subscription_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran WMI-Event-Subscription_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Suspicious-File-Characteristics-due-to-Missing-Fields_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Suspicious-File-Characteristics-due-to-Missing-Fields_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Weak-Encryption-Enabled-and-Kerberoast_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Weak-Encryption-Enabled-and-Kerberoast_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule UAC-Bypass-via-Event-Viewer_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran UAC-Bypass-via-Event-Viewer_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Ursnif_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Ursnif_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule NTLM-Logon_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran NTLM-Logon_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Ran Multiple-Failed-Logins-with-Different-Accounts-from-Single-Source-System_1 from 2019-05-17 22:38 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Suspicious-PowerShell-Keywords_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Suspicious-PowerShell-Keywords_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule DHCP-Callout-DLL-installation_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran DHCP-Callout-DLL-installation_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Microsoft-Malware-Protection-Engine-Crash_0 from 2019-05-18 15:23 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Microsoft-Malware-Protection-Engine-Crash_0 from 2019-05-18 15:23 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule DHCP-Server-Loaded-the-CallOut-DLL_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran DHCP-Server-Loaded-the-CallOut-DLL_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Hacktool-Use_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Hacktool-Use_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Sticky-Key-Like-Backdoor-Usage-2_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Sticky-Key-Like-Backdoor-Usage-2_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule PowerShell-Rundll32-Remote-Thread-Creation_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran PowerShell-Rundll32-Remote-Thread-Creation_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Suspicious-Program-Location-with-Network-Connections_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Suspicious-Program-Location-with-Network-Connections_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Kerberos-Manipulation_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Kerberos-Manipulation_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Suspicious-Powershell-commands_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Suspicious-Powershell-commands_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Cobalt-Strike-Named-Pipes_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Windows-Psexec-Powershell-Module_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 / 0 hits
INFO:elastalert:Ran Windows-Psexec-Powershell-Module_0 from 2019-05-18 15:19 UTC to 2019-05-18 15:34 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Sleeping for 58.543264 seconds

Do you think this is an Elasticsearch issue at this point?. This is my updated Elastalert config:
rules_folder: rules
run_every:
    minutes: 1
buffer_time:
    minutes: 15
es_host: helk-elasticsearch
es_port: 9200
alert_time_limit:
    days: 1
scroll_keepalive: 1m
writeback_index: elastalert_status
alert_text: "Index: {0} \nEvent_Timestamp: {1} \nBeat_Name: {2} \nUser_Name: {3} \nHost_Name: {4} \nLog_Name: {5} \nOriginal_Message: \n\n{6}"
alert_text_type: alert_text_only
alert_text_args: ["_index","@timestamp","beat.name","user_name","host_name","log_name","z_original_message"]

This is my latest scroll context info from Elasticsearch:
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "helk-cluster",
  "nodes" : {
    "Y9alb0wgTwO1Dj_8yEJwYw" : {
      "timestamp" :<PHONE_NUMBER>858,
      "name" : "helk-1",
      "transport_address" : "<IP_ADDRESS>:9300",
      "host" : "<IP_ADDRESS>",
      "ip" : "<IP_ADDRESS>:9300",
      "roles" : [
        "master",
        "data",
        "ingest"
      ],
      "attributes" : {
        "ml.machine_memory" : "13549408256",
        "xpack.installed" : "true",
        "ml.max_open_jobs" : "20"
      },
      "indices" : {
        "search" : {
          "open_contexts" : 490,
          "query_total" : 74786,
          "query_time_in_millis" : 3959,
          "query_current" : 0,
          "fetch_total" : 11022,
          "fetch_time_in_millis" : 3548,
          "fetch_current" : 0,
          "scroll_total" : 32160,
          "scroll_time_in_millis" :<PHONE_NUMBER>,
          "scroll_current" : 490,
          "suggest_total" : 0,
          "suggest_time_in_millis" : 0,
          "suggest_current" : 0
        }
      }
    }
  }
}

I started ingesting logs now to my ELK stack in real time and I started getting the error messages regarding scrolling:
INFO:elastalert:Ran Backup-Catalog-Deleted_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule Sticky-Key-Like-Backdoor-Usage_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 / 0 hits
INFO:elastalert:Ran Sticky-Key-Like-Backdoor-Usage_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule User-Added-to-Local-Administrators_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 / 0 hits
INFO:elastalert:Ran User-Added-to-Local-Administrators_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Ran Rare-Schtasks-Creations_0 from 2019-05-17 22:38 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule WMI-Persistence_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 / 0 hits
INFO:elastalert:Ran WMI-Persistence_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule PsExec-Tool-Execution_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 / 0 hits
INFO:elastalert:Ran PsExec-Tool-Execution_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Queried rule DNS-Server-Error-Failed-Loading-the-ServerLevelPluginDLL_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 / 0 hits
INFO:elastalert:Ran DNS-Server-Error-Failed-Loading-the-ServerLevelPluginDLL_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=1m&size=10000 [status:500 request:0.008s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran CMSTP-Execution-2_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-system-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=1m&size=10000 [status:500 request:0.005s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran SAM-Dump-to-AppData_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=1m&size=10000 [status:500 request:0.003s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran Active-Directory-User-Backdoors_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=1m&size=10000 [status:500 request:0.005s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran NetNTLM-Downgrade-Attack-2_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-sysmon-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=1m&size=10000 [status:500 request:0.004s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran Detection-of-SafetyKatz_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-system-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=1m&size=10000 [status:500 request:0.008s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran Eventlog-Cleared_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=1m&size=10000 [status:500 request:0.004s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran Remote-Task-Creation-via-ATSVC-named-pipe_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=1m&size=10000 [status:500 request:0.008s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran Usage-of-Sysinternals-Tools-2_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=1m&size=10000 [status:500 request:0.004s]
ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [500]. This limit can be set by changing the [search.max_open_scroll_context] setting.')
INFO:elastalert:Ran Unauthorized-System-Time-Modification_0 from 2019-05-18 15:27 UTC to 2019-05-18 15:42 UTC: 0 query hits (0 already seen), 0 matches, 0 alerts sent
WARNING:elasticsearch:GET http://helk-elasticsearch:9200/logs-endpoint-winevent-security-*/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=1m&size=10000 [status:500 request:0.004s]

I dont know whats going on from an Elastalert perspective. I will open an issue in elasticsearch too and linked this

This is one of the rules triggering the scroll errors:
alert:
- debug
description: Detects programs with network connections running in suspicious files
  system locations
filter:
- query:
    query_string:
      query: (event_id:"3" AND process_path.keyword:(*\\$Recycle.bin *\\Users\\All\
        Users\\* *\\Users\\Default\\* *\\Users\\Public\\* *\\Users\\Contacts\\* *\\Users\\Searches\\*
        C\:\\Perflogs\\* *\\config\\systemprofile\\* *\\Windows\\Fonts\\* *\\Windows\\IME\\*
        *\\Windows\\addins\\*))
index: logs-endpoint-winevent-sysmon-*
name: Suspicious-Program-Location-with-Network-Connections_0
priority: 2
realert:
  minutes: 0
type: any

I also opened an issue/topic in Elastic Discuss : https://discuss.elastic.co/t/elasticsearch-7-0-1-trying-to-create-too-many-scroll-contexts-must-be-less-than-or-equal-to-500/181687

Thanks for posting all this info. I would definitely recommend against wildcard indexes if you're using time based index names.
The default scroll_keepalive should be 30 seconds: https://github.com/Yelp/elastalert/blob/master/elastalert/config.py#L477 so it's odd that changing it to 1m would fix things.

Same error with ELK stack 7.1.0 and elastalert>=0.2.0b
May 22 12:32:12 elastalert node[7215]: {"name":"elastalert-server","hostname":"elastalert","pid":7215,"level":50,"msg":"ProcessController:  WARNING:elasticsearch:GET http://pbxkibana:9200/pbxes-*/_search?ignore$
May 22 12:32:12 elastalert node[7215]: {"name":"elastalert-server","hostname":"elastalert","pid":7215,"level":50,"msg":"ProcessController:  ERROR:root:Error running query: TransportError(500, u'search_phase_exe$

same error with ELK 7.1.0 and elastalert>=0.2.0b
May 22 12:52:14 elastalert node[7645]: {"name":"elastalert-server","hostname":"elastalert","pid":7645,"level":50,"msg":"ProcessController:  WARNING:elasticsearch:GET http://pbxkibana:9200/pbxes-/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=60s&size=10000 [status:500 request:0.032s]\n","time":"2019-05-22T12:52:14.992Z","v":0}
May 22 12:52:14 elastalert node[7645]: {"name":"elastalert-server","hostname":"elastalert","pid":7645,"level":50,"msg":"ProcessController:  ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [1000]. This limit can be set by changing the [search.max_open_scroll_context] setting.')\n","time":"2019-05-22T12:52:14.993Z","v":0}
May 22 12:53:10 elastalert node[7645]: {"name":"elastalert-server","hostname":"elastalert","pid":7645,"level":50,"msg":"ProcessController:  WARNING:elasticsearch:GET http://pbxkibana:9200/pbxes-/_search?ignore_unavailable=true&_source_includes=%40timestamp%2C%2A&scroll=60s&size=10000 [status:500 request:0.033s]\n","time":"2019-05-22T12:53:10.304Z","v":0}
May 22 12:53:10 elastalert node[7645]: {"name":"elastalert-server","hostname":"elastalert","pid":7645,"level":50,"msg":"ProcessController:  ERROR:root:Error running query: TransportError(500, u'search_phase_execution_exception', u'Trying to create too many scroll contexts. Must be less than or equal to: [1000]. This limit can be set by changing the [search.max_open_scroll_context] setting.')\n","time":"2019-05-22T12:53:10.305Z","v":0}


Node polyfills incorrectly included due to aliasFields

Environment
Using haul bundler 0.14.x.
Description
It appears by adding aliasFields for react-native for #653 it causes other packages to include node dependencies.
Reproducible Demo
The fix is to add additional fallbacks for aliasFields in the defaultConfig for @haul-bundler/preset-0.59:
	aliasFields: ['react-native', 'browser', 'main']

Please upgrade the preset (@haul-bundler/preset-0.59 or @haul-bundler/preset-0.60) to 0.14.2


Run-time dependency Boost found: NO (tried system)

Now able to install properly, any tips would be helpful?
meson --prefix ~/tools/align-clr-to-ccs-0.1.0
The Meson build system
Version: 0.61.1
Source dir: /tools/align-clr-to-ccs-0.1.0
Build dir: /tools/align-clr-to-ccs-0.1.0/build
Build type: native build
Project name: actc
Project version: 0.1.0
C++ compiler for the host machine: c++ (gcc 7.5.0 "c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0")
C++ linker for the host machine: c++ ld.bfd 2.30
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C++ supports arguments -Werror=strict-aliasing: YES
Compiler for C++ supports arguments -Wno-unused-parameter: YES
Compiler for C++ supports arguments -Wno-unused-local-typedef: NO
Run-time dependency threads found: YES
Found pkg-config: /usr/bin/pkg-config (0.29.1)
Run-time dependency Boost found: NO (tried system)
../meson.build:29:0: ERROR: Dependency "boost" not found, tried system
A full log can be found at /tools/align-clr-to-ccs-0.1.0/build/meson-logs/meson-log.txt
Although boost is installed, already :
b2 -h
Invalid option: -h
usage: b2 [ options ] targets...
-a      Build all targets, even if they are current.
-dx     Set the debug level to x (0-13,console,mi).
-fx     Read x instead of bootstrap.
-jx     Run up to x shell commands concurrently.
-lx     Limit actions to x number of seconds after which they are stopped.
-mx     Maximum target output saved (kb), default is to save all output.
-n      Don't actually execute the updating actions.
-ox     Mirror all output to file x.
-px     x=0, pipes action stdout and stderr merged into action output.
-q      Quit quickly as soon as a target fails.
-sx=y   Set variable x=y, overriding environment.
-tx     Rebuild x, even if it is up-to-date.
-v      Print the version of jam and exit.
--x     Option is ignored.

Hi. I'm not supporting build problems. You can find a fully static binary under releases for those who can't build it themselves.


Mistral

Great job, awasome.
Did you plan to use Mistral models, ot it's already possible ?
Thanks

Hello @bacoco! Did you figure out a way to use mistral models? It seems currently s-lora only support llama models. So, if you have figured out a way to load other models, can you share your approach? I am also trying to use this framework for gpt-neox models.

Any success with Mistral?

Adding Mistral would be fantastic!


Refactor add to cart process

I refactored the add to cart process just to follow a bit closer to our typical api post/reducer pattern. Few changes made, but hopefully the process is a bit more straightforward for anyone reading or using the code

There are duplicate orderLists in the store after you add an item to the cart. However, when you go to the cart page, the store is reset and looks accurate again.
We can address this issue in a follow up, but it shouldn't be a problem for now.


packet_inspector: fix UnexpectedEof errors deep inside packet decoding causing perfectly good connections to be aborted. fixes #64

fixes #64
Test plan
start a vanilla mc server
docker run -e EULA=TRUE -e ONLINE_MODE=false -d -p 25565:25565 --name mc itzg/minecraft-server

run packet inspector
cargo r -r -p packet_inspector -- <IP_ADDRESS>:25566 <IP_ADDRESS>:25565

Open mc and connect to localhost:25566
See that you can load into the world and play as normal.

Need to investigate further to make sure this doesn't cause issues elsewhere

I don't think it would affect #54 at all? This is just a change to the packet inspector, not to valence itself. Funnily enough, I can't seem to reproduce that issue with this setup.

Don't think this is needed anymore due to #146. Closing.


Add show-projects command to CommentOps

It would be great to have a digger show-projects command to show all the impacted projects by the PR on the comment, and use that info to run other commands like digger plan -p <project_name> or digger apply -p <project_name> without looking at the digger.yaml file. We currently define a lot of projects in the digger.yaml file and cannot remember the relationship between dir and name of the project.
Something like this
digger.yaml:
projects:
- name: foo_develop
  dir: foo
  workflow: develop
- name: foo_staging
  dir: foo
  workflow: staging
- name: foo_product
  dir: foo
  workflow: product
- name: bar_develop
  dir: bar
  workflow: develop
- name: bar_staging
  dir: bar
  workflow: staging
- name: bar_product
  dir: bar
  workflow: product
- name: baz
  dir: baz
  workflow: default
...

GitHub comment output:
Following projects are impacted by pull request #999
- foo_develop
- foo_staging
- foo_product
- baz

If this feature request is acceptable, I will send the PR do the implementation. (WIP: https://github.com/toVersus/digger-lib-orchestrator/commit/10387e241c7c7ff28f4075b865c4c26229758120, https://github.com/toVersus/digger/commit/bf59142c84267f6b1a42efeeb357347d58d80245)

Hey @toVersus thanks for filing this issue! It makes sense to me to have this command. I think this would be useful also when we are generating projects at runtime or as you mentioned having alot of projects in digger.yml
Please go ahead and create the pull requests for review 👍

Thanks! I submitted the PRs to implement this feature.


Support identifiers with . in them

Which issue does this PR close?
Closes https://github.com/apache/arrow-datafusion/issues/1432
Rationale for this change
Field names containing period such as f.c1 cannot be named in SQL query
What changes are included in this PR?

Directly construct Expr::Column references rather than trying to parse a string using col() when converting SQL

Are there any user-facing changes?
columns with periods can now be used

FYI @liukun4515


Tab Section

ToneHunt should expand its capabilities by incorporating tabs from platforms like MuseScore and Pro Guitar, etc. (or even make its own standard and convert everything to it). Additionally, it should offer a dedicated section for free tabs on the website, allowing users to view and download them. To ensure accuracy, the community should be able to collaboratively edit tabs, similar to how Wikipedia allows page edits with the ability to ban misbehaving accounts. The platform should also provide a playback feature while previewing tabs, and the option to apply IRs and NAM files to ensure accurate sound reproduction (as in the community could make an IR and NAM for each song but you could always change it if you wanted just for yourself) . This combination would allow users to effortlessly download tabs with the appropriate IR and NAM files, making it a remarkable and accessible resource. This would be a game-changer for guitarists everywhere. We would probably need to make a CLAP plugin for bass, guitar, drums, piano, etc,, for the playback so that could be its own issue which I'll make now.

Tab Format and Conversion
Tab Section on website
Editing
Playback
CLAP plugins / Other means for the instruments
Downloading the song with the IRs and NAM as a pack


Bump OpenTracing-api to 0.21.0.

This changes Tag.PEER_PORT from short to int.

Sorry, I think I left the "organize imports" flag on in intellij when committing, so some imports were moved around too.

@sjoerdtalsma LGTM - thanks!


fix documentation for symlink

The target path is actually relative to the link, not to the current
working directory. This is in line with the ln shell command, so changing
the documentation to match the behavior seems in order.
Also changing some wording, and un-highlighting "symlink" throughout as
it never refers to the function.

Auto-merge toggled on


OpenAPIV3Parser setting null instead of parsing default enum values

Hi,
I've discovered what seems to be an issue when OpenAPIV3Parser parses specifications containing default values for enum types.
When io.swagger.v3.parser.OpenAPIV3Parser#read(java.lang.String) parses default values for enums, the returned OpenAPI model schemas contain null instead of default values. This applies for both defaults set in enums themselves or inside schemas referencing the enums. The same specification that exposes this issue is valid and properly shows defaults inside Swagger Editor. This seems to be in line with Schema Object in the OpenAPI specification.
I'm providing a test class with unit tests that confirm my claims alongside an example OpenAPI specification that is used in tests and can be run inside Swagger Editor. The tests should be easy to add to your existing OpenAPIV3ParserTest. This is tested using swagger-parser versions 2.0.14 and 2.0.22.
Can you please look into this and see if my findings are correct?
Test class
The class contains various examples of setting default values for enums that do or do not set the default values themselves. Assertions are based on behavior in Swagger Editor. Ignored tests are the ones exposing the issue.
package io.swagger.v3.parser.test;

import io.swagger.v3.oas.models.OpenAPI;
import io.swagger.v3.parser.OpenAPIV3Parser;
import org.junit.*;

import java.io.File;

import static org.assertj.core.api.BDDAssertions.then;

public class OpenAPIV3ParserTest {

    private static final String GIVEN_SPEC_PATH = "src/test/resources/openapis/openapi-default-enum-bug.json";
    private static final String GIVEN_LOCATION = new File(GIVEN_SPEC_PATH).getAbsolutePath();

    private OpenAPIV3Parser openAPIV3Parser;

    @Before
    public void setUp() {
        openAPIV3Parser = new OpenAPIV3Parser();
    }

    @Test
    public void shouldParseDefaultForEnumSchema() {
        // when
        OpenAPI openAPI = openAPIV3Parser.read(GIVEN_LOCATION);

        // then
        then(openAPI.getComponents()
                .getSchemas()
                .get("Enum")
                .getDefault()
        ).isNull();
    }

    @Test
    public void shouldParseDefaultForEnumWithDefaultSchema() {
        // when
        OpenAPI openAPI = openAPIV3Parser.read(GIVEN_LOCATION);

        // then
        then(openAPI.getComponents()
                .getSchemas()
                .get("EnumWithDefault")
                .getDefault()
        ).isEqualTo("ENUM_DEFAULT");
    }

    @Test
    public void shouldParseDefaultForSchemaWithEnum() {
        // when
        OpenAPI openAPI = openAPIV3Parser.read(GIVEN_LOCATION);

        // then
        then(openAPI.getComponents()
                .getSchemas()
                .get("SchemaWithEnum")
                .getDefault()
        ).isNull();
    }

    @Ignore("This test fails due to OpenAPIV3Parser bug.")
    @Test
    public void shouldParseDefaultForSchemaWithDefaultAndEnum() {
        // when
        OpenAPI openAPI = openAPIV3Parser.read(GIVEN_LOCATION);

        // then
        then(openAPI.getComponents()
                .getSchemas()
                .get("SchemaWithDefaultAndEnum")
                .getDefault()
        ).isEqualTo("SCHEMA_DEFAULT");
    }

    @Ignore("This test fails due to OpenAPIV3Parser bug.")
    @Test
    public void shouldParseDefaultForSchemaWithEnumWithDefault() {
        // when
        OpenAPI openAPI = openAPIV3Parser.read(GIVEN_LOCATION);

        // then
        then(openAPI.getComponents()
                .getSchemas()
                .get("SchemaWithEnumWithDefault")
                .getDefault()
        ).isEqualTo("ENUM_DEFAULT");
    }

    @Ignore("This test fails due to OpenAPIV3Parser bug.")
    @Test
    public void shouldParseDefaultForSchemaWithDefaultAndEnumWithDefault() {
        // when
        OpenAPI openAPI = openAPIV3Parser.read(GIVEN_LOCATION);

        // then
        then(openAPI.getComponents()
                .getSchemas()
                .get("SchemaWithDefaultAndEnumWithDefault")
                .getDefault()
        ).isEqualTo("SCHEMA_DEFAULT");
    }

}

Example specification
This specification is used in the test above and is named openapi-default-enum-bug.json for testing purposes.
{
  "openapi": "3.0.1",
  "info": {
    "title": "Here be enum bugs",
    "description": "OpenAPI spec to test the enum parsing bugs.",
    "version": "1.0.0"
  },
  "paths": {
    "/enum/parse/bug": {
      "post": {
        "summary": "Test enum parse bug",
        "requestBody": {
          "content": {
            "application/json": {
              "schema": {
                "$ref": "#/components/schemas/TestRequest"
              }
            }
          }
        },
        "responses": {
          "default": {
            "description": "Successful response"
          }
        }
      }
    }
  },
  "components": {
    "schemas": {
      "TestRequest": {
        "type": "object",
        "description": "This request body schema depicts various enum and default value combinations inside schemas of the following properties.",
        "properties": {
          "schemaWithEnum": {
            "$ref": "#/components/schemas/SchemaWithEnum"
          },
          "schemaWithDefaultAndEnum": {
            "$ref": "#/components/schemas/SchemaWithDefaultAndEnum"
          },
          "schemaWithEnumWithDefault": {
            "$ref": "#/components/schemas/SchemaWithEnumWithDefault"
          },
          "schemaWithDefaultAndEnumWithDefault": {
            "$ref": "#/components/schemas/SchemaWithDefaultAndEnumWithDefault"
          }
        }
      },
      "SchemaWithEnum": {
        "type": "object",
        "properties": {
          "enumProperty": {
            "description": "This schema with no default values set behaves properly in Swagger Editor and when parsed with OpenAPIV3Parser.",
            "allOf": [
              {
                "$ref": "#/components/schemas/Enum"
              }
            ]
          }
        }
      },
      "SchemaWithDefaultAndEnum": {
        "type": "object",
        "properties": {
          "enumProperty": {
            "description": "This schema with set default enum value and enum ref shows default \"SCHEMA_DEFAULT\" value in Swagger Editor, but is null when parsed with OpenAPIV3Parser.",
            "allOf": [
              {
                "$ref": "#/components/schemas/Enum"
              }
            ],
            "default": "SCHEMA_DEFAULT"
          }
        }
      },
      "SchemaWithEnumWithDefault": {
        "type": "object",
        "properties": {
          "enumProperty": {
            "description": "This schema with ref to enum with set default value shows default \"ENUM_DEFAULT\" value in Swagger Editor, but is null when parsed with OpenAPIV3Parser.",
            "allOf": [
              {
                "$ref": "#/components/schemas/EnumWithDefault"
              }
            ]
          }
        }
      },
      "SchemaWithDefaultAndEnumWithDefault": {
        "type": "object",
        "properties": {
          "enumProperty": {
            "description": "This schema with set default enum value and ref to enum with set default value shows default \"SCHEMA_DEFAULT\" value in Swagger Editor, but is null when parsed with OpenAPIV3Parser.",
            "allOf": [
              {
                "$ref": "#/components/schemas/EnumWithDefault"
              }
            ],
            "default": "SCHEMA_DEFAULT"
          }
        }
      },
      "Enum": {
        "type": "string",
        "enum": [
          "SCHEMA_DEFAULT",
          "NOT_DEFAULT"
        ]
      },
      "EnumWithDefault": {
        "type": "string",
        "enum": [
          "SCHEMA_DEFAULT",
          "ENUM_DEFAULT",
          "NOT_DEFAULT"
        ],
        "default": "ENUM_DEFAULT"
      }
    }
  }
}

could you check with latest changes. one part of the issue is fixed. default values are set for enums.
EnumWithDefault has its default value set.

@r-sreesaran unfortunately, tests provided by @hkozacinski are still failing with swagger-parser version 2.0.25.
I've created a little maven project that you can use to easily re-run the tests: swagger-parser-issue-1454. Hope this helps!

@r-sreesaran the problem still exists.
As far as I can tell, it comes from io.swagger.v3.parser.util.OpenAPIDeserializer.getSchema(ObjectNode node, String location, ParseResult result) .
After line if (node.get("default") != null), there is a tentative to get the actual default value, based on the type of the node. The type is null because of the allOf. So the default value is simply ignored.
On top of that, at the time the default value is ignored, the Enum schema has not been parsed yet, so we don't even know the type of the thing.
A fix would probably require to remember those undecidable cases, then try to decide after the whole spec has been parsed.

@r-sreesaran  No sorry: the cause of the problem was in ResolverFully.java
I will issue a pull request.

@tmeckel my first comment was the result of the inspection of the source code of 2.0.x. Sorry.
Version 2.1.2 is working much better in this regard, and I think the little fix in ResolverFully is just the icing on the cake.

I've just added all the tests and made an adjustement to ResolverFully so it sets a default value when all combined models have the same default value.
It is almost completely fixed now.
I'm saying "almost" because I'm not sure the SchemaWithEnumWithDefault case is supposed to be working when resolve=false and aggregateCombinators=false. I don't think so (and it isn't working). In my opinion that's the purpose of the resolve=true and aggregateCombinators=true, and it's working with those parameters as shown by new unit tests.


Add support for simple let bindings

Issue
Closes #80
Description of the Change
Add support for simple let bindings of the structure:
 let x1 = e1
      . . .
      xn = en
      in e

where x1 .... xn are variable pattern and e1 ... en are expressions.

I marked this pull as "ready to review". However, there is still a bug that i cannot identify.
The scenario is as simple as it can get.
let b = True in b
The occurring error is "Identifier not in scope: b"


Flash Attention

Adapts the code @VHellendoorn wrote for GPT-Neox to add Flash Attention to NeMo.
Currently has @Quentin-Anthony 's TFLOPS logging code added in. Not clear that the FLOPS or parameters are being calculated precisely correctly, but it should be good enough for a comparison. Here is a sample of the logs with FlashAttention:
Epoch 0:   0%|   | 190/150750 [01:46<23:23:17,  1.79it/s, loss=7.08, v_num=38, TFLOPS_per_gpu=249.0, reduced_train_loss=7.050, global_step=139.0, consumed_samples=8896.0, val_loss=7.360]
Epoch 0:   0%|          | 191/150750 [01:46<23:24:11,1.79it/s, loss=7.08, v_num=38, TFLOPS_per_gpu=249.0, reduced_train_loss=7.050, global_step=139.0, consumed_samples=8896.0, val_loss=7.360]
Epoch 0:   0%|          | 191/150750 [01:46<23:24:11,  1.79it/s, loss=7.07, v_num=38, TFLOPS_per_gpu=248.0, reduced_train_loss=7.020, global_step=140.0, consumed_samples=8960.0, val_loss=7.360]
Epoch 0:   0%|          | 192/150750 [01:47<23:25:04,  1.79it/s, loss=7.07, v_num=38, TFLOPS_per_gpu=248.0, reduced_train_loss=7.020, global_step=140.0, consumed_samples=8960.0, val_loss=7.360]
Epoch 0:   0%|  | 192/150750 [01:47<23:25:04,  1.79it/s, loss=7.06, v_num=38, TFLOPS_per_gpu=248.0, reduced_train_loss=7.070, global_step=141.0, consumed_samples=9024.0, val_loss=7.360]
Epoch 0:   0%| 0: | 193/150750 [01:48<23:25:56,  1.78it/s, loss=7.06, v_num=38, TFLOPS_per_gpu=248.0, reduced_train_loss=7.070, global_step=141.0, consumed_samples=9024.0, val_loss=7.360]
Epoch 0:   0%|          | 193/150750 [01:48<23:25:56,  1.78it/s, loss=7.05, v_num=38, TFLOPS_per_gpu=248.0,
reduced_train_loss=6.800, global_step=142.0, consumed_samples=9088.0, val_loss=7.360]

And here is a sample with NeMo's standard CoreAttention:
Epoch 0:   0%|          | 188/150750 [01:53<25:19:06,  1.65it/s, loss=7.11, v_num=39, TFLOPS_per_gpu=222.0, reduced_train_loss=7.120, global_step=136.0, consumed_samples=8704.0, val_loss=7.340]
Epoch 0:   0%|          | 188/150750 [01:53<25:19:07,  1.65it/s, loss=7.09, v_num=39, TFLOPS_per_gpu=223.0, reduced_train_loss=6.730, global_step=137.0, consumed_samples=8768.0, val_loss=7.340]
Epoch 0:   0%|          | 189/150750 [01:54<25:20:07,  1.65it/s, loss=7.09, v_num=39, TFLOPS_per_gpu=223.0, reduced_train_loss=6.730, global_step=137.0, consumed_samples=8768.0, val_loss=7.340]
Epoch 0:   0%|          | 189/150750 [01:54<25:20:07,  1.65it/s, loss=7.08, v_num=39, TFLOPS_per_gpu=222.0, reduced_train_loss=6.910, global_step=138.0, consumed_samples=8832.0, val_loss=7.340]
Epoch 0:   0%|          | 190/150750 [01:55<25:21:08,  1.65it/s, loss=7.08, v_num=39, TFLOPS_per_gpu=222.0, reduced_train_loss=6.910, global_step=138.0, consumed_samples=8832.0, val_loss=7.340]
Epoch 0:   0%|          | 190/150750 [01:55<25:21:08,  1.65it/s, loss=7.07, v_num=39, TFLOPS_per_gpu=
221.0, reduced_train_loss=7.030, global_step=139.0, consumed_samples=8896.0, val_loss=7.340]
Epoch 0:   0%|          | 191/150750 [01:55<25:22:08,  1.65it/s, loss=7.07, v_num=39, TFLOPS_per_gpu=221.0, reduced_train_loss=7.030, global_step=139.0, consumed_samples=8896.0, val_loss=7.340]
Epoch 0:   0%|          | 191/150750 [01:55<25:22:08,  1.65it/s, loss=7.06, v_num=39, TFLOPS_per_gpu=222.0, reduced_train_loss=7.020, global_step=140.0, consumed_samples=8960.0, val_loss=7.340]
Epoch 0:   0%|          | 192/150750 [01:56<25:23:08,  1.65it/s, loss=7.06, v_num=39, TFLOPS_per_gpu=222.0, reduced_train_loss=7.020, global_step=140.0, consumed_samples=8960.0, val_loss=7.340]

merging now


fixed formatting

What's changed

Fixed formatting in Docker for Mac FAQ's per feedback issue referenced below.

Related issues
#2209
Reviewers, fyi
@hairyhenderson
Signed-off-by: Victoria Bialas<EMAIL_ADDRESS>

LGTM


Can't set headers after they are sent.

I use your lib in production and I noticed that when there's many USSD requests, they will end up without being resolved throwing this error
app[web.1]: Error: Can't set headers after they are sent.
app[web.1]:     at validateHeader (_http_outgoing.js:491:11)
app[web.1]:     at ServerResponse.setHeader (_http_outgoing.js:498:3)
app[web.1]:     at ServerResponse.header (/app/node_modules/express/lib/response.js:767:10)
app[web.1]:     at ServerResponse.send (/app/node_modules/express/lib/response.js:170:12)
app[web.1]:     at UssdMenu.menu.run.ussdResult [as onResult] (/app/src/ussd/index.ts:33:11)
app[web.1]:     at UssdMenu.callOnResult (/app/node_modules/ussd-menu-builder/lib/ussd-menu.js:20:18)
app[web.1]:     at UssdMenu.con (/app/node_modules/ussd-menu-builder/lib/ussd-menu.js:26:14)
app[web.1]:     at menuOps (/app/src/ussd/registration.ts:222:17)
app[web.1]:     at <anonymous>
app[web.1]: (node:27) UnhandledPromiseRejectionWarning: Error: Can't set headers after they are sent.
app[web.1]:     at validateHeader (_http_outgoing.js:491:11)
app[web.1]:     at ServerResponse.setHeader (_http_outgoing.js:498:3)
app[web.1]:     at ServerResponse.header (/app/node_modules/express/lib/response.js:767:10)
app[web.1]:     at ServerResponse.send (/app/node_modules/express/lib/response.js:170:12)
app[web.1]:     at UssdMenu.menu.run.ussdResult [as onResult] (/app/src/ussd/index.ts:33:11)
app[web.1]:     at UssdMenu.callOnResult (/app/node_modules/ussd-menu-builder/lib/ussd-menu.js:20:18)
app[web.1]:     at UssdMenu.end (/app/node_modules/ussd-menu-builder/lib/ussd-menu.js:31:14)
app[web.1]:     at endMenu (/app/src/ussd/registration.ts:232:8)
app[web.1]:     at menuOps (/app/src/ussd/registration.ts:226:5)
app[web.1]:     at <anonymous>

It's the same request resolved twice or more. I'm working on a fix but I have so far hit a dead end.

@musale thanks for opening this and sorry for the late reply, I've just seen this.
Could you show the snippet of code you're using to register the ussd handler?

@musale I think I have an idea of what could be causing that error:
The library is most likely used by creating one instance of the UssdMenu and using this single instance to define states. And each ussd request triggers a call to the menu.run(args, resultHandler) method.
However there's a flaw in the design: while the menu.run() is called separately for each request and possibly with a different result handler each time, the result handler is actually attached to the menu instance as menu.onResult object. So A new request will replace this function with the new handler. So if one request takes time to complete, its handler might be replaced by that of a more recent request, so when it'll trigger the wrong handler when it's complete. And if it triggers a handler that has already been called, it will cause res.send to be called twice, which would cause the error you're referring to.

@musale I have an idea for a fix. Instead of using menu.con, and menu.end and an onResult object that's attached to the instance, I'll add con and end functions to the args passed to the state.run function and ensure that they call the correct handler.
I will also deprecate menu.con, menu.end and menu.onResult, I could keep them for backwards compatibility but add a warning against their usage.
I will try to work on an update this coming week.

@musale until I have an update ready, I could propose the following temporary fix:
Create a new menu instance for each request, that way you avoid the issue of the handler that gets replaced.
Instead of:
const menu = new UssdMenu();
// your states
menu.startState(...);
menu.state(...);

// register handler
app.post('/ussd', (req, res) => {
    menu.run(req.body, msg => res.send(msg);
}

Try this:
function createMenu () {
  const menu = new UssdMenu();
  // your states
  menu.startState(...);
  menu.state(...);

  return menu;
}

// register handler
app.post('/ussd', (req, res) => {
    // creates new instance each time
    const menu = createMenu();
    menu.run(req.body, msg => res.send(msg);
}

@musale if you have a better suggestion for a fix, let me know. Also, if you see a potential problem or flaw with my proposed update, let me know as well.
Thanks for pointing this out.

@habbes thank you for the follow-up. I haven't had time to try out the suggestions as I've been very ill. I was declaring a global menu instance and as you've explained, that was the cause of the requests being terminated. I will try creating the menu for each instance and then give you the feedback for it. However, this might not be soon. I was looking for a fix but like I mentioned, I got sidetracked. I will update you if anything works(or not hehe) thank you!

@musale I've been away for a while and have been bogged down by work. I will try to get back to this issue this coming week or the next.

@Marcosmaliki you can have a go at it. I no longer work on the project that required this dependency, I doubt I will pursue it any further 😺


Add displayName

The display name is not currently configured. This PR adds displayName to package.json to make the name shown in the HomeBridge UI prettier.

Hi @davidjbradshaw thanks for taking the time to make a PR.
In fact, I like the name of the plugin as Homebridge Govee in the UI.
Recent changes to the UI have made this shorter in mobile devices so in this case the plugin will display as just Govee.
So in this case I will close this PR without merge, but again, thanks for the thought and time!


OSX Sonoma 14.7.1 Giving "Operation not permitted" when trying to update firmware for Advantage 360 Pro

I'm on an old version of the firmware, trying to pull in the latest factory default.  My system is a pretty vanilla osx machine,
# sw_vers
ProductName:		macOS
ProductVersion:		14.7.1
BuildVersion:		23H222

and my keyboard has (what I believe to be) the factory defaults:
# cat /Volumes/ADV360PRO/INFO_UF2.TXT
UF2 Bootloader 0.4.0-41-g103bd32-dirty lib/nrfx (v2.0.0) lib/tinyusb (0.9.0-22-g7cdeed54) lib/uf2 (remotes/origin/configupdate-9-gadbb8c7)
Model: ADV360PRO
Board-ID: nRF52840-ADV360PRO-v1
SoftDevice: S140 version 6.1.1
Date: Jun 29 2022

When I try to update the firmware per the instructions, to the new factory defaults
# cp left.uf2 /Volumes/ADV360PRO/
cp: /Volumes/ADV360PRO/left.uf2: Operation not permitted

And a similar error appears when attempting to copy in the UI.
I'm guessing there's some "how to get to v2" step I'm missing, as most of the documentation I can find talks about how to get from v2 to v3.  Is there some extra step I am missing?

Nevermind - switching to a different machine (with the same setup) seems to be the trick.


mobile: open link in specific app

Not sure if possible but will improve mobile UX
Subwallet

mobile.subwallet.app

Nova

no

@roiLeo any link I can test with when attempting this?

nvm


unable to set widths of scrollbars individually

Expected behavior and actual behavior.
Can't find any way to individually set the height of the scrollbars independently.
setting the width property as you suggested does so for both scrollbars
possibly  adding additional public properties to override this logic?
self.scrollBox.horizontalBarVisible = self.style.width !== 'auto' && dataWidth > self.scrollBox.width;            self.scrollBox.verticalBarVisible = self.style.height !== 'auto' && dataHeight > self.scrollBox.height;
Steps to reproduce the problem.
try to set scrollbar width to 0
Specifications like the version of the project, operating system, or hardware.

It's easy enough to add the extra styles, I'll stick it in the next release.

I didn't put it in the next release :( I am pond scum.  Next one for sure!

understand believe me.
i know how it goes
On Mon, Jul 23, 2018 at 5:47 PM, Tony Germaneri<EMAIL_ADDRESS>wrote:

I didn't put it in the next release :( I am pond scum. Next one for sure!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/TonyGermaneri/canvas-datagrid/issues/148#issuecomment-407224882,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFzyln-AiprA9KxlXwrPwuruji4CKGxXks5uJlJkgaJpZM4VXW5E
.

I've added overflow-x and overflow-y properties on the style object.  They behave as you might expect.  I'll have a release later today.

If these overflow settings are not fixing your problem please reopen the ticket.

doesn't seem to be working quite right
http://jsfiddle.net/qr1cythe/241/
I swear it has removed the horizontal scroll bar once or twice but re-rendering the screen brings it back.
Could be issues with the JS Fiddle site I don't know

http://jsfiddle.net/cwevk79g/5/  example also does not work

Order matters..whether it should or not is another question
grid.style.overflowX = "hidden"
grid.setColumnWidth(0, 60);
grid.setColumnWidth(1, 200);
grid.setColumnWidth(2, 200);
grid.style.width = "316px";
grid.style.height ="200px"
works
grid.setColumnWidth(0, 60);
grid.setColumnWidth(1, 200);
grid.setColumnWidth(2, 200);
grid.style.width = "316px";
grid.style.height ="200px"
grid.style.overflowX = "hidden"
does not
On Thu, Sep 6, 2018 at 1:35 PM Tony Germaneri<EMAIL_ADDRESS>wrote:

If these overflow settings are not fixing your problem please reopen the
ticket.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/TonyGermaneri/canvas-datagrid/issues/148#issuecomment-419198006,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFzylsqgx6v5Ud97KXwjx8fsNjrn8uiJks5uYWr_gaJpZM4VXW5E
.