Constant Update Notification

Hi all, I ran Driveshare a couple of times and it hasnt said there was an update. Now it does, I download the 3.4 version, but it seems its the same as the one I have already, and after installing, it still says theres an update available

Hi friend, looks like this is because we are using the version tracked at the master branch to notify you that there is an update and master is currently tracking v0.4.1. Releases have not yet been packaged for all platforms, which why you downloaded the version you already had! :smile:
The Updater class should be changed to look at remote tags instead of what's in master to avoid this in the future. I'll fix this.

Retest passed. I get only a message if a new release is published. Fix will come with version 0.4.2.

OK ty
On Dec 11, 2015 7:26 PM, "littleskunk"<EMAIL_ADDRESS>wrote:

Retest passed. I get only a message if a new release is published. Fix
will come with version 0.4.2.
—
Reply to this email directly or view it on GitHub
https://github.com/Storj/driveshare-gui/issues/90#issuecomment-164093718
.


fix 'empty request scheme' (close #522)

Codecov Report

Merging #529 (0a5b3b5) into main (72065c9) will decrease coverage by 0.15%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #529      +/-   ##
==========================================
- Coverage   80.59%   80.44%   -0.16%
==========================================
  Files          87       87
  Lines       10103    10113      +10
==========================================
- Hits         8143     8135       -8
- Misses       1510     1531      +21
+ Partials      450      447       -3

Impacted Files
Coverage Δ

pkg/object/mqttproxy/session.go
74.23% <0.00%> (-7.98%)
:arrow_down:

pkg/cluster/mutex.go
79.31% <0.00%> (-4.91%)
:arrow_down:

pkg/cluster/syncer.go
76.28% <0.00%> (-1.29%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 72065c9...0a5b3b5. Read the comment docs.


Keep double underscores on index names

Summary
Index names are composed of index.table name, index.columns names and
the "index" suffix. If index.table or index.columns names cointained two
underscores, they were replaced by a single underscore. This caused
issues when applying unique index constraints on changesets.
Additional info
Fixes #2311

:heart: :green_heart: :blue_heart: :yellow_heart: :purple_heart:


[Feature] Control the zoom/drag-to-zoom speed on large data

What problem does this feature solve?
Hi,
I wanna ask if there any way to control the speed/sensitivity of the zoom in/out function, because for the large data, its quite hard to zoom to the specific location in the chart.
Thanks
What does the proposed API look like?
Can we have a props in dataZoom like sensitivity/ speed in order to have more control on zoom function.
For example, adding the speed and sensitivity props like chartjs-plugin-zoom did:
zoom: { enabled: true, drag: false, mode: "x", speed: 0.1, sensitivity: 0.01 },

Hi,
I am also interested to know if there is any way to control the speed/sensitivity when zooming in and out of the graph (in my case, a scatter plot graph). I tried to increase the throttle value to make it look slower, but it does not have the smooth transition/animation that I am looking for (patchy imo).
Would be awesome if there is another solution, or a new feature to be considered in the future. :)

Hi guys,
i also landed here in search for a zoom speed option. Any news here?

Also looking for a solution

don't know if the throttle parameter could work.
    dataZoom: [
      {
        type: 'inside',
        throttle: 100,
      }
    ]

https://echarts.apache.org/en/option.html#dataZoom-inside.throttle

Would love this.

+1.  Zooming on a time series chart is currently sensitive, making it difficult to control via mouse interaction.
@bbeas - Throttle seems to control the animation speed, but from my local testing doesn't seem to impact the zoom sensitivity.


EMR Serverless Adding Option to Boto3 for Glue Catlog

AWS has announced AWS EMR CLI
https://aws.amazon.com/blogs/big-data/build-deploy-and-run-spark-jobs-on-amazon-emr-with-the-open-source-emr-cli-tool/

I have tried and CLi works great simplifies submitting jobs

However, could you tell us how to enable the Glue Hive meta store when submitting a job via CLI or in Boto3
i have looked at documentation i don't see an argument for supplying use Glue CatLog option on boto3

Here is a sample of how we are submitting jobs qith EMR-CLI
emr run     --entry-point entrypoint.py
    --application-id      --job-role <arn>
    --s3-code-uri s3:///emr_scripts/     --spark-submit-opts "--conf spark.jars=/usr/lib/hudi/hudi-spark-bundle.jar --conf spark.serializer=org.apache.spark.serializer.KryoSerializer"
    --build `
    --wait

Created A Github Issue https://github.com/awslabs/amazon-emr-cli/issues/18
If you can kindly get back to us on issue that would be great 😃

I've been able to get this to work by specifying in sparkSubmitParameters. For example:
job_run = client.start_job_run(
    applicationId=application_id,
    executionRoleArn=job_role_arn,
    jobDriver={
        'sparkSubmit': {
            'entryPoint': 's3://bucket/your_script.py',
            'entryPointArguments': [],
            'sparkSubmitParameters': '--conf spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory'
        },
    }
)


Error: Server at <IP_ADDRESS>:11211 not available

LOG
Error: Server at <IP_ADDRESS>:11211 not available
at Client.memcachedCommand [as command] (/home/hello.com/node_modules/memcached/lib/memcached.js:306:70)
at Client.get (/home/hello.com/node_modules/memcached/lib/memcached.js:830:10)

But Memcached active.
# service memcached status
● memcached.service - memcached daemon
   Loaded: loaded (/lib/systemd/system/memcached.service; disabled)
   Active: active (running) since Thu 2017-03-23 08:35:35 EDT; 10s ago
 Main PID: 1538 (memcached)
   CGroup: /system.slice/memcached.service
           └─1538 /usr/bin/memcached -m 64 -p 11211 -u memcache -l <IP_ADDRESS>

Mar 23 08:35:35 trampamp systemd[1]: Starting memcached daemon...
Mar 23 08:35:35 trampamp systemd[1]: Started memcached daemon.

# telnet <IP_ADDRESS> 11211
Trying <IP_ADDRESS>...
Connected to <IP_ADDRESS>.
Escape character is '^]'.
stats
STAT pid 1538
STAT uptime 876
STAT time<PHONE_NUMBER>
STAT version 1.4.21
STAT libevent 2.0.21-stable
STAT pointer_size 64
STAT rusage_user 0.025996
STAT rusage_system 0.006998
STAT curr_connections 5
STAT total_connections 8
STAT connection_structures 6
STAT reserved_fds 20
STAT cmd_get 0
STAT cmd_set 0
STAT cmd_flush 2
STAT cmd_touch 0
STAT get_hits 0
STAT get_misses 0
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 46
STAT bytes_written 15
STAT limit_maxbytes 67108864
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT hash_power_level 16
STAT hash_bytes 524288
STAT hash_is_expanding 0
STAT malloc_fails 0
STAT bytes 0
STAT curr_items 0
STAT total_items 0
STAT expired_unfetched 0
STAT evicted_unfetched 0
STAT evictions 0
STAT reclaimed 0
STAT crawler_reclaimed 0
STAT lrutail_reflocked 0
END

Is project dead?

@4h0y Did you solve this problem? I have the same issue.

@zhenfan0753 I used a different port. For example 11212

Looks like I got it!
this is how my memcached is started
memcached -p 11212 -m 256 -c 1024 -I 20m

and this was the client creation
new Memcached(
    options.host + ':' + options.port, {
        maxValue: 33554432
    }
);

when i corrected maxValue to the value <= 20Mb, for example 10485760
in logs instead of those unexpected errors i sometimes see for too big items
Error: The length of the value is greater than 10485760

which is perfectly fine and all other write operations succeed

Any updates on this ?

Can anyone help me out over this problem ?

any updates on this ???!

any updates on this ???!

Any updates?

@WilliamNurmi I get the same "Connection readyState is set to readOnly" during heavy loads.
Were you able to address this?

Have you solved this problem now？I also encountered it

Initialising the client like below fixed my problem:
new MemcachedClient(this.serverEndpoint, { maxValue: 10485760, timeout: 10 * 1000 /* 10 sec. */});

I think I managed to consistently reproduce Error: Connection readyState is set to readOnly locally.
Basically, you need memcached server running + stress test it with parallel writes.
For me it starts to break at 40 parallel writes.
Considering that nothing else uses this memcached server & I have only one connection instance, I think its a library fault that it doesn't try to queue writes.. nor limit how many writes can do at a time & instead seems to just kill memcache.
Steps:
const MIN_FAILING_THREADS = 100;
it(`native memcache fails at ${MIN_FAILING_THREADS} concurrent writes`, async () => {
	const Memcached = require('memcached');

	// ACT
	const ops = [];

	const connection = new Memcached(['localhost:11211'], {
		// timeout: 500, the time after which Memcached sends a connection timeout (in milliseconds)
		timeout: 500,
		// retries: 5, the number of socket allocation retries per request.
		retries: 2,
		// failures: 5, the number of failed-attempts to a server before it is regarded as 'dead'.
		failures: 5,
		// retry: 30000, the time between a server failure and an attempt to set it up back in service.
		// the maximum size of the connection pool.
		poolSize: 200,
		// if true, authorizes the automatic removal of dead servers from the pool.
		remove: true,
		idle: 5000,
		maxValue: 524288, // default is 1048576
		encoding: 'binary',
	});

	connection.on('failure', async () => {
		console.error('Lost connection');
	});

	for (let i = 0; i < MIN_FAILING_THREADS; i++) {
		ops.push(
			new Promise((resolve, reject) => {
				connection.set(`x${i}`, i, 2, (err, result) => {
					if (err) {
						return reject(err);
					}

					return resolve(result);
				});
			}),
		);
	}

	// overload memcache
	Promise.all(ops);

	const result = await new Promise((resolve, reject) =>
		connection.getMulti(['x1'], (err, data) => {
			if (err) {
				reject(err);
			}

			resolve(data);
		}),
	);

	// ASSERT
        expect(result).toEqual({ x1: 1 });
});

Install memcached in window works for my case.
https://stackoverflow.com/questions/59476616/install-memcached-on-windows

Can somebody explain how to reproduce this? I followed the jest code, all it tells is a ERROR response was received


TypeError: Path must be a string. Received -1

Hi,
The above error is thown when linting. Here is my config
gulp-tslint version: 8.1.1
tslint version: 5.4.3
Operating system: Windows 8.1, 64Bits
Example gulp configuration (if applicable):
gulp.task('lint', (cb) => {
  pump([
    gulp.src(config.allTs),
    gulpTslint({ formatter: "verbose" }),
    gulpTslint.report()
  ], cb);
});

Error console output:
TypeError: Path must be a string. Received -1
    at assertPath (path.js:7:11)
    at Object.join (path.js:468:7)
    at findFile (H:\my-project\node_modules\tslint\lib\configuration.js:137:98)
    at findup (H:\my-project\node_modules\tslint\lib\configuration.js:118:19)
    at findConfigurationPath (H:\my-project\node_modules\tslint\lib\configuration.js:95:30)
    at Object.findConfiguration (H:\my-project\node_modules\tslint\lib\configuration.js:48:22)
    at H:\my-project\node_modules\gulp-tslint\index.js:101:64
    at wrappedMapper (H:\my-project\node_modules\map-stream\index.js:83:19)
    at Stream.stream.write (H:\my-project\node_modules\map-stream\index.js:95:21)
    at write (H:\my-project\node_modules\vinyl-fs\node_modules\readable-stream\lib\_stream_readable.js:623:24)

TypeScript example code (if applicable):

What is config.allTs?

Hi @panuhorsmalahti
config.allTs is a glob pattern to identify the *.ts files to lint, it contains 'src/**/!(*.spec).ts'.
A sample project can be found here: https://github.com/pratikthecook/my-sample-lib
If you upgrade the packages gulp-tslint version to  8.1.1 and tslint  to 5.4.3, you should have the above mentioned error when linting (via gulp lint )...
Tested on node 7

I get no error from the following:
glit clone https://github.com/pratikthecook/my-sample-lib.git
ncu -u
npm i
gulp lint

I'm using Mac OS X and 8.0.0. This problems appears to be Windows specific, maybe even Windows 8. I'll try Windows 10 later.
(Note: ncu is the npm-check-updates tool).


Clue Compass hotkeys not working

Some Clue Compass hotkeys are not working. For example Y, P doesn't work, but O does.

I don't know if this issue is present with other menus.

I think you have a hidden disabled entry for P that is taking precedence

fixed in 7c4d3dfa0d1f24cfc4b19933ff93e145d5006669 by not matching disabled entries. I've pushed this as 1.5.5, which should be live in a few minutes.


Review trips: species_counted

trips: species_counted indicates what species are counted with an integer code:
https://github.com/ices-tools-dev/esas/blob/0d294b4ea678e34f52409a16602f9133ac179dcf/_data/table-schemas/trips.yaml#L235-L255

[ ] @ices-tools-dev/data-and-information Does ICES already have a controlled vocabulary for this?
[x] @nicolasvanermen Wouldn't human readable values like all, all_not_larus be better, or are the integer codes well established?

I would hold on to the integer codes, species_counted often gets drawn-out
descriptions.
Nicolas Vanermen
Wetenschappelijk medewerker
Instituut voor Natuur- en Bosonderzoek
Havenlaan 88, bus 73, 1000 Brussel
0486/361.582
On Wed, Jan 6, 2021 at 12:31 PM Peter Desmet<EMAIL_ADDRESS>wrote:

trips: species_counted indicates what species are counted with an integer
code:
https://github.com/ices-tools-dev/esas/blob/0d294b4ea678e34f52409a16602f9133ac179dcf/_data/table-schemas/trips.yaml#L235-L255

@ices-tools-dev/data-and-information
https://github.com/orgs/ices-tools-dev/teams/data-and-information
Does ICES already have a controlled vocabulary for this?
@nicolasvanermen https://github.com/nicolasvanermen Wouldn't human
readable values like all, all_not_larus be better, or are the integer
codes well established?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ices-tools-dev/esas/issues/19, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYKGXOC4H23OIQCZMHX533SYRC2BANCNFSM4VXNBRAA
.

I would hold on to the integer codes, species_counted often gets drawn-out
descriptions.
Nicolas Vanermen
Wetenschappelijk medewerker
Instituut voor Natuur- en Bosonderzoek
Havenlaan 88, bus 73, 1000 Brussel
0486/361.582
On Wed, Jan 6, 2021 at 12:31 PM Peter Desmet<EMAIL_ADDRESS>wrote:

trips: species_counted indicates what species are counted with an integer
code:
https://github.com/ices-tools-dev/esas/blob/0d294b4ea678e34f52409a16602f9133ac179dcf/_data/table-schemas/trips.yaml#L235-L255

@ices-tools-dev/data-and-information
https://github.com/orgs/ices-tools-dev/teams/data-and-information
Does ICES already have a controlled vocabulary for this?
@nicolasvanermen https://github.com/nicolasvanermen Wouldn't human
readable values like all, all_not_larus be better, or are the integer
codes well established?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ices-tools-dev/esas/issues/19, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYKGXOC4H23OIQCZMHX533SYRC2BANCNFSM4VXNBRAA
.

need new vocab

need new vocab

Question: what does 'All species' mean?

This means that all species encountered are counted, opposed to the other
codes that all indicate a selection of species (not) being counted.
Nicolas Vanermen
Wetenschappelijk medewerker
Instituut voor Natuur- en Bosonderzoek
Havenlaan 88, bus 73, 1000 Brussel
0486/361.582
On Tue, Aug 10, 2021 at 4:08 PM Anna Osypchuk @.***>
wrote:

Question: what does 'All species' mean?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ices-tools-dev/esas/issues/19#issuecomment-896060549,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABYKGXIGMVAVU7TJMNHJVZLT4EXHNANCNFSM4VXNBRAA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email
.


Error connecting to CertStream - Connection is already closed

I implemented this tool, following the given examples. I found out that sometimes ( usally every 2 minutes, more or less) the connection drops and it takes almost a minute to reconnect to the stream.
I don't know why this happens and this could probably lead to missing some hits i'm looking for.
Here's an example of log ( integrated with python logging module )

[ERROR:certstream] 2020-09-02 16:20:31,002 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:20:36,097 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:23:10,796 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:23:15,908 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:25:11,015 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:25:16,182 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:27:17,051 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:27:22,194 - Connection established to CertStream! Listening for events...
[...]
[ERROR:certstream] 2020-09-02 16:29:01,343 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2020-09-02 16:29:06,522 - Connection established to CertStream! Listening for events...

Any suggestion about how to fix this?

Facing the same issue again!

Hi. Might be related to https://github.com/CaliDog/certstream-server/issues/59
@Fitblip have you found the time to check this issue already? As far as I can see, your library is sending pings/keepalives over the websocket to indicate that it's still being used.
I haven't been able to find the reason behind the random disconnects yet.

Hi. Might be related to https://github.com/CaliDog/certstream-server/issues/59
@Fitblip have you found the time to check this issue already? As far as I can see, your library is sending pings/keepalives over the websocket to indicate that it's still being used.
I haven't been able to find the reason behind the random disconnects yet.

Hmm interesting, are you folks using the certstream.calidog.io server, or is this happening with private servers as well? I wonder if you happen to be using an older version of the library (pinned in your requirements.txt) or something? As long as it's sending something to the server though it should be fine.

Hmm interesting, are you folks using the certstream.calidog.io server, or is this happening with private servers as well? I wonder if you happen to be using an older version of the library (pinned in your requirements.txt) or something? As long as it's sending something to the server though it should be fine.

I for my part am using your certstream.calidog.io server - at least for development.
$ pip freeze
[...]
certstream==1.11
websocket-client==0.57.0
[...]

The error message is always "Connection is already closed.". There are basically two places where that exception is raised:
websocket/_app.py
    def send(self, data, opcode=ABNF.OPCODE_TEXT):
        [...]

        if not self.sock or self.sock.send(data, opcode) == 0:
            raise WebSocketConnectionClosedException(
                "Connection is already closed.")

websocket/_socket.py
def recv(sock, bufsize):
    if not sock:
        raise WebSocketConnectionClosedException("socket is already closed.")

    def _recv():
        [...]

    try:
        if sock.gettimeout() == 0:
            bytes_ = sock.recv(bufsize)
        else:
            bytes_ = _recv()
    except socket.timeout as e:
        [...]
    except SSLError as e:
        [...]

    if not bytes_:
        raise WebSocketConnectionClosedException(
            "Connection is already closed.")

For me the disconnects only appear within the latter (recv) call.
It reads 0 bytes

This is the full stack trace:

I for my part am using your certstream.calidog.io server - at least for development.
$ pip freeze
[...]
certstream==1.11
websocket-client==0.57.0
[...]

The error message is always "Connection is already closed.". There are basically two places where that exception is raised:
websocket/_app.py
    def send(self, data, opcode=ABNF.OPCODE_TEXT):
        [...]

        if not self.sock or self.sock.send(data, opcode) == 0:
            raise WebSocketConnectionClosedException(
                "Connection is already closed.")

websocket/_socket.py
def recv(sock, bufsize):
    if not sock:
        raise WebSocketConnectionClosedException("socket is already closed.")

    def _recv():
        [...]

    try:
        if sock.gettimeout() == 0:
            bytes_ = sock.recv(bufsize)
        else:
            bytes_ = _recv()
    except socket.timeout as e:
        [...]
    except SSLError as e:
        [...]

    if not bytes_:
        raise WebSocketConnectionClosedException(
            "Connection is already closed.")

For me the disconnects only appear within the latter (recv) call.
It reads 0 bytes

This is the full stack trace:

I'm facing the same issue when running certstream --json --full >> certstream.json. I've installed certstream using pip installed certstream
[INFO:certstream] 2021-01-27 13:34:15,660 - Connection established to CertStream! Listening for events...
[ERROR:certstream] 2021-01-27 13:36:58,324 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/websocket/_app.py", line 306, in run_forever
    dispatcher.read(self.sock.sock, read, check)
  File "/usr/local/lib/python3.9/site-packages/websocket/_app.py", line 66, in read
    if not read_callback():
  File "/usr/local/lib/python3.9/site-packages/websocket/_app.py", line 272, in read
    op_code, frame = self.sock.recv_data_frame(True)
  File "/usr/local/lib/python3.9/site-packages/websocket/_core.py", line 344, in recv_data_frame
    frame = self.recv_frame()
  File "/usr/local/lib/python3.9/site-packages/websocket/_core.py", line 378, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/usr/local/lib/python3.9/site-packages/websocket/_abnf.py", line 375, in recv_frame
    payload = self.recv_strict(length)
  File "/usr/local/lib/python3.9/site-packages/websocket/_abnf.py", line 396, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "/usr/local/lib/python3.9/site-packages/websocket/_core.py", line 453, in _recv
    return recv(self.sock, bufsize)
  File "/usr/local/lib/python3.9/site-packages/websocket/_socket.py", line 114, in recv
    raise WebSocketConnectionClosedException(
websocket._exceptions.WebSocketConnectionClosedException: Connection is already closed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/certstream", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/certstream/cli.py", line 64, in main
    certstream.listen_for_events(_handle_messages, args.url, skip_heartbeats=True)
  File "/usr/local/lib/python3.9/site-packages/certstream/core.py", line 54, in listen_for_events
    c.run_forever(ping_interval=15, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/websocket/_app.py", line 312, in run_forever
    teardown()
  File "/usr/local/lib/python3.9/site-packages/websocket/_app.py", line 231, in teardown
    if thread and thread.isAlive():
AttributeError: 'Thread' object has no attribute 'isAlive'

I'm facing the same issue when running certstream --json --full >> certstream.json. I've installed certstream using pip installed certstream
[INFO:certstream] 2021-01-27 13:34:15,660 - Connection established to CertStream! Listening for events...
[ERROR:certstream] 2021-01-27 13:36:58,324 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/websocket/_app.py", line 306, in run_forever
    dispatcher.read(self.sock.sock, read, check)
  File "/usr/local/lib/python3.9/site-packages/websocket/_app.py", line 66, in read
    if not read_callback():
  File "/usr/local/lib/python3.9/site-packages/websocket/_app.py", line 272, in read
    op_code, frame = self.sock.recv_data_frame(True)
  File "/usr/local/lib/python3.9/site-packages/websocket/_core.py", line 344, in recv_data_frame
    frame = self.recv_frame()
  File "/usr/local/lib/python3.9/site-packages/websocket/_core.py", line 378, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/usr/local/lib/python3.9/site-packages/websocket/_abnf.py", line 375, in recv_frame
    payload = self.recv_strict(length)
  File "/usr/local/lib/python3.9/site-packages/websocket/_abnf.py", line 396, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "/usr/local/lib/python3.9/site-packages/websocket/_core.py", line 453, in _recv
    return recv(self.sock, bufsize)
  File "/usr/local/lib/python3.9/site-packages/websocket/_socket.py", line 114, in recv
    raise WebSocketConnectionClosedException(
websocket._exceptions.WebSocketConnectionClosedException: Connection is already closed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/certstream", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/certstream/cli.py", line 64, in main
    certstream.listen_for_events(_handle_messages, args.url, skip_heartbeats=True)
  File "/usr/local/lib/python3.9/site-packages/certstream/core.py", line 54, in listen_for_events
    c.run_forever(ping_interval=15, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/websocket/_app.py", line 312, in run_forever
    teardown()
  File "/usr/local/lib/python3.9/site-packages/websocket/_app.py", line 231, in teardown
    if thread and thread.isAlive():
AttributeError: 'Thread' object has no attribute 'isAlive'

I for my part am using your certstream.calidog.io server - at least for development.
$ pip freeze
[...]
certstream==1.11
websocket-client==0.57.0
[...]

The error message is always "Connection is already closed.". There are basically two places where that exception is raised:
websocket/_app.py
    def send(self, data, opcode=ABNF.OPCODE_TEXT):
        [...]

        if not self.sock or self.sock.send(data, opcode) == 0:
            raise WebSocketConnectionClosedException(
                "Connection is already closed.")

websocket/_socket.py
def recv(sock, bufsize):
    if not sock:
        raise WebSocketConnectionClosedException("socket is already closed.")

    def _recv():
        [...]

    try:
        if sock.gettimeout() == 0:
            bytes_ = sock.recv(bufsize)
        else:
            bytes_ = _recv()
    except socket.timeout as e:
        [...]
    except SSLError as e:
        [...]

    if not bytes_:
        raise WebSocketConnectionClosedException(
            "Connection is already closed.")

For me the disconnects only appear within the latter (recv) call.
It reads 0 bytes

This is the full stack trace:

I exactly have same issue with using calidog server.
certificate_update: 192647cert [13:19, 261.07cert/s]Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
certificate_update: 212013cert [14:58, 21.30cert/s]Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...

Error occurred every 1~2 minutes so that connection is not stable.
Hope this will be solved.

I for my part am using your certstream.calidog.io server - at least for development.
$ pip freeze
[...]
certstream==1.11
websocket-client==0.57.0
[...]

The error message is always "Connection is already closed.". There are basically two places where that exception is raised:
websocket/_app.py
    def send(self, data, opcode=ABNF.OPCODE_TEXT):
        [...]

        if not self.sock or self.sock.send(data, opcode) == 0:
            raise WebSocketConnectionClosedException(
                "Connection is already closed.")

websocket/_socket.py
def recv(sock, bufsize):
    if not sock:
        raise WebSocketConnectionClosedException("socket is already closed.")

    def _recv():
        [...]

    try:
        if sock.gettimeout() == 0:
            bytes_ = sock.recv(bufsize)
        else:
            bytes_ = _recv()
    except socket.timeout as e:
        [...]
    except SSLError as e:
        [...]

    if not bytes_:
        raise WebSocketConnectionClosedException(
            "Connection is already closed.")

For me the disconnects only appear within the latter (recv) call.
It reads 0 bytes

This is the full stack trace:

I exactly have same issue with using calidog server.
certificate_update: 192647cert [13:19, 261.07cert/s]Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
certificate_update: 212013cert [14:58, 21.30cert/s]Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...

Error occurred every 1~2 minutes so that connection is not stable.
Hope this will be solved.

I am having the same problem with 1.11 from pypi.
[ERROR:certstream] 2021-02-06 15:33:51,563 - Error connecting to CertStream - Connection is already closed. - Sleeping for a few seconds and trying again...
Traceback (most recent call last):
  File "/home/chaz/.local/lib/python3.9/site-packages/websocket/_app.py", line 306, in run_forever
    dispatcher.read(self.sock.sock, read, check)
  File "/home/chaz/.local/lib/python3.9/site-packages/websocket/_app.py", line 66, in read
    if not read_callback():
  File "/home/chaz/.local/lib/python3.9/site-packages/websocket/_app.py", line 272, in read
    op_code, frame = self.sock.recv_data_frame(True)
  File "/home/chaz/.local/lib/python3.9/site-packages/websocket/_core.py", line 344, in recv_data_frame
    frame = self.recv_frame()
  File "/home/chaz/.local/lib/python3.9/site-packages/websocket/_core.py", line 378, in recv_frame
    return self.frame_buffer.recv_frame()
  File "/home/chaz/.local/lib/python3.9/site-packages/websocket/_abnf.py", line 375, in recv_frame
    payload = self.recv_strict(length)
  File "/home/chaz/.local/lib/python3.9/site-packages/websocket/_abnf.py", line 396, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "/home/chaz/.local/lib/python3.9/site-packages/websocket/_core.py", line 453, in _recv
    return recv(self.sock, bufsize)
  File "/home/chaz/.local/lib/python3.9/site-packages/websocket/_socket.py", line 114, in recv
    raise WebSocketConnectionClosedException(
websocket._exceptions.WebSocketConnectionClosedException: Connection is already closed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/chaz/.local/bin/certstream", line 8, in <module>
    sys.exit(main())
  File "/home/chaz/.local/lib/python3.9/site-packages/certstream/cli.py", line 64, in main
    certstream.listen_for_events(_handle_messages, args.url, skip_heartbeats=True)
  File "/home/chaz/.local/lib/python3.9/site-packages/certstream/core.py", line 54, in listen_for_events
    c.run_forever(ping_interval=15, **kwargs)
  File "/home/chaz/.local/lib/python3.9/site-packages/websocket/_app.py", line 312, in run_forever
    teardown()
  File "/home/chaz/.local/lib/python3.9/site-packages/websocket/_app.py", line 231, in teardown
    if thread and thread.isAlive():
AttributeError: 'Thread' object has no attribute 'isAlive'

This is fixed upstream https://github.com/websocket-client/websocket-client/pull/594

@Chaz6 the linked PR does not fix the issue for me. I applied the patch locally but am still getting the same issue every two minutes or so.

@Chaz6 the linked PR does not fix the issue for me. I applied the patch locally but am still getting the same issue every two minutes or so.

What version of Python do you have?

@Chaz6 the linked PR does not fix the issue for me. I applied the patch locally but am still getting the same issue every two minutes or so.

What version of Python do you have?

I am currently developing and testing on Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32

I am currently developing and testing on Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32

OK, I am using python 3.9 so it is possible there is a different fix required.

@Chaz6 I'm facing the same problem. I'm running certstream in the docker container on AWS ECS and python version is 3.9.5.

@Fitblip mind looking into this? This is "easy to reproduce" because it literally happens (reliably) every time I use certstream.
Your help is much appreciated! Thanks in advance.

Just as a point of reference, today I updated to certstream 1.12 (python 3.9.6 on Linux 64bit) and it works fine.
$ pip3 install -U --user certstream
Requirement already satisfied: certstream in /home/chaz/.local/lib/python3.9/site-packages (1.11)
Collecting certstream
  Downloading certstream-1.12.tar.gz (9.6 kB)
Collecting websocket-client>=0.58.0
  Using cached websocket_client-1.2.0-py2.py3-none-any.whl (52 kB)
Requirement already satisfied: termcolor in /home/chaz/.local/lib/python3.9/site-packages (from certstream) (1.1.0)
Building wheels for collected packages: certstream
  Building wheel for certstream (setup.py) ... done
  Created wheel for certstream: filename=certstream-1.12-py3-none-any.whl size=4731 sha256=6c8a70c8ba6c8510511ba16ff26432014fc95fc546223645cf302f31a89c7266
  Stored in directory: /home/chaz/.cache/pip/wheels/b9/d5/4b/157f4ada54a1b76c9509a7c00191014478d1039d390eda57c3
Successfully built certstream
Installing collected packages: websocket-client, certstream
  Attempting uninstall: websocket-client
    Found existing installation: websocket-client 0.57.0
    Uninstalling websocket-client-0.57.0:
      Successfully uninstalled websocket-client-0.57.0
  Attempting uninstall: certstream
    Found existing installation: certstream 1.11
    Uninstalling certstream-1.11:
      Successfully uninstalled certstream-1.11
Successfully installed certstream-1.12 websocket-client-1.2.0

$ certstream
[INFO:certstream] 2021-08-12 09:52:33,821 - Connection established to CertStream! Listening for events...
[2021-08-12T09:52:33.875015] ct.googleapis.com/testtube/ - www01.abn.ing.rabo.abn.accounts.betaling.berlingskemedia.net
[2021-08-12T09:52:33.876959] ct.googleapis.com/testtube/ - docker-test.aggeler.eu
[2021-08-12T09:52:33.878495] ct.googleapis.com/testtube/ - load.12-s.de
[2021-08-12T09:52:33.880872] ct.googleapis.com/testtube/ - og.blog.blog.magento.rainbow.idealnastrona.pl
[2021-08-12T09:52:33.882442] ct.googleapis.com/testtube/ - blog.blog.dev.sitemap.owlstown.net
[2021-08-12T09:52:33.883426] ct.googleapis.com/testtube/ - www.cheybeauty.org
[2021-08-12T09:52:33.884452] ct.googleapis.com/testtube/ - www.emilyasheyphoto.com

Hi folks, apologies for the delay in looking into this. I think there's a few things in this one issue to the point where it's over-loaded.
Install/packaging/isAlive issues
I'm able to use certstream as expected with docker and the lastest python, which leads me to believe this is not an issue with certstream, but a local issue:
$ docker pull python
Using default tag: latest
latest: Pulling from library/python
d272b0d8f7c4: Pull complete
81fabd82f026: Pull complete
0193c0cdceae: Pull complete
967a8f138596: Pull complete
bc1dbd38cda7: Pull complete
80a46c137b0c: Pull complete
f41525227e2c: Pull complete
1354066e326c: Pull complete
ffb5ac5ec588: Pull complete
Digest: sha256:44e7cbba204f9a81ec5bfa1b5de1de176b2590c01d55a02bd299551593517c7d
Status: Downloaded newer image for python:latest
docker.io/library/python:latest

$ docker run --rm -it python bash

root@1477dfc03834:/ $ pip install certstream
Collecting certstream
  Downloading certstream-1.12.tar.gz (9.6 kB)
Collecting websocket-client>=0.58.0
  Using cached websocket_client-1.2.0-py2.py3-none-any.whl (52 kB)
Collecting termcolor
  Using cached termcolor-1.1.0-py3-none-any.whl
Building wheels for collected packages: certstream
  Building wheel for certstream (setup.py) ... done
  Created wheel for certstream: filename=certstream-1.12-py3-none-any.whl size=4746 sha256=bf05687d2cfdbd224c5873cf0d6d229d6cd4176fa7cd5cd1ace19638ac45ee8c
  Stored in directory: /root/.cache/pip/wheels/b9/d5/4b/157f4ada54a1b76c9509a7c00191014478d1039d390eda57c3
Successfully built certstream
Installing collected packages: websocket-client, termcolor, certstream
Successfully installed certstream-1.12 termcolor-1.1.0 websocket-client-1.2.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.3; however, version 21.2.4 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

root@1477dfc03834:/ $ certstream
[INFO:certstream] 2021-08-12 20:17:57,734 - Connection established to CertStream! Listening for events...
[2021-08-12T20:17:57.799850] ct.googleapis.com/logs/argon2021/ - imap.sangse.com
[2021-08-12T20:17:57.801696] ct.googleapis.com/logs/argon2021/ - universe.manawac.com
[2021-08-12T20:17:57.803194] ct.googleapis.com/logs/argon2021/ - inspire.devmantra.uk
[2021-08-12T20:17:57.804575] ct.googleapis.com/logs/argon2021/ - mail.airlab.id
[2021-08-12T20:17:57.805963] ct.googleapis.com/logs/argon2021/ - staging.191.com.au

Disconnections
The certstream server is written in Elixir, using the excellent Cowboy http server. I believe when it doesn't receive any data from the client after a specific timeout (60s according to the docs), it assumes the client is dead and reaps the connection.
The fix for this is to have the client spawn a thread to send some data up to the server every 30s or so (a simple ws.send('.') will work), outlined in the websocket-client docs.
I'm not sure I have time to implement this but if someone wants to take a stab at it I'd happily look at a PR! :)

Thank you @Fitblip for your reply! I followed your steps:
root@server:~# docker run --rm -it python bash

root@755dee80470a:/# pip install certstream
Collecting certstream
  Downloading certstream-1.12.tar.gz (9.6 kB)
Collecting websocket-client>=0.58.0
  Using cached websocket_client-1.2.0-py2.py3-none-any.whl (52 kB)
Collecting termcolor
  Using cached termcolor-1.1.0-py3-none-any.whl
Building wheels for collected packages: certstream
  Building wheel for certstream (setup.py) ... - \ done
  Created wheel for certstream: filename=certstream-1.12-py3-none-any.whl size=4746 sha256=44a26af2b4f43e01bb24b32fdc267c5057832012c75988d239b7fb5c48d7aa30
  Stored in directory: /root/.cache/pip/wheels/b9/d5/4b/157f4ada54a1b76c9509a7c00191014478d1039d390eda57c3
Successfully built certstream
Installing collected packages: websocket-client, termcolor, certstream
Successfully installed certstream-1.12 termcolor-1.1.0 websocket-client-1.2.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

root@755dee80470a:/# certstream
[INFO:certstream] 2021-08-12 23:58:00,412 - Connection established to CertStream! Listening for events...
[2021-08-12T23:58:03.549530] ct.googleapis.com/skydiver/ - sbza7hkhrd6mmtk.westus2.atlas-test.cloudapp.azure.com
[2021-08-12T23:58:03.551558] ct.googleapis.com/skydiver/ - wus2-fd497270b7.wus2.backend-instance-dev.azgrafana-test.io
[2021-08-12T23:58:04.663397] ct.googleapis.com/logs/xenon2022/ - c8577f8edcf9.database.azure.com
[...]
[2021-08-13T00:00:22.213534] nessie2022.ct.digicert.com/log/ - sni.cloudflaressl.com
[2021-08-13T00:00:22.215210] nessie2022.ct.digicert.com/log/ - sni.cloudflaressl.com
[ERROR:certstream] 2021-08-13 00:01:01,141 - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2021-08-13 00:01:06,226 - Connection established to CertStream! Listening for events...
[2021-08-13T00:01:08.407135] ct.googleapis.com/logs/argon2022/ - sni.cloudflaressl.com
[2021-08-13T00:01:08.408785] ct.googleapis.com/logs/argon2022/ - sni.cloudflaressl.com

As you can see I still get the error Error connecting to CertStream - Connection to remote host was lost, which is another error to the previously mentioned Connection is already closed.
Also in my custom software where I use certstream as a library:
2021-08-13 02:03:04,394 - certstream - ERROR - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
2021-08-13 02:03:05,534 - certstream - INFO - Connection established to CertStream! Listening for events...
2021-08-13 02:03:50,008 - certleak.core.certstreamwrapper - INFO - Processed 1486 updates (2881 unique domains) during the last 60 seconds. 48.0 domains/s, 24.8 certs/s
2021-08-13 02:03:50,008 - certleak.core.certstreamwrapper - INFO - Queue length: 1
2021-08-13 02:04:11,178 - certstream - ERROR - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
2021-08-13 02:04:12,323 - certstream - INFO - Connection established to CertStream! Listening for events...
2021-08-13 02:04:50,032 - certleak.core.certstreamwrapper - INFO - Processed 1854 updates (2946 unique domains) during the last 60 seconds. 49.1 domains/s, 30.9 certs/s
2021-08-13 02:04:50,032 - certleak.core.certstreamwrapper - INFO - Queue length: 1
2021-08-13 02:05:05,796 - certstream - ERROR - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
2021-08-13 02:05:06,899 - certstream - INFO - Connection established to CertStream! Listening for events...
2021-08-13 02:05:50,025 - certleak.core.certstreamwrapper - INFO - Processed 1857 updates (2790 unique domains) during the last 60 seconds. 46.5 domains/s, 30.9 certs/s
2021-08-13 02:05:50,025 - certleak.core.certstreamwrapper - INFO - Queue length: 1
2021-08-13 02:05:54,115 - certstream - ERROR - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
2021-08-13 02:05:55,213 - certstream - INFO - Connection established to CertStream! Listening for events...
2021-08-13 02:06:38,937 - certstream - ERROR - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
2021-08-13 02:06:40,167 - certstream - INFO - Connection established to CertStream! Listening for events...
2021-08-13 02:06:50,013 - certleak.core.certstreamwrapper - INFO - Processed 1721 updates (2789 unique domains) during the last 60 seconds. 46.5 domains/s, 28.7 certs/s
2021-08-13 02:06:50,013 - certleak.core.certstreamwrapper - INFO - Queue length: 1
2021-08-13 02:07:30,673 - certstream - ERROR - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
2021-08-13 02:07:31,862 - certstream - INFO - Connection established to CertStream! Listening for events...
2021-08-13 02:07:50,027 - certleak.core.certstreamwrapper - INFO - Processed 1818 updates (2731 unique domains) during the last 60 seconds. 45.5 domains/s, 30.3 certs/s
2021-08-13 02:07:50,027 - certleak.core.certstreamwrapper - INFO - Queue length: 1
2021-08-13 02:08:22,420 - certstream - ERROR - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
2021-08-13 02:08:23,620 - certstream - INFO - Connection established to CertStream! Listening for events...

 So this could only be fixed with a regular ws.send('.')?
If so, I'll try to implement this as soon as I find some time.

Thank you @Fitblip @d-Rickyy-b. Even though I updated to certstream 1.12, I still get the same error. It is much appreciated if someone can implement above (ws.send('.')).

@Fitblip @d-Rickyy-b I implemented ws.send in cli.py as following.
def _on_open(self, ws):
    certstream_logger.info("Connection established to CertStream! Listening for events...!")
    if self.on_open_handler:
        self.on_open_handler()
    t_ws_send = threading.Thread(target=self._ws_send, args=(ws,))
    t_ws_send.start()

def _ws_send(self, ws):
    while ws:
        try:
            ws.send('.')
            certstream_logger.info("Connection continue to CertStream! data sending...")
            time.sleep(10)
        except Exception as e:
            certstream_logger.error(e)
            break

But, still get the same error.
[INFO:certstream] 2021-08-23 08:26:48,303 - Connection established to CertStream! Listening for events...!
[INFO:certstream] 2021-08-23 08:26:48,304 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:26:58,314 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:27:08,320 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:27:18,330 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:27:28,341 - Connection continue to CertStream! data sending...
[ERROR:certstream] 2021-08-23 08:27:28,663 - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2021-08-23 08:27:34,248 - Connection established to CertStream! Listening for events...!
[INFO:certstream] 2021-08-23 08:27:34,248 - Connection continue to CertStream! data sending...
[ERROR:certstream] 2021-08-23 08:27:38,351 - Connection is already closed.
[INFO:certstream] 2021-08-23 08:27:44,259 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:27:54,269 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:28:04,284 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:28:14,296 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:28:24,304 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:28:34,315 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:28:44,325 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:28:54,336 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:29:04,346 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:29:14,357 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:29:24,367 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:29:34,374 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:29:44,385 - Connection continue to CertStream! data sending...
[ERROR:certstream] 2021-08-23 08:29:44,948 - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2021-08-23 08:29:50,540 - Connection established to CertStream! Listening for events...!
[INFO:certstream] 2021-08-23 08:29:50,540 - Connection continue to CertStream! data sending...
[ERROR:certstream] 2021-08-23 08:29:54,395 - Connection is already closed.
[INFO:certstream] 2021-08-23 08:30:00,551 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:30:10,561 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:30:20,572 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:30:30,580 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:30:40,591 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:30:50,601 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:31:00,612 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:31:10,617 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:31:20,627 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:31:30,638 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:31:40,648 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:31:50,658 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:32:00,669 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:32:10,679 - Connection continue to CertStream! data sending...
[INFO:certstream] 2021-08-23 08:32:20,690 - Connection continue to CertStream! data sending...
[ERROR:certstream] 2021-08-23 08:32:20,829 - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2021-08-23 08:32:26,459 - Connection established to CertStream! Listening for events...!
[INFO:certstream] 2021-08-23 08:32:26,459 - Connection continue to CertStream! data sending...
[ERROR:certstream] 2021-08-23 08:32:30,700 - Connection is already closed.
[INFO:certstream] 2021-08-23 08:32:36,460 - Connection continue to CertStream! data sending...

I would like to prevent data loss caused by this error, do you have any ideas?

@Fitblip @d-Rickyy-b I would like to know whether this disconnection happens only in my environment or not. Do you have any success cases (environments) that don't make this data loss error? or do you have any ideas to collect certificates data without data loss using other tools?

I have changed the python version from 3.9.7 to 3.8.12 It working fine without the error

@katesaikishore Thank you for your advice. I have changed the python to python:3.8.12-buster, but it still has the error.
ubuntu@ip-10-0-0-241:~/work/certstream_documentdb$ docker run 8ed9a1d5f1f8
[INFO:certstream] 2021-09-06 05:48:17,937 - Connection established to CertStream! Listening for events...
[ERROR:certstream] 2021-09-06 05:49:47,094 - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2021-09-06 05:49:52,666 - Connection established to CertStream! Listening for events...
[ERROR:certstream] 2021-09-06 05:52:38,244 - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2021-09-06 05:52:43,799 - Connection established to CertStream! Listening for events...
[ERROR:certstream] 2021-09-06 05:58:10,727 - Error connecting to CertStream - Connection to remote host was lost. - Sleeping for a few seconds and trying again...
[INFO:certstream] 2021-09-06 05:58:16,304 - Connection established to CertStream! Listening for events...

Could you show the other libraries versions? My environment is as following
certstream==1.12
pymongo==3.12.0
python-dateutil==2.8.2
PyYAML==5.4.1
schedule==1.1.0
six==1.16.0
termcolor==1.1.0
websocket-client==1.2.1

certstram==1.11
websocket-client==0.57.0
And use 3.8.12
Try to deploy it in heroku cloud. @kres20

@katesaikishore Than you for your reply. We used the same versions of the libraries except for heroku (we have to use AWS), but it could not work well. I'll try it in local environment, not cloud. I'll also check the another certstream like Go, JAVA.

@kres20 Oh, I have deployed it in heruko. Even though it will disconnect and reconnect itself. Something is really fishy.

@katesaikishore I've just created the environment in locally (in VM) without docker, but it still has the error. @Fitblip Do you think that this problem is caused by server side (wss://certstream.calidog.io/)?

@katesaikishore @Fitblip I set up the certstream server too (https://github.com/CaliDog/certstream-server), and connect there like certstream.listen_for_events(print_callback, url='ws://localhost:4000/'), and it works fine without any errors. Therefore this above error is caused by certstream.calidog.io.

Argh, well this is disappointing. :-( I am hitting this problem too and can only reach the same conclusion as @kres20 that the certstream.calidog.io service is just basically unusable.
I noticed certstream.calidog.io is actually Cloudflare. In Wireshark packet capture on my client, I can see the Cloudflare frontend server is just suddenly closing the TCP connection for some reason. No shutting down of the Websocket session or the TLS session, but a (mostly) normal 4-way close handshake of the outer TCP connection. (Abnormally, the Cloudflare server waits for exactly 10 seconds before sending its very last ACK, despite my client dutifully attempting several retransmissions of its last FIN that the server is waiting for.)
When I add sslopt={'suppress_ragged_eofs': False} to the certstream.listen_for_events parameters, the issue causes an exception that matches what I saw in packet capture: ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2633).
Traceback (most recent call last):
  File "C:\Users\m\dev\watchcertstream\watchcertstream.py", line 72, in <module>
    main()
  File "C:\Users\m\dev\watchcertstream\watchcertstream.py", line 68, in main
    certstream.listen_for_events(certstream_message_callback, url='wss://certstream.calidog.io/', on_error=certstream_error_callback, sslopt={"suppress_ragged_eofs": False})
  File "C:\Program Files\Python39\lib\site-packages\certstream\core.py", line 54, in listen_for_events
    c.run_forever(ping_interval=15, **kwargs)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 371, in run_forever
    self._callback(self.on_error, e)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 412, in _callback
    self.on_error(self, e)
  File "C:\Program Files\Python39\lib\site-packages\certstream\core.py", line 47, in _on_error
    self.on_error_handler(ex)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 407, in _callback
    callback(self, *args)
  File "C:\Program Files\Python39\lib\site-packages\certstream\core.py", line 47, in _on_error
    self.on_error_handler(ex)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 369, in run_forever
    dispatcher.read(self.sock.sock, read, check)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 70, in read
    if not read_callback():
  File "C:\Program Files\Python39\lib\site-packages\websocket\_app.py", line 335, in read
    op_code, frame = self.sock.recv_data_frame(True)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_core.py", line 396, in recv_data_frame
    frame = self.recv_frame()
  File "C:\Program Files\Python39\lib\site-packages\websocket\_core.py", line 435, in recv_frame
    return self.frame_buffer.recv_frame()
  File "C:\Program Files\Python39\lib\site-packages\websocket\_abnf.py", line 351, in recv_frame
    payload = self.recv_strict(length)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_abnf.py", line 372, in recv_strict
    bytes_ = self.recv(min(16384, shortage))
  File "C:\Program Files\Python39\lib\site-packages\websocket\_core.py", line 519, in _recv
    return recv(self.sock, bufsize)
  File "C:\Program Files\Python39\lib\site-packages\websocket\_socket.py", line 113, in recv
    bytes_ = _recv()
  File "C:\Program Files\Python39\lib\site-packages\websocket\_socket.py", line 90, in _recv
    return sock.recv(bufsize)
  File "C:\Program Files\Python39\lib\ssl.py", line 1226, in recv
    return self.read(buflen)
  File "C:\Program Files\Python39\lib\ssl.py", line 1101, in read
    return self._sslobj.read(len)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2633)

When I use my preferred public DNS resolver (<IP_ADDRESS>), certstream.calidog.io resolves to <IP_ADDRESS> for me. This is a Cloudflare IP. Traceroute from my client and from a couple other places suggests that it's an anycast IP that exists at/near multiple different Internet exchanges.
When I use other public DNS resolvers, certstream.calidog.io resolves to different Cloudflare IPs for me - <IP_ADDRESS> and <IP_ADDRESS>. When I force my client to connect to these IPs, the problem still happens, but anecdotally seems to occur far less frequently than with <IP_ADDRESS>.
I'm skeptical that IP anycast is a suitable CDN strategy for services requiring permanent TCP connections, such as the Calidog Certstream service. IMO this problem is likely a completely normal side-effect of the chosen CDN strategy that the Calidog service is using. :-(

@mikenor Yep things are currently hosted behind Cloudflare, and I agree it's not ideal given the CDN nature of cloudflare and the persistent connections for the operation of certstream. Ultimately https://certstream.calidog.io is a service to try out certstream but if you need production grade quality (for analytics/etc) then you should absolutely run your own server. I chose Cloudflare because it was a way for me to spin up a SSL fronted service without much headache and to cut down on potential botspam.
For the public instance we also currently fully saturate a gigabit link (https://twitter.com/fitblip/status/1203058921583398914) and pump about a petabyte of traffic into the internet every 3 months, so inevitably that means that certs are dropped and you shouldn't rely on the service to see 100% of everything.
Ultimately I'm not interested in spending more time or money per month to host this in a way where it's a rock solid production quality service (certstream.calidog.io, the server software should be solid), but if someone wanted to donate some servers and/or run the service and run it as a cluster with multi-gigabit links then I'm absolutely game to point the DNS records at it and help with the effort!


updated the wrong categories

fixed AGCORDOVA-112

@edewit should this be filed against the upstream repo ?


Added root go file for client-go for dep dependency manager

What this PR does / why we need it:
Added root go file for client-go for dep dependency manager
Special notes for your reviewer:
Just the same as we already have for kubevirt's root dir.
Already done on client-go v0.19.0 and tested on HCO
Release note:
NONE

/hold
@slintes I think it needs a make generate when looking on the travis error.

@slintes I think it needs a make generate when looking on the travis error.

yes, new BUILD.bazel file was missing...

Travis is happy now
/hold cancel


Add most shared topic

Current calculation could also use the number of post shares. Adding this as a todo item to implement in the near future.

Added via 854ad7675c0be1f292c997bd745c43240ab3b2b2


Improve handling of #base and #include directives

Hello! 👋
This PR makes two relatively small changes to improve handling of #base and #include directives.
I've been working with some .res files that have #base or #include directives which point to files that don't exist, and this PR silently skips those files. To my knowledge, this is how the Source engine treats such directives. Additionally, I've been working with some files that consist ONLY of these directives, which is also valid in the Source engine. This PR handles both cases.
Let me know if you'd like any changes or additions to this PR. I thought about adding tests but was kind of scared off by the test structure. I'd appreciate some help on that front if tests are desired.
Thanks!

Does it not work if your FileLoader simply returns an empty object? This needs a test too.

Does it not work if your FileLoader simply returns an empty object?

It does not, an error is thrown:
Unhandled exception. System.InvalidOperationException: Stack empty.
   at System.Collections.Generic.Stack`1.ThrowForEmptyStack()
   at System.Collections.Generic.Stack`1.Peek()
   at ValveKeyValue.Deserialization.KVMergingObjectBuilder.FinalizeState() in C:\Users\vanca\Documents\git\SteamDatabase\ValveKeyValue\ValveKeyValue\ValveKeyValue\Deserialization\KVMergingObjectBuilder.cs:line 20
   at ValveKeyValue.Deserialization.KVObjectBuilder.GetObject() in C:\Users\vanca\Documents\git\SteamDatabase\ValveKeyValue\ValveKeyValue\ValveKeyValue\Deserialization\KVObjectBuilder.cs:line 20
   at ValveKeyValue.KVSerializer.Deserialize(Stream stream, KVSerializerOptions options) in C:\Users\vanca\Documents\git\SteamDatabase\ValveKeyValue\ValveKeyValue\ValveKeyValue\KVSerializer.cs:line 45
   at REDACTED

This needs a test too.

Tests have been added 🙂

Perhaps instead of removing all the checks there should be a way for file loader to accept empty objects.
Have you tried an empty root object too?

Perhaps instead of removing all the checks there should be a way for file loader to accept empty objects.

I'm not sure what you mean.

Have you tried an empty root object too?

Yes. If a file that has #base or #include directives has even an empty root object, it would succeed without this PR. This PR only helps in the case where a file has only #base and #include directives with no root object.

I'd rather keep the checks that you removed, and instead explicitly support empty files/objects (consumers can provide empty object at the moment).

I'm gonna have to dig out my VKV-vs-tier1 validation tool and check exactly how Valve's handling works.


No token warning

Per discussion here.

Very nice. Although this "no token warning" only covers our main example.
Adding this much code to every minimal example would be too much.
As an alternative we might actually want to consider this as a PR to react-map-gl itself. @abmai.

Closing in favor of #513.


DOCKER_HOST with docker-socket-proxy

I'm running Beszel in Docker. Everything works if I mount the docker socket but if I use a socket proxy and add DOCKER_HOST: tcp://socket-proxy:2375 the agent reports the error;
Closing idle connections. Error: Get "[http://localhost/containers/json":](http://localhost/containers/json%22:) dial tcp: lookup socket-proxy on <IP_ADDRESS>:53: server misbehaving
Could this be because the hub is on a separate network so it can't communicate via docker container names like socket-proxy:2375. If I use the containers IP it works <IP_ADDRESS>:2375.
Is this the intended behaviour or am I doing something wrong?
Here's my compose;
services:
  beszel:
    container_name: beszel
    image: henrygd/beszel:latest
    restart: unless-stopped
    networks:
      - t3_proxy
    security_opt:
      - no-new-privileges:true
    ports:
      - "2183:8090"
    volumes:
      - /appdata/beszel:/beszel_data
    extra_hosts:
      - host.docker.internal:host-gateway

  beszel-agent:
    container_name: beszel-agent
    image: henrygd/beszel-agent
    restart: unless-stopped
    network_mode: host
    environment:
      PORT: 45876
      KEY: "xxxxxxxxxxxxxx"
      FILESYSTEM: /dev/sda3
      DOCKER_HOST: tcp://socket-proxy:2375 # NOT WORKING
      # DOCKER_HOST: tcp://<IP_ADDRESS>:2375 # WORKING

  socket-proxy:
    container_name: socket-proxy
    image: tecnativa/docker-socket-proxy:latest
    restart: always
    networks:
      socket_proxy:
        ipv4_address: <IP_ADDRESS>
    ports:
      - "<IP_ADDRESS>:2375:2375"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
    environment:
      - LOG_LEVEL=info
      - EVENTS=1
      - PING=1
      - VERSION=1
      - AUTH=0
      - SECRETS=0
      - POST=1 # Watchtower
      - BUILD=0
      - COMMIT=0
      - CONFIGS=0
      - CONTAINERS=1 # Traefik, portainer, etc.
      - DISTRIBUTION=0
      - EXEC=0
      - IMAGES=1 # Portainer
      - INFO=1 # Portainer
      - NETWORKS=1 # Portainer
      - NODES=0
      - PLUGINS=0
      - SERVICES=1 # Portainer
      - SESSION=0
      - SWARM=0
      - SYSTEM=0
      - TASKS=1 # Portainer
      - VOLUMES=1 # Portainer

Could this be because the hub is on a separate network so it can't communicate via docker container names like socket-proxy:2375. If I use the containers IP it works <IP_ADDRESS>:2375.

That appears to be the case. I don't think docker containers can resolve other containers by name unless they're in the same network. And the agent is in host network mode, so it's not in the same network as your proxy despite being in the same compose file.
Using the IP address looks like a good workaround though.

Closing this issue, but let me know if you have any other questions.


Infinitely looping nodes

still possible + the whole app crashed

Unresponsive Nodes is currently expected behaviour, they cannot be moved because touching them would result in displaying their infinite regular expression. That's because circles are in fact allowed, but only if they do not contribute to the regular expression. However the app just crashing is a problem.

Maybe that was just caused by my escalating behavior for testing node editors. But as you can see in the first image: The editor freezes in the state of connecting the nodes.

I was able to crash the application using this node setup. Let's dive into the code again

Ok I think I found the error. The algorithm works, but it is too generous, thus the detection takes too long. With the mentioned setup, it increases (probably exponentially) in computation time, depending on the node graph complexity. Lemme fix that

The "cycle detection" works by having a cost attached to each node while generating the regular expression, and cycles being infinite will always exceed the maximum cost. This means that overly complex (non-cyclic) nodes also will be rejected, and the message text currently incorrectly says it's because of cycles in the node graph. This will be improved later but it has a low priority.


Record for tenant hosted in France

When will it be possible to record meeting for tenant hosted in France ?
Thank you in advance :)

thank you for submitting feedback.
I think the best way forward is if you open a service ticket in your tenant so this can get resolved ASAP. Based on the outcome let me know if it is something that can be called out in the docs.
Please follow this link to contact support for business products:https://docs.microsoft.com/office365/admin/contact-support-for-business-products
Please keep us posted here on the resolution so we can feed whatever information you discover into the content.


add fixture 'stairville/generic-rgb-par'

add fixture 'stairville/generic-rgb-par'
update register.json

Thank you @null!

Export files validity
(Output of test script tests/github/exports-valid.js.)
Test the exported files of selected fixtures against the plugins' export tests.
You can run a plugin's export tests by executing:
$ node cli/run-export-test.js -p <plugin name> <fixtures>
Fixture stairville/generic-rgb-par changed in this PR

d-light

attributes-correctness

:white_check_mark: stairville/generic-rgb-par-Standard.xml

ecue
ofl
qlcplus

xsd-schema-conformity

:x: Stairville-Generic-RGB-PAR.qxf
Show errors

Element '{http://www.qlcplus.org/FixtureDefinition}Dimensions', attribute 'Width': '0' is not a valid value of the atomic type 'xs:positiveInteger'.
Element '{http://www.qlcplus.org/FixtureDefinition}Dimensions', attribute 'Height': '0' is not a valid value of the atomic type 'xs:positiveInteger'.
Element '{http://www.qlcplus.org/FixtureDefinition}Dimensions', attribute 'Depth': '0' is not a valid value of the atomic type 'xs:positiveInteger'.
Element '{http://www.qlcplus.org/FixtureDefinition}Dimensions', attribute 'Weight': [facet 'minExclusive'] The value '0' must be greater than '0'.
Element '{http://www.qlcplus.org/FixtureDefinition}Dimensions', attribute 'Weight': '0' is not a valid value of the atomic type '{http://www.qlcplus.org/FixtureDefinition}weightType'.

Thanks for creating this fixture to whoever it was! :+1:
We do have Generic fixtures for cases like these, but our Generic RGB Fader lacks the dimmer ability. It'll be a good idea adding a mode with an Intensity channel to that fixture or creating a new generic dimmable fixture, but I can't merge this PR as is.
I created the issue #318 for this problem – if the author of this PR who didn't want to unveil his identity reads this: Feel free to fork this repository, create a generic fixture and contribute to this project!


Wrong insert SQL command order

The schema has following structure:
User <--Join Inheritance-- Store <--OneToMany/ManyToOne-- Stock  --ManyToOne/undirectional--> Address.
Right insert order is: User, Store, Address, Stock
But doctrine tries to insert objects in this order: Address, Stock, etc. In Stock insertion SQL command fails because store_id is NOT NULL field.
It seems like it doesn't take into account that Stock's store_id field  has NOT NULL flag so Store should be created before Stock.

A test case is needed here.

Maybe related to #8349 and all its previous issues. Closing due to lack of details.


Doesn't block CoinNebula

Hi!
NoCoin v0.4.14 for Chrome is not blocking CoinNebula, at least in powvideo.net.
Regards!

Thanks for the report. I found an example video and saw some mining ressources loaded. Will add the following URIs to the blacklist soon.
*://*.1q2w3.website/lib/*
ws://*1q2w3.website/proxy*
wss://*1q2w3.website/proxy*

Cheers!

Awesome! Thanks for your reply.
Cheers! :)

@keraf Shouldn't *:// cover all URIs? http,https,spdy,http2, ws:// as well as wss:// too?


[feature] implement squash function by selecting commits to squash

Issue for https://github.com/Murmele/Gittyup/discussions/434
It would be awesome to have a simple squash functionality by selecting multiple commits.
You would select (with shift+click) the commits you want to squash, then squash them.

squash is the basic functionality of working with commits


Move data initialization out from notebooks into separate py file

I did a quick implementation in my branch's data.py. It works now. Tomorrow (Mon 11/7) I will clean it up and then request a pull.


coordinates lost after uploading obs with private geoprivacy

Make an photoless obs with coordinates and set geoprivacy to private. Should look fine before you upload
Upload
Look at obs detail and obs edit again in the app: no coordinates. On the website it looks fine.

Build 205.

Dang it sorry, wrong repo.


✨ Sidebar should be revealed when mousing to edge of screen in full screen when hidden

Description
When the user takes CodeEdit full screen, after hiding the sidebar, it should temporarily reveal the sidebar while hovering over the sidebar. When mousing out it should disappear.
Additional Context
I am not sure if this is a built-in feature or it is custom but it seems to happen to all of Apple's apps like Finder, Mail and Calendar.

[!NOTE]
This works as expected (at least for the left sidebar) if the new windowing mode feature flag is on, but this breaks other things such as the Source control navigator list item heights are now too tall and file saving seems to break. Ideally we could get the new windowing mode working properly.

Screenshots
https://github.com/CodeEditApp/CodeEdit/assets/806104/594d30d0-bd66-4061-8faf-9b2e845f22b3
https://github.com/CodeEditApp/CodeEdit/assets/806104/c24e6172-34d0-48cc-87d2-b12a4e8dfbfe

Just thinking about it, this may be an issue with CodeEditTextView or CodeEditSourceEditor.


added Copyright queston to commit checklist

Fixed issue #937

[x] Is it mergeable?
[x] Did it pass the tests?
[x] If it introduces new functionality in scripts/ is it tested?
Check for code coverage with make clean diff-cover
[x] Is it well formatted? Look at make pep8, make diff_pylint_report,
make cppcheck, and make doc output. Use make format and manual
fixing as needed.
[x] Did it change the command-line interface? Only additions are allowed
without a major version increment. Changing file formats also requires a
major version number increment.
[x] Is it documented in the ChangeLog?
http://en.wikipedia.org/wiki/Changelog#Format
[x] Was a spellchecker run on the source code and documentation after
changes were made?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Ok to test, Jenkins.

Jenkins, ok to test

Otherwise this looks good to me, thanks!

@mr-c Ready for review!

@sguermond Congratulations on your first commit to the khmer project! Your name will be included in the release notes for the next version and you'll be listed amongst our other contributors in the next software release paper.


Question: What is the status of System.Reflection.TypeLoader?

Hi.
Can you let me know the status of System.Reflection.TypeLoader?  There doesn't seem to have been any activity in the last two years.
Is it acceptable to use this library in an application? If so, where can I find the package?  I checked the CoreFxLab NuGet feed and the package doesn't seem to be present.
Thanks
-marc

Consider this question closed.  I have discovered that this functionality will be in the System.Reflection.Metadata namespace in .NET 5.


[1pt] PR: Adding step to trim catchments from levelpath

Addressing #560 , this fix in run_by_branch clips catchments which do not intersect the NWM levelpath they were derived for.
Highlights:

No modification to I/O data formats
slight changes to outputs: smaller extent of inundation
Extra intermediates generated:

toy vrt called $branch/trim_catch
a copy of the catchments prior to trimming called &branch/gw_catchments_reaches_untrimmed$current_branch_id.gpkg

Cleanup:

[x] Matched code to style of repo
[x] verified across multiple catchments including NY, HI, and AL
[ ] determined if results impact FIM performance scores
[ ] Explore the dropping of very small segments

Geospatial context:

Known limitations:

As outlined above, it is uncertain that this modification improves or degrades FIM performance but should represent an improvement in process.
Small(er) branches still exercise a fairly large spatial footprint
Clipping uses st_intersects, which has known issues with geometry precision and subsequently may include catchments at the boarders where a true intersection would not. ex: the extra green catchment in the purple branch on the right side of the watershed.  The alternative (to break this off into a separate process and buffer the linestring geo), is probably a bit excessive.

During merge with Dev (which now included the new 'branch 0" system, we had to move the new "trim catchments" code from run_by_branch.sh to delineate_hydros_and_produce_HAND.sh.

After much consternation and iteration, here's that solution.
Highlights:

trim_catchments.py was dropped in favor of modifying split_flows.py.
all processing was relocated to delineate_hydros_and_produce_HAND.sh
No modification to I/O data formats, built off last dev push so you should be able to roll this in without conflict
Changes to outputs: smaller extent of inundation
split_flows.py now requires an extra input flag, (-n --nwm-streams-filename) for the nwm_subset_streams_levelPaths_ geopackage that contaions 'levpa_id' attribute
Extra intermediates generated: none
Time added to processing: minimal but untimed, this code is only hit when processing branches higher than 0

Cleanup:

[x] Matched code to style of repo
[x] verified across multiple catchments including NY, HI, and AL, one huc 6 and multiple huc 8's
[ ] determined if results impact FIM performance scores
[ ] Explore the dropping of very small segments

Geospatial context:
Left: DEM streamlines and branch envelopes -- Center: Branch 0 outputs -- Right: Overlaid branch catchments

Known limitations:

As outlined above, it is uncertain if this modification improves or degrades FIM performance but should represent an improvement in process.
There are several assumptions made in this particular iteration, the most impactful being that:

I did it the right way :)
The streamlines (both NWM and DEM_derived) are digitized such that the first point is the headwater and the tail is the outlet.  This allows me to assume that, when dissolved by levpa_id the last point in the line is the most downstream point.
Along those lines, I have assumed that none of my operations (dissolve, split) alter linestring order.
Python precision issues required me to buffer that point to split.  While the code should handle edge cases, this has not yet been tested at scale.

Small(er) branches still exercise a fairly large spatial footprint, and seem to be far larger than the FR catchment sizes generated in branch 0.

Let me know if you have any questions.

I (Rob) was bad and renamed a enum class. Was named "FIM_system_exit_codes" and I changed it to FIM_exit_codes.   But this won't be seen until my PR (coming today or tomorrow) for gms_pipeline comes in.  We will need to make 3 minor edits to your split_flows.py but the question will be when. (depends on who gets merged first).  If you get merged first, then I will change it in my PR. If I get merged first, then we need to change yours.

Sorry Jim.  I found a fix that is needed.  Granted it happened when I ran HUC 07080101 with all stream orders (aka.. no filtering), but I feel pretty sure that it would show up in other HUCs even with stream orders 3 and higher.
The error:
##########################################################################
Split Derived Reaches 07080101<PHONE_NUMBER>
##########################################################################
Fri Jul 29 21:33:34 UTC 2022
Loading data ...
trimming DEM stream to NWM branch terminus
Traceback (most recent call last):
File "/foss_fim/src/split_flows.py", line 306, in
split_flows(max_length, slope_min, lakes_buffer_input, **args)
File "/foss_fim/src/utils/shared_functions.py", line 247, in wrapper
func(*args, **kwargs)
File "/foss_fim/src/split_flows.py", line 112, in split_flows
snapped_point['geometry'] = snapped_point.apply(lambda row: target_snapline.interpolate(target_snapline.project( row.geometry)), axis=1)
File "/usr/local/lib/python3.8/dist-packages/geopandas/geodataframe.py", line 850, in setitem
value = _ensure_geometry(value, crs=self.crs)
File "/usr/local/lib/python3.8/dist-packages/geopandas/geodataframe.py", line 42, in _ensure_geometry
out = from_shapely(data, crs=crs)
File "/usr/local/lib/python3.8/dist-packages/geopandas/array.py", line 166, in from_shapely
return GeometryArray(vectorized.from_shapely(data), crs=crs)
File "/usr/local/lib/python3.8/dist-packages/geopandas/_vectorized.py", line 104, in from_shapely
arr[:] = data
ValueError: could not broadcast input array from shape (0,) into shape (1,)
Command exited with non-zero status 1

the commands I ran:  (both missing the -s switch)
-- gms_run_unit.sh -n rob_jim_4_07080101 -u 07080101 -c /foss_fim/config/params_template.env -j 40 -d /foss_fim/config/deny_gms_unit_default.lst -o
-- gms_run_unit.sh -n rob_jim_4_07080101 -u 07080101 -c /foss_fim/config/params_template.env -j 40 -d /foss_fim/config/deny_gms_unit_default.lst -o

IMPORTANT: Do not deleted branch if this gets merged before a different issue is resolved.
In testing this issue, it exposed another issue with run_test_case.py (as part of alpha testing).  It may be a bit hard to replicate the bug, so lets keep the branch (and this current commit) so the error can be replicated.  The new issue number is 647.

I also just found this error happen when doing GMS for streams 3+ in HUC 01090004. It looks like a coastal HUC, so it's a tricky one. I think the right decision is to let this error happen and to define an exit code for this case like we discussed.
We will address this HUC boundary overlap bug at the start of \gms\run_by_unit.sh when the NWM streamlines are subset in to the HUC8. That way, there won't be any level path defined for these streams and the HAND catchments will be associated with the proper streams. I opened a new issue #650 that Matt will be addressing.

@JamesColl-NOAA I did find a an error that needs to be addressed. It appears to happen a few times in HUC 11110202 in branches 2, 3, 7, and 8. Please take a look at these and make a determination of what's causing the error. I ran the testing domain and the outputs are here if you want to take a look:
/data/outputs/_fim4_catch_trim/branches/11110202
##########################################################################
Split Derived Reaches 11110202<PHONE_NUMBER>
##########################################################################
Wed Aug  3 19:30:35 UTC 2022
Loading data ...
trimming DEM stream to NWM branch terminus
Traceback (most recent call last):
  File "/foss_fim/src/split_flows.py", line 306, in <module>
    split_flows(max_length, slope_min, lakes_buffer_input, **args)
  File "/foss_fim/src/utils/shared_functions.py", line 247, in wrapper
    func(*args, **kwargs)
  File "/foss_fim/src/split_flows.py", line 112, in split_flows
    snapped_point['geometry'] = snapped_point.apply(lambda row: target_snapline.interpolate(target_snapline.project( row.geometry)), axis=1)
  File "/usr/local/lib/python3.8/dist-packages/geopandas/geodataframe.py", line 850, in __setitem__
    value = _ensure_geometry(value, crs=self.crs)
  File "/usr/local/lib/python3.8/dist-packages/geopandas/geodataframe.py", line 42, in _ensure_geometry
    out = from_shapely(data, crs=crs)
  File "/usr/local/lib/python3.8/dist-packages/geopandas/array.py", line 166, in from_shapely
    return GeometryArray(vectorized.from_shapely(data), crs=crs)
  File "/usr/local/lib/python3.8/dist-packages/geopandas/_vectorized.py", line 104, in from_shapely
    arr[:] = data
ValueError: could not broadcast input array from shape (2,) into shape (1,)

After the test domain run and a tag up with @CarsonPruitt-NOAA, it was determined that I continue to overcomplicate this.  GMS allows for out-of-catchment flooding, and prior to splitting at the watershed bounds both the NWM levelpath and DEM derived reach are singular features.  The extra slicing and error checking I was previously performing was unnecessary and has been removed here.  This update should remove all errors with split flows experienced in the test domain, and masks the errors discovered in #650.  This was tested by running the split_flows.py modification over all failed branches in the _fim4_catch_trim outputs.


Cleanup dependencies

I looked through crates where we compile two versions and changed this:

Remove BigDecimal where it isn't needed and import types through num
instead when possible
Remove default warp features (multipart and websocket)

The BigInteger conversion can go away once sqlx updates their BigDecmial dependency which I have made a PR for.
Test Plan
CI

Codecov Report

Merging #1361 (d8f1427) into main (bd346c5) will decrease coverage by 0.01%.
The diff coverage is 96.15%.

@@            Coverage Diff             @@
##             main    #1361      +/-   ##
==========================================
- Coverage   66.71%   66.69%   -0.02%
==========================================
  Files         140      140
  Lines       29767    29766       -1
==========================================
- Hits        19858    19853       -5
- Misses       9909     9913       +4


Add support for SSL Client Authentication

Currently phantomjs is not able to provide a client certificate for secured pages.
I think the QT code is able to create SSL sockets using a client key and certificate.
The only missing part is some configuration options and some handling to feed the QSslConfiguration.
I propose some configuration options like:
--ssl-client-certificate-file Similar to --ssl-certificates-path, but should just point to a single certificate file. As PEM seems to be used as format, the certificate file might contain a secret key as well.
--ssl-client-key-file Needed in case client certificate file does not contain a key.
--ssl-client-key-passphrase The passphrase to decode the secret key.

@ariya any reason not to merge this? Would love to see this in the next release of PhantomJS!

Error: Unknown option: ssl-client-certificates-file  for windows build with ffray changes on latest phantomjs

Unfortunately without this feature I cannot use PhantomJS on my project.

Hi, can someone review the code and merge this PR please!! It's more then 2 years since it has been submitted. @ariya Can you have a look?

Thank you! Landed in https://github.com/ariya/phantomjs/commit/37a3c2dd22838e5b88c7163c499d4c0f4dae3ff4!

Could you please add an example to use the feature ?

Still not merged into main branch, no releases with this patch to get it tested either.. any chance someone could compile this and let us test it?

It is in the master branch - https://github.com/ariya/phantomjs/commit/e4c0d9fa5c3da1148d17c758e2f6cc54560ba4ca

With the latest windows download (downloaded earlier today) of version 2.0.0. using these new options generates errors: "Error: Unknown option: ssl-client-certificates-file", the options also do not display with --help parameters and neither are they listed on http://phantomjs.org/api/command-line.html
The file was compiled on 4th february this year - i'll guess the compiled versions have not yet been updated? is there a bleeding edge release i can find anywhere?

@JernejL yes, this feature will be in 2.0.1 release

Sorry for being a pain, but is there any chance to get 2.0.1. earlier? i would really need it and will be able to help out with testing the features before release too.

Vitallium: thank you, i've done some testing and came to some possible bug / issue:
2015-08-07T09:19:20 [DEBUG] Network - Resource request error: 99 ( "Unable to init SSL Context: " ) URL: "https://removed"
command line:
phantomjs.exe --ignore-ssl-errors=true --cookies-file=R:\phantom\bin\cookies.txt --ssl-certificates-path=R:\phantom\bin --ssl-client-certificate-file=R:\phantom\bin\cert.crt --debug=true test.js
debug shows the ssl section properly:
2015-08-07T09:26:09 [DEBUG]      18 sslProtocol : "default"
2015-08-07T09:26:09 [DEBUG]      19 sslCiphers : "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-RSA-RC4-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DH
E-RSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:RC4-SHA:RC4-MD5"
2015-08-07T09:26:09 [DEBUG]      20 sslCertificatesPath : "R:\phantom\bin/*"
2015-08-07T09:26:09 [DEBUG]      21 sslClientCertificateFile : "R:\phantom\bin\cert.crt"
2015-08-07T09:26:09 [DEBUG]      22 sslClientKeyFile : ""
2015-08-07T09:26:09 [DEBUG]      23 sslClientKeyPassphrase : ""
User and all other certificates are base64 encoded PEM files.
I've followed the error to this:
http://stackoverflow.com/questions/14989135/qt-downloading-a-file-doesnt-work
And then downloaded this + put it into phantomjs bin folder:
https://code.google.com/p/openssl-for-windows/downloads/detail?name=openssl-0.9.8k_WIN32.zip&can=2&q=
I copied the dlls and it did not appear to change anything.
Using process monitor i can see phantomjs reading the certificate file i specified, but it does not appear to attempt to read any other ssl certificates in the directory.
However in debug one thing stands out - "sslCertificatesPath : "R:\phantom\bin/*"" shows a forward slash instead of backslash in the end, if i specify --ssl-certificates-path=R:\phantom\bin or --ssl-certificates-path=R:\phantom\bin\ in both cases it'll output a forward slash in the debug output  - perhaps this could be a issue / bug in windows build and causes it to not be able to iterate the target directory for certificates?

Could you please file a new issue?

Done: https://github.com/ariya/phantomjs/issues/13472


WIP adding support for running spec tests against jruby in travis

Finished in 20 minutes 16 seconds
22466 examples, 406 failures, 42 pending

Finished in 19 minutes 14 seconds
22351 examples, 293 failures, 42 pending

CLA signed by all contributors.

Previous commit took care of all the obvious fork cases.
Finished in 31.22 seconds (files took 8.76 seconds to load)
Finished in 18 minutes 47 seconds
22269 examples, 221 failures, 57 pending

previous commit took care of tests that require SSL things
Finished in 21.68 seconds (files took 6.1 seconds to load)
Finished in 18 minutes 54 seconds
22113 examples, 73 failures, 58 pending

Now at
Finished in 34.79 seconds (files took 8.08 seconds to load)
Finished in 20 minutes 1 second
22101 examples, 66 failures, 58 pending

Finished in 23.42 seconds (files took 6.74 seconds to load)
Finished in 15 minutes 48 seconds
22101 examples, 64 failures, 58 pending

Finished in 30.42 seconds (files took 6.69 seconds to load)
Finished in 17 minutes 14 seconds
22096 examples, 51 failures, 58 pending

Finished in 2 minutes 17.1 seconds (files took 8.44 seconds to load)
Finished in 18 minutes 29 seconds
22096 examples, 33 failures, 74 pending

@melissa @jhelwig  someone will have to take over this PR, this is the latest of what I have done.

New WIP PR: https://github.com/puppetlabs/puppet/pull/6790


Movie not playing

Issue description
After attempting to upload and play a movie (not happening with TV series), the movie does not play, displaying a "Internal Server Error" message in a red popup.
Inspecting the HTTP request reveals that the failing request is made to the URL/video/show.json?id=1. Interestingly enough, requests to /movie/show.json?id=1 are evaluated correctly (only on other pages, it does not happen in the player). Stack trace is at the end of this report.
When assigning the same file to a TV series episode, the file is played correctly, and requests to /video/show.json?id=3 do not fail.

Sometimes the connection to the DB times out as well, with the following message in MariaDB:
[Warning] Aborted connection 34 to db: 'streama' user: 'streama' host: '<IP_ADDRESS>' (Got timeout reading communication packets)

The result of that is the whole application responding with {"message":"Internal server error","error":500} to ANY HTTP request, causing the web interface to not show anymore either. I forgot to save the stack trace of that, however I remember seeing: com.mysql.cj.exceptions.CJCommunicationsException: The last packet successfully received from the server was 636,743 milliseconds ago.  The last packet sent successfully to the server was 636,743 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
Steps to Reproduce

Create a movie
Upload a file for the movie
Attempt to play the movie

Expected Behaviour
Movie playing (without any other unexpected behavior)
Actual Behaviour
Movie does not play; sometimes DB connection times out, doesn't reconnect, causing the whole server to respond with 500 to anything.
Environment Information

Operating System: Docker container of 'gkiko/streama' (Linux container, host OS irrelevant)
Streama version: v1.6.0-FINAL
DB: MariaDB 10.3.12

Here's the docker-compose.yml used:
version: '2'

services:
    streama:
        image: 'gkiko/streama:v1.6.0-FINAL'
        working_dir: '/app'
        entrypoint: 'java'
        command: ["-Dgrails.env=mysql", "-jar", "streama.war"]
        hostname: streama
        ports:
            - 8080:8080
        environment:
            - ACTIVE_PROFILE=mysql
            - MYSQL_HOST=mysql
            - MYSQL_DATABASE=streama
            - MYSQL_USER=streama
            - MYSQL_PASSWORD=streama_password
        restart: always
        volumes:
            - ./data:/data/streama
        links:
            - mysql
        depends_on:
            - mysql
    mysql:
        image: 'mariadb:10'
        hostname: mysql
        restart: always
        volumes:
            - ./mysql_data:/var/lib/mysql
        environment:
            - MYSQL_ROOT_PASSWORD=password
            - MYSQL_DATABASE=streama
            - MYSQL_USER=streama
            - MYSQL_PASSWORD=streama_password
        expose:
            - 3306

Stack trace
https://pastebin.com/3Jj5mk84

thanks for looking into this!
Can you reproduce this on the demo-version of the streama as well? https://demo.streamaserver.org/login/auth
that way, i could look into the stacktrace as well

Sorry I forgot about this one! I can attempt replicating it today, I'll report back with any potential findings.
What database does the demo server run on? If it's H2, I'd assume this issue will not be replicable there

Hello
I am also having this issue. I am running this on Docker with the same docker-compose.yml posted above (with a few extra labels). This happens for any action (not just watching a movie) and only happens after around 20 minutes of the container running.
I have added a dirty fix by using an autoheal container alongside this one and adding a healthcheck to the streama docker-compose.yml but obviously this isn't ideal.

Figured this out.
I noticed I wasn't using the latest image so I changed my image in my docker-compose.yml to image: 'gkiko/streama:latest'.
Also changed the ACTIVE_PROFILE environment variable to docker rather than mysql and changed the command to command: ["-Dgrails.env=docker", "-jar", "streama.jar"] (notice since it's using the latest version it uses .jar rather than .war and the -Dgrails.env is set to docker now.
I also mounted a volume for the database because it won't be using the mariadb provided in the docker-compose.yml anymore.
That's it - no more 500 errors after 20 mins of runtime. Hope this helps anyone else using the same docker-compose file.


version v2.11.0 doesn't support condition for chart requirements.yaml

Output of helm version:
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

Output of kubectl version:
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T22:29:25Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:05:37Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.):
Docker CE 18.06.1-ce-mac73 (26764)

BTW, version v2.2.2 worked.

Did you mean to say 2.12.0?

Did you mean to say 2.12.0 instead of 2.2.2?

No, I mean v2.2.2.

Can you please provide an example chart that doesn’t work?

Hey @zzxwill - could you give us some details on how to reproduce this issue. I was unable to reproduce this issue.
These are the steps that I took:

Added a requirements.yaml file with a condition specified in the conditions section of the docs to a simple chart with a key that evaluates to false in the values.yaml file

ran $ helm dep up to get the dependency in the charts/

ran $ helm install mychart/ -n example and I did not see the dependency be installed because the condition evaluated to false in the values.yaml

Then I ran $helm install mychart/ -n example2 --set subchart.enabled=true to make sure the condition field as expected and my dependency was installed along with the parent chart as expected.

Please let us know if there are different steps we should follow to reproduce the issue you're seeing.
Thank you.

We are also running into this issue with v2.12.2. After running helm dep up, the condition field from the requirements.yaml is not present in requirements.lock. Can provide instructions to reproduce the issue in a bit

closing as inactive.


multiple destinations

Hey...
I have no idea how to code, but I've been looking for something like this app for a while...
You might not know, but if you search for multiple destinations, and start or end your trip in a random place, you can end up with much cheaper flights.
Eg.
HKG-PRG-HKG Dec 21-Jan 2 on British airways... HKD5,600
KUL-HKG in October, then HKG-PRG-HKG Dec 21 - Jan 2... HKD 4,563
Also, Kayak runs searches if you just input the correct URL, without needing to fill in a form on the actual page to run the search....
I'm trying to work out how to automatically run a bunch of searches, with randomised one way flights either before or after the "main trip", then dump the results into an Excel document for easy comparison.
What I'm doing at the moment is running a script in VBA which opens a heap of internet explorer windows with a different search in each window.
Any advice you can give would help a lot!
Ps. Sorry for doing this as an 'issue'...i couldn't find your email address.

I like the idea about writing the results to an excel document, I think I'll start with that(once I have time)
I'm not sure how your suggestion would work, in your example you want to travel HKG-PRG, instead you book KUL-HKG and then HKG-PRG-HKG in the same booking? can you send me an example url of a page with that search?

Thanks for getting back to me...
Here's some URLs with different searches...
https://www.kayak.com.hk/flights/Sin-HKG/2017-09-29/HKG-PRG/2017-12-21/PRG-HKG/2018-01-02/2adults/children-11
https://www.kayak.com.hk/flights/kul-HKG/2017-09-29/HKG-PRG/2017-12-21/PRG-HKG/2018-01-02/2adults/children-11
https://www.kayak.com.hk/flights/Bkk-HKG/2017-09-29/HKG-PRG/2017-12-21/PRG-HKG/2018-01-02/2adults/children-11
On 21 Mar 2017, at 04:09, Shadi-A <notifications<EMAIL_ADDRESS>wrote:
I like the idea about writing the results to an excel document, I think I'll start with that(once I have time)
I'm not sure how your suggestion would work, in your example you want to travel HKG-PRG, instead you book KUL-HKG and then HKG-PRG-HKG in the same booking? can you send me an example url of a page with that search?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/Shadi-A/ExpediaCrawler/issues/1#issuecomment-287882301, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AZSolzV38kcOpKq9LYdycyUY7qCAHMo7ks5rntzwgaJpZM4MhsLH.

I made a change to the script, there is an optional parameter --file that you can use to append the output to a csv file, see the updated readme file for how to use it.