Bug: Connection terminated unexpectedly (postgres)

Environment
Knex version: 0.20.1
Database + version: postgres:11.5
OS: Alpine Linux
Additional information:

Database: I've confirmed the bug to happen in postgres versions at least up from 9.5. The database is running on a container, currently postgres:11.5.
Knex environment: Knex is currently running on node v12, though I've confirmed the bug to happen on v8 and v10 too. It's running on a container, currently node:12.13.0-alpine. The current pg version is 7.12.1; though I've seen this happen with previous versions, I haven't documented this bit.
Knex configuration object:

({
  client: 'pg',
  connection: 'postgres://user:password@server:5432/db',
  pool: { min: 2, max: 10 }
})

Bug
I am getting a Connection terminated unexpectedly error in production, after long periods of inactivity (think +1h) during which no queries are made, causing the first query after that period to fail. Any query made after that first failing query will succeed.
This was first reported on issue #3258. In that issue, it was reported happening on knex 0.16.5, so this has been going on at least since then. It was also stated that a downgrade to 0.13.0 fixes the issue, which met the suggestion that maybe newer knex versions logged the error while previous ones didn't. I've confirmed this not to be the case. Downgrading to knex 0.13.0 does in fact fix the issue, with queries being successful after long periods of inactivity.
A typical error message will look like this:

message: select \"user\".* from \"user\" where \"email\" ilike $1 - Connection terminated unexpectedly
stack: Error: Connection terminated unexpectedly\n at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:252:9)\n at Object.onceWrapper (events.js:299:28)\n at Connection.emit (events.js:210:5)\n at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:77:10)\n at Socket.emit (events.js:210:5)\n at TCP.<anonymous> (net.js:658:12)

node-postgres #1324 might help providing additional insight on the matter. #3046 might also be related.

Would be good if you can provide test code to verify that. For example simple example project with docker-compose file and knex app that waits long enough to encounter that error.
We have got various reports of that bug, but no-one has been able to show if the problem is in user's code or in knex.
If you can catch that error with env variable DEBUG=knex:* set it would also be helpful to verify that the error is not thrown from transaction that has been left open accidentally etc.

I have good news! After further investigating the issue, I believe I can point towards a solution -I'll leave that for last, as there's a bit to unpack.
Unfortunately, I haven't been able to reproduce the error locally, even with the same setup and compose configuration as the remote server. It is quite a strange thing, as the only difference I can appreciate between both environments is I'm running docker-compose locally, while the server is running a docker swarm. To reiterate, I've set up a repo that reproduces this error on a remote production server, but (somehow) not locally. I will post the logs below (last), so for the purpose of understanding them, I've made the repo available here. As you can see there, I'm making a first successful query right after the process starts, making a 2nd query after 20 minutes (which fails), and a 3rd query immediately after (which succeeds).
Some additional info, just for the sake of future reference:

It seems the idle time needed for the connection error to happen after querying is somewhere between 15 and 20 minutes. Queries after 15 minutes of inactivity will succeed; while after 20, they will fail.
knex 0.13.0 doesn't show this issue, while every version after, including 0.14.0, does. However, on 0.14.0 a new connection won't be acquired on the next query, so they will fail indefinitely, which is solved on 0.15.0.

And now, for the useful bit: it turns out pg is not emitting the connection error until after the query has been made, but before it has resolved. This is the sequence of events:

knex public api query is made.
knex's dialects/postgres/index.js _query is called with a pg connection, which calls pg's connection.query.
pg's connection emits an 'error' event, which is received by connection.on('error', cb) at knex's dialects/postgres/index.js.
pg's connection.query calls its callback with the same error that had just been emitted, hence knex's dialects/postgres/index.js _query rejects with said error.
pg's connection emits an 'end' event, which is received by connection.on('end', cb) at knex's dialects/postgres/index.js.
knex releases connection to pool.
The next time knex receives a query to be made, it will acquire a new pg connection (acquireRawConnection) and succeed.

So essentially, it could be argued that maybe there's an issue with the containerized production environment/setup, and/or with pg, which is not reporting a connection termination until the time at which a query is made -though I don't know whether reporting it earlier would be feasible. But regardless, there's a (very much fixable) issue with knex: if a connection errors out in the time that goes between a query is made and its resolution, the query will fail. I would imagine this would be the case not just with postgres, but with other dialects too.
Hence I would suggest that, in this event, a new connection is acquired and the query retried. It might make sense to pass that error (make the query fail with it), though, in the event of the connection having being acquired just before it was made. So, to recap, I'd propose that if a query errors out and the connection was just acquired for the purpose of the query, it rejects with the error, but otherwise it checks whether the connection has failed, and if so, acquires a new connection and retries the query.
Last, the logs. These are for v0.20.1, but I have logs for every minor version starting on 0.13.0.

First query, right after the process starts:

2019-11-09T09:22:02.057558944Z Elapsed: 0 ms
2019-11-09T09:22:02.058104661Z 2019-11-09T09:22:02.057Z knex:client acquired connection from pool: __knexUid1
2019-11-09T09:22:02.059072595Z {
2019-11-09T09:22:02.059140593Z method: 'select',
2019-11-09T09:22:02.059150803Z options: {},
2019-11-09T09:22:02.059156884Z timeout: false,
2019-11-09T09:22:02.059162835Z cancelOnTimeout: false,
2019-11-09T09:22:02.059169678Z bindings: [<EMAIL_ADDRESS>],
2019-11-09T09:22:02.059176932Z __knexQueryUid: '0ab6fbe6-f4d0-497a-87b0-057033964950',
2019-11-09T09:22:02.059183233Z sql: 'select * from "user" where "email" = ?'
2019-11-09T09:22:02.059191860Z }
2019-11-09T09:22:02.059483679Z 2019-11-09T09:22:02.059Z knex:query select * from "user" where "email" = ? trx2
2019-11-09T09:22:02.059543883Z 2019-11-09T09:22:02.059Z knex:bindings [<EMAIL_ADDRESS>] trx2
2019-11-09T09:22:02.063050947Z 2019-11-09T09:22:02.062Z knex:client releasing connection to pool: __knexUid1
2019-11-09T09:22:02.063380849Z Result []

Second query, after 20 minutes:

2019-11-09T09:42:02.067012525Z Elapsed: 1200000 ms
2019-11-09T09:42:02.071066942Z 2019-11-09T09:42:02.069Z knex:client acquired connection from pool: __knexUid1
2019-11-09T09:42:02.071878431Z {
2019-11-09T09:42:02.071909690Z method: 'select',
2019-11-09T09:42:02.071916663Z options: {},
2019-11-09T09:42:02.071921983Z timeout: false,
2019-11-09T09:42:02.071926913Z cancelOnTimeout: false,
2019-11-09T09:42:02.071931722Z bindings: [<EMAIL_ADDRESS>],
2019-11-09T09:42:02.071936431Z __knexQueryUid: '229e5c4a-9ed3-4d82-a25e-b048e7cb9c0d',
2019-11-09T09:42:02.071941269Z sql: 'select * from "user" where "email" = ?'
2019-11-09T09:42:02.071946579Z }
2019-11-09T09:42:02.072055515Z 2019-11-09T09:42:02.071Z knex:query select * from "user" where "email" = ? trx2
2019-11-09T09:42:02.072125507Z 2019-11-09T09:42:02.072Z knex:bindings [<EMAIL_ADDRESS>] trx2
2019-11-09T09:42:02.081042312Z 2019-11-09T09:42:02.080Z knex:client releasing connection to pool: __knexUid1
2019-11-09T09:42:02.082578957Z Error: Connection terminated unexpectedly
2019-11-09T09:42:02.082619874Z at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:252:9)
2019-11-09T09:42:02.082627337Z at Object.onceWrapper (events.js:299:28)
2019-11-09T09:42:02.082633188Z at Connection.emit (events.js:210:5)
2019-11-09T09:42:02.082638899Z at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:77:10)
2019-11-09T09:42:02.082645932Z at Socket.emit (events.js:210:5)
2019-11-09T09:42:02.082650612Z at TCP.<anonymous> (net.js:658:12)

Third query, right after the 2nd:

2019-11-09T09:42:02.084190783Z Elapsed: 0 ms
2019-11-09T09:42:02.084868650Z Connection Error: Connection ended unexpectedly
2019-11-09T09:42:02.097410334Z 2019-11-09T09:42:02.097Z knex:client acquired connection from pool: __knexUid3
2019-11-09T09:42:02.098051381Z {
2019-11-09T09:42:02.098071179Z method: 'select',
2019-11-09T09:42:02.098076118Z options: {},
2019-11-09T09:42:02.098080576Z timeout: false,
2019-11-09T09:42:02.098085165Z cancelOnTimeout: false,
2019-11-09T09:42:02.098089503Z bindings: [<EMAIL_ADDRESS>],
2019-11-09T09:42:02.098094572Z __knexQueryUid: '6e702fd9-b0d5-4c8e-9ee7-0b6404fec6d6',
2019-11-09T09:42:02.098099522Z sql: 'select * from "user" where "email" = ?'
2019-11-09T09:42:02.098104000Z }
2019-11-09T09:42:02.098367957Z 2019-11-09T09:42:02.098Z knex:query select * from "user" where "email" = ? undefined
2019-11-09T09:42:02.098583893Z 2019-11-09T09:42:02.098Z knex:bindings [<EMAIL_ADDRESS>] undefined
2019-11-09T09:42:02.103675485Z 2019-11-09T09:42:02.103Z knex:client releasing connection to pool: __knexUid3
2019-11-09T09:42:02.104059078Z Result []

Thank you very much. Those findings are really useful. Looks like things are happening as I expected so that error is not either happening or registered before it is taken away from pool and data is actually sent over the connection.
One can probably work around this issue by setting minimum pool size to 0 and set idle timeout to be like 5 minutes or so. In that way connections will be discarded from pool before that.

But regardless, there's a (very much fixable) issue with knex: if a connection errors out in the time that goes between a query is made and its resolution, the query will fail.

Actually connection error might happen at any time (or any other surprise error) and app should be written to handle unexpected errors causing queries to fail. Automatic retrying will not resolve it, except for some very specific cases, where connection error gets fixed automatically in some time frame and when transactions are not used.
So knex is intentionally not trying to retry failed queries automatically. Retrying would also cause other problems, like a delay for app to be able to notice that your DB connection is down (and after noticing you would still have to have that error management for unexpectedly failed query).

Hi, thank you for taking the time ;) In general, I can see where you're coming from, however:

Actually connection error might happen at any time (or any other surprise error) and app should be written to handle unexpected errors causing queries to fail. Automatic retrying will not resolve it, except for some very specific cases, where connection error gets fixed automatically in some time frame and when transactions are not used.

I'm failing to get the point here. When a connection error occurs, knex will discard it and acquire a new one, hence there's a expectation that connection errors (not other kind of query errors, but yes, connection errors) will be handled, when possible to reconnect, by knex. This case fits there, with the only difference that the connection happens to emit the error after the query has been made (but before it's been resolved). Hence my proposal to log the connection error, discard the query error, get a new connection, and retry the query -not for every case, but only for this particular case in which the connection has emitted an error after querying but before resolving. This seems the most consistent to me with the current behavior otherwise, given the above expectation.

Retrying would also cause other problems, like a delay for app to be able to notice that your DB connection is down (and after noticing you would still have to have that error management for unexpectedly failed query).

On most cases, I'd assume that the dialect's connection will emit a connection error at any time (so not during an ongoing query), in which case, the current behavior applies, so no harm done. As pointed out, this would only apply in cases when the connection error is emitted during a query, hence both because the # of times that would happen by random chance is statistically very low, and because the delay would be realistically unnoticeable (given it's not something that accumulates, but rather a singularly occurring event), I'd say the downside would be more than justified.

One can probably work around this issue by setting minimum pool size to 0 and set idle timeout to be like 5 minutes or so. In that way connections will be discarded from pool before that.

It might be sensible to set defaults that avoid this issue or, at least, document it in some way. But regardless, I am somewhat confused with the current defaults. I'm assuming when you say set idle timeout to be like 5 minutes or so you're referring to tarn.js idleTimeoutMillis, to be set in the pool key object within knex's configuration object. Unless knex is passing a different default value (is it?), idleTimeoutMillis seems to default to 30000ms, 30 seconds, which should have avoided this error altogether. Any insight on this?

I have explained why knex does not retry on connection errors or on any other errors so many times to so many people that I should probably just find a link to some older discussion 😬

'm failing to get the point here. When a connection error occurs, knex will discard it and acquire a new one, hence there's a expectation that connection errors (not other kind of query errors, but yes, connection errors) will be handled, when possible to reconnect, by knex.

If there is a connection error when connection is in pool we know that it is not used by anyone so it is owned by knex and we can do what ever we want to with it (actually we don't discard it until someone tries to get it from pool and validation phase sees that connection was broken while being in the pool).
Actually now that I think of it if I recall correctly even pool does not retry for connection errors. So if connection is broken when connection was in pool, then it will be discarded and fresh connection is returned instead, but if that fresh connection fails because of connection error, it will not be retried, but the error is passed to the user. This is also for the same reason, because we don't want knex to hide those connection errors which happen during queries.

Hence my proposal to log the connection error, discard the query error, get a new connection, and retry the query -not for every case, but only for this particular case in which the connection has emitted an error after querying but before resolving. This seems the most consistent to me with the current behavior otherwise, given the above expectation.

It doesn't make sense to make code more complicated and add single retry for this single case. Expectation is that when connection is got from pool it is usable... if there is connection error happening afterwards, then it is passed to the user. We don't want to mask errors, you still can have them coming at any time which is not even related to this issue and then you would need to retry for that case too (because it will be hard to distinct from other case). Also as I mentioned above you would still need to have error handling for the case where connection error comes and it takes longer that just one query or if it happens during transaction.
For this particular case I would love to find fix the root cause of the problem, but until then setting pool min to 0 should do just fine. It is pretty uncommon case where database is without use for hours and that you would still want to have pool populated with connections during that idle time.

On most cases, I'd assume that the dialect's connection will emit a connection error at any time (so not during an ongoing query), in which case, the current behavior applies, so no harm done. As pointed out, this would only apply in cases when the connection error is emitted during a query, hence both because the # of times that would happen by random chance is statistically very low, and because the delay would be realistically unnoticeable (given it's not something that accumulates, but rather a singularly occurring event), I'd say the downside would be more than justified.

I really didn't understand what you tried to say with that. Only benefit that I see with having that retrying logic is that it will work as a hack to kind of fix something that does not seem to be a knex bug. Preventing pool from keeping idle connections there for long periods of time is a lot better workaround for this (happens with some very specific environment).

Unless knex is passing a different default value (is it?), idleTimeoutMillis seems to default to 30000ms, 30 seconds, which should have avoided this error altogether. Any insight on this?

Pool min size 0 was the key part of that work around. If pool size min is bigger that 0 then idle timeout will not do anything after pool has already reached its minimum size.

Hi, thank you for taking the time of explaining the reasoning for this (yet again). I know it can get tiring; thank you.
From your previous statements I got the idea that passing errors and not retrying queries was intentional, but not necessarily not handling connection errors. Your latest explanation makes it clear that it is your intent that connection errors are explicitly managed by the user in all cases ("if connection is broken when connection was in pool, then it will be discarded and fresh connection is returned instead, but if that fresh connection fails because of connection error, it will not be retried, but the error is passed to the user"). I had previously assumed this wasn't the case. With that in mind, I think what you're saying is consistent with the expectations set, hence no changes should be made, as regardless of whether they happen during or before a query, they will be passed either way.
However, it might be that I didn't look carefully enough, but I think it would be very good if this information regarding the handling of broken connections vs. errored connections had a prominent place in knex's documentation in order for devs to have certainty on the expectations. On a side note, I also couldn't locate the default values for the configuration options in the docs, which would also be helpful.
Anyhow, I have confirmed that setting the pool.min to 0 prevents this from happening. I wonder though, why is it 0 not the default value?

I'm having the same problem. It's very hard to reproduce since it's only occurring on the system of my remote colleagues (no docker). I can run the server here and have it idle for hours and it still doesn't happen. Also no signs of it on our dockerized ECS instance. They all connect to the same Postgres server. I'll try the workaround.

However, it might be that I didn't look carefully enough, but I think it would be very good if this information regarding the handling of broken connections vs. errored connections had a prominent place in knex's documentation in order for devs to have certainty on the expectations. On a side note, I also couldn't locate the default values for the configuration options in the docs, which would also be helpful.

Certainly that would be good to have 👍

Same here. Local instance of nodejs12 + pg 12 on docker. Connection terminated if not used for approx 1 hour.

@iw2rmb please send test code for that... for example docker-compose file would be helpful. This problem has not been able to be reproduced outside of AWS lambda so this would make a huge difference for being able to resolve the bug. Though I'm pretty sure that error you are experiencing is actually not the same issue.... Is that terminated connection returned from pool or is it automatically refreshed before next query after 12 hours is sent to DB?

I'm experiencing similar issue with in a docker swarm cluster.
select "users".* from "users" where "email" = $1 limit $2 - Connection terminated unexpectedly
Using postgres 12, node 12.3.1 and knex 0.19.5.
knex configuration pool connections min:2, max:10
I have Sentry set up in my environment. I can match the timing of the error with several of my docker services.
postgres service reports nothing.
my api service logs knex connections.
Here's the log at that same time:
2020-03-12T13:04:18.716Z knex:client acquired connection from pool: __knexUid1
2020-03-12T13:04:18.716Z knex:query select "users".* from "users" where "email" = ? limit ? trx
12
2020-03-12T13:04:18.716Z knex:bindings [<EMAIL_ADDRESS>2 ] trx12
2020-03-12T13:04:18.719Z knex:client releasing connection to pool: __knexUid1
2020-03-12T13:04:37.964Z pg-listen:paranoid Paranoid connection check ok

It looks like the query was correctly executed, right?

@NelsonFrancisco I don't see anythin in logs which would refer that query was ran successfully... It could have as well got broken connection from pool try to run query and then return it.

I was assuming that if an error occurred, knex would report it.

If query fails for some reason, it is just normal behaviour and not specially logged. If you like to log results / errors you need to setup for example knex.on('query-error',... ) and knex.on('query-response',... ) event handlers.

you application logic should anyways have proper error handling for those cases, for example with REST API return some 500 error.

I agree. Currently, it is timing out, and returning 502 bad request. But it is very strange.
I'm using objection with knex. Every error that happens when I execute a query is properly catch in a middleware - and ultimately, if the error is not known, a fallback middleware returns 500.
I'm pretty sure the exception is not being thrown when I "await" for objection's "where" method.i.e, when I await knex's "select" method. Because if it would, then the API would catch the error. So, maybe the promise is taking too long OR it is being resolved and the error happens asynchronously

I'm getting the same problem in docker and Kubernetes environment.
When trying to connect to Postgres,  the first call failed, and then request immediately next call works.
{ Error: read ECONNRESET
at TLSWrap.onStreamRead (internal/stream_base_commons.js:111:27) errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'read' }
Connection Error: Connection ended unexpectedly

I'm getting the same problem in docker and Kubernetes environment.
The first call failed if few hours not use, and the next request works.
Connection Error: Connection ended unexpectedly

Similar issue happens to me. First call in a long time fails. I can't exactly say the time interval.

app.get('/all', function (req, res, next) {
knex.select().table('main').then(data => {
res.json(data);
}).catch(next);
});

In your case, is "next" callback executed?

@NelsonFrancisco  yes, the second call executed successfully.
The connection timeout interval here seems to be 20 min

@jamesdixon  guess James is the most experienced person on this issue

That's good, because the same happens to me

@NelsonFrancisco  I've set the min pool size to 0,  did a test again this morning, it seems works.
I don't see that error message from log again

@NelsonFrancisco I've set the min pool size to 0, did a test again this morning, it seems works.
I don't see that error message from log again

I can say, without a doubt, that fixed the problem.
One week and going without this issue

I hid bunch of comments about a separate issue, when that connection closed error is shown, but which didn't  cause any queries to fail (cases where broken connections were actually never returned from pool).

Any updated on this? I see this issue has been open for a year.
I'm facing the same problem while running a nodejs app in google cloud-run.
Queries are failing in production when the service has been left idle for a long time. The error message is
select ... - Connection terminated unexpectedly

Here are my connection pool settings for Knex.
pool: {
    min: 1,
    max: 5,
    acquireTimeoutMillis: 60000,
    idleTimeoutMillis: 600000,
}

I've solved it by setting minimal pool size to zero: pool: { min: 0, max: 10 }

@lynxtaa I changed my pool settings to the following and the error has not occurred in a couple days, so I guess this fixes the problem. I'll comment again if it resurfaces. Thanks.
pool: {
    min: 0,
    max: 5,
    acquireTimeoutMillis: 60000,
    idleTimeoutMillis: 600000,
}

Still an issue. I suggest that the docs be changed to reflect the suggestions given above.
My stack trace:
{"level":50,"time":1614497262294,"pid":1,"hostname":"843348d00255","reqId":194,"errno":-110,"code":"ETIMEDOUT","syscall":"read","stack":"Error: read ETIMEDOUT\n    at TCP.onStreamRead (internal/stream_base_commons.js:209:20)","type":"Error","msg":"select...REDACTED_SQL_QUERY) - read ETIMEDOUT"}

Yes this still happen to me... instead of setting pool, I think it's better if we have know how to catch time out and recover properly.

you can reproduce the bug in any environment by entering
    "pool": {       "min": 0,       "max": 1     },     "acquireConnectionTimeout": "10"
a max pool to 1 or an acquireConnectionTimeout to 10ms and call several requests in a row

I got some timers running and making queries. First query fails if settings are as @ariden84 said
KnexTimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
    at Client_PG.acquireConnection (/home/whatever/node_modules/knex/lib/client.js:349:26)
    at runNextTicks (internal/process/task_queues.js:62:5)
    at listOnTimeout (internal/timers.js:518:9)
    at processTimers (internal/timers.js:492:7) {
  sql: 'select 1',
  bindings: undefined
}

I got this problem today.
If I use pooling and let the app sit for a couple of hours, then make a request I get "Connection Error: Connection ended unexpectedly". After that first failing request the second request works.

@Zyles, I've the same problem.

I'm on the same boat as everyone here, first request fails but after that everything works as expected

I'm on the same boat as everyone here, first request fails but after that request, everything works as expected
Seems like min : 0gives us a solution given that tarn.js will remove every idle instance given that the minimum number of instances in the pool is determined by the minKeep and maxDestroy instantiate inside the check() method on the Pool object. (https://github.com/Vincit/tarn.js/blob/master/src/Pool.ts#L247)
So, passing a 0 will not retain instances inside Pool object and you won't have idle instances. So, this is more like a bandaid to the problem than actually a solution for it but works at the end since the check() function inside Pool.js on tarn.js will use this minimum size to keep instances alive and given that the minimum size is 0 will retain 0 instances alive

Do you have any idea how to fix this from the root cause ?

I'm on the same boat as everyone here, first request fails but after that request, everything works as expected
Seems like min : 0gives us a solution given that tarn.js will remove every idle instance given that the minimum number of instances in the pool is determined by the minKeep and maxDestroy instantiate inside the check() method on the Pool object. (https://github.com/Vincit/tarn.js/blob/master/src/Pool.ts#L247)
So, passing a 0 will not retain instances inside Pool object and you won't have idle instances. So, this is more like a bandaid to the problem than actually a solution for it but works at the end since the check() function inside Pool.js on tarn.js will use this minimum size to keep instances alive and given that the minimum size is 0 will retain 0 instances alive

Do you have any idea how to fix this from the root cause ?. We can patch-package knex.js if we have a fix...

I was really packed at work that day, but I can try to dig into today and see what I can do

I might have found something, correct me please If I'm wrong because this is my first time digging the code:
On the method acquireRawConnection() on postgres/index.js dialect, we are using an event handler for the event type "end" which I think is incorrect because we dispose of the error to the pipe think this cause the "idle" error we currently see (correct me here If I'm wrong and I'm not taking in count other use cases).
dialect/postgres/index.js line 78
connection.on('end', (err) => {
          connection.__knex__disposed = err || 'Connection ended unexpectedly';
});

pg.Client documentation (https://node-postgres.com/api/client)
client.on('end') => void
When the client disconnects from the PostgreSQL server it will emit an end event once.

Given docs from pg,  this event is emitted only on a client disconnection, so what could be happening here is that we left the connection "open" on the pool but closed on the Postgres pool, so when we try to communicate with the connection using the pg driver we receive that error.
Aside from this, also we should extend the validate connection method inside the pool to check if the connection already exists or is it valid. As far as I saw on the code while digging, correct me if this is on purpose, the validate method for the pool config inside client.js is quite silly (not using connection object to check if it's valid)

Maybe in conjunction with the end event listening commented on the first part of this comment, and the validate extension to something like the following could do the job:
  validateConnection(connection) {
    return connection && connection.connected;
  }

Currently, I'm trying to replicate it using the repository posted by @rafamel but without luck (I'm waiting for idle connections up to 20 minutes but nothing happens)

Wow, this is great start. This week I don't have any bandwidth to do the digging, pardon. This bug is hard to replicate because we have to wait for the timeout, we don't have any reliable way to confirm whether this is fixed or not... Maybe we can take look another postgres library how they did the pool ?

I know of an easy way to reproduce this error, since I forward the remote port via ssh from the server, then I just interrupt the forwarding and start reconnect. I suppose the same behavior will happen with a simple restart of the database.

I could reproduce the error emulating a postgresql database based on @intech comments (but here it's 100% offline)
Steps to reproduce the bug:
1- Emulate and expose a Postgres instance using docker-compose
version: '3.4'
services:
  db:
    image: 'postgres:11-alpine'
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
      POSTGRES_DB: test

2- Run a Node.JS script with a setInternal() to hit the database constantly while we are doing step 3

const knex = Knex({
  client: 'pg',
  connection: 'postgresql://dev:dev@localhost:5432/dev', //use the public URL not the docker one, in order to have more flexibility at the time of debugging
  pool: { min: 1, max: 5 },
  debug: true
});

setInterval(async () => {
  await knex.select().from('some-table'); //you could use postgres common tables, not need to create a migration
}, 2000);

2- Run, as a superuser, this query (will shutdown all connections associated with the database test
SELECT
  pg_terminate_backend(pid)
FROM
  pg_stat_activity
WHERE
  pid <> pg_backend_pid()
  AND datname = 'test';

As you can see in the following picture, after running the script of step 2, while in parallel, we are running the query of step 3, we will get idle connections (dead ones) with the expected error message (Connection terminated unexpectedly). This is helpful to replicate and fix this issue

@Warkanlock use patch-package.

Wow, if you fixed it, can you share the .patch file in patches directory ? That file is generated by patch-package.

@Warkanlock Wouldn't it be better to fix the problem upstream in the library itself? What changes are you doing via that patch?

Wow, if you fixed it, can you share the .patch file in patches directory ? That file is generated by patch-package.

I didn't use patch-package but npm link instead, I'm doing a PR with the changes but before pushing something. I'm looking for advice here, quite unsure about the results. I don't know at 100% if this is the way to replicate the issue (despite the fact the errors return the same message and status) nor if this bug is the bug we all saw before.
But here a demonstration of the connection reconnecting with the pool after an unexpected shutdown (or idle behaviour). As we can see, the first request hit correctly, the second one does not but at the end resolve either way since the pool is re-created

What I did here is a change on the validateConnection element, where we check if the connection is actually alive (using the events object). Aside of this, I modified my database timeouts to handle only 10ms for idle connections
validateConnection(connection) {
    return connection._events != null;
}

Also, I modify the end listener to end the connection after the reading in order to trigger the validate method on the tarn pool.
connection.on('end', async () => {
  connection.end();
  connection.__knex__disposed = 'Connection terminated unexpectedly'; //optional here
});

If I print the error that comes from the Postgres engine when the instance is shut down, we see the mythical error that everyone saw:

At least with my replication queries seems to work after the first fail (error on orange), but as I said, I'm not sure if this solve the core issue of this thread for idle statuses (I don't have time to wait for an hour for open connections)

@Warkanlock thank you so much for the insight and provided solution. As far as I can help, I've made some tests.

I was previously using postgres 12, but instead made the tests with version 11, so we are on the same page;
Same knex configuration;
I ran the tests in a similar environment as you: making requests within an interval. I had more intervals, that's the only difference;

Results: exactly the same as you did, but the error I get before your patch is not "Connection terminated unexpectedly" but instead "terminating connection due to administrator command", I have no idea why. After the patch, I get no error and the queries run just fine.
So, later I changed to postgres 12, and I'm sorry to say that your fix does not work. It throws the error anyway, whereas with postgres 11 it keeps doing queries.
But, anyways, make the PR and explain your doubts. I'm sure people will test and give feedback as I did.

very much still an issue, seems to be happening to all M1 macs, suggested workarounds not resolving

Does knex have any built-in retry behavior for queries that fail with Connection terminated unexpectedly. It seems like that should exist, because network issues can happen at any time. Basic retries with exponential backoff and a maxRetries config setting would be great.

@david-wb Does knex have any built-in retry behavior for queries that fail with Connection terminated unexpectedly? It seems like that should exist, because network issues can happen at any time. Basic retries with exponential backoff and a maxRetries config setting would be great.

No, the root cause should be fixed. Retrying is pretty much impossible, because there is no way to know what was that connections state when it happened. For example if there was transaction going on etc. Also your app should be always prepared that any of your SQL queries may fail at any moment.
@Warkanlock and others so were you really able to reproduce this problem, killing db connections while knex was doing other stuff?
I have tried earlier following methods: killing queries with pg_terminate_backend, shut down DB  and restart, using toxiproxy between knex and DB and closing connections with it while queries running at certain moment or at certain amount of bytes sent through proxy (allows disconnections at very weird moments during the queries).
Some tests code that I was using for stress testing drivers and their error handling / memory leaks / connection leaks is found here https://github.com/knex/knex/tree/master/scripts/stress-test
If you are able to create any test case replicating this behavior there it would be really great.

@elhigu I get quite a lot of errors with that stress-test script after just a minute or two.
$ node knex-stress-test.js| grep -A 3 -- "- STATS "
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: connect ETIMEDOUT <IP_ADDRESS>:25432
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 40365)
(node:45754) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 40402)
(node:45754) UnhandledPromiseRejectionWarning: Error: Response status was not 200: 409
    at Proxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Proxy.js:109:23)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Proxy.js:5:65)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 40403)
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 41391)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 11.5, results: 8, errors: 3.5 } }
------------------------------- EOS ------------------------------
KILLER ERROR: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 43664)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
KILLER ERROR: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 8.5, results: 5.5, errors: 3 } }
------------------------------- EOS ------------------------------
KILLER ERROR: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 48523)
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 51548)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 54765)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 55706)
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 55707)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
KILLER ERROR: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 57520)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
KILLER ERROR: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 64802)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 5.5, results: 3, errors: 2.5 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 11.5, results: 7, errors: 4.5 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 79412)
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
KILLER ERROR: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 86561)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 3, results: 0, errors: 3 } }
------------------------------- EOS ------------------------------
KILLER ERROR: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 11, results: 6.5, errors: 4.5 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 3.5, results: 2, errors: 1.5 } }
------------------------------- EOS ------------------------------
KILLER ERROR: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Response status was not 200: 404
    at Proxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Proxy.js:109:23)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Proxy.js:5:65)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 99436)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 14.5, results: 10, errors: 4.5 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 108585)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 2.5, results: 1, errors: 1.5 } }
------------------------------- EOS ------------------------------
Connection Error: error: terminating connection due to administrator command
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 393.5, results: 339, errors: 54.5 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
KILLER ERROR: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Response status was not 200: 404
    at Proxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Proxy.js:109:23)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Proxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 122042)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 2.5, results: 1, errors: 1.5 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 126079)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
KILLER ERROR: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 130461)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 9, results: 4.5, errors: 4.5 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 417, results: 389, errors: 28 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 132374)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 3.5, results: 2, errors: 1.5 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 477.5, results: 429, errors: 48.5 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
KILLER ERROR: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 5, results: 2.5, errors: 2.5 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 0, results: 0, errors: 0 } }
------------------------------- EOS ------------------------------
PSQL TO: Error: Connection terminated unexpectedly
(node:45754) UnhandledPromiseRejectionWarning: Error: Proxy postgresql already exists
    at Toxiproxy.<anonymous> (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:31:27)
    at Generator.throw (<anonymous>)
    at rejected (/Users/cmcgregor/Source/public/knex/node_modules/toxiproxy-node-client/dist/Toxiproxy.js:5:65)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
(node:45754) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 158028)
--
------------------------ STATS PER SECOND ------------------------
{ pg: { queries: 3.5, results: 1.5, errors: 2 } }
------------------------------- EOS ------------------------------
PSQL: Error: Connection terminated unexpectedly
^C

I cut down the script to only include pg tests and the docker compose file to only run a postgres:12.4-alpine image container to match what we're using. I also noticed knex doesn't include a package-lock.json file anywhere in git, so running against master isn't necessarily a true representation of how a release version will behave..
Can I ask, what kind of output indicates nominal or problematic behaviour? The errors here are very much what I've seen during a standby failover (via RDS, or RDS Proxy) though knex sometimes doesn't manage to recover. It'll get stuck in a weird state where every or most connections report these kinds of errors until the entire knex client is destroyed & reinitialised (or the service is restarted). Without sticking my fingers into internals of knex and tarn, touching properties I probably shouldn't, it's quite difficult to instrument adequately to chart behaviour via logs or metrics.
Unrelated note.. is there any plan to support self-management of connections & pooling? I think I've seen a few people say they've solve their issues using pg and/or pg-pool directly (sacrificing use of knex), and I'd imagine that to be a desirable escape hatch for some folks.

I haven't followed the progress of this issue, but for those who arrive here searching for solutions to the Connection ended unexpectedly errors -
the reason is that node-postgres disables keepAlive by default.
In our case, TCP connections idle for more than 15min got killed and the first query after 15min always failed with Connection ended unexpectedly
A workaround is to set keepAlive to true:
const knex = Knex({
    client: 'pg',
    connection: {
        host: '',
        user: '',
        password: '',
        keepAlive: true,
    },
    pool: {
        min: 2,
        max: 20
    }
})

Is there anything we can learn from pg-pool?
https://github.com/brianc/node-postgres/blob/master/packages/pg-pool/index.js

Could it be linked to this: https://github.com/moby/moby/issues/37466 ?
workaround here: https://github.com/moby/moby/issues/37466#issuecomment-405537713

Is the issue still open i am getting bellow error with knex 0.21.21 and pg 8.7.1 sometimes i production with the configuration {min: 2, max: 10}
Error: Connection terminated
at Connection. (/app/node_modules/pg/lib/client.js:132:36)
at Object.onceWrapper (events.js:519:28)
at Connection.emit (events.js:412:35)
at Connection.emit (domain.js:475:12)
at Socket. (/app/node_modules/pg/lib/connection.js:57:12)
at Socket.emit (events.js:400:28)
at Socket.emit (domain.js:475:12)
at TCP. (net.js:686:12)
at TCP.callbackTrampoline (internal/async_hooks.js:130:17)

In my server I have this comment, part of the initialization
pool: { min: 0, max: 2 }  // IMPORTANT https://knexjs.org/guide/#pool

I also dont seem to have the problem these days, I think maybe the key is
min:0 ?
On Tue, Jul 11, 2023 at 8:07 AM sshettyyara @.***>
wrote:

Is the issue still open i am getting bellow error with knex 0.21.21 and pg
8.7.1 sometimes i production with the configuration {min: 2, max: 10}
Error: Connection terminated
at Connection. (/app/node_modules/pg/lib/client.js:132:36)
at Object.onceWrapper (events.js:519:28)
at Connection.emit (events.js:412:35)
at Connection.emit (domain.js:475:12)
at Socket. (/app/node_modules/pg/lib/connection.js:57:12)
at Socket.emit (events.js:400:28)
at Socket.emit (domain.js:475:12)
at TCP. (net.js:686:12)
at TCP.callbackTrampoline (internal/async_hooks.js:130:17)
—
Reply to this email directly, view it on GitHub
https://github.com/knex/knex/issues/3523#issuecomment-1630704710, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AHTCBFXQIM3BJMOJ2P64L2TXPU6WRANCNFSM4JJ7VOZQ
.
You are receiving this because you commented.Message ID:
@.***>

It seems like the previous issue pertained to running Knex inside a Docker Swarm, but our application is running in Kubernetes. Will setting { min: 0 } also resolve the issue for us in this Kubernetes environment ??
within kubernetes yes its running as docker containers

The document linked in my previous post suggests that zero is the safest
setting and the default of 2 is "for legacy" so I would go with zero and do
some testing
On Thu, Jul 13, 2023 at 11:36 AM sshettyyara @.***>
wrote:

It seems like the previous issue pertained to running Knex inside a Docker
Swarm, but our application is running in Kubernetes. Will setting { min: 0
} also resolve the issue for us in this Kubernetes environment ??
within kubernetes yes its running as docker containers
—
Reply to this email directly, view it on GitHub
https://github.com/knex/knex/issues/3523#issuecomment-1634462476, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AHTCBFW7YV765HIEYTZSEU3XQAIXHANCNFSM4JJ7VOZQ
.
You are receiving this because you commented.Message ID:
@.***>

Coming here after a while, setting this to zero it's the most failover practice you could use on knex.js to avoid this kind of issue.
So far, using it on production for at least 20 different servers, and none of them is throwing the error anymore (after setting it to pool : { min : 0, max: N }
Maybe the issue has been resolved on tarn.js + knex library, but still, good to have it mind if you are experimenting this kind of issues


Add LDAP_DEBUG env var

This was just added to the base config in SODAR v0.14.1. This helps in debugging LDAP issues, obviously.

Done.


Improve the ability to run tests inside the docker container.

Use iptables to allow girder tests to reach the mongo container as if it were on localhost from inside the main docker container.
Install some extra utilities, such as npx and eslint in the main docker to allow testing.
With this change, after starting the docker group with ./deploy_docker.py, the following should result in all HistomicsTK tests passing:
docker exec -it histomicstk_histomicstk bash -c "cd ../build && cmake -DRUN_CORE_TESTS:BOOL=OFF -DTEST_PLUGINS:STRING=HistomicsTK ../girder && make -j && ctest --output-on-failure"

@cdeepakroy, @fbudin69500 This should result in dockers that will work for cmake testing directly.

I confirm that all the tests pass with the changes made in this PR.
I am yet to check if the tests pass if i mount in a local clone of HistomicsTK into /opt/histomicstk in the histomicstk_histomicstk container.

Works on my computer, even after mounting the source code from the host computer!


bug(select/ngOptions): switching to a valid model value via $setViewValue doesn't update selected value

In 1.3.1, when I call $setViewValue inside a directive and the value I'm switching to is a valid ng-option value, the select switches to the correct value: http://jsfiddle.net/8mgc0dv2/.
In 1.3.2+, given the same behavior, the select doesn't switch to the correct value: http://jsfiddle.net/g15o09t2/
In both 1.3.1 and 1.3.2+ (including 1.4 snapshot), if I set the model value directly inside a controller after a $timeout, the select switches to the correct value.  I looked at the changelog and this may have been introduced per https://github.com/angular/angular.js/commit/9e305948e4965fb86b0c79985dc6e8c59a9c66af, but I'm not 100% sure.

Update:
I was able to solve this by basically doing $parse(attrs.ngModel).assign(scope, newValue) instead of ngModel.$setViewValue(newValue): http://jsfiddle.net/LLsvdq02/.
Basically, it looks like in the $setViewValue case, when the ngModelWatch occurs, the control's modelValue and the actual modelValue are in sync so it never calls render() to go and update the select.  And in the $parse case, since the model is being set directly outside of any directive, the control's modelValue and the actual modelValue are different so render() is called.
After thinking about this a bit, I think the 1.3.2+ behavior actually makes more sense because $setViewValue assumes the user has taken over "ui updating" responsibilities.  $parse seems cleaner as well.  What do you guys think?

In 1.3.2, we changed a specific check to use === instead of ==.
However, it makes sense to me that it doesn't work now. When your model is '20000' and you are setting the $viewValue to 20000 you are skipping the complete ngOptions matching phase, and the $render phase. Changing the select should either be done by the user or driven completely by the model.
I also can't see that the select value is changing in 1.3.1 when calling $setViewValue inside the parser - the select changes when setting the model because we didn't use strict equality checking in 1.3.1
'

Thank you for the response Narretz.  I realized that $setViewValue wasn't even the cause of the select updating in 1.3.1, but it was rather the equality check like you were saying.  And yeah, only a change event or a $render invocation (via model change) causes the dropdown to update.  The $parse/completely model driven approach seems to be the correct way of handling those types of situations.


Question about CancellationTokens

I'm not exactly sure where to ask this question, so maybe you can help me direct this question to the correct team if this is not the place. But my question is about the CancellationToken passed into C# functions: https://docs.microsoft.com/en-us/azure/azure-functions/functions-dotnet-class-library#cancellation-tokens
We have observed that occasionally when we are in the middle of running a function, the CancellationToken is raised and causes all our async calls to stop. If a CancellationToken is signaled, we do expect this behavior, but I have a more general question about what causes this CancellationToken to be signaled in the first place. According to the documentation, I am left with a sort of vague feeling of understanding on the topic.
It seems like it just can happen any time the operating system says "it is time to shut down what we are working on". But I have no idea under which conditions this can and will happen when hosting functions within Azure. Does this ever happen as a result of functions running on the consumption plan needs to "scale out" under the hood? If not, what exactly (in the context of hosting within Azure on the consumption plan) causes this to happen?
Further more, should we be passing the CancellationToken along to async calls? I feel like the answer is yes, but maybe this is our mistake in the first place. Maybe we aren't supposed to pass the CancellationToken along at all, but it would feel so strange.
For a little more context on how we came across this issue/question in the first place:

We have a TimerTrigger function that runs once a day which queues tens of thousands of messages onto an Azure Storage Queue using the Azure.Storage.Queues nuget package.
We have a QueueTrigger function that picks up messages from the queue and makes calls to an HTTP dependency.

What we observed is that our TimerTrigger function was failing often after a couple thousand messages, with exceptions along the lines of TaskCanceledException. My first guess was that the function was timing out after 10 minutes, but it usually happens only a minute into execution.
My working theory is that after queuing messages begins, the QueueTrigger function begins to process them with high levels of parallelization. However, after a certain threshold the underlying infrastructure (ie: the consumption plan VMs) determine that in order to handle the load it should scale out. If this is the case, maybe the functions that were started on the original machine (ie, the TimerTrigger function) are signaled to wrap up what they are doing via the CancellationToken since the VM is being free'ed up/abandoned.
I realize that the internal workings of Azure are not something that are typically discussed in regards to how functions are scaled, etc. But I would really like to know what exactly is causing my TimerTrigger's CancellationToken to be signaled.
For anyone that runs across this issue and wonders how we ended up solving at (at least in the short term): We decided to take our logic in the TimerTrigger and refactor into a durable function. So basically the TimerTrigger simply kicks off an OrchestrationTrigger and we have an ActivityTrigger that queues a much smaller number of messages at a time. We call that ActivityTrigger from the OrchestrationTrigger with a retry policy, which allows the process to complete gracefully even when a CancellationToken is signaled in the ActivityTrigger function (which does happen about 10-15% of the time, from my initial testing).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 7a4532b8-8c27-ca15-edc6-85fc38cab341
Version Independent ID: e06de78b-9271-25a4-439d-2a024f31693e
Content: Azure Functions C# developer reference
Content Source: articles/azure-functions/functions-dotnet-class-library.md
Service: azure-functions
GitHub Login: @ggailey777
Microsoft Alias: glenga

Hello @APIWT - Thank you for your valuable feedback. Since this channel is for driving improvements towards MS Docs, the more appropriate channel for this type of discussion is the Functions host repository where the Functions team can weigh in on your feedback: https://github.com/Azure/azure-functions-host/issues

@mike-urnun-msft Thank you!
Closing, discussion continued here: https://github.com/Azure/azure-functions-host/issues/6195


Spurious import of ()

Hi,
When using a proto which defines a service taking or returning google.protobuf.Empty, by default prost will convert that to () which it then supports directly.  Sadly this means that the generated service code from tower-grpc-build ends up with an attempt to import super::() in a number of places.
I could litter the server.rs and client.rs with loads of if foobartype != "()" { around the calls to .import_type() and .import() but this feels clunky.  It'd be nice if there were a cleaner fix, but if there is, I can't see it.  My dodgy hack will let me proceed, and I can reflow it and supply it in a PR if desired, but essentially input_type and output_type from the prost_build::Module need to be checked wherever they might be imported, so that we don't generate an import of ().
An example proto which uses this is google.longrunning.operations in the Operations service there's DeleteOperation which returns google.protobuf.Empty as does CancelOperation at the same level.  (Proto example is here)

Also, since it doesn't seem sane to hide it from you, you can find my shonky commit on my fork at https://github.com/kinnison/tower-grpc

Oh and again, I should note that this is only important if you're using prost master (since August 1st when the type mapping was amended using their extern_path logic)

Hi @kinnison, if you're interested, I think a PR would be welcome!

I think that makes a lot of sense yes.  I will see if I can get some time to look at it, but if someone else looks at what I did and writes a better patch I won't feel bad :-)

I believe this should be done thanks to https://github.com/tower-rs/tower-grpc/pull/140/files#diff-d15ac6177e75577c93c4041346f98ed0R182


Unable to switch windows

Hi @gpakosz. First off, thank you very much for this amazing conf! I've just came across it today.
My issue is, I simply cannot switch windows when using it. Everything else works as expected, such as <prefix> Tab and even switching panes. However <prefix> n or <prefix> p simply gives me nothing. Unlinking the conf file brings me back to plain tmux and I have no problem creating and switching to next/previous windows at all.
I'm boggled. Any help is appreciated!

<prefix> C-h and <prefix> C-l let you navigate windows (default <prefix> n and <prefix> p are unbound)

Yet another case of not being thorough on the RTFM principle. I'll just leave this here.

I struggle with this choice. Why the default bindings are unbounded?


[Bug] Query service process record repeatedly

Search before asking

[X] I searched in the issues and found nothing similar.

Paimon version
Paimon 0.9
Compute Engine
Flink 1.18
Minimal reproduce step

Create paimon catelog
CREATE CATALOG paimon WITH (  'type'='paimon',  'warehouse'='file:/tmp/paimon' );
USE CATALOG paimon;
Create table and insert one record
CREATE TABLE DIM (k INT, v STRING, PRIMARY KEY(k) NOT ENFORCED) WITH ('bucket' = '1', 'deletion-vectors.enabled' = 'true');
INSERT INTO DIM VALUES (1, 'a');
Call Query Service
CALL sys.query_service('default.DIM', 1);

Insert another record
INSERT INTO DIM VALUES (2, 'b');
It shows records sent 4

What doesn't meet your expectations?
Records sent should consistent with table record count
Anything else?
No response
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

Is there any progress?


deprecation warnings for tensorflow

Using a pretty up-to-date container https://github.com/jupyter/docker-stacks/tree/master/tensorflow-notebook I encountered the following Deprecation Warnings... there are a lot of them.
WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py:1624: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From variational-autoencoder/train_variational_autoencoder_tensorflow.py:98: Normal.__init__ (from tensorflow.python.ops.distributions.normal) is deprecated and will be removed after 2019-01-01.
Instructions for updating:
The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/distributions/normal.py:160: Distribution.__init__ (from tensorflow.python.ops.distributions.distribution) is deprecated and will be removed after 2019-01-01.
Instructions for updating:
The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
WARNING:tensorflow:From variational-autoencoder/train_variational_autoencoder_tensorflow.py:106: Bernoulli.__init__ (from tensorflow.python.ops.distributions.bernoulli) is deprecated and will be removed after 2019-01-01.
Instructions for updating:
The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
WARNING:tensorflow:From variational-autoencoder/train_variational_autoencoder_tensorflow.py:132: kl_divergence (from tensorflow.python.ops.distributions.kullback_leibler) is deprecated and will be removed after 2019-01-01.
Instructions for updating:
The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2019-09-11 20:55:51.252283: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-09-11 20:55:51.274283: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency:<PHONE_NUMBER> Hz
2019-09-11 20:55:51.275538: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x5638c76eb780 executing computations on platform Host. Devices:
2019-09-11 20:55:51.275599: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From variational-autoencoder/train_variational_autoencoder_tensorflow.py:151: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please write your own downloading logic.
WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please use urllib or similar directly.
Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/dat/train-images-idx3-ubyte.gz
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/dat/train-labels-idx1-ubyte.gz
WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.one_hot on tensors.
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Extracting /tmp/dat/t10k-images-idx3-ubyte.gz
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting /tmp/dat/t10k-labels-idx1-ubyte.gz
WARNING:tensorflow:From /opt/conda/lib/python3.7/site-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.

Then this one follows for each iteration summary:
Saving TensorBoard summaries and images to: /tmp/log/
Iteration: 0 ELBO: -529.328 s/iter: 2.030e-04
Lossy conversion from float32 to uint8. Range [0, 1]. Convert image to uint8 prior to saving to suppress this warning.

@mathematicalmichael can you take a look at https://github.com/altosaar/variational-autoencoder/commit/b43325e297498269be86b281a02e2ef79664f273 please?
It should suppress most of the warnings. Is that better now? Any other ways to make it easier to understand? I'm not sure I'm using tf.datasets in the simplest way, but I think feed_dict's are more intuitive than tfrecords...
I also added an environment.yml file for anaconda with the latest tensorflow version.


clue scrolls should not be variant cross-difficulty.

Why not? What if you have multiple clue scrolls banked. I think they should remain as 6 separate items.

I don't know what you mean. Currently all clue scroll difficulties are grouped in one group in runelite's ItemVariationMapping and this makes it impossible to consistently place different clue difficulties in different spots in the bank - it will be based off of the order they are in in your regular bank. This is bad and I don't like it.

Ooooh I see, I misunderstood 👍

dupe #13.


Add numpy and scipy convolution wrappers

Replaces #2662; incorporates some ideas from #1831
I'm punting on mode='same' from scipy.signal.convolve2d, because I'm having trouble replicating the internal logic it uses to pad the inputs. In particular, the behavior is not internally consistent between 1d and 2d versions of the function:
from scipy import signal
x = np.arange(1, 5)
signal.correlate(x, x, mode='same')
# array([11, 20, 30, 20])
signal.correlate2d(x[None, :], x[None, :], mode='same')
# array([[20, 30, 20, 11]])

Turned out mode='same' was not as difficult as I thought...


Blazor samples debug configuration

Q
A

Bug fix?
yes

New feature?
no

New sample?
no

Related issues?
NA

What's in this Pull Request?
Adds config settings to support debugging C# Blazor from Visual Studio.

Fixes #549


 I'm able to reproduce this failures locally against that but not against any other spark released version so far

it appears commit https://github.com/NVIDIA/spark-rapids/commit/a42e328b7ce8c01a53722cb0f06d9b58ddc450f1 which added Databricks 11.3 is when this started to fail.

the issue is reproduceEmptyStringBug got set to true in that pr when it should be set to false.
that change originated from https://issues.apache.org/jira/browse/SPARK-39107 and it was fixed in 3.1.4 only in the 3.1.x line.


class.name isn't specific enough as a key

class Foo:
    @execTimeList
    def do_soemthing():
           pass

@execTimeList
def do_soemthing():
    pass

Foo.do_soemthing()
do_soemthing()

display_performances()
#  -- do_soemthing --
# Number of calls : 2
# Total execution time :   4.0531158447265625e-05 s
# Average execution time : 0.020265579223632812 ms

Thanks ! It should be fixed with str(method).
2e4885111111a4584424fa119633ec6fddde124b