"Received signal 11" in app_analyze

I finally caught a glimpse as to why analysis sometimes hangs:
2023-01-21T18:57:54.937913401Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] Analyzing (limit: 25) ... [component: analyze]
2023-01-21T18:57:54.983740764Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] Checkout directory: /checkouts [component: analyze]
2023-01-21T18:57:54.987633060Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] pulling https://github.com/yonaskolb/Stringly.git in /checkouts/github.com-yonaskolb-stringly [component: analyze]
2023-01-21T18:57:55.021965033Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] pulling https://github.com/dagronf/DSFActionBar.git in /checkouts/github.com-dagronf-dsfactionbar [component: analyze]
2023-01-21T18:57:55.029915226Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] pulling https://github.com/ordo-one/package-frostflake.git in /checkouts/github.com-ordo-one-package-frostflake [component: analyze]
2023-01-21T18:57:55.271462231Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] pulling https://github.com/chojnac/Dumpling.git in /checkouts/github.com-chojnac-dumpling [component: analyze]
2023-01-21T18:57:55.273393230Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] pulling https://github.com/raymccrae/swift-jsonpatch.git in /checkouts/github.com-raymccrae-swift-jsonpatch [component: analyze]
2023-01-21T18:57:55.277729726Z app_analyze.1.t1dix7fv4zph@p3    | Received signal 11. Backtrace:
2023-01-21T18:58:04.984287787Z app_analyze.1.t1dix7fv4zph@p3    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection timeout. [component: server, database-id: psql]
2023-01-21T18:58:04.984746687Z app_analyze.1.t1dix7fv4zph@p3    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection timeout. [component: server, database-id: psql]

The backtrace itself isn't in the logs but hopefully the crash is reproducible with one of the packages in question.

None of these packages individually are causing a signal 11 when running locally (macOS/arm64). It's going to be difficult to run this on Linux/x86 but maybe Linux/arm64 can reproduce it.

Doesn't crash on Linux/arm64 either.

It's not clear to me why the backtrace isn't captured by our logging. We're logging both stderr and stdout and yet there's no trace (😅) of the output beyond Received signal 11. Backtrace:.

I'll close this for now, will reopen and add more details if it happens again.

Another lock-up, different error and we got a partially symbolicated stack trace:
2023-01-27T15:37:57.557322522Z app_analyze.1.9jb9jx9vkshk@p2    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your con[130/1048]
meout. [component: server, database-id: psql]
2023-01-27T15:37:57.557341622Z app_analyze.1.9jb9jx9vkshk@p2    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection ti
meout. [component: server, database-id: psql]
2023-01-27T15:37:57.557377022Z app_analyze.1.9jb9jx9vkshk@p2    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection ti
meout. [component: server, database-id: psql]
2023-01-27T15:37:57.557428923Z app_analyze.1.9jb9jx9vkshk@p2    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection ti
meout. [component: server, database-id: psql]
2023-01-27T15:37:57.557445323Z app_analyze.1.9jb9jx9vkshk@p2    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection ti
meout. [component: server, database-id: psql]
2023-01-26T13:50:08.693743753Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78b36187, closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install(signals: Swift.Array<Swift.Int32>) -> () at /build/.build/checkouts/swift-backtrace/Sources/Backt
race/Backtrace.swift:80
2023-01-26T13:50:08.693750253Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78b36187, @objc closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install(signals: Swift.Array<Swift.Int32>) -> () at /build/<compiler-generated>:79
2023-01-26T13:50:08.693754653Z app_analyze.1.luku8u31ozjl@p2    | 0x7f4d69f8841f
2023-01-26T13:50:08.693758753Z app_analyze.1.luku8u31ozjl@p2    | 0x561f799f0571
2023-01-26T13:50:08.694152755Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79533bd0, closure #4 () throws -> Swift.String in (extension in ShellOut):Foundation.Process.(launchBash in _839723A297212BDF262C1834C3E29C1F)(with: Swift.String, outputHandle:
 Swift.Optional<Foundation.FileHandle>, errorHandle: Swift.Optional<Foundation.FileHandle>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:442
2023-01-26T13:50:08.694178555Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79533bd0, partial apply forwarder for closure #4 () throws -> Swift.String in (extension in ShellOut):Foundation.Process.(launchBash in _839723A297212BDF262C1834C3E29C1F)(with:
 Swift.String, outputHandle: Swift.Optional<Foundation.FileHandle>, errorHandle: Swift.Optional<Foundation.FileHandle>) throws -> Swift.String at /build/<compiler-generated>:0
2023-01-26T13:50:08.694185355Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7982a3eb
2023-01-26T13:50:08.694189055Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79829285
2023-01-26T13:50:08.694192655Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7982d0fb
2023-01-26T13:50:08.694196255Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79824068
2023-01-26T13:50:08.694202855Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79840cd9
2023-01-26T13:50:08.694206355Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7982d071
2023-01-26T13:50:08.694210455Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7982914b
2023-01-26T13:50:08.694214055Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79829594
2023-01-26T13:50:08.694217755Z app_analyze.1.luku8u31ozjl@p2    | 0x561f795318e0, (extension in ShellOut):Foundation.Process.(launchBash in _839723A297212BDF262C1834C3E29C1F)(with: Swift.String, outputHandle: Swift.Optional<Foundation.FileHandle>,
errorHandle: Swift.Optional<Foundation.FileHandle>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:441
2023-01-26T13:50:08.694237255Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79531d27, ShellOut.shellOut(to: Swift.String, arguments: Swift.Array<Swift.String>, at: Swift.String, process: Foundation.Process, outputHandle: Swift.Optional<Foundation.FileH
andle>, errorHandle: Swift.Optional<Foundation.FileHandle>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:40
2023-01-26T13:50:08.694242255Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79531d27, ShellOut.shellOut(to: ShellOut.ShellOutCommand, at: Swift.String, process: Foundation.Process, outputHandle: Swift.Optional<Foundation.FileHandle>, errorHandle: Swift
.Optional<Foundation.FileHandle>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:106
2023-01-26T13:50:08.694566856Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7880f4d8, closure #1 (ShellOut.ShellOutCommand, Swift.String) throws -> Swift.String in variable initialization expression of static App.Shell.live : App.Shell at /build/Source
s/App/Core/AppEnvironment.swift:291
2023-01-26T13:50:08.694694157Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78830c1d, App.Shell.run(command: ShellOut.ShellOutCommand, at: Swift.String) throws -> Swift.String at Sources/App/Core/AppEnvironment.swift:283
2023-01-26T13:50:08.694709557Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78830c1d, function signature specialization <Arg[2] = Dead> of static App.Git.revisionInfo(_: App.Reference, at: Swift.String) throws -> App.Git.RevisionInfo at /build/Sources/
App/Core/Git.swift:70
2023-01-26T13:50:08.694851657Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78812395, static App.Git.revisionInfo(_: App.Reference, at: Swift.String) throws -> App.Git.RevisionInfo at /build/<compiler-generated>:0
2023-01-26T13:50:08.694873258Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78812395, implicit closure #12 (App.Reference, Swift.String) throws -> App.Git.RevisionInfo in implicit closure #11 (App.Git.Type) -> (App.Reference, Swift.String) throws -> Ap
p.Git.RevisionInfo in variable initialization expression of static App.Git.live : App.Git at /build/<compiler-generated>:271
2023-01-26T13:50:08.694901558Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78812395, partial apply forwarder for implicit closure #12 (App.Reference, Swift.String) throws -> App.Git.RevisionInfo in implicit closure #11 (App.Git.Type) -> (App.Reference
, Swift.String) throws -> App.Git.RevisionInfo in variable initialization expression of static App.Git.live : App.Git at /build/<compiler-generated>:0
2023-01-26T13:50:08.696367663Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7878eefe, closure #3 (App.Reference) throws -> App.Version in static App.Analyze.getIncomingVersions(client: Vapor.Client, logger: Logging.Logger, package: App.Joined<App.Packa
ge, App.Repository>) async throws -> Swift.Array<App.Version> at /build/Sources/App/Commands/Analyze.swift:438
2023-01-26T13:50:08.696417564Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7878eefe, function signature specialization <Arg[1] = Owned To Guaranteed, Arg[2] = Owned To Guaranteed> of function signature specialization <Arg[0] = [Closure Propagated : cl
osure #3 (App.Reference) throws -> App.Version in static App.Analyze.getIncomingVersions(client: Vapor.Client, logger: Logging.Logger, package: App.Joined<App.Package, App.Repository>) async throws -> Swift.Array<App.Version>, Argument Types : [Swi
ft.StringApp.Joined<App.Package, App.Repository>]> of generic specialization <Swift.Array<App.Reference>, App.Version> of (extension in Swift):Swift.Collection.map<A>((A.Element) throws -> A1) throws -> Swift.Array<A1> at /build/<compiler-generated
>:0
2023-01-26T13:50:08.696437964Z app_analyze.1.luku8u31ozjl@p2    | 0x561f787920c7, function signature specialization <Arg[0] = [Closure Propagated : closure #3 (App.Reference) throws -> App.Version in static App.Analyze.getIncomingVersions(client: V
apor.Client, logger: Logging.Logger, package: App.Joined<App.Package, App.Repository>) async throws -> Swift.Array<App.Version>, Argument Types : [Swift.StringApp.Joined<App.Package, App.Repository>]> of generic specialization <Swift.Array<App.Refe
rence>, App.Version> of (extension in Swift):Swift.Collection.map<A>((A.Element) throws -> A1) throws -> Swift.Array<A1> at /build/<compiler-generated>:0
2023-01-26T13:50:08.696444064Z app_analyze.1.luku8u31ozjl@p2    | 0x561f787920c7, (1) suspend resume partial function for function signature specialization <Arg[3] = Dead> of static App.Analyze.getIncomingVersions(client: Vapor.Client, logger: Logg
ing.Logger, package: App.Joined<App.Package, App.Repository>) async throws -> Swift.Array<App.Version> at /build/Sources/App/Commands/Analyze.swift:437
2023-01-26T13:50:08.696486964Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79e6520d
2023-01-26T13:50:08.696503764Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79e658fb
2023-01-26T13:50:08.696508064Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79837424
2023-01-26T13:50:08.696511764Z app_analyze.1.luku8u31ozjl@p2    | 0x561f798371d2
2023-01-26T13:50:08.696515264Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79842ac1
2023-01-26T13:50:08.696518764Z app_analyze.1.luku8u31ozjl@p2    | 0x7f4d69f7c608
2023-01-26T13:50:08.696538164Z app_analyze.1.luku8u31ozjl@p2    | 0x7f4d69ac0132
2023-01-26T13:50:08.696542264Z app_analyze.1.luku8u31ozjl@p2    | 0xffffffffffffffff
2023-01-26T13:50:08.846038854Z app_analyze.1.luku8u31ozjl@p2    | /bin/bash: line 1: 20617 Illegal instruction     (core dumped) ./Run analyze --env prod --limit 25
2023-01-26T13:50:28.883543917Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] Analyzing (limit: 25) ... [component: analyze]
2023-01-26T13:50:28.953630593Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] Checkout directory: /checkouts [component: analyze]
2023-01-26T13:50:28.958182711Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/genius/future.git in /checkouts/github.com-genius-future [component: analyze]
2023-01-26T13:50:28.987394827Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/xiiagency/SwiftConcurrency.git in /checkouts/github.com-xiiagency-swiftconcurrency [component: analyze]
2023-01-26T13:50:29.001886784Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/JUSTINMKAUFMAN/SwiftUIPhone.git in /checkouts/github.com-justinmkaufman-swiftuiphone [component: analyze]
2023-01-26T13:50:29.226366270Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/patchthecode/JTAppleCalendar.git in /checkouts/github.com-patchthecode-jtapplecalendar [component: analyze]
2023-01-26T13:50:29.231274989Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/RougeWare/Swift-Optional-Tools.git in /checkouts/github.com-rougeware-swift-optional-tools [component: analyze]
2023-01-26T13:50:29.254494181Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/apple/swift-corelibs-xctest.git in /checkouts/github.com-apple-swift-corelibs-xctest [component: analyze]

Common theme:
Connection request timed out. This might indicate a connection deadlock in your application.

Not sure if that's a symptom or the cause, but worth investigating.

It's been a while since we've seen this, so I'm going to close this for now.

After increasing log level to DEBUG we're seeming interesting details:
│   [ DEBUG ] Connection pool exhausted on this event loop, adding request to waitlist [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:206)                                                                             │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Connection pool exhausted on this event loop, adding request to waitlist [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:206)                                                                             │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] updateStatus ops: 25 [component: analyze] (App/Common.swift:48)                                                                                │
│   [ DEBUG ] REFRESH MATERIALIZED VIEW recent_packages [] [component: server, database-id: psql] (FluentPostgresDriver/FluentPostgresDatabase.swift:164)                                                                                │

The Connection pool exhausted on this event code path is one step from our deadlock error message:
https://github.com/vapor/async-kit/blob/a61da00d404ec91d12766f1b9aac7d90777b484d/Sources/AsyncKit/ConnectionPool/EventLoopConnectionPool.swift#L206

After reducing the batch size to 5 (from 25) the messages are still there but there are fewer of them:
│   id: psql] (FluentPostgresDriver/FluentPostgresDatabase.swift:33)                                                                                │
│   [ DEBUG ] UPDATE "packages" SET "status" = 'ok', "score" = $1, "processing_stage" = 'analysis', "updated_at" = $2 WHERE "packages"."id" = $3 [60, 2023-04-18 09:52:42 +0000, E11AC48B-A2C8-48FB-B496-D05FE9368F93] [component: server, database-   │
│   id: psql] (FluentPostgresDriver/FluentPostgresDatabase.swift:33)                                                                                │
│   [ DEBUG ] Connection pool exhausted on this event loop, adding request to waitlist [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:206)                                                                             │
│   [ DEBUG ] Connection pool exhausted on this event loop, adding request to waitlist [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:206)                                                                             │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] Fulfilling connection waitlist request [component: server, database-id: psql] (AsyncKit/EventLoopConnectionPool.swift:275)                                                                                │
│   [ DEBUG ] updateStatus ops: 5 [component: analyze] (App/Common.swift:48)                                                                                │
│   [ DEBUG ] REFRESH MATERIALIZED VIEW recent_packages [] [component: server, database-id: psql] (FluentPostgresDriver/FluentPostgresDatabase.swift:164)                                                                                │

This could possible prevent us from running into the deadlock but I'll also check if we can get rid of them entirely by increasing the pool size.

One notable observation: the last couple of times this happened I didn't see any backtrace. I'm not sure if this means the process isn't crashing.
Note also that in January '23 when I opened this issue we were running on Swift 5.7 and now we're on 5.8, so that might be an explanation.

Happened again yesterday (alert from April 26 2023, 13:42 CET) but no additional detail in logs:

I didn't get a chance to check the state of the container before a re-deploy "fixed" the issue, so I can't be 100% sure it was the same error.

hang-prod-2023-04-27.logs.zip

Happened again on dev just now.

Happened again on dev this morning.

Happened in prod this morning.

And now on dev.

The analysis job is also running git commands. Johannes Weiß mentioned in passing on the Swift Open Source slack that they saw "crazy stuff" due to FileManager.defaul.removeItems(path) taking very long due to a large directory being remove. I wonder if our case is a symptom of that same issue.
EventLoop latency logging is a recommended diagnostic tool, more details here: https://github.com/apple/swift-nio/issues/2410

I mean, we do run checkouts of new packages for instance, and those can take > 10s if they're large.

Hang on dev

Hang on prod

Hang on prod

Ok, I've actually managed to reproduce this locally on my slow network:

And it's back, but in slightly different form. It always bothered me that initially I saw Received signal 11 crashes in the logs when the hangs happened but then it turned into the connection deadlock errors.
Turns out, the connection deadlock errors just masked the crashes. Now that they're gone, we see them, although it's signal 4, at least this time around:
Received signal 4. Backtrace:
0x559b0531a222, Backtrace.(printBacktrace in _B82A8C0ED7C904841114FDF244F9E58E)(signal: Swift.Int32) -> () at /build/.build/checkouts/swift-backtrace/Sources/Backtrace/Backtrace.swift:66
0x559b0531a497, closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install(signals: Swift.Array<Swift.Int32>) -> () at /build/.build/checkouts/swift-backtrace/Sources/Backtrace/Backtrace.swift:80
0x559b0531a497, @objc closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install(signals: Swift.Array<Swift.Int32>) -> () at /build/<compiler-generated>:79
0x7f7a21f5b41f
0x559b06336951
0x559b05d58b69, closure #4 () throws -> Swift.String in (extension in ShellOut):Foundation.Process.(launchBash in _839723A297212BDF262C1834C3E29C1F)(with: Swift.String, outputHandle: Swift.Optional<Foundation.FileHandle>, errorHandle: Swift.Optional<Foundation.FileHandle>, environment: Swift.Optional<Swift.Dictionary<Swift.String, Swift.String>>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:476
0x559b05d59981, partial apply forwarder for closure #4 () throws -> Swift.String in (extension in ShellOut):Foundation.Process.(launchBash in _839723A297212BDF262C1834C3E29C1F)(with: Swift.String, outputHandle: Swift.Optional<Foundation.FileHandle>, errorHandle: Swift.Optional<Foundation.FileHandle>, environment: Swift.Optional<Swift.Dictionary<Swift.String, Swift.String>>) throws -> Swift.String at /build/<compiler-generated>:0
0x559b0614e43b
0x559b0614d265
0x559b061515bb
0x559b06147fe8
0x559b06165269
0x559b06151041
0x559b0614d12b
0x559b0614d574
0x559b05d55a9e, (extension in ShellOut):Foundation.Process.(launchBash in _839723A297212BDF262C1834C3E29C1F)(with: Swift.String, outputHandle: Swift.Optional<Foundation.FileHandle>, errorHandle: Swift.Optional<Foundation.FileHandle>, environment: Swift.Optional<Swift.Dictionary<Swift.String, Swift.String>>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:475
0x559b05d55d12, ShellOut.shellOut(to: ShellOut.SafeString, arguments: Swift.Array<ShellOut.Argument>, at: Swift.String, process: Foundation.Process, outputHandle: Swift.Optional<Foundation.FileHandle>, errorHandle: Swift.Optional<Foundation.FileHandle>, environment: Swift.Optional<Swift.Dictionary<Swift.String, Swift.String>>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:42
0x559b05d55d12, ShellOut.shellOut(to: ShellOut.ShellOutCommand, at: Swift.String, process: Foundation.Process, outputHandle: Swift.Optional<Foundation.FileHandle>, errorHandle: Swift.Optional<Foundation.FileHandle>, environment: Swift.Optional<Swift.Dictionary<Swift.String, Swift.String>>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:76
0x559b04fafe87, closure #1 (ShellOut.ShellOutCommand, Swift.String) throws -> Swift.String in variable initialization expression of static App.Shell.live : App.Shell at /build/Sources/App/Core/AppEnvironment.swift:282
0x559b04fdbe54, App.Shell.run(command: ShellOut.ShellOutCommand, at: Swift.String) throws -> Swift.String at Sources/App/Core/AppEnvironment.swift:274
0x559b04fdbe54, function signature specialization <Arg[2] = Dead> of static App.Git.revisionInfo(_: App.Reference, at: Swift.String) throws -> App.Git.RevisionInfo at /build/Sources/App/Core/Git.swift:70
0x559b04fafdf5, static App.Git.revisionInfo(_: App.Reference, at: Swift.String) throws -> App.Git.RevisionInfo at /build/Sources/App/Core/AppEnvironment.swift:0
0x559b04fafdf5, implicit closure #6 (App.Reference, Swift.String) throws -> App.Git.RevisionInfo in variable initialization expression of static App.Git.live : App.Git at /build/Sources/App/Core/AppEnvironment.swift:262
0x559b04f39e63, closure #3 (App.Reference) throws -> App.Version in static App.Analyze.getIncomingVersions(client: Vapor.Client, logger: Logging.Logger, package: App.Joined<App.Package, App.Repository>) async throws -> Swift.Array<App.Version> at /build/Sources/App/Commands/Analyze.swift:421
0x559b04f39e63, function signature specialization <Arg[1] = Owned To Guaranteed, Arg[2] = Owned To Guaranteed> of function signature specialization <Arg[0] = [Closure Propagated : closure #3 (App.Reference) throws -> App.Version in static App.Analyze.getIncomingVersions(client: Vapor.Client, logger: Logging.Logger, package: App.Joined<App.Package, App.Repository>) async throws -> Swift.Array<App.Version>, Argument Types : [Swift.StringApp.Joined<App.Package, App.Repository>]> of generic specialization <Swift.Array<App.Reference>, App.Version> of (extension in Swift):Swift.Collection.map<A>((A.Element) throws -> A1) throws -> Swift.Array<A1> at /build/<compiler-generated>:0
0x559b04f3d28f, function signature specialization <Arg[0] = [Closure Propagated : closure #3 (App.Reference) throws -> App.Version in static App.Analyze.getIncomingVersions(client: Vapor.Client, logger: Logging.Logger, package: App.Joined<App.Package, App.Repository>) async throws -> Swift.Array<App.Version>, Argument Types : [Swift.StringApp.Joined<App.Package, App.Repository>]> of generic specialization <Swift.Array<App.Reference>, App.Version> of (extension in Swift):Swift.Collection.map<A>((A.Element) throws -> A1) throws -> Swift.Array<A1> at /build/<compiler-generated>:0
0x559b04f3d28f, (1) suspend resume partial function for function signature specialization <Arg[3] = Dead> of static App.Analyze.getIncomingVersions(client: Vapor.Client, logger: Logging.Logger, package: App.Joined<App.Package, App.Repository>) async throws -> Swift.Array<App.Version> at /build/Sources/App/Commands/Analyze.swift:420
0x559b06788d9d
0x559b0678947b
0x559b0615b7e3
0x559b0615b591
0x559b06167071
0x7f7a21f4f608
0x7f7a21a92132
0xffffffffffffffff
/bin/bash: line 1: 21985 Illegal instruction     (core dumped) ./Run analyze --env dev --limit 25

There are lots of occurrences in this log grab and I failed to add timestamp when dumping them, so it's hard to tell if they're all closely related in time.
The last one has a ShellOut error happening just before:
[ INFO ] fetch failed: Shell command failed:
command: "git fetch --tags --prune-tags --prune"
path:    "/checkouts/github.com-firebase-firebase-ios-sdk"
message: "ShellOut encountered an error
Status code: 1
Message: "From https://github.com/firebase/firebase-ios-sdk
 - [deleted]             (none)     -> origin/nc/csh-analyzer-fixes
 - [deleted]             (none)     -> origin/pb-apnstokentype
   cead23010..927ae355b  master                    -> origin/master
 * [new branch]          abt-exp-meta              -> origin/abt-exp-meta
   80357b4e3..01cbad1f0  appcheck-lite             -> origin/appcheck-lite
   35d2acbc4..d3eb5c805  auth-swift                -> origin/auth-swift
   0f47aa608..11ddd49c8  markduckworth/sum-avg-cpp-api -> origin/markduckworth/sum-avg-cpp-api
   fcfb8f5a9..e18205268  samedson/wowlocal-urgent-deadlock -> origin/samedson/wowlocal-urgent-deadlock
 + 46a10002b...7105602ef themis/swiftDemangle      -> origin/themis/swiftDemangle  (forced update)
 ! [rejected]            CocoaPods-10.10.0.nightly -> CocoaPods-10.10.0.nightly  (would clobber existing tag)"
Output: """ [component: analyze]
[ INFO ] removing directory [component: analyze]
[ INFO ] cloning https://github.com/firebase/firebase-ios-sdk.git to /checkouts/github.com-firebase-firebase-ios-sdk [component: analyze]
Received signal 4. Backtrace:

That may be significant. I.e. it might be that the earlier signal 4 errors are being recovered - actually I think that's in fact the case. There are sections in the logs where processing is normal:
[ INFO ] Analyzing (limit: 25) ... [component: analyze]
[ INFO ] Checkout directory: /checkouts [component: analyze]
[ INFO ] pulling https://github.com/theappstudiollc/ASEnterprise.git in /checkouts/github.com-theappstudiollc-asenterprise [component: analyze]
[ INFO ] pulling https://github.com/cxa/Piz.git in /checkouts/github.com-cxa-piz [component: analyze]
[ INFO ] pulling https://github.com/BeAppOnline/AFBilling.git in /checkouts/github.com-beapponline-afbilling [component: analyze]
[ INFO ] pulling https://github.com/heckj/MeshGenerator.git in /checkouts/github.com-heckj-meshgenerator [component: analyze]
[ INFO ] pulling https://github.com/SocketMobile/swift-package-capturesdk.git in /checkouts/github.com-socketmobile-swift-package-capturesdk [component: analyze]
[ INFO ] pulling https://github.com/chrispomeroyhale/csv-dialect-swift.git in /checkouts/github.com-chrispomeroyhale-csv-dialect-swift [component: analyze]
[ INFO ] pulling https://github.com/dagronf/DSFActionBar.git in /checkouts/github.com-dagronf-dsfactionbar [component: analyze]
[ INFO ] pulling https://github.com/swift-kipple/Core.git in /checkouts/github.com-swift-kipple-core [component: analyze]
[ INFO ] pulling https://github.com/apocolipse/Thrift-Swift.git in /checkouts/github.com-apocolipse-thrift-swift [component: analyze]
[ INFO ] pulling https://github.com/ParkGwangBeom/TransitionableTab.git in /checkouts/github.com-parkgwangbeom-transitionabletab [component: analyze]
[ INFO ] pulling https://github.com/apple/swift-foundation-icu.git in /checkouts/github.com-apple-swift-foundation-icu [component: analyze]
[ INFO ] pulling https://github.com/StanfordBDHG/CardinalKitFHIRtoFirestoreAdapter.git in /checkouts/github.com-stanfordbdhg-cardinalkitfhirtofirestoreadapter [component: analyze]
[ INFO ] pulling https://github.com/dmrschmidt/QRCode.git in /checkouts/github.com-dmrschmidt-qrcode [component: analyze]
[ INFO ] pulling https://github.com/driven-app/porsche-connect.git in /checkouts/github.com-driven-app-porsche-connect [component: analyze]
[ INFO ] pulling https://github.com/timsearle/euclid.git in /checkouts/github.com-timsearle-euclid [component: analyze]
[ INFO ] pulling https://github.com/danielsaidi/Sheeeeeeeeet.git in /checkouts/github.com-danielsaidi-sheeeeeeeeet [component: analyze]
[ INFO ] pulling https://github.com/Carthage/ReactiveTask.git in /checkouts/github.com-carthage-reactivetask [component: analyze]
[ INFO ] pulling https://github.com/tonystone/tracelog.git in /checkouts/github.com-tonystone-tracelog [component: analyze]
[ INFO ] pulling https://github.com/amosavian/FileProvider.git in /checkouts/github.com-amosavian-fileprovider [component: analyze]
[ INFO ] pulling https://github.com/theleftbit/BSWInterfaceKit.git in /checkouts/github.com-theleftbit-bswinterfacekit [component: analyze]
[ INFO ] pulling https://github.com/m-barthelemy/DockerSwift.git in /checkouts/github.com-m-barthelemy-dockerswift [component: analyze]
[ INFO ] pulling https://github.com/SwiftyFinch/Rugby.git in /checkouts/github.com-swiftyfinch-rugby [component: analyze]
[ INFO ] pulling https://github.com/mapbox/mapbox-directions-swift.git in /checkouts/github.com-mapbox-mapbox-directions-swift [component: analyze]
[ INFO ] pulling https://github.com/Kitura/Kitura.git in /checkouts/github.com-kitura-kitura [component: analyze]
[ INFO ] pulling https://github.com/fxm90/GradientLoadingBar.git in /checkouts/github.com-fxm90-gradientloadingbar [component: analyze]
[ INFO ] Analyzing (limit: 25) ... [component: analyze]
[ INFO ] Checkout directory: /checkouts [component: analyze]
[ INFO ] Analyzing (limit: 25) ... [component: analyze]
[ INFO ] Checkout directory: /checkouts [component: analyze]

This is after an earlier signal 4 error. So that's not what's making it hang. I take that as an indication that the git fetch --tags --prune-tags --prune error might be close to the root cause.
I grabbed the repo in question (github.com-firebase-firebase-ios-sdk) but I did this after restarting the service and it might have been too late. In fact, running git fetch --tags --prune-tags --prune on a copy does not show that same error.
Full logs:
hang-dev-2023-05-10.logs.zip

This is happening with release 2.83.13, for backtrace line number purposes.

We're actually also still seeing the connection deadlock, despite the attempted fix deployed yesterday. It could be that it's in another part though. I.e. while I moved the git clone/pull out of the db transaction, the 10s timeout could be exceeded by some other synchronous operation.
The logs from prod also show that there are Received signal 4 crashes, and it's clear that they don't hang the process.
hang-prod-2023-05-10.log.zip

The signal 4 crashes might be related to this:
https://askubuntu.com/questions/1420966/method-https-has-died-unexpectedly-sub-process-https-received-signal-4-after
I'll attempt a fix to try and remove that part from the equation.

Turns out this is somewhat tricky for uninteresting reasons. What gives me pause is that the nature of the env variable (and the bug reports) indicate this is an issue when running network requests. However, we see this fail on git operations like
git rev-list --count HEAD

or
git log -n1 --format=format:"%H-%ct" foo

which should run locally. Might limit it to just changing these two commands for now to see if it has an effect before applying it to all git commands we run.

Late note: it doesn't seem to have anything to do with the git commands. The one git command I'd instrumented still appeared in the crashes, it did nothing: https://github.com/SwiftPackageIndex/SwiftPackageIndex-Server/pull/2381

Increasing the connection pool timeout in #2386 seems have fixed the hangs - we haven't had any on either envs since then (a week ago). I don't think the underlying issue is addressed (as detailed here: https://github.com/vapor/console-kit/issues/179#issuecomment-1539516353) but at least it doesn't impact our processing any more, at least for the moment.
Closing this for now.

Just happened again on prod. The timeout increase just made it a lot less frequent (as expected).

For the record, it hasn't happened since May 26 🤞(it isn't fixed but it seems to be rare enough not to be an issue for the moment).

Happened again this morning on both envs within a few hours.

Happened again at ~18:00 CET, July 3 2023.

Happened again at ~5:30 CET, July 11 2023.

~12:00 CET, July 22 2023

prod, ~5:40 CET, July 30 2023

prod, ~21:00 CET, July 31 2023

prod, ~1:30 CET, Aug 12 2023

prod, 9:19 CET, Sep 3 2023

We've had a hang again, with all the latest ShellOut fixes, at 16:18 CET today, Sep 6 2023.
It's very clear now that the cause is an actual crash:

2023-09-06T13:36:58.114865247Z Received signal 11. Backtrace:
2023-09-06T13:36:58.615927201Z 0x55ba0c45b4f1, Backtrace.(printBacktrace in _B82A8C0ED7C904841114FDF244F9E58E)(signal: Swift.Int32) -> () at /build/.build/checkouts/swift-backtrace/Sources/Backtrace/Backtrace.swift:66
2023-09-06T13:36:58.615971303Z 0x55ba0c45b7a7, closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install(signals: Swift.Array<Swift.Int32>) -> () at /build/.build/checkouts/swift-backtrace/Sources/Backtrace/Backtrace.swift:80
2023-09-06T13:36:58.615994404Z 0x55ba0c45b7a7, @objc closure #1 (Swift.Int32) -> () in static Backtrace.Backtrace.install(signals: Swift.Array<Swift.Int32>) -> () at /build/<compiler-generated>:79
2023-09-06T13:36:58.615999404Z 0x7fb0bf00941f
2023-09-06T13:36:58.616003205Z 0x55ba0df75896
2023-09-06T13:36:58.616006905Z 0x55ba0df7ceb9
2023-09-06T13:36:58.616011105Z 0x55ba0df7ccb5
2023-09-06T13:36:58.616014905Z 0x55ba0df6c60b
2023-09-06T13:36:58.616018605Z 0x55ba0df7575b
2023-09-06T13:36:58.616022306Z 0x55ba0df6941d
2023-09-06T13:36:58.616470727Z 0x55ba0c0fa23a, implicit closure #4 (Swift.String) throws -> Swift.Array<Swift.String> in implicit closure #3 (Foundation.FileManager) -> (Swift.String) throws -> Swift.Array<Swift.String> in variable initialization expression of static App.FileManager.live : App.FileManager at /build/Sources/App/Core/AppEnvironment.swift:270
2023-09-06T13:36:58.616499028Z 0x55ba0c0fa23a, partial apply forwarder for implicit closure #4 (Swift.String) throws -> Swift.Array<Swift.String> in implicit closure #3 (Foundation.FileManager) -> (Swift.String) throws -> Swift.Array<Swift.String> in variable initialization expression of static App.FileManager.live : App.FileManager at /build/<compiler-generated>:0
2023-09-06T13:36:58.620597122Z 0x55ba0c0404fe, App.FileManager.contentsOfDirectory(atPath: Swift.String) throws -> Swift.Array<Swift.String> at Sources/App/Core/AppEnvironment.swift:258
2023-09-06T13:36:58.620625524Z 0x55ba0c0404fe, function signature specialization <Arg[0] = Dead> of static App.Analyze.trimCheckouts() throws -> () at /build/Sources/App/Commands/Analyze.swift:97
2023-09-06T13:36:58.620662926Z 0x55ba0c037a2c, static App.Analyze.trimCheckouts() throws -> () at /build/<compiler-generated>:0
2023-09-06T13:36:58.620772931Z 0x55ba0c037a2c, (3) suspend resume partial function for App.Analyze.Command.run(using: ConsoleKit.CommandContext, signature: App.Analyze.Command.Signature) async throws -> () at /build/Sources/App/Commands/Analyze.swift:64
2023-09-06T13:36:58.620926738Z 0x55ba0de0123e
2023-09-06T13:36:58.620943239Z 0x55ba0de0191f
2023-09-06T13:36:58.620947939Z 0x55ba0de162f8
2023-09-06T13:36:58.620951539Z 0x55ba0de1610e
2023-09-06T13:36:58.620955639Z 0x55ba0de21135
2023-09-06T13:36:58.620959240Z 0x7fb0beffd608
2023-09-06T13:36:58.621011242Z 0x7fb0becfc132
2023-09-06T13:36:58.621028243Z 0xffffffffffffffff
2023-09-06T13:36:58.763773210Z /bin/bash: line 1: 28473 Segmentation fault      (core dumped) ./Run analyze --env prod --limit 25
2023-09-06T14:02:12.447331859Z Received signal 11. Backtrace:

Note the timestamp of the last entry at 14:02 UTC (16:00 CET) is a single entry and then it hangs. Our alerts trigger when we've not seen logs for 20 mins, so that aligns with the alert at 16:18 CET.
I see signal 11 crashes every few hours but it seems they don't always lead to a hang. Haven't looked at the logs in more detail yet, attached below.
analyze-2023-09-06-stderr.logs.zip
analyze-2023-09-06.logs.zip
FYI @gwynne 😢

@finestructure So, from that crash log, looks to me like odds are decent you're hitting the ARC miscompile crash in 5.8.1. My very unorthodox suggestion is to try building with 5.9, for three reasons:

5.9 is now very close to release so can be reasonably considered at least heading towards stable. I've seen several people using it in production (at least for smaller stuff) already, and we're doing so in Vapor for generating DocC docs.
5.9 is known to have the fix for the miscompile in question, whereas it's pretty much a foregone conclusion that it won't be backported to 5.8 at this point.
5.9 now has built-in Backtrace support, and Vapor has been updated accordingly, meaning if it does crash again, it's likely the log will be more helpful.

Mmm, we're actually compiling with a 5.9 toolchain, from Sep 1 or so.
https://gitlab.com/finestructure/spi-base/-/blob/main/Dockerfile

Ooof... then my suggestion is to update your dependencies (and bump your 5.9 to the latest snapshot, which at the moment appears to be sha256:76911e1da04bce21683872c23033f6e2c07f296df7465fce0bf1e1bd01835001.) This will get you the new native backtracking support (if you don't already have it), plus the updated Vapor that doesn't overwrite it with the old version.

That image is from yesterday, dependencies updated on Monday - we should be pretty much up to date, I'd say.
I'm not sure why the backtrace is only partially working. It's been like this for a long time. It's weird 🤔

(I'll def check in the morning that we've got the Vapor update you're referring to.)

For reference, the update in question is Vapor 4.81.0, released ~12 hours ago at the time of this writing.

I just realised we'd updated to that latest Swift 5.9 and run the package updates but it's not deployed yet. It's the revision just after so what was running had just the ShellOut changes!

Another prod hang, 15:57 CET, Sep 9 2023. stderr is just
2023-09-09T10:01:16.343649916Z /bin/bash: line 1: 13830 Segmentation fault      (core dumped) ./Run analyze --env prod --limit 25

stdout logs below - I don't see any backtrace or other bits that indicate issues, except the infamous
2023-09-09T14:09:20.520006338Z [ ERROR ] Connection request (ID 25 timed out. This might indicate a connection deadlock in your application. If you have long-running requests, consider increasing your connection timeout. [component: server, database-id: psql]

analyze-2023-09-09-stderr.logs.zip

Hold on, that segfault is from 10:01 this morning, so certainly not the issue.

This is on prod with Vapor 4.81.0, compiled with Swift 5.9 from Sep 1.
 ################################################################
 #                                                              #
 # Swift Nightly Docker Image                                   #
 # Tag: swift-5.9-DEVELOPMENT-SNAPSHOT-2023-09-01-a                     #
 #                                                              #
 ################################################################

@finestructure If the corefile (indicated by (core dumped)) is still present in the container that runs the command, it can be used to get a backtrace of the crash.

prod, 16:33 CET, Sep 14 2023
I missed the first alert and only checked on this now (21:37 CET). I've pulled the logs and the closest Seg fault is at 13:28 UTC
2023-09-14T13:28:13.410953838Z /bin/bash: line 1: 29712 Segmentation fault      (core dumped) ./Run analyze --env prod --limit 25

with processing continuing until 14:18 UTC while the connection timeout messages leading to the hang are appearing:
2023-09-14T14:18:09.519182656Z [ ERROR ] Connection request (ID 6 timed out. This might indicate a connection deadlock in your application. If you have long-running requests, consider increasing your connection timeout. [component: server, database-id: psql]
2023-09-14T14:18:09.529500376Z [ ERROR ] Connection request (ID 7 timed out. This might indicate a connection deadlock in your application. If you have long-running requests, consider increasing your connection timeout. [component: server, database-id: psql]

The are 15 seg faults in the log file I pulled (ranging from Sep 11 to Sep 14). I think it's safe to say that the seg faults aren't the cause of the hangs.
I looked for a core file in the running container but there was none in the executable's directory nor in a few other places I checked (/var/cache/abrt, /var/spool/abrt, /var/crash). Core dump size it unlimited:
root@fce97f658c3d:/app# ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited

It's late and so I restarted the container for now. Can look into where it ends up some other time - these seem to happen frequently enough (15 times in 3 days).
BTW, the logs do not contain any stack trace info, despite the latest Vapor and Swift 5.9 🤔:
2023-09-14T13:28:08.682435359Z [ INFO ] pulling https://github.com/pointfreeco/swift-dependencies.git in /checkouts/github.com-pointfreeco-swift-dependencies [component: analyze]
2023-09-14T13:28:08.684063885Z [ INFO ] pulling https://github.com/Alamofire/AlamofireImage.git in /checkouts/github.com-alamofire-alamofireimage [component: analyze]
2023-09-14T13:28:09.417655788Z [ WARNING ] stderr: From https://github.com/team-telnyx/telnyx-webrtc-ios
2023-09-14T13:28:09.417694189Z  * [new tag]         0.1.10     -> 0.1.10 [component: analyze]
2023-09-14T13:28:10.842271007Z [ INFO ] throttled 1 incoming revisions [component: analyze]
2023-09-14T13:28:10.937562626Z [ INFO ] throttled 1 incoming revisions [component: analyze]
2023-09-14T13:28:11.961565948Z [ INFO ] Updating 25 packages for stage 'analysis' (errors: 0) [component: analyze]
2023-09-14T13:28:13.410953838Z /bin/bash: line 1: 29712 Segmentation fault      (core dumped) ./Run analyze --env prod --limit 25
2023-09-14T13:28:33.536090365Z [ INFO ] Analyzing (limit: 25) ... [component: analyze]
2023-09-14T13:28:33.592712560Z [ INFO ] Checkout directory: /checkouts [component: analyze]
2023-09-14T13:28:33.593526773Z [ INFO ] Updating 0 packages for stage 'analysis' (errors: 0) [component: analyze]
2023-09-14T13:28:55.544915191Z [ INFO ] Analyzing (limit: 25) ... [component: analyze]
2023-09-14T13:28:55.619826266Z [ INFO ] Checkout directory: /checkouts [component: analyze]

@finestructure For next time, I'd suggest just searching the entire filesystem, e.g. find / -name core (if that finds nothing, I'd give it one more short with a more permissive search, like find / -iname '*core*').

Closing this as fixed now - we haven't had a hang since we removed the TaskGroup in analysis in #2656 (released as 2.91.9).
Finally 🙂🎉
Huge thanks again to Gwynne for all the help!

Closing this as fixed now - we haven't had a hang since we removed the TaskGroup in #2656 (released as 2.91.9).
Finally 🙂🎉
Huge thanks again to Gwynne for all the help!


Unable to run tests on iOS simulator

When running tests on iOS simulator without "flutterSystemPort" we need to consider DEVICE_PORT_RANGE and not SYSTEM_PORT_RANGE
Server logs:
2024-07-11 09:29:38:270 [WD Proxy] Proxying [GET /status] to [GET http://<IP_ADDRESS>:10000/status] with no body
2024-07-11 09:29:38:272 [WD Proxy] connect ECONNREFUSED <IP_ADDRESS>:10000

Simulator Logs:
Appium Server started on 9000
utter) flutter: [APPIUM FLUTTER]  Appium flutter server is listening on port 9000

#51


Rooms Implementation + Clean Up

What
The user can now join/create their own rooms, and message other users in said room
Modifications

Input functions to promises for better flow
Socket logic removed from input.js and into app.js
Room filtering logic move to the server
Room name modifications to follow standard
general -> chat

Try it out guys 😄

Where do u quit the application now? I noticed u got rid of the process.exit but I cant see how u replaced it


Fix git mod sync

Codecov Report

Merging #87 (9b14cfb) into master (993fd01) will increase coverage by 0.05%.
The diff coverage is 35.18%.

@@            Coverage Diff             @@
##           master      #87      +/-   ##
==========================================
+ Coverage   31.62%   31.68%   +0.05%
==========================================
  Files          63       63
  Lines       12523    12628     +105
  Branches      199      200       +1
==========================================
+ Hits         3961     4001      +40
- Misses       8363     8427      +64
- Partials      199      200       +1

Impacted Files
Coverage Δ

src/clj/skylobby/fx/battles_buttons.clj
42.51% <0.00%> (ø)

src/clj/skylobby/fx/bottom_bar.clj
7.14% <0.00%> (ø)

src/clj/skylobby/fx/channel.clj
19.17% <0.00%> (-0.83%)
:arrow_down:

src/clj/skylobby/fx/mod_sync.clj
3.06% <0.00%> (-0.17%)
:arrow_down:

src/clj/skylobby/fx/sync.clj
26.31% <0.00%> (+0.45%)
:arrow_up:

src/clj/spring_lobby.clj
12.19% <12.50%> (-0.02%)
:arrow_down:

src/clj/skylobby/fx/settings.clj
3.18% <22.22%> (-0.11%)
:arrow_down:

src/clj/skylobby/fx/battle.clj
64.14% <71.42%> (+0.17%)
:arrow_up:

src/clj/skylobby/resource.clj
65.38% <90.90%> (+1.74%)
:arrow_up:

src/clj/skylobby/fx.clj
96.82% <95.45%> (+1.26%)
:arrow_up:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 993fd01...9b14cfb. Read the comment docs.


Syntax Error

`node_modules\react-app-rewire-less\index.js:14
);
^
SyntaxError: Unexpected token )
at Object.exports.runInThisContext (vm.js:73:16)
at Module._compile (module.js:543:28)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:488:32)
at tryModuleLoad (module.js:447:12)
at Function.Module._load (module.js:439:3)
at Module.require (module.js:498:17)
at require (internal/module.js:20:19)
at Object. (D:\workspace\reactjs\ant-design\create-react-app-antd                                                                                \create-react-app-antd\config-overrides.js:2:20)
at Module._compile (module.js:571:32)
at Object.Module._extensions..js (module.js:580:10)
at Module.load (module.js:488:32)
at tryModuleLoad (module.js:447:12)
at Function.Module._load (module.js:439:3)
at Module.require (module.js:498:17)
at require (internal/module.js:20:19)
npm ERR! Windows_NT 10.0.15063
npm ERR! argv "C:\Program Files\nodejs\node.exe" "C:\Program Files\nodejs\                                                                                node_modules\npm\bin\npm-cli.js" "start"
npm ERR! node v7.5.0
npm ERR! npm  v4.1.2
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>start: react-app-rewired start
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>start script 'react-app-rewir                                                                                ed start'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the create-react-app-antd                                                                                package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     react-app-rewired start
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs create-react-app-antd
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls create-react-app-antd
npm ERR! There is likely additional logging output above.`
It worked when the trailing comma is removed. Not sure if that is the correct solution.

Show me your config-overrides.js,  react-app-rewired version and react-app-rewire-less version

Same error basically.
From package-lock:
react-app-rewire-less: 2.0.7
react-app-rewired: 1.3.3

From package:
rewire-less: ^2.0.7
rewired: ^1.2.5

config-overrides.js:
const theme = require('./src/theme')`

const { injectBabelPlugin } = require('react-app-rewired');
const rewireLess = require('react-app-rewire-less');

module.exports = function override(config, env) {
  config = injectBabelPlugin(['import', {
    libraryName: 'antd',
    style: true
  }], config);

  config = rewireLess(config, env, { modifyVars: theme });

  return config;
};

theme.js if it matters:
module.exports = {
    "@primary-color": "#58C4D9"
}

Removing ^ from rewire-less in package.json fixes it anyway.

Please update to
react-app-rewire-less: 2.0.8
react-app-rewired: 1.3.5

The usage of  react-app-rewire-less  has changed, because we add a new api compose, detail see react-app-rewire-less .
Now you can use react-app-rewire-less, like:
// with loaderOptions
compose(
  rewireLess.withLoaderOptions(someOptions),
  ...// other rewires
)
// without loaderOptions
compose(
  rewireLess,
  ...// other rewires
)

the bug happened in this code :
const fileLoader = getLoader(
config.module.rules,
rule =>
rule.loader &&
typeof rule.loader === 'string' &&
rule.loader.indexOf(${path.sep}file-loader${path.sep}) !== -1, // the , is unnecessary
);
solution: remove  ,;  just like this:
const fileLoader = getLoader(
config.module.rules,
rule =>
rule.loader &&
typeof rule.loader === 'string' &&
rule.loader.indexOf(${path.sep}file-loader${path.sep}) !== -1   //this  ,
);

I was also telling the same. It worked when the trailing comma is removed. Not sure if that is the correct solution.

Please try - react-app-rewire-less: 2.0.9

react-app-rewire-less: 2.0.9 shows no syntax errors due to this change. Not sure if there are runtime errors. Since this issue is titled "syntax error" then we can close this issue, I guess.


feat: add icq register ts NTRN-352

task: https://p2pvalidator.atlassian.net/browse/NTRN-352
Whereas I liked the auditors highlighted the problem described in the task, I didn't like the solution they suggested. What I particularly didn't like was the fact that even a freshly registered query without any results submitted would have some value assigned to the LastSubmittedResultLocalHeight field. This is not clean, so I've come up with a more gentle solution to the problem to add a dedicated field RegisteredAtHeight which has the very purpose to prevent such exploits.
I've added a test to the ICQ Keeper's tests file called TestRemoveFreshlyCreatedICQ and run it before making any changes revealing the exploit. The changes made in this PR made the test passing.
relative PRs:

https://github.com/neutron-org/neutron-integration-tests/pull/100

tests run: https://github.com/neutron-org/neutron-tests/actions/runs/4416009334

just noticed: please add the new field to neutron-sdk

just noticed: please add the new field to neutron-sdk

done in this PR and in sdk as well: https://github.com/neutron-org/neutron-sdk/pull/86


Event times specified in Encoder

Allow the application to assign event times via the event_time_ns method on the Decoder.
WIP.

@pzel When tests pass, you can merge.


Rate limiting of PacketIn messages should be enforced in the dataplane

Describe the bug
Antrea uses the PacketIn feature of OVS to redirect some packets from the datapath to the local controlplane (Agent). This is required to implement the following Antrea features:

NetworkPolicy Audit Logging
NetworkPolicy Reject Action
Traceflow

In theory, it is possible for someone to generate a large amount of packets in the Pod network that will be sent as PacketIn messages to the Antrea Agent. For example, if a NetworkPolicy rule has been defined to reject UDP traffic from Pod A to Pod B, and Pod A is aware of this policy and is "malicious", Pod A can keep crafting new UDP packets that will match the NetworkPolicy rule and be sent to the Antrea Agent.
This is is an issue because this can greatly increase the CPU usage of the Agent, and can cause other unrelated PacketIn messages to be dropped (e.g. Traceflow messages). It may even interfere with OpenFlow Bundle messages used by the Agent to program the OVS datapath (not confirmed).
There is some irony here: one defines a NetworkPolicy to protect Pod network communications but in doing so (if the NetworkPolicy uses Logging or the Reject action) it can potentially open up the network to DDOS attacks, from the very Pods to which the NetworkPolicy applies.
In Antrea v1.0.0, some steps were taken to mitigate this: https://github.com/vmware-tanzu/antrea/pull/2015. The idea was to limit the maximum CPU usage that can be incurred in the Agent by PacketIn messages, along with using different queues for Traceflow messages and NetworkPolicy messages in the Agent. However, this is a very limited solution. PacketIn messages must be filtered in the OVS datapath if we want a full-proof solution. The current plan is to leverage OpenFlow meters to rate limit PacketIn messages in the datapath, with different meter objects for different types of packets (this will ensure that Traceflow requests are not impacted when a large number of packets match NetworkPolicy logging / reject rules). We want to implement this by Antrea v1.1.
To Reproduce
See https://github.com/vmware-tanzu/antrea/pull/2015 for some benchmarks.
The following steps can generally be followed:

create a NetworkPolicy which uses logging or the reject action
generate synthetic traffic (e.g. with hping3) which will match the NetworkPolicy rule
observe the Agent's CPU usage on the appropriate Node (where traffic is logged or rejected), attempt a Traceflow request where this Node is the source / destination Node

Expected

CPU usage should remain reasonable
Traceflow requests should consistently succeed

Actual behavior

CPU usage can take up a full core
Traceflow requests will sometimes fail

Versions:
Antrea v1.0.0
Additional context
For more information about OpenFlow meters, refer to the OpenFlow 1.3 specification.

Thank you @antoninbas for describing this in such detail. Working on it.


cicd: add Ruff linting and formatting

(making an issue for a github action to detect the presence of the word "gene")


Update pureconfig to 0.17.4

About this PR
📦 Updates com.github.pureconfig:pureconfig from 0.17.3 to 0.17.4
📜 GitHub Release Notes - Changelog - Version Diff
Usage
✅ Please merge!
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

🔍 Files still referring to the old version number
The following files still refer to the old version number (0.17.3).
You might want to review and update them manually.
CHANGES.md

⚙ Adjust future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "com.github.pureconfig", artifactId = "pureconfig" } ]

Or, add this to slow down future updates of this dependency:
dependencyOverrides = [{
  pullRequests = { frequency = "30 days" },
  dependency = { groupId = "com.github.pureconfig", artifactId = "pureconfig" }
}]

labels: library-update, early-semver-minor, semver-spec-patch, old-version-remains, commit-count:1

Codecov Report

Merging #2573 (01e7f55) into main (325f21c) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #2573      +/-   ##
==========================================
- Coverage   78.49%   78.47%   -0.02%
==========================================
  Files         445      445
  Lines       15626    15626
  Branches     2534     2534
==========================================
- Hits        12265    12263       -2
- Misses       3361     3363       +2

see 3 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Question: Embed c:soqlDatatable in other LWC component called from Aura ?

Hi,
I'm trying to add a Quick Action (Aura Component) that includes my own LWC Component, and this LWC component would include a soqlDatatable.
If I add <c:soqlDatatable></c:soqlDatatable> to the Aura component directly, it works, but if I add it to my LWC component, nothing gets rendered.
Do you have any idea why ? (I do not have any error in the console or logs)
Thanks

<template>

The table below should be displayed but does not.

  <c:soqlDatatable query-string="SELECT Id FROM Account"></c:soqlDatatable>
</template>

<aura:component implements="force:lightningQuickActionWithoutHeader,force:hasRecordId">

    <c:soqlDatatable queryString="SELECT Id FROM Account"></c:soqlDatatable>

    <c:myComponent></c:myComponent>
</aura:component>

@Esya
Your LWC shouldn't compile like that, make sure it's using kebab-case:
<template>

The table below should be displayed but does not.

  <c-soql-datatable query-string="SELECT Id FROM Account"></c-soql-datatable>
</template>

FWIW, an Account SELECT works for me, so it's likely a malformed queryString (and then I didn't catch the error somewhere) or your specific org's data coming back is somehow breaking the columns in a way that I didn't anticipate.
I would check a few things:

Remove extra spaces or line breaks from queryString.
Add a LIMIT statement.

I attempt to add one for you, called ROW_LIMITER but maybe 500 is too high still for your dataset.

Start with just the Id, Name columns.

Closing for inactivity, feel free to create a new issue with more detail!


Can't access site because of GraphQL redirects

What I'm trying to achieve
Access website
Steps to reproduce the problem

Installed and migrated everything
After I ran a site it always redirects to graphql, no matter where I would go

What I expected to happen
To load site?
Screenshots

System information
Operating system:
Windows 10 64 BIT

This might be helpful to mention in the docs as this is a change from the prior stable versions of saleor (ie 2.9.1)

Saleor core does not have any other views
This is definitely not obvious and intuitive at all. The docs do not detail nor explain this behaviour, which confuses newcomers (me) that expect the guide to help me setup a local server to be able to test the site out.
The state of the introductory docs right now presents an unnecessary pain point for newcomers and could hinder adoption.

Related issue: #5237

Completely agree with yanxun827. Great piece of code.

Thank you for your feedback. We're working on improvements in the docs and planning to release it this week.

Thank you kindly for the prompt reply. Saleor is really great. I would like
to know if there is documentation around customising the react frontend.
Cheers.
On Mon, 20 Apr 2020, 2:47 pm Marcin Gębala<EMAIL_ADDRESS>wrote:

Thank you for your feedback. We're working on improvements in the docs and
planning to release it this week.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/mirumee/saleor/issues/5191#issuecomment-616345775,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB2GFDRTL2UQYKR2YJIKDYDRNPVYPANCNFSM4KKVQYUQ
.

We don't have docs for the storefront yet. The plan is to finish the SDK (which allows you to interact with the API through JS functions and objects). Once it's done and we fully migrate to using it, it will be also much easier to customize the visual components. Then we'll also release docs for the SDK and basic theming of the app.


unexpected '$httpAdapter' error when i try to send an email

`require_once ‘vendor/autoload.php’;
use SparkPost\SparkPost;
use Ivory\HttpAdapter\CurlHttpAdapter;

$httpAdapter = new CurlHttpAdapter();
$sparky = new SparkPost($httpAdapter, ’API’);
try {
$results = $sparky->transmission->send([
'from' => [
'name' => 'pavneet',
'email' => 'email',
],
'recipients' => [
[
'address' => [
email=> 'email',
],
],
],
'template' => 'suggest_link',
]);
echo 'Congrats! You sent an email using SparkPost!';
} catch (\Exception $exception) {
echo $exception->getAPIMessage()."\n";
echo $exception->getAPICode()."\n";
echo $exception->getAPIDescription()."\n";
}
?>`
I am using the following code to make a request to send an email and i have added the required request to my composer file.
But everytime i execute the file i get unexpected '$httpAdapter' on line 33 error where i try to define the adapter.
What is that i am doing wrong

Hi @pavneet9 - this line:
$sparky = new SparkPost($httpAdapter, ’API’);

Needs to look like this:
$sparky = new SparkPost($httpAdapter, ['key' => $config['YOUR API KEY']]);

Let me know if that helps.

I'm going to go ahead and close this as answered - feel free to re-open if necessary.


Add support for user-specific settings (e.g. ProjectName.fsproj.user)

Some project settings are user-specific, such as the debugger options in a project (e.g. StartAction, StartProgram, etc.).

In CSharp and in VB .NET, changing these settings will graciously create a ProjectName.csproj.user or ProjectName.vbproj.user alongside the project, to store these settings.
FSharp projects don't follow the same behavior, and instead, save the changes inside the project itself (e.g. ProjectName.fsproj)
Repro steps

Create a new Visual F# Library

Open the project properties, and go to the Debug tab

Change the Start Action to "Start external program", and type in a valid path for an executable, e.g. C:\Windows\Notepad.exe

Save the project

Open Windows Explorer and go to the folder where the F# project is located, and notice that a .user file alongside the project was not created

Open the .fsproj and notice that the debug settings have been added to the project file

bin\$(Configuration)\$(AssemblyName).XML
Program
C:\Windows\Notepad.exe

Repeat steps 1 through 4, but now with a C# or VB .NET project, then open Windows Explorer and go to the folder where the C# or VB .NET project is located, and notice that the debug settings were stored in a .user file created alongside the project file

Expected behavior
Debug settings are saved in a separate .user file.
Actual behavior
Debug settings are saved in the .fsproj file directly.
Known workarounds
I don't know of any workarounds at the moment.
Related information

Windows 10
master branch
.NET Runtime
Visual Studio 2017

Duplicate of https://github.com/Microsoft/visualfsharp/issues/91


IDEA import fails with buildSrc project and org.gradle.java.home specified

Given a simple project with:

a buildSrc project
a top-level gradle.properties setting org.gradle.java.home to a JDK 9+

When attempting to import/sync in Intellij IDEA, the daemon process cannot be started.
Expected Behavior
Build will sync/import without error.
Current Behavior
Sync fails after initial "CONFIGURE SUCCESSFUL:
> Task :buildSrc:compileJava UP-TO-DATE
> Task :buildSrc:compileGroovy NO-SOURCE
> Task :buildSrc:processResources NO-SOURCE
> Task :buildSrc:classes UP-TO-DATE
> Task :buildSrc:jar UP-TO-DATE
> Task :buildSrc:assemble UP-TO-DATE
> Task :buildSrc:compileTestJava NO-SOURCE
> Task :buildSrc:compileTestGroovy NO-SOURCE
> Task :buildSrc:processTestResources NO-SOURCE
> Task :buildSrc:testClasses UP-TO-DATE
> Task :buildSrc:test NO-SOURCE
> Task :buildSrc:check UP-TO-DATE
> Task :buildSrc:build UP-TO-DATE

CONFIGURE SUCCESSFUL in 0s
Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.
Please refer to the user guide chapter on the daemon at https://docs.gradle.org/5.1.1/userguide/gradle_daemon.html
Please read the following process output to find out more:
-----------------------
Error: Could not find or load main class java.base.java.lang=ALL-UNNAMED
Caused by: java.lang.ClassNotFoundException: java.base.java.lang=ALL-UNNAMED

Context
This appears to be an issue when IDEA attempts to separately get the model for the buildSrc build.
Symptoms are similar to #7045 but I suspect the underlying cause is different. This is an issue starting the Gradle daemon itself, whereas #7045 is related to starting a Groovy compiler daemon.
Possibly related issues:
https://youtrack.jetbrains.com/issue/IDEA-202975
https://youtrack.jetbrains.com/issue/IDEA-203429
Steps to Reproduce (for bugs)
Clone https://github.com/baron1405/gradle7045 and follow instructions in the README.
Thanks to @baron1405 for this reproducer.

I have verified that this has been fixed in IntelliJ IDEA 2018.3.5. This issue can be closed.


Delete useless links

Motivation

Right now we keep links for translations which are outdated or archived in the database. However, there is no reason to do so.
We have a lot of link objects in the database, and removing unused links will be a nice performance improvement.
If we can rely on only useful links existing, we could also simplify some database queries.
Proposed Solution

Remove all links for content objects which are archived or outdated (There may be more cases where we don't want to have links)
Alternatives

Do nothing
Additional Context

https://chat.tuerantuer.org/digitalfabrik/pl/ykbppm73hi8pim98khpgja6z9h

This should be an infra ticket. There links for old versions are not kept in the database on purpose, but they are a side effect of the link check library not being race condition safe.
If you find a specific bug, where old links are not deleted as they should, this ticket would be valid, but last time I checked, I did not find such a problem.


Default value for  first_element_offset_px

During development with this plugin, I noticed that when all friends were excluded, first_element_offset_px was triggering a javascript error (since array all_friends) was empty. so i just set the offset in that case to be 0.

Closing as may be stale.


Can't upgrade from 4012 -> 4013

I'm unable to upgrade to 4013. Get the following "Download Failed" error:
An error occured while updating: You do not have permission to access the requested resource.
This is on El Capitan, if it makes a difference.

Works now. It's possible that this was caused by a corporate firewall issue.

Didn’t have El Cap until after I had upgraded to 1.3.1, but I went through the Quicksilver update in a VM and it worked as expected. I agree that it sounds like a temporary network issue.


dataset for freebase-likq sample does not exist

FireBase Sample dataset does not exist.
https://graphengine.blob.core.windows.net/public-data"
freebase-full-dataset-20170410.zip file does not exist.
System.AggregateException
HResult=0x80131500
Message=One or more errors occurred. (The remote server returned an error: (404) The specified resource does not exist..)
Source=System.Private.CoreLib
StackTrace:
at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
at System.Threading.Tasks.Task.Wait()
at freebase_likq.Program.DownloadDataFile() in O:\TrinityGraphEngine\GraphEngine\samples\freebase-likq\freebase-likq\Program.cs:line 122
at freebase_likq.Program.Initialize() in O:\TrinityGraphEngine\GraphEngine\samples\freebase-likq\freebase-likq\Program.cs:line 58
at freebase_likq.Program.Main(String[] args) in O:\TrinityGraphEngine\GraphEngine\samples\freebase-likq\freebase-likq\Program.cs:line 29
This exception was originally thrown at this call stack:
[External Code]
Inner Exception 1:
WebException: The remote server returned an error: (404) The specified resource does not exist..

This is a duplicate - please detele!

This is a duplicate of 341. This one will be closed.


BluetoothDevice.FromIdAsync  Why is it empty? I already know the ID of Bluetooth.

Why is it empty? I already know the ID of Bluetooth.

What platform are you running on? For example on Windows the ID can be a device id which is different from a Bluetooth address. Please post the code you are using and some detail about the devices you are using.

What platform are you running on? For example on Windows the ID can be a device id which is different from a Bluetooth address. Please post the code you are using and some detail about the devices you are using.

I have three Bluetooth devices, two of which can be connected by ID, but one can't, but this device can be paired manually, and then get the devices in the pairing list, and then connect. It's strange.

Not all devices can be connected without pairing. The library doesn't provide programmatic pairing for Bluetooth LE devices. It's certainly something which is possible on most platforms and so I hope to add it in a future version.

Not all devices can be connected without pairing. The library doesn't provide programmatic pairing for Bluetooth LE devices. It's certainly something which is possible on most platforms and so I hope to add it in a future version.

Sometimes, for example, after searching, you can use ID connection?