[Bug]: Theme ID mismatch

Please confirm that you have:

[x] Searched existing issues to see if your issue is a duplicate. (If you’ve found a duplicate issue, feel free to add additional information in a comment on it.)
[x] Reproduced the issue in the latest CLI version.

In which of these areas are you experiencing a problem?
Theme
Expected behavior
When I run theme dev and follow the preview URL (http://<IP_ADDRESS>:9292), I expect to preview my development theme locally and for the CLI process to not exit.
Actual behavior
CLI exits after following error:
Theme ID mismatch: expected<PHONE_NUMBER>95 but got<PHONE_NUMBER>43.
[0] │  Request ID: 7f690ce7-9d82-4419-9777-9a26e7efeb24-1733758184
[0] │  URL: REDACTED
[0] │
[0] │  This is likely related to an issue in upstream Shopify APIs.
[0] │  Please try again in a few minutes and report this issue:
[0] │  https://github.com/Shopify/cli/issues/new?template=bug-report.yml

Note this doesn't happen using the full preview URL, only the local URL.
Verbose output

Verbose output
2024-12-10T14:15:18.228Z:
Running system process:
· Command: npm prefix
· Working directory: /Users/marianne/Downloads/dawn-main 4

2024-12-10T14:15:18.339Z: Obtaining the dependency manager in directory /Users/marianne/Downloads/dawn-main 4...
2024-12-10T14:15:18.563Z: Request to https://ultrafine-codepen.myshopify.com/admin/api/unstable/themes/139056316590/assets/bulk.json completed in 380 ms
With response headers:
- content-type: application/json; charset=utf-8
- server-timing: processing;dur=153, cfRequestDuration;dur=358.000040, ipv6
- x-request-id: 0f60fc01-e31b-4796-a9fd-c4cccfb745b4-1733840118

2024-12-10T14:15:18.563Z: File Upload Results:
-sections/header-group.json: success
2024-12-10T14:15:18.563Z: Uploading the following files:
-templates/404.json
-templates/article.json
-templates/cart.json
-templates/password.json
-templates/product.json
-templates/customers/account.json
-templates/customers/activate_account.json
-templates/customers/addresses.json
-templates/customers/order.json
-templates/customers/register.json
2024-12-10T14:15:18.564Z: Uploading the following files:
-templates/customers/reset_password.json
-templates/customers/login.json
2024-12-10T14:15:18.640Z: Request to https://monorail-edge.shopifysvc.com/v1/produce completed in 204 ms
With response headers:
- x-request-id: fcfd6bc2-f0ca-40be-af4c-7fa72863660e

2024-12-10T14:15:18.640Z: Analytics event sent: {
"command": "theme dev",
"time_start":<PHONE_NUMBER>909,
"time_end":<PHONE_NUMBER>224,
"total_time": 3315,
"success": true,
"cli_version": "3.70.0",
"ruby_version": "",
"node_version": "22.12.0",
"is_employee": false,
"uname": "darwin arm64",
"env_ci": false,
"env_plugin_installed_any_custom": false,
"env_plugin_installed_shopify": "[\"@shopify/cli\"]",
"env_shell": "zsh",
"env_device_id": "5adba125250ffb656acf6734e3c045ce9e33d646",
"env_cloud": "localhost",
"env_package_manager": "unknown",
"env_is_global": true,
"env_auth_method": "device_auth",
"cmd_app_warning_api_key_deprecation_displayed": false,
"cmd_all_timing_network_ms": 3139,
"cmd_all_timing_prompts_ms": 0,
"cmd_all_launcher": "unknown",
"cmd_all_topic": "theme",
"cmd_all_plugin": "@shopify/theme",
"cmd_all_force": false,
"cmd_all_verbose": true,
"cmd_all_path_override": true,
"cmd_all_path_override_hash": "39d54f2ef5770121a6b1608021ed33d2b4bc617d",
"cmd_all_last_graphql_request_id": "7e945e2a-d62d-4be9-b16f-c80058587279-1733840117",
"cmd_all_timing_active_ms": 175,
"cmd_all_exit": "ok",
"user_id": "f2c7e177-175b-464e-8234-0f4d94628b70",
"args": "--store ultrafine-codepen --verbose",
"env_plugin_installed_all": "[\"@shopify/cli\"]",
"metadata": "{\"extraPublic\":{},\"extraSensitive\":{}}"
}
2024-12-10T14:15:18.640Z: Completed command theme dev
2024-12-10T14:15:19.232Z: Request to https://ultrafine-codepen.myshopify.com/admin/api/unstable/themes/139056316590/assets/bulk.json completed in 668 ms
With response headers:
- content-type: application/json; charset=utf-8
- server-timing: processing;dur=365, cfRequestDuration;dur=555.999756, ipv6
- x-request-id: 1a503c05-2868-498e-a706-9812f04e2518-1733840118

2024-12-10T14:15:19.234Z: File Upload Results:
-templates/customers/reset_password.json: success
-templates/customers/login.json: success
2024-12-10T14:15:20.388Z: Request to https://ultrafine-codepen.myshopify.com/admin/api/unstable/themes/139056316590/assets/bulk.json completed in 1824 ms
With response headers:
- content-type: application/json; charset=utf-8
- server-timing: processing;dur=1511, cfRequestDuration;dur=1805.999994, ipv6
- x-request-id: fe1ca2a2-e279-4898-b062-bb01914aa2cf-1733840118

2024-12-10T14:15:20.389Z: File Upload Results:
-templates/404.json: success
-templates/article.json: success
-templates/cart.json: success
-templates/password.json: success
-templates/product.json: success
-templates/customers/account.json: success
-templates/customers/activate_account.json: success
-templates/customers/addresses.json: success
-templates/customers/order.json: success
-templates/customers/register.json: success
2024-12-10T14:15:20.391Z: Files to be deleted:

2024-12-10T14:15:35.728Z: → Rendering https://ultrafine-codepen.myshopify.com/?_fd=0&pb=0 (with )...
2024-12-10T14:15:36.135Z: Request to https://ultrafine-codepen.myshopify.com/?_fd=0&pb=0 completed in 406 ms
With response headers:
- content-type: text/html; charset=utf-8
- etag: "cacheable:e569a60cd343b1511164f23d6115a9b2"
- server-timing: processing;dur=248;desc="gc:14", db;dur=64, db_async;dur=1.586, edge_cart;dur=8.27;desc="trips=1", render;dur=68, asn;desc="12353", edge;desc="LIS", country;desc="PT", theme;desc="136432025774", pageType;desc="index", servedBy;desc="7mlc", requestID;desc="db042b07-5996-4422-9c59-79399c300c2d-1733840135", cfRequestDuration;dur=336.000204, ipv6
- x-request-id: db042b07-5996-4422-9c59-79399c300c2d-1733840135

2024-12-10T14:15:36.135Z: ← 200 (request_id: db042b07-5996-4422-9c59-79399c300c2d-1733840135)
• 14:15:36 Request »    GET 200 / 336ms
╭─ error ──────────────────────────────────────────────────────────────────────────────╮
│                                                                                │
│  Theme ID mismatch: expected<PHONE_NUMBER>90 but got<PHONE_NUMBER>74.                      │
│  Request ID: db042b07-5996-4422-9c59-79399c300c2d-1733840135                         │
│  URL: https://ultrafine-codepen.myshopify.com/?_fd=0&pb=0                            │
│                                                                                │
│  This is likely related to an issue in upstream Shopify APIs.                        │
│  Please try again in a few minutes and report this issue:                            │
│  https://github.com/Shopify/cli/issues/new?template=bug-report.yml

Reproduction steps

Run shopify theme dev
Follow local preview link
Notice error in terminal

Operating System
macOS 14.6.1
Shopify CLI version (shopify --version)
@shopify/cli/3.71.4
Shell
zsh
Node version (run node -v if you're not sure)
v20.9.0
What language and version are you using in your application?
No response

Hi 👋  can you try with 3.70.0 and let me know if it works with that version?
Also, are you using theme-kit-access? Perhaps you can update the issue with a full --verbose log so that we can see this kind of detail 🙏

@karreiro I remember we had this issue some months ago and it was related so some inadvertent change in SFR. Do you remember what you checked that time?

Hi @frandiox , yes it's still happening with 3.70.0 and I'm not using theme-kit-access – just previewing a clean copy of Dawn on my dev store, ultrafine-codepen.myshopify.com.
For context I uninstalled the CLI via Homebrew and reinstalled 3.70.0 via NPM this time. I also updated my node version in order to to that. Running shopify --version now returns the following: @shopify/cli/3.70.0 darwin-arm64 node-v22.12.0.
I added the verbose output above.
Let me know if you need any more info, thanks!

Any advancements on this issue? It's been tormenting me for some time now.

Same, actually I can't use the CLI at all for a while now. For some reasons the CLI is requesting urls I don't even navigate too (could be an app), such as GET 200 /a/pwa/[myshopifyurl]/manifest 576ms and then I get the Theme ID mismatch. Before I used to be able to restart the command process, but now everytime it breaks.

@MaxDesignFR Could you open an issue for that behaviour please?
Might be a different issue (in combination of with the Theme ID mismatch) so I want to capture as much information as possible.
For anyone else facing this issue, can you please try:

Verify that you don't have multiple installs from different package managers
Clear legacy data rm -rf ~/.shopify-app-cli
Clear any locally stored data rm -rf ~/Library/Preferences/shopify-cli-{development-theme-config,kit,theme-conf,theme-store-password}-nodejs

@xmadscientist can you let us know if these steps, or the steps @marianneciara tried work for you?

@jamesmengo I did another thing which seemed to temporarily fix it.

Run shopify theme dev -e=...
Instead of directly going to the URL of the page I'm working on, I first go to <IP_ADDRESS>:9292 (root)
Then I manually navigate to the page
Dev seems to work again

However, when running shopify theme dev -e=..., proceed to directly reloading the page that I'm working on, it immediately throws the theme ID mismatch error in the console (not even triggered from saving a file—it happens on the GET request). Every time!
I will try the steps above and see if reloading directly on the page that I'm working on works afterwards.

@jamesmengo the steps that you posted seemed to do the trick!!!! 🥳


high packet loss from continuous peer reconnections in some cases with windows to windows connections

Hi, from a windows 10 pro device with netbird 0.23.7 connecting to a windows 10 pro I saw frequent freeze of rdp session and a ping packet lost during these problems.
I updated destination device to netbird 0.23.7 but problem persist and 12% of packet lost
Statistiche Ping per <IP_ADDRESS>:
    Pacchetti: Trasmessi = 49, Ricevuti = 43,
    Persi = 6 (12% persi),
Tempo approssimativo percorsi andata/ritorno in millisecondi:
    Minimo = 16ms, Massimo =  1717ms, Medio =  59ms

I tried connetting to a linux device and no issue or packet lost detected
I tried connecting to windows 2019 standard, netbird 0.23.7 and same issue, even if less frequest still not usable and 7% packet lost
Statistiche Ping per <IP_ADDRESS>:
    Pacchetti: Trasmessi = 107, Ricevuti = 99,
    Persi = 8 (7% persi),
Tempo approssimativo percorsi andata/ritorno in millisecondi:
    Minimo = 15ms, Massimo =  3390ms, Medio =  75ms

I tried to check the client.log but there are only these lines on the test (and issues) time range:
windows 10 pr source
2023-10-05T15:43:51+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:54500
2023-10-05T15:45:41+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:57599
2023-10-05T15:46:01+02:00 INFO client/internal/peer/conn.go:337: connected to peer dbrzwmy0grg6A18e4h4QPG32b/g2B68boB5SZ1lwPEo=, endpoint address: <IP_ADDRESS>:55932
2023-10-05T15:46:17+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:54166
2023-10-05T15:46:49+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:53421
2023-10-05T15:48:08+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:58581
2023-10-05T15:48:44+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:58032
2023-10-05T15:49:39+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:50940
2023-10-05T15:50:04+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:64547
2023-10-05T15:50:47+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:52384
2023-10-05T15:51:50+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:64640
2023-10-05T15:52:22+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:56654
2023-10-05T15:52:52+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:60795
2023-10-05T15:53:45+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:54856
2023-10-05T15:54:05+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:57528
2023-10-05T15:54:44+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:56627
2023-10-05T15:55:04+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:56431
2023-10-05T15:55:24+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:59815
2023-10-05T15:58:08+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:62078
2023-10-05T15:58:46+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:53470
2023-10-05T16:00:08+02:00 INFO client/internal/peer/conn.go:337: connected to peer gTjUAU9z/zwMkmd9iaIY/6UDPL3NgyYIZSq9Bb8kNGY=, endpoint address: <IP_ADDRESS>:51862

windows 2019 destination
2023-10-05T15:43:49+02:00 INFO client/internal/peer/conn.go:337: connected to peer dbrzwmy0grg6A18e4h4QPG32b/g2B68boB5SZ1lwPEo=, endpoint address: <IP_ADDRESS>:51820
2023-10-05T15:43:51+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50782
2023-10-05T15:44:04+02:00 INFO client/internal/peer/conn.go:337: connected to peer QEpV+NKRdFU82Sw/wBvCVn2AOxyEVUilicveHQ6kqgc=, endpoint address: <IP_ADDRESS>:50248
2023-10-05T15:45:42+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50205
2023-10-05T15:46:18+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50826
2023-10-05T15:46:50+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50349
2023-10-05T15:48:08+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50572
2023-10-05T15:48:45+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:49612
2023-10-05T15:49:39+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:49890
2023-10-05T15:50:05+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50210
2023-10-05T15:50:47+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50968
2023-10-05T15:51:50+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:49554
2023-10-05T15:52:22+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:49404
2023-10-05T15:52:53+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50642
2023-10-05T15:53:46+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:49490
2023-10-05T15:54:05+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50317
2023-10-05T15:54:44+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50758
2023-10-05T15:55:04+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:49770
2023-10-05T15:55:24+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50484
2023-10-05T15:58:09+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:50704
2023-10-05T15:58:46+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:49220
2023-10-05T16:00:08+02:00 INFO client/internal/peer/conn.go:337: connected to peer 659jTAbdpL8qPoe8nwvmu3Z5zJRdUd46KFr+t0FOyUA=, endpoint address: <IP_ADDRESS>:49619

I don't know if there is something useful for you
if you need more info tell me how to debug and provide useful data
thanks for any reply and sorry for my bad english

Hello @Fantu can you share more details about your NetBird configuration? Do you have network routes involved and what are the networks the two peers are part of?

@mlsmaycon thanks for reply, netbird server is selfhosted, there are no network routes on netbird configuration and the devices of the tests have 2 groups ("All" and "M2R")
on access control page I see: "M2R"<->"M2R" protocol ALL ports ALL
the Default related to "All"<->"All" is disabled instead
sorry if I have almost no knowledge of netbird and have only done a few quick things when my boss told me to install and then 2 times (including this one) where he reported problems to fix
tell me if the info provided is wrong or incomplete

Thanks @Fantu , we see a lot of reconnection in your logs. Can you share more details about the network of both machines? Are they in the same office, data center, or cloud?
Regarding the connection, they also seem to be using relay, can you ensure that the server where you are running NetBird services, have the following ports accessible from the internet:

Open TCP ports 80, 443, 33073, 10000 (Dashboard HTTP & HTTPS, Management gRCP & HTTP APIs, Signal gRPC API respectively) on your server.
Coturn is used for relay using the STUN/TURN protocols. It requires a listening port, UDP 3478, and range of ports, UDP 49152-65535, for dynamic relay connections. These are set as defaults in setup file, but can be configured to your requirements.

netbird server and destination devices are in the same LAN on office
and here my first question: why these reconnections on LAN connections which in theory should have no problems?
source device is in different network on my home and use internet, anyway both internet connection works correctly and using a normal wireguard vpn to office network instead of netbird I can work on same devices without any issue and 0 packet lost
netbird server seems have all ports ok FWIK and with port forwarding on NAT of gateway. there was an issue on custom signal port on start but was solved with https://github.com/netbirdio/netbird/commit/8d18190c9412268aa0279a4e89d4e02cbd1bdb46
about coturn range is custom but range in coturn configuration is correct and udp ports list viewed with "ss" command are all in the correct range
I don't know a way to test and check are really working correctly, is there?
anyway I have another question: connection with coturn should be only the relay one when direct connection is not possible or on start when direct connection are not started and after should use direct connections between devices or am I wrong?
at least take a quick look at the documentation
but from the logs I don't seem to see direct connections between the devices and I also don't understand why in the source device logs there is always the loopback IP as an endpoint unlike the devices in the LAN which have the IP of the netbird server or of other devices. shouldn't there be the internet IP of the netbird server instead on the source device?
is there a cli command to view all devices, the active connections and their details (or if are missed connection) or some other useful command for debugging? for example of have a small experience with zerotier and on it there is zerotier-cli peers useful for similar thing
thanks for any reply and sorry for my bad english

I found the cli command to check peers connection status netbird status -d --filter-by-ips ... and I did some tests
with both devices (source and destination) on LAN (or connected with other stable VPN) there is P2P connetion stable
Peers detail:
 administration.netbird.selfhosted:
  NetBird IP: <IP_ADDRESS>
  Public key: xxx
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): srflx/host
  Last connection update: 2023-10-09 12:22:54

connection from remote instead have a relay connection that continue to disconnect/reconnect:
Peers detail:
 administration.netbird.selfhosted:
  NetBird IP: <IP_ADDRESS>
  Public key: xxx
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  Last connection update: 2023-10-09 14:38:31

...

Peers detail:
 administration.netbird.selfhosted:
  NetBird IP: <IP_ADDRESS>
  Public key: xxx
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  Last connection update: 2023-10-09 14:39:06

...

Peers detail:
 administration.netbird.selfhosted:
  NetBird IP: <IP_ADDRESS>
  Public key: xxx
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  Last connection update: 2023-10-09 14:41:19

for the brief moments it remains connected it seems works well and packets seems are lost only during disconnections
Is it perhaps that it keeps trying to connect in P2P but fails and goes back to relay or are there other problems?
If you need further information/tests let me know

I did some tests with custom build with updated ICE (https://github.com/M2Rbiz/netbird/commit/3b00ee9a4241b26f313a6cdfcef8d7891a4a7809) but issue persist :(
issue seems happen only when one or both peer is inside a network with symmetrical nat (pfsense, freebsd don't support fullcone for now https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219803) and double nat (the router is fritzbox that have fullcone nat)
I suppose other users have symmetrical nat, nobody saw this issue before?
on major of cases netbird is more stable that zerotier and faster to connect/reconnect between peers except this case where with netbird is unusable and with zerotier even if with packet loss, and occasional rdp disconnect is enough usable
at the moment I don't know what else to do and my boss seems willing to abandon the use of netbird
any advice is appreciated
thanks for any reply and sorry for my bad english

here a client log debug on a peer with the issue:
https://paste.debian.net/1295455/
netbird status -d --filter-by-ips <IP_ADDRESS>
Peers detail:
 administration.netbird.selfhosted:
  NetBird IP: <IP_ADDRESS>
  Public key: dbrzwmy0grg6A18e4h4QPG32b/g2B68boB5SZ1lwPEo=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  Last connection update: 2023-10-18 18:44:52

peer dbrzwmy0grg6A18e4h4QPG32b/g2B68boB5SZ1lwPEo= is one with reconnect loop issue
I don't understand what the problem could be looking the logs, maybe I don't have enough experience, does anyone have any ideas/advice please?

From a fast test with managed netbird service on same device the issue don't happen
So seems happen only when one windows peer in lan with netbird server connect to external peer.
I also saw the other cases, using the same selfhosted netbird server, where windows peer of one external network connect to peers of other external network don't have the issue and even if stay in relay is stable.
I'll probably do more tests next week to be sure.
small question OT:
I tried managed netbird for this fast test and I saw only peers list and is only possible add peers with login,
I didn't saw these feature listed in free plan:  Access controls, Private DNS, Network routes, Management activity logging
I suppose the cause is my user result with role "user" and not possible to change to admin
should not be set "admin", as the first user of the selfhosted?

Hello @Fantu it should be set as admin, but is very likely that there is an admin account for your domain already.
Can you reach out on slack so we can discuss your case in detail?

thanks for the reply, there is another user with the same domain who had signed up before and there were no peers because he only had that signup/login by mistake. I would recommend adding something in the managed service interface to make sure that cases like these are seen/understood
returning in topic:
from the first tests this morning with the same selfhost server adding peers from another external network with pfsense (therefore nat symmetrical and double like the office) and was in relay without problems it seemed confirmed that the problem was with windows peer in lan with netbird server.
instead then trying from other peers from 2 other networks (both always w10) towards an office peer (with which we had problems) in those cases there were no problems and one also has a "worse" wifi connection instead of vdsl or FTTH but he had no disconnections, no packet loss but only some rare packets with latency of about 1 second
so at the moment I have no idea what other tests to do and under what conditions this problem occurs

Having the exact same issue.

We basically only use one Peer as exit node for a bunch of services and have a self-hosted Management/Admin server with a Public IP and the required port-openings.
Our exit node is behind a PFSense and at this moment I think it's related to STUN/ICE udp-holepunching not working.
I'll keep troubleshooting and update if I have any findings that might help you out.

About the issue in my office it seems to be related to the netbird selfhost server (more exactly issue seems related to coturn) in a network with pfsense behind another nat and without access to a public ip. using an external coturn with single nat with public IP and the managed service there were no problems.
After my employer has started to put netbird using the managed service on some devices of some customers (I think he will want to completely replace zerotier with that) and problems of frequent disconnections have started on some windows devices even using the managed service.
After some debugging attempts without being able to find the cause and in which the netbird debug logs did not give any details regarding the disconnection or possible problems to work to try to resolve or improve yesterday thanks to the help of Pascal Fisher I finally managed to find something which could be useful. Initially try to enable ICE debug logs was difficult because trying $env:NB_LOG_LEVEL="debug"; $env:PIONS_LOG_DEBUG="all"; netbird service run | Tee-Object -FilePath "disconnect.logs" was not working, as seems don't possible run netbird in foreground like with linux. I tried to add env. variable also in the windows service with regedit and in other way but netbird client.log was always without ICE debug.
Pascal Fisher found a solution hardcoding it in custom build I after used for debug on peers with disconnect issue.
Here a cut from one ICE debug log with strange thing spotted after disconnect:
ice WARNING: 2023/12/21 15:54:07 Discard message from (udp4 relay <IP_ADDRESS>:58033 related <IP_ADDRESS>:54256), unknown TransactionID 0x8f16f56aba969acc059bffeb
turnc DEBUG: 15:54:07.929646 allocation.go:132: Refresh timer 1 expired
turnc DEBUG: 15:54:07.945809 allocation.go:173: Set new nonce with 16 bytes
turnc DEBUG: 15:54:07.945809 allocation.go:56: Refresh allocation: 438, got new nonce.
turnc DEBUG: 15:54:07.971915 allocation.go:127: Refresh permissions successful
ice WARNING: 2023/12/21 15:54:11 Discard message from (udp4 relay <IP_ADDRESS>:58033 related <IP_ADDRESS>:54256), unknown TransactionID 0x13a8d2430fae1880ea81004d
ice INFO: 2023/12/21 15:54:17 Setting new connection state: Disconnected
turnc DEBUG: 15:54:17.797803 allocation.go:77: Send refresh request (dontWait=true)
ice WARNING: 2023/12/21 15:54:17 Failed to read from candidate udp4 relay <IP_ADDRESS>:37089 related <IP_ADDRESS>:64981: read udp <IP_ADDRESS>:37089: use of closed network connection
turnc DEBUG: 15:54:17.798374 allocation.go:84: Refresh request sent
turnc DEBUG: 15:54:17.798374 client.go:177: Failed to read: read tcp4 <IP_ADDRESS>:64981-><IP_ADDRESS>:443: use of closed network connection. Exiting loop
ice INFO: 2023/12/21 15:54:17 Setting new connection state: Closed
ice DEBUG: 15:54:25.123955 agent.go:397: Started agent: isControlling? true, remoteUfrag: "OicdTxJdOSmLLGcq", remotePwd: "mwlAbQuRQYPTcrZReOiJwwAWuPJZXLzw"
ice INFO: 2023/12/21 15:54:25 Setting new connection state: Checking
ice WARNING: 2023/12/21 15:54:25 Failed to ping without candidate pairs. Connection is not possible yet.
ice DEBUG: 15:54:25.147639 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.147639 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.180758 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.180758 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.182850 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.182850 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.221630 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.221630 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: turn.netbird.io:443: lookup turn.netbird.io: no such host
ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: stun.netbird.io:5555: lookup stun.netbird.io: no such host
ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.315599 agent.go:817: Ignore duplicate candidate: udp4 srflx <hiddenip>:51820 related <IP_ADDRESS>:51820
turnc DEBUG: 15:54:25.364673 client.go:119: Resolved TURN server turn.netbird.io:443 to <IP_ADDRESS>:443
turnc DEBUG: 15:54:25.411742 udp_conn.go:66: Initial lifetime: 600 seconds
turnc DEBUG: 15:54:25.411742 udp_conn.go:81: Started refresh allocation timer
turnc DEBUG: 15:54:25.411742 udp_conn.go:84: Started refresh permission timer
turnc DEBUG: 15:54:25.462244 udp_conn.go:438: Channel binding successful: <IP_ADDRESS>:51820 16384
ice INFO: 2023/12/21 15:54:25 Failed to send packet: CreatePermission error response (error 403: Forbidden IP)
turnc DEBUG: 15:54:25.509529 udp_conn.go:438: Channel binding successful: <IP_ADDRESS>:51820 16385
turnc DEBUG: 15:54:25.533151 udp_conn.go:438: Channel binding successful: <hiddenip>:61335 16386
ice DEBUG: 15:54:25.533698 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true

the strange thing found is:
ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: turn.netbird.io:443: lookup turn.netbird.io: no such host
ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: stun.netbird.io:5555: lookup stun.netbird.io: no such host

resolution on windows works correctly, I also tried to change windows dns server and restart netbird but the error still happen.
I tried for example to keep only <IP_ADDRESS> (google) as dns server and keep active ping on it and turn.netbird.io and was both without packet lost when netbird peer disconnect and have same error in ICE debug log.
On linux peers disconnect issue is rare, but I tried to debug also it with:
sudo bash -c "NB_LOG_LEVEL=debug PIONS_LOG_DEBUG=all netbird service run" | tee disconnect.logs
and on one ubuntu 20.04 after a disconnect I had the same resolve error.
I don't know if this problem is linked to that of disconnections, which are a significant problem on some Windows peers, or just causes a possible delay in the connection/reconnection between the peers.
Is there an issue with netbird internal dns resolution?
Any answer and advice are welcome.
If there are other debugging possibilities that might be useful that I don't know about let me know please.

Working ICE debug method on Windows, thanks to @mlsmaycon :

download psexec: https://learn.microsoft.com/en-us/sysinternals/downloads/psexec
add PION_LOG_TRACE=all as system environment variables
stop netbird: netbird service stop
run the following in a privileged PowerShell session: .\PsExec64.exe -s cmd.exe /c "netbird up -F -l debug > c:\windows\temp\netbird.out.log 2>&1"
So the netbird.out.log will contain netbird debug log and ICE trace logs

I have the same problem, I change NB_ICE_DISCONNECTED_TIMEOUT_SEC to 13，solved this problem.
on centos: vi /etc/sysconfig/netbird
# NB_LOG_LEVEL=debug
# PION_LOG_DEBUG=all
NB_ICE_DISCONNECTED_TIMEOUT_SEC=13

i think iceDisconnectedTimeoutDefault should be more than iceKeepAliveDefault * 2：
if iceDisconnectedTimeoutDefault <  iceKeepAliveDefault * 2

ping -> lost, -> disconnected -> netbird clean peer and reconnect

if iceDisconnectedTimeoutDefault >  iceKeepAliveDefault * 2:

ping -> lost, ping -> lost, -> disconnected -> netbird reconnect
ping -> lost, ping -> pong:  no disconnected, netbird ok  <--

https://github.com/netbirdio/netbird/blob/main/client/internal/peer/env_config.go#L14
	envICEKeepAliveIntervalSec   = "NB_ICE_KEEP_ALIVE_INTERVAL_SEC"
	envICEDisconnectedTimeoutSec = "NB_ICE_DISCONNECTED_TIMEOUT_SEC"

https://github.com/netbirdio/netbird/blob/main/client/internal/peer/conn.go#L27
	iceKeepAliveDefault           = 4 * time.Second
	iceDisconnectedTimeoutDefault = 6 * time.Second # <-- may be 10  * time.Second ( 2.5 * iceKeepAliveDefault)

https://github.com/pion/ice/blob/master/agent_config.go#L21
	// keepaliveInterval used to keep candidates alive
	defaultKeepaliveInterval = 2 * time.Second

	// defaultDisconnectedTimeout is the default time till an Agent transitions disconnected
	defaultDisconnectedTimeout = 5 * time.Second

https://github.com/pion/ice/blob/master/agent.go#L631
	case a.disconnectedTimeout != 0 && disconnectedTime > a.disconnectedTimeout:
		a.updateConnectionState(ConnectionStateDisconnected)

Thanks for the information, I did some tests with keepalive lower without result, but I hadn't thought carefully about the disconnect timeout.
I tried today, starting with NB_ICE_DISCONNECTED_TIMEOUT_SEC to 30 on one case with continuous nearby disconnections, setting it on both "source" and "destination" peer and solved.
After I tried NB_ICE_DISCONNECTED_TIMEOUT_SEC to 10 which seems more reasonable to not increase the reconnection time too much in case the connection actually needs to be re-established and also with that 0 disconnection and 0 packet lost in for the test period.
Unfortunately, however, I have not yet been able to test with the ongoing problem on some networks which are occasionally very problematic and I have some doubts whether increasing the disconnection timeout will resolve.

i got the same problem... any news how to fix it?

For example, if you have frequent disconnections between 2 peer windows, on both peers open PowerShell as administrator and do:
[Environment]::SetEnvironmentVariable("NB_ICE_DISCONNECTED_TIMEOUT_SEC", "10", "Machine")
After restart netbird service:
netbird service stop
netbird service start

After you will try, can you write if increase disconnect timeout to 10 seconds has solved your problem, please?

Only the Period between the interrupts increase, but its not completely gone. I tried multiple Values on both Side from 5 to 30. Also I tried a Debian 12 that connect to the Windows Server and there is no Interruption.
My current Setup: I use Windows 11 (22H2) that connects to a Windows Server 2202 (21H2) over RDP. The Netbird Instance is Self-Hosted on an ESXi-Hosts that got an OPNSense as a Firewall to NAT the Inbound needed Ports.

Probably is the same other issue I have on my office using netbird selfhosted server in LAN behind pfsense, the problem seems double nat and with only 1 public ipv4 I can't assign one on pfsense.
Increase timeout decreased the issue, specify external-ip in the turnserver.conf seems also helped.

its not only 1 public IP-Address, but in the same subnet range...
the office network send the information e.g. <IP_ADDRESS> (my windows 11 machine + Netbird Instance) over a Palo Alto Firewall to the e.g. <IP_ADDRESS> (windows server) that has the OPNSense... i will try later in the evening to modify the turnserver.conf.

@Fantu did you make any progress here? I'm having some terribly annoying inconsistencies trying to stay connected to a remote Linux box behind a NAT'd 4G connection from my Windows 11 PC. Are we saying that the connection lifetime is too short on Windows to properly handle navigating NAT delays?

Hey @webash , we have had good results setting the following environment variable on the client machines
Variable: NB_ICE_DISCONNECTED_TIMEOUT_SEC
Value: 10
This can be set in powershell using
[Environment]::SetEnvironmentVariable("NB_ICE_DISCONNECTED_TIMEOUT_SEC", "10", "Machine")

Do you know how to set this on Linux, too? I think its the sporadic nature of the 4G connection, but being the target that is the issue; so I think its disconnecting too often when actually it should just be a bit more patient. In some cases I seem to be able to keep an SSH session open to do a bunch of stuff, other times it drops rapidly.

Set disconnect timeout to 10 seconds solve the problem?
For Windows device, see here: https://github.com/netbirdio/netbird/issues/1195#issuecomment-1962321207
On Linux device I didn't remember if export of the environment variable and restart of netbird is working, I only remember I used env. variable only on manual foreground testing that can be different from the service

Thanks for the pointer to the part in the code that shows the defaults! That also lead me to this part of the documentation that describes how to set the environment variables for the Linux netbird daemon:
https://docs.netbird.io/how-to/troubleshooting-client#on-linux-with-systemd
Which I did such, which seems to have improved reliability so far, but I think the 4G connection is less constrained at this time of night.
sudo mkdir -p /etc/sysconfig
echo 'NB_ICE_KEEP_ALIVE_INTERVAL_SEC=1' | sudo tee -a /etc/sysconfig/netbird
echo 'NB_ICE_DISCONNECTED_TIMEOUT_SEC=10' | sudo tee -a /etc/sysconfig/netbird
sudo systemctl restart netbird


Add tests for override formatter formatException

Closes https://github.com/microsoft/picologging/issues/103

Codecov Report

Merging #140 (bbc1fe9) into main (783acf1) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #140   +/-   ##
=======================================
  Coverage   89.05%   89.05%
=======================================
  Files          11       11
  Lines        2119     2119
=======================================
  Hits         1887     1887
  Misses        232      232

Impacted Files
Coverage Δ

src/picologging/handlers.py
94.55% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Use ubuntu-22.04

The repositories for ubuntu-21.04 are not available anymore and return 404 errors. We with @briannaromasky tested it with ubuntu-22.04, and the installation and startup were successful.

Don't want it on the fork. Will close and reopen for the main repo.


Is it there a way to copy the files from Snapshot to local machine

Hi,
I am trying to copy the files from the running exe file created using pkg and i want few files from the snapshot of the pkg to be copied to local machine where the exe file is currently running.
Thanks.

Does fs.copyFile work?

I am using fs.copySync to copy the files and it does throws me the error
Error: ENOENT: no such file or directory, open 'C:/snapshot/app/dist/config.json'

Hi,
Any suggestions ion the same ?
I am using fs.copySync to copy the files and it does throws me the error
Error: ENOENT: no such file or directory, open 'C:/snapshot/app/dist/config.json'

Hi,
Could please provide me some guidance here ?
I tried both fs.copySync and fs.copyFile but neither one works.
Thanks

try to read from srcpath then write to destpath

I can confirm that @fujinxiang approach works. I bundled a binary file that my code needed later on, and it couldn't run from the snapshot file system, so it needed to be extracted to the "real" file system:
`let copy = fs.createReadStream(pathInSnapshotFileSystem).pipe(fs.createWriteStream(target), { end: true });
copy.on('close', () => {
//here the file writing is done and the file is avalible
}
`

Thanks @carlhaman  for the confirmation, I will surely give it a try and update the issue here.

Is there a way to do this for entire directories as well?

@carlhaman method works, using ncp works out of the box for coping folders @FinlayDaG33k, probably using the mechanism under the hood.

@jonathanasquier Yea, in the end I just used a glob and a for loop to extract all files (I want to minimize the amount of extra dependencies):
// Get a list of all config files
const files: string[] = await globResources('**/*');

// Extract each of the config files
for(let file of files) {
      // Build our destination path
      let dest = path.resolve(file).replace(Data.getResourceDir(), Data.getProjectPath());

      // Copy the file from our resources to the destination
      await copy(file, dest);
    }


UNRESOLVED DEPENDENCIES

When I run sbt -DsocksProxyHost=<IP_ADDRESS> -DsocksProxyPort=1080 run, I got this:
[info] Loading global plugins from /Users/lingchax/.sbt/0.13/plugins
[info] Loading project definition from /Volumes/work/develop/scala/webapp/project
[info] Set current project to webapp (in build file:/Volumes/work/develop/scala/webapp/)
[info] Updating {file:/Volumes/work/develop/scala/webapp/}webapp...
[info] Resolving com.twitter.common#stats;0.0.74 ...
[warn] 	module not found: com.twitter.common#stats;0.0.74
[warn] ==== local: tried
[warn]   /Users/lingchax/.ivy2/local/com.twitter.common/stats/0.0.74/ivys/ivy.xml
[warn] ==== public: tried
[warn]   http://repo1.maven.org/maven2/com/twitter/common/stats/0.0.74/stats-0.0.74.pom
[warn] ==== Twitter: tried
[warn]   http://maven.twttr.com/com/twitter/common/stats/0.0.74/stats-0.0.74.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] downloading http://maven.twttr.com/com/twitter/common/metrics/0.0.9/metrics-0.0.9.jar ...
[warn] 	[FAILED     ] com.twitter.common#metrics;0.0.9!metrics.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/metrics/0.0.9/metrics-0.0.9.jar did not indicate a success. See log for more detail. (1329ms)
[warn] 	[FAILED     ] com.twitter.common#metrics;0.0.9!metrics.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/metrics/0.0.9/metrics-0.0.9.jar did not indicate a success. See log for more detail. (1329ms)
[warn] ==== Twitter: tried
[warn]   http://maven.twttr.com/com/twitter/common/metrics/0.0.9/metrics-0.0.9.jar
[info] downloading http://maven.twttr.com/com/twitter/common/base/0.0.67/base-0.0.67.jar ...
[warn] 	[FAILED     ] com.twitter.common#base;0.0.67!base.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/base/0.0.67/base-0.0.67.jar did not indicate a success. See log for more detail. (1340ms)
[warn] 	[FAILED     ] com.twitter.common#base;0.0.67!base.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/base/0.0.67/base-0.0.67.jar did not indicate a success. See log for more detail. (1340ms)
[warn] ==== Twitter: tried
[warn]   http://maven.twttr.com/com/twitter/common/base/0.0.67/base-0.0.67.jar
[info] downloading http://maven.twttr.com/com/twitter/common/quantity/0.0.52/quantity-0.0.52.jar ...
[warn] 	[FAILED     ] com.twitter.common#quantity;0.0.52!quantity.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/quantity/0.0.52/quantity-0.0.52.jar did not indicate a success. See log for more detail. (1027ms)
[warn] 	[FAILED     ] com.twitter.common#quantity;0.0.52!quantity.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/quantity/0.0.52/quantity-0.0.52.jar did not indicate a success. See log for more detail. (1027ms)
[warn] ==== Twitter: tried
[warn]   http://maven.twttr.com/com/twitter/common/quantity/0.0.52/quantity-0.0.52.jar
[info] downloading http://maven.twttr.com/com/twitter/common/stats-util/0.0.30/stats-util-0.0.30.jar ...
[warn] 	[FAILED     ] com.twitter.common#stats-util;0.0.30!stats-util.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/stats-util/0.0.30/stats-util-0.0.30.jar did not indicate a success. See log for more detail. (1427ms)
[warn] 	[FAILED     ] com.twitter.common#stats-util;0.0.30!stats-util.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/stats-util/0.0.30/stats-util-0.0.30.jar did not indicate a success. See log for more detail. (1427ms)
[warn] ==== Twitter: tried
[warn]   http://maven.twttr.com/com/twitter/common/stats-util/0.0.30/stats-util-0.0.30.jar
[info] downloading http://maven.twttr.com/com/twitter/common/util-executor-service-shutdown/0.0.36/util-executor-service-shutdown-0.0.36.jar ...
[warn] 	[FAILED     ] com.twitter.common#util-executor-service-shutdown;0.0.36!util-executor-service-shutdown.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/util-executor-service-shutdown/0.0.36/util-executor-service-shutdown-0.0.36.jar did not indicate a success. See log for more detail. (1229ms)
[warn] 	[FAILED     ] com.twitter.common#util-executor-service-shutdown;0.0.36!util-executor-service-shutdown.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/util-executor-service-shutdown/0.0.36/util-executor-service-shutdown-0.0.36.jar did not indicate a success. See log for more detail. (1229ms)
[warn] ==== Twitter: tried
[warn]   http://maven.twttr.com/com/twitter/common/util-executor-service-shutdown/0.0.36/util-executor-service-shutdown-0.0.36.jar
[info] downloading http://repo1.maven.org/maven2/com/twitter/jsr166e/1.0.0/jsr166e-1.0.0.jar ...
[info] 	[SUCCESSFUL ] com.twitter#jsr166e;1.0.0!jsr166e.jar (5049ms)
[info] downloading http://maven.twttr.com/com/twitter/common/util-system-mocks/0.0.50/util-system-mocks-0.0.50.jar ...
[warn] 	[FAILED     ] com.twitter.common#util-system-mocks;0.0.50!util-system-mocks.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/util-system-mocks/0.0.50/util-system-mocks-0.0.50.jar did not indicate a success. See log for more detail. (953ms)
[warn] 	[FAILED     ] com.twitter.common#util-system-mocks;0.0.50!util-system-mocks.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/util-system-mocks/0.0.50/util-system-mocks-0.0.50.jar did not indicate a success. See log for more detail. (953ms)
[warn] ==== Twitter: tried
[warn]   http://maven.twttr.com/com/twitter/common/util-system-mocks/0.0.50/util-system-mocks-0.0.50.jar
[info] downloading http://maven.twttr.com/com/twitter/common/collections/0.0.55/collections-0.0.55.jar ...
[warn] 	[FAILED     ] com.twitter.common#collections;0.0.55!collections.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/collections/0.0.55/collections-0.0.55.jar did not indicate a success. See log for more detail. (1367ms)
[warn] 	[FAILED     ] com.twitter.common#collections;0.0.55!collections.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/collections/0.0.55/collections-0.0.55.jar did not indicate a success. See log for more detail. (1367ms)
[warn] ==== Twitter: tried
[warn]   http://maven.twttr.com/com/twitter/common/collections/0.0.55/collections-0.0.55.jar
[info] downloading http://maven.twttr.com/com/twitter/common/util/0.0.75/util-0.0.75.jar ...
[warn] 	[FAILED     ] com.twitter.common#util;0.0.75!util.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/util/0.0.75/util-0.0.75.jar did not indicate a success. See log for more detail. (974ms)
[warn] 	[FAILED     ] com.twitter.common#util;0.0.75!util.jar: The HTTP response code for http://maven.twttr.com/com/twitter/common/util/0.0.75/util-0.0.75.jar did not indicate a success. See log for more detail. (974ms)
[warn] ==== Twitter: tried
[warn]   http://maven.twttr.com/com/twitter/common/util/0.0.75/util-0.0.75.jar
[warn] 	[NOT FOUND  ] com.twitter.common#jdk-logging;0.0.32!jdk-logging.jar (0ms)
[warn] ==== Twitter: tried
[warn]   http://maven.twttr.com/com/twitter/common/jdk-logging/0.0.32/jdk-logging-0.0.32.jar
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: com.twitter.common#stats;0.0.74: not found
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::              FAILED DOWNLOADS            ::
[warn] 	:: ^ see resolution messages for details  ^ ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: com.twitter.common#metrics;0.0.9!metrics.jar
[warn] 	:: com.twitter.common#base;0.0.67!base.jar
[warn] 	:: com.twitter.common#quantity;0.0.52!quantity.jar
[warn] 	:: com.twitter.common#collections;0.0.55!collections.jar
[warn] 	:: com.twitter.common#util-system-mocks;0.0.50!util-system-mocks.jar
[warn] 	:: com.twitter.common#stats-util;0.0.30!stats-util.jar
[warn] 	:: com.twitter.common#util;0.0.75!util.jar
[warn] 	:: com.twitter.common#util-executor-service-shutdown;0.0.36!util-executor-service-shutdown.jar
[warn] 	:: com.twitter.common#jdk-logging;0.0.32!jdk-logging.jar
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
sbt.ResolveException: unresolved dependency: com.twitter.common#stats;0.0.74: not found
download failed: com.twitter.common#metrics;0.0.9!metrics.jar
download failed: com.twitter.common#base;0.0.67!base.jar
download failed: com.twitter.common#quantity;0.0.52!quantity.jar
download failed: com.twitter.common#collections;0.0.55!collections.jar
download failed: com.twitter.common#util-system-mocks;0.0.50!util-system-mocks.jar
download failed: com.twitter.common#stats-util;0.0.30!stats-util.jar
download failed: com.twitter.common#util;0.0.75!util.jar
download failed: com.twitter.common#util-executor-service-shutdown;0.0.36!util-executor-service-shutdown.jar
download failed: com.twitter.common#jdk-logging;0.0.32!jdk-logging.jar
	at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:213)
	at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
	at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
	at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:116)
	at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:104)
	at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:51)
	at sbt.IvySbt$$anon$3.call(Ivy.scala:60)
	at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
	at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
	at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
	at xsbt.boot.Using$.withResource(Using.scala:10)
	at xsbt.boot.Using$.apply(Using.scala:9)
	at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
	at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
	at xsbt.boot.Locks$.apply0(Locks.scala:31)
	at xsbt.boot.Locks$.apply(Locks.scala:28)
	at sbt.IvySbt.withDefaultLogger(Ivy.scala:60)
	at sbt.IvySbt.withIvy(Ivy.scala:101)
	at sbt.IvySbt.withIvy(Ivy.scala:97)
	at sbt.IvySbt$Module.withModule(Ivy.scala:116)
	at sbt.IvyActions$.update(IvyActions.scala:121)
	at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1144)
	at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1142)
	at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1165)
	at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$73.apply(Defaults.scala:1163)
	at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
	at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1167)
	at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1162)
	at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
	at sbt.Classpaths$.cachedUpdate(Defaults.scala:1170)
	at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1135)
	at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1113)
	at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
	at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:42)
	at sbt.std.Transform$$anon$4.work(System.scala:64)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:237)
	at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
	at sbt.Execute.work(Execute.scala:244)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
	at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:237)
	at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
	at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
[error] (*:update) sbt.ResolveException: unresolved dependency: com.twitter.common#stats;0.0.74: not found
[error] download failed: com.twitter.common#metrics;0.0.9!metrics.jar
[error] download failed: com.twitter.common#base;0.0.67!base.jar
[error] download failed: com.twitter.common#quantity;0.0.52!quantity.jar
[error] download failed: com.twitter.common#collections;0.0.55!collections.jar
[error] download failed: com.twitter.common#util-system-mocks;0.0.50!util-system-mocks.jar
[error] download failed: com.twitter.common#stats-util;0.0.30!stats-util.jar
[error] download failed: com.twitter.common#util;0.0.75!util.jar
[error] download failed: com.twitter.common#util-executor-service-shutdown;0.0.36!util-executor-service-shutdown.jar
[error] download failed: com.twitter.common#jdk-logging;0.0.32!jdk-logging.jar
[error] Total time: 68 s, completed 2015-4-19 11:57:09

But I can download them from my web browser, my build.sbt:
name := "webapp"

version := "0.0.1-SNAPSHOT"

scalaVersion := "2.10.3"

libraryDependencies ++= Seq(
  "com.twitter" %% "finatra" % "1.5.3"
)

resolvers +=
  "Twitter" at "http://maven.twttr.com"

And my sbt version:
sbt launcher version 0.13.8

It works now, close it.


Remove old command from update sample, plus a minor consistency tweak

LGTM.
Merging doc change with oncall's approval.


UP-4436 Groovy Script to convert XML PAGS to entity PAGS files

https://issues.jasig.org/browse/UP-4436
Groovy can probably be improved or made more idiomatic.  I'd love to hear feedback on improvements.
Can invoke script 2 ways:

using Groovy (requires installing Groovy and having it on path)

groovy utilities/scripts/xmlPagsToEntityPags.groovy uportal-war/src/main/resources/properties/groups/PAGSGroupStoreConfig.xml uportal-war/src/main/data/my_entities/pags-group
Output example at https://issues.jasig.org/secure/attachment/14737/groovyExecutionResults.txt

via ant (ugh - not as nice, made script less clear, but done on request)

ant -Dmaven.test.skip=true -Ddir=uportal-war/src/main/data/quickstart_entities/pags-group pagsXmlToEntity
Output similar to groovy execution but wrapped in ant execution.
Note

As mentioned at end of output example, after running the groovy script you still have to either initdb, or import the entity files and run a generated SQL script to update existing PAGS group references in the DB.
I debated creating an ant task to allow running the generated SQL script, but I think low value for the effort.

Coverage increased (+0.02%) to 22.77% when pulling 8abe1c798e98f487a072b57915c2294e99e273c2 on UP-4436ScriptConvertXmlPags into 815dba5af8ff0fa8379bf17c37d35c775fb620f6 on master.

Coverage increased (+0.02%) to 22.77% when pulling a6338a002dac0bdc419af7e762230c57b614288c on UP-4436ScriptConvertXmlPags into 815dba5af8ff0fa8379bf17c37d35c775fb620f6 on master.

Will invoke lazy consensus on Wed 4/15.

Coverage decreased (-0.01%) to 22.76% when pulling fee75648ca1fc82d22c81db5876fa7bfdc56a96f on UP-4436ScriptConvertXmlPags into b58926304c32d4f74324b31dd841785f2c38723e on master.


Spelling fix

https://github.com/VirtualFlyBrain/geppetto-vfb/issues/61

Thanks. Sorry I didn't find the right place to submit this. Is this where I should submit v2 bugs in future?

Nope you had the right place


Remove TODOs from comments

As discussed in #446, some kapp-controller structs have TODO comments that are then used as descriptions in schemas:

https://github.com/vmware-tanzu/carvel-kapp-controller/blob/babe7624f2c72c54e5f83860df410470f2b1bd71/config/crds.yml#L227
https://github.com/vmware-tanzu/carvel-kapp-controller/blob/babe7624f2c72c54e5f83860df410470f2b1bd71/config/crds.yml#L350
https://github.com/vmware-tanzu/carvel-kapp-controller/blob/babe7624f2c72c54e5f83860df410470f2b1bd71/config/crds.yml#L360

The comment about supporting Docker config format was implemented in v0.19.0 of kapp-controller: https://carvel.dev/kapp-controller/docs/latest/app-overview/#image-and-imgpkgbundle-authentication. So this comment can be removed.
There is another question about removed the jsonnet/kustomize properties since they are not implemented. It may be confusing for users to see this information in the schema. We have not even decided if we will support these options yet, so we should consider removing them until they are implemented.
To update the schema descriptions above, the comments above the properties in the go structs need to be changed and then the build/generator scripts need to be run. An example of doing this would be as follows:

Update the comments:

https://github.com/vmware-tanzu/carvel-kapp-controller/blob/babe7624f2c72c54e5f83860df410470f2b1bd71/pkg/apis/kappctrl/v1alpha1/types_fetch.go#L46

Run ./hack/build.sh to regenerate the crds.yml file

Run the generator scripts: ./hack/gen.sh and ./hack/gen-apiserver.sh. NOTE: protobuf must be installed to successfully run these scripts. See install directions for more information: https://grpc.io/docs/protoc-installation/.

In speaking with @joe-kimmel-vmw, we think it would be a reasonable approach to remove jsonnet/kustomize options as part of this work.

If anyone is not contributing to this issue then I would like to contribute.

/assign

@danielhelfand I think the branch where we have to commit changes is not available


Add MODULATE builtin to canvas item shaders (3.2)

This adds a new builtin for canvas item shaders,  in color MODULATE. Its value is the final modulate color that would be applied to the item.
Usually, the modulate color is applied after the user fragment shader code has been run. This allows you to use the modulate yourself however you want. Therefore, if you use MODULATE, the default behavior won't happen so you are responsible of multiplying at the end (that would be replicating the default behavior, so quite pointless) or doing something fancier with it.
My use case: I have an almost finished game in Godot 2.1 that uses canvas item shaders to create rich looks by using normal maps and applying the modulate color in 3D-like ways, so to speak. After porting it to 3.2, it wasn't possible anymore to do that. This PR re-adds that ability.
There will be a separate PR for master, in case the shading model for 4.0 makes it necessary.

I've spoken to RandomShaper and it requires a slight tweak of the batching to work with it (because we bake the final modulate and vertex colors together in some batches), but it looks quite straightforward to get it working, we can just detect the MODULATE keyword and disable translating to colored verts for that item. :+1:
Just as soon as I can remember where @clayjohn showed me how to detect keywords in the shaders lol :grin: ...

I've updated the PR with a change: MODULATE can be read from the vertex shader as well. That's useful if you want to use it to generate the COLOR somehow from it. That way you don't need to do it per every fragment. Or if you just want to pre-modulate the color, just just have to do it per vertex.
The same semantics of inhibiting the default post-modulate behavior still hold.
@lawnjelly, I guess this doesn't change anything for our plans. :)

Thanks!
Don't forget to do the matching PR for 4.0 :)

It seems a PR for 4.0 is not needed now. How canvas item shaders will work there still has some open questions and maybe in the end we don't need the distinction between COLOR and MODULATE anymore in 4.0 and a simpler model can be used there. MODULATE was needed in 3.2 to more control without breaking compatibility.
What I'll do is documenting MODULATE in the docs repo.

I was thinking about this last night, and there may be some other effects with color in a custom shader, I need to examine this in detail. It may be worth either swapping to / or adding a vertex format including color and final modulate separately for this purpose. It's not a big deal, it's just the vertex format will be larger, so there will be some small drop in efficiency doing that. The shader itself will be doing the same work.


tests: add operator deployment on ephemeral cluster

Enable further integration testing by deploying the operator at the beginning of the testing pipeline.
Copy-pasting the sample from here:
https://github.com/konflux-ci/pipeline-samples/blob/main/pipelines/integration/deploy-operator.yaml

This doesn't work at the moment because our deployment doesn't have v4.16 test clusters.


ability to enable cni portmap plugin

Hi,
Can you add the possibility of enable portmap plugin on all host  on kubernetes
This will solve the hostport problems
Regards :)

Something like this

Yeah but it's dont work :/

Any more details?

I have compiled it for my setup, add the binary on the /opt/cni/bin
Add
{
    "cniVersion": "0.3.1",
    "name": "mynet",
    "plugins": [
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

And restart kubelet and cni pod
After that the cni pod restart with no problems but ... all others pod stay stuck on container creating :/

You can't run with just a portmap plugin; you need a network too.
By analogy to the example I pointed to, for Weave Net it would look something like:
{
    "cniVersion": "0.5.1",
    "name": "mynet",
    "plugins": [
        {
            "type": "weave",
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

caution: I haven't tried this

this conflist ovewrite other conf file ? it's not realy clear for me :)
I'll give a try this night

Yes, Kubernetes will only read one CNI config file, the first one it comes to (probably in alphanumeric order)

@bboreham I tried the following, but this sadly did not work:

place https://github.com/projectcalico/cni-plugin/releases/download/v1.9.0/portmap in /opt/cni/bin/portmap
configure /etc/cni/net.d/10-weave.conf like this:

{
    "cniVersion": "0.5.1",
    "name": "weave",
    "plugins": [
        {
            "type": "weave",
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {
                "portMappings": true
            },
            "snat": true
        }
    ]
}

restart the VM

result is that kube-dns no longer starts - when I revert the /etc/cni/net.d/10-weave.conf to
{
            "type": "weave",
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
}

kube-dns works just like before.
The kube-dns error is quite generic: Pod sandbox changed, it will be killed and re-created.

Nop :) it's not work
{     "cniVersion": "0.3.1",     "name": "mynet",     "plugins": [ 	{     	    "name": "weave",             "type": "weave-net",     	    "hairpinMode": true 	},         {             "type": "portmap",             "capabilities": {"portMappings": true},             "snat": true         }     ] }

\o/ working :)
{
    "cniVersion": "0.3.0",
    "name": "mynet",
    "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}

@ekozan can you -vvv your steps a little? download portmap and save your content to /etc/cni/net.d/10-mynet.conflist?

@bboreham Will writing this new CNI spec happen automatically in the future?

@luxas that would be best, yes.  Need to think about how to detect if the user has the portmap plugin installed (or provide a way for Weave Net to install it).
PRs welcome :slightly_smiling_face:

Maybe just do something like hit /version of the API Server?

full step
$ git clone https://github.com/containernetworking/plugins
$ cd plugins
$ ./build.sh
$ sudo cp bin/* /opt/cni/bin/
$ sudo sh -c 'cat >/etc/cni/net.d/10-mynet.conflist <<-EOF
{
    "cniVersion": "0.3.0",
    "name": "mynet",
      "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}
EOF'
$ rm /etc/cni/net.d/10-weave.conf

@ekozan how did you verify the hostPort is actually working?

Maybe just do something like hit /version of the API Server?

@luxas as I see it two things are key:

does the machine have the portmap CNI plugin installed on disk?
does the local kubelet have the code to pass details of hostPort mappings?

The API server version doesn't indicate either of these things.  I don't think there is a published API to get the kubelet version.
I think at present we would need a manual switch for the operator to turn on if they are sure the prerequisites are met.

FTR, I got this working without requiring git or go (but adding a dependency on wget 😏) on the target system with a small tweak to the instructions in https://github.com/weaveworks/weave/issues/3016#issuecomment-313174867 to use the released cni-plugins v0.6.0-rc1 package:
$ cd /tmp && mkdir cni-plugins && cd cni-plugins && \
    wget https://github.com/containernetworking/plugins/releases/download/v0.6.0-rc1/cni-plugins-amd64-v0.6.0-rc1.tgz && \
    tar zxfv cni-plugins-amd64-v0.6.0-rc1.tgz
$ sudo cp /tmp/cni-plugins/portmap /opt/cni/bin/
$ sudo sh -c 'cat >/etc/cni/net.d/10-mynet.conflist <<-EOF
{
    "cniVersion": "0.3.0",
    "name": "mynet",
      "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {"portMappings": true},
            "snat": true
        }
    ]
}
EOF'
$ rm /etc/cni/net.d/10-weave.conf
$ kubectl delete po --all --namespace=kube-system

@lukemarsden why you need to restart all pods in kube-system?
Maybe just kubectl delete po -n kube-system -l name=weave-net will do the trick?

@Bregor I was just copying https://github.com/weaveworks/weave/issues/3016#issuecomment-313174867 😄 I'm not sure what you need to delete exactly to get kubelet to restart CNI, would be good if you could run some tests to find out?

Restarting pods has no effect on CNI.
I might have expected you need to restart kubelet, but if not, that's cool.

@lukemarsden, @ekozan  which kubernetes version did you use?
I use kubernetes-1.6.8 with no success at all.
I tried following CNI + Plugins combinations:

cni-0.5.1 + plugins-0.6.0-rc1
cni-0.5.2 + plugins-0.6.0-rc1
cni-0.5.1 + plugins-0.6.0
cni-0.5.2 + plugins-0.6.0
cni-0.6.0 + plugins-0.6.0-rc1
cni-0.6.0 + plugins-0.6.0-rc1
cni-0.6.0 + plugins-0.6.0
cni-0.6.0 + plugins-0.6.0

Also tried all combinations above with weave-1.9.8 and 2.0.1.
No success at all.
Events:
nginx     2017-08-21 15:10:15 +0300 MSK   2017-08-21 15:00:48 +0300 MSK   45        nginx-3341093162-f3kkq   Pod                 Warning   FailedSync   kubelet, <IP_ADDRESS>   Error syncing pod, skipping failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"

Kubelet logs:
Aug 21 15:11:47 kube01 kubelet[24531]: W0821 15:11:47.377124   24531 docker_sandbox.go:263] NetworkPlugin cni failed on the status hook for pod "nginx-3341093162-f3kkq_nginx": Cannot find the network namespace, skipping pod network status for container {"docker" "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933"}
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.379565   24531 cni.go:278] Error deleting network: no plugin name provided
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.382972   24531 remote_runtime.go:109] StopPodSandbox "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933" from runtime service failed: rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod "nginx-3341093162-f3kkq_nginx" network: no plugin name provided
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383066   24531 kuberuntime_manager.go:784] Failed to stop sandbox {"docker" "1369eaf944618152e832a70ae221e808f888700082f1f712d05bef871b2e0933"}
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383285   24531 kuberuntime_manager.go:573] killPodWithSyncResult failed: failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"
Aug 21 15:11:47 kube01 kubelet[24531]: E0821 15:11:47.383340   24531 pod_workers.go:182] Error syncing pod 3b3bc4a9-8664-11e7-a06b-264916ab5742 ("nginx-3341093162-f3kkq_nginx(3b3bc4a9-8664-11e7-a06b-264916ab5742)"), skipping: failed to "KillPodSandbox" for "3b3bc4a9-8664-11e7-a06b-264916ab5742" with KillPodSandboxError: "rpc error: code = 2 desc = NetworkPlugin cni failed to teardown pod \"nginx-3341093162-f3kkq_nginx\" network: no plugin name provided"

Contents of /etc/cni/net.d/10-weave.conf:
{
    "cniVersion": "0.3.0",
    "name": "weave",
    "plugins": [
        {
            "name": "weave",
            "type": "weave-net",
            "hairpinMode": true
        },
        {
            "type": "portmap",
            "capabilities": {
                "portMappings": true
            },
            "snat": true
        }
    ]
}

What am I doing wrong?..

Ah, nevermind.
Just realized, that ability to use NetworkConfigList became available only from 1.7.0-alpha3 :(

@bboreham when I run the kube add-on, it automatically generates /etc/cni/net.d/10-weave.conf. If I have 10-weave.conflist, CNI will pick up the first and miss my generated conflist.
Short of giving it a lower number (higher priority), is there any way to get it to not generate 10-weave.conf?
Conversely, if there were a way to tell the add-on to add the portmap - generate the conflistwith portmap instead of the single conf - perhaps via env var, that would be valuable.
You mentioned earlier that it has an issue detecting; this would give at least a partial resolution.

Hi,
@deitch your right and kubelet pick-up the first file and only one.
In fact, Kubelet CNI implementation supports Json, Conf and ConfList (https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/cni/cni.go#L99). You cannot specifiy multiple plugins in a conf file but you can in a conflist, I think you can do both in Json.
@deitch did the lower priority for the config file work ?

@ArchiFleKs yes. I used the following config manually.
        {
            "cniVersion": "0.3.0",
            "name": "mynet",
              "plugins": [
                {
                    "name": "weave",
                    "type": "weave-net",
                    "hairpinMode": true
                },
                {
                    "type": "portmap",
                    "capabilities": {"portMappings": true},
                    "snat": true
                }
            ]
        }

I saved it as /etc/cni/net.d/00-weave.conflist. This way, even though weave automatically adds 10-weave.conf, it takes precedence.
I would have preferred to have it auto-generate like weave normally does, but for the future, I guess.

Nice @deitch so I wonder - what is blocking weave from adding this to their default configuration?

what is blocking weave from adding this to their default configuration?

I do not know. @bboreham ?

See https://github.com/weaveworks/weave/issues/3016#issuecomment-313644623

Thanks.
To summarize: As weave, we don't know if we can enable portmap, thus it should be left to the admin to enable it.
best is to make this enablement easy.

This is actually nasty, I wonder if this can be solved in the realm of Kube. After all portmap is a common thing whenever NodePort is used.

@fabiand actually it is needed only if you are using HostPort directly on a container, without Services nor NodePort

Hm. That's a good hint, and it then sounds as if this is not the issue I am seeing.
I'm using kubelet with CNI and weave, and using NodePort - or the definition used here kubevirt/kubevirt#542 - then the user defined ports will not be opened (only the randomly assigned ones).
The very same definition works with docker networking.

@fabiand I replied to you here also https://github.com/kubevirt/kubevirt/issues/542

@bboreham

does the machine have the portmap CNI plugin installed on disk?

maybe we could provide the whole CNI package and plugins as packaged in https://github.com/containernetworking/cni/releases at least the will take care of this.

does the local kubelet have the code to pass details of hostPort mappings?

For this one I don't know which kubernetes version introduced HostPort mapping but it is not a very new feature i guess.

maybe we could provide

who is 'we' ?

which kubernetes version introduced HostPort mapping

...via CNI was 1.7; 30th June 2017

who is 'we' ?

I meant weave, the CNI plugin are symlink when running weave setup right ? maybe the portmap plugin could also be shipped / symlinked

...via CNI was 1.7; 30th June 2017

Oh OK. Is it really an issue ? because even before portmap plugin support, I believe it failed silently if you tried to specified an hostPort into a manifest, it just had no effect.

Yes, we could ship a copy of the portmap plugin.  Not completely sold on the idea, though, as it complicates our release process.
The support in kubelet I am referring to is for new CNI config file syntax; all CNI calls will fail if you supply a config that lists plugins and kubelet doesn't understand that.

The portmap plugin will be available in most places in v1.9, as k8s v1.9 ships with CNI v0.6.0

Hello all,
In my Kubernetes cluster, SlaveIP:80 can be shown by netstat and accessed by browser.
but MasterIP:80 can't be opened.
I think portmap plugin have already enabled by slave side.
But, why doesn't on master side?
My environment condition:
Ubuntu: 16.04 LTS
Kubernetes: 1.9.3
Docker-ce: 17.03.2~ce-0~ubuntu-xenial
Weave Net: 2.2.1
I follow the official Kubernetes ingress page.
https://github.com/kubernetes/ingress-nginx/tree/master/deploy
In default "with-rbac.yaml", I appended "hostnetwork=true" and "hostport=80" within
https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml
/////////////////////////////////////////////////////////////////////////////////////
...
...
spec:
serviceAccountName: nginx-ingress-serviceaccount
hostNetwork: true
containers:
...
...
ports:
- name: http
hostPort: 80
containerPort: 80
...
...
/////////////////////////////////////////////////////////////////////////////////////
Is there something parameter or step that need to enable master portmap in Kubernetes cluster?
Thanks very much.
Hong

@yanhongwang please open a new issue rather than muddling an existing conversation.
However hostNetwork: true means that the pod will use the host network, rather than Weave Net (or any other CNI plugin), so this is not a Weave Net issue at all.

@bboreham
Thanks.
Sorry for muddling.
Hong

@bboreham I see you tagged "help wanted" on here.  Do you have any thoughts how you would solve this?  I'm using kops to spin up our cluster, but we need portmap.  It seems the way Calico solved this with kops was to package the portmap binary with their docker image and then copy it onto the host through the /opt/cni/bin mount.
I'm wondering if this needs to be a combined fix between kops and weave.  We could have kops install the portmap binary (it already has the full CNI tarball available since it loads other CNI binaries), then update weave to install the proper /etc/cni/net.d/10-weave.conflist.  The current conf doesn't include portmapper obviously.

Do you have any thoughts how you would solve this?

Not really. Probably the whole thing needs taking apart and doing again.
The current implementation is a bit minamalist and simplest-thing-that-works. If I remember correctly, at the time nobody was installing add-ons purely by running a daemonset; the state of the art was to modify the Kubernetes in-tree saltstack.
The inside of a DameonSet is a terrible place to make decisions about what is installed and where. To that extent, places like kops and kubeadm are much better.

The inside of a DameonSet is a terrible place to make decisions about what is installed and where. To that extent, places like kops and kubeadm are much better.

The problem with places outside is that they are, well, outside. There is something great about it all just being in a single yml you can kubectl apply, and keeping it clean.
FWIW, the last installation I did was a mix of the cloud-init downloading the standard CNI distro and installing it, and a modified yml for the DaemonSet with an init container that installs an overriding config file:
          initContainers:
            # set up our correct conflist in /etc/cni/net.d
            - name: init-conflist
              image: busybox
              command: ['sh','-c','mkdir -p /host/etc/cni/net.d/ && cp /conf/00-weave.conflist /host/etc/cni/net.d/']
              volumeMounts:
              - name: cni-conf
                mountPath: /host/etc
              - name: config-volume
                mountPath: /conf

Where the 00-weave.conflist looks like:
        {
            "cniVersion": "0.3.0",
            "name": "mynet",
              "plugins": [
                {
                    "name": "weave",
                    "type": "weave-net",
                    "hairpinMode": true
                },
                {
                    "type": "portmap",
                    "capabilities": {"portMappings": true},
                    "snat": true
                }
            ]
        }

I rely on the ordering of numbers 00 to get it pulled in. It is somewhat brittle, but at least everything "weave-ish" is in that one place, and everything "standard" is in my cloud-init.

My guess is we might have to change this check/add portmap to the list
https://github.com/kubernetes/kops/blob/master/nodeup/pkg/model/network.go#L46
to place portmap in the CNI bin directory.

@murali-reddy Yes, it is a one-liner change in that file but we also need to update the weave install template for kops (https://github.com/kubernetes/kops/blob/master/upup/models/cloudup/resources/addons/networking.weave/k8s-1.6.yaml.template).  However, it would be nice for weave to support generating the conflist rather than the conf in /etc/cni/net.d/.  I guess for now, we could use the hack suggested by @deitch.
@bboreham As an initial rework to weave, could we do something similar to flannel and use a configmap to generate the /etc/cni/net.d/ file?  Currently, the CNI conf is hardcoded in the weave script.

configmap fine as long as it's optional.
It was so long ago when I did it, optional configmaps didn't exist in Kubernetes...

Yeah the optional configmaps are nice.   So maybe implement the logic "if cni configmap exists, use it... otherwise, default to the hardcoded 10-weave.conf.

Hello,
Do you know this version will be publish to DockerHub.
Can't wait to test it !

Do you know this version will be publish to DockerHub.

@fjudith support for portmap will be part of 2.5 release
I just need to add a integration test to finish https://github.com/weaveworks/weave/pull/3356
Let me know if you would like to test out the changes in https://github.com/weaveworks/weave/pull/3356, i will share the docker image.

I have a lab for this puposal running Containerd (1.2.0-Beta.2) and Docker (17.03.2) along with Kubernetes 1.11.2.
I'm open for further testings.

@fjudith Great! Please use the image muralireddy/weave-kube:portmap. Please note that once you use this image its going to remove /etc/cni/net.d/10-weave.conf on all nodes and replace with /etc/cni/net.d/10-weave.conflist. So if you have revert back please delete .conflist file.

That is exactly why I'd go with an at least in part backwards compatible version if you're going to remove the 10-weave.conf file. Use 11-weave.conflist and then on downgrade the old Weave image will still work because it'll create the 10-weave.conf again and as it takes precedence, it'll not crash. Upgrading to your beta image will drop it again and use the conflist again. Might be worth the effort.

@murali-reddy .
Once hostport pods applied it breaks systemd-resolved, leading to ErrImagePull for any further pods scheduled on the node, as well for general DNS resolution.
It seems this issue specifically targets  Ubuntu Bionic (18.04).
ref: https://github.com/kubernetes/kubernetes/issues/66067
The problem has been observed on Docker, Containerd and Cri-o.
systemctl status systemd-resolved

● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2018-09-02 11:48:51 CEST; 35min ago
     Docs: man:systemd-resolved.service(8)
           https://www.freedesktop.org/wiki/Software/systemd/resolved
           https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 798 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 2299)
   CGroup: /system.slice/systemd-resolved.service
           └─798 /lib/systemd/systemd-resolved

Sep 02 12:24:18 proxy01 systemd-resolved[798]: Got packet on unexpected IP range, refusing.
Sep 02 12:24:20 proxy01 systemd-resolved[798]: Got packet on unexpected IP range, refusing.
 There is nothing in particular need to be done. When upgrade is done to new release 10-weave.conf is deleted and 10-weave.conflist will be used further. If Weave need to be downgraded to old release, 10-weave.conf will be created. In case there are multiple conf file first one in lexicographical order is chosen. So  10-weave.conf will be used.


Add triu and tril Tensor Method

Feature description
Add a triu and tril methods to return the upper/lower of a matrix (2-D tensor) or batch of matrices, while setting the rest to 0.
Feature motivation
Operation supported by other ML frameworks

Added tril as well

@antimora can you assign me here?

@antimora can you assign me here?

Done!

PR for this now submitted #1002

Closing as the PR is merged now.


Crash in RLMObservation.hpp

<_NSCallStackArray 0x1500eff40>(
0   ???                                 0x0000000105b7cc64 0x0 +<PHONE_NUMBER>,
1   My App                          0x00000001002cde64 main + 0,
2   Realm                               0x0000000100e5c8c8 _Z12RLMDidChangeRKNSt3__16vectorIN5realm14BindingContext13ObserverStateENS_9allocatorIS3_EEEERKNS0_IPvNS4_IS9_EEEE + 332,
3   Realm                               0x0000000100ec5cfc _ZN12_GLOBAL__N_121RLMNotificationHelper10did_changeERKNSt3__16vectorIN5realm14BindingContext13ObserverStateENS1_9allocatorIS5_EEEERKNS2_IPvNS6_ISB_EEEE + 48,
4   Realm                               0x0000000100ed3d04 _ZN5realm5_impl11transaction7advanceERNS_11SharedGroupEPNS_14BindingContextENS2_9VersionIDE + 264,
5   Realm                               0x0000000100e32690 _ZN5realm5_impl16RealmCoordinator16advance_to_readyERNS_5RealmE + 200,
6   Realm                               0x0000000100ee08e0 _ZZN5realm5_impl17WeakRealmNotifierC1ERKNSt3__110shared_ptrINS_5RealmEEEbEN3$_08__invokeEPv + 48,
7   CoreFoundation                      0x0000000180a3d09c <redacted> + 24,
8   CoreFoundation                      0x0000000180a3cb30 <redacted> + 540,
9   CoreFoundation                      0x0000000180a3a830 <redacted> + 724,
10  CoreFoundation                      0x0000000180964c50 CFRunLoopRunSpecific + 384,
11  GraphicsServices                    0x000000018224c088 GSEventRunModal + 180,
12  UIKit                               0x0000000185c4e088 UIApplicationMain + 204,
13  My App                          0x00000001002cdef4 main + 144,
14  libdyld.dylib                       0x00000001805028b8 <redacted> + 4
)

Version of Realm and Tooling
Realm version: v1.0.0
Xcode version: 7.3.1
iOS/OSX version: iOS 9.2
Dependency manager + version: Carthage 0.16.2

I was able to avoid this crash by using a let constant instead of a var for the realm instance I'm observing via KVO. FWIW I'm using the PMKVObserver library.

If this is a reproducible crash, could you please share what steps we can take to trigger this on our end?
Thanks!

@kharmabum do you have any updates on this? Thanks!

@stel working on putting together a project to repro this and https://github.com/realm/realm-cocoa/issues/3412 Will try to get something together by the end of the work.

@kharmabum thank you!

Is there any update on this issue? I'm experiencing the same symptoms in Obj-C when using KVO to monitor changes in my data model. My use case is for monitoring download progress of an item where track progress complete on my data model. This is needed because downloads happen in the background and I provide views that allow the user to monitor download progress, cancel downloads, etc. Any info or recommendations you have are appreciated.
Add observer to my model object

Observe Changes - Removing observer when progress reaches 100%

The error I receive every time I attempt to remove self as the observer

Thank you,
Travis

That specific stacktrace is the one which was fixed by https://github.com/realm/realm-cocoa/pull/3983, so it should work if you use master.

How stable is master? And when do you suspect this fix will make it in an official release (say 1.0.3)?

I've noticed I get this issue when I have multiple observers on a realm object. Using version 2.1.1.

@kharmabum since we've fixed a bug that causes this exact stack trace months ago, if you're still experiencing something similar you'll need to share a repro case to differentiate and for us to make progress on it.

Closing this issue. If this crash occurs to any users who might be reading this thread, please open a new issue.


WebTorrent 1.0 checklist

@feross from our conversation yesterday:

[x] Growth tracking: # of active users, month over month growth
[ ] UI polish. Improve the CSS (#802, #813, ...)

[ ] Clarify download status (see #715)
[ ] Clarify that torrents can be expanded / collapsed

[x] Cast menu

Cast menu should be modal instead of being a toggle menu

[ ] Technical debt cleanup

[x] Split up renderer/main.js
[ ] Eliminate uncaught exceptions as best we can (#804, ...)
State object refactor. The state object is not as simple and clean as it could be. We should think carefully and only change it once, since keeping backward compatibility with multiple config.json formats would be ugly.

[x] Telemetry

[x] Report uncaught exceptions
[x] Maybe report % of times the play button Just Works, % times out, % shows unsupported codec error

[ ] Performance

[x] Profile and see what's making WebTorrent sluggish
[ ] Reduce CPU use

[ ] Better codec support

[ ] "Play in VLC" should allow casting instead; the cast devices have AC3 support
[ ] Ideally, we should just support AC3. On Windows and Mac, we could use the system codec support

Also:

[ ] Bounds refactor. Save the latest non-player window bounds, and restore to those whenever existing the player. This should fix #648 as well as a bug I just discovered: play video > start casting > stop casting > hit ESC returns to the torrent list as expected, but does not restore the original window bounds.

[ ] Autoplay files within a torrent. Until we can do this we don't really support audiobooks.

[ ] Cast refactor. Simplify state (state.playing / state.devices etc)

[ ] When casting to a device fails, the corresponding cast menu item shouldn't be checked
[ ] New cast devices should show up without rebooting the app. Same for removing cast devices. Currently this doesn't work reliably for DLNA.
[ ] When you're eg. streaming to Chromecast from WTD, then you unplug the Chromecast, it should kick you back to the regular player and remove the corresponding cast menu item

Just shipped WebTorrent v0.11. https://github.com/feross/webtorrent-desktop/releases/tag/v0.11.0
Hoping we can do one more release (v0.12) in the next few days, and then hopefully we're ready for v1.0!

Hi guys,
Regarding:

[ ] Clarify download status (see #715), we could use a 'pause' or 'stop' button. I read over the issue and saw @dcposch comment about the pause button being confusing next to a Play button, but wanted to try it anyway. For me is more clarifying than the current status. Also, it helps the fact that the red play button never becomes a pause button (as it only opens the player). I leave some screenshots:

When torrent is already paused, it will show the current download/upload icons.
For

[ ] Clarify that torrents can be expanded / collapsed, I am trying with some arrows before the current buttons, I leave some screenshots:

What do you think? kind to open PR on this

I still think it's confusing having a big play button with the pause/stop one. Maybe a solution would be having them in different places instead of next to each other?

@iovis9 Agree, maybe having the torrenting button more aligned with the torrent metadata and leave the 'play' button alone would be more clarifying. This would need a bit more of design, though.
As for now, I have tried with the pause/reload button just on hover, so the torrenting status is still visible at first sight. You can have a look here https://github.com/bertofer/webtorrent-desktop

I was thinking something more on the lines of this (2-minute hack, just an idea).

Don't post those torrents 🙉 !
I like the idea, yes. I think it can be combined with having pause icon only on hover, so you still see if torrent is downloading/seeding, and as a meaning of what the button is about.

@iovis9 We don't support the use of WebTorrent for copyright infringement, and cannot help you.

@bertofer Thanks for the ideas. Let's move the torrent list designs to their own issue.

I think we're really close to 1.0. I just updated the checkboxes. Better codec support is the most important remaining issue, IMO.

and heavy CPU usage too, when i lunch webtorrent on MBP i start hearing fans inside (qbittorrent work quiet)


feature: add rpc client

This pr adds both the axon client which handles communication with axon and the ckb client which handles rpc requests to ckb_indexer.
The axon client is not fully finished. The endpoint to send the metadata to axon is left unimplemented so far.

Why not directly rely on emitter-lib, but choose to copy the code？


Missing alarm(3) in posix.unistd?

Hi there!
Before I go off implementing something silly, is there any particular reason alarm(2) isn't exposed in ext/posix/unistd.c?
It seems like a trivial patch to do (and I'd be happy to write one up), just making sure I'm not about to do something (obviously) stupid...

By all means.  Assuming you mean this:
http://pubs.opengroup.org/onlinepubs/009695399/functions/alarm.html

...then a patch is welcome, and the only reason for omission thus far is that no client currently uses it, or has sent back a suitable implementation!

Submitted pr 226.  Closing this issue...


JSESSIONID not present in the set of Cookies in the PhantomJS browser

Hi
I am running a test which is passing in firefox browser (Both Headless and non Headless) but it is failing in PhantomJS (phantomjsdriver-1.1.0.jar Driver).
This page contains several AJAX requests. In Firefox, Server is authenticating while opening the page and use JSESSIONID for subsequent request.
When I am printing set of Cookies present using following code
Set cookieSet = driver.manage().getCookies();
for (Cookie c: cookieSet){
System.out.println("Cookie: " + c.getName() + "with value: " + c.getValue());
}
I am not seeing JSESSIONID present in case of PhantomJS browser. Because of this it every request from PhantomJS is getting authenticated.
I am using Arquillian to run the test. Below in config for that.

phantomjs
true

Please suggest if I am missing something.

Arquillian Config

phantomjs
true

I'm experiencing a similar behavior which might be related.
PhantomJS: 1.9.7
Selenium-WebDriver: 2.46.2
I noticed that cookies with the secure flag set are not returned by: driver.manage.all_cookies (ruby). But they are returned when using firefox.
I.e. If your server sets these two cookies:
Set-Cookie: secure=foo; Expires=Thu, 01-Jan-2020 00:00:01 GMT; Path=/; Secure
Set-Cookie: insecure=bar; Expires=Thu, 01-Jan-2020 00:00:01 GMT; Path=/

Only the second one can be read:
driver.manage.all_cookies
=> [{:name=>"insecure", :value=>"bar", :path=>"/", :domain=>"localhost", :expires=>#<DateTime: 2020-01-01T00:00:00+00:00 ((2458850j,0s,999999919n),+0s,2299161j)>, :secure=>false}]

A workaround is to set a cookie file using the phantomjs.cli.args capabilities (e.g. --cookies-file=/tmp/cookies.txt), and parse that file to find the cookie you need.