pebble SDK emulator is not fully functional

It is possible to install and run the pebble SDK command line tool in Bash on Ubuntu on Windows     using this guide: https://developer.pebble.com/sdk/install/linux/.
It is also possible to start the emulator (with X-ming running) but there is an error when trying to install any apps.
The error is [Errno 22] Invalid argument.
Using strace -f it appears that the problem comes from a setsockopt call (TCP_KEEPCNT option on a IPPROTO_IP socket).
Is the TCP_KEEPCNT option supported for IPPROTO_IP?
If not: When will it be?
Here is the relevant part of the trace:
[pid 32276] socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 32276] fcntl(3, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 32276] fcntl(3, F_SETFL, O_RDWR)   = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [30], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [10], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPCNT, [3], 4) = -1 EINVAL (Invalid argument)
[pid 32276] close(3)                    = 0
[pid 32276] select(0, NULL, NULL, NULL, {0, 500000}) = 0 (Timeout)
[pid 32276] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
[pid 32276] open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
[pid 32276] fstat(3, {st_mode=S_IFREG|0644, st_size=221, ...}) = 0
[pid 32276] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7bf040000
[pid 32276] read(3, "<IP_ADDRESS> localhost\n\n# The follo"..., 512) = 221
[pid 32276] read(3, "", 512)            = 0
[pid 32276] close(3)                    = 0
[pid 32276] munmap(0x7fd7bf040000, 4096) = 0
[pid 32276] socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 32276] fcntl(3, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 32276] fcntl(3, F_SETFL, O_RDWR)   = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [30], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [10], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPCNT, [3], 4) = -1 EINVAL (Invalid argument)
[pid 32276] close(3)                    = 0
[pid 32276] select(0, NULL, NULL, NULL, {0, 500000}) = 0 (Timeout)
[pid 32276] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
[pid 32276] open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
[pid 32276] fstat(3, {st_mode=S_IFREG|0644, st_size=221, ...}) = 0
[pid 32276] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7bf040000
[pid 32276] read(3, "<IP_ADDRESS> localhost\n\n# The follo"..., 512) = 221
[pid 32276] read(3, "", 512)            = 0
[pid 32276] close(3)                    = 0
[pid 32276] munmap(0x7fd7bf040000, 4096) = 0
[pid 32276] socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 32276] fcntl(3, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 32276] fcntl(3, F_SETFL, O_RDWR)   = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [30], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [10], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPCNT, [3], 4) = -1 EINVAL (Invalid argument)
[pid 32276] close(3)                    = 0
[pid 32276] select(0, NULL, NULL, NULL, {0, 500000}) = 0 (Timeout)
[pid 32276] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
[pid 32276] open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
[pid 32276] fstat(3, {st_mode=S_IFREG|0644, st_size=221, ...}) = 0
[pid 32276] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7bf040000
[pid 32276] read(3, "<IP_ADDRESS> localhost\n\n# The follo"..., 512) = 221
[pid 32276] read(3, "", 512)            = 0
[pid 32276] close(3)                    = 0
[pid 32276] munmap(0x7fd7bf040000, 4096) = 0
[pid 32276] socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 32276] fcntl(3, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 32276] fcntl(3, F_SETFL, O_RDWR)   = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [30], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [10], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPCNT, [3], 4) = -1 EINVAL (Invalid argument)
[pid 32276] close(3)                    = 0
[pid 32276] select(0, NULL, NULL, NULL, {0, 500000}) = 0 (Timeout)
[pid 32276] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
[pid 32276] open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
[pid 32276] fstat(3, {st_mode=S_IFREG|0644, st_size=221, ...}) = 0
[pid 32276] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7bf040000
[pid 32276] read(3, "<IP_ADDRESS> localhost\n\n# The follo"..., 512) = 221
[pid 32276] read(3, "", 512)            = 0
[pid 32276] close(3)                    = 0
[pid 32276] munmap(0x7fd7bf040000, 4096) = 0
[pid 32276] socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 32276] fcntl(3, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 32276] fcntl(3, F_SETFL, O_RDWR)   = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [30], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [10], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPCNT, [3], 4) = -1 EINVAL (Invalid argument)
[pid 32276] close(3)                    = 0
[pid 32276] select(0, NULL, NULL, NULL, {0, 500000}) = 0 (Timeout)
[pid 32276] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
[pid 32276] open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
[pid 32276] fstat(3, {st_mode=S_IFREG|0644, st_size=221, ...}) = 0
[pid 32276] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7bf040000
[pid 32276] read(3, "<IP_ADDRESS> localhost\n\n# The follo"..., 512) = 221
[pid 32276] read(3, "", 512)            = 0
[pid 32276] close(3)                    = 0
[pid 32276] munmap(0x7fd7bf040000, 4096) = 0
[pid 32276] socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 32276] fcntl(3, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 32276] fcntl(3, F_SETFL, O_RDWR)   = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [30], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [10], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPCNT, [3], 4) = -1 EINVAL (Invalid argument)
[pid 32276] close(3)                    = 0
[pid 32276] select(0, NULL, NULL, NULL, {0, 500000}) = 0 (Timeout)
[pid 32276] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
[pid 32276] open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
[pid 32276] fstat(3, {st_mode=S_IFREG|0644, st_size=221, ...}) = 0
[pid 32276] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7bf040000
[pid 32276] read(3, "<IP_ADDRESS> localhost\n\n# The follo"..., 512) = 221
[pid 32276] read(3, "", 512)            = 0
[pid 32276] close(3)                    = 0
[pid 32276] munmap(0x7fd7bf040000, 4096) = 0
[pid 32276] socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 32276] fcntl(3, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 32276] fcntl(3, F_SETFL, O_RDWR)   = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [30], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [10], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPCNT, [3], 4) = -1 EINVAL (Invalid argument)
[pid 32276] close(3)                    = 0
[pid 32276] select(0, NULL, NULL, NULL, {0, 500000}) = 0 (Timeout)
[pid 32276] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
[pid 32276] open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
[pid 32276] fstat(3, {st_mode=S_IFREG|0644, st_size=221, ...}) = 0
[pid 32276] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7bf040000
[pid 32276] read(3, "<IP_ADDRESS> localhost\n\n# The follo"..., 512) = 221
[pid 32276] read(3, "", 512)            = 0
[pid 32276] close(3)                    = 0
[pid 32276] munmap(0x7fd7bf040000, 4096) = 0
[pid 32276] socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 32276] fcntl(3, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 32276] fcntl(3, F_SETFL, O_RDWR)   = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [30], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [10], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPCNT, [3], 4) = -1 EINVAL (Invalid argument)
[pid 32276] close(3)                    = 0
[pid 32276] select(0, NULL, NULL, NULL, {0, 500000}) = 0 (Timeout)
[pid 32276] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
[pid 32276] open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
[pid 32276] fstat(3, {st_mode=S_IFREG|0644, st_size=221, ...}) = 0
[pid 32276] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7bf040000
[pid 32276] read(3, "<IP_ADDRESS> localhost\n\n# The follo"..., 512) = 221
[pid 32276] read(3, "", 512)            = 0
[pid 32276] close(3)                    = 0
[pid 32276] munmap(0x7fd7bf040000, 4096) = 0
[pid 32276] socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 32276] fcntl(3, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 32276] fcntl(3, F_SETFL, O_RDWR)   = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [30], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [10], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPCNT, [3], 4) = -1 EINVAL (Invalid argument)
[pid 32276] close(3)                    = 0
[pid 32276] stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=114, ...}) = 0
[pid 32276] open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
[pid 32276] fstat(3, {st_mode=S_IFREG|0644, st_size=221, ...}) = 0
[pid 32276] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd7bf040000
[pid 32276] read(3, "<IP_ADDRESS> localhost\n\n# The follo"..., 512) = 221
[pid 32276] read(3, "", 512)            = 0
[pid 32276] close(3)                    = 0
[pid 32276] munmap(0x7fd7bf040000, 4096) = 0
[pid 32276] socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 32276] fcntl(3, F_GETFL)           = 0x2 (flags O_RDWR)
[pid 32276] fcntl(3, F_SETFL, O_RDWR)   = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPIDLE, [30], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPINTVL, [10], 4) = 0
[pid 32276] setsockopt(3, SOL_TCP, TCP_KEEPCNT, [3], 4) = -1 EINVAL (Invalid argument)
[pid 32276] close(3)                    = 0
[pid 32276] write(2, "[Errno 22] Invalid argument\n", 28[Errno 22] Invalid argument
) = 28

@MorrisTimm - Thanks for the detailed trace and log information. As you have already figured out, TCP_KEEPCNT is currently not supported on WSL sockets. We are aware of this and have a task to track it, but we don't have an ETA at the moment. We will continue to provide updates through our release notes whenever these fixes make it to the release. And, thanks for trying out WSL and the feedback.

You can hack the pebble SDK to work if do this:
cd .env/lib/python2.7/site-packages/websocket/_socket.py
nano _socket.py
comment out the two lines referencing TCP_KEEPCNT
python -m compileall .
With this change both installs to a physical pebble and even the emulator work!

@lukeforeman: Good catch! I searched for the TCP_KEEPCNT option but didn't find it. I didn't look in the .env directory :)
It seems to be working now. Instead of "python -m compileall" I just deleted the _socket.pyc file. It is then recreated automatically.

Had the same problem and solution from lukeforeman worked for me too. Thanks!

Just an update that the Windows Networking team has made the change to TCP_KEEPCNT parameter tunable. We have checked in the equivalent fix to expose this parameter in WSL. The fix is in our dev branch, but should soon hit the release branch. Keep an eye out on the release notes. And, thanks for trying out WSL and giving us your valuable feedback.

This should be fixed in 15002. Please validate and let us know if the issue persists.

@sunilmut  I don't know if you are considering that a workaround for this was implemented in the Pebble Tool a while ago by @Katharine.

@sGerli - I am indicating that TCP_KEECNT is now supported in WSL, starting build 15002. See the release notes. So, the patch should no longer be needed. Please validate at your end and let us know.

Closing this out. If anyone is still (on builds 15002+) having problems, please let us know and we can reopen.


rspec-mocks + minitest 5.6 dont play nice together

see https://github.com/jekyll/jekyll/issues/3656
Minitest 5.6 now defines a expect method, which collides with rspec-mocks.
What we can do about it:

Dont do anything, but I am sure people will open issues here confused.
Lock minitest version in rspec-mocks.
Give a warning to the user, so they will better understand the error and the solution.

thoughts?

Good to know.  Ideally, I'd like to find a way to make rspec-mocks and minitest 5.6+ play nice together.  For rspec-expectations, we added a file that takes care of integrating it with minitest awhile back:
https://github.com/rspec/rspec-expectations/blob/v3.2.1/lib/rspec/expectations/minitest_integration.rb
https://github.com/rspec/rspec-expectations/blob/v3.2.1/features/test_frameworks/minitest.feature
I think we could similarly add a rspec/mocks/minitest_integration file that minitest users who use rspec-mocks could use to deal with integrating the two and handling the dual role of expect.  We'll need to think carefully about how to make expect compatible with both libs, though.

Here's a simple way to make them compatible:

Move the to and not_to methods currently defined on RSpec::Mocks::ExpectationTarget into a module (which is included into RSpec::Mocks::ExpectationTarget to keep the method available there).
In the new rspec/mocks/minitest_integration file, we would also include the module in Minitest::Expectation if Minitest::Expectation is defined.

BTW, you can see the relevant minitest change here:
https://github.com/seattlerb/minitest/commit/9e78cc974f3ef0d9716f1cca2675753cf5f648d0

[related https://github.com/jekyll/jekyll/pull/3658]

Attaching myself to this ticket so I get notified if you guys make any changes and can remove our hacks from Jekyll.  Thanks @arthurnn and @myronmarston!

I think , to fix this, we could disable expecations on minitest using that env var https://github.com/seattlerb/minitest/blob/master/lib/minitest/spec.rb#L327

I think , to fix this, we could disable expecations on minitest using that env var https://github.com/seattlerb/minitest/blob/master/lib/minitest/spec.rb#L327

Maybe I'm reading the code wrong, but I don't see how that will help.  The expect method is defined in Minitest::Spec::DSL::InstanceMethods and will be unaffected by that environment variable.  Even if it were to solve the problem, I don't think it's a solution I'd be happy with; users should be able to use minitest-spec's obj.must_xyz syntax and rspec-mocks in the same project if they wish.  (Some users may already be doing that and preventing the must_xyz methods from being added to Object would be a breaking change for them).
As I said above, I think the solution is to define a rspec/mocks/minitest_integration file that, among other things, includes a module in Minitest::Expectation that provides the to and not_to methods so that it works properly.

I agree with @myronmarston.  While I wouldn't complain if it did happen that way (because it is an rspec lib after all) it would bum me out that it did have to happen that way.

I've whipped together a simple patch for this problem, which fixes the problem for me. An rspec/mocks/minitest_integration file is added, which needs to be loaded after minitest.
For a future improvement, the refactoring suggested by @myronmarston (allowing the Minitest::Expectation class to handle #to, #not_to & #to_not directly) would save allocating an RSpec::Mocks::ExpectationTarget  for each call.


Improvements around forc plugins command

Hey, so I was getting started with Sway, and tried calling forc plugins command while I was exploring the functionality.
This lead to a crash:
~/dev/sway/target/debug$ forc plugins
Installed Plugins:
thread 'main' panicked at 'Could not get plugin description.: Os { code:
2, kind: NotFound, message: "No such file or directory" }',
forc/src/cli/commands/plugins.rs:43:10
stack backtrace:
   0: rust_begin_unwind
at
/rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
at
/rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panicking.rs:143:14
   2: core::result::unwrap_failed
at
/rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/result.rs:1785:5
   3: core::result::Result<T,E>::expect
at
/rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/result.rs:1035:23
   4: forc::cli::commands::plugins::parse_description_for_plugin
at
/home/joao/dev/sway/forc/src/cli/commands/plugins.rs:40:16
   5: forc::cli::commands::plugins::format_print_description
at
/home/joao/dev/sway/forc/src/cli/commands/plugins.rs:81:23
   6: forc::cli::commands::plugins::print_plugin
at
/home/joao/dev/sway/forc/src/cli/commands/plugins.rs:97:5
   7: forc::cli::commands::plugins::exec
at
/home/joao/dev/sway/forc/src/cli/commands/plugins.rs:28:21

The first commit deals with this, which happens due to not using the full path while exec'ing the plugin, so it just changes the function to receive the full path. The docs say unwrap should not generally be used because it can lead to a panic, but here we know we have a valid path, so I thought it was okay. If that is not the case, let me know.
After that, we get:
Installed Plugins:
forc-cd4662b67fa2a5c9
forc-7550923f7d96c76d
forc-7eada75316346bfc
forc-fac57f493a01e50b
forc-lsp
forc-fmt-v2
forc-explore
forc-fmt
forc-gm

The second commit just skips those executables from being considered plugins. While I generally don't like hardcoding the deps folder in the check, it seemed like a sensible solution for this not too frequent edge case, since it will probably only happens for developers. What do you think?

@tritao Does this PR have an associated issue?

@tritao Does this PR have an associated issue? I see the PR fixes something (from your PR description), but trying to see what the originating issue is/was

No, found the issue today and sent the PR directly. Should I create one?

@tritao For housekeeping purposes, yes that would be great if you could create one :)

My review was requested again recently, however I still have some outstanding Qs here. To clarify, the premise for some of these changes is that it may be common to have the deps of a target directory on a user's PATH, however I'm not so sure that this is the common case and I suspect these errors might result from misuse of PATH, hence the Qs in the linked comment.

My review was requested again recently, however I still have some outstanding Qs here. To clarify, the premise for some of these changes is that it may be common to have the deps of a target directory on a user's PATH, however I'm not so sure that this is the common case and I suspect these errors might result from misuse of PATH, hence the Qs in the linked comment.

I've updated the PR with just the fix to use the full path to the plugin, which seems like the correct thing to do either way.

Poking @ra0x3 for re-review


What to do with typos when answering script questions?

Backspace doesn't seem to work in the console and I usually don't want to exit from the script. What do I do if I make a typo and want to correct it?
It then becomes easy to be "exited" prematurely from the script rather than having a chance to type a module name correctly.
Thanks!

I'm open to listening to any ideas you might have. I can't think of any good solutions that let you go back to a previous field in a CLI application.


Release

This release will:
Close #84
Close #85

:tada: This PR is included in version 1.9.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Can the library evaluate Excel formula when reading + return type of cell data?

Hi,
I am interested to try out the OpenXLSX library for reading and writing data from/to XLSX.
Q1. Can the library evaluate excel formulas?
If e.g. a cell contains a formula "=2+3".
When reading, will the OpenXLSX library return a number (5) or a string ("=2+3")? Or can it do both?

Q2. Can the library return the type of the cell data (float, string...)
If yes - which types are supported yet?

In the documentation I could not find this information.
Thanks,
STU

Hi @stu0408,
Regarding your first question, the answer is no. OpenXLSX does not evaluate cell formulas, and to be honest it's not something I intend to implement. Formulas can be very long and complicated, so implementing cell formula evaluation is almost like implementing a parser for a programming language. If someone else wants to implement it, I will be happy to merge into the project, but it's not something I'm going to do. Also, I know of no other Excel library (in any programming language) that supports formula evaluation, which indicates that it is a complicated matter and/or it's not something a lot of people need.
As for your second question, the answer is yes. You can access it through the XLCellValue class. E.g., if you have an XLCell object called myCell, you can do the following:
auto cellType = myCell.value().type();

This returns an XLValueType enum, which has the following possible values:
enum class XLValueType { Empty, Boolean, Integer, Float, Error, String };

/Kenneth

Hi @troldal ,
thank you very much for the quick response. This helps further.

"Also, I know of no other Excel library (in any programming language) that supports formula evaluation, which indicates that it is a complicated matter and/or it's not something a lot of people need."
The OpenPyXl library can do it, also the Aspose.Cells.
https://openpyxl.readthedocs.io/en/stable/api/openpyxl.reader.excel.html => (data_only=True)
classopenpyxl.reader.excel.ExcelReader(fn, read_only=False, keep_vba=False, data_only=False, keep_links=True
Parameters:

filename (string or a file-like object open in binary mode c.f., zipfile.ZipFile) – the path to open or a file-like object
read_only (bool) – optimised for reading, content cannot be edited
keep_vba (bool) – preseve vba content (this does NOT mean you can use it)
data_only (bool) – controls whether cells with formulae have either the formula (default) or the value stored the last time Excel read the sheet
keep_links (bool) – whether links to external workbooks should be preserved. The default is True

Kind regards,
STU

Interesting. I didn't know that.
With the pricetag on the Aspose.Cells library, I guess they have a lot more resources to implement that kind of functionality. But I'm impressed that OpenPyXl can do it (although it is only limited support for formula evaluation), considering that it is an open-source library.

Yes, it is a cool feature. As much I understand OpenPyXl is just reading the cached formula values that Excel provides. Everytime Excel is opened it does the dirty job (formula evaluating) for OpenPyXl.

https://stackoverflow.com/questions/36116162/python-openpyxl-data-only-true-returning-none
https://stackoverflow.com/questions/53271690/openpyxl-how-to-read-formula-result-after-editing-input-data-on-the-sheet-data

STU

Short tests showed that OpenXLSX is reading the cached data from Excel already.
If in a cell e.g. following is defined ="(2+3)*3" then returned value will be 15 and not a string that represents the formula.
Therefore I will close this issue. It works as expected.
STU


Add Documentation on setting Flask

Type
Advancement
Description
Create a Documentation File using Markdown and add it into a specific Documentation/ directory to allow new contributors to get started with setting Flask. The features to be added in the documentation includes:

Give a basic idea about Flask as a Micro Web-Framework.
Give a step-wise plan of setting Flask on Windows/Linux/OSX Environment.
Add Screenshots wherever possible.
Add a sample Flask Code with a mock endpoint to show the Server up and running.

Tools

Markdown
Python
Pip/Conda

Have you read the Contributing Guidelines on Pull Requests?
Yes
Motivation
To contribute to the Wiki of this Project in the longer run.

I would like to work on this @HarshCasper

I have worked in flask, so i can work on the documentation

Assigned to you @rjn01


Selected color

Hey !
Is there a way to change current blue color for selection in Select ? Thxx

Hi @9Samzer ,
The current select is under the pointer style class, you can change it like this.
from InquirerPy import inquirer
from InquirerPy.base.control import Choice
from InquirerPy.utils import get_style

action = inquirer.select(
    message="Select an action:",
    choices=[
        "Upload",
        "Download",
        Choice(value=None, name="Exit"),
    ],
    default=None,
    style=get_style({"pointer": "yellow"}),
).execute()

This will change the selection to yellow. Please refer to doc for more info.
Thanks,
Kevin


Fix sitemaps api

in py2, robot.sitemaps returns a list. in py3 it returns whatever comes out of map which is some kind of iter.
this pr changes it to always return a list.
Fixes #109

The change looks good, but I'm not sure why the submodule got updated? Was it out of date?

I know nothing about submodules.

I can push a branch preserving your commits, but resolving the submodule issue. Otherwise, you could give this a shot:
# Ensure all the submodules are initialized and checked out at the current SHAs
git submodule update --init --recursive
(
  cd reppy/rep-cpp
  git pull --rebase origin master
  # Update the googletest submodule
  git submodule update --init --recursive
)

git add reppy
git commit --amend

And thanks for the PR, btw!

Hm, that left me with a repo that can't push -- not really worth debugging, though.


chore: Update VS Code settings for Ruff and file nesting

Pull Request
Description
This change updates the VS Code settings to enhance the development environment:

Adds Ruff configuration path, pointing to the analyser's pyproject.toml file.

Enables file nesting in the Explorer view, with specific patterns for:

pyproject.toml: Nesting related Python project files
package.json: Nesting related JavaScript/Node.js project files

Sets the file nesting to be collapsed by default.

These changes aim to improve code organisation and streamline the project structure within VS Code.
fixes #170

#169  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @JackPlowman and the rest of your teammates on  Graphite

Merge activity

Oct 29, 4:15 AM EDT: A user merged this pull request with Graphite.


Validate primary column uniqueness if migration function is not specified

This PR is related to the second issue in https://github.com/realm/realm-cocoa/issues/4370: in case if the migration function is not specified the validation of the primary column should still be performed.
/cc @bdash, @jpsim

/cc @tgoyne, as he's most familiar with the migration code.

I'm not sure this makes sense. The purpose of re-checking all primary keys is to detect bugs in the migration function, since primary key uniqueness is not enforced during a migration. I think instead we would want to require that any schema changes which could produce duplicate PKs include a migration function to set them. I think this is just changing a PK from optional to non-optional and creating an entirely new column which is the new PK.

Created a separate issue for this and closing this PR.


python312Packages.tencentcloud-sdk-python: 3.0.1126 -> 3.0.1127

Diff: https://github.com/TencentCloud/tencentcloud-sdk-python/compare/refs/tags/3.0.1126...3.0.1127
Changelog: https://github.com/TencentCloud/tencentcloud-sdk-python/blob/3.0.1127/CHANGELOG.md
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 303221 run on x86_64-linux 1

4 packages built:

python311Packages.tencentcloud-sdk-python
python311Packages.tencentcloud-sdk-python.dist
python312Packages.tencentcloud-sdk-python
python312Packages.tencentcloud-sdk-python.dist


Add lint

Add ESLint support to the project
Links that maybe can help:
https://eslint.org/
https://khalilstemmler.com/blogs/typescript/eslint-for-typescript/

Working on it

@MounicaChanda0809 Thanks for your interesting! Go ahead! 🚀🚀

@MounicaChanda0809 Thanks for your interesting! Go ahead! 🚀🚀

Created PR. Please take a look when you have time :)

@MounicaChanda0809 Very nice! I'll review it soon! Thank you 😍😍

Resolved in #23


Sorting feature lists before alignment in batch mode

Is your feature request related to a problem? Please describe.
Sorting of feature lists before alignment in batch mode was possible in MZmine2. We are really missing this feature to reconstruct the original order of samples.
Describe the solution you'd like
Add the sort feature lists to batch mode again, please.
Describe alternatives you've considered
An alternative could be an optional switch in the alignment. But a general method is preferable.
Additional context
Add any other context or screenshots about the feature request here.

I also miss the "sort feature lists" method from MZmine2 and would like to see it in MZmine3!

closing for now, as this is a duplicate.


App stats should read from DB not the json files

Description
In the API file pages/api/statistics/totals.js it still counts the json files but as these are loaded into the DB, we should be using the DB not the files
Screenshots
No response
Additional information
No response

Could you please assign the issue to me?

Sure, assigned 👍 (btw you don't need to tag me, other maintainers can assign also)


Once mods are added the server doesn't start listening

It's a completely fresh install.
I actually switched from a manual install to arkservertools as I couldn't get any mods running. Maybe this is a general issue ATM ?
I could also add a complete history output with important commands.
That would also help ppl. to get a complete installation instruction for new environments (Ubuntu 18.04).
I used the fix for Linux Servers described in #1063
Starting the server without any addons doen't yield an Issue.
But adding an addon to /etc/arkmanager/instances/main.cfg  breaks the server.
I've tested different mods for example adding only<PHONE_NUMBER> or<PHONE_NUMBER>.
Running command 'status' for instance 'main'
 Server running:   Yes
 Server PID:   5891
 Server listening:   No
 Server build ID:   4776339

cat ShooterGame-b...
[2020.03.16-10.37.49:789][  0]Log file open, 03/16/20 11:37:49
[2020.03.16-10.37.49:789][  0]ARK Version: 307.4
[2020.03.16-10.37.49:789][  0]PID: 5891
[2020.03.16-10.38.22:514][  0]Primal Game Data Took 31.69 seconds

cat /var/log/arktools/arkmanager.log:
2020-03-16 11:50:09: [main] The server is starting...
2020-03-16 11:50:09: [main] The server is now running, and should be up within 10 minutes

cat /var/log/arktools/arkserver.log
2020-03-16 11:50:09: [main] 2020-03-16 11:50:09: start
2020-03-16 11:50:09: [main] 2020-03-16 11:50:09: Running /home/steam/ARK/ShooterGame/Binaries/Linux/ShooterGameServer Valguero_P\?RCONEnabled=True\?RCONPort=32330\?SessionName=HIDDENNAME\ YOLO\?Port=7777\?QueryPort=27015\?ServerPassword=HIDDENPW\?ServerAdminPassword=keyboardcat\?MaxPlayers=20\?GameModIds=1786522940\?listen -log -log
2020-03-16 11:50:09: [main] 2020-03-16 11:50:09: Server PID: 12700
2020-03-16 11:50:10: [main] [S_API FAIL] SteamAPI_Init() failed; SteamAPI_IsSteamRunning() failed.
2020-03-16 11:50:11: [main] Setting breakpad minidump AppID = 346110

Additional information (whole setup on a fresh server) executed as root:
Note after writing:
Looks like this installation is working on a fresh server. So you can use this to setup on ubuntu 18.
# system requirements
$ apt-get update
$ apt-get upgrade
$ apt-get install perl-modules curl lsof libc6-i386 lib32gcc1 bzip2 libcompress-raw-zlib-perl

# steamcmd
$ add-apt-repository multiverse
$ dpkg --add-architecture i386
$ add-get install steamcmd

# firewall
$ iptables -I INPUT -p udp --dport 27016 -j ACCEPT
$ iptables -I INPUT -p tcp --dport 27016 -j ACCEPT
$ iptables -I INPUT -p udp --dport 7778 -j ACCEPT
$ iptables -I INPUT -p tcp --dport 7778 -j ACCEPT
$ iptables -I INPUT -p tcp --dport 32330 -j ACCEPT

# ark server tools
$ curl -sL http://git.io/vtf5N | sudo bash -s steam
$ arkmanager --version // doesnt find the executables as the default ubuntu config is in /usr/games
$ nano /etc/arkmanager/arkmanager.cfg  // changed executable to /usr/games
$ arkmanager --version // should work now
$ nano /etc/arkmanager/instances/main.cfg // changed map to Valguero_P, name, password

# installing the server
$ arkmanager --verbose install
>.... (a lot of steam output)
>Update state (0x101) committing, progress: 88.75<PHONE_NUMBER>1 /<PHONE_NUMBER>7)
>Success! App '376030' fully installed.
>CWorkThreadPool::~CWorkThreadPool: work processing queue not empty: 2 items discarded.

# starting the server
$ arkmanager start
$ arkmanager status
>Running command 'status' for instance 'main'
>Server running:   Yes
>Server PID:   16435
>Server listening:   Yes
>Server Name: SomeServerName - (v307.4)
>Players: 0 / 20
>Active Players: 0
>Server online:   Yes
>ARKServers link:   http://arkservers.net/server/<IP_ADDRESS>:27015
>Steam connect link:   steam://connect/<IP_ADDRESS>:27015
>Server build ID:   4776339

$ arkmanager stop

Now if I add an mod. In this case Awesome Spyglass. Is one of the most subscribed so it should work.
https://steamcommunity.com/sharedfiles/filedetails/?id=1404697612
# install mod
$ arkmanager installmod<PHONE_NUMBER>
>Running command 'installmod' for instance 'main'
>Downloading mod<PHONE_NUMBER> ... Mod<PHONE_NUMBER> downloaded
>Copying files to /home/steam/ARK/ShooterGame/Content/Mods/1404697612
>Mod<PHONE_NUMBER> installed

$ ls -al /home/steam/ARK/ShooterGame/Content/Mods/1404697612
>drwxr-xr-x  3 steam steam  4096 Mar 16 14:18 111111111
>-rwxr-xr-x  1 steam steam   273 Mar 16 14:12 111111111.mod
>drwxr-xr-x  5 steam steam  4096 Mar 16 14:24<PHONE_NUMBER>
>-rw-r--r--  1 steam steam   310 Feb 14  2019<PHONE_NUMBER>.mod
>drwxr-xr-x  4 steam steam 20480 Mar 16 14:18 Ragnarok
>drwxr-xr-x  5 steam steam  4096 Mar 16 14:18 TheCenter
>drwxr-xr-x  4 steam steam  4096 Mar 16 14:18 Valguero

# enable the mod
$ arkmanager enablemod<PHONE_NUMBER>
>Running command 'enablemod' for instance 'main'
>/usr/local/bin/arkmanager: line 2813: /etc/arkmanager/instances/main.cfg: Permission denied
# this is quite interesting and may be an issue.
# I just used nano to chenge it and add<PHONE_NUMBER> to gameModIDS

Now comes the server starting part
# restart server
$ arkmanager start

$ cat /var/log/arktools/arkserver.log
>Running /home/steam/ARK/ShooterGame/Binaries/Linux/ShooterGameServer Valguero_P\?>RCONEnabled=True\?RCONPort=32330\?SessionName=NAME\?Port=7777\?>QueryPort=27015\?ServerPassword=PASSWORD\?ServerAdminPassword=keyboardcat\?MaxPlayers=20\?>GameModIds=1404697612\?listen
>2020-03-16 14:32:29: [main] 2020-03-16 14:32:29: Server PID: 24535
>2020-03-16 14:32:30: [main] [S_API FAIL] SteamAPI_Init() failed; SteamAPI_IsSteamRunning() failed.
>2020-03-16 14:32:30: [main] Setting breakpad minidump AppID = 346110
>2020-03-16 14:34:15: [main] 2020-03-16 14:34:15: server is up

$ arkmanager status
>Running command 'status' for instance 'main'
>Server running:   Yes
>Server PID:   24535
>Server listening:   Yes
>Server Name: SomeServerName - (v307.4)
>Players: 0 / 20
>Active Players: 0
>Server online:   Yes
>ARKServers link:   http://arkservers.net/server/<IP_ADDRESS>:27015
>Steam connect link:   steam://connect/<IP_ADDRESS>:27015
>Server build ID:   4776339

Uff .. looks like it's working now ? As you can see the Mod is In GameModIds when starting + server is listening
Just a quick check on another mod. I also installed the Valguero extension<PHONE_NUMBER>) like the Awesome Spyglass above.
https://steamcommunity.com/sharedfiles/filedetails/?id=1786522940
$ arkmanager start
>Running command 'status' for instance 'main'
>Server running:   Yes
>Server PID:   29879
>Server listening:   Yes
>Server Name: SomeServerName - (v307.4)
>Players: 0 / 20
>Active Players: 0
>Server online:   Yes
>ARKServers link:   http://arkservers.net/server/<IP_ADDRESS>:27015
>Steam connect link:   steam://connect/<IP_ADDRESS>:27015
>Server build ID:   4776339

I used the fix for Linux Servers described in #1063

The fix in #1063 was only required for a short time after Genesis dropped, as when Genesis dropped Wildcard forgot to include the Linux libraries when they pushed out the update.

$ arkmanager enablemod<PHONE_NUMBER>
>Running command 'enablemod' for instance 'main'
>/usr/local/bin/arkmanager: line 2813: /etc/arkmanager/instances/main.cfg: Permission denied
# this is quite interesting and may be an issue.

enablemod and disablemod require that the instance config be writable (they append or replace the arkmod_{modid}=game option in the instance config).

The fix in #1063 was only required for a short time after Genesis dropped, as when Genesis dropped Wildcard forgot to include the Linux libraries when they pushed out the update.

That could be why the second try setting up from scratch worked as I didn't include the fix anymore.


Remove Near-related code from the node

We have code that is related to NEAR at it has some problems and throws errors.
We need to clean it up and to remove that code.
We use a pallete created by our partners WEBB, who forked Rainbow Bridge code but don't support it anymore.
Artur might help to define the scope.
DOD:|
There should be no Near related code and messages in node and telemetry


response: Drop the use of EqualReader for TransferEncoding::Identity

It's purpose is unclear, and it causes the entire reader to be consumed,
even when client has disconnected and won't get the content.
If the application needs to flush the reader for some side-effect, that
can still be achieved by the application itself.
Resolves: #182

I'd like to support this pull request. I can't see a rationale for the existence of EqualReader either.
It effectively prevents streaming without chunking (even though chunking was inveneted for the purpose, some devices can(t handle chunking when streaming).
If there is a reason for having it, I still think it should be possible to disable it.

Since I have endorsement, and long time without review, I'll merge this.


[Feature request] When Shift-Selecting frames keep up preview with last selected frame

In the editor window when shift-selecting frames keep up the main preview with the last selected frame. Currently it always shows the first selected frame, and it would be useful te see a full sized version of the last selected in the frame range, so one could see what was being used for the operation.
This actually works pretty well when shift-selecting backwards say from frame 100->80 with [Shift]-[Left Arrow] but selecting forward, like say, frame 100->120 will no longer update the full size preview.
Also if I shift-select with the arrow keys all works fine, but if I shift select with the mouse from say frame 10->20 and and repeatedly press [Shift]-[Left Arrow] to deselect, it will change selection to 10->9, 10->8, 10->7 instead of 10->19, 10->18, 10->17 etc.
If I do it in reverse order and use the shit+mouse to select the frame range 20->10 it all works as expected, pressing [Shift]-[Right Arrow] will correctly update selection to 20->11, 20->12, 20->13, etc.

Understood :)

Done! (release 2.5, when available)
Shift+Left/Right will preview the current focused frame.
Shift+Click and after that Shift+Left/Right will arrange the selection based on the latest selection.

Cool thanks for the fix! Looking forward to the next release :)


Keep information about production values

Information like:

Lifetime energy Production
This week energy Production
Todays Energy Production
Yesterdays Energy Production
Production itself

Then, when the inverter is offline, we keep seeing these values:

Previous behaviour was to set the values to 0, if i compare it with my non-test env.

maybe also keep the power conversion error information available?

Indeed it seems like a good idea to keep all the counters. I do like that the other measurements go to unknown instead of 0 to not mess up graphs and min/max/average calculations etc.

yeah, i agree on the unknown part for the other counters, so this is just a bit of fine-tuning :)

Closed in favor of #180


calico-node pods are failing after upgrade from 3.22 to 3.23: felix is not ready: readiness probe reporting 503

Expected Behavior

Calico is working after upgrade to version 3.23.
Current Behavior

calico-node pods are failing after upgrade from 3.22 to 3.23:
$ kubectl describe po calico-node-pg52g
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  2m35s                  default-scheduler  Successfully assigned kube-system/calico-node-pg52g to dev-k8s-worker-r1
  Normal   Pulling    2m35s                  kubelet            Pulling image "docker.io/calico/cni:v3.23.3"
  Normal   Pulled     2m31s                  kubelet            Successfully pulled image "docker.io/calico/cni:v3.23.3" in 3.507665044s
  Normal   Created    2m31s                  kubelet            Created container install-cni
  Normal   Started    2m31s                  kubelet            Started container install-cni
  Normal   Pulling    2m28s                  kubelet            Pulling image "docker.io/calico/node:v3.23.3"
  Normal   Pulled     2m24s                  kubelet            Successfully pulled image "docker.io/calico/node:v3.23.3" in 3.680538035s
  Normal   Created    2m24s                  kubelet            Created container mount-bpffs
  Normal   Started    2m24s                  kubelet            Started container mount-bpffs
  Normal   Pulled     2m23s                  kubelet            Container image "docker.io/calico/node:v3.23.3" already present on machine
  Normal   Created    2m23s                  kubelet            Created container calico-node
  Normal   Started    2m23s                  kubelet            Started container calico-node
  Warning  Unhealthy  2m21s (x2 over 2m22s)  kubelet            Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused
  Warning  Unhealthy  2m15s                  kubelet            Readiness probe failed: 2022-07-27 18:18:21.875 [INFO][438] confd/health.go 180: Number of node(s) with BGP peering established = 10
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  2m5s  kubelet  Readiness probe failed: 2022-07-27 18:18:31.882 [INFO][665] confd/health.go 180: Number of node(s) with BGP peering established = 10
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  115s  kubelet  Readiness probe failed: 2022-07-27 18:18:41.881 [INFO][975] confd/health.go 180: Number of node(s) with BGP peering established = 10
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  105s  kubelet  Readiness probe failed: 2022-07-27 18:18:51.904 [INFO][1252] confd/health.go 180: Number of node(s) with BGP peering established = 10
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  95s  kubelet  Readiness probe failed: 2022-07-27 18:19:01.907 [INFO][1565] confd/health.go 180: Number of node(s) with BGP peering established = 10
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  85s  kubelet  Readiness probe failed: 2022-07-27 18:19:11.894 [INFO][1810] confd/health.go 180: Number of node(s) with BGP peering established = 10
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  79s  kubelet  Readiness probe failed: 2022-07-27 18:19:17.203 [INFO][2018] confd/health.go 180: Number of node(s) with BGP peering established = 10
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  75s  kubelet  Readiness probe failed: 2022-07-27 18:19:21.868 [INFO][2146] confd/health.go 180: Number of node(s) with BGP peering established = 10
calico/node is not ready: felix is not ready: readiness probe reporting 503
  Warning  Unhealthy  43s (x4 over 65s)  kubelet  (combined from similar events): Readiness probe failed: 2022-07-27 18:19:51.889 [INFO][3017] confd/health.go 180: Number of node(s) with BGP peering established = 10
calico/node is not ready: felix is not ready: readiness probe reporting 503

The only logs that are not INFO:
2022-07-27 18:19:21.025 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.025 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.025 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.025 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.025 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.025 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.026 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.026 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.027 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.027 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.027 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.027 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.028 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.028 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.028 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.028 [WARNING][2094] felix/l3_route_resolver.go 645: Unable to create route for IP; the node it belongs to was not recorded in IPAM IP=<IP_ADDRESS>
2022-07-27 18:19:21.066 [WARNING][2094] felix/daemon.go 1209: IPIP and/or VXLAN encapsulation changed, need to restart.
2022-07-27 18:19:21.066 [WARNING][2094] felix/daemon.go 715: Felix is shutting down reason="encapsulation changed"
2022-07-27 18:19:21.868 [WARNING][2094] felix/health.go 211: Reporter is not ready. name="int_dataplane"
2022-07-27 18:19:21.869 [WARNING][2094] felix/health.go 173: Health: not ready
2022-07-27 18:19:21.898 [WARNING][2094] felix/health.go 211: Reporter is not ready. name="int_dataplane"

Full log: https://gist.github.com/r0bj/1df72959f5f992efba3544fa5eb89d47
Calico manifest: https://projectcalico.docs.tigera.io/manifests/calico-etcd.yaml
Steps to Reproduce (for bugs)

Running calico in version 3.22.1
Upgrade calico to version 3.23.3 from manifest https://projectcalico.docs.tigera.io/manifests/calico-etcd.yaml
calico-node pods are failing

Your Environment

Calico version: 3.23.3
Orchestrator version (e.g. kubernetes, mesos, rkt):
kubernetes:
Client Version: v1.24.3
Kustomize Version: v4.5.4
Server Version: v1.24.3
Operating System and version:
Ubuntu 18.04

We ran into this as well, modifying the default-ipv4-ippool ippools.crd.projectcalico.org object to add vxlanMode: Never fixed it for us.  We're probably going to roll back to 3.22 though for for now until this is fixed.
This did not happen in recent cluster but did happen when rolling back our code about 6 months and upgrading a cluster to our current configs.

Sounds like this might have been introduced by this PR? https://github.com/projectcalico/calico/pull/5576
Fits the code area as well as the time frame. Perhaps we're not properly handling the difference between vxlanMode: "" and vxlanMode: Never for older clusters that don't have that field set.

CC @coutinhop - WDYT?

Sounds like this might have been introduced by this PR? https://github.com/projectcalico/calico/pull/5576
Fits the code area as well as the time frame. Perhaps we're not properly handling the difference between vxlanMode: "" and vxlanMode: Never for older clusters that don't have that field set.

@caseydavenport very possibly so, I'm trying to understand exactly how that's happening. Currently taking a look at the logs, will try to figure this out an fix it ASAP.

@r0bj or @mikesplain could you post the output for kubectl get felixconfigurations.crd.projectcalico.org -o yaml and kubectl get ippools.crd.projectcalico.org -o yaml so I can try to repro/better understand the issue?
It would also help if you could enable Debug logging on felix and post those.
Thanks!

I'm using calico with etcd datastore (installed from the manifest https://projectcalico.docs.tigera.io/manifests/calico-etcd.yaml) so there are no felixconfigurations.crd.projectcalico.org and ippools.crd.projectcalico.org CRDs in my clusters. There are just no calico related CRDs in this YAML manifest to install.
Can I change Felix log level without felixconfig CRD?

I think you can still get those with calicoctl:
calicoctl get felixconfigurations -o yaml and calicoctl get ippools -o yaml, could you try that?

Sure, this is the data:
$ calicoctl get felixconfigurations -o yaml
apiVersion: projectcalico.org/v3
items:
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2019-02-11T16:01:23Z"
    name: default
    resourceVersion: "7"
    uid: 47412911-2e16-11e9-92ae-5254008bbbc5
  spec:
    bpfLogLevel: ""
    ipipEnabled: true
    logSeverityScreen: Info
    reportingInterval: 0s
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2022-05-02T19:03:00Z"
    name: node.dev-k8s-controller-r1
    resourceVersion: "2998691"
    uid: 7a77cd98-4f36-410c-9bcd-1ffbe9720f91
  spec:
    bpfLogLevel: ""
    defaultEndpointToHostAction: Return
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2019-02-11T16:01:23Z"
    name: node.dev-k8s-controller-r2
    resourceVersion: "16"
    uid: 475bb9a6-2e16-11e9-9e0d-5254008403d4
  spec:
    bpfLogLevel: ""
    defaultEndpointToHostAction: Return
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2019-02-11T16:01:24Z"
    name: node.dev-k8s-controller-r3
    resourceVersion: "24"
    uid: 47df2ca4-2e16-11e9-9e0d-525400bfbe5d
  spec:
    bpfLogLevel: ""
    defaultEndpointToHostAction: Return
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2021-07-28T18:00:18Z"
    name: node.dev-k8s-external-lb-a-r1
    resourceVersion: "2857508"
    uid: 8008bab6-94cf-4f90-a7e4-536808fafdaf
  spec:
    bpfLogLevel: ""
    defaultEndpointToHostAction: Return
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2022-04-18T21:05:14Z"
    name: node.dev-k8s-external-lb-a-r2
    resourceVersion: "2988703"
    uid: 92ab10a1-9646-4b26-a22d-4c3b22b73fa9
  spec:
    bpfLogLevel: ""
    defaultEndpointToHostAction: Return
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2020-03-20T17:21:33Z"
    name: node.dev-k8s-storage-r5
    resourceVersion: "2593494"
    uid: a7a278d5-5ed2-4670-bb82-cd2d34dc79ce
  spec:
    bpfLogLevel: ""
    defaultEndpointToHostAction: Return
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2020-03-20T17:21:34Z"
    name: node.dev-k8s-storage-r6
    resourceVersion: "2593504"
    uid: ed3066fe-d697-49a6-9b55-56bc322d0825
  spec:
    bpfLogLevel: ""
    defaultEndpointToHostAction: Return
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2020-03-20T17:20:08Z"
    name: node.dev-k8s-storage-r7
    resourceVersion: "2593478"
    uid: f719f361-4417-42fd-82d1-9f4fbcb0adb7
  spec:
    bpfLogLevel: ""
    defaultEndpointToHostAction: Return
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2019-02-11T16:14:19Z"
    name: node.dev-k8s-worker-r1
    resourceVersion: "32"
    uid: 15c183fa-2e18-11e9-93df-002590d66e06
  spec:
    bpfLogLevel: ""
    defaultEndpointToHostAction: Return
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2019-02-11T16:16:46Z"
    name: node.dev-k8s-worker-r2
    resourceVersion: "89"
    uid: 6d804a97-2e18-11e9-b2a9-0cc47a0ae03c
  spec:
    bpfLogLevel: ""
    defaultEndpointToHostAction: Return
- apiVersion: projectcalico.org/v3
  kind: FelixConfiguration
  metadata:
    creationTimestamp: "2021-06-09T20:49:15Z"
    name: node.dev-k8s-worker-r3
    resourceVersion: "2830390"
    uid: 63664c62-4288-4f32-a8c3-8afbe0bff0cd
  spec:
    bpfLogLevel: ""
    defaultEndpointToHostAction: Return
kind: FelixConfigurationList
metadata:
  resourceVersion: "3046937"

# calicoctl get ippools -o yaml
apiVersion: projectcalico.org/v3
items:
- apiVersion: projectcalico.org/v3
  kind: IPPool
  metadata:
    creationTimestamp: "2019-02-11T16:01:23Z"
    name: default-ipv4-ippool
    resourceVersion: "5"
    uid: 47417284-2e16-11e9-92ae-5254008bbbc5
  spec:
    allowedUses:
    - Workload
    - Tunnel
    blockSize: 26
    cidr: <IP_ADDRESS>/16
    ipipMode: Always
    natOutgoing: true
    nodeSelector: all()
kind: IPPoolList
metadata:
  resourceVersion: "3046937"

Thank you all for jumping in!  Let me know if theres anything else we can do to help.
# kubectl get felixconfigurations.crd.projectcalico.org -o yaml
apiVersion: v1
items:
- apiVersion: crd.projectcalico.org/v1
  kind: FelixConfiguration
  metadata:
    annotations:
      projectcalico.org/metadata: '{"uid":"8c9cd4c8-ab43-4a00-9032-03d6146c8686","creationTimestamp":"2022-07-28T15:49:16Z"}'
    creationTimestamp: "2022-07-28T15:49:16Z"
    generation: 3
    name: default
    resourceVersion: "595230"
    uid: 8c9cd4c8-ab43-4a00-9032-03d6146c8686
  spec:
    bpfLogLevel: ""
    floatingIPs: Disabled
    ipipEnabled: true
    logSeverityScreen: Debug
    reportingInterval: 0s
kind: List
metadata:
  resourceVersion: ""

and:
# kubectl get ippools.crd.projectcalico.org -o yaml

apiVersion: v1
items:
- apiVersion: crd.projectcalico.org/v1
  kind: IPPool
  metadata:
    annotations:
      projectcalico.org/metadata: '{"uid":"e21999c6-97f1-46de-b690-dd87600ef07a","creationTimestamp":"2022-07-28T15:49:16Z"}'
    creationTimestamp: "2022-07-28T15:49:16Z"
    generation: 2
    name: default-ipv4-ippool
    resourceVersion: "1463"
    uid: 98a9c83a-c0bb-4b95-9f77-458d5d9eb5a1
  spec:
    blockSize: 26
    cidr: <IP_ADDRESS>/11
    ipipMode: CrossSubnet
    natOutgoing: true
kind: List
metadata:
  resourceVersion: ""

Thanks @r0bj and @mikesplain! @r0bj, if it's not too much to ask, could you send the full calico-log after enabling LogSeverityScreen:"Debug"? Thanks!

@caseydavenport do you mean the problem could be that pool.Spec.VXLANMode could be "" here?
https://github.com/projectcalico/calico/blob/8bfde6eabb506bd5973901f0ac6509be7d77f989/felix/calc/encapsulation_resolver.go#L201
That line is run at felix startup (in daemon.go) on all IP pools retrieved from the client.
I was lead to believe that it would never be an empty string (it would be defaulted to "Never" as the comment says):
https://github.com/projectcalico/calico/blob/8bfde6eabb506bd5973901f0ac6509be7d77f989/api/pkg/apis/projectcalico/v3/ippool.go#L53-L55
Let's look at the full debug logs to be sure, but do you think changing the check to pool.Spec.VXLANMode == apiv3.VXLANModeAlways || pool.Spec.VXLANMode == apiv3.VXLANModeCrossSubnet (and the same for IPIP) would solve it? (either that or pool.Spec.VXLANMode != "" && pool.Spec.VXLANMode != apiv3.VXLANModeNever)

@coutinhop Sure, there is calico-node log with LogSeverityScreen:"Debug":
calico-node.log

@caseydavenport do you mean the problem could be that pool.Spec.VXLANMode could be "" here?

Yep, if the pool was created prior to that field existing or created / edited with an older version of calicoctl that isn't aware of the field (or potentially another reason).
  spec:
    allowedUses:
    - Workload
    - Tunnel
    blockSize: 26
    cidr: <IP_ADDRESS>/16
    ipipMode: Always
    natOutgoing: true
    nodeSelector: all()

^ Can see it's not set here.

I'm using calico with etcd datastore

and

kubectl get felixconfigurations.crd.projectcalico.org -o yaml

One thing I'm not sure about - does the cluster use etcd mode or k8s CRD mode? Or are we talking about two different clusters here?
Do you recall what the original version of Calico that was installed on these clusters was? Like, the version used when the cluster was originally provisioned?

One thing I'm not sure about - does the cluster use etcd mode or k8s CRD mode? Or are we talking about two different clusters here?
@r0bj is using etcd mode and used calicoctl to get the felixconfig and ip pools, @mikesplain is using kdd and used kubectl, so yeah 2 different clusters with the same issue, I think...

One thing I'm not sure about - does the cluster use etcd mode or k8s CRD mode? Or are we talking about two different clusters here?

In my case it's etcd mode so I used calicoctl get felixconfigurations -o yaml

Do you recall what the original version of Calico that was installed on these clusters was? Like, the version used when the cluster was originally provisioned?

Git history for my cluster shows that it was created 6 years ago (it's a bare metal cluster upgraded in place) with calico in version 1.4.2 at that time.

One thing I'm not sure about - does the cluster use etcd mode or k8s CRD mode? Or are we talking about two different clusters here?

Confirming, we are using whatever the default was in kops now. At the time I think it may have been etcd but currently I believe it's crd mode.

Do you recall what the original version of Calico that was installed on these clusters was? Like, the version used when the cluster was originally provisioned?

Our git history shows the cluster is 3 years old, initially created with kops 1.11.1 & calico v3.3.1.  It was installed with this config file:
https://github.com/kubernetes/kops/blob/1.11.1/upup/models/cloudup/resources/addons/networking.projectcalico.org/k8s-1.7-v3.yaml.template

Perfect, so that supports the theory that these pools were created prior to VXLANMode being an option and the newest release is just not properly handling that case, so I think @coutinhop's fix for this in #6494 is probably good.
@coutinhop it occurs to me that we should look at doing read-time defaulting of that field in case there is any other code that might be hit by the same issue. We should be able to handle that in the Calico client code so any users of the client see "VXLANMode: Never" even if the underlying data doesn't include the field.

@caseydavenport makes sense, will look into doing that!

@coutinhop looks like we already have a good hook to do this in: https://github.com/projectcalico/calico/blob/0bfeb0f2c7e77b3cf21eaebd5996e514e28d242c/libcalico-go/lib/clientv3/ippool.go#L243-L244

As per @caseydavenport 's comment on the fix PR
This was cherry-picked to v3.24 and v3.23 - should be present in v3.23.4+

Hey i am getting this issue and with these logs
EKS Version: 1.24
Calico Node: 3.25
When i enable BPF mode i get readiness probe 503 and below warning logs and pod is stuck starting
2023-03-01 08:42:18.393 [WARNING][1132] felix/daemon.go 719: Felix is shutting down reason="config changed"
2023-03-01 08:42:19.911 [WARNING][1132] felix/health.go 266: Reporter is not ready. name="CalculationGraph"
2023-03-01 08:42:19.911 [WARNING][1132] felix/health.go 266: Reporter is not ready. name="InternalDataplaneMainLoop"
2023-03-01 08:42:19.911 [WARNING][1132] felix/health.go 228: Health: not ready
2023-03-01 08:42:20.465 [WARNING][1165] felix/config_params.go 1049: Proceeding with `RouteTableRange` config option. This field has been deprecated in favor of `RouteTableRanges`.
2023-03-01 08:42:20.501 [WARNING][1165] felix/daemon.go 719: Felix is shutting down reason="config changed"
2023-03-01 08:42:22.598 [WARNING][1184] felix/config_params.go 1049: Proceeding with `RouteTableRange` config option. This field has been deprecated in favor of `RouteTableRanges`.
2023-03-01 08:42:22.630 [WARNING][1184] felix/daemon.go 719: Felix is shutting down reason="config changed"
2023-03-01 08:42:24.698 [WARNING][1203] felix/config_params.go 1049: Proceeding with `RouteTableRange` config option. This field has been deprecated in favor of `RouteTableRanges`.
2023-03-01 08:42:24.732 [WARNING][1203] felix/daemon.go 719: Felix is shutting down reason="config changed"
2023-03-01 08:42:26.802 [WARNING][1222] felix/config_params.go 1049: Proceeding with `RouteTableRange` config option. This field has been deprecated in favor of `RouteTableRanges`.
2023-03-01 08:42:26.835 [WARNING][1222] felix/daemon.go 719: Felix is shutting down reason="config changed"
2023-03-01 08:42:28.010 [WARNING][1222] felix/health.go 266: Reporter is not ready. name="CalculationGraph"
2023-03-01 08:42:28.010 [WARNING][1222] felix/health.go 266: Reporter is not ready. name="InternalDataplaneMainLoop"
2023-03-01 08:42:28.902 [WARNING][1242] felix/config_params.go 1049: Proceeding with `RouteTableRange` config option. This field has been deprecated in favor of `RouteTableRanges`.
2023-03-01 08:42:28.957 [WARNING][1242] felix/daemon.go 719: Felix is shutting down reason="config changed"
2023-03-01 08:42:29.947 [WARNING][1242] felix/health.go 266: Reporter is not ready. name="InternalDataplaneMainLoop"
2023-03-01 08:42:29.947 [WARNING][1242] felix/health.go 266: Reporter is not ready. name="CalculationGraph"
2023-03-01 08:42:29.947 [WARNING][1242] felix/health.go 228: Health: not ready
2023-03-01 08:42:31.027 [WARNING][1276] felix/config_params.go 1049: Proceeding with `RouteTableRange` config option. This field has been deprecated in favor of `RouteTableRanges`.
2023-03-01 08:42:31.072 [WARNING][1276] felix/daemon.go 719: Felix is shutting down reason="config changed"
2023-03-01 08:42:33.141 [WARNING][1295] felix/config_params.go 1049: Proceeding with `RouteTableRange` config option. This field has been deprecated in favor of `RouteTableRanges`.
2023-03-01 08:42:33.182 [WARNING][1295] felix/daemon.go 719: Felix is shutting down reason="config changed"


[Snyk] Fix for 2 vulnerabilities

Snyk has created this PR to fix 2 vulnerabilities in the maven dependencies of this project.
Snyk changed the following file(s):

master/pom.xml

Vulnerabilities that will be fixed with an upgrade:

Issue
Score
Upgrade

Improper Handling of Case Sensitivity SNYK-JAVA-ORGSPRINGFRAMEWORK-8230368
  53
org.springframework:spring-webmvc:  5.3.18 -> 6.1.14 Major version upgrade No Known Exploit

Improper Handling of Case Sensitivity SNYK-JAVA-ORGSPRINGFRAMEWORK-8230365
  40
org.springframework:spring-beans:  5.3.18 -> 6.1.14 Major version upgrade No Known Exploit

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

:tada: Snyk checks have passed. No issues have been found so far.
:white_check_mark: code/snyk check is complete. No issues have been found. (View Details)


Icon Request: css.map

[x] I'm using Visual Studio Code Version [1.4.0] [Stable/Insider]
[x] I'm using the extension version [1.8.5]
[x] I've regarded to the FAQ and they did not help.
[x] My system is: [Mac OS Sierra]
[x] I'm sure this issue is not a duplicate?
[x] I want to create an icon request(if not, remove these lines below):
- Icon Name: [css.map]
- Sample original Icon(32x32/png/transparent whenever possible): [url]
- Extensions: only one, *.map.css

We can use the same map icon for the js.map. Just make it blue to match the css icon, and replace JS with CSS.

Do you make the icon or should I?

@robertohuertasm  implemented the js.map one in #154 , so it would be easier for him to make since he should have the template.

@iRoachie @jens1o no problem, I'll do it!


[WIP] docs: add authoring template for openshift quick starts

@secondsun @wtrocki Just giving this PR a bump. If you get a chance to review, I'd appreciate it.
Thanks!

@secondsun any reason why we did not reviewed those changes?

@wtrocki Just the WIP label. Changes look like a good, template for quickstarts. If it is ready to merge I'll merge it

@jbyrne-redhat Can you add more info why, what and remove WIP label.
@secondsun I remember that we had concerns to not include that into the official operator templates.
Would renaming it to _template fix that?

The only problem having the template here is that if someone runs oc apply -f ./olm/quickstarts the command would try to install the template. The operator supplies a list of templates to install when it runs.

The only problem having the template here is that if someone runs oc apply -f ./olm/quickstarts the command would try to install the template. The operator supplies a list of templates to install when it runs.

Ah, OK. I thought we explicitly specified a list of quick starts to be installed with the Operator and therefore this would be exempt. In that case, can I add a templates sub-dir and drop it into that?


REST API: Security - Authentication

Changes

create section or authentication part of the REST API
update example app for anonymous access

👍


*: Add custom verbose flag

It can use the code to change the value of verbose flag.

LGTM

LGTM


Using system wide xdg-open to open browser, fixes #425

Short description of what this resolves:
Browser open doesn't work on unix system because of a reference to a missing external script (vendor/xdg-open).
Changes proposed in this pull request:
This is a quick fix what uses xdg-open installed on the system. Works on Ubuntu system and probably in a lot of Linux distribution because xdg-open is quite popular.
I also added a warning message if the system command throws an error.

Fixes: #425

@marcorosi,
Thanks!
Thanks,
Dan


why USDT have more than one property id?

Many articles says that the USDT's property id is 31?
What about 192,330 and 341?
I found them on office assets page

Any id except id=31 could be just fake ones, because anyone can create his/her own token with any additional info and token name.

@vagabondan  thank you for answer

I added a wiki entry for this question:
https://github.com/OmniLayer/omnicore/wiki/Why-are-there-multiple-tokens-with-similar-names,-e.g.-TetherUS%3F

thanks


Unable to create new case from existing case data

Bug
Work Environment

Question
Answer

OS version (server)
Ubuntu 18.04 LTS

OS version (client)
Ubuntu 18.04 LTS

TheHive4py version / git hash
1.6.0

Problem Description
After retrieving a case from a Hive instance with find_cases(), the case object cannot be passed back to create another case to the create_case() method. We get this error message:
TypeError: create_case() got an unexpected keyword argument 'flag'
Looks like creating a case with a flag key is not supported
Steps to Reproduce

Create a basic case in TheHive
Run this code to retrieve the specific case from thehive4py with the case ID:

response = org_api.find_cases(query=Id(caseId), range='all', sort=[])
if response.status_code == 200:
  if len(response.json()) > 0:
    dest_api.create_case(**response.json()[0])
  else:
    app.logger.debug("No case to create, was it a deletion?")
else:
  app.logger.debug('Cannot retrieve case: {}/{}'.format(response.status_code, response.text))

Case is properly retrieved, response data is correct but the case creation fails with the error mentioned before

Possible Solutions
Allow the creation of a case with the 'flag' keyword argument
Complementary information
I haven't tested creating a case from scratch with the 'flag' keyword argument but I suppose it fails also

Code formatting doesn't work at all... Not impressed by GitHub there

Multiline code needs to be wrapped by triple backquotes. I've fixed the formatting anyways

Thanks, learnt something today.
On Mon, Apr 15, 2019 at 4:28 PM Nabil Adouani<EMAIL_ADDRESS>wrote:

Multiline code needs to be wrapped by triple backquotes. I've fixed the
formatting anyways
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/TheHive-Project/TheHive4py/issues/110#issuecomment-483157616,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ar1U9L5OxRK2jHltQyXmwDMzcITjZJSGks5vhDg7gaJpZM4cvZhN
.

--
[image: Grab] https://htmlsig.com/t/000001BKA99J
[image: Twitter] https://twitter.com/GrabMY  [image: Facebook]
https://www.facebook.com/Grab/?fref=ts  [image: LinkedIn]
https://www.linkedin.com/company/grabapp  [image: Instagram]
https://www.instagram.com/grabmy
Pierre Neihouser / Security<EMAIL_ADDRESS>/ +6592207795
Grab1 Wallich Street, Guoco Tower #35-01Singapore 078881http://www.grab.com
--
Grab is hiring. Learn more at https://grab.careers
https://grab.careers/
By communicating with Grab Inc and/or its
subsidiaries, associate companies and jointly controlled entities (“Grab
Group”), you are deemed to have consented to processing of your personal
data as set out in the Privacy Notice which can be viewed at
https://grab.com/privacy/ https://grab.com/privacy/
This email contains
confidential information and is only for the intended recipient(s). If you
are not the intended recipient(s), please do not disseminate, distribute or
copy this email and notify Grab Group immediately if you have received this
by mistake and delete this email from your system. Email transmission
cannot be guaranteed to be secure or error-free as any information therein
could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or
contain viruses. Grab Group do not accept liability for any errors or
omissions in the contents of this email arises as a result of email
transmission. All intellectual property rights in this email and
attachments therein shall remain vested in Grab Group, unless otherwise
provided by law.

I'm also getting TypeError: create_case() got an unexpected keyword argument 'description' after removing the flag. Something must be wrong with my code.

the create_case methods needs a Case object as input, not a JSON object.
Try dest_api.create_case(Case(**response.json()[0]))

Confirm, it works as expected with the Case object, sorry for the trouble


Document install instructions

[ ] docker
[ ] docker-compose
[ ] standalone

Note that pulling, even public, images from GitHub's infrastructure requires authentication. The GitHub documentation describes how to set that up.
Pull image from the command line:
docker pull docker.pkg.github.com/frankgrimm/omen/omen:latest
Use the image as a base image for other Dockerfiles.
FROM docker.pkg.github.com/frankgrimm/omen/omen:latest

The configuration is, by default, stored in a JSON formatted file called ./config.json.
Configuration
Users can override the location of the configuration file by setting the environment variable OMEN_CONFIG_FILENAME.
Users can override any configuration value by passing the respective equivalent as environment variables. Option keys can be translated to environment variables by:

Prefixing the key with OMEN_, e.g. option key => OMEN_option key.
Replacing all spaces with underscores, e.g. OMEN_option key => OMEN_option_key.
Converting the whole key to uppercase letters, e.g. OMEN_option_key => OMEN_OPTION_KEY.

Settings
table here

https://github.com/FrankGrimm/omen/issues/9#issuecomment-654093576

https://github.com/FrankGrimm/omen/blob/master/docs/omen-configuration.md


Linking large amount of files with mslink fails

Version of SCons

3.04

Version of Python

2.7 trough 3.7

Which python distribution if applicable (python.org, cygwin, anaconda, macports, brew,etc)

How you installed SCons

Nuitka inline copy of 3.0.4

What Platform are you on? (Linux/Windows and which version)

Windows with VS 2017 and earlier.

How to reproduce your issue?  Please include a small self contained reproducer. Likely a SConstruct should do for most issues.

Again no none, sorry.
We have had to make a change, because while Scons avoids one limit, namely the OS limit of command line length, the mslink parser also has a line length limit. Scons joins the files with spaces, producing a single line with, which then mslink.exe can refuse. Our change avoid this:
-        os.write(fd, bytearray(" ".join(args) + "\n",'utf-8'))
+        # Nuitka: Avoid super long lines beyond the parser limit of the MS tool.
+        os.write(fd, bytearray("\n".join(args) + "\n",'utf-8'))

With this, a single filename is put per line, and that works perfectly. No scalability limit is known on Windows with MSVC. Seen my other issue #3348 for MinGW64 issue.

Fixed in master.

Is this mslink and also msvc? (new line better than space)
Or other tools that you are aware of as well.
I'm going to default it to space and then only change it for tools which work with newlines.

Ich have used this with MSVC 2008 and earlier. Not sure now what distinction you are drawing.

mslink (link.exe) and msvc (cl.exe) are separate tools and executables.
Safe to assume that all tools on all platforms do not accept newlines in their command files.
So ideally we'd only make this change for tools which do.

Oh @bdbaddog I might be confused, but this is not at all about command line length anymore. I think there is already a tempfile employed to be passed to link.exe and the issue is with that input file of the linker.
Seems the linker parses the file line wise, and in that has a length limit for the line, which Nuitka users hit with large compilations, when all filenames are put on a single line. Separating filenames with new line in that file is working dramatically better than spaces.
Has nothing to do with cl.exe and command line at all. Or so I am thinking.
It just gives link.exe an easier time in its parser, which seems line oriented. To me, this patch we have given should be absolutely uncritical.
Yours,
Kay

I find this quote on the internet, not an official doc but no real reason to doubt:

The command file is parsed for directives line by line, for as long as lines can be read. It is a fatal error (LNK1170) for a command file to have too long a line. The present limit is 128KB.

Microsoft's own docs contain this:

In the command file, arguments can be separated by spaces or tabs (as on the command line) and by newline characters.

I find this quote on the internet, not an official doc but no real reason to doubt:

The command file is parsed for directives line by line, for as long as lines can be read. It is a fatal error (LNK1170) for a command file to have too long a line. The present limit is 128KB.

Microsoft's own docs contain this:

In the command file, arguments can be separated by spaces or tabs (as on the command line) and by newline characters.

Can you add URL's for such?
TEMPFILE is used or can be used on any platform for any tool.
So making a universal change from space to newline may not be valid.
As such making a per-tool change seems reasonable.
That's why I'm asking which tools you've tried your change with.

I added the URLs.  GNU linker scripts have their own syntax anyway, and I don't see that scons generates them: lines of the form INPUT(file1, file2, file3,...)

O.k. pull request to resolve this available.
Please take a look.
I can prepare a pre-release test pypi package if that's helpful.


Fix issue with ProgressRing not animating after being unloaded

Description

When the ProgressRing control is being unloaded, the underlying AnimatedVisualPlayer unloads its content. This results in the animation not rendering anymore.
Motivation and Context

Closes #6354
How Has This Been Tested?

Tested manually.
Screenshots (if appropriate):

/azp run


Explain how to use types

This package already includes types, need to add README section how to use it

https://on.cypress.io/typescript


Version Packages

This PR was opened by the Changesets release GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated.
Releases
<EMAIL_ADDRESS>Patch Changes

51beeae: enhanced docs

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
105

Relevant Lines:
105

💛  - Coveralls


Prevent RPCs to return sensitive data if wallet is locked

There are many RPCs like ListMarket (the very first one that I noticed, the complete list will follow) that I guess they shouldn't do anything but returning an error in case the wallet hasn't been unlocked yet.
In every of these RPCs, after validating the eventual arguments, we should retrieve the vault and check is it's still locked and return an error just in case.
I wonder if we could create a kind of middleware to avoid having repetion of code.

Wallet service RPCs are ok.
The only available RPC of the trader service is Markets but I guess this might be ok. At interface level, we should "rename" the error to something more meaningful for the trader like service not reachable or something like that instead of the chanonical wallet must be unlocked to perform this operation.
Basically none of the operator service's RPCs are ok at the moment.
This is the list of completely unsafe RPCs, ie. those that do their job data even if the wallet is locked:

ListMarket
UpdateMarketFee
UpdateMarketStrategy
ListSwaps
ReportMarketFee
ReloadUtxos
ListUtxos
DropMarket

The other ones might seem ok but just because at a certain point they use the internal vault which errors because it's still locked. This means that all the other operations made before involving the vault can potentially cause changes to other domains/repositories unexpectedly.
In general, checking the status of the vault should be the very first thing to do (before or after validating the eventual arguments?) in order to prevent unexpected behaviors later in the future.


SOAP problem?

Steps to reproduce the problem:

mitmdump --mode reverse:https://<IP_ADDRESS>:443 -p 8082 -v -w ./vsphere.log --ssl-insecure -s ./pbtest.py > shell.log
start the client application (talks to <IP_ADDRESS>) and attempt to register <IP_ADDRESS>. This is done by pointing the client to the IP:PORT where mitmdump (or mitmproxy) is running.

Any other comments? What have you tried so far?
The server at <IP_ADDRESS>:443 is a VMware vSphere server. The client registration of this server involves client use of the vSphere SDK APIs. My understanding is that these APIs talk "SOAP".
The shell.log file contains this:
Proxy server listening at http://*:8082
<IP_ADDRESS>:51319: clientconnect
::ffff:<IP_ADDRESS>:51319: serverconnect
  -> ('<IP_ADDRESS>', 443)
::ffff:<IP_ADDRESS>:51319: Establish TLS with server
::ffff:<IP_ADDRESS>:51319: Certificate verification error for <IP_ADDRESS>: unable to get local issuer certificate (errno: 20, depth: 0)
::ffff:<IP_ADDRESS>:51319: Ignoring server verification error, continuing with connection
::ffff:<IP_ADDRESS>:51319: ALPN selected by server: -
::ffff:<IP_ADDRESS>:51319: Establish TLS with client
 << Invalid header line: b'SOAPAction\r\n'
::ffff:<IP_ADDRESS>:51319: serverdisconnect
  -> ('<IP_ADDRESS>', 443)
<IP_ADDRESS>:51319: HTTP protocol error in client request: Invalid header line: b'SOAPAction\r\n'
<IP_ADDRESS>:51319: ProtocolException("HTTP protocol error in client request: Invalid header line: b'SOAPAction\\r\\n'",)
<IP_ADDRESS>:51319: clientdisconnect

System information
Mitmproxy: 4.0.3
Python:    3.6.5
OpenSSL:   OpenSSL 1.1.0h  27 Mar 2018
Platform:  Linux-2.6.32-642.11.1.el6.x86_64-x86_64-with-centos-6.9-Final
Grateful for any light you can shed on this.
-Paul

Suppose that the vSphere SOAP protocol does not return a valid HTTP response or, at least that it doesn't return HTTP headers where they are expected to be.
Is it possible to provide a script to mitmproxy or  mitmdump that tolerates this, e.g., don't process the response headers - and don't choke on them. Instead, simply return the data to the requesting client?
If so, can you provide an example of how to do this?
Thanks.

Hi,
This indeed looks like you are not dealing with HTTP traffic. You can use mitmproxy's raw tcp mode to exempt traffic from parsing (https://docs.mitmproxy.org/stable/concepts-protocols/#raw-tcp-tcp-proxy-fallback), although that limits you in what you can do.

Hi,
Thank you for getting back to me about this.
In fact, I did get this working by means of raw TCP. In the process I learned that, in fact, the initial client request in this conversation contains what purports to be an HTTP header that consists of 'SOAPAction' just as I wrote it - no colon after the term, and no value.
I may also have stumbled over the mitmproxy code that splits on the colon and complains about lack of a value.
While I suppose that this header's structure represents a protocol violation, it might not be a bad idea if you could tolerate it. Just a thought.
I will likely have some more questions for you about how to proxy an NFS server, but I'll open them in a different issue.
Thanks again!

While I suppose that this header's structure represents a protocol violation, it might not be a bad idea if you could tolerate it. Just a thought.

Yes, slippery slope unfortunately. If we accept lines without colons, many not HTTP-ish protocols may suddenly be parsed by our HTTP parser, which probably creates a bunch of cunfusing errors somewhere else. At this point I'm rather inclined to say no to this as long as it stays a rather obscure phenomenon.
Thanks for the quick feedback!


Network Block Device Implementation

Reasoning
Current booting options with our stack are limited to a large initramfs or similar. This module allows for whole block devices to be exported for read/write.
Features

Copy-on-write, page granularity

In memory
On disk

Multiple client support
Copy block device to ram for serving

Using
A complete Arch Linux system can be booted using this system. This module needs to be added to the initcpio, as instructed here.
An iPXE configuration script has been provided to show how to use the image once created.
Performance is perfectly acceptable for one client, multiple clients using the copy-on-write mechanism should see similar performance.
Issues

Locking had to be implemented when using the shared block device open file. Seek location is shared between threads, so seeks followed by reads and writes are wrapped in a single shared lock.
There are a few debug messages (disk reads/writes) which should be a higher level of debugging, which has yet to be implemented (mentioned in #59)

Future expansion

COW pages in memory or on disk could be compressed to save space in memory or on disk respectively. Overwriting pages causing compression size changes create implementation difficulties for this.

Merging
I don't want this merged until we've completed #59 and maybe #65 and have updated the master branch with those updates.
I've opened this PR for testing and code quality changes.
Comparison to NFS
This deals at a significantly lower level than NFS, we really only care about "read x bytes from y offset", or "write data x to y offset", whereas NFS is a complete filesystem. There is many orders of magnitude lower complexity to this system than NFS.
That being said, I will still attempt to complete my NFS server as they have slightly different use cases, and NFS is used for PXE booting more than NBD is.

As per our conversation, this will not be merged into development until #59 is merged and development is merged into master. Please keep in mind it may need to be rebased once #59 is merged.

This PR is also now rebased. There are argument change conflicts with #65 so another rebase will be required for whichever is merged second.
This also fixes #80, but I am open to changes. Here is the current output (missing those added in #65)

WARNING: Not root. Servers will probably fail to bind.

usage: pypxe-server.py [-h] [--ipxe] [--http] [--no-tftp] [--debug MODE_DEBUG]
                       [--config JSON_CONFIG] [--syslog SYSLOG_SERVER]
                       [--syslog-port SYSLOG_PORT] [--dhcp | --dhcp-proxy]
                       [-s DHCP_SERVER_IP] [-p DHCP_SERVER_PORT]
                       [-b DHCP_OFFER_BEGIN] [-e DHCP_OFFER_END]
                       [-n DHCP_SUBNET] [-r DHCP_ROUTER] [-d DHCP_DNS]
                       [-c DHCP_BROADCAST] [-f DHCP_FILESERVER]
                       [-a NETBOOT_DIR] [-i NETBOOT_FILE]
                       [--nbd NBD_BLOCKDEVICE] [--nbd-write] [--nbd-cow]
                       [--nbd-cowinmem] [--nbd-copytoram]
                       [--nbd-server NBD_SERVER_IP] [--nbd-port NBD_PORT]

Set options at runtime. Defaults are in pypxe-server.py

optional arguments:
  -h, --help            show this help message and exit
  --ipxe                Enable iPXE ROM (default: False)
  --http                Enable built-in HTTP server (default: False)
  --no-tftp             Disable built-in TFTP server, by default it is enabled
                        (default: True)
  --debug MODE_DEBUG    Comma Seperated (http,tftp,dhcp). Adds verbosity to
                        the selected services while they run. Use 'all' for
                        enabling debug on all services (default: )
  --config JSON_CONFIG  Configure from a JSON file rather than the command
                        line (default: )
  --syslog SYSLOG_SERVER
                        Syslog server (default: None)
  --syslog-port SYSLOG_PORT
                        Syslog server port (default: 514)
  -a NETBOOT_DIR, --netboot-dir NETBOOT_DIR
                        Local file serve directory (default: netboot)
  -i NETBOOT_FILE, --netboot-file NETBOOT_FILE
                        PXE boot file name (after iPXE if --ipxe) (default: )

DHCP:
  Arguments relevant to the DHCP server

  --dhcp                Enable built-in DHCP server (default: True)
  --dhcp-proxy          Enable built-in DHCP server in proxy mode (implies
                        --dhcp) (default: False)
  -s DHCP_SERVER_IP, --dhcp-server-ip DHCP_SERVER_IP
                        DHCP Server IP (default: <IP_ADDRESS>)
  -p DHCP_SERVER_PORT, --dhcp-server-port DHCP_SERVER_PORT
                        DHCP Server Port (default: 67)
  -b DHCP_OFFER_BEGIN, --dhcp-begin DHCP_OFFER_BEGIN
                        DHCP lease range start (default: <IP_ADDRESS>)
  -e DHCP_OFFER_END, --dhcp-end DHCP_OFFER_END
                        DHCP lease range end (default: <IP_ADDRESS>)
  -n DHCP_SUBNET, --dhcp-subnet DHCP_SUBNET
                        DHCP lease subnet (default: <IP_ADDRESS>)
  -r DHCP_ROUTER, --dhcp-router DHCP_ROUTER
                        DHCP lease router (default: <IP_ADDRESS>)
  -d DHCP_DNS, --dhcp-dns DHCP_DNS
                        DHCP lease DNS server (default: <IP_ADDRESS>)
  -c DHCP_BROADCAST, --dhcp-broadcast DHCP_BROADCAST
                        DHCP broadcast address (default: <broadcast>)
  -f DHCP_FILESERVER, --dhcp-fileserver DHCP_FILESERVER
                        DHCP fileserver IP (default: <IP_ADDRESS>)

Network Block Device:
  Arguments relevant to the NBD server

  --nbd NBD_BLOCKDEVICE
                        Enable the NDB server with a specific block device
                        (Can be a disk image) (default: )
  --nbd-write           Enable writes on the NBD device (default: False)
  --nbd-cow             Enable copy-on-write for the NBD device (Non-
                        persistent changes) (default: True)
  --nbd-cowinmem        Store copy-on-write pages in memory (default: False)
  --nbd-copytoram       Copy the NBD device to memory before serving clients
                        (default: False)
  --nbd-server NBD_SERVER_IP
                        NBD Server IP (default: <IP_ADDRESS>)
  --nbd-port NBD_PORT   NBD Server Port (default: 10809)

Now that we have the JSON config file, do we still want the non-obvious short options for DHCP?
Furthermore, do we still want all of the options in the usage:  line at the beginning of the output? We have a lot of options now and this is beginning to get cluttered.

@psychomario can you just run through the issue tracker and comment with which issues this fixes/closes? I think it closes 2-3 of them but it's missing that in the commit messages so I might have to close them manually. If you leave a list here it makes it easier later. Thanks!

Fixes: #80

@psychomario all ready to go?

@mmattioli Looks like it, yes


Update Composer bin to 1.5.2

Problem:
The composer script is actually an alias for a docker image that contains composer. Although this works, it is not necessary as we already have a PHP image in the project.
Solution:
In addition to updated the composer version, I modified the project so that the composer binary is not in version control. Instead it is installed via a command in init_project. The composer file that is installed is the normal phar, and is simply executed against the bin/php Docker image since the #!/usr/bin/env php at the beginning of the composer file will resolve to ./bin/php if ./bin is sufficiently high in the users PATH list (which it should be).

Testing file changes were not found.
Paging @fortoulrodolfo to review the back-end work in this pull request!
Paging @jose-ochoa to review the testing work in this pull request!
I couldn't recognize who these files were relevant to:
.gitignore
bin/composer
bin/init/install_composer.sh
bin/init_project

This is dependent on https://github.com/Medology/FlexibleMink/pull/123. It will not pass until that is merged.