Use mini_portile2 to build libmagic

This pull request introduces significant changes to the build process:

Use mini_portiile2 gem to build the C extension.
Clean up unneeded files after build.
Switch the download archive to pull from the S3 endpoint.
By default, build a static library of libmagic and link it.
Support disabling of static library generation via the --disable-static argument.
Support for using the system library via the --use-system-libraries argument.
Update Travis CI to use Ubuntu 18.04 (bionic) instead of 16.04 (xenial).

This pull request lifts much of the extconf.rb code from Nokogiri.

Hi @stanhu, thank you for this! This is amazing!
Before you move deeper into the proverbial "rabbit hole" (but I hope it won't be that bad, though) of trying to get this to work, I need to ask - are you sure you want to spent time on this?
An odd question to ask, but... I believe you were working on solving GitLab incident, and it also seems that the Ruby on Rails team is about to select the mini_mime Ruby Gem as their preferred solution (see https://github.com/rails/marcel/commit/5040f4875e63b09455e9ef99e5943d561d773848).
To put it differently - are you OK with helping me? I would really hate to waste your time, especially after when Ruby on Rails project fixes their problem, but in the same time I appreciate your help!
Krzysztof

To put it differently - are you OK with helping me? I would really hate to waste your time, especially after when Ruby on Rails project fixes their problem, but in the same time I appreciate your help!

@kwilczynski Of course. This gem has value, even if Rails decides to use mini_mime. It looks like the transition mini_mime is not quite done yet, and the last release was in 2019. In the meantime, I'll do what I can to push this forward.

I'm going to abandon this pull request because I'm having a hard time making mini_portile link against the compiled .so instead of the libmagic.
I figured out that statically compiling the binary is possible, but you need to use pkg-config --libs --static libmagic to get the linker flags needed for your system. For macOS, this adds -lmagic -llzma -lbz2 -lz, while on Linux it's only -l magic -lz. I couldn't find an easy way to tell mini_portile where to look for the .pc file other than to set PKG_CONFIG_PATH.

Hi @stanhu, thank you for taking a look!
I wonder whether mini_portile is doing something awkward behind the scenes, as the "vanilla", so to speak, approach of downloading the libmagic and building a shared object, setting the environment, building ruby-magic ensuring that everything is properly linked, would be (one would assume) almost the same as the steps you captured in the extconf.rb at the moment.
Krzysztof

Ok, finally got this to work. To do:

Document and test --disable-static flag
Consider adding -enable-system-libraries flag.
Add clean step

Hi @stanhu, thank you for all the work here thus far! This is amazing!
We have some failures when using system libraries as the libmagic version is quite old (version 5.25), and it does not support the MAGIC_PARAM_BYTES_MAX that was added in version 5.27.
I used to support older versions of libmagic. I will bring the code I removed back.
Sorry about this!
Krzysztof

@kwilczynski No problem! We could also just use your existing script and test this that way. I don't have a strong opinion whether backwards compatibility needs to be preserved, especially since I think we might want to vendor the libmagic source directly in this repo.

Hi @stanhu,

I don't have a strong opinion whether backwards compatibility needs to be preserved, especially since I think we might want to vendor the libmagic source directly in this repo.

This is something I am not yet sure about, to be honest, as I am not sure if storing tarballs directly here would be something we should do. I need to think about this some more. I hope you don't mind.
Krzysztof

@kwilczynski It turns out Ubuntu 16.04 (xenial) will reach end-of-life in April 2021, so I just updated .travis-ci.yml to Ubuntu 18.04 (bionic). This explains why I was able to build locally. I think this pull request is ready for review!

Wild, with --use-system-libraries:
irb(main):002:0> Magic.new
=> #<Magic:0x000055ea69401f38 @flags=0, @paths=["/etc/magic", "/usr/share/misc/magic"]>
irb(main):003:0> Magic.new.flags
=> 64

Hi @stanhu,

Wild, with --use-system-libraries:
irb(main):002:0> Magic.new
=> #<Magic:0x000055ea69401f38 @flags=0, @paths=["/etc/magic", "/usr/share/misc/magic"]>
irb(main):003:0> Magic.new.flags
=> 64

This is rather worrying, not something I would imagine to see. Let me check what is going on here.
Krzysztof

Hi @stanhu,
Seems it's not us - a good news in a way - as per:
Using the following version of libmagic:
   kwilczynski@workstation  /tmp  $ apt list libmagic1
Listing... Done
libmagic1/focal,now 1:5.38-4 amd64 [installed,automatic]
libmagic1/focal 1:5.38-4 i386

   kwilczynski@workstation  /tmp  $ cat test.c
#include <stdio.h>
#include <magic.h>

int main(int argc, char **argv)
{
	magic_t cookie = magic_open(MAGIC_NONE);

	magic_load(cookie, NULL);
	printf("magic = %s\n", magic_getpath(NULL, 0));
	printf("error = %s\n", magic_error(cookie));
	printf("version = %d\n", magic_version());
	printf("flags = %d\n", magic_getflags(cookie));

	return 0;
}
   kwilczynski@workstation  /tmp  $ gcc -Wall -Wpedantic test.c -lmagic
   kwilczynski@workstation  /tmp  $ ./a.out
magic = /etc/magic:/usr/share/misc/magic
errno = 0, error = (null)
version = 538
flags = 0

Now, when I comment out magic_load(), then we get this:
   kwilczynski@workstation  /tmp  $ gcc -Wall -Wpedantic test.c -lmagic
   kwilczynski@workstation  /tmp  $ ./a.out
magic = /etc/magic:/usr/share/misc/magic
errno = 0, error = (null)
version = 538
flags = 0

Building same version from the tarball and linking against it shows the same behaviour:
   kwilczynski@workstation  /tmp  $ gcc -Wall -Wpedantic test.c -I/tmp/magic/usr/local/include -L/tmp/magic/usr/local/lib -Wl,--rpath /tmp/magic/usr/local/lib -lmagic -o test
   kwilczynski@workstation  /tmp  $ ./test
magic = /usr/local/share/misc/magic
errno = 0, error = could not find any valid magic files!
version = 538
flags = 64

The above is very true, so let's provide a proper path to look for the Magic database:
   kwilczynski@workstation  /tmp  $ cat test.c
#include <stdio.h>
#include <magic.h>

int main(int argc, char **argv)
{
	magic_t cookie = magic_open(MAGIC_NONE);

	magic_load(cookie, "/tmp/magic/usr/local/share/misc/magic.mgc");
	//magic_load(cookie, "/usr/share/file/magic");
	//magic_load(cookie, NULL);
	printf("magic = %s\n", magic_getpath(NULL, 0));
	printf("errno = %d, error = %s\n", magic_errno(cookie), magic_error(cookie));
	printf("version = %d\n", magic_version());
	printf("flags = %d\n", magic_getflags(cookie));

	return 0;
}

   kwilczynski@workstation  /tmp  $ gcc -Wall -Wpedantic test.c -I/tmp/magic/usr/local/include -L/tmp/magic/usr/local/lib -Wl,--rpath /tmp/magic/usr/local/lib -lmagic -o test
   kwilczynski@workstation  /tmp  $ ./test
magic = /usr/local/share/misc/magic
errno = 0, error = (null)
version = 538
flags = 0

Now, the flag is set to 0, as it should be.
On macOS, the story is completely different, as it does work without any issues, as per:
   kwilczynski@rocinante  /tmp  $ cat test.c
#include <stdio.h>
#include <magic.h>

int main(int argc, char **argv)
{
	magic_t cookie = magic_open(MAGIC_NONE);

	magic_load(cookie, NULL);
	printf("magic = %s\n", magic_getpath(NULL, 0));
	printf("errno = %d, error = %s\n", magic_errno(cookie), magic_error(cookie));
	printf("version = %d\n", magic_version());
	printf("flags = %d\n", magic_getflags(cookie));

	return 0;
}
   kwilczynski@rocinante  /tmp  $ gcc -Wall -pedantic test.c -lmagic
   kwilczynski@rocinante  /tmp  $ gcc -Wall -pedantic test.c -lmagic -o test
   kwilczynski@rocinante  /tmp  $ ./test
magic = /usr/local/Cellar/libmagic/5.39/share/misc/magic
errno = 0, error = (null)
version = 539
flags = 0

Back to Linux (Ubuntu 20.04), the following is the location of the Magic database:
   kwilczynski@workstation  /tmp  $ file -v
file-5.38
magic file from /etc/magic:/usr/share/misc/magic

And having a closer look reveals:
   kwilczynski@workstation  /tmp  $ find /usr/share/misc/magic -ls
   131142      0 lrwxrwxrwx   1 root     root           13 Jan 16  2020 /usr/share/misc/magic -> ../file/magic
   kwilczynski@workstation  /tmp  $ find /usr/share/misc -ls
   135635      4 drwxr-xr-x   2 root     root         4096 Mar 27 06:09 /usr/share/misc
   131129     36 -rwxr-xr-x   1 root     root        36136 Feb 24  2018 /usr/share/misc/config.sub
   131128     44 -rwxr-xr-x   1 root     root        44283 Feb 24  2018 /usr/share/misc/config.guess
   136942      0 lrwxrwxrwx   1 root     root           25 Mar 19  2020 /usr/share/misc/usb.ids -> /var/lib/usbutils/usb.ids
   131142      0 lrwxrwxrwx   1 root     root           13 Jan 16  2020 /usr/share/misc/magic -> ../file/magic
   131141      0 lrwxrwxrwx   1 root     root           24 Jan 16  2020 /usr/share/misc/magic.mgc -> ../../lib/file/magic.mgc
   136936   1184 -rw-r--r--   1 root     root      1210291 Mar 20  2020 /usr/share/misc/pci.ids
   kwilczynski@workstation  /tmp  $ ls -l /usr/share/misc/
total 1264
-rwxr-xr-x 1 root root   44283 Feb 24  2018 config.guess
-rwxr-xr-x 1 root root   36136 Feb 24  2018 config.sub
lrwxrwxrwx 1 root root      13 Jan 16  2020 magic -> ../file/magic
lrwxrwxrwx 1 root root      24 Jan 16  2020 magic.mgc -> ../../lib/file/magic.mgc
-rw-r--r-- 1 root root 1210291 Mar 20  2020 pci.ids
lrwxrwxrwx 1 root root      25 Mar 19  2020 usb.ids -> /var/lib/usbutils/usb.ids
   kwilczynski@workstation  /tmp  $ ls -l /usr/share/file/magic
total 0
   kwilczynski@workstation  /tmp  $ ls -l /usr/share/file/
total 4
drwxr-xr-x 2 root root 4096 Jan 16  2020 magic
lrwxrwxrwx 1 root root   24 Jan 16  2020 magic.mgc -> ../../lib/file/magic.mgc

I am not sure why would "magic_load(cookie, NULL)" be failing to load the Magic database from the default location when using packages provided by the Linux distribution. However, when I install both file and libmagic form the tarball system-wide, then everything works fine:
   kwilczynski@workstation  /tmp  $ which file
/usr/bin/file
   kwilczynski@workstation  /tmp  $ file -v
file-5.38
magic file from /usr/share/misc/magic
   kwilczynski@workstation  /tmp  $ cat test.c
#include <stdio.h>
#include <magic.h>

int main(int argc, char **argv)
{
	magic_t cookie = magic_open(MAGIC_NONE);

	magic_load(cookie, NULL);
	printf("magic = %s\n", magic_getpath(NULL, 0));
	printf("errno = %d, error = %s\n", magic_errno(cookie), magic_error(cookie));
	printf("version = %d\n", magic_version());
	printf("flags = %d\n", magic_getflags(cookie));

	return 0;
}
   kwilczynski@workstation  /tmp  $ gcc -Wall -Wpedantic test.c -lmagic -o test ; ./test
magic = /usr/share/misc/magic
errno = 0, error = (null)
version = 538
flags = 0
   kwilczynski@workstation  /tmp  $ gcc -Wall -Wpedantic test.c -lmagic -o test
   kwilczynski@workstation  /tmp  $ ./test
magic = /usr/share/misc/magic
errno = 0, error = (null)
version = 538
flags = 0
   kwilczynski@workstation  /tmp  $ ldd ./test
	linux-vdso.so.1 (0x00007ffeefb62000)
	libmagic.so.1 => /lib/libmagic.so.1 (0x00007a60c2f72000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007a60c2d80000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007a60c2d64000)
	/lib64/ld-linux-x86-64.so.2 (0x00007a60c2faa000)

The above works and there are no issues - which is why I haven't caught this in testing as the CI machines used libmagic tarball when building, compiling and then installing everything prior to running tests suite.
There are definitely few things that I would not expect, for instance:

magic_getflag() returns a value in a case of an error originating after using magic_load() which is not something I would expect to have place - a potential bug?
magic_load() might not be able to either resolve this marriage of symbolic links that Debian maintainers use, or does not simply follow symbolic links at all
Debian (and Ubuntu most likely inherits everything) does something really silly with the default paths, etc.

I am adding @zoulasc for visibility and to see if he has some opinion on what is going in here.
Krzysztof

@kwilczynski It looks like https://github.com/file/file/blob/7dea9d7eede587e2b37571d9388a81d49020cd86/src/apprentice.c#L1379 is responsible for setting the flags to 64. I think we can skip this test for the flags because it seems to be an internal state: https://github.com/file/file/blob/7dea9d7eede587e2b37571d9388a81d49020cd86/src/magic.h.in#L39.
When only a single file is provided via magic_load(0, apprentice_map() in https://github.com/file/file/blob/7dea9d7eede587e2b37571d9388a81d49020cd86/src/apprentice.c#L461 appears to return a non-null value. When a search path is provided (e.g. /etc/magic:/usr/share/misc/magic, which can be provided via the MAGIC env variable), then this apprentice_load() gets called.

@kwilczynski I updated the test, but I'm not sure why Travis CI stopped working. Does something need to be done to migrate to travis-ci.com?

Hi @stanhu,

@kwilczynski I updated the test, but I'm not sure why Travis CI stopped working.

Looks like it's still running things on push to this branch, but why it's not showing here on GitHub anymore I am not sure.

Does something need to be done to migrate to travis-ci.com?

I honestly don't know. I haven't looked into it, as I considered Travis CI somewhat broken after the change of management. Having said that, it kept (the free version) working fine even with some occasional timeouts and issues here and there.
I also wonder, what would be the real difference between free plan on travis-ci.com and the one here on travis-ci.org - if the experience is the same, then I would be disappointed. Otherwise, I can't afford $70 per month to use the commercial plan :-(
But, let me have a look into switching over.
Krzysztof

Hi @stanhu, sorry for late reply!
Thank you for all the detective work here! I haven't looked at the libmagic source code at the time when I did my initial investigation. It got late for me and I called it a day. Apologies!

@kwilczynski It looks like<EMAIL_ADDRESS>is responsible for setting the flags to 64.

Good to know!
I would consider this potentially an issue to handle on your side (or generally a bug, if this is not an intended behaviour). Why? Primarily because an internal state - as you also pointed out - is leaking to the user of libmagic in a case of an error. This is a problem since libmagic does not use a copy of current flags that the magic_t object holds while doing some of its internal operations, but rather muate the current value adding 0x40 to it - which in a case of an error is never cleared. This would be a bit of a surprise to the user since the bitmask holding flags would have been updated outside of user's knowledge.

I think we can skip this test for the flags because it seems to be an internal state<EMAIL_ADDRESS>
This particular flag is also a part of what I consider to be a public interface of libmagic since these are available in magic.h header file. Thus, if possible, I would like to support it - even though, there is magic_check(), setting MAGIC_CHECK could potentially reveal issues during the time of loading and parsing of the Magic library. Having said that, most users will probably never use this flag, unless they are developing their own database or want to troubleshooting something.

When only a single file is provided to magic_load(), apprentice_map() in<EMAIL_ADDRESS>appears to return a non-null value. When a search path is provided (e.g. /etc/magic:/usr/share/misc/magic, which can be provided via the MAGIC env variable), then this apprentice_load() gets called: (file/file@7dea9d7/src/apprentice.c#L465).

Good find!
I need have a look into this a bit closer. A potential test cases would be:

magic_load() using NULL to force it to load Magic database from a default location
A single valid database passed to magic_load() directly and via the MAGIC environment variable
An invalid database provided (the exact opposite of the above case)
Multiple valid database files passed to magic_load()
A set of magic databases was provided - one valid and one invalid

I think, this unexpected (let's call it like that) behaviour we've seen when using libmagic provided by the distribution would be due to the "/etc/magic" file (usually an empty or with some comments) being included in the search path for libmagic to use. Since it's empty, it probably fails to parse and sets an error (to be verified), even though the second location provided contains a path to a valid compiled Magic database. I suppose on the "happy path" through the code (when the second valid database has been loaded successfully) a previous/existing error isn't cleared, etc.
And, of course, I found a patch from Debian that changes loading of Magic database files, as per:
local.support-local-definitions-in-etc-magic.patch
I wonder if this is what is causing this interesting behaviour. Need to check.

Alternatively, we could workaround the issue by running the tests within a known environment:
MAGIC=/usr/share/misc/magic.mgc bundle exec rake test

Sure. To unblock us I see why not (I saw you already did that, thank you!).
Krzysztof

Hi @stanhu, all of this looks amazing!
I did some testing.
Building with --use-system-libraries works fine on both macOS and Linux. I then run into problems when building vendored libmagic - not unexpected.
For example, on macOS libmagic failed to build as per:
   kwilczynski@rocinante  Ruby/ruby-magic   pr/5 ✔  $ ruby ext/magic/extconf.rb
Building ruby-magic using packaged libraries.
Static linking is enabled.
Cross build is disabled.
Using mini_portile version 2.5.0
Downloading file-5.39.tar.gz (100%)
Extracting file-5.39.tar.gz into tmp/x86_64-apple-darwin19.6.0/ports/libmagic/5.39... OK
Running 'configure' for libmagic 5.39... OK
Running 'compile' for libmagic 5.39... ERROR, review '/Users/kwilczynski/Development/Projects/Personal/Ruby/ruby-magic/tmp/x86_64-apple-darwin19.6.0/ports/libmagic/5.39/compile.log' to see what happened. Last lines are:
========================================================================
cd . && /bin/sh ./config.status config.h
config.status: creating config.h
config.status: config.h is unchanged
/Library/Developer/CommandLineTools/usr/bin/make  all-recursive
Making all in src
sed -e "s/X.YY/$(echo 5.39 | tr -d .)/" < ../src/magic.h.in > magic.h
/Library/Developer/CommandLineTools/usr/bin/make  all-am
gcc -DHAVE_CONFIG_H -I. -I..  -DMAGIC='"/Users/kwilczynski/Development/Projects/Personal/Ruby/ruby-magic/ports/x86_64-apple-darwin19.6.0/libmagic/5.39/share/misc/magic"'  -fvisibility=hidden -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith        -Wmissing-declarations -Wredundant-decls -Wnested-externs        -Wsign-compare -Wreturn-type -Wswitch -Wshadow        -Wcast-qual -Wwrite-strings -Wextra -Wunused-parameter -Wformat=2 -fPIC -c -o file.o file.c
In file included from file.c:32:
./file.h:37:10: fatal error: 'config.h' file not found
#include <config.h>
         ^~~~~~~~~~
1 error generated.
make[3]: *** [file.o] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
========================================================================

Does it build fine for you? I wonder if this is my runtime environment that is causing some issues here. I am going to build vanilla libmagic to see what is going on.
On Linux, default build (linking libmagic statically) is working fine, as per:
   kwilczynski@workstation  Ruby/ruby-magic   pr/5 ✘  $ file ./magic.so
./magic.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=42f37d7651c2a48f3c4aee23a8ead25dd7459e89, with debug_info, not stripped
   kwilczynski@workstation  Ruby/ruby-magic   pr/5 ✘  $ patchelf --print-rpath magic.so
/home/kwilczynski/.rbenv/versions/2.7.2/lib:/home/kwilczynski/Development/Projects/Personal/Ruby/ruby-magic/ports/x86_64-pc-linux-gnu/libmagic/5.39/lib
   kwilczynski@workstation  Ruby/ruby-magic   pr/5 ✘  $ patchelf --print-needed magic.so
libruby.so.2.7
libc.so.6
libz.so.1

But it seems that mkmf is having some troubles while checking for symbols when building with --disable-static, as per:
   kwilczynski@workstation  Ruby/ruby-magic   pr/5 ✘  $ ruby ext/magic/extconf.rb --disable-static
Building ruby-magic using packaged libraries.
Static linking is disabled.
Cross build is disabled.
Using mini_portile version 2.5.0
Extracting file-5.39.tar.gz into tmp/x86_64-pc-linux-gnu/ports/libmagic/5.39... OK
Running 'configure' for libmagic 5.39... OK
Running 'compile' for libmagic 5.39... OK
Running 'install' for libmagic 5.39... OK
Activating libmagic 5.39 (from /home/kwilczynski/Development/Projects/Personal/Ruby/ruby-magic/ports/x86_64-pc-linux-gnu/libmagic/5.39)...
checking for ruby.h... yes
checking for rb_thread_call_without_gvl()... yes
checking for rb_thread_blocking_region()... no
checking for magic.h... yes
checking for -lmagic... yes
checking for magic_getpath()... no

 Your version of libmagic(3) appears to be too old.

 Please, consider upgrading to at least version 5.29 or newer,
 if possible. For more information about file(1) command and
 libmagic(3) please visit the following web site:

   https://www.darwinsys.com/file

Interesting. I wonder what is mkmf doing here not to be able to detect his function.
Krzysztof

I am going to build vanilla libmagic to see what is going on.

 ✘    kwilczynski@rocinante  archives/file-5.39   pr/5 ✘  $ ./configure --disable-silent-rules --disable-dependency-tracking --enable-fsect-man5 1>/dev/null
   kwilczynski@rocinante  archives/file-5.39   pr/5 ✘  $ make
cd . && /bin/sh ./config.status config.h
config.status: creating config.h
config.status: config.h is unchanged
/Library/Developer/CommandLineTools/usr/bin/make  all-recursive
Making all in src
/Library/Developer/CommandLineTools/usr/bin/make  all-am
gcc -DHAVE_CONFIG_H -I. -I..  -DMAGIC='"/usr/local/share/misc/magic"'  -fvisibility=hidden -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith        -Wmissing-declarations -Wredundant-decls -Wnested-externs        -Wsign-compare -Wreturn-type -Wswitch -Wshadow        -Wcast-qual -Wwrite-strings -Wextra -Wunused-parameter -Wformat=2 -g -O2 -c -o file.o file.c
In file included from file.c:32:
./file.h:37:10: fatal error: 'config.h' file not found
#include <config.h>
         ^~~~~~~~~~
1 error generated.
make[3]: *** [file.o] Error 1
make[2]: *** [all] Error 2
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

That is a bit odd.
 ✘    kwilczynski@rocinante  archives/file-5.39   pr/5 ✘  $ ./config.status config.h
config.status: creating config.h
config.status: config.h is unchanged
   kwilczynski@rocinante  archives/file-5.39   pr/5 ✘  $ ls -l config*
-rwxr-xr-x  1 kwilczynski  staff   44977 Jun 15  2020 config.guess
-rw-r--r--  1 kwilczynski  staff   11526 Jun 15  2020 config.h.in
-rw-r--r--  1 kwilczynski  staff   90412 Mar 29 14:05 config.log
-rwxr-xr-x  1 kwilczynski  staff   61723 Mar 29 14:05 config.status
-rwxr-xr-x  1 kwilczynski  staff   31560 Jun 15  2020 config.sub
-rwxr-xr-x  1 kwilczynski  staff  510342 Jun 15  2020 configure
-rw-r--r--  1 kwilczynski  staff    6623 Jun 15  2020 configure.ac
   kwilczynski@rocinante  archives/file-5.39   pr/5 ✘  $ ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating magic/Makefile
config.status: creating tests/Makefile
config.status: creating doc/Makefile
config.status: creating python/Makefile
config.status: creating libmagic.pc
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
config.status: executing libtool commands
   kwilczynski@rocinante  archives/file-5.39   pr/5 ✘  $ ls -l config.h
ls: config.h: No such file or directory

Seems like autoconf script is not copying config.h for some reason.
   kwilczynski@rocinante  archives/file-5.39   pr/5 ✘  $ ./config.status config.h --debug
config.status: creating config.h
config.status: config.h is unchanged
   kwilczynski@rocinante  archives/file-5.39   pr/5 ✘  $ ls -l confAXId7R/config.h
-rw-r--r--  1 kwilczynski  staff  12128 Mar 29 14:21 confAXId7R/config.h
   kwilczynski@rocinante  archives/file-5.39   pr/5 ✘  $ ln -sf confAXId7R/config.h .
   kwilczynski@rocinante  archives/file-5.39   pr/5 ✘  $ make
/Library/Developer/CommandLineTools/usr/bin/make  all-recursive
Making all in src
sed -e "s/X.YY/$(echo 5.39 | tr -d .)/" < ../src/magic.h.in > magic.h
/Library/Developer/CommandLineTools/usr/bin/make  all-am
(...)
Making all in magic
../src/file -C -m magic
Making all in tests
make[2]: Nothing to be done for `all'.

Interesting.
Krzysztof

@kwilczynski I've found that I often have to clean up the ports/, tmp/x86_64-apple-darwin19.6.0/ports, and .o files when switching between different modes. We might need to investigate that or just run the clean step for even development environments.

Hi @stanhu,

@kwilczynski I've found that I often have to clean up the ports/, tmp/x86_64-apple-darwin19.6.0/ports, and .o files when switching between different modes. We might need to investigate that or just run the clean step for even development environments.

Ah, good to know! I've been running rake clobber just in case. This reminds me, we probably need to update .gitignore at some point too. :)
Krzysztof

I added some files to the rake clobber list. I tried building with macOS with --use-system-libraries and without it, interspersed with rake clobber, and it worked for me. I'm curious if it solves your problem.

Hi @stanhu,

@kwilczynski I updated the test, but I'm not sure why Travis CI stopped working.

Looks like it's still running things on push to this branch, but why it's not showing here on GitHub anymore I am not sure.

Does something need to be done to migrate to travis-ci.com?

I honestly don't know. I haven't looked into it, as I considered Travis CI somewhat broken after the change of management. Having said that, it kept (the free version) working fine even with some occasional timeouts and issues here and there.
I also wonder, what would be the real difference between free plan on travis-ci.com and the one here on travis-ci.org - if the experience is the same, then I would be disappointed. Otherwise, I can't afford $70 per month to use the commercial plan :-(
But, let me have a look into switching over.

We are not on travis-ci.com!
Initially, the migration from travis-ci.org to travis-ci.com didn't work. It took awhile for things to synchronise or whatever they've been doing behind the scenes.
Some things seem broken e.g., it keep asking me for authentication against GitHub every time, even though it first time everything worked. Perhaps an issue with their integration or some caching issue, not sure.
Sadly, for free plan the number of credits is quite low. I've done a build of the project, and we are down 200 credits from available 10000 already. It seems, this will run out very quickly.
Krzysztof

@kwilczynski Ok, thanks for migrating. Anything else in this pull request you would like to review?

Hi @stanhu,

@kwilczynski Ok, thanks for migrating.

No problem! Sorry for the delay.

Anything else in this pull request you would like to review?

Not that I can think of. I tested all the changes, and it works flawlessly on Linux. I still can't build it on macOS, but I believe that something is up with my runtime environment, so nothing that would stop this from being merged.
Having said that - once you merge, don't release new version just yet, as I have some local changes pending. :)
Krzysztof

Hi @stanhu,
[...]

But it seems that mkmf is having some troubles while checking for symbols when building with --disable-static, as per:
   kwilczynski@workstation  Ruby/ruby-magic   pr/5 ✘  $ ruby ext/magic/extconf.rb --disable-static
Building ruby-magic using packaged libraries.
Static linking is disabled.
Cross build is disabled.
Using mini_portile version 2.5.0
Extracting file-5.39.tar.gz into tmp/x86_64-pc-linux-gnu/ports/libmagic/5.39... OK
Running 'configure' for libmagic 5.39... OK
Running 'compile' for libmagic 5.39... OK
Running 'install' for libmagic 5.39... OK
Activating libmagic 5.39 (from /home/kwilczynski/Development/Projects/Personal/Ruby/ruby-magic/ports/x86_64-pc-linux-gnu/libmagic/5.39)...
checking for ruby.h... yes
checking for rb_thread_call_without_gvl()... yes
checking for rb_thread_blocking_region()... no
checking for magic.h... yes
checking for -lmagic... yes
checking for magic_getpath()... no

 Your version of libmagic(3) appears to be too old.

 Please, consider upgrading to at least version 5.29 or newer,
 if possible. For more information about file(1) command and
 libmagic(3) please visit the following web site:

   https://www.darwinsys.com/file

I am happy to report that this is no longer an issue with the recent changes! Thank you!
Krzysztof

Hi @stanhu,

Ok, thanks!

No. Thank YOU!
Krzysztof


Release source code.

Please release the source code for repetition. THANKS! 😊

Hello, thanks for your attention. Code will be released as soon as possible when Accepted.

Hi. Could you please leave your WeChat or email address for further communication?

It's possible to get my email address in my GitHub personal homepage :-)


Squash-merged PRs now merged instead of closed

for a longer explanation of what's going on here, see: https://cohost.org/Gabriella439/post/1650311-i-got-this-working-t

also, i should mention that i've been test-driving this on this repository: https://github.com/MercuryTechnologies/merge-test
specifically, i tested the following scenarios:

squash merging a batch of two PRs
this works correctly

adding new commits after the PR is part of a batch before it is merged
bors-ng cancels the PR and removes it from the batch if this happens


Linux port using IBM's libraries

Related to #346
This should work on macOS, iOS and Linux. I couldn't manage to run Autobahn|Testsuite on it, though. Can someone help? Also I'm not sure how to test on other platforms like tvOS and watchOS.
I'm not sure replacing SSCommonCrypto (like I did) is the best approach. Would you rather have a #if os(Linux) around the Cryptor usage, keeping the old code on the #else? Or maybe go for something a little more fancy like what was done in WSStream to allow polymorphism? This polymorphic route is a bit bumpy because it could mean dynamic dependencies on Package.json and Cartfile or coming up with a plugin mechanism involving multiple repos.
I personally like the idea of just using IBM's cross-platform libraries "everywhere" as the base while the Server Work Group (or Apple itself) doesn't come up with an official solution. For that we would have to make sure IBM's BlueCryptor (and later on BlueSocket) work on all targeted platforms.
I tried to add BlueCryptor to Cartfile, but apparently BlueCryptor does not support Carthage. Is that an impediment to using it in Starscream?

This looks promising, may have some time to test tomorrow!

this is looking promising! I'll take it for a spin tomorrow

Although I managed to get it to compile on Linux and the connection seems successful, it's unable to send or receive messages.
I solved my problem using the WebSocketClient on https://github.com/vapor/engine instead, and I don't intend on moving forward with this port, so I'm closing this PR.
If @wbarksdale or anyone else wants to continue this work, the code will be available at https://github.com/caiofbpa/starscream/tree/support-cryptor


error NotReadableError Could not start video source

The first is always ok, when i close the screen and try to reopen. it can not work any more.
Maybe it can not close the old one. at below link.
https://blog.addpipe.com/common-getusermedia-errors/
Can you check again on the close method.

It throw error NotReadableError

it through error "NotReadableError"

@chitainguyen For helping rsolving this issue, could you tell more specific information how to reproducre it.
Which OS and OS version did you use?
Which browser and browser version did you use?
Was it normal website (SPA) or PWA App?
How did you "close the screen"? Can you tell me about your component lifecycles when this error occured? How mean how did you use it.
Giving me some example code can help me find the problem sooner.
Thanks.

Dear,
Thanks for your respond,

Which OS and OS version did you use?
Answer: OS is android, version 6.0. And on iOS, it is not working too.

Which browser and browser version did you use?
Answer: browser is chrome.

Was it normal website (SPA) or PWA App?
Answer: i build with cordova android.
How did you "close the screen"? Can you tell me about your component lifecycles when this error occured? How mean how did you use it.

Open it on first time -> it work ok
Turn off screen OR switch to another app OR device idle for the couple of time.
Switch to app again. -> open QR reader -> throw this error

Giving me some example code can help me find the problem sooner.
My Code:
<QrReader
delay={delay}
onError={this.handleError}
onScan={this.handleScan}
style={{width: '100%'}}
/>

What information can i supply for this case, is there any result?.
Thanks alot

Sorry I haven't had any time to look into Cordova. I think this is Cordova specific problem. I'll try to find time to look into this. Example codes and PRs are also welcome.


account for OpenShift 4.x on RHEL 7

Discovered this was possible when viewing the output of "generate-genesis-dumps" in CI

E2E failure unrelated: https://github.com/stackrox/scanner/pull/1105


Added Builder interface to generated code to have a common type for Builders in the OpenLineage client

Problem
Working on the implementation outlined in https://github.com/OpenLineage/OpenLineage/pull/437 , it's difficult to reference InputDatasetBuilders and OutputDatasetBuilders in an abstract way, since they have no common ancestor. Adding a Builder<T> interface to the generated code means that those builders now implement Builder<InputDataset> and Builder<OutputDataset> respectively, which makes it easier to write code that deals with both of those in a generic way.
The generated code for OutputDatasetBuilder now looks like the following
  public static final class OutputDatasetBuilder implements Builder<OutputDataset> {
    private String namespace;

    private String name;

    private DatasetFacets facets;

    private OutputDatasetOutputFacets outputFacets;

    /**
     * @param namespace The namespace containing that dataset
     * @return this
     */
    public OutputDatasetBuilder namespace(String namespace) {
      this.namespace = namespace;
      return this;
    }

    /**
     * @param name The unique name for that dataset within that namespace
     * @return this
     */
    public OutputDatasetBuilder name(String name) {
      this.name = name;
      return this;
    }

    /**
     * @param facets The facets for this dataset
     * @return this
     */
    public OutputDatasetBuilder facets(DatasetFacets facets) {
      this.facets = facets;
      return this;
    }

    /**
     * @param outputFacets The output facets for this dataset
     * @return this
     */
    public OutputDatasetBuilder outputFacets(OutputDatasetOutputFacets outputFacets) {
      this.outputFacets = outputFacets;
      return this;
    }

    public OutputDataset build() {
      OutputDataset __result = new OutputDataset(namespace, name, facets, outputFacets);
      return __result;
    }
  }

Note: All schema changes require discussion. Please link the issue for context.

[ ] Your change modifies the core OpenLineage model
[ ] Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).
Checklist

[x] You've signed-off your work
[x] Your pull request title follows our guidelines
[ ] Your changes are accompanied by tests (if relevant)
[x] Your change contains a small diff and is self-contained
[ ] You've updated any relevant documentation (if relevant)
[ ] You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
[ ] You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)

Codecov Report

Merging #443 (57a2772) into main (380e1ee) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #443   +/-   ##
=======================================
  Coverage   84.17%   84.17%
=======================================
  Files          33       33
  Lines        1984     1984
=======================================
  Hits         1670     1670
  Misses        314      314

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 380e1ee...57a2772. Read the comment docs.


Compose protocol extensions

IRCv3.2 base, add extensions on top of it, make extensions optional.

Framework is done, a CAP manager enables and disables extensions (which register and unregister handlers).


Add Tracing v2 schema and spec

This PR adds a tracing v2 interface specification. V1 is skipped as a v1 libapi already exists and contains no changes in schema or spec, but some backwards-incompatible API changes with respect to v0.
Reference implementation at:
https://github.com/canonical/tempo-k8s-operator/pull/65

tandem PR: https://github.com/canonical/tempo-k8s-operator/pull/71
as soon as that closes, we can un-WIP this one, remove the tempo branches from charms.yaml and proceed.

tandem PR: canonical/tempo-k8s-operator#71 as soon as that closes, we can un-WIP this one, remove the tempo branches from charms.yaml and proceed.

This was merged, can we rebase? It is hard to review as is

tandem PR: canonical/tempo-k8s-operator#71 as soon as that closes, we can un-WIP this one, remove the tempo branches from charms.yaml and proceed.

This was merged, can we rebase? It is hard to review as is

@PietroPasotti
This PR still seem to contain 70+ file changes, can we only have the tracing content here?

changes, can we only have the tracing

The large diff is not due to a missing rebase, but to the fact that I bumped pydantic to v2 and had to make a number of adjustments to the models due to model now being a protected field.
I needed to bump to v2 because tempo uses v2 and it would have been impossible to test otherwise.


Consider broadening ark service account permissions to cluster admin

The service account we use is currently allowed to list/watch/create all resources in all namespaces. While this is sufficient for all backup operations, it is only partially so for restore operations.
While most restore operations will succeed, attempting to restore roles and/or clusterroles that have greater privileges than the ark service account fail.
We might want to broaden the privileges to cluster-admin. WDYT @mattmoyer @jbeda @skriss?

xref #23

Would this be necessary if #18 were addressed?  As I understood that, the goal was to have Ark simply impersonate the credentials of the given user.
In the short term I do think that it would be more straightforward to increase the scope of to ClusterAdmin so that the tool works as the user would expect.  One of the unfortunate things about the current model would be that an operator would not know it wasn't working properly until they were trying to test the restore process itself and a few of theses services may not be immediately apparent that they were not restored properly.

I think as part of #18 we will need to revisit this, but for now, it's a short-term win.


Incorrect warning in "Working with files" documentation

As discussed on the forum, there are a couple warnings in the "Working with files" section of the doc, about using new File() with relative paths.
In Single files and directories:

Never use new File(relative path) because this creates a path relative to the current working directory (CWD)

In File collections:

Although the files() method accepts File instances, never use new File(relative path) with it because this creates a path relative to the current working directory (CWD)

Given that the code snippets showcasing file() and files() both include a call to those functions with a new File(relativePath) as argument, I believe those warnings should be rephrased to include that it's ok to pass File objects with relative paths to those functions, and related functions like from().

Improvement will be released in Gradle 7.6


deploy command fails consistently

I've played around a bit with this tool, but reverted back to manual/CFN deployment options as the tool fails with
Invalid type for parameter CatalogName, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
However, I came back to this tool, due to the Compute Optimiser Dashboard only being available via cid-cmd.
See below for error log
[cloudshell-user@ip-10-0-20-208 ~]$ cid-cmd -vv deploy
CLOUD INTELLIGENCE DASHBOARDS (CID) CLI 0.1.30 Beta

Logging level set to: DEBUG
Loading plugins...
        Core loaded

Checking AWS environment...
        profile name: default
        accountId: <my account id>
        AWS userId: <my user id>
        Region: eu-west-1

? [dashboard-id] Please select dashboard to install: [compute-optimizer-dashboard] Compute Optimizer Dashboard
Discovering deployed dashboards...  [####################################]  100%  "Trusted Advisor Organizational View" (ta-organiza

Required datasets:
 - compute_optimizer_all_options

Looking by DataSetId defined in template...complete

There are still 1 datasets missing: compute_optimizer_all_options
Creating dataset: compute_optimizer_all_options
Traceback (most recent call last):
  File "/home/cloudshell-user/.local/bin/cid-cmd", line 8, in <module>
    sys.exit(main())
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/cid/cli.py", line 25, in wrapper
    res = func(ctx, **kwargs)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/cid/cli.py", line 102, in deploy
    ctx.obj.deploy(**kwargs)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/cid/common.py", line 261, in deploy
    self.create_datasets(required_datasets, dashboard_datasets, recursive=recursive, update=update)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/cid/common.py", line 824, in create_datasets
    if self.create_or_update_dataset(dataset_definition, dataset_id, recursive=recursive, update=update):
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/cid/common.py", line 925, in create_or_update_dataset
    self.athena.discover_views(required_views)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/cid/helpers/athena.py", line 349, in discover_views
    self.get_table_metadata(TableName=view_name)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/cid/helpers/athena.py", line 179, in get_table_metadata
    'DatabaseName': self.DatabaseName,
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/cid/helpers/athena.py", line 74, in DatabaseName
    athena_databases = self.list_databases()
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/cid/helpers/athena.py", line 137, in list_databases
    return self.client.list_databases(CatalogName=self.CatalogName).get('DatabaseList')
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/botocore/client.py", line 508, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/botocore/client.py", line 875, in _make_api_call
    api_params, operation_model, context=request_context
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/botocore/client.py", line 936, in _convert_to_request_dict
    api_params, operation_model
  File "/home/cloudshell-user/.local/lib/python3.7/site-packages/botocore/validate.py", line 381, in serialize_to_request
    raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Invalid type for parameter CatalogName, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
[cloudshell-user@ip-10-0-20-208 ~]$

Hi
Yes, output shows it is there
    "DataCatalogsSummary": [
        {
            "CatalogName": "AwsDataCatalog",
            "Type": "GLUE"
        },
        {
            "CatalogName": "aws-config-snapshot",
            "Type": "GLUE"
        }
    ]
}

I also have run through the data collection lab, so that should be fine.  At least I can see a rightsizing/ folder in the costoptimizationdata bucket

There was a bug when we had multiple catalogs. Thanks for raising this


Fix MySQL parsing of GRANT, REVOKE, and CREATE VIEW

Support MySQL style 'user'@'host' syntax.
Support wildcards on grant objects: *.*.
Make CASCADE/RESTRICT in revoke statements optional since MySQL doesn't accept it.
Support optional MySQL parameters for create view: algorithm, definer (requires user@host syntax), and security context.

Would appreciate feedback on the naming the MySQL-specific params types (would a generic/non-branded name be better?) and the API change for wildcard support (didn't want to change all 107 parse_object_name callsites for an option that is so niche, but happy to be convinced otherwise).

@mvzink it seems there's a couple conflicts in the branch could you take a look when you have some time?


fix: move watcher to node watch

to run the executable of the cli now you can do
bun build src/index.ts --compile --outfile scalar
./scalar <commands>

That’s probably better! Just want to mention that we had node watch and it was really, really slow (like 3s delay) in Node. I don’t know enough about bun: Do we switch to bun completely to publish the CLI? If so, let’s go! Otherwise, we might need to check alternative solutions or use both approaches.


chore: more information for exceed timeout error

fix #839

You know I'm deeply afraid of code changes that have no test coverage :) Can you make sure this code is covered by a unit test?

Sure, @alexander-fenster added a unit test to verify the error message. Based on the information from #839, add function name, total time it takes, and retry settings to the message.

Shoud it say "fix #741" instead of "fix #839"? :)
If it's for #741 then in its current form it does not help unfortunately, because what the user specifically asked for is

I tried to debug which API call and which library emit this error message, but I couldn't because the error message and stack trace don't provide any information other than 'Retry total timeout exceeded before any response was received'.

We still don't print which API is being called...

pass the apiName to the createApiCall and print out the useful information from the error message. This also requires setting api name from micro-generator side.


Bug in "Average Elemental Stress on a given volume"

Before submitting the issue

[X] I have checked for Compatibility issues
[X] I have searched among the existing issues
[X] I am using a Python virtual environment

Description of the bug
In the example Average Elemental Stress on a given volume, the values_to_sum_field are correctly visualized in the first plot.
However, the second and third plots (regarding divide and out variables, respectively) are not correctly displayed.
Note that I tried to isolate the last plot and create a python script that recreates only the third plot. When doing that, I was able to plot the results of the operator volume_stress.
Steps To Reproduce
Follow the example.
Which Operating System are you using?
Windows
Which Python version are you using?
3.10
Installed packages
aiohttp==3.8.3
aiosignal==1.2.0
-e git+https://github.com/pyansys/pydpf-core@3977c3952a6d67ceb67390fe7ea7a862b8389c6c#egg=ansys_dpf_core
ansys-dpf-gate==0.2.3
ansys-dpf-gatebin==0.2.3
ansys-grpc-dpf==0.6.1
appdirs==1.4.4
async-timeout==4.0.2
attrs==22.1.0
cachetools==5.2.0
certifi==2022.9.24
charset-normalizer==2.1.1
colorama==0.4.5
contourpy==1.0.5
cycler==0.11.0
fonttools==4.37.4
frozenlist==1.3.1
google-api-core==2.10.2
google-api-python-client==2.64.0
google-auth==2.12.0
google-auth-httplib2==0.1.0
googleapis-common-protos==1.56.4
grpcio==1.49.1
httplib2==0.20.4
idna==3.4
imageio==2.22.1
kiwisolver==1.4.4
matplotlib==3.6.1
multidict==6.0.2
numpy==1.23.4
packaging==21.3
Pillow==9.2.0
protobuf==4.21.7
psutil==5.9.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.9
python-dateutil==2.8.2
pyvista==0.36.1
requests==2.28.1
rsa==4.9
scooby==0.6.0
six==1.16.0
tqdm==4.64.1
uritemplate==4.1.1
urllib3==1.26.12
vtk==9.2.2
wslink==1.8.4
yarl==1.8.1

Hi @PProfizi,
I just realized that now the 2nd and 3rd figures are properly displayed in the example.
However, the first figure has changed now. When I run the example locally, I get:

as opposed to what it is displayed in the example

Hi @GuillemBarroso, I've released 0.6.2 yesterday, so the documentation has been updated.
No idea why that would have changed something here, yet if the third plot is right then that is good news as it is the result of an operator.
If the first is wrong, then it might be a problem with the script.

Hi @PProfizi,
Yes, the third plot looks fine. The operator shouldn't be the problem since when I isolated the last part of the example, I could visualize the results correctly (see issue description). It looks like the release fixed the visualization of the last two plots.
I will keep going through all examples in detail to see if I realize what's the issue with the first plot. Please, let me know if you find something on your side. Thanks!

@PProfizi Why is there a factor of 10 when 'volume_stress' operator is used?

Hi @rlagha, I see you are the one who implemented this operator, can you tell us more about its behavior?
I can also see it's been switched to private in March 2023 for some reason... if private, then it should not appear in an example.
@cbellot000 I see you are the one who added the paragraph concerning the private operator in the example, maybe we should switch it to public?


Fence weapon durability discrepancy

General information:
Project issue is for:
Server - Build b6ff89
Expected result:
Fence selling weapons with 90-ish max durability and around 40-50 durability
Actual result:
Fence weapons are less than 50 max durability and very low durability
Fence weapon durability is much lower than what is found in live
How to reproduce:

Go to fence trader screen
Pick any weapon(s) and inspect
Notice very low durability

Log files:
I'm not sure if logs are necessary.
Screenshots:
3 random weapons from live fence

3 weapons from spt

Attachments

image.png (701 KiB)
image.png (924 KiB)

Originally written by CrocodileJonesy

Should be massively improved with: https://dev.sp-tarkov.com/SPT-AKI/Server/commit/973b74bd98e9c01c87a47b9454177f0066305752

Originally written by chomp

Max weapon durability for all weapons on fence have been 85-99 for all weapons for 3 refreshes. The weapon's 'current' durability is still wildly between 2 & 90 but max is looking accurate now. (you can just repair the crap current durability ones)

Originally written by acidphantasm

Improved with: https://dev.sp-tarkov.com/SPT-AKI/Server/commit/f70bee2fd2e44091367de7fb50562229e67726fc

Originally written by chomp

Fence now has a good mixture of 40-100 current durability weapons! Should be good to classify as fixed.

Originally written by acidphantasm


HADOOP-17365. Contract test for renaming over existing file is too lenient

What changes were proposed in this pull request?
Add some assertions to make sure the result of rename() over existing file
more closely matches expectations defined by contract options
rename-returns-false-if-dest-exists and rename-overwrites-dest.
https://issues.apache.org/jira/browse/HADOOP-17365
How was this patch tested?
Ran related contract tests in Hadoop project:
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.025 s - in org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractRename
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.993 s - in org.apache.hadoop.fs.contract.localfs.TestLocalFSContractRename
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.654 s - in org.apache.hadoop.fs.contract.hdfs.TestHDFSContractRename
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.14 s - in org.apache.hadoop.fs.contract.router.TestRouterHDFSContractRename
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.295 s - in org.apache.hadoop.fs.contract.router.web.TestRouterWebHDFSContractRename

(also some others for S3A and Azure, but tests in those were skipped).
Also ran Ozone rename contract test with various combinations of these two contract options after overriding testRenameFileOverExistingFile().

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Logfile
Comment

+0 :ok:
reexec
0m 33s

Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s

No case conflicting files found.

+1 :green_heart:
@author
0m  0s

The patch does not contain any @author tags.

+1 :green_heart:

0m  0s
test4tests
The patch appears to include 1 new or modified test files.

_ trunk Compile Tests _

+1 :green_heart:
mvninstall
37m 29s

trunk passed

+1 :green_heart:
compile
21m 59s

trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
compile
17m 14s

trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

+1 :green_heart:
checkstyle
0m 52s

trunk passed

+1 :green_heart:
mvnsite
1m 31s

trunk passed

+1 :green_heart:
shadedclient
17m 41s

branch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m  4s

trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
1m 37s

trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

+0 :ok:
spotbugs
2m 19s

Used deprecated FindBugs config; considering switching to SpotBugs.

+1 :green_heart:
findbugs
2m 16s

trunk passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
0m 55s

the patch passed

+1 :green_heart:
compile
19m 18s

the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javac
19m 18s

the patch passed

+1 :green_heart:
compile
17m  9s

the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

+1 :green_heart:
javac
17m  9s

the patch passed

-0 :warning:
checkstyle
0m 51s
/diff-checkstyle-hadoop-common-project_hadoop-common.txt
hadoop-common-project/hadoop-common: The patch generated 1 new + 5 unchanged - 2 fixed = 6 total (was 7)

+1 :green_heart:
mvnsite
1m 31s

the patch passed

+1 :green_heart:
whitespace
0m  1s

The patch has no whitespace issues.

+1 :green_heart:
shadedclient
14m 51s

patch has no errors when building and testing our client artifacts.

+1 :green_heart:
javadoc
1m  1s

the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS>

+1 :green_heart:
javadoc
1m 36s

the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

+1 :green_heart:
findbugs
2m 23s

the patch passed

_ Other Tests _

+1 :green_heart:
unit
9m 50s

hadoop-common in the patch passed.

+1 :green_heart:
asflicense
0m 56s

The patch does not generate ASF License warnings.

175m 12s

Subsystem
Report/Notes

Docker
ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2447/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hadoop/pull/2447

Optional Tests
dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle

uname
Linux c5ae75eea207 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/bin/hadoop.sh

git revision
trunk / ae7b00a9988

Default Java
Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

Multi-JDK versions
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu<IP_ADDRESS> /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10

Test Results
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2447/1/testReport/

Max. process+thread count
3113 (vs. ulimit of 5500)

modules
C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common

Console output
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2447/1/console

versions
git=2.17.1 maven=3.6.0 findbugs=4.1.3

Powered by
Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

makes sense.
you couldn't run the s3a/abfs versions of these tests could you?

Thanks @steveloughran for taking a look.

you couldn't run the s3a/abfs versions of these tests could you?

Those were skipped by Maven, due to missing auth-keys.xml I guess.  Is there a way to run them locally without real cloud credentials?

Is there a way to run them locally without real cloud credentials?

Afraid not, except in the special case you have a minio s3 server.
If you don't have them, I'll merge them, and if something does fail then we'll decide whether to revert or address the failure

Thanks @steveloughran for reviewing and committing it.

I managed to run ITestS3AContractRename against Ozone S3 and got the following failure:
expected rename(s3a://rename/test/source-256.txt, s3a://rename/test/dest-512.txt) to be rejected with exception, but got false

Inspecting S3AFileSystem, it seems rename file over existing file is intended to return false:
https://github.com/apache/hadoop/blob/fc961b63d145ef17fb45fa8a2bcb1c3318b8bff7/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1383-L1386
and
https://github.com/apache/hadoop/blob/fc961b63d145ef17fb45fa8a2bcb1c3318b8bff7/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1452-L1458
I think it should be declared in the contract xml.

Yeah, I'd just seen that breakage too.
[ERROR] Tests run: 22, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 49.852 s <<< FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename
[ERROR] testRenameFileOverExistingFile[auth=false](org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)  Time elapsed: 1.58 s  <<< FAILURE!
java.lang.AssertionError: expected rename(s3a://stevel-ireland/fork-0004/test/source-256.txt, s3a://stevel-ireland/fork-0004/test/dest-512.txt) to be rejected with exception, but got false
	at org.junit.Assert.fail(Assert.java:88)
	at org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileOverExistingFile(AbstractContractRenameTest.java:131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

[ERROR] testRenameFileOverExistingFile[auth=true](org.apache.hadoop.fs.contract.s3a.ITestS3AContractRename)  Time elapsed: 1.457 s  <<< FAILURE!
java.lang.AssertionError: expected rename(s3a://stevel-ireland/fork-0004/test/source-256.txt, s3a://stevel-ireland/fork-0004/test/dest-512.txt) to be rejected with exception, but got false
	at org.junit.Assert.fail(Assert.java:88)
	at org.apache.hadoop.fs.contract.AbstractContractRenameTest.testRenameFileOverExistingFile(AbstractContractRenameTest.java:131)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)

Created https://issues.apache.org/jira/browse/HADOOP-17376.

ok, so the FS contract says rename-over-file should be rejected. S3A doesn't do that, but then we have to worry about delayed consistency, where a newly deleted file may still be found with HEAD, so we have to go with allowing the rename to succeed. Pity

yes, contract.xml should be fixable. If you could supply a patch with the option that'd be great
consider the fact that you've caught where s3a is wrong to be a success of the test case


Integrate AWS IAM with Kubernetes authentication

What type of PR is this?
/kind feature
What this PR does / why we need it
Integrate AWS IAM with Kubernetes authentication.
Which issue(s) this PR fixes
Fixes #77
Special notes for your reviewer
Does this PR introduce a user-facing change?

Integrated AWS IAM with Kubernetes Cluster

Allow autentication to whole AWS account
Result: Inconclusive
Description and results: We proceed to grant authentication permission using Pulumi. to a whole AWS account but I want to know the steps a normal user should do to generate kubeconfig and login from scratch
Corrective action: Need more info from @pasqualet. Please, explain steps to test correctly this functionality

The user can generate or not the kubeconfig using the AWS CLI only if he has permission to do it. In case he doesn't have permission in AWS, it must use a pre-generated kubeconfig.

Allow autentication to whole AWS account
Result: Partially OK
Description and results: We created a role in AWS and proceed to grant authentication permission using Pulumi. We update kubeconfig using aws eks update-kubeconfig --name --role-arn and loging to cluster output expected message Error from server (Forbidden): nodes is forbidden: User "kaka-role" cannot list resource "nodes" in API group "" at the cluster scope.
The action of generating kubeconfig with aws depend on user used.
Corrective action: Need more info from @pasqualet. Please, explain steps to test correctly this functionality and the scope of the roles (e.g if this feature allow authentication for all accounts or only one). Moreover, how a normal user would proceed to generate kubeconfig.

Again if the user doesn't have permission to generate the kubeconfig, he must use a pre-generated kubeconfig.


Item not released NAIP over Time

id:c725e126a53b4fbab72157dd77c0c29d
Not released but used in content from National Security Demo.
@kerryrobinson FYI..

This or any other time-enabled map will do. This was the best one I could find at the time.

I don't see any image change when running hitting the play button.   This app is part of "What is Web GIS?" on "Operational Data" in the "Time Aware Data - Imagery over Time" section.
Running straight out of NSEC Portal does not seem to work either?
Tested with Chrome and FF.
Portal Web App

@pshowalter The NAIP Over Time app was working last week, although it took some time to refresh.  It does not seem to be working now so I'm not sure what has changed.
Do we know of any other time-enabled map that is released on the Ops Server?

@NatalieCampos Just republished this service. We must have lost the time enabled properties.
I will review a fix. No need to replace this.

This app is now functioning, however the web map that the app is using is not released.
Map = id:43774ea0de014514917b2167691b494e
Service = ca7d6030cb3e4f7d9131c661d432c0c8
Assigned to @nfeuerstein to release the app and the service

The web map, the app, and the image service are now shared with Demo Review:
App =c725e126a53b4fbab72157dd77c0c29d
Map = 43774ea0de014514917b2167691b494e
Service = ca7d6030cb3e4f7d9131c661d432c0c8

Access and Use Constraints are misleading.  It states "none"; however the link takes you to the USDA disclaimer that states "The Originator asks to be credited in derived products". USDA is cited in the credits. Should the Access and Use Constraints contain this statement, "The information presented in this template or demonstration is based on standard Esri Commercial off-the-Shelf (COTS) technologies and commercial data. All scenario data is notional." instead of "None"?  @karlterrey What do you advise?

@lfunkhouser @nfeuerstein Credit the sources in the Description with a text notation of where the data came from.
In Access and Use Constraints say... "The information presented in this template or demonstration is based on standard Esri Commercial off-the-Shelf (COTS) technologies and commercial data. Any scenario data is notional".
Keep the Credits as they are.
We have the NAIP permission on Box already.

Assigning to @karlterrey to update access and use constraints

@lfunkhouser Updated Access and Use Constraints and Description.

All items released by DemoIntelligence

Verified.


Mark disk wwn and nvram arguments as computed

These changes are meant to keep the libvirt provider idempotent, i.e. running terraform apply twice with no changes in between should not result in any updated resources.
The disk WWN is randomly generated if it is not specified, which means that the disk.wwn argument will always be different when it is regenerated. This difference causes Terraform to perform an in-place update.
When specifying the firmware argument, libvirt will automatically generate the corresponding NVRAM definitions. Since the generated definitions differ from the unspecified nvram argument in the resource and the nvram argument is marked as force new, Terraform will replace the entire libvirt domain.
The nvram.file argument was required, but this is not necessary for libvirt. Libvirt will autogenerate the location of the NVRAM variables file even if only the template is specified.

Thanks for the contribution.


chore(mount): simplify mount operation

Found this approach in main @vuejs/core repo
https://github.com/vuejs/core/blob/main/packages/runtime-core/__tests__/rendererTemplateRef.spec.ts#L23
I think this might be a good approach to simplify our codebase too!

@cexbrayat that was fast, I've even was unable to push another version (full one), since my internet connection got flaky (I'm commuting between two cities via train :train: )
Now this one is a final state :)


Fixes stability issues with bonsai, session start/stop, and two minor issues

Pull Request instructions:

Please follow the update protocol
Answer the questions below in detail. Your responses will be emailed to experimenters.
If the experimenters must do anything new, provide detailed step by step instructions on the wiki
Use markdown syntax in order for your comments to be rendered reliably in the email: "1." instead of "1)", use four spaces for indents.
If you use the keyword "skip email" in the title, it will skip the email updates
Merges from "production_testing" into "main" should use the keyword "production merge" in the title for reliable indexing of updates

Describe changes:

Changes relative import statements in Foraging.py and Dialogs.py to absolute import statements
Adds logging information about session start/stop and trial start/stop
When Bonsai reconnects, and trials have been generated already. The user is warned to start a new session. Otherwise bonsai hangs in a loop.
When trials start, either at the start of a new session, or after a pause, we check if the bonsai connection is still active. If not, we stop trials, and warn the user. We specifically check for a ConnectionAbortedError.
Catches errors with trial initiation and alerts the user
Alerts the user if it has been 5 minutes since the last trial started (we can adjust that duration). The use case here is bonsai disconnects, or hangs, or crashes. Or the trial generation loop gets stuck in an infinite loop. The user is prompted to either keep waiting, or stop the trials.
Changes the Box with console.bat files to stay open after the GUI closes. This is valuable if the GUI crashes and error information is displayed
Fixes a bug with "Tower" information being requested in the NWB saving step
Sets the title for the black console window with the current box name

What issues or discussions does this update address?

resolves #45
resolves #167
resolves #166
resolves #168

Describe the expected change in behavior from the perspective of the experimenter

The GUI should be more stable if Bonsai crashes.
If Bonsai crashes when trials are not running, click "Control" then "reconnect Bonsai", then "new session"
If Bonsai crashes when trials are running, a timeout window will popup and let you force stop the session. Then you can save the session, and then reconnect to bonsai with "Control", then "reconnect Bonsai"

Describe the outcome of testing this update on a rig in 447

[x] tested in 447
[x] Updated the startup .bat files in 447

@XX-Yin This is ready for you to review again, I've made a lot of changes.

@XX-Yin This is ready for you to review again, I've made a lot of changes.
One question for you: What should the time-out duration be? Right now I set it to 5 minutes without the trial ending.

5m is good. We will not have trial longer than that

@XX-Yin identified a bug in my code where if the user hits the start/stop button many times the "StopCurrentSession" process will eat up memory the GUI will crash. To be clear this will only happen if the user is spamming the button 100+ times. We stopped this by only triggering the "StopCurrentSession" process when we want to start a new session.
@XX-Yin I tested our update, and everything looks good. If you approve I will merge this in after the weekly production merge


In macos visual selection is not happening when doing v  and .

https://user-images.githubusercontent.com/18016032/127732657-90ae81e1-3cde-49b6-81a4-ddd964e8e2d3.mp4

@shekhars094 How did you resolve this?

Also occured in windows.

Will be fixed soon. Fixes are in master but have a bug.


Pretrained models for car NeRFs

Hi, thanks for your great work. I downloaded pretrained models and run render scripts following README, but it outputs:

Where can I find pretrained models for car NeRFs and latents in the below?

Looking forward to your reply.

Hi! @yanjh97
Thanks for your reply! You can find the download link in README.md.
https://drive.google.com/drive/folders/1E4YjMwkDbRsF4Hb1UK0iBDkz-tFVx3Me


35% smaller wikipedia icon

It's a new icon based on an official wiki logo. Because it's look different I'm using pull request instead of direct push so you can approve this changes.

I think this is the official one - https://commons.wikimedia.org/wiki/File:Wikipedia's_W.svg
Looks more like the original to me.

I've used this: https://commons.wikimedia.org/wiki/File:Wikipedia-W-bold-in-square.svg

Interesting. They seem to use the thin version for their mobile apps.
https://play.google.com/store/apps/details?id=org.wikipedia&hl=en_GB
https://en.m.wikipedia.org/wiki/List_of_Wikipedia_mobile_applications
Let's keep the styling thin for now.

Looks like this image should be used as a reference: https://upload.wikimedia.org/wikipedia/commons/b/b3/Wikipedia-logo-v2-en.svg

I'm going to China for a week, so won't be able to access GitHub.


Typo in repo/include/ example

Fix typo in Json example
Consistency of comma use.

Thanks for the fix, actually aptly is using relaxed JSON here to allow comments, but first comma was a mistake.


Correct class naming for the inspector

Doing trick like proposed we are passing correct names to the inspector.
What is not good, I don't know how to check inspector behaviour correctly. Really I've checked it in the browser manually, so you might wish to improve the testcases.

This seems intriguing.  I'd like to see some performance numbers on it though, since it introduces a string eval, and I'm not sure how that plays with VMs

The string eval may take some performance penalty, but I think it should not affect the production code noticable in this case, because this penalty will aply only to class (metaclass object) creation code, while the instance creation code should not be affected.

I believe that's true too, but I would like to see it tested and verified.


Can't extract content from huffington post (?)

Hi all,
Got into trouble today when I tried to extract the content from some huffington posts.
Here are two URLs that don't receive a 'cleaned_text' after the extraction :

http://www.huffingtonpost.com/ryan-scott/the-best-volunteer-progra_b_7566478.html?utm_hp_ref=small-business&ir=Small+Business
http://www.huffingtonpost.com/steve-mariotti/how-to-get-from-here-to-t_b_7556636.html

I'll try to understand where it comes from, but no time at the moment.

Sorry, it's a duplicate of #231

I got a look at #224 (Goose fails on NYTimes) and applied the patch. But it doesn't work yet for Huff.Post.

Hi, I'm not sure if you're still having this problem, but I was able to resolve it using the following fix:
        from goose import Goose
        from cookielib import CookieJar
        import zlib
            ...
              other imports and code
            ...
        g = Goose({'enable_image_fetching':False}) # Optionally disable image fetching
        yummy = CookieJar()
        cookieSesh = urllib2.build_opener(urllib2.HTTPCookieProcessor(yummy))
        raw_html = cookieSesh.open(url).read() # HuffPo and several other sites return compressed text.
        raw_html = zlib.decompress(raw_html, 16 + zlib.MAX_WBITS) # Uncompress it
        b = g.extract(url=url, raw_html=raw_html)
        print "{0} - {1}".format(b.title,b.cleaned_text)


stop watching elm-test generated code

Right now if you have elm-test watching and elm-live watching, elm-live ends up restarting a whole bunch of times from the changes that elm-test makes in elm-stuff/generated-code.

Hey Noah, thanks a lot for the PR! Sorry for the delay here, life totally sucked me in over the last couple of weeks (big renovation project for my family). Luckily it’s mostly sorted now so I’m back to elm-live again. Thanks for the patience, I’ll get the build fixed in master and get back to you here in no time.


Update smoke test to use common prow e2e script

Can we make this script run locally for local development?

I am not sure I understand your question. You can run this script locally for local development.


Sets default gas price to 0.02 szabo

to better mimic current livenet and testnet's gas price.
I understand that gas price is changeable via option, however I feel that a much larger default than the current 1 would probably be better even during development.

A more reasonable default is still better than 1 wei, which is multiple orders of magnitude smaller than livenet's value.
Regarding keeping up with changes, I am fine with either of the follow 2 approaches:

Keep up with live net's hardcoded value, or
Do not keep up, but have the default at a a more sensible value than 1 wei.

Noticed the PR is now having conflicts. Let me know if I should resubmit.

It's a simple PR; not sure why there's conflicts. No need to resubmit. We'll take care of it if it's accepted.
@area and I have been talking about fixing #130 in the next big release (3.0). This feature is similar. @area - can I get your view on it?

Yeah, I can't see any reason why this isn't a net benefit. Even if the price changes on the live net, we can certainly be a lot more correct than ten orders of magnitude wrong.
A knock-on effect that might want to be considered is increasing the default balance accounts start with, given accounts will be spending a lot more on gas now, but I could equally see that being left alone - 100 Ether is very reasonable.

There was a conflict because gasPriceVal was moved to the state manager.
I've merged this into the develop branch. Will close this ticket to show 3.0 milestone progress.
Regarding the default gas price, there's a discussion in #109. I'm on the fence about raising it or leaving it where it is. If 100 ETH is plenty, we should probably stick with that for now. However, if there are reports that people are running out of funds during unit tests than I'd definitely consider raising it.


Fix double test parameter call

For dockstore/dockstore#3871
Kind of reverting https://github.com/dockstore/dockstore-ui2/pull/1079 but not quite because of the tests

Kind of reverting #1079 but not quite because of the tests

If you're kind of reverting #1079, does that mean that bug #3758 is coming back?

Kind of reverting #1079 but not quite because of the tests

If you're kind of reverting #1079, does that mean that bug #3758 is coming back?

It's possible. But I handled the error in the original calls, so it shouldn't. The test from before remains. Though tester should test both issues.


No Results feedback

It would be nice being able to give the user feedback in case there's no result found.

@DERIIIFranz I just released react-autosuggest 2.0 which supports a controlled behaviour. You

This will be possible in 3.0. Check out the 3.0 branch for work in progress, if interested.

@DERIIIFranz  Just wanted to let you know that 3.0 alpha version is available on npm:
npm install<EMAIL_ADDRESS>--save

@moroshko Heya, is this possible yet? I am using 3.0.0.

@hughlomas It's not built-in, but you can easily implement this yourself.

@moroshko Thank you for the example, that worked nicely.


Stopped Loading on Ubuntu 17.10

Hi, in the last few days the app has stopped loading in Ubuntu 17.10, it just changes the cursor to a spinner for a few seconds and then nothing. Rebooting had no effect, same with removing and installing the latst version on the website, the app simply doesn't load.
I've had it installed for  a few months before that and its worked fine.
Any suggestions?

Nope and its worked fine before. Google Music also works fine if run from Chrome. It literally stopped working over night.

Try installing libgconf-2-4.  I was having the same issue and saw this suggested in another thread.  It worked for me.

@johndeaton  - You're a star, that worked!

Great!  I'm glad it worked for you too.

This worked for me too on Ubuntu 18.04.

The deb package should mark libgconf-2-4 as dependency, in my opinion. I've run into the issue just now and this could be avoided by that dependency marker.

Out of the frying-pan into the fire...

I'm getting the same Seg Fault error as speedstyle after installing libgconf-2-4 on Ubuntu 19.04

I can also confirm @speedstyle, @MichalKrasno on 19.04.
I already had libgconf-2-4 installed and got:

Segmentation fault (core dumped)

Then i removed the lib just for kicks and got:

google-play-music-desktop-player: error while loading shared libraries: libgconf-2.so.4: cannot open shared object file: No such file or directory

Same issue here on Ubuntu 19.04. The application worked just fine on Ubuntu 18.04, but after upgrading, I get that same Segmentation Fault.

@mjohnson8165  @MichalKrasno  @speedstyle Looks like #3553 is related.

OK, #3117 can be resolved by installing libgconf-2.4 and the segfault is separate and documented in #3553.


[BUG] Machine stuck in "Deleting" state

Rancher Server Setup

Rancher version: v2.7.1
Installation option : Helm Chart

If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc):

v1.24.10+k3s1

Proxy/Cert Details: None

Information about the Cluster

Kubernetes version: v1.24.8+rke2r1
Cluster Type: Downstream

If downstream, what type of cluster? Infrastructure Provider, using a custom docker-machine driver:

docker-machine-driver-outscale

User Information

What is the role of the user logged in? Admin

Describe the bug
I'm creating a cluster with the Help of this Helm Chart using docker-machine-driver-outscale (version 0.2.0).

Cluster creation :

Worker is faster to deploy than controller, so sometime the worker get deleted before the controller is up&running. In that case, the worker get stuck in "Deleting" state

Cluster removal :

This is done with an helm uninstall, usually both Machine (node) get stuck in "Deleting" state, no matter if I use Helm chart or Rancher Web UI. The underlying VM is however deleted

To Reproduce

Install outscale docker driver version 0.2.0

https://github.com/outscale/docker-machine-driver-outscale/releases/download/v0.2.0/docker-machine-driver-outscale_0.2.0_linux_amd64.zip
https://oos.eu-west-2.outscale.com/rancher-ui-driver-outscale/v0.2.0/component.js
bb539ed4e2b0f1a1083b29cbdbab59bde3efed0a3145fefc0b2f47026c48bfe0
oos.eu-west-2.outscale.com

Configure a cloud credentials for Outscale
Create a cluster with the Outscale node driver (wait for the node to be ready)
Remove the cluster

Result
Machine is in "Deleting" state
apiVersion: cluster.x-k8s.io/v1beta1
kind: Machine
metadata:
  annotations:
    machine.cluster.x-k8s.io/exclude-node-draining: 'true'
  creationTimestamp: '2023-03-14T15:02:29Z'
  deletionGracePeriodSeconds: 0
  deletionTimestamp: '2023-03-14T15:55:44Z'
  finalizers:
    - machine.cluster.x-k8s.io
  generateName: fhn-dev-rancher-dev-zone-dev2-control-5fdcdc86dd-
  generation: 4
  labels:
    cattle.io/os: linux
    cluster.x-k8s.io/cluster-name: fhn-dev-rancher-dev-zone-dev2
    cluster.x-k8s.io/control-plane: 'true'
    cluster.x-k8s.io/deployment-name: fhn-dev-rancher-dev-zone-dev2-control
    machine-template-hash: '19856474288'
    rke.cattle.io/cluster-name: fhn-dev-rancher-dev-zone-dev2
    rke.cattle.io/control-plane-role: 'true'
    rke.cattle.io/etcd-role: 'true'
    rke.cattle.io/machine-id: 2b85d046acdf976a09d093b35bdc327e792dd3868d8bf1c4031a97b552842de
    rke.cattle.io/rke-machine-pool-name: control
  name: fhn-dev-rancher-dev-zone-dev2-control-5fdcdc86dd-5cnns
  namespace: dev-zone-dev2
  ownerReferences:
    - apiVersion: cluster.x-k8s.io/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: MachineSet
      name: fhn-dev-rancher-dev-zone-dev2-control-5fdcdc86dd
      uid: 4db62934-5cc0-4f9a-a748-2a58e6117b01
  resourceVersion: '2549832'
  uid: 844ef00e-4221-41a4-ae55-3c3225cd1a2e
spec:
  bootstrap:
    configRef:
      apiVersion: rke.cattle.io/v1
      kind: RKEBootstrap
      name: fhn-dev-rancher-dev-zone-dev2-bootstrap-template-r6668
      namespace: dev-zone-dev2
      uid: 3b869958-8bea-4383-8ce5-ca0f34ab4d98
    dataSecretName: fhn-dev-rancher-dev-zone-dev2-bootstrap-template-r6668-ma-7c6bf
  clusterName: fhn-dev-rancher-dev-zone-dev2
  infrastructureRef:
    apiVersion: rke-machine.cattle.io/v1
    kind: OutscaleMachine
    name: fhn-dev-rancher-dev-zone-dev2-control-b25bf7ee-25sbx
    namespace: dev-zone-dev2
    uid: 52da89cd-b663-468f-982e-445739e81b0b
  nodeDeletionTimeout: 10s
  providerID: aws:///eu-west-2a/i-d874bc09

Expected Result
The Machine is deleted
Screenshots

We finally managed  to solve this issue.
I spotted that I did not have this issue with other CNI (canal) the issue happened with Cilium mainly.
Here is a specific configuration for Ubuntu 22.04 published on Cilium doc

systemd-based distributions
Some distributions need to be configured not to manage “foreign” routes. This is the case in Ubuntu 22.04, for example (see GitHub issue 18706). This can usually be done by setting

ManageForeignRoutes=no
ManageForeignRoutingPolicyRules=no

I'm having a similar issue.
I find what you are referring to in [the v1.13 docs]https://docs.cilium.io/en/v1.13/operations/system_requirements/#systemd-based-distributions) but not in the v1.14 docs onward. Wonder if this has been resolved for newer versions of Cilium. Do you know which version you were on when you had the issue?


Is an internet connection needed for proper web-interface

In files like the skelleton.css, I see lines that point to internet url's, like:
src: local('Raleway Light'), local('Raleway-Light'), url(http://fonts.gstatic.com/s/raleway/v11/ZKwULyCG95tk6mOqHQfRBCEAvth_LlrfE80CYdSH47w.woff2) format('woff2');
Does that mean that, for a proper operation, the client needs access to internet so it can also get those data?
I ask this because when I connect my laptop or smartphone (only) to the esp8266 Accesspoint, there are intermitting failures when loading the webpages.

Ah found this: https://github.com/spacehuhn/wifi_ducky/issues/30


Camel - Import of the Camel Model Definitions

The Camel model schema is imported into the old UI for icon lookup and model validation.
Maybe instead of importing it into a vendor directory, we should publish it as a separate npm package that is then brought in as a yarn dependency?

Playing with some more am finding the overhead of the model being a module outweighs the reasons for it to be a separate package.

Actually I second your initial proposal. Making the Camel model JS a separate module sounds reasonable, especially considering that the module would require an extra dev dependency to interact with Maven artifacts from Camel as hawtio-integration currently does. Keeping it separate can make the @hawtio/react package simple.
Since we are already migrated to monorepo, why not just having the module under another package packages/camel-model within the hawtio-next project?  Then I don't see extra overhead of having it as a separate module.


Generalisation of snapshots

snapshot are limited to one thread producer and one thread consumer. It will be handy to remove this limitation. However it is not easy as one major feature of a snapshot is to avoid any copy and avoid any waiting for data availability (even if theoretically there is a wait loop for a CAS...)
To do so, instead of a triple buffer, a 1+C+P buffer is needed with C the number of thread consumers and P the number of thread producers. The API should allow to keep track of the used buffers by the different threads, and the different threads should be able to lock/unlock a particular buffer (This a quite similar to a shared pointer mechanism to keep track of the used buffers and the free ones).

API is likely to be like a little bit more complex:
const type *_read_begin(void);
void _read_end(const type*p);
type *_write_begin(void);
void _write_end(type*p);


docs(gateway/proxy) add more detailed description of the regex matching

behavior. add description of URI normalization methods used by the router.

Direct review link: https://deploy-preview-2768--kongdocs.netlify.app/gateway-oss/2.4.x/proxy/#request-path


Postgres Auto Dropping the DB Randomly - database "template0" is not currently accepting connections

Hi, I'm facing this error from multiple days, it happens on postgres inside docker.

ERROR: database "template0" is not currently accepting connections
FATAL: database "template0" can only be updated by autovacuum worker processes

It happens mostly after a hour,
the below is the  service inside docker compose
  db:
    image: postgres
    restart: always
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: postgres
    ports:
      - "5432:5432"
    volumes:
      - ./init.sql:/docker-entrypoint-initdb.d/init.sql
      - database_data:/var/lib/postgresql/data

init.sql
-- Create the EC2 Fleet Instance Management table
CREATE TABLE IF NOT EXISTS ec2_fleet_instance_management (
    fleet_id VARCHAR PRIMARY KEY,
    instance_id VARCHAR,
    state VARCHAR,
    requested_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    fleet_group VARCHAR,
    instance_detail JSONB,
    instance_status VARCHAR,
    system_status VARCHAR,
    llm_status VARCHAR,
    public_ip_address VARCHAR
);

-- Create the Request History table
CREATE TABLE IF NOT EXISTS request_history (
    request_id VARCHAR PRIMARY KEY,
    instance_handled VARCHAR,
    request_details JSONB,
    response_details JSONB,
    requested_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
    request_type VARCHAR,
    instance_detail JSONB,
    status_code VARCHAR,
    public_ip_address VARCHAR
);

-- Indexes for the EC2 Fleet Instance Management table for quicker search
CREATE INDEX IF NOT EXISTS idx_ec2_fleet_instance_management_on_fleet_group ON ec2_fleet_instance_management (fleet_group);
CREATE INDEX IF NOT EXISTS idx_ec2_fleet_instance_management_on_state ON ec2_fleet_instance_management (state);

-- Indexes for the Request History table for quicker search
CREATE INDEX IF NOT EXISTS idx_request_history_on_request_type ON request_history (request_type);
CREATE INDEX IF NOT EXISTS idx_request_history_on_requested_at ON request_history (requested_at);

the connection im making
def get_db_connection():
    conn = psycopg2.connect(
        dbname="postgres",
        user="postgres",
        password="*****",
        host="db",
        port="5432"
    )
    return conn

Any where i'm doing it wrong?

Unless you require access from another machine, then remove the ports: config. If the database host is accessible on the internet, then anyone can attempt to connect. Which is what I would guess is happening; there isn't anything in the container that would auto drop tables or databases. If you do need other hosts to access it, then choose an extremely strong password and setup something like AWS security groups to limit network access (local solutions like firewalld and iptables will not work since docker works with/around those to make the port exposed when you use ports).

@yosifkit yes i understood that, but the issue is the ports is for the local docker container to access, and the instance is under vpc, so no outer people can access it,
the issue is, after some time, the error i mentioned (database "template0" is not currently accepting connections), is coming up and then the dropping is happening, to confirm this, i tried multiple other times with same config, same error is happening,
but other pg under same docker where i access tables with prisma, its not happening, so i was guessing there is issue in my config


An error that should go silent

Sometimes I get this on saving files (very rare in general, but often when I :Gwrite:
Error detected while processing function ale#events#SaveEvent[6]..ale#fix#Fix[25]..ale#job#Stop:
line    9:
E474: Invalid argument

I think even if it can't be avoided it should be silent and don't show error.
ALEInfo has paths of my computer I'm not comfortable to share. I also suggest removing those paths from it.

You can redact paths before you share whatever output.
Do you have any steps for repeating that bug? That ought not to happen.

This is the ALEInfo:

 Current Filetype: python
Available Linters: ['flake8', 'mypy', 'pycodestyle', 'pylint']
  Enabled Linters: ['flake8', 'mypy', 'pylint']
 Linter Variables:

let g:ale_python_flake8_executable = 'flake8'
let g:ale_python_flake8_options = ''
let g:ale_python_flake8_use_global = 0
let g:ale_python_mypy_executable = 'mypy'
let g:ale_python_mypy_options = ''
let g:ale_python_mypy_use_global = 0
let g:ale_python_pylint_executable = 'pylint'
let g:ale_python_pylint_options = '--rcfile=~/.config/pylint.conf --load-plugins pylint_django'
let g:ale_python_pylint_use_global = 0
 Global Variables:

let g:ale_echo_cursor = 1
let g:ale_echo_msg_error_str = 'Error'
let g:ale_echo_msg_format = '[%linter%] %s'
let g:ale_echo_msg_warning_str = 'Warning'
let g:ale_enabled = 1
let g:ale_fix_on_save = 1
let g:ale_fixers = {'javascript': ['eslint', 'remove_trailing_lines', function('<lambda>1')], 'python': ['autopep8', 'isort', 'remove_trailing_lines', function('<lambda>2'), function('<lambda>3')]}
let g:ale_keep_list_window_open = 0
let g:ale_lint_delay = 200
let g:ale_lint_on_enter = 1
let g:ale_lint_on_save = 1
let g:ale_lint_on_text_changed = 'always'
let g:ale_linter_aliases = {}
let g:ale_linters = {}
let g:ale_open_list = 0
let g:ale_set_highlights = 1
let g:ale_set_loclist = 1
let g:ale_set_quickfix = 0
let g:ale_set_signs = 1
let g:ale_sign_column_always = 1
let g:ale_sign_error = '✖'
let g:ale_sign_offset = 1000000
let g:ale_sign_warning = '⚠'
let g:ale_statusline_format = ['✖ %d', '⚠ %d', '']
let g:ale_warn_about_trailing_whitespace = 1
  Command History:

(finished - exit code 0) ['/bin/sh', '-c', '''flake8'' --format=default --stdin-display-name ''/blahblah/somepath.py'' - < ''/var/folders/sx/995cq4316_57jgdyhwjvf9rw0000gn/T/nvimZbbbKV/435/routes.py''']

<<<NO OUTPUT RETURNED>>>

(executable check - failure) mypy
(finished - exit code 1) ['/bin/sh', '-c', '''/blahblah/.virtualenvs/somevf/bin/pylint'' --rcfile=~/.config/pylint.conf --load-plugins pylint_django --output-format text --msg-template="{path}:{line}:{column}: {msg_id} ({symbol}) {msg}" --reports n ''/blahblah/somepath.py''']

<<<NO OUTPUT RETURNED>>>

(finished - exit code 0) ['/bin/sh', '-c', '''flake8'' --format=default --stdin-display-name ''/blahblah/somepath.py'' - < ''/var/folders/sx/995cq4316_57jgdyhwjvf9rw0000gn/T/nvimZbbbKV/440/routes.py''']

<<<NO OUTPUT RETURNED>>>

(executable check - failure) mypy
(finished - exit code 1) ['/bin/sh', '-c', '''/blahblah/.virtualenvs/somevf/bin/pylint'' --rcfile=~/.config/pylint.conf --load-plugins pylint_django --output-format text --msg-template="{path}:{line}:{column}: {msg_id} ({symbol}) {msg}" --reports n ''/blahblah/somepath.py''']

<<<NO OUTPUT RETURNED>>>

(started) ['/bin/sh', '-c', '''flake8'' --format=default --stdin-display-name ''/blahblah/somepath.py'' - < ''/var/folders/sx/995cq4316_57jgdyhwjvf9rw0000gn/T/nvimZbbbKV/441/routes.py''']
(executable check - failure) mypy
(finished - exit code 0) ['/bin/sh', '-c', '''flake8'' --format=default --stdin-display-name ''/blahblah/somepath.py'' - < ''/var/folders/sx/995cq4316_57jgdyhwjvf9rw0000gn/T/nvimZbbbKV/442/routes.py''']

<<<NO OUTPUT RETURNED>>>

(executable check - failure) mypy
(finished - exit code 0) ['/bin/sh', '-c', '''autopep8'' - < ''/var/folders/sx/995cq4316_57jgdyhwjvf9rw0000gn/T/nvimZbbbKV/443/routes.py''']
(finished - exit code 0) ['/bin/sh', '-c', '''/blahblah/.virtualenvs/somevf/bin/isort'' - < ''/var/folders/sx/995cq4316_57jgdyhwjvf9rw0000gn/T/nvimZbbbKV/444/routes.py''']
(started) ['/bin/sh', '-c', '''flake8'' --format=default --stdin-display-name ''/blahblah/somepath.py'' - < ''/var/folders/sx/995cq4316_57jgdyhwjvf9rw0000gn/T/nvimZbbbKV/445/routes.py''']
(executable check - failure) mypy
(started) ['/bin/sh', '-c', '/usr/local/bin/add-trailing-comma --py35-plus --py36-plus ''/var/folders/sx/995cq4316_57jgdyhwjvf9rw0000gn/T/nvimZbbbKV/446/routes.py''']
(finished - exit code 0) ['/bin/sh', '-c', '''flake8'' --format=default --stdin-display-name ''/blahblah/somepath.py'' - < ''/var/folders/sx/995cq4316_57jgdyhwjvf9rw0000gn/T/nvimZbbbKV/447/routes.py''']

<<<NO OUTPUT RETURNED>>>

(executable check - failure) mypy
(finished - exit code 0) ['/bin/sh', '-c', '''flake8'' --format=default --stdin-display-name ''/blahblah/somepath.py'' - < ''/var/folders/sx/995cq4316_57jgdyhwjvf9rw0000gn/T/nvimZbbbKV/458/routes.py''']

<<<NO OUTPUT RETURNED>>>

(executable check - failure) mypy
(finished - exit code 1) ['/bin/sh', '-c', '''/blahblah/.virtualenvs/somevf/bin/pylint'' --rcfile=~/.config/pylint.conf --load-plugins pylint_django --output-format text --msg-template="{path}:{line}:{column}: {msg_id} ({symbol}) {msg}" --reports n ''/blahblah/somepath.py''']

<<<NO OUTPUT RETURNED>>>

If it doesn't provide you the information you need then we can go with reproducing plan. I almost always get this error if I save file with :Gwrite from fugitive.

If you can provide some steps for reproducing the bug, I'll have a look at this. Please also provide the full error message that you see, if there are any other lines output for the error message.

There no more lines, that's the complete error message. What about making that error silent? I think it can be ignored.
This is the list of my python fixers for ale:
            \ 'python': [
            \   'autopep8',
            \   'isort',
            \   'remove_trailing_lines',
            \   {buffer -> {
            \     'command': '/usr/local/bin/add-trailing-comma --py35-plus --py36-plus %t',
            \     'read_temporary_file': 1,
            \   }},
            \   {buffer, lines -> map(lines, 'substitute(v:val, ''^\s*$'', '''', '''')')},
            \ ]

If and only if I remove autopep8, isort, and the custom add-trailing-comma fixers it doesn't raise the error. If only one of these 3 are there it always raises the error with :Gwrite.

That error can certainly be fixed. I haven't looked at this yet.

Great, I suggested making it silent in case it's not easy to fix.

I also receive this error primarily when running the Gwrite fugitive command or on changing buffers (when I have autowrite set to on) when editing a file which uses prettier as a fixer(the only fixer i have enabled).
PS: just wanted to take the opportunity to also say thanks for the amazing plugin!!

I should have fixed this now by just changing call jobstop(a:job_id) to silent! call jobstop(a:job_id).

I tested it and it doesn't raise an error anymore. Thanks.