Segfaults when under pressure?

I'm trying to debug a segfault while using Gatsby that is potentially in the realm of using sharp.
(I'm a performance eng for Gatsby)
It seems that sites with lots of images can trigger a segfault and we're not sure when exactly this happens.
I get this fairly frequently with a simple site that just has many images (~1500 for my machine) on the example site posted (in http://github.com/gatsbyjs/gatsby/pull/6291) as a repro;
Clone https://github.com/jp887/gatsby-issue6291-repro
Update package.json to set all gatsby deps to *
Clear the npm cache (~/.npm)
Clear node_modules, in case that's relevant
npm install
yarn gen-images 1500 (It triggers with 1500 images for me, this will try to process 9000 images in total)
gatsby build
For me, often it segfaults roughly between 25% and 75%.
I'm on ubuntu (xfce) 19.10
Node v8.16.2 (through nvm)
I went through some of the older Gatsby issues relating to this bug. I am able to reproduce it with .simd(true) and .simd(false), and same for .cache(true) and .cache(false).
Reducing the available memory to node does not seem to make a difference (I set it to 500mb, half of the default, but saw no increase in crash rate). Was also able to repro it running single core.
Please let me know how I can help :)

Hi Peter, I previously helped one of the organisations experiencing the problems reported in https://github.com/gatsbyjs/gatsby/issues/6291 and there's a summary of the findings in https://github.com/gatsbyjs/gatsby/issues/6291#issuecomment-505097465
Thanks for bringing it to my attention again. By stressing my machine with other CPU-bound processes at the same time until it overheated and throttled the clock rate, I finally managed to reproduce a crash and get a backtrace using the latest sharp v0.23.3, its vendored copy of libvips v8.8.1 and the latest Node.js LTS v12.13.1.
(sharp:16801): GLib-GObject-WARNING **: 16:50:33.759: gtype.c:4265: type id '0' is invalid

(sharp:16801): GLib-GObject-WARNING **: 16:50:33.759: can't peek value table for type '<invalid>' which is not currently referenced

(sharp:16801): GLib-GObject-WARNING **: 16:50:33.762: gvalue.c:188: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation
[Thread 0x7fff9affd700 (LWP 7136) exited]

Thread 10 "node" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd6ffc700 (LWP 17345)]
0x00007fffd52c27d7 in g_type_check_value () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libgobject-2.0.so.0
(gdb) bt
#0  0x00007fffd52c27d7 in g_type_check_value () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libgobject-2.0.so.0
#1  0x00007fffd52c54e3 in g_value_transform () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libgobject-2.0.so.0
#2  0x00007fffd5a3fe97 in ?? () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#3  0x00007fffd5a3ff07 in ?? () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#4  0x00007fffd5a50485 in vips_slist_map2 () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#5  0x00007fffd5a405a4 in vips.image_copy_fields_array () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#6  0x00007fffd5a3b262 in vips_image_pipeline_array () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#7  0x00007fffd5a3b363 in vips_image_pipelinev () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#8  0x00007fffd5939a16 in ?? () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#9  0x00007fffd5a31b29 in vips_object_build () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#10 0x00007fffd5a3cf98 in vips_cache_operation_buildp () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#11 0x00007fffd5a4339d in vips_call_required_optional () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#12 0x00007fffd5a43eaf in vips_call_split () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#13 0x00007fffd5939d89 in vips_copy () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#14 0x00007fffd5a383dd in vips_image_decode () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#15 0x00007fffd589e075 in ?? () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#16 0x00007fffd5a31b29 in vips_object_build () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#17 0x00007fffd5a3cf98 in vips_cache_operation_buildp () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#18 0x00007fffd5a4339d in vips_call_required_optional () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#19 0x00007fffd5a43eaf in vips_call_split () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#20 0x00007fffd589e9f7 in vips_reducev () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#21 0x00007fffd58954e5 in ?? () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#22 0x00007fffd5a31b29 in vips_object_build () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#23 0x00007fffd5a3cf98 in vips_cache_operation_buildp () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips.so.42
#24 0x00007ffff41fe905 in vips::VImage::call_option_string(char const*, char const*, vips::VOption*) ()
   from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips-cpp.so.42
#25 0x00007ffff4210ee3 in vips::VImage::resize(double, vips::VOption*) const () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/../../vendor/lib/libvips-cpp.so.42
#26 0x00007ffff444d7b1 in PipelineWorker::Execute() () from /gatsby-issue6291-repro/node_modules/sharp/build/Release/sharp.node
#27 0x00000000012d1c0e in worker (arg=0x0) at ../deps/uv/src/threadpool.c:122
#28 0x00007ffff7c42182 in start_thread (arg=<optimised out>) at pthread_create.c:486
#29 0x00007ffff7b6bb1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

From what I can tell, the Gatsby build process is queuing filesystem-based input images one-at-a-time then attempting to concurrently generate 6 output images to memory, resized in various dimensions. All of these images are JPEGs.
sharp uses libuv-managed threads, which by default will limit this to 4 concurrently (and queue the other 2). It looks like Gatsby calls sharp.concurrency() and therefore vips_concurrency_set with the physical core count, so I think that's 12 concurrent threads accessing the same input image on a typical "4 core" CPU with hyper-threading.
If I set the GATSBY_CPU_COUNT=1 environment variable, which Gatsby uses to call sharp.concurrency(1) and therefore vips_concurrency_set(1) I cannot get it to crash, which suggests a possible threading problem. Setting GATSBY_CPU_COUNT=1 was part of the summary of https://github.com/gatsbyjs/gatsby/issues/6291#issuecomment-505097465 although in that context was due to CircleCI incorrectly reporting physical CPU core counts.
@jcupitt Can you see any possible thread-safety issues in this backtrace? Perhaps something in vips__image_copy_fields_array and vips__gslist_gvalue_merge?
I'll attempt to get it to crash with a debug build of libvips to try to discover which functions the ?? in the backtrace refer to.

Hi. I'm really happy to hear you were able to get a repro AND trace, as I was unable to get the trace.
I will try to go through the code in Gatsby tomorrow and see how we can improve that situation. I don't think it should be hammering the process like that in the first place as that has different perf implications as well.

By stressing my machine with other CPU-bound processes at the same time until it overheated and throttled the clock rate, I finally managed to reproduce a crash

That's interesting because on my machine, this happens all the time (whenever I trigger any computation) and have these messages in my logs every time. So perhaps the problem is related to clock throttling and stressing.
Anyways. Happy to hear you have a repro and hoping it'll improve the situation :)

(Perhaps my test with single cpu was botched, I can't say for sure right now, so take that with a grain of salt)

Here's a more detailed backtrace:
#0  0x00007ffff405f047 in g_type_check_value () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#1  0x00007ffff4061d53 in g_value_transform () at /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0
#2  0x00007ffff420a9d7 in meta_new (image=0x7fffc8068640, name=0x7fffb40008d0 "\020\"", value=0x7fffd002de70) at header.c:325
#3  0x00007ffff420aa47 in meta_cp_field (meta=<optimised out>, dst=<optimised out>) at header.c:916
#4  0x00007ffff42144f5 in vips_slist_map2 (list=<optimised out>, fn=fn@entry=0x7ffff420aa30 <meta_cp_field>, a=a@entry=0x7fffc8068640, b=b@entry=0x0) at util.c:129
#5  0x00007ffff420b114 in meta_cp (src=0x7fffc00aa4d0, dst=0x7fffc8068640) at header.c:934
#6  0x00007ffff420b114 in vips__image_copy_fields_array (out=out@entry=0x7fffc8068640, in=in@entry=0x7fffd7ffae00) at header.c:981
#7  0x00007ffff4205e92 in vips_image_pipeline_array (image=image@entry=0x7fffc8068640, hint=hint@entry=VIPS_DEMAND_STYLE_THINSTRIP, in=in@entry=0x7fffd7ffae00) at generate.c:395
#8  0x00007ffff4205f9f in vips_image_pipelinev (image=0x7fffc8068640, hint=hint@entry=VIPS_DEMAND_STYLE_THINSTRIP) at generate.c:435
#9  0x00007ffff4153459 in vips_copy_build (object=0x7fff60070380) at copy.c:183
#10 0x00007ffff41fbdd9 in vips_object_build (object=0x7fff60070380) at object.c:367
#11 0x00007ffff4207ce8 in vips_cache_operation_buildp (operation=0x7fffd7ffd160) at cache.c:866
#12 0x00007ffff4207ce8 in vips_cache_operation_buildp (operation=operation@entry=0x7fffd7ffd160) at cache.c:842
#13 0x00007ffff420dd5d in vips_call_required_optional (operation=operation@entry=0x7fffd7ffd160, required=required@entry=0x7fffd7ffd190, optional=optional@entry=0x7fffd7ffd270) at operation.c:880
#14 0x00007ffff420e466 in vips_call_by_name (operation_name=<optimised out>, option_string=option_string@entry=0x0, required=required@entry=0x7fffd7ffd190, optional=optional@entry=0x7fffd7ffd270)
    at operation.c:920
#15 0x00007ffff420e8b9 in vips_call_split (operation_name=operation_name@entry=0x7ffff4245ff6 "copy", optional=optional@entry=0x7fffd7ffd270) at operation.c:1024
#16 0x00007ffff41537c6 in vips_copy (in=in@entry=0x7fffc00aa4d0, out=out@entry=0x7fffd00d8e30) at copy.c:399
#17 0x00007ffff4202f6d in vips_image_decode (in=in@entry=0x7fffc00aa4d0, out=out@entry=0x7fffd00d8e30) at image.c:2910
#18 0x00007ffff40edf45 in vips_reducev_build(VipsObject*) (object=0x7fffd00d84c0) at reducev.cpp:860
#19 0x00007ffff41fbdd9 in vips_object_build (object=0x7fffd00d84c0) at object.c:367
#20 0x00007ffff4207ce8 in vips_cache_operation_buildp (operation=0x7fffd7ffd550) at cache.c:866
#21 0x00007ffff4207ce8 in vips_cache_operation_buildp (operation=operation@entry=0x7fffd7ffd550) at cache.c:842
#22 0x00007ffff420dd5d in vips_call_required_optional (operation=operation@entry=0x7fffd7ffd550, required=required@entry=0x7fffd7ffd580, optional=optional@entry=0x7fffd7ffd660) at operation.c:880
#23 0x00007ffff420e466 in vips_call_by_name (operation_name=<optimised out>, option_string=option_string@entry=0x0, required=required@entry=0x7fffd7ffd580, optional=optional@entry=0x7fffd7ffd660)
    at operation.c:920
#24 0x00007ffff420e8b9 in vips_call_split (operation_name=operation_name@entry=0x7ffff423f420 "reducev", optional=optional@entry=0x7fffd7ffd660) at operation.c:1024
#25 0x00007ffff40ee574 in vips_reducev(VipsImage*, VipsImage**, double, ...) (in=in@entry=0x7fffc00aa4d0, out=out@entry=0x7fffd0045b60, <EMAIL_ADDRESS>at reducev.cpp:976
#26 0x00007ffff40e64d5 in vips_resize_build (object=0x7fff6c0c08b0) at resize.c:227
#27 0x00007ffff41fbdd9 in vips_object_build (object=0x7fff6c0c08b0) at object.c:367
#28 0x00007ffff4207ce8 in vips_cache_operation_buildp (operation=0x7fffd7ffd810) at cache.c:866
#29 0x00007ffff4207ce8 in vips_cache_operation_buildp (operation=operation@entry=0x7fffd7ffd810) at cache.c:842
#30 0x00007ffff4581b50 in vips::VImage::call_option_string(char const*, char const*, vips::VOption*) (operation_name=<optimised out>, option_string=0x0, options=0x7fffd002d8f0) at VImage.cpp:513
#31 0x00007ffff4597e43 in vips::VImage::resize(double, vips::VOption*) const (this=0x7fffd7ffd908, scale=0.9375, options=<optimised out>) at ../cplusplus/include/vips/VImage8.h:108
#32 0x00007ffff45fab5c in PipelineWorker::Execute() () at /gatsby-issue6291-repro/node_modules/sharp/build/Release/sharp.node
#33 0x00000000012d1c0e in worker (arg=0x0) at ../deps/uv/src/threadpool.c:122
#34 0x00007ffff7c42182 in start_thread (arg=<optimised out>) at pthread_create.c:486
#35 0x00007ffff7b6bb1f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

That's very interesting. I'll have a look today.

I think I found what's making those message before the crash:
https://github.com/libvips/libvips/blob/master/libvips/iofuncs/header.c#L311
If you change that to be:
        /*
        if( G_VALUE_TYPE( value ) == G_TYPE_STRING )
                g_value_init( &meta->value, VIPS_TYPE_REF_STRING );
        else
                g_value_init( &meta->value, G_VALUE_TYPE( value ) );
         */

        g_value_init( &meta->value, 0 );

And try vips copy x.jpg y.jpg you see:
$ vips copy k2.jpg x.v

(vips:14583): GLib-GObject-WARNING **: 16:33:11.184: ../../../gobject/gtype.c:4268: type id '0' is invalid

(vips:14583): GLib-GObject-WARNING **: 16:33:11.184: can't peek value table for type '<invalid>' which is not currently referenced

(vips:14583): GLib-GObject-WARNING **: 16:33:11.184: ../../../gobject/gvalue.c:185: cannot initialize GValue with type '(null)', this type has no GTypeValueTable implementation

The same set of errors, so probably the value param to meta_new() is pointing at a lot of zeros.
meta_cp takes a shallow copy of the list of metadata items on each input image, then calls meta_new() for each item. A metadata item is disappearing (probably) some time between the shallow copy of the list and calling meta_new().
This could be caused by an image being unreffed during the copy, but that seems unlikely to me, the chaos would be even worse chaos.
It's more likely that some other thread is changing the metadata on the input image during the copy. Could this be happening?
libvips images have a list of arbitrary name - value pairs attached to them, and these are inherited as images are processed. They are used for things like ICC profiles and EXIF data, but user code can add extra metadata too if it's convenient. Image metadata is supposed to be immutable: after the image has been constructed, it's all frozen and must not be changed.
At the moment, there's no lock mechanism to prevent you changing image metadata. How about adding something to enforce this? It could be something very simple, eg:
void
vips_image_set( VipsImage *image, const char *name, GValue *value )
{
        g_assert( name );
        g_assert( value );

        /* If this image is shared, block metadata changes.
         */
        if( G_OBJECT( image )->ref_count > 1 ) {
                g_info( "can't set metadata \"%s\" on shared image", name );
                return;
        }

        meta_init( image );
        (void) meta_new( image, name, value );

... etc.

Thanks John, amazing debugging skills there as always.
If I add your suggestion to libvips and run the sharp test suite against  it I see things like:

can't set metadata "exif-data" on shared image
can't set metadata "hide-progress" on shared image
can't set metadata "offset" on shared image

which led me to a few possible sources within libvips:
https://github.com/libvips/libvips/blob/master/libvips/foreign/jpeg2vips.c#L613
https://github.com/libvips/libvips/blob/master/libvips/foreign/dzsave.c#L599
https://github.com/libvips/libvips/blob/master/libvips/convolution/convi.c#L841
Next step (for me) is to try to reproduce the original crash with this extra debugging in place.

Yes, bits of the test suite fail too, I'll fix them.

Thanks John, I also added this debug to vips_image_remove() and found the following occurrence:
https://github.com/libvips/libvips/blob/master/libvips/colour/profile_load.c#L208

Commit bb15cd9 ensures sharp always takes a copy before updating/removing metadata.

I made a PR to gather up any changes libvips needs for this. Please push any related patches there. I put your nice _remove one in.
https://github.com/libvips/libvips/pull/1483

I found a couple of places in libvips which were breaking the "metadata is frozen" rule.
The big one was vips__exif_update(): this merges other metadata fields into the exif block and is called before a save to formats that support exif. I think this is a plausible cause of these crashes: if one thread updates the exif block while another thread is trying to take a copy of the image metadata you'd see something like this.
I'll backport https://github.com/libvips/libvips/pull/1483 (but not the locks on metadata changes) to current stable 8.8 and it should be easy to test.

OK, the head of 8.8 (it's 8.8.4 at the moment) includes this change to metadata handling and (I hope!) should be easy to test.

Brilliant, thanks John, I've been unable to reproduce a segfault using the check-metadata-changes branch of libvips with the latest master branch of sharp.

That's great!
@pvdz would you be able to test 8.8.4? It should be a drop-in replacement for your current libvips.

(or are there related sharp changes that would be needed too, Lovell?)

I found a couple of places in sharp that could potentially cause this problem, addressed in https://github.com/lovell/sharp/commit/bb15cd906717ba054d9c23db2971e94d2e250969, which will be in sharp v0.23.4.

I'll wait for that release and check it. Sounds like great progress! Thanks :purple_heart:
I've also fixed a bug in Gatsby which has a side effect that, if we were hammering the binary before, we're certainly going to be smashing it now :p (The fix removes a perf problem relating to artificial delays caused by the cli output, so there's even way less delay now between scheduling image jobs)

Should I push out 8.8.4? Or should we test a bit first?

@kleisauke net-vips might need testing for this change too.

I found a few places in NetVips' test suite that changes metadata without first doing a copy(), which are resolved with https://github.com/kleisauke/net-vips/commit/3cd1bdf55119c9107c2de38d726cbcd5d3a6e436.
Our service also seems to have a few potential problems with modifying metadata, which are resolved with https://github.com/weserv/images/commit/e72b8fde66558f502437c016bab7e15c0117a49b. The segfault does not (yet) seem to have taken place in our production environment.
Note that I still see some GLib warnings while running libvips' test suite, see comment https://github.com/libvips/libvips/pull/1483#issuecomment-559860643.

Thanks for pointing those out, Kleis. I'll fix them over the weekend.

sharp v0.23.4 now available with the changes in commit bb15cd9. I suspect the changes to libvips discussed here will also be required to fully deal with this.
(I expect the future sharp v0.24.0 to provide a prebuilt version of a future libvips with these fixes, either v8.8.4 or v8.9.0.)

There's a pre-release of 8.8.4 here:
https://github.com/libvips/libvips/releases/tag/v8.8.4
I'll add some win binaries and test on mac, but it should be good to go.

@lovell with libvips v8.9.1 available now, can we get a sharp v0.24.1 containing the updated dependency? i'd love to see these segfaults gone ;)

@oezi sharp will always look for and use a globally-installed libvips in preference to its prebuilt copy to allow you to use a newer libvips with an older sharp.

If you could also release 8.9.1 in https://github.com/lovell/sharp-libvips/releases I can test it. Or another easy way to temporarily get a contained build (on linux, ubuntu) to test drive. I'm fine with inline replacing the cached 8.9.0 file with a new version as a workaround.
Not sure what your release schedule is/was here? :)

@lovell can you give us any update on this, please?

The yield branch that will become the forthcoming sharp v0.25.0 depends on and provides libvips v8.9.1.
npm i lovell/sharp#yield

sharp v0.25.0 will also include the N-API migration (see #1282) as well as the return of support for 32-bit node.exe on Windows to allow for easier use with Azure.

It seems I can no longer trigger the problem on this branch. 30 or 40 runs on high system load (aka a Zoom call, lol) without signs of trouble. Thank you!

Can confirm, no more segfaults with the yield branch.

This is great news, thank you very much for testing!

sharp v0.25.0 with libvips v8.9.1 is now available; enjoy!

Awesome. Thanks for the effort on your end(s)! :D We expect to upgrade soon and ship this as well.

We've now bumped Sharp in Gatsby, affecting the following versions. Thanks again!

<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>


Add Hatchbox Integration

Add Hatchbox Integration
Overview
This PR adds core Hatchbox integration components following our established GitHub integration patterns.
Core Components

Connection - Handles authentication and API access

API token management
Request authorization
Environment configuration

Client - Coordinates deployment operations

Application creation
Environment configuration
Response handling and validation

Application - Domain representation

Maps Hatchbox API responses
Encapsulates application data
Follows instance-oriented architecture

Implementation Details

Follows Three C's pattern (Connect, Consume, Coerce)
Maintains Sandi Metz principles
Complete test coverage
MVP focused on deployment essentials

File Changes
lib/jumpstart_deploy/hatchbox/
├── connection.rb - API authentication
├── client.rb     - Operation coordination
└── application.rb - Domain representation

spec/lib/jumpstart_deploy/hatchbox/
├── connection_spec.rb
├── client_spec.rb
└── application_spec.rb

Testing

Connection authentication and validation
Client operations and error handling
Application object mapping
Comprehensive integration tests

Next Steps

Add deployment operations
Enhance error handling
Add integration with existing GitHub components
Complete CLI implementation

Closes #ISSUE_NUMBER

Here's a non-technical summary of what we've accomplished:
"We've built the foundation for automatically deploying our apps to Hatchbox. Specifically:

We've established a secure and reliable way to communicate with Hatchbox's service:

Added proper security authentication
Built in automatic retries if the connection fails
Added error handling to give clear feedback if something goes wrong

We're following the same pattern we successfully used with GitHub integration, which means:

Code is consistent and maintainable
Future developers will find it familiar and easy to work with
We can easily add more features as needed

We've written comprehensive tests to ensure everything works correctly.

The next step will be building the features to actually deploy apps, but we now have the secure foundation in place to do that."
If they ask about specifics of the changes, you can mention we:

Switched to a more robust communication library (Faraday) that's an industry standard
Added proper error handling and retry logic for reliability
Made sure all the code is thoroughly tested

Would you like me to:

Add any other points to this summary?
Explain any part in more detail?
Draft a more technical version for the development team?


Fix test failure on rubocop 1.40.0

Perhaps the following change is the cause.

https://github.com/rubocop/rubocop/pull/11247

I ran into this failure at the following Pull Request:

https://github.com/rubocop/rubocop-rspec/pull/1509
https://github.com/rubocop/rubocop-rspec/actions/runs/3652710709/jobs/6171400421

Before submitting the PR make sure the following are checked:

[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[ ] Added tests.
[ ] Updated documentation.
[ ] Added an entry to the CHANGELOG.md if the new code introduces user-observable changes.
[x] The build (bundle exec rake) passes (be sure to run this locally, since it may produce updated documentation that you will need to commit).

Thank you!


Preconditions.checkState(x != null)

In 0.7.8 it's not quite smart enough to follow this
    public AppEngineWebXmlProjectStageConfiguration build() {
      Preconditions.checkState(sourceDirectory != null, "No source directory supplied");
      Preconditions.checkState(stagingDirectory != null, "No staging directory supplied");

      return new AppEngineWebXmlProjectStageConfiguration(
          this.sourceDirectory,
          this.stagingDirectory,
          this.dockerfile,
          this.enableQuickstart,
          this.disableUpdateCheck,
          this.enableJarSplitting,
          this.jarSplittingExcludes,
          this.compileEncoding,
          this.deleteJsps,
          this.enableJarClasses,
          this.disableJarJsps,
          this.runtime);
    }

That is, in a Builder inner class I'm using Preconditions.checkState (from Guava) instead of Preconditions.checkNotNull because an IllegalStateException is slightly more appropriate here than a NullPointerException. Whatever the exception type, though, the return line can't be reached when sourceDirectory or stagingDirectory are null. However Nullaway thinks these can be reached in that case.

[X ] If you think you found a bug, please include a code sample that reproduces the problem. A test case that reproduces the issue is preferred. A stack trace alone is ok but may not contain enough context for us to address the issue.

[ X] Please include the library version number, including the minor and patch version, in the issue text.

Hi @elharo, thanks for the report!  At first glance, I don't think it will easy to catch this one without a special-case hack.  We have ways to specify functions that check the nullness of an argument, but nothing yet to inspect boolean arguments more deeply when a function checks their truth.  I think adding this kind of support would be low priority as we haven't seen it much before.
You can work around this issue by writing your own version of checkNotNull that throws an IllegalStateException instead, and annotating that method with @Contract("null -> fail") (see docs).  I'll leave this issue open in case we do decide to add more specialized support in the future.

@lazaroclapp we can close this one yes?

I believe we can, this should be a specific case of the support for checkArgument/checkState in #608 🎉


Add name=None argument to roll function

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template
System information

TensorFlow version 2.0:
Are you willing to contribute it Yes:

Describe the feature and the current behavior/state.
tensorflow.python.ops.manip_ops.roll does not take name=None as parameter the current API
does not allow that
Will this change the current api? How?
Yes, roll function in tensorflow.python.ops.manip_ops.roll will take name=None as argument which will return a tensor with name=name and not the default one roll_n
File where roll exists
manip_ops.py

There's an open PR, no need to assign the issue.

There's an open PR, no need to assign the issue.

@Gurpreetsingh9465 We see that PR #27075 is merged , please let us know if we can close this issue ?


Remove max_length from RewardDataCollatorWithPadding

What does this PR do?
Fixes #2118
max_length isn't used by tokenizer.pad when no truncation strategy not padding='max_length'. So we can remove it.
it raises the following warning:
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:2888: UserWarning: `max_length` is ignored when `padding`=`True` and there is no truncation strategy. To pad to max length, use `padding='max_length'`.

Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[x] Did you read the contributor guideline,
Pull Request section?
[x] Was this discussed/approved via a GitHub issue? Please add a link
to it if that's the case.
[x] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines.
[ ] Did you write any new necessary tests?

Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.


Emissions Reduction/Removals (ERRs)Calculation Pre-Calculator in Guardian

Problem description
The current process does not make it clear to stakeholders the estimated ERR's in Guardian
Requirements
Based on input data a estimated ERRs should be able to be queried in project development.
Definition of done
ER calculation should be visible in the Guardian in real time based on input data by all parties
Acceptance criteria
The ERR calculation should be consistent with defined formula by the methodology/registry from baseline through issuance.

@anvabr this request comes from a registry^

This should work similar to a Nerd Wallet like quick calculator, effectively for key inputs (1-2 levels lower in the schema tree) to manually bypass all data entry to determine projected credits. For example in a forestry (ARR/IFM/REDD+) methodology a project proponent may want to determine their carbon yield with key assumptions (not manually entering each data point for a monitoring report. This could bypass to an output of calculations that feed the higher order methodology formula leading to a projected credit issuance.

Ideally it could use a similar interface for bottom up attributes to allow PP to click into a schema tree UI and determine which level of detail would enable a pre-issuance calculator for a published methodology. Instead of creating an attribute (tag) for a production scenario, this would determine an output in dry run. For clarity this feature would be on published policies in dry run, it should be enabled to allow project proponents to save and ideally export this to add more details later (maybe in excel if that's easily feasible).


Implement Term through proxy path

I actually changed my mind about #3: rather than implementing and returning instances of (our own or another) Term, a proxy path should implement the Term properties and methods.
So we basically need handlers for:

[ ] termType
[ ] value
[ ] equals
[ ] language
[ ] datatype

and these should defer to subject internally.
We can implement all of them with a resolver, which I would call DataResolver. Then we wire it up like new DataResolver('subject'), and this resolver tries to resolve all properties from the internal subject field. Optionally, we might want to pass a second parameter to limit it to the 5 keys above.
Alternatively, given that it is only 5 fields, we might wire them up as individual handlers as new DataHandler('subject', 'termType'). On second thought, that might actually be my preference.

Implemented by #13.


User Avatar not displayed in Notification

Hi,
when I kill the app via launcher, the next notification will always have the default avatar. If you open the app again, notifications seem to be normal again.
Reproduceable on the latest development branch.
Best regards,
Markus

Hi @markusschweitzer ,
Yes, this is a known issue (by the developer). Hopefully this is fixed in RiotX, but sadly there is a few chance that it will be fixed in the good old Riot-Android

Hi @markusschweitzer ,
Yes, this is a known issue (by the developer). Hopefully this is fixed in RiotX, but sadly there is a few chance that it will be fixed in the good old Riot-Android

Any clue why the avatar image is not shown? I could fix it myself and create a pull request, but someone needs to point me in the right direction :)


Enable post-hoc evaluations through defining and using output value in TestSuite

What this PR is trying to solve
Currently node module evaluate() works by running the values defined in TestSuite.tests through the prompts or providers defined in TestSuite.prompts and TestSuite.providers, and then evaluating the output through the assertions defined in TestSuite.assert.
This evaluation process works well when we are running offline experiments or tests. However, this process would not allow us to efficiently evaluate LLM systems that are already in production (post-hoc or post-execution evaluations) where the output for each of the inputs/tests are already produced in production, as evaluate() expects to obtain those output values by first running the inputs/tests through a prompt or through a provider defined by the user in TestSuite (instead of leveraging the output values already produced in production and stored in a database or file).
IMO we should allow users to define the output value for a given TestCase, and if it is defined by the user then evaluate() should directly treat that as the output value in ProviderResponse and continue running runEval without calling callApi first. Allowing this would open up the ability for evaluate() to more efficiently evaluate the scenario described above, while taking advantage of all the assertions available today.
How does this PR solves it

Add an output field to TestCase where user can define the output that they have for a given test case
Add a check in runEval whether a given test case being handled already contained an output value or not. If that is the case then skip calling the callApi function. Otherwise continue to run callApi as per usual.
Continue with the rest of the runEval function.

LGTM - maybe we should rename it to providerOutput

@typpo I modified the element name from output to providerOutput


Gitignore node_modules

The first step of DEVELOPING.md is to run npm install, which fills up node_modules with dependency artifacts that shouldn't be tracked by git.

Target 0.6.x for this? Otherwise LGTM

Oh boy, turns out changing the base on GitHub's interface loads doesn't do what I expected it to. I'll just close this PR and reopen it with a commit based on 0.6.x.


0.16 math errors

I just ran some code that finds the roots of numbers (reals and complex) that runs perfectly well in 0.15 but 0.16  gives this error:
jzakiya@localhost ~ $ crystal "Crystal Projects"/roots.cr
13
cannot raise an integer to a negative integer power, use floats for that (ArgumentError)
[134718745] *CallStack::unwind:Array(Pointer(Void)) +121
[134718607] *CallStack#initialize:Array(Pointer(Void)) +31
[134718549] *CallStack::new:CallStack +69
[134718431] *Exception +63
[134718332] *ArgumentError#initialize<ArgumentError, String>:CallStack +60
[134718251] *ArgumentError::new:ArgumentError +139
[134735792] *Int32 +64
[134734974] *Int32 +366
[134669731] ???
[134692141] main +77
[3075092222] __libc_start_main +222
[134666897] ???
I haven't tried to track down where it starts giving this error, but of course you can raise any number to a negative exponent (integer, real, or complex), which, again, was done correctly in 0.15.

Hi,
After each release I recommend reading the changelog and see what are the breaking changes:

(breaking change) Int#**(Int) now returns an integer, and raises if the argument is negative. Use a float base or exponent for negative exponents to work

And I think it makes more sense this way. For example 3 / 2 gives an Int, 3 * 4 gives an int, etc. In general operations between ints give ints. You can also read this issue for the full discussion.

I guess, the main complaint here is that the backtrace dump is not very readable.


chore: mapping still mention usage

🧭 What and Why
🎟 JIRA Ticket: -
Changes included:
just realized this one is still there

omg

I forgot to ai validate this file


Azure AD connect with MFA enabled.

Hi,
Currently Microsoft is enforcing MFA for all users, so i wanted to know is there a way where we can connect samba4 and azure ad with MFA enabled.

it must be possible
everything is played here with the token :
https://github.com/sfonteneau/AADInternals_python/blob/90dd2c0d7db4bea64f2a6709ca4b957f85fda0e9/AADInternals.py#L33C24-L33C24
if I replace UserasCredentials with IDTokenCredentials it should be possible :
https://learn.microsoft.com/en-us/python/api/msrestazure/msrestazure.azure_active_directory.aadtokencredentials?view=azure-python
but it's a little blurry for me

Works in aadinternal
use old token for obtain new token

from adal import AuthenticationContext

    def __init__(self, mail=None, password=None,proxies={}):
        self.proxies=proxies

        cache_file=r'C:\waptdev\test'

        if not os.path.isfile(cache_file):
            self.credentials = UserPassCredentials(mail, password, resource="https://graph.windows.net",proxies=proxies)
            token_response = self.credentials.token
            print(token_response)
            with open(cache_file,'w') as f:
                f.write(json.dumps(self.credentials.token))

        else:

            with open(cache_file,'r') as f:
                old_token=json.loads(f.read())

            context = AuthenticationContext("https://login.microsoftonline.com/" + old_token['tenant_id'])
            token_response = context.acquire_token_with_refresh_token(
                old_token['refresh_token'],
                old_token['_client_id'],
                old_token['resource']
                )
            token_response['tenant_id'] = old_token['tenant_id']
            token_response['resource'] = old_token['resource']
            token_response['_client_id'] = old_token['_client_id']
            token_response['refresh_token'] = old_token['refresh_token']

            token_response['access_token'] = token_response['accessToken']

            with open(cache_file,'w') as f:
                f.write(json.dumps(token_response))

        self.tenant_id = token_response['tenant_id']
        self.token = token_response['access_token']

to be continued ...

tennantid = "000000-00000-00000-00000-00000"

TEMPLATE_AUTHZ_URL = ('https://login.windows.net/{}/oauth2/authorize?' +
                      'response_type=code&client_id={}&redirect_uri={}&' +
                      'state={}&resource={}')

authorization_url = TEMPLATE_AUTHZ_URL.format(
    tennantid),
    "04b07795-8ddb-461a-bbee-02f9e1bf7b46",
    "http://localhost",
    12345,
    "2ff814a6-3304-4ab8-85cb-cd0e6f879c1d")

print(authorization_url)

code = "codefakeinurl"

context = AuthenticationContext("https://login.microsoftonline.com/" + tennantid)
token_response = context.acquire_token_with_authorization_code(code,"http://localhost",'https://graph.windows.net',"04b07795-8ddb-461a-bbee-02f9e1bf7b46")

For first time login with MFA

use AADTokenCredentials for continue use GraphRbacManagementClient

Works : https://github.com/sfonteneau/AADInternals_python/commit/18d6a39be1eb29f37ccf9236270d60ea561b737c
remains to be implemented in AzureADConnect_Samba4

Works :
https://github.com/sfonteneau/AzureADConnect_Samba4/commit/2680395a53f32048709067492f45148b44d3780b
Please note that if the token is not used regularly, it may expire and require multi-factor authentication again.
If used in crontab it should work


upgrading to latest paho client, fixing error message comparison

I was running into issues when a connection timeout would occur between the mqtt writer and broker, it wouldn't successfully reconnect. Paho has released a pretty big upgrade (over a year) with a ton of fixes.
There is one particular change that I believe benthos does not need to worry about but would like you to confirm. It's something that they call out in this release - specifically changing from buffered to unbuffered channels. https://github.com/eclipse/paho.mqtt.golang/releases/tag/v1.3.0
These changes fixed the reconnection issue that I was facing
Without fix (sorry I stripped the timestamp in my notes):
mqtt: [net]     outgoing waiting for an outbound message
mqtt: [client]  enter Publish
mqtt: [client]  sending publish message, topic:dev/devices/bb4d12d6-7afc-42f6-b284-5c329cd5dbc7
mqtt: [net]     obound msg to write0
mqtt: [net]     outgoing obound reporting error write tcp <IP_ADDRESS>:53761-><IP_ADDRESS>:1883: i/o timeout
mqtt: [net]     outgoing waiting for an outbound message
mqtt: [client]  Connect comms goroutine - error triggeredwrite tcp <IP_ADDRESS>:53761-><IP_ADDRESS>:1883: i/o timeout
Failed to send message to mqtt: write tcp <IP_ADDRESS>:53761-><IP_ADDRESS>:1883: i/o timeout
mqtt: [client]  internalConnLost called
mqtt: [client]  stopCommsWorkers called
mqtt: [pinger]  keepalive stopped
mqtt: [net]     startIncomingComms: ibound complete
mqtt: [client]  stopCommsWorkers waiting for workers
mqtt: [client]  internalConnLost waiting on workers
mqtt: [net]     incoming complete
mqtt: [client]  enter Publish
mqtt: [client]  startCommsWorkers output redirector finished
mqtt: [net]     outgoing waiting for an outbound message
mqtt: [net]     outgoing waiting for an outbound message
mqtt: [router]  matchAndDispatch exiting
mqtt: [net]     startIncomingComms goroutine complete
mqtt: [client]  stopCommsWorkers waiting for comms
mqtt: [net]     outgoing waiting for an outbound message
mqtt: [net]     outgoing comms stopping
mqtt: [client]  sending publish message, topic:dev/devices/e1615984-57b6-4e83-ae91-a51586c5dc46
mqtt: [net]     startComms closing outError
mqtt: [client]  incoming comms goroutine done
mqtt: [client]  stopCommsWorkers done
mqtt: [client]  internalConnLost workers stopped
mqtt: [msgids]  cleaned up
mqtt: [client]  internalConnLost complete
mqtt: [client]  Disconnect() called but not connected (disconnected/reconnecting)
mqtt: [client]  stopCommsWorkers called
mqtt: [client]  stopCommsWorkers done (not running)
Connection lost due to: write tcp <IP_ADDRESS>:53761-><IP_ADDRESS>:1883: i/o timeout
Failed to send message to mqtt: publish was broken by timeout
mqtt: [client]  enter Publish
Failed to send message to mqtt: not Connected
mqtt: [client]  enter Publish
Failed to send message to mqtt: not Connected
mqtt: [client]  enter Publish
Failed to send message to mqtt: not Connected

With fix
2021-01-19T14:44:34-06:00 mqtt: [net]     outgoing waiting for an outbound message
2021-01-19T14:44:34-06:00 mqtt: [client]  enter Publish
2021-01-19T14:44:34-06:00 mqtt: [client]  sending publish message, topic:dev/devices/6f8e5a8d-b0c4-4cad-9e42-8d30a65d17cd
2021-01-19T14:44:34-06:00 mqtt: [net]     obound msg to write0
2021-01-19T14:44:34-06:00 mqtt: [net]     obound wrote msg, id:0
2021-01-19T14:44:34-06:00 mqtt: [net]     outgoing waiting for an outbound message
2021-01-19T14:44:34-06:00 mqtt: [client]  enter Publish
2021-01-19T14:44:34-06:00 mqtt: [client]  sending publish message, topic:dev/devices/6f8e5a8d-b0c4-4cad-9e42-8d30a65d17cd
2021-01-19T14:44:34-06:00 mqtt: [net]     obound msg to write0
2021-01-19T14:44:35-06:00 mqtt: [net]     outgoing obound reporting error write tcp <IP_ADDRESS>:63272-><IP_ADDRESS>:1883: i/o timeout
2021-01-19T14:44:35-06:00 mqtt: [net]     outgoing waiting for an outbound message
2021-01-19T14:44:35-06:00 Failed to send message to mqtt: write tcp <IP_ADDRESS>:63272-><IP_ADDRESS>:1883: i/o timeout
2021-01-19T14:44:35-06:00 mqtt: [client]  Connect comms goroutine - error triggeredwrite tcp <IP_ADDRESS>:63272-><IP_ADDRESS>:1883: i/o timeout
2021-01-19T14:44:35-06:00 mqtt: [client]  internalConnLost called
2021-01-19T14:44:35-06:00 mqtt: [client]  stopCommsWorkers called
2021-01-19T14:44:35-06:00 mqtt: [pinger]  keepalive stopped
2021-01-19T14:44:35-06:00 mqtt: [net]     incoming complete
2021-01-19T14:44:35-06:00 mqtt: [client]  enter Publish
2021-01-19T14:44:35-06:00 mqtt: [client]  internalConnLost waiting on workers
2021-01-19T14:44:35-06:00 mqtt: [client]  stopCommsWorkers waiting for workers
2021-01-19T14:44:35-06:00 mqtt: [net]     startIncomingComms: ibound complete
2021-01-19T14:44:35-06:00 mqtt: [net]     startIncomingComms goroutine complete
2021-01-19T14:44:35-06:00 mqtt: [client]  sending publish message, topic:dev/devices/b62e573c-f86a-445f-9d6b-64bb5b5f27d8
2021-01-19T14:44:35-06:00 mqtt: [net]     outgoing waiting for an outbound message
2021-01-19T14:44:35-06:00 mqtt: [net]     outgoing waiting for an outbound message
2021-01-19T14:44:35-06:00 mqtt: [net]     outgoing waiting for an outbound message
2021-01-19T14:44:35-06:00 mqtt: [net]     outgoing comms stopping
2021-01-19T14:44:35-06:00 mqtt: [client]  startCommsWorkers output redirector finished
2021-01-19T14:44:35-06:00 mqtt: [client]  incoming comms goroutine done
2021-01-19T14:44:35-06:00 mqtt: [net]     startComms closing outError
2021-01-19T14:44:35-06:00 mqtt: [client]  stopCommsWorkers waiting for comms
2021-01-19T14:44:35-06:00 mqtt: [client]  stopCommsWorkers done
2021-01-19T14:44:35-06:00 mqtt: [router]  matchAndDispatch exiting
2021-01-19T14:44:35-06:00 mqtt: [client]  internalConnLost workers stopped
2021-01-19T14:44:35-06:00 mqtt: [msgids]  cleaned up
2021-01-19T14:44:35-06:00 mqtt: [client]  internalConnLost complete
2021-01-19T14:44:35-06:00 mqtt: [client]  Disconnect() called but not connected (disconnected/reconnecting)
2021-01-19T14:44:35-06:00 mqtt: [client]  stopCommsWorkers called
2021-01-19T14:44:35-06:00 mqtt: [client]  stopCommsWorkers done (not running)
2021-01-19T14:44:35-06:00 Connection lost due to: write tcp <IP_ADDRESS>:63272-><IP_ADDRESS>:1883: i/o timeout
2021-01-19T14:44:36-06:00 Failed to send message to mqtt: publish was broken by timeout
2021-01-19T14:44:36-06:00 mqtt: [client]  enter Publish
2021-01-19T14:44:36-06:00 mqtt: [client]  Connect()
2021-01-19T14:44:36-06:00 mqtt: [store]   memorystore initialized
2021-01-19T14:44:36-06:00 mqtt: [client]  about to write new connect msg
2021-01-19T14:44:36-06:00 mqtt: [client]  socket connected to broker
2021-01-19T14:44:36-06:00 mqtt: [client]  Using MQTT 3.1.1 protocol
2021-01-19T14:44:36-06:00 mqtt: [net]     connect started
2021-01-19T14:44:39-06:00 mqtt: [net]     received connack
2021-01-19T14:44:39-06:00 mqtt: [client]  startCommsWorkers called
2021-01-19T14:44:39-06:00 mqtt: [client]  client is connected/reconnected
2021-01-19T14:44:39-06:00 mqtt: [net]     incoming started
2021-01-19T14:44:39-06:00 mqtt: [net]     startIncomingComms started
2021-01-19T14:44:39-06:00 mqtt: [net]     outgoing started
2021-01-19T14:44:39-06:00 mqtt: [net]     startComms started
2021-01-19T14:44:39-06:00 mqtt: [client]  startCommsWorkers done
2021-01-19T14:44:39-06:00 mqtt: [store]   memorystore wiped
2021-01-19T14:44:39-06:00 mqtt: [client]  exit startClient
2021-01-19T14:44:39-06:00 mqtt: [net]     logic waiting for msg on ibound
2021-01-19T14:44:39-06:00 mqtt: [net]     startIncomingComms: inboundFromStore complete
2021-01-19T14:44:39-06:00 mqtt: [net]     logic waiting for msg on ibound
2021-01-19T14:44:39-06:00 mqtt: [client]  enter Publish
2021-01-19T14:44:39-06:00 mqtt: [client]  sending publish message, topic:dev/devices/df3c61d9-8c02-47ea-b2f6-fc2f28fb7b90
2021-01-19T14:44:39-06:00 mqtt: [pinger]  keepalive starting
2021-01-19T14:44:39-06:00 mqtt: [net]     outgoing waiting for an outbound message
2021-01-19T14:44:39-06:00 mqtt: [net]     obound msg to write0
2021-01-19T14:44:39-06:00 mqtt: [net]     obound wrote msg, id:0
2021-01-19T14:44:39-06:00 mqtt: [net]     outgoing waiting for an outbound message
2021-01-19T14:44:39-06:00 mqtt: [client]  enter Publish
2021-01-19T14:44:39-06:00 mqtt: [client]  sending publish message, topic:dev/devices/6f8e5a8d-b0c4-4cad-9e42-8d30a65d17cd
2021-01-19T14:44:39-06:00 mqtt: [net]     obound msg to write0
2021-01-19T14:44:39-06:00 mqtt: [net]     obound wrote msg, id:0
2021-01-19T14:44:39-06:00 mqtt: [net]     outgoing waiting for an outbound message

Thanks @nwest1, we should be fine in regards to the non-buffered channels.

Thanks @nwest1, we should be fine in regards to the non-buffered channels.


Added Twiddler product instances up to arity 26

There should be less issue with row decoding.
I view that as a band-aid as a more generic solution base on NonEmptyTuple would have been nicer. On my radar but right now the compiler is not kind with me.

FYI in Scala 3 you can use .pmap and .pcontramap, which don't need Twiddler.

Thanks for your quick reply. That's perfect :)
Do we want to close this or keep it as a superset of  #656 ?

@vbergeron Thanks! I lean towards closing this one and sticking with 16 as max built-in arity unless folks really need it for cross-compilation with Scala 2 code using Skunk. For Scala 3 only code bases, we should encourage use of *: and pmap.


11.0 port from 10.0

Travis reports:
************* Module website_debranding
website_debranding/__init__.py:1: [E7991(manifest-image), ] Module's image is absent or it's not in images/ folder. Check attribute "images" in module's __manifest__.py
website_debranding/__init__.py:1: [E7992(absent-index-html), ] File: static/description/index.html is absent in module. Contact Technical Writers for assistance

wontfix

I approve to merge it now


examples: Fix 2-stage install CoreOS and etcd_proxy setup

4GB disk is too small and routinely fails in libvirt
Depending on build env, dnsmasq ACI can have an old /etc/resolv.conf
DNS entry embedded. Can rebuild to fix, but use --dns=<IP_ADDRESS> too
Fix issue where etcd_proxy profile failed to pull metadata
so etcd_proxy wasn't configured after CoreOS install

@gyuho @xiang90 FYI, https://github.com/coreos/coreos-baremetal/blob/master/Documentation/getting-started-rkt.md with the coreos-install.yaml config validates a 3-node etcd and proxy setup on bare metal.


Implementing Linear Model of Coregionalization (LMC)

Hello,
I've been trying to implement an LMC (e.g. equations 18-20 in https://arxiv.org/abs/1106.6251 -Kernels for Vector-Valued Functions: a Review), but I'm having trouble specifying multiple latent functions in my model.
To be specific, I have a single dimensional input of times and a multi-dimensional output of values. I'm trying to set up a coregionalized model such that:
f_1 = a_1 u_1(t) + b_1 u_2(t)
f_2 = a_2 u_1(t) + b_2 u_2(t)
f_3 = a_3 u_1(t) + b_3 u_2(t)
where f_i are my outputs, u_q are my latent functions (Q=2 here) and a/b are variables that compose each coregionalized matrix B_q. Each latent random function needs a covariance kernel specified for it though, and in the context of GPflow it looks like I can only specify one covariance kernel:
in_dim, out_dim = 1, 3
coreg = gpflow.kernels.Coregion(in_dim, output_dim=out_dim, rank=out_dim, active_dims=[1])
k1 = gpflow.kernels.Exponential(1)
kern = k1 * coreg

m = gpflow.svgp.SVGP(X, Y, kern=kern, likelihood=lik, num_latent=2, Z=X[::3].copy())

If I try to make kern a list, e.g. [k1*coreg, k2*coreg], I get an error, and making k a sum of kernels (e.g. k = gpflow.kernels.Exponential(1) + gpflow.kernels.Matern32(1)) seems to just complexify a single latent function, vs. specify kernels for each latent function. I saw this notebook - http://gpflow.readthedocs.io/en/latest/notebooks/GPLVM.html - and looks like it might be useful, but I couldn't quite follow it. I see no mention of ekernels in the documentation, and to my knowledge active_dims specify the input dimension (not output dimension), which in my case is 1.
Is someone able to advise me how to proceed?
Thanks in advance,
Ari

You were on the right track with a list of kernels. In order to let GPflow know about these parameters, you should store them in a ParamList.

@markvdw Thanks for your speedy reply. If I give that a try:
in_dim, out_dim = 1, 3
coreg = gpflow.kernels.Coregion(in_dim, output_dim=out_dim, rank=out_dim, active_dims=[1])
k1 = gpflow.kernels.Exponential(1) * coreg
k2 = gpflow.kernels.Matern32(1) * coreg
kern = ParamList([k1,k2])

m = gpflow.svgp.SVGP(X, Y, kern=kern, likelihood=lik, num_latent=2, Z=X[::3].copy())
m.optimize(maxiter=500)

I end up with an error:
/storage/home/ajs725/tf-ml/lib/python2.7/site-packages/gpflow/conditionals.pyc in conditional(Xnew, X, kern, f, full_cov, q_sqrt, whiten)
     62     num_data = tf.shape(X)[0]  # M
     63     num_func = tf.shape(f)[1]  # K
---> 64     Kmn = kern.K(X, Xnew)
     65     Kmm = kern.K(X) + tf.eye(num_data, dtype=float_type) * settings.numerics.jitter_level
     66     Lm = tf.cholesky(Kmm)

/storage/home/ajs725/tf-ml/lib/python2.7/site-packages/gpflow/param.pyc in __getattribute__(self, key)
    730         will have aname scope applied.
    731         """
--> 732         o = object.__getattribute__(self, key)
    733
    734         # if _tf_mode is False, or there is no _tf_mode, just return the object as normal.

AttributeError: 'ParamList' object has no attribute 'K'

Any idea?
Thanks very much

Ah yes, SVGP does not know how to use multiple kernels for different latent GPs. You'll have to adapt SVGP in your own way for that.
I'm starting to see what you want to do. You can definitely implement this in GPflow, but it may require deriving your own inference, and implementing your own model. Unless your case completely subsumed by the coregionalization kernel? By placing the output as an argument on the input, you can build a large covariance matrix for all outputs, which allows using SVGP, I believe. But sadly, I don't know the details, so someone else will have to chip in here.

There might be some fancy way to subsume all the latent kernels into one master kernel and feed that to SVGP, I'd have to think about it. If possible that would probably be the quickest answer to my problem. Thanks for the replies, hopefully someone else will chime in.

Perhaps #502 is relevant

@hughsalimbeni thanks for the suggestion. I think I'm getting close but still not quite. My current code is:
X = np.vstack((t,l)).T          #(100,2) matrix: first column=X values, second column=dimension number (0,1,2)
Y = np.vstack((y,l)).T         #(100,2) matrix: first column=Y values, second column=dimension number (0,1,2)

idim, odim = 1, 3    #input, output dimensions
coreg1 = gpflow.kernels.Coregion(idim, output_dim=odim, rank=1, active_dims=[1])   #coregionalization matrix
coreg2 = gpflow.kernels.Coregion(idim, output_dim=odim, rank=1, active_dims=[1])
coreg3 = gpflow.kernels.Coregion(idim, output_dim=odim, rank=1, active_dims=[1])

kern_list = [gpflow.kernels.Exponential(1)*coreg1,
             gpflow.kernels.Matern32(1)*coreg2,
             gpflow.kernels.Exponential(1)*coreg3]

lik = gpflow.likelihoods.SwitchedLikelihood([gpflow.likelihoods.StudentT()]*odim)

model_multi = SVGPMultiKernel(X, Y, kern_list, lik, X.copy(), num_latent=len(kern_list))

when I try to optimize, I get a long error that starts with (I could post the full error if you're interested):
InvalidArgumentError: Incompatible shapes: [100,20] vs. [300,20]
When I change the number of kernels in kern_list to X (i.e. the number of latent functions, each with it's own unique kernel) then the error changes accordingly to InvalidArgumentError: Incompatible shapes: [100,20] vs. [X00,20]. So it seems like gpflow is not interpreting these kernels quite correctly, and multiplying the number of outputs by each kernel. Any idea how to proceed?
Thanks.

Upon further testing, it seems like this issue has something to do with num_latent not being properly understood. If I make a single latent co-regionalized model:
#arrange data
X = np.vstack((t,l)).T
Y = np.vstack((y,l)).T

idim, odim = 1, 3
coreg = gpflow.kernels.Coregion(idim, output_dim=odim, rank=1, active_dims=[1])
kern = gpflow.kernels.Exponential(1) * coreg
lik = gpflow.likelihoods.SwitchedLikelihood([gpflow.likelihoods.StudentT()]*odim)

m = gpflow.svgp.SVGP(X, Y, kern=kern, likelihood=lik, num_latent=1, Z=X[::3].copy())

everything compiles and optimizes fine. However, if I change num_latent=3 and execute the exact same code I get the error from above, i.e. InvalidArgumentError: Incompatible shapes: [100,20] vs. [X00,20] where X=num_latent.

Hi @silburt ,
We should document num)_latent better. This is the number of independent, aligned GPs that the model has. It's mostly useful in the GPLVM and in multi-class classification.
Let's ignore inference (GPR/SVGP) and likelihoods for a second, and implement the intrinsic and linear models of coregionalization through a kernel. You should be able to apply the kernel to any inference class (GPR, ...) if you're okay with a fixed likelihood.
I'm looking at eqn. 18 in https://arxiv.org/pdf/1106.6251.pdf, and letting
D=3, Q=2, R_q=[1, 2]
D=3
Q=2
R = [1, 2]
input_dim=1

coregions = [gpflow.kernels.Coregion(input_dim, output_dim=D, rank=R[q], active_dims=[1])
             for q in range(Q)]
base_kernels = [gpflow.kernels.Matern32(input_dim, lengthscales=np.random.randn(1)
                for q in range(Q)]
k = coregions[0] * base_kernels[1] + coregions[1] * base_kernels[1]

Notes:

I used a random lengthscale to initialize each kernel. Otherwise, there will be a symmetry issue with the optimization, and you'll find that the two Materns always have the same lengthscale

Our implementation is slightly different to Alvarez's: the B matrix in our Coregion kernel is always positive definite, constructed as

$ B[i, j] = \sum_r W[i, r] * W[j, r] + \delta_{i,j}\kappa_i$
Alvarez does not have those kappa terms. You could always force them to be zero (or small) and untrainable.

Inducing points may work for your application, or you may want to have a look a Kronecker structured algebra, if your outputs are all observed simultaneously. GPflow doesn't have any support for that at the moment, but I think it's a feature we'd welcome!

@jameshensman - last question: I still get the same error shown in https://github.com/GPflow/GPflow/issues/563#issuecomment-346080533 if num_latent > 1. You mention that num_latent is the number of independent, aligned GPs that the model has, which in my case I thought would be equal to Q? Maybe num_latent is referring to something different than Q, the number of latent functions in my model?

@silburt Could you solve the issue?

@orpia I have a notebook showing a version of what I got able to work here, feel free to take a look - https://github.com/silburt/gp_nn/blob/master/Coregion-py3.ipynb

Hi,
Thanks!
On Thu, Aug 16, 2018 at 10:09 AM Ari Silburt<EMAIL_ADDRESS>wrote:

@orpia https://github.com/orpia I have a notebook showing a version of
what I got able to work here, feel free to take a look -
https://github.com/silburt/gp_nn/blob/master/Coregion-py3.ipynb
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GPflow/GPflow/issues/563#issuecomment-413599441, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AI4lsJqO4lYHCU1e5MMnw7e7iNPYX04hks5uRZlFgaJpZM4QiU08
.


ContactListCacheTest Done Sayfullina Zarina

Coverage increased (+11.9%) to 62.581% when pulling 40a20ccaa262398540aeb61875916cac384f87c6 on elBroom:lecture04 into 1f005511cdff2990e3c484daf5158496734cc1e5 on rybalkinsd:lecture04.


Add new listener for config change

Description of the Change
Add a new listener {onDidUpdate} in atom.config, it's different from {:onDidChange} in that it will always be called, regardless whether the new value is different than the old. This new listener is used to solve the problem that when a layered project specific settings override the global settings, the {:onDidChange} won't work cause the global settings was override and the callback never occur even if the user changes the setting in menu or global config file. The new listener, however, will observe the global config change as desired.
Alternate Designs

Why Should This Be In Core?
This should be in core because recently atom merged a pull request (#16845 ) that allow per-project settings through {:resetProjectSettings}, the project settings layers the global settings by owning higher priority, the corresponding config change listener should be changed cause when user change the layered global settings the listener should be fired correctly.
Benefits
This change add a config change listener that can behave correctly.
Possible Drawbacks
Verification Process
Applicable Issues
This situation origins from our packages which is currently developed for per-project settings based on {:resetProjectSettings}, user can have a local config file in current project folder, our package will parse it and apply the config on the current window, our plan is when user changes the settings in global, our package should be disabled temporarily so as to apply the global config setting changes that the user made, so we used {:atom.config.onDidChange} to listen to the behavior, but it didn't work as described above.

onDidChange is almost never the right event to subscribe to, in virtually every case atom.config.observe is the right choice as it is fired when initially called and when the value changes.
Does that not work with the project settings?

it's different from {:onDidChange} in that it will always be called, regardless whether the new value is different than the old.

I'm not entirely sure what you mean by this, if the values haven't changed, why should any listener be fired if the value isn't different?

What are the benefits for creating a new listener? What are the drawbacks of changing the existing listeners?

Thank you for your PR!
We haven't gotten a response to the reviews in the comments above. I'm going to close this due to inactivity, but don't hesitate to reach out if you have the time to resolve the conflicts and review comments, we'll be happy to reopen the PR.


Add required_ruby_version to gemspec

The use of the double splat in lib/jsonb_accessor/macro.rb restricts the use of this gem to version 2x of ruby.

Thanks!


Missing levels folder, When we run this application we got the error

You should put your levels in assets/levels folder.
Each level should be .png file (maybe different format, not sure) .|
For example path for 1. level should be "assets/levels/1.png"


Compatibility with rdflib version

It seems changes in rdflib after 6.1.1 removed the preferredLabel method -- we should specify a maximum version here in setup.cfg

@ricklupton
Hi Rick, I've made a new branch with changes to setup.cfg and directives.py (for changes to probs prefix). Can you give me access to push to repository. Thanks

Hi @stephenjboyle thanks for doing this. Unfortunately as per https://github.com/probs-lab/petrochemical-integrated-data/issues/39 I think we might need to fix it and allow versions >= 6.2, <7.
Can you just copy the definition of preferredLabel from here:
https://github.com/RDFLib/rdflib/blob/2fc55b38c40a0f7df6b44989cd52a6b3ce5d1540/rdflib/graph.py#L722-L799
into the file here in this repository, just above where it is used?
And update setup.cfg to specify at least 6.2, less than 7 (to avoid any other possible breaking changes that came in there).
This is a bit of an unusual case, it looks like these methods were supposed to be removed for the major version 6.0, but were missed, which is why they went ahead and removed them for a minor version. Normally we should be "safe" specifying a dependency as a particular major version.
Thanks,
Rick

@stephenjboyle I've invited you to this repository

Closed by #5


UI support of custom values.yaml in the catalog apps

What kind of request is this (question/bug/enhancement/feature request):
UI feature request
Steps to reproduce (least amount of steps as possible):

go to the catalog apps page and click one of the app e.g, DockerRegistry

add an option to use custom user-supplied values.yaml file, e.g,

click launch and ensure the user-supplied values.yaml values will override the default one.

Result:
Ensure the user-supplied values.yaml's values will override the default one.
Environment information

Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): rancher:master
Installation option (single install/HA): single & HA

**Related PR **
https://github.com/rancher/rancher/pull/17576

Version: Master 2.2 (2/19/19)
Confirmed enhancement.
This now treats "Edit as YAML" as actual YAML rather than key pairs as we were doing before. I inspected the POST (creation) and POST (upgrading/editing) of the docker-registry app and the http/API looks good. This adds "valuesYaml" and it will send proper YAML with all the whitespaces/carriage returns.
I tested by Editing as YAML and putting in custom YAML such as replicaCount: 2 and then validate in UI that we have two pods created for example.
Screenshot below. You can see my added YAML below. I tested a few other things as well.


Docs fix: Replace all kubernetes API links

Seems like all docs of v1.22 has been removed from kubernetes docs site,
Right now minimum available version for which docs is available is 1.23.
So I have replaced all the links

Hi @Fluder-Paradyne Thanks for your code. but we use the elastic/crd-ref-docs to generate API docs, So I think just update ./crd-reference-config.yaml and running make manifests will be work.
And could you please change your git commit log to pass the lint check? You can see this document

Hey sorry about that, have fixed it now


DM-45287 Introduce script details page to CM web app

This PR add a new script details page and link script names all over the app to the page.
Group dictionaries are now rendered correctly in the grid in step details page
Collections are filtered in all pages rather than displaying collections from the whole tree down

@mr-asher @fritzm @eacharles Can you please review that PR when you get a chance?


k8s: Added script for setting up k8s nodes

This script allows to configure k8s node disks and tune networking.
Priviledged pods should run it as a entrypoint, and provide two environmental variables:

ROOT_DISK - where Scylla root directory should be mounted
ROOT_MOUNTPOINT - path to directory where host filesystem is mounted

cc. @mmatczuk

@zimnx just for some context, there was some mail thread (or any other form of agreement, with @slivne or @avikivity) that this belongs here ?

@zimnx just for some context, there was some mail thread (or any other form of agreement, with @slivne or @avikivity) that this belongs here ?

This is needed for scylla-operator example for EKS, where we want to configure node using privileged container running as a DeamonSet.

should I point this PR to next branch or master?

should I point this PR to next branch or master?

Should point it to next


npm install node-sass doesn't install latest pre-release

$ npm install --save-dev node-sass
><EMAIL_ADDRESS>install /Users/abc/front/node_modules/node-sass
> node scripts/install.js
# etc

It seems like it does install latest pre-release that is v3.0.0-alpha.0.

npm install node-sass@next

Ok, so, to be clear, alpha.0 is after pre?

Yes.
You can see the list of releases here: https://github.com/sass/node-sass/releases.
The current release is v3.0.0-alpha.1.
You can also subscribe to atom feed (RSS'ish): https://github.com/sass/node-sass/releases.atom and also get email notification via sibbell (see the philosophy behind Sibbell project).

Doesn't P sort after A?
Anyway, thanks for answering. That explains it.

Preview and Alpha are interchangeable, we decided later to use alpha instead of pre.
And last time I checked, it has nothing to do with sorting order..

Ok, so it's the exact same code? They're not interchangeable from a semver perspective.
Precedence for two pre-release versions with the same major, minor, and patch version MUST be determined by comparing each dot separated identifier from left to right until a difference is found as follows: identifiers consisting of only digits are compared numerically and identifiers with letters or hyphens are compared lexically in ASCII sort order.
From http://semver.org/ also see https://github.com/haf/semver/blob/master/spec/semver_spec.rb#L174-L186 for real-life tests for it.

Btw, you should unpublish the pre-release if npm supports that.

Not the exact code, see the release changelog.
They are interchangeable as in you can use either pre or alpha, they imply the same meaning.
As I mentioned earlier, it was later we decided to go with alpha instead of pre, and sticking with alpha in future.
I understand this is against the principles of semver, but the damage has done.

Just use this command https://docs.npmjs.com/cli/unpublish

Or choose another greek character, like zeta.


Redirect HTTP to HTTPS?

Is there a recommended path to redirect all HTTP traffic to HTTPS in AKS?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ada58160-dc50-b977-d3bc-65f92406e16d
Version Independent ID: 5c596e04-fddb-effe-3b72-575f4df42d4e
Content: Create an HTTPS ingress with Azure Kubernetes Service (AKS) cluster
Content Source: articles/aks/ingress-tls.md
Service: container-service
GitHub Login: @iainfoulds
Microsoft Alias: iainfou

We can achieve this using Kubernetes ingress Controllers. In the example we used Nginx as the ingress controller.  We can add annotation to ingress or add options in Config maps to control the behaviour.
If SSL termination happens at the ingress, By default all HTTP requests will be redirected (302) HTTPS.
Annotation which controls this setting is "ingress.kubernetes.io/ssl-redirect"
config map key is ssl-redirect
Default value is True
If SSL termination is happening at the Load balancer, Then ingress will receive all the traffic in HTTP. In that case we can set an annotation which redirects to HTTPS if the request actually came to Load balancer with HTTP (with header "http_x_forwarded_proto" equal to HTTP )
Annotation which controls this setting is "nginx.org/redirect-to-https"
config map key is redirect-to-https
default value is False
All customization options for nginx ingress controller is available here.
Also note that we have 3 nginx ingress controllers. 2 from NGINX Inc and one from Kubernetes.
Differences are listed here

As @jakaruna-MSFT notes, the NGINX ingress controller default behavior is to redirect all traffic to HTTPS. This isn't AKS specific behavior, rather how NGINX decided (somewhat correctly) to redirect all traffic to HTTPS connections.
An example of how to override this behavior is shown in the basic ingress example - https://docs.microsoft.com/en-us/azure/aks/ingress-basic#create-an-ingress-route. On line 7 of the YAML manifest, the ssl-direct is set to false. Otherwise, all traffic would be redirected. If you use a different ingress controller, the behavior may be different, but there are likely similar annotations you can set to control how traffic is redirected.
As there is no doc action here, #please-close


what do i nee to do to be able to modify the Designer UI file and convert to .py file and work when i run main.py?

I try to run the command "pyside6-uic main.ui> ui_main.py" but when i try to run main.py i get the error
> python .\main.py Traceback (most recent call last):   File "C:\Users\iosorio\Code\Modern_GUI_PyDracula_PySide6_or_PyQt6-master\Modern_GUI_PyDracula_PySide6_or_PyQt6-master\main.py", line 170, in <module>     window = MainWindow()   File "C:\Users\iosorio\Code\Modern_GUI_PyDracula_PySide6_or_PyQt6-master\Modern_GUI_PyDracula_PySide6_or_PyQt6-master\main.py", line 38, in __init__     self.ui.setupUi(self)   File "C:\Users\iosorio\Code\Modern_GUI_PyDracula_PySide6_or_PyQt6-master\Modern_GUI_PyDracula_PySide6_or_PyQt6-master\modules\ui_main.py", line 16, in setupUi     MainWindow.setMinimumSize(QtCore.QSize(940, 560)) TypeError: 'PySide6.QtWidgets.QWidget.setMinimumSize' called with wrong argument types:   PySide6.QtWidgets.QWidget.setMinimumSize(QSize) Supported signatures:   PySide6.QtWidgets.QWidget.setMinimumSize(PySide6.QtCore.QSize)   PySide6.QtWidgets.QWidget.setMinimumSize(int, int)
what am i doing wrong? I'm trying to add functionality to the ui but i'd need to rebuild the UI designer file and convert it to .py file

Make sure the imports on top are
from PySide6.QtCore import *
from PySide6.QtGui import *
from PySide6.QtWidgets import *

from . resources_rc import *

it might be importing for example import PySide6.QtCore.QSize, which you then need to use it as PySide6.QtCore.QSize, while it tries to use it as QSize.

the issue here is not the importing the issue is how to generate ui_main.py from main.ui and I am in the similar boat, usually I use pyuic5 but since this is pyside6 i dont know

the answer is you must run this command and then replace ui_main.py in the modules folder but you must modify a line
pyside6-uic  main.ui -o ui_main.py

In the new ui_main.py
import resources_rc <---- this line changes into
from . resources_rc import * <-- into this


Memory leak trying to create a while loop with promises

I'm trying to create a while loop using promises and I'm getting a memory leak.
An example of the pattern I'm using is
var promiseFor = BPromise.method(function(delay, condition, action, value) {
  if (!condition(value)) {
    return value;
  } else {
    return action(value).delay(delay).then(promiseFor.bind(null, delay, condition, action));
  }
});

var condition = function(value) {
  return value != 100000;
};

var action = function(value) {
  return BPromise.delay(10)
    .then(function() {
      console.log('value:' + value);
      return value + 1;
    });
};

var delay = 0;

promiseFor(delay, condition, action, 0)
  .then(function() {
    console.log('done');
  })
  .catch(function(e) {
    console.error(e.message);
  });

When I run this the memory used by node keeps increasing.
I'm using bluebird 2.9.10.
I've tried another approach to avoid the recursive nature of the approach above but it also has a leak.
Any suggestions on how to correctly implement a promise while loop ?

Here is how I'd implement a while loop using bluebird:
var promiseWhile  = BPromise.coroutine(function(delay, condition, action, value) {
    while(!condition(value){
          value = yield condition(action(value));
          yield Promise.delay(delay);
    }
});

Looks nice but unfortunately for now we are using node 0.10 so we can't use generators.

There is no need to implement any helper in the first place, just use recursion
(function loop(value) {
  if (value != 100000) {
    return Promise.delay(10).then(function() {
      console.log('value:' + value);
      return value + 1;
    }).then(loop);
  }
})(0);

@petkaantonov unfortunately this approach also has a leak. On windows the node process gained about 14 MB in 20 mins. I'm looking for some way to keep running a promise function over a long period of time without leaking memory.

@colmaengus It appears you are not actually using 2.9.10 then. Can you log this for me and let us know what it logs:
console.log(BPromise.prototype._rejectCallback + "");

function (reason, synchronous, shouldNotMarkOriginatingFromRejection) {
    if (!shouldNotMarkOriginatingFromRejection) {
        util.markAsOriginatingFromRejection(reason);
    }
    var trace = util.ensureErrorObject(reason);
    var hasStack = trace === reason;
    this._attachExtraTrace(trace, synchronous ? hasStack : false);
    this._reject(reason, hasStack ? undefined : trace);
}

Ok so we are surely using the same version then. I cannot reproduce any memory leak. Just to make sure we are on the same page here, can you run this:
var Promise = require('bluebird');

function immediate() {
    return new Promise(function(resolve) {
        setImmediate(resolve);
    });
}

(function loop(value) {
    return immediate().then(function() {
      if (value % 10000 === 0) {
        console.log(process.memoryUsage().heapUsed, value);
      }
      return value + 1;
    }).then(loop);
})(0);

I ran it until 1110000 (equal to having the original code run for 3 hours) and the heap usage is almost same as when it started.

That seems to be pretty stable. The node process memory climbed to 44,304 K and pretty much stays there.

The heapUsed also seems pretty stable.
There are a few places I need to use this and in some cases the check condition uses a class function so I need a clean way to pass 'this' around while trying to keep the code readable.

So do you agree there is no memory leak?

Ok somehow the original code is not equivalent to the straight-forward recursion example and it indeed leaks : https://gist.github.com/petkaantonov/b73d52b9092b6f857271

I agree there is no memory leak leaving the interesting question of how to use while loop in practice.

Well you can use your original code or my code.
It leaks when there is a .then() added for the console.log("Done"); but this didn't leak in 2.5.2 which fixed the original memory leak issue with promises in general.

Many thanks for your help. I let you know how I get on.

This memory leak was introduced in 2.8.2 -> 2.9.0.
So you could use earlier version before I fix it (2.8.2)

Fixed in 2.9.12

I hate to pile on, but I think I'm running into another form of the same bug, using recursion as a while loop: https://gist.github.com/robey/de6717a41734d802c864
20.26 mb 1
34.39 mb 2
47.85 mb 3
67.94 mb 4
80.96 mb 5
99.04 mb 6
110.18 mb 7
123.17 mb 8
136.16 mb 9
156.67 mb 10
169.56 mb 11
182.55 mb 12
194.06 mb 13

removing the "catch" block also removes the leak. weird.

This variant also works around it, which I'm using now:
    return get().catch(function (error) {
      active = false;
    }).then(function () {
      return active ? loop() : Promise.resolve();
    });

@robey That is not a similar bug or a bug at all. In the original bug all the promises have settled before starting the next iteration -  there is no way to reach them and any memory used by them is therefore considered memory leak.
In your code you are creating an additional promise before moving on to the next iteration. This additional promise you create can only be settled after the loop has ended and the code will then reach all of them therefore all those promises have to be kept around thus it's not a memory leak.
Let's write 5 iterations of your loop manually with logs placed:
function get() {
    return Promise.delay(5);
}

return get().then(function (v) {
    return get().then(function (v) {
        return get().then(function (v) {
            return get().then(function (v) {
                return get().then(function (v) {
                    console.log("finished loop");
                }).then(function (error) {
                    console.log("1 reached");
                });
            }).then(function (error) {
                console.log("2 reached");
            });
        }).then(function (error) {
            console.log("3 reached");
        });
    }).then(function (error) {
        console.log("4 reached");
    });
}).then(function (error) {
    console.log("5 reached");
});

This will log:
finished loop
1 reached
2 reached
3 reached
4 reached
5 reached

Which clearly shows that the promises (and their callbacks) must be reachable after the loop ends.

@robey also using .catch for every iteration as you have done has actually no good purpose here. Rejection during any iteration of the loop will already stop the loop even if you place no catch at all.

so my workaround fixes my own bug by making sure the loop is the last call in the promise. makes sense; thanks for digging into it! :)


Appstudio update test-component-pac-cqxb

Pipelines as Code configuration proposal

Pipelines as Code CI/test-component-pac-cqxb-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
13 seconds
init

✅ Succeeded
19 seconds
clone-repository

✅ Succeeded
42 seconds
build-container

✅ Succeeded
16 seconds
deprecated-base-image-check

✅ Succeeded
13 seconds
inspect-image

✅ Succeeded
3 minutes
clair-scan

✅ Succeeded
40 seconds
clamav-scan

✅ Succeeded
11 seconds
sbom-json-check

✅ Succeeded
9 seconds
optional-label-check

✅ Succeeded
9 seconds
label-check

✅ Succeeded
14 seconds
show-summary

Pipelines as Code CI/test-component-pac-cqxb-on-pull-request has successfully validated your commit.

StatusDurationName

✅ Succeeded
5 seconds
init

✅ Succeeded
14 seconds
clone-repository

✅ Succeeded
29 seconds
build-container

✅ Succeeded
15 seconds
inspect-image

✅ Succeeded
18 seconds
deprecated-base-image-check

✅ Succeeded
8 seconds
clair-scan

✅ Succeeded
32 seconds
clamav-scan

✅ Succeeded
9 seconds
sbom-json-check

✅ Succeeded
8 seconds
optional-label-check

✅ Succeeded
8 seconds
label-check

✅ Succeeded
5 seconds
show-summary


HLS Support.

According to issues, HLS is supported. Readme.md should be updated accordingly.

Fixed.


Can't render null fields

Prerequisites

[x] I have read the documentation;
[x] In the case of a bug report, I understand that providing a SSCCE example is tremendously useful to the maintainers.
[ ] Ideally, I'm providing a sample JSFiddle or a shared playground link demonstrating the issue.

Description
Providing a schema with a property of type 'null' will yield the error 'Unknown field type null', this is inspite of it being able to validate this property.
Steps to Reproduce

Create a schema in which one of the properties has type null, such as:

{
  "type": "object",
  "properties": {
    "static": {
      "type": "null",
      "title": "Age"
    }
  }
}

Render it via RJSF

Expected behavior
The title 'Age' is rendered and nothing else for that property.
Actual behavior
The title is rendered but the following error appears:

Unsupported field schema for field root_static: Unknown field type null.

Version
v1.3.0

react-jsonschema-form has not previously supported null fields. I think the behavior you described (rendering the title and nothing else) makes sense, though. Could you make a PR implementing this?

I don't have much time on my hands unfortunately, you'd have to give me a week or two.


Worker Exception: cannot read properties of null (reading '0')

When trying to execute the worker with correct credentials, the following exception will occur:
 "exceptions": [
    {
      "name": "TypeError",
      "message": "Cannot read properties of null (reading '0')",
      "timestamp":<PHONE_NUMBER>426
    }
  ],

(For simpler reproduction, test with the following curl command: curl -X POST -u<REDACTED> -v https://dyndns.<REDACTED>.workers.dev/update)

Further debugging shows:
TypeError: Cannot set properties of undefined (setting 'content')
    at t.exports.updateRecord (worker.js:10:13629)
    at worker.js:1:1608


Update Composer dependencies (2022-08-10-00-11)

Visual regression test passed!
View the visual regression test report


fix the race condition in test logger

What type of PR is this?
/kind cleanup
What this PR does / why we need it:
When using test logger in an unsynchronized goroutine, we will meet a race condition such as: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/93/pull-kueue-test-unit-main/1499496533626720256 (in https://github.com/kubernetes-sigs/kueue/pull/93)
We have to synchronize the goroutines in the unit test to solve the problem.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
/assign @alculquicondor
/cc @ahg-g

Can you summarize how you concluded that this is the problem? And what experiments did you conduct to verify that this fixes the problem.
The logger itself is thread-safe. So I was thinking that the problem might be when trying to access a value passed to the logger.

Can you summarize how you concluded that this is the problem? And what experiments did you conduct to verify that this fixes the problem?
The logger itself is thread-safe. So I was thinking that the problem might be when trying to access a value passed to the logger.

hi @alculquicondor !
First, I tried to reproduce the issue with go test ./pkg/scheduler/... -race -count 1000. Then I found that sometimes I can see the "Log in goroutine after..." message when a race condition is generated.
We can found that this panic is generated in https://github.com/golang/go/blob/63bd6f68e6cbb237b46a99775103758afaee370a/src/testing/testing.go#L887 and with the help of related comments we can conclude that it will be illegal to use test.logger in a test case that has already completed.
That's to say, before executing the next test case, we have to make sure that no test.logger is still being used anywhere.
By adding a channel to synchronize the goroutines, we can ensure that test.logger will only be used in each individual testcase (no two testcases are using test.logger at the same time, especially if one of them is considered complete even though it still has a goroutine running).
We found the cause of the problem, and we can solve it directly through coroutine synchronization. I verified the above through multiple test commands as well.

Thanks for the explanation. That makes sense.
However, I think we should implement it differently. Use sync.WaitGroup and do not use wantScheduled as the number of routines we are waiting for. If we introduce a bug in the scheduler, the number of goroutines would be different from wantScheduled. Instead, do wg.Add everytime that a routine is launched.

Sound great to use sync.WaitGroup instead of channel. And I have updated the code.
In addition, I personally think it is worthwhile to create a function level logger, which can play a role in many similar scenarios. This function logger is located in pkg/util/function/logger.go . What is your opinion please?

/retest

/retest

Sounds good, it might be useful long term.
I like the routine.Wrapper name.
/lgtm
/approve


Reporting a vulnerability

Hello!
I hope you are doing well!
We are a security research team. Our tool automatically detected a vulnerability in this repository. We want to disclose it responsibly. GitHub has a feature called Private vulnerability reporting, which enables security research to privately disclose a vulnerability. Unfortunately, it is not enabled for this repository.
Can you enable it, so that we can report it?
Thanks in advance!
PS: you can read about how to enable private vulnerability reporting here: https://docs.github.com/en/code-security/security-advisories/repository-security-advisories/configuring-private-vulnerability-reporting-for-a-repository

Sorry for the late response and thanks for reaching out. The feature is now enabled for this repo.


UI-8712 - Fix widget mappings not created during migration from 4.3 to 5.0 when the axis names are not fully uppercase in their query

Backport of https://github.com/activeviam/atoti-ui-migration/commit/63436c114e3125394db172c49615baacf057fd1a.

@Nouzbe I solved the problems that appear when upgrading the version of the SDK in another PR: https://github.com/activeviam/atoti-ui-migration/pull/106
So in fact we could close this one and keep only #106

Closed in favor of #106


nhentai and chan.sankaku

i can bypass the internet positif, but the image didnt shown up, all the image shown is broken image

Fixed both, thx for reporting the bugs


[WIP] Named pipe cleanup

Potential fix for #59.

@blink1073, what to do about this PR? Do you want to close or merge it?

Closing, not sure what else to try here.


Ensure http 401 in case of missing security

While using pyramid-swagger for a personal project I've noticed that:
missing authentication information specified in the security section of the swagger specs results in a HTTP/400 response
This was caused by the fact that call to bravado_core.request.unmarshal_request is wrapped to reraise SwaggerMappingError as RequestValidationError.
In the specific case of missing authentication information, bravado_core raises a SwaggerSecurityValidationError exception.
The goal of this PR is to catch such exception and to re-raise it as RequestAuthenticationError (which extends the base HTTPUnauthorized)

Coverage increased (+0.007%) to 99.572% when pulling 6607eb58c762e550bc732b2c1403d520ce504280 on macisamuele:maci-ensure-HTTP-401-in-case-of-missing-security into 820a01f95d2cd4f794d73baa752ca2043ee89297 on striglia:master.

Coverage increased (+0.007%) to 99.572% when pulling 6607eb58c762e550bc732b2c1403d520ce504280 on macisamuele:maci-ensure-HTTP-401-in-case-of-missing-security into 820a01f95d2cd4f794d73baa752ca2043ee89297 on striglia:master.


Figure out what is happening with the counties for American Samoa

Describe the bug
America Samoa is missing the territory name in the beta spreadsheet (I think). can we make sure this doesn't happen for 1.0? beta was missing the population count but this appears to be fixed for 1.0. the other issue is that neither version has counties which apparently are available for the territory.
https://en.wikipedia.org/wiki/Outline_of_American_Samoa
To Reproduce
Steps to reproduce the behavior:

Go to 'AS'
Click on 'a census tract'
Scroll down to 'county info'
See error

Expected behavior
Populations and counties should appear for American Samoa, both on the front and back end. The territory name should also appear in the spreadsheet.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

I looked quickly in our code, and we use the a census gazetteer file for our county names. It doesn't include AS, nor do any other obvious files in that gazetteer directory :/
This is our counties file: https://www2.census.gov/geo/docs/maps-data/data/gazetteer/Gaz_counties_national.zip

closing, not currently supported.


Improve translations by using named fomatting

What is the motivation for adding / enhancing this feature?
Translation strings which are composed of multiple pieces like for example
https://github.com/DivanteLtd/vue-storefront/blob/38fb1579c60d1aa352d353c06c1f0d9af7f79937/src/themes/default/components/core/GenericSelector.vue#L6 or
https://github.com/DivanteLtd/vue-storefront/blob/6995f88dab30cbd2a10dcd6874891f6566124320/src/themes/default/components/core/ColorSelector.vue#L5
force the pieces to be composed in this exact order. So, for example first "Select " and then always followed by the label. Considering the following German translation, this results in "Auswählen Größe" which sounds really stupid as it is simply incorrect grammar:

"Select " translated with "Auswählen "
label having a value of "Größe"

Correct would be "Größe auswählen".
What are the acceptance criteria
A translation string needs to be able to define the order of how the pieces get put together.
Can you complete this feature request by yourself?

[x] YES
[ ] NO

Which Release Cycle state this refers to?

[x] This is a normal feature request. This should be available on https://test.storefrontcloud.io and then after tests this can be added to next Vue Storefront version. In this case Developer should create branch from develop branch and create Pull Request 2. Feature / Improvement back to develop.
[ ] (Pick this option only if you're sure) This is an important improvement request for current Release Candidate version on https://next.storefrontcloud.io and should be placed in next RC version. In this case Developer should create branch from release branch and create Pull Request 3. Stabilisation fix back to release.
[ ] (Pick this option only if you're sure) This is a critical improvement request for current Stable version on https://demo.storefrontcloud.io and should be placed in next stable version. In this case Developer should create branch from hotfix or master branch and create Pull Request 4. Hotfix back to hotfix.

Additional information
I propose that all translations which are composed of multiple strings/translations use named formatting. This feature is already available (vue-i18n), only need to refactor existing code to take advantage of it.
Translations for the above mentioned examples would look like this:
"Select {label}","{label} auswählen"
"Select color {label}","Farbe '{color}' auswählen"

I'd offer to take this issue over but don't know how to assign it to me.

In general I suggest the translations should be in the formal instead of informal way.

Hi @ca1zr, is your last comment just a general statement regarding translations? If it has to do with the approach requested in the initial description then I'm not quite sure if I get the link ... could you clarify more how it relates to named formatting?

Hi @phoenixdev-kl, it was a general request. Some translations sound pretty weird and choppy, so  named formatting is a good thing.

@phoenixdev-kl this is a pretty good idea, why not to implement it to 2.0?

@pkarw great! sorry, the process is unfortunately not clear to me. if this feature is planned for 2.0, from which branch do i create a branch in my fork (release/v2.0 or develop)? and when the refactoring is finished, am i supposed to create the pull request in release/2.0?


[riscv64] Refactor bsp framework

Hi @vvaltchev
I have recently successfully ported tilck to the Sipeed MaixCAM (profile: https://wiki.sipeed.com/hardware/en/maixcam/index.html,  purchase: https://www.aliexpress.com/store/911876460 (It seems that the aliexpress official sipeed store is temporarily out of stock. But I can bought it at taobao official sipeed store)), which is based on the licheerv-nano core board (profile: https://wiki.sipeed.com/hardware/en/lichee/RV_Nano/1_intro.html) and has a 2.3-inch screen and a camera. Of course, you also need a USB to serial module.

About this PR:
Refactor bsp framework, and introduces the BOARD environment variable (default is 'qemu-virt'), which specifies the board level target to be compiled.

Hi @aenrbes,
this is an outstanding achievement! Making Tilck to work on real hardware is what really matters at the end of the day! :-)
I'll consider buying the same hardware as well.
Unfortunately, I'm super tired and cannot make a full review for this change today, but I'll try to do that in the following days.


Document necessary dependencys with versions

Create a requirements.txt with all relevant information to help people build faster and make the usability more reliable.


chore(assortedscans): update icon

Checklist:

[x] Updated source's version for individual source changes
[ ] Updated all sources' versions for template changes
[ ] Set appropriate nsfw value
[ ] Did not change id even if a source's name or language were changed
[ ] Tested the modifications by running it on the simulator or a test device

I knew I'd missed something…

Done.


Uninstall fails trying to delete ListView from standard objects

I'm not sure how CumulusCI is working despite this...perhaps you don't use standard objects?
We add custom fields to various standard objects.  For example, User.object.  This is what is happening:

The sf:retrieve code retrieves all defined list views.
The macro to generate destructive changes adds entries for the stock, standard list views, like User.ActiveUsers.
Later, the deploy of the destructive changes fails, because it tries to delete all of the standard ListView records.  Fail.

Looks like what we need it to do is add any custom list views, but exclude those standard ones that we don't own and shouldn't touch anyway.  In our case, we don't add any list views to standard objects ourselves...so we could just remove all of them from the downloaded uninstallsrc/object files and be good.  But that isn't a solution for everyone.

Clearing out old issues to get ready for CumulusCI 2 related issues.
For future reference, I believe this was about the uninstallUnmanaged target which is no longer the preferred way to do deletes in CumulusCI.  The preferred method is to use an unmanaged package as a container and then use the uninstallPackaged Ant target in old CumulusCI or cci task run uninstall_packaged  or cci task run uninstall_packaged_incremental in CumulusCI 2.  That will only uninstall the metadata contained in the package and leave all the other unmanaged metadata in place.


Issue/2237 broken resource details link (#2243)

Pull request opened by the merge tool on behalf of #2243

Merged in c9e5b7b0a393326879fc021710281579f1023220