Always break chained pipe infix operators

Improve printing of chained |> (#1066, #1045)
Result
let x = foo |> z;

let x =
  foo
  |> f
  |> g;

let x =
  foo
  |> somelongfunctionname "foo"
  |> anotherlongfunctionname "bar" 1
  |> somelongfunction
  |> bazasdasdad;

let code =
  JSCodegen.Code.(
    create
    |> lines
         Requires.(
           create
           |> import_type local::"Set" source::"Set"
           |> import_type local::"Map" source::"Map"
           |> import_type local::"Immutable" source::"immutable"
           |> require local::"invariant" source::"invariant"
           |> require local::"Image" source::"Image.react"
           |> side_effect source::"monkey_patches"
           |> render_lines
         )
    |> new_line
    |> new_line
    |> new_line
    |> new_line
    |> render
  );

let code = JSCodegen.Code.(create |> render);

Overview
/* (1) */
let x = foo |> f;

/* (2) */
let x =
  foo
  |> f
  |> g;

We have two cases here: the first shouldn't break, the second should always break.
Note that |> results in some sort of recursive structure 'going deeper' on the 'left side' of the printed result.
The printing is some sort of left fold.
let x =  foo |> f |> g;

/* is going to be parsed as: */
((foo |> f) |> g)

This is important because at a certain moment in the unparsing recursion,
we're going to see foo |> f. This is exactly the same as the first case!
If we just recursively print everything where the last one is treated as the first case,
we would get:
let x =
  foo |> f
  |> g;

How do we make a difference between the two cases?
(* list containing all infix operators with the 'left-fold' printing behaviour *)
let leftRecInfixOperators = ["|>"]

While unparsing an expression, check if a printedIdent is a member of the above list.
If so, we check if the left child of the application arguments contains another
application with the same printedIdent (isLeftInfixChain)
Is (foo |> f) another Pexp_apply with a |> in ((foo |> f) |> g) ?
Yes, it is.

If so mark the unparsing with the parameter ~infixChain.
~infixChain indicates we're chaining an application with an identifier ("|>" here).
Based on the value of ~infixChain, we know we always have to break.
The innermost (foo |> f) of ((foo |> f) |> g), will break and result in the following:
let x =
  foo
  |> f
  |> g;

/* we won't get
let x =
  foo |> f
  |> g;
*/

Future work
This PR works as advertised for |>, but it would be nice if we could standardize
printing for all infix operators. >>= & co has the same problem, but prints as a right fold instead
of a left fold. (>>= has some too-much-parens problems too).
Can we drop the leftRecInfixOperators in the future? Is there a way to detect those?
Note that we can't just take all infix operators with left associativity here, because of e.g. "1 + 2 + 3"
Printing all left associative infix application as |> would result in:
let x =
  1
  + 2
  + 3

With |>, it always breaks as soon as there's more than one |> chained.
In the case of +, we want break IfNeed style.

/* is going to be printed as: */

I think you mean "parsed" as?

What did you mean by this?

"Note that we can't just take all infix operators with left associativity here, because of e.g. "1 + 2 + 3"

By the way, this is a very good explanation of the problem/challenges.

"Note that we can't just take all infix operators with left associativity here, because of e.g. "1 + 2 + 3"

I thought of printing all left associative infix application as |>, but clearly this won't with + for example.
let x =
  1
  + 2
  + 3

With |>, it always breaks as soon as there's more than one |> chained.
In the case of +, we want break IfNeed style.

For typical use cases of |> would it be bad to format on an IfNeed basis? I genuinely don't know.
I don't mind + wrapping like this:
let x =
  1
  + 2
  + 3

as long as it's on an IfNeed basis.
Also, I believe that >>= is actually left associative right? If so, then it is one case where we don't want the infix to dock to the left, despite being left associative.

Should I change it to an IfNeed basis?
let uri = req |> Request.uri |> Uri.to_string;
let meth = req |> Request.meth |> Code.string_of_method;
let headers = req |> Request.headers |> Header.to_string;

In the above it absolutely makes sense, but when I'm writing
let nextState =
    state
    |> updateFlappy
    |> updateTime;

I want it to break. Hmm, food for thought.

I don't feel strongly. @bordoley, our resident |> connoisseur may have opinions.

@IwanKaramazow why do you want line breaks in your nextState case?
Looking through my own code, I think the IfNeed basis would accurately layout most code the same way I would by hand.

In the nextState case it's just for readability, and there's a big chance I might extend it. It would eventually break. The IfNeed basis might be harder since the Auto breaking of labels here originally didn't work & spawned the issue 😄
Back to the drawing board...

The original issue here which was caused by IfNeed was due to the fact that they were multiple nested labels. I had originally had something kind of like the "concrete parse tree" representation that we just discussed, but I removed it in favor of this implementation (labels) to first focus on correctness, with the intent of eventually evolving it back into a kind of "concrete parse tree".

@IwanKaramazow ready? =D

It works, I just need to rename some functions so it makes more sense. I can do that this weekend.
@jordwalke should I wait & do this on top of Fred's new diff?

I think this should get in before the breaking change, since current users can benefit

Would love to see this get merged, this is my biggest issue with the formatter at the moment :)

I want to make sure @let-def is aware of this one.

This got out of sync since we've re-merged back reason-parser. Would be great to have it working again! Number one request for formatting right now =)

Rebased on master, now continuing

This should be something we get in before the potential function syntax refactor. cc @let-def just in case

What ended up happening with infix operators like >>=? I can't quite follow what happens to them in the change list.
While I agree it can't be "all left associative infix operators", any infix operator that's more than one character is probably worth applying this to.

@TheSpyder
>>= should be printed a lot better. Check the result of some classic cohttp code:
let server = {
  let callback _conn req body => {
    let uri =
      req
      |> Request.uri
      |> Uri.to_string
      |> Code.string_of_uri
      |> Server.respond
      |> Request.uri;
    let meth =
      req
      |> Request.meth
      |> Code.string_of_method;
    let headers =
      req |> Request.headers |> Header.to_string;
    body
    |> Cohttp_lwt_body.to_string
    >|= (
          fun body =>
            Printf.sprintf
              "okokok" uri meth headers body
        )
    >>= (
          fun body =>
            Server.respond_string
              ::status ::body ()
        )
  };
  Server.create
    ::mode (Server.make ::callback ())
};

The parens are unfortunate but it sounds like they're pretty difficult to remove.
@IwanKaramazow This is so much better that the current output - thank you!

This is getting a bit off topic for the pipe operators, so I'll log a new task if you'd prefer, but in the case of infix operators could we avoid the indent inside parens? Or perhaps just when the brackets contain a single function?
e.g.
>|= (fun body =>
      Printf.sprintf
        "okokok" uri meth headers body
    )

@TheSpyder I addressed the weird indentation. 😁

Soooo, which one should we merge first, @let-def's function diff, or this one? =P

@chenglou This one! Fixes some significant paper cuts before opening up a world of fun from overhauling the entire function and constructor syntax. It will also (hopefully) keep the conversation around the big @let-def function change focused on just that change.

@chenglou just checked, this PR & the js-syntax-branch shouldn't cause any crazy merging conflicts. Had to touch some parts of the code let-def didn't touch ⛷

Alright I've thought about this. We're gonna do right by the community and make sure you don't feel conned into using the function application PR just for better pipes lol.
So I'll merge this PR, make a release, before we get crazy with the function diff. cc @let-def, I'll re-rebase your diff on top of this one. This way everyone's happy.
As always, thanks @IwanKaramazow!

Thanks @chenglou!


rustls  https errors on atomicdata.dev

The server keeps going down since running 29.2, and it throws these errors:
[2021-12-11T23:44:04Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-11T23:44:04Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-11T23:44:04Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-11T23:44:04Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }

Tried removeing the https folder from config, since a few restarts did not help.
After that, the server got stuck on [2021-12-11T23:49:07Z WARN  atomic_server::https] HTTPS TLS Cert init sucesful! Stopping HTTP server, starting HTTPS...

I think the problem was that the HTTP port was not set, so defaulted to the new 9883 value, instead of 80.

Hmm it went down again... Not sure if it was the HTTPS error, though, forgot to save logs

Yes, it's that same issue.
[2021-12-12T11:30:13Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-12T11:30:13Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-12T11:30:13Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-12T11:30:13Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-12T11:30:15Z WARN  rustls::conn] Sending fatal alert HandshakeFailure
[2021-12-12T11:30:15Z WARN  rustls::conn] Sending fatal alert NoApplicationProtocol
[2021-12-12T11:30:15Z WARN  rustls::conn] Sending fatal alert ProtocolVersion
[2021-12-12T11:30:15Z WARN  rustls::conn] Sending fatal alert HandshakeFailure
[2021-12-12T11:30:27Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-12T11:30:27Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-12T11:30:27Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-12T11:30:27Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-12T11:30:45Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-12T11:30:49Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-12T11:30:53Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }
[2021-12-12T11:30:56Z ERROR rustls::conn] TLS alert received: AlertMessagePayload {
        level: Fatal,
        description: BadCertificate,
    }

https://github.com/actix/actix-web/issues/2507

I'm seriously considering rolling back actix-web to v3.x. Will break Tauri builds, but that's a nice to have anyway.

Just noticed this on atomicdata.dev, after the server went down without a specific panic log
[2021-12-21T04:36:19Z WARN  rustls::server::tls13] Resumption ignored, DHE_KE not offered

This comment is interesting:
https://github.com/joepio/atomic-data-rust/commit/6bb7412539171b54fcdc6231298e8f49feb9a154#r62189445

This fixes it, had nothing to do with actix:
https://github.com/joepio/atomic-data-rust/pull/265/commits/77bbfb404963b4f1f293e40dbdbdde86b21c07f9

It's still here...
2021-12-30T13:43:13.838283Z  WARN rustls::conn: Sending fatal alert HandshakeFailure
2021-12-30T13:43:59.125224Z  WARN rustls::conn: Sending fatal alert HandshakeFailure
2021-12-30T13:44:42.298872Z  WARN rustls::msgs::handshake: Illegal SNI hostname received [49, 48, 56, 46, 54, 49, 46, 50, 53, 50, 46, 53, 48]


Fix bug and broken tests from VSMT updates

It seems we had taken out the larger test runner from the CI and so we didn't realise we had broken some tests (it's only testing the mixer client).
These tests fail.
test smt::tests::test_vsmt_prove_verif ... FAILED
test time_based_rewarding::tests::test_time_based_reward_gadget_verification ... FAILED

My investigation shows that update method on the SMT does not properly add leaves into leaf_indices afaik. So these modules try to unwrap that at some point and end up with an empty collection.

leaf_indices are populated inside add_leaves, but it would be a good idea to do it inside update to make it future proof.


Unbound ONG balance unstable in v0.9.6

OWallet was working well for over a month. Now, for past day, the Unbound ONG balance keeps changing and sticking at the wrong number. As example, I have approx 3 showing in wallet which is correct based on previous balances. Then sometimes it jumps to 0.15, then jumps to 0.65, then back to 3. It sticks at a wrong number for 10 seconds, or 10 minutes, or 30 minutes. I closed and reopened wallet many times, but no difference.

OWallet Unbound ONG not stable.docx

OWallet Unbound ONG not stable.docx

The balance seem much more stable now, but:

I have lost approx 4 Unbound ONG
Will this keep happening?
Note, never any problem with balance of ONT, ONG, Claimable ONG. It is only the Unbound ONG balance which has issues. Anyone else seen this instability before?

Hello, dont' worry. Your assets are not lost. The instability is due to the load balance of our explorer backend server. We are fixing it now.

Thanks Mick. Keep up the great work.Best Regards,Paul
On Wednesday, March 13, 2019, 10:48:07 PM EDT, MickWang<EMAIL_ADDRESS>wrote:

Hello, dont' worry. Your assets are not lost. The instability is due to the load balance of our explorer backend server. We are fixing it now.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Thanks Mick, keep up the great work. Best Regards.
On Wednesday, March 13, 2019, 10:48:07 PM EDT, MickWang<EMAIL_ADDRESS>wrote:

Hello, dont' worry. Your assets are not lost. The instability is due to the load balance of our explorer backend server. We are fixing it now.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

I am experiencing the same issue with the Unbound ONG figures are not being updated since last 2 months or so.  What should I do to have this figure refreshed or recalculated correctly please?  Thanks in advance.  Anil


fix(bug): incorrect kokoro configuration for running the lint presubmit

This is a fix for getting the master branch presubmitting again on Kokoro:
https://github.com/GoogleCloudPlatform/nodejs-getting-started/issues/359
There is a Kokoro-side fix that is needed as well, but it won't work without this in place first.

Looks like this is finally passing :)


fix: Enable NVIDIA GPU support for Ollama in Docker

added NVIDIA GPU support for OLLAMA in docker-compose.yaml

can make the latest docker files that compatible with the current changes.

This code throws error. if nvidia gpu container toolkit is not present ( which will prevent non-gpu from running docker).
I would suggest to handle it gracefully using "runtime" and a environment variable in code, so that it will give flexibity for a non-GPU user also to run & also it won't throw errors if nvidia-gpu-container- toolkit is not present.

So it can handle in better way? Or I just skip this pr currently?


updated to 11.1.2 and the browser project does not build because it does not find the namespace avalonia.browser

Describe the bug

using Avalonia.Browser; <- The namespace does not exist in the namespace avalonia.
To Reproduce
Upgrade to the latest version
Expected behavior
No response
Avalonia version
11.1.2
OS
WebAssembly
Additional context
No response

@simsfarrow can you provide a repro? Freshly created project from the templates builds.


[RFC?] Read board revision and serial number

Is it possible to use rpiboot (or enhance it) to get a dump of the connected CM's board revision and serial number? This would be really handy to use in production...

This information is reported in the USB string descriptor from bootcode4.bin but it also reported by the new mass-storage-gadget. Since the descriptor is generated by a Linux script you can put anything there
https://github.com/timg236/cm4-scripts/blob/master/usr/local/bin/configure-gadgets#L33
cd mass-storage-gadget ../rpiboot -d .
From USB Device Tree Viewer https://www.uwe-sieber.de/
             ------ String Descriptor 3 ------
bLength                  : 0x22 (34 bytes)
bDescriptorType          : 0x03 (String Descriptor)
Language 0x0409          : "10000000b29d2da0"
Data (HexDump)           : 22 03 31 00 30 00 30 00 30 00 30 00 30 00 30 00   ".<IP_ADDRESS>.0.0.0.
                           30 00 62 00 32 00 39 00 64 00 32 00 64 00 61 00   0.b.2.9.d.2.d.a.
                           30 00                                             0.

bootcode4.bin could probably be updated to dump string descriptors e.g. boot the CM4 without a firmware .elf  so that it just sits there and waits
mkdir serial
cp msd/bootcode4.bin serial
cd serial
../rpiboot -d .

bit busy to look at this right now but hopefully that's enough to hack something in main.c

@timg236

Ok, I now see that I can get the serial number by finding the right node and reading /sys/bus/usb/devices/*/serial So what about the board revision?
The rpiboot -vv output does print a serial number, but not the right one that is available in the USB string descriptor.

@timg236 This forum post seems to indicate that rpiboot used to provide some functionality that it no longer does... at least not the correct serial number for CM4.

The board revision (and serial number) can be read from /proc/cpuinfo.
I don't think rpiboot has ever automatically extracted these values on 2711 and it might require a patch to bootcode4.bin to make this work nicely. I think it's useful but can't promise when we'll be able to do this.

This can be done using the metadata collection flags.
Set up your config txt as per https://github.com/raspberrypi/rpi-sb-provisioner/blob/b8dfe1550e06f85aabddfb3ed59c4a81cd0e3ef2/service/rpi-sb-provisioner.sh#L417
And then invoke rpiboot with the -j flag, as per https://github.com/raspberrypi/rpi-sb-provisioner/blob/b8dfe1550e06f85aabddfb3ed59c4a81cd0e3ef2/service/rpi-sb-provisioner.sh#L432
Closing as 'Fixed'. It's pretty useful.


Configure nginx worker timeout

This change is

Coverage increased (+0.03%) to 44.768% when pulling 106cfca1a64c4a69d4277927117dd47ec4868ff7 on aledbf:worker-timeout into 27f447f88aa7efa64a77968e8b52a36def492e17 on kubernetes:master.

Looking forward to this! 😄
I'm trying to deploy a WebSocket application to Kubernetes and use NGINX to reverse proxy the WebSocket connections to my application pods, but I observed a very odd memory usage behavior in my ingress pods ever since I deployed this to production:

After struggling for a while, I noticed that the increased memory usage was due to lots of worker processes stuck in worker process is shutting down state, probably due to active WebSocket connections still being handled by those workers:
root       507   490  0 16:09 ?        00:00:00 /usr/bin/dumb-init -v /nginx-ingress-controller --default-backend-service=kube-system/broken-bronco-nginx-ingress-be --configmap=kube-system/broken-bronco-nginx-ingress-conf --ingress-class=nginx-ingress-prd
root       521   507  0 16:09 ?        00:00:38 /nginx-ingress-controller --default-backend-service=kube-system/broken-bronco-nginx-ingress-be --configmap=kube-system/broken-bronco-nginx-ingress-conf --ingress-class=nginx-ingress-prd
root       543   521  0 16:09 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nobody     562   543  1 16:09 ?        00:01:17 nginx: worker process is shutting down
nobody    4868   543  5 17:45 ?        00:00:08 nginx: worker process
core      5838  5826  0 17:48 pts/0    00:00:00 grep --colour=auto nginx
nobody   10001   543  0 16:34 ?        00:00:23 nginx: worker process is shutting down
nobody   12630   543  2 16:41 ?        00:01:34 nginx: worker process is shutting down
nobody   23713   543  0 17:10 ?        00:00:01 nginx: worker process is shutting down
nobody   24172   543  0 17:11 ?        00:00:05 nginx: worker process is shutting down
nobody   24867   543  0 17:13 ?        00:00:07 nginx: worker process is shutting down
nobody   25864   543  2 17:15 ?        00:00:40 nginx: worker process is shutting down
nobody   30571   543  0 17:28 ?        00:00:11 nginx: worker process is shutting down
nobody   31906   543  4 17:32 ?        00:00:44 nginx: worker process is shutting down
...

Stracing one of those worker process showed that it was still handling WebSocket traffic:
Process 10001 attached
gettimeofday({1502304533, 387656}, NULL) = 0
epoll_wait(24, {{EPOLLIN|EPOLLOUT, {u32=31428513, u64=31428513}}}, 512, 3940) = 1
gettimeofday({1502304534, 692450}, NULL) = 0
recvfrom(45, "\301\203n\0\3514\\\2\351", 4096, 0, NULL, NULL) = 9
sendto(48, "\301\203n\0\3514\\\2\351", 9, 0, NULL, 0) = 9
epoll_wait(24, {{EPOLLIN|EPOLLOUT, {u32=31431873, u64=31431873}}}, 512, 10000) = 1
gettimeofday({1502304534, 694349}, NULL) = 0
recvfrom(48, "\201\0013", 4096, 0, NULL, NULL) = 3
sendto(45, "\201\0013", 3, 0, NULL, 0)  = 3
epoll_wait(24, {{EPOLLIN|EPOLLOUT, {u32=31431873, u64=31431873}}}, 512, 9998) = 1
gettimeofday({1502304536, 762742}, NULL) = 0
recvfrom(48, "\201~\1R42/quaestio,[\"insert\",{\"text"..., 4096, 0, NULL, NULL) = 342
sendto(45, "\201~\1R42/quaestio,[\"insert\",{\"text"..., 342, 0, NULL, 0) = 342

So, is specifying a worker shutdown timeout (what this PR does) the only way to avoid having worker nodes unable to shut down due to active WebSocket connections, or do you guys know other ways of handling this?

@danielfm you can use this image quay.io/aledbf/nginx-ingress-controller:0.173 to test the new parameter

@aledbf I tried this image, but could not find the worker_shutdown_timeout  parameter in the generated configuration.

Sorry about that. Please use quay.io/aledbf/nginx-ingress-controller:0.174

It did not seem to work for me; I still see several workers stuck in shutting down:
root     17755 17739  0 19:47 ?        00:00:00 /usr/bin/dumb-init /nginx-ingress-controller --default-backend-service=kube-system/broken-bronco-nginx-ingress-be --configmap=kube-system/broken-bronco-nginx-ingress-conf --ingress-class=nginx-ingress-prd
root     17765 17755  0 19:47 ?        00:00:08 /nginx-ingress-controller --default-backend-service=kube-system/broken-bronco-nginx-ingress-be --configmap=kube-system/broken-bronco-nginx-ingress-conf --ingress-class=nginx-ingress-prd
root     17776 17765  0 19:47 ?        00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
nobody   18866 17776  0 19:49 ?        00:00:05 nginx: worker process is shutting down
nobody   19466 17776  0 19:51 ?        00:00:01 nginx: worker process is shutting down
nobody   19698 17776  0 19:51 ?        00:00:05 nginx: worker process is shutting down
nobody   20331 17776  0 19:53 ?        00:00:05 nginx: worker process is shutting down
nobody   20947 17776  0 19:54 ?        00:00:03 nginx: worker process is shutting down
nobody   21390 17776  1 19:55 ?        00:00:05 nginx: worker process is shutting down
nobody   22139 17776  0 19:57 ?        00:00:00 nginx: worker process is shutting down
nobody   22251 17776  0 19:57 ?        00:00:01 nginx: worker process is shutting down
nobody   22510 17776  0 19:58 ?        00:00:01 nginx: worker process is shutting down
nobody   22759 17776  0 19:58 ?        00:00:01 nginx: worker process is shutting down
nobody   23038 17776  1 19:59 ?        00:00:03 nginx: worker process is shutting down
nobody   23476 17776  1 20:00 ?        00:00:01 nginx: worker process is shutting down
nobody   23738 17776  1 20:00 ?        00:00:01 nginx: worker process is shutting down
nobody   24026 17776  2 20:01 ?        00:00:02 nginx: worker process is shutting down
nobody   24408 17776  4 20:01 ?        00:00:01 nginx: worker process

Any suggestion on how to find out what's going on?

@danielfm how are you testing this?

I simply deployed the image you provided and, when the ingress deployment was fully rolled out, I changed the DNS record for this application to shift traffic to the one deployed in Kubernetes.
As soon as I switch the DNS, client applications start connecting to the WebSocket application in Kubernetes via the NGINX ingress load balancer. When I do that, I can almost immediately see the memory usage for the ingress pods going up:

And when I log into any machine where some NGINX pod is running, I can see several workers unable to shutdown (see the ps -ef listing in my previous comment).
When I rollback the DNS change, the memory usage stabilizes:

I'm also seeing lots of reload requests (twice a second, in avg) made by the ingress controller that I'm unable to explain, this might give you some insight:
I0809 19:47:29.748303       6 leaderelection.go:184] successfully acquired lease kube-system/ingress-controller-leader-nginx-ingress-prd
I0809 19:47:30.186002       6 controller.go:419] backend reload required
I0809 19:47:30.439873       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:47:46.152766       6 controller.go:419] backend reload required
I0809 19:47:46.283154       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:48:12.824363       6 controller.go:419] backend reload required
I0809 19:48:12.959181       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:48:46.157339       6 controller.go:419] backend reload required
I0809 19:48:46.286852       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:49:16.150587       6 controller.go:419] backend reload required
I0809 19:49:16.345406       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:49:42.816188       6 controller.go:419] backend reload required
I0809 19:49:42.959675       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:51:12.817276       6 controller.go:419] backend reload required
I0809 19:51:12.947618       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:51:42.854549       6 controller.go:419] backend reload required
I0809 19:51:43.054739       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:53:16.152390       6 controller.go:419] backend reload required
I0809 19:53:16.378867       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:54:44.605780       6 controller.go:419] backend reload required
I0809 19:54:44.850715       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:55:44.543525       6 controller.go:419] backend reload required
I0809 19:55:44.739965       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:57:29.511057       6 controller.go:419] backend reload required
I0809 19:57:29.664816       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:57:42.815175       6 controller.go:419] backend reload required
I0809 19:57:42.893928       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:58:12.815656       6 controller.go:419] backend reload required
I0809 19:58:12.947353       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:58:42.814878       6 controller.go:419] backend reload required
I0809 19:58:42.892335       6 controller.go:428] ingress backend successfully reloaded...
I0809 19:59:12.823478       6 controller.go:419] backend reload required
I0809 19:59:12.954918       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:00:16.150933       6 controller.go:419] backend reload required
I0809 20:00:16.339993       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:00:46.154790       6 controller.go:419] backend reload required
I0809 20:00:46.280207       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:01:12.827212       6 controller.go:419] backend reload required
I0809 20:01:12.962812       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:01:16.149813       6 controller.go:419] backend reload required
I0809 20:01:16.352509       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:01:58.773523       6 controller.go:419] backend reload required
I0809 20:01:58.955574       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:01:59.494158       6 controller.go:419] backend reload required
I0809 20:01:59.652722       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:03:03.463374       6 controller.go:419] backend reload required
I0809 20:03:03.659459       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:03:08.989399       6 controller.go:419] backend reload required
I0809 20:03:09.158166       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:03:55.041454       6 controller.go:419] backend reload required
I0809 20:03:55.175486       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:04:42.592096       6 controller.go:419] backend reload required
I0809 20:04:42.749416       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:04:49.475018       6 controller.go:419] backend reload required
I0809 20:04:49.640061       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:06:05.272085       6 controller.go:419] backend reload required
I0809 20:06:05.377370       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:07:29.501766       6 controller.go:419] backend reload required
I0809 20:07:29.664621       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:08:16.148507       6 controller.go:419] backend reload required
I0809 20:08:16.277656       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:08:42.817247       6 controller.go:419] backend reload required
I0809 20:08:43.051563       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:09:12.872480       6 controller.go:419] backend reload required
I0809 20:09:13.147934       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:09:46.151076       6 controller.go:419] backend reload required
I0809 20:09:46.275197       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:10:46.159073       6 controller.go:419] backend reload required
I0809 20:10:46.281841       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:11:12.873691       6 controller.go:419] backend reload required
I0809 20:11:13.065928       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:12:02.823197       6 controller.go:419] backend reload required
I0809 20:12:03.054718       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:12:34.720417       6 controller.go:419] backend reload required
I0809 20:12:34.867959       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:12:44.543628       6 controller.go:419] backend reload required
I0809 20:12:44.674828       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:17:29.494224       6 controller.go:419] backend reload required
I0809 20:17:29.641063       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:17:46.149205       6 controller.go:419] backend reload required
I0809 20:17:46.269999       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:18:12.862447       6 controller.go:419] backend reload required
I0809 20:18:13.062000       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:19:42.816781       6 controller.go:419] backend reload required
I0809 20:19:42.956621       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:20:12.822210       6 controller.go:419] backend reload required
I0809 20:20:12.956532       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:20:42.822127       6 controller.go:419] backend reload required
I0809 20:20:42.959075       6 controller.go:428] ingress backend successfully reloaded...
I0809 20:21:12.815628       6 controller.go:419] backend reload required
I0809 20:21:12.951984       6 controller.go:428] ingress backend successfully reloaded...
...

I seem to have found the culprit in my case: a misconfigured socket.io ping interval / timeout. After adjusting these according to NGINX read/send timeout (and ELB idle timeout, since I'm running this on AWS), this is how things are working now:

Clients connect to the WebSocket-based application
Ingress controller reloads the configuration (i.e. when the endpoints for some service change), spawning new workers
The old workers will not be terminated until all open connections handled by them are closed. Since we are talking about WebSockets, these connections might take a long time to finish; the new worker_shutdown_timeout parameter does not have any effect on this, as far as I could tell

From the client PoV, this is nice because we avoid dropping WebSockets at every configuration reloads (which can happen quit frequently in larger deployments).
However, from the server PoV, you might end accumulating workers in 'shutting down' state (which might cause the elevated memory consumption as I showed earlier) depending on how long the WebSocket connections are kept open and how many times the configuration gets reloaded by the ingress controller.
After spending some time on this problem, it seems like the only way to mitigate this is to find a way to keep the rate of configuration reloads as low as possible. One way of achieving this is running a dedicated ingress deployment just for this WebSocket app, but it seems a bit overkill.
@aledbf What do you think?
Sorry for hijacking this thread, but it has gone too far already. 😅

Ingress controller reloads the configuration (i.e. when the endpoints for some service change), spawning new workers

Can you increase the log level to 2? (flag --v=2) so you can see the diff of the configuration and which service is the guilty of the changes? The number of reloads is not normal (I have more than 300 apps and I cannot reproduce that)

@danielfm are you using slack (k8s channel)?

@aledbf Yes, my handle is 'danielmartins'.
I've increased the logs and, as far as I can tell, the only thing triggering configuration reloads are changes in upstreams. (Well, not exactly, since apparently the list of endpoints and servers are identical, the only thing that changed was the order in which they got rendered):
I0810 23:14:47.866939       5 nginx.go:300] NGINX configuration diff
I0810 23:14:47.866963       5 nginx.go:301] --- /tmp/a072836772	2017-08-10 23:14:47.000000000 +0000
+++ /tmp/b304986035	2017-08-10 23:14:47.000000000 +0000
@@ -163,32 +163,26 @@

     proxy_ssl_session_reuse on;

-    upstream production-chimera-production-pepper-80 {
+    upstream upstream-default-backend {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:8080 max_fails=0 fail_timeout=0;
     }

-    upstream production-gabarito-production-80 {
+    upstream production-landings-production-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
     }

     upstream production-sisu-production-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
         server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-    }
-
-    upstream production-lap-production-worker-80 {
-        # Load balance algorithm; empty for round robin, which is the default
-        least_conn;
-        server <IP_ADDRESS>:9292 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:9292 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
     }

     upstream production-passepartout-production-80 {
@@ -201,61 +195,67 @@
     upstream production-lap-production-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server <IP_ADDRESS>:8000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:8000 max_fails=0 fail_timeout=0;
         server <IP_ADDRESS>:8000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:8000 max_fails=0 fail_timeout=0;
         server <IP_ADDRESS>:8000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:8000 max_fails=0 fail_timeout=0;
     }

-    upstream production-desauth-production-80 {
+    upstream production-chimera-production-pepper-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
     }

-    upstream production-live-production-80 {
+    upstream production-gabarito-production-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server <IP_ADDRESS>:5000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:5000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:5000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:5000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
     }

-    upstream upstream-default-backend {
+    upstream production-chimera-production-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server <IP_ADDRESS>:8080 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
     }

-    upstream production-landings-production-80 {
+    upstream production-lap-production-worker-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:9292 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:9292 max_fails=0 fail_timeout=0;
     }

-    upstream production-chimera-production-80 {
+    upstream production-desauth-production-80 {
         # Load balance algorithm; empty for round robin, which is the default
         least_conn;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
-        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:3000 max_fails=0 fail_timeout=0;
+    }
+
+    upstream production-live-production-80 {
+        # Load balance algorithm; empty for round robin, which is the default
+        least_conn;
+        server <IP_ADDRESS>:5000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:5000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:5000 max_fails=0 fail_timeout=0;
+        server <IP_ADDRESS>:5000 max_fails=0 fail_timeout=0;
     }

     server {

All other configuration reloads looks like this.

@danielfm , maybe it is not too late, but "--sort-backends=true" should help :)

@redbaron Haha thanks, after I hit this, I noticed the latest version introduced that flag and started using it right away. 😄


better default reverb & chorus settings

FluidSynth's default reverb and chorus settings are... kind of gross. Fortunately, it is possible to alter these in the plugin GUI, but why not also make them better out-of-the box? Here are the settings I will be suggesting for use with all of my SoundFonts:
synth.reverb.damp = 0.3
synth.reverb.level = 0.7
synth.reverb.room-size = 0.5
synth.reverb.width = 0.8
synth.chorus.depth = 3.6
synth.chorus.level = 0.5
synth.chorus.nr = 4
synth.chorus.speed = 0.36

Here is audio comparing the original with my revised settings. The instrument featured is the FM Electric Piano from my upcoming Fluidity GM, and the reverb and chorus levels are set as CC91 = 117 and CC93 = 87.
Note that default reverb & chorus.flac had to be attenuated 6 dB to avoid clipping due to the massive amount of volume that the default chorus adds.

Hello @mrbumpy409,
thanks for your input!
But I am a bit hesitant to change the default settings of the plugin and I'll try to explain my reasons.
The central idea of the FluidSynth plugin is that  - according to the documentation - it tries to

exactly reproduce an external fluidsynth rendition

Once I am changing the defaults of the plugin to some more "convenient" values, this is hard to do: when I wanted to mimic the external fluidsynth command line processor, I would have to override settings in the plugin to compensate those changes.  Another way would be to use some settings file for fluidsynth with the exact same settings.
Both variants are not very practical.
And normally when you are working in a DAW, you have plugins available that are much more elaborate than the built-in reverb or chorus of FluidSynth.  Examples are the free chorus Azurite or the ReaPlugs stock plugins of Reaper with Reaper ReaVerb even using impulse responses.  And if you really need DAW plugins that have command-line counterparts, my notorious SoXPlugins could also be helpful (although there is no chorus there...).
Chains of specialized plugins can be tuned to your individual instrument much better; I guess that some standard one-size-fits-all fluidsynth setting will not be very helpful once you are trying to optimize the sound for an electric bass or a pad synth.
So my standard technique is even to switch off reverb and chorus completely in the FluidSynth plugin and in the external rendering chain (via synth.chorus.active = 0 and synth.reverb.active = 0) and then use other means to produce audio enhancements.
If you really want to use the fluidsynth chorus and reverb, you could - as a workaround - take your settings and make a preset of them for initialization.  The disadvantage of that approach is that once you load that preset, all other data is dismissed.  Another strategy is to just use copy-and-paste: this strategy benefits from the simplistic text interface... Or even combine both approaches by temporarily cutting the text parts you want to keep and then load your preset.
So this time it seems I have to pass on your request.
But maybe you have further convincing arguments I cannot refuse...
Best regards
Prof. Spock

I cannot argue with your logic. Also, it appears that FluidSynth will be changing the default reverb and chorus to settings similar to mine in version 2.4 anyway. I don't mind pasting my custom settings into each plugin instance where I want to use them. I just figured the change might be nice for newcomers to the plugin wondering why the default fx engine sounds so gnarly.
Peace :smile:

Hello @mrbumpy409,
well, whenever Fluidsynth 2.4 comes out, you simply drop in the DLL or SO file and the settings you obviously recommended upstream will be available as the new default. And then, of course, this will be the new mandatory reference for the FluidSynth plugin.
Be aware that fluidsynth has a different usage profile, because people use it to just render complete MIDI files to audio. On the other hand a DAW plugin normally transforms a single voice in a DAW having a multitude of other plugins available for postprocessing.
Nevertheless I have no problem with that upcoming change at all, but probably I shall also shut those off by silencing chorus and reverb for the reasons given.
I only want to ensure that the FluidSynth plugin is - as much as possible - conforming to the fluidsynth command-line program. This was the original intention: a "bit-exact" reproduction of fluidsynth in a DAW...
The only problem I am going to have is that people will complain that version X of the FluidSynth plugin will not sound identical to version X+1 when using default settings, although I have not changed anything.
But I can live with that :wink:
Best regards,
Prof. Spock

well, whenever Fluidsynth 2.4 comes out, you simply drop in the DLL or SO file and the settings you obviously recommended upstream will be available as the new default.

Well, they didn't end up using my settings, but their settings are pretty close. Well enough to make a huge improvement anyway. I like the final iteration of their reverb, but the chorus is a bit too tame for my taste.

Nevertheless I have no problem with that upcoming change at all, but probably I shall also shut those off by silencing chorus and reverb for the reasons given.

I only want to ensure that the FluidSynth plugin is - as much as possible - conforming to the fluidsynth command-line program. This was the original intention: a "bit-exact" reproduction of fluidsynth in a DAW...

I will be recommending FluidSynthPlugin in my documentation as the best way to experience my SoundFonts in a DAW. Some people may still want something like juicysfplugin due to the GUI, but that plugin's timing resolution is limited to the current ASIO buffer size, it can't adjust reverb/chorus/interpolation settings or load only the active preset to memory, and it appears to have been pretty much abandoned. There's also no working Linux version. All of which just makes me all the more appreciative of your efforts!

[juicysfplugin has] also no working Linux version.
All of which just makes me all the more appreciative of your efforts!

Thanks for your kind words, that's music to my ears :smile:
And there is even a MacOS AU version :wink:, so the main x86-64 platforms are covered (but I have to admit that there is no MacOS ARM version, which is somewhat careless).

Hello mrbumpy409,
the new release v0.7.5 provides the fluidsynth library 2.4.0 (at least for Windows and MacOSX, see your issue https://github.com/prof-spock/FluidSynthPlugin/issues/17).
Since the FluidSynth team has added your issue to that release, the problem should be solved.
Please let me know, whether this is correct or not.
Best regards,
Prof. Spock


Socket hang up when trying to login to my Azure account

Azure Data Studio Version:

Version: 1.48.1 (user setup)
Commit: 8825ec21e67a34ede00e4f02a4afc692eb6cab61
Date: 2024-06-13T21:07:19.253Z
VS Code: 1.82.0
Electron: 25.9.8
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.19045

OS Version: Windows 10 22H2, B19045.4529

Steps to Reproduce:

Install Azure Data Studio
Login with my Azure account

After I selected my account from the browser, I got a message saying "socket hang up" on the browser and in the Data Studio.
I have tried to re-install the application and delete AppData/Roaming/azuredatastudio folder.
In the browser, the URL looks like:
http://<IP_ADDRESS>:56969/callback?code=...&state=...&session_state=...

The port number changes every time I restart the Data Studio.

Does this issue occur when all extensions are disabled?: Yes

Hi @davidshen84 can you please attach logs and also check if there are any errors in the Developer Tools Console of ADS? (F12 -> Console)

@cssuh , sorry for the late reply.
F12 does nothing for me. So I tried the Toggle Developer Tools from the Help menu. It has tons of warning messages. When I try to do the login, I got only one error message.

ERR Error while adding account: Error: socket hang up

The Azure Accounts.log file content is almost the same thing.
I also tried to reinstall the app, but not helping.
Version: 1.49.1 (user setup)
Commit: 2f696aa95a13ccabd7e0a4431d1d759e3a7b8cfd
Date: 2024-08-13T18:12:38.831Z
VS Code: 1.82.0
Electron: 25.9.8
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.19045

1-Azure Accounts.log


Meilisearch should accept gzipped requests

We would like Meilisearch to accept user queries that are content-encoded in gzip. This way, users will be able to send their payloads directly in gzip to the engine, and the engine should be able to receive it blindly.
We are using actix-web for our HTTP engine, there already is a middleware that is made explicitly for that. The only setting that must be specified is the fact that we only want the engine to accept the gzip compression format (look into the features of the lib).
Another functionality that this middleware should support is compressing the output of the engine too. It is useful when requesting many documents or facets, for example.
We want this feature to be tested to ensure that nothing wrong can happen when a user sends compressed documents or wants to receive a compressed HTTP response. You can look at the actix-web tests to start.

Edit from @curquiza
Impacted team
@meilisearch/docs-team this is something we would like to see mentioned in the documentation but NOT detailed. I mean, people should find the information about "the engine can accept gzip content" when typing "gzip" or "compressed". No need to detailed, this is something people already know if they need it, and 90% of the job is done by the browser
TODO

[ ] Implement the changes and merge to the main branch
[ ] Update the related spec

I'll try solving this!

Hello @amab8901
Thanks for your involvement 😇

that makes sense. Also if someone finishes it before me, then I can use their solution as feedback on what I missed in my own attempts

I have run the below cURL and the index did seem to have created.
echo '{"uid": "sai-test-temp","primaryKey": "id"}' | gzip | curl -v -i --data-binary @- -H "Content-Encoding: gzip"  -H "Content-Type: application/json" http://localhost:7700/indexes
Below is the full verbose mode of this cURL
` $ echo '{"uid": "sai-test-temp","primaryKey": "id"}' | gzip | curl -v -i --data-binary @- -H "Content-Encoding: gzip"  -H "Content-Type: application/json" http://localhost:7700/indexes

Trying ::1:7700...
TCP_NODELAY set
connect to ::1 port 7700 failed: Connection refused
Trying <IP_ADDRESS>:7700...
TCP_NODELAY set
Connected to localhost (<IP_ADDRESS>) port 7700 (#0)

POST /indexes HTTP/1.1
Host: localhost:7700
User-Agent: curl/7.68.0
Accept: /
Content-Encoding: gzip
Content-Type: application/json
Content-Length: 59

upload completely sent off: 59 out of 59 bytes
Mark bundle as not supporting multiuse
< HTTP/1.1 202 Accepted
HTTP/1.1 202 Accepted
< content-length: 129
content-length: 129
< access-control-allow-origin: *
access-control-allow-origin: *
< content-type: application/json
content-type: application/json
< vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
vary: Origin, Access-Control-Request-Method, Access-Control-Request-Headers
< date: Sun, 02 Oct 2022 18:11:59 GMT
date: Sun, 02 Oct 2022 18:11:59 GMT

<

Connection #0 to host localhost left intact
{"taskUid":7,"indexUid":"sai-test-temp","status":"enqueued","type":"indexCreation","enqueuedAt":"2022-10-02T18:11:59.154494098Z"}
`
Based on what I've understood, if we were to compress a request using gzip compression, then the request is expected to either not work, or fail. This does not seem to be happening here.

Is my interpretation correct?

Hey @psvnlsaikumar,
It is strange indeed as we never changed anything on our side to accept compressed requests. This issue was raised because we wanted our users to be able to send documents, e.g. CSV, JSON, JSON-lines, in a compressed manner. On the other hand, we would also like to compress a server answer if it is possible.

Is my interpretation correct?

Don't you think that it would be possible that cURL would do a preflight call to the server to check that it support gzip compressed content and uncompress it on the fly? I would find it very strange. I am just guessing...

@Kerollmops
if the cURL would do a preflight call, and then uncompress it before sending the request, the content-encoding header of the request would have been changed is what I am expecting
I am not sure if it would uncompress the requests while in transit though. or maybe there is some change in the actix?
Were there any logs that we can look at when the request failed from the time when we created the issue?

@Kerollmops
I guess the request works fine for creating indexes, but fails on creating documents. I just tried sending request for documents, and it seems to be failing.
Reference cURL
cat movies.json | gzip | curl -v -i --data-binary @- -H "Content-Type: application/json" -H "Content-Encoding: gzip" -X POST http://localhost:7700/indexes/sai-test-temp/documents

response :
"message":"The `json` payload provided is malformed. `Couldn't serialize document value: expected value at line 1 column 1`.","code":"malformed_payload","type":"invalid_request","link":"https://docs.meilisearch.com/errors#malformed_payload

Why do you have to write --data-binary on the curl? Isn't the data already binary from the gzip?

Oh! I found how to reproduce the problem for POST method (inserting docs into the database):
cat meteorites.json | gzip | curl -vi --data-binary @- -H "Content-Type: application/gzip" -X POST http://localhost:7700/indexes/meteorites/documents

Here is the output:
{"message":"The Content-Type `application/gzip` is invalid. Accepted values for the Content-Type header are: `application/json`, `application/x-ndjson`, `text/csv`","code":"invalid_content_type","type":"invalid_request","link":"https://docs.meilisearch.com/errors#invalid_content_type"}

That looks about right!

The reason why some requests support various content-encodings and others do not lie in structures representing requests. Most of the bodies are represented by actix built-in JSON-related structs, which utilize decompression logic (via Payload wrapping). Still, API for documents uses custom-implemented stream-like structures (I think to avoid huge instant allocations).

so what is left to do, now that a commit has been submitted?

Hello @amab8901 we will review the PR soon, and we'll let you know if something is missing and still need to be implemented 😄

For meili teams: I've added the spec freshly opened to the issue description
https://github.com/meilisearch/specifications/pull/178


Simplify the reindexing process

It would be nice to have the following code snippet (taken from the docs) wrapped in a single method call:
// Block until the SchemaStatus transitions from INSTALLED to REGISTERED
registered = false
before = System.currentTimeMillis()
while (!registered) {
    Thread.sleep(500L)
    mgmt = g.getManagementSystem()
    idx  = mgmt.getGraphIndex("mixedExample")
    registered = true
    for (k in idx.getFieldKeys()) {
        s = idx.getIndexStatus(k)
        registered &= s.equals(SchemaStatus.REGISTERED)
    }
    mgmt.rollback()
}

Suggestion:
// Block until the SchemaStatus transitions from INSTALLED to REGISTERED
mgmt = g.getManagementSystem()
mgmt.getGraphIndex("mixedExample").await() // perhaps with an optional timeout:
//   .await(10000) throwing a TimeoutException if the process doesn't finish within the
// given time span

discussed with @mbroecheler, it's best to close and reopen ManagementSystem often; it does cache, and it should not be held open longer than necessary to avoid obstructing tx-closure-dependent ops

I added a pair of utility classes for this:

GraphIndexStatusWatcher
RelationIndexStatusWatcher

To make their use in the shell a bit more concise, I also added public static ManagmentSystem methods:

ManagementSystem.awaitGraphIndexStatus
ManagementSystem.awaitRelationIndexStatus

An example invocation from the docs:
mgmt.awaitGraphIndexStatus(g, "mixedExample").call()

There are setters for controlling the target status (defaults to REGISTERED), timeout (defaults to 1 minute), and fixed polling interval (defaults to 500 ms).


SELECT doesnt not use index for OR clause not on the main level

v 2.2.3
SELECT ... WHERE f2=y or f3=z is OK
SELECT ... WHERE f1=x and (f2=y or f3=z) will not use index for f2 and f3 - why?

Hi @vantonov1
It's a known limitation, we are working on it for next versions
Thanks
Luigi

@luigidellaquila is this already supported in 3.0?

Yes, it's already supported since v 3.0 M1
Closing, thanks
Luigi


[#445] Add jest

close: #445

We need some documentation (doc/TESTING.md), and an example on how to write a test case - f.ex. a test case that verifies the front page


Fix arguments of Auth0JWT::decode call for 1.x.x

Hey, I need to fix the minimum requirement for the auth0-php sdk (I will push the change right now).
If you want to use the latest version, please use the latest auth0-php sdk version too (`~3.3).
Otherwise, use jwt-auth-bundle 1.2.8
I will let you know the new release version

try with 2.0.2 to use latest


markdown file can't generate the header with and end with

I want to generate the header with
some thing

Can you solve it ? thinks

I had solved it by add
let b:block_comment = 1

and changed the comment_start to comment_begin


make exportmap procedures static instead of monkeypatching exportmap

The following procedures:
ExportMap.get
ExportMap.for
ExportMap.parse

are conceptually static methods of ExportMap, but instead of using the static keyword they are monkey-patched onto the ExportMap class
ExportMap.get = function (...) {...}

which is surprising.
Lets make them static methods proper.

Tip for the reviewer/maintainer:
Use the "Hide whitespace" feature of GitHub's diff.

you're right that this repo doesn't enforce that rule for some reason; i'll fix it shortly :-)


Pagination issues for blog use

I tried to use clean blog,default and just pico themes but unable to get pagination working.
I am using the Pico Pagination plugin and set $config['pagination_limit'] in config file, I get the older posts link but when I click on it, it doesn't go to the next page and show the other posts for a blog.
Appreciate if someone can help with this
Thanks

Works just fine for me. Make sure to strictly follow the theme's and rewdy/Pico-Pagination's installation instructions (rewdy/Pico-Pagination e.g. only considers pages with a Date meta value). Please note that themes (like BesrourMS/clean-blog) must know about the pagination plugin, i.e. neither the default theme nor blocknotes/just-pico won't ever show a pagination until you actually modify the theme accordingly.
If you still can't make it working, please refer to the 3rd party theme and/or plugin developers:

@rewdy's Pico-Pagination plugin
@BesrourMS's clean-blog theme

Hi,
Thanks for the response, much appreciated.
I am using clean-blog and rewdy/Pico-Pagination and it is showing pagination but if I limit 4 articles to the first page and want to see the next 4 articles by clicking on the older posts then it does not show those old 4 articles and maybe i am missing something and will review it again.
Could you please let me know what theme you are   using since pagination works fine for you?
Thanks again.

BesrourMS/clean-blog with rewdy/Pico-Pagination. However, I'm just making sure that this is neither a issue with Pico, nor an obvious issue with the theme or plugin. Please refer to the 3rd party theme and/or plugin developer for getting help.

Hi,
Thank you so much for your help.
I found out what the problem was, when i was using the built in php web server it didnt work but when i started using apache web server pagination started working, so I think i had configured it properly but maybe there was something different in the way i was running it with the built in  php web server.
I am using BesrourMS/clean-blog with rewdy/Pico-Pagination.
It is all working fine, thanks for your time and help.
Thanks again.


a fix to a bug in wix ui tpa and stylable mixing

there is a bug that only happening in stylable mixin and wix-ui-tpa
for example if you use them with this definition
-st-mixin: TPAText(
    MainTextFont '"fallback(--someVar, font({theme: 'Body-M', size: '16px', lineHeight: '1.33em'}))"'
  );

then the fallback inside is messed up

are you aware that this is the old library? you need to change in tpa-style-webpack-plugin if you need it in OOI

Apparently not


Doesn't compile both less and sass files

I have following configuration:
mix.js('resources/assets/js/app.js', 'public/js').extract(['vue'])
.less('resources/assets/less/theme.less', 'public/css')
.sass('resources/assets/sass/app.scss', 'public/css');
However less files are not compiled, it works fine if I comment out .sass line.

Support for this was added just recently. Update your laravel-mix dependency to 0.6.0


DiscretePredictionStatistics.getRecall() Incorrect

DiscretePredictionStatistics.getRecall() is giving some incorrect numbers.

Fixed. Division by integers was replaced by division by doubles.
Also includes test classes, and style changes.


more tutorial tweaks

moderate overhaul of freesurfer background tutorial; the rest is just cleanup of some old TODO items, and removal of no-longer-required :class: tags along the way.

This is ready for review/merge.


[BUG] : Small Screen Footer is not aligned, left margin need even-ness

on squeezing the screen size footer also a lot attached to left side, adding margin or padding on left would make it look more appealing.

@Sudhanshu2306 Proceed

I see this is already resolved, and currently the footer looks fantastic, so no need for the issue. Thank you!


Rename "make" to "make.sh"

I think that renaming "make" to "make.sh" would be useful to clarify that it is a shell script. Many tools also do not recognize the file to be a shell script without the ".sh" suffix - syntax highlighters, in particular.
I also suggest adding a "shebang" (#!/bin/bash) to the top of the file.

Retracting this PR per advice in https://google.github.io/styleguide/shell.xml#File_Extensions

Executables should have no extension (strongly preferred) or a .sh extension.


Implement a way to get/set window state

[Upstream issue](Implement a way to get/set window state)

Your link seems broken.

Fixed, thank you!


[IDEA] Separate "views" and "components"

We have two types of renderers:

Components (follow the standard component pattern); data is bound to the selection passed to them and configuration is set using methods on the component
Views (are composed of components); both data and configuration properties are bound to the selection passed to them, there are no methods for configuration

https://github.com/ScottLogic/d3fc-showcase/commit/b8d21c337a58a8476b2425c7a1c624539471d980 separates the views into their own namespace - is this a more logical/intuitive grouping?

:+1:

We have two types of renderers

Maybe that's the issue. If they only differ in the way state is passed in, I feel we should use components for the charts too.

Maybe that's the issue. If they only differ in the way state is passed in, I feel we should use components for the charts too.

Following the chat earlier, are we happy with the approach of separate 'views' and 'components' (as defined above)? Naming probably isn't the best, so happy to rename if there are better ideas...
I'll also strip any methods from the models (also happy to rename 'models' if there's a better name).

[ ] Remove methods from models

With 'views' rename - :+1:

[ ] Remove methods from models
[ ] Rename view to chart


Added tests for DRILL-4862

@kfaraaz, I closed your previous PR, rebased your master over mapr/master, created a new branch with your commits for DRILL-4862, and have raised this as a new PR.
Please verify that the commit here looks good. Also, please schedule a new regression test and once you have a clean run, please add a comment in here. I'll have this PR committed.

@Agirish I have verified, and we have a clean pre-commit test run. Please merge. I will share link to pre-commit run. There are some failing tests that are not related to this commit.
Thanks,
Khurram

Thanks.


Test upstream branch release-1.15

This demonstrates that the upstream branch release-1.15, which uses C++17, can be compiled in the conda feedstock.
This is only for documentation purposes and should not be merged.
Also note that we will still need to create a pre-check PR once tiledb 2.27 is available as a conda binary.
xref: https://github.com/single-cell-data/TileDB-SOMA/pull/3389, https://github.com/TileDB-Inc/tiledbsoma-feedstock/pull/234

CI passed!


Gate accessory vs garage

I'm using the garage example as the basis for a gate controller. It works well. However Siri doesn't know what a gate is and when I ask Siri to "close the gate" she responds with "your garage door is closing" - which is disconcerting at times as I have a commercial HomeKit garage controller installed as well. I see no mention of a gate accessory type in Apple's HomeKit documentation, so I guess this is expected behaviour..?

Are you sure it's not because you have given them such names? What name have you assigned?

The gate controller is called "Gate"..

Ok. And just to clarify: are you saying that your commercial HomeKit gate opener behaves differently?

Sorry if that wasn't clear. No, my commercial controller is connected to my garage - so it's fine. It's just sometimes confusing because when I ask Siri to "shut the gate" her response is "your garage is closing" - because the gate controller is registered as a garage accessory type.
Not sure there's anything actionable here other than confirming there is no "gate accessory" type available...

Not sure I understand "my commercial controller is connected to my garage - so it's fine". What is fine? What is "my garage"? If it's fine, so what is the problem? If you're referring to your own controller "not being fine", then why wouldn't you connect it to "your garage" as well, so it could also be "fine"?

because the gate controller is registered as a garage accessory type
What is "the gate controller" ? There is not such thing as "garage" accessory type. There is a "Garage Door Opener" accessory type (at least in non-commercial specification. If you have an example that proves it wrong, I would incorporate new services definitions into this library).

OK, forget about the commercial garage controller. It's attached to a different structure and tangential to the story..
When I said "garage accessory type" I meant specifically "homekit_accessory_category_garage". I'm assuming there is no "homekit_accessory_category_gate" defined (I don't see it anywhere - either in your source, or the Apple doco), which is why Siri refers to my device as a garage even though it's being used on a gate (and is called "gate").
Anyway, I think we're going in circles. I'll close this.
FWIW, Siri works very well for me...

@pocketscience, The best approach here is to change the name of your commercial Garage door controller in Home app and test it with Siri. If the response is the same as you mentioned above, so this issue should be addressed by the Siri developers.


Output Calibration for multi-label and multi-class settings

Label smoothing https://arxiv.org/abs/1512.00567, https://arxiv.org/abs/1906.02629
Label relaxation (https://ojs.aaai.org/index.php/AAAI/article/view/17041), and
Laplace approximation (https://arxiv.org/abs/2106.14806)

(1) and (2) are ready to use in the 1vsAll training strategy (multi-class setting).
(1) can be readily used in KvsAll training stategy (multi-label setting) as commonly done in the literature. Yet, It is not very obvious to me how to extend (2) for multi-label classification.
Laplace approximation will be added after the model's interface is modifed,

We could not complete (3) due to the problem described in https://github.com/AlexImmer/Laplace/issues/90

from laplace import Laplace
from laplace.utils.subnetmask import ModuleNameSubnetMask
from laplace.utils import ModuleNameSubnetMask
from laplace import Laplace

subnetwork_mask = ModuleNameSubnetMask(model, module_names=['emb_rel_real'])
subnetwork_mask.select()
subnetwork_indices = subnetwork_mask.indices
# User-specified LA flavor
la = Laplace(model, 'classification',
             subset_of_weights='subnetwork',
             hessian_structure='full',
             subnetwork_indices=subnetwork_indices)
la.fit(dataset.train_dataloader())

Works like charm!


async support

adapter = await  casbin_pymongo_adapter.Adapter('mongodb://localhost:27017/', "dbname")
Do you have plan to adopt async features.??

@Zxilly

will working On this

Is this issue resolved or not? @Zxilly   I'm applying this plan and wanna to take this story.

@AmosChenYQ It's welcome.

@Zxilly Sorry for the late response, I just found that connect to a database or get a collections is NO I/O needed....


Fixes Machine Item Consumption

Description
Fixes Machine Item consumption, which leads to infinite recipe reloading when the output is full trying to find a recipe that could be placed.
Closes #409

Todo

[ ] Things that are yet to be completed for this PR to no longer be a draft.

Checklist:

[ ] My code follows the style guidelines of this project (.editorconfig, most IDEs will use this for you).
[ ] I have performed a self-review of my own code.
[ ] I have commented my code in areas it may be challenging to understand.
[ ] I have made corresponding changes to the documentation.
[ ] My changes are ready for review from a contributor.

I'm double check, but this does seem to be a leftover. What this check does is look if the item can be placed as an output, and if not, it will restart the crafting task. However now we can always start the task, but we just need to wait to place the outputs until there's room. For other machines, this doesn't matter, as the recipe will always have the same outputs. In that case the task never starts until room is made.


chore: bdd-spring-1573168004 to 0.0.1

chore: Promote bdd-spring-1573168004 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Bug fix: html.parser was not working

html.parser was not working.
python version = 3.9.1
output =
Traceback (most recent call last):
File "/home/d.py", line 3, in
ret = mtranslate.translate("hello","tr","en")
File "/home/mtranslate/core.py", line 87, in translate
result = unescape(re_result[0])
File "/home/mtranslate/core.py", line 57, in unescape
return (parser.unescape(text))
AttributeError: 'HTMLParser' object has no attribute 'unescape'

Thank you for your contribution !

Thank you for your contribution !

You're welcome

You're welcome


docs: 📝 Fix spelling error on README.md

Unable to sign the commits? Did something happen with your GPG keys?


add search depth for WindowSpecification as a search_criteria

when an app has  lots of elements and big tree, it's slow to find element, add depth like print_control_identifiers(self, depth=None)

Implemented in #348. Currently in master branch. Will be included into 0.6.3 coming soon.


Codecamp not accepting my answers

Describe your problem and how to reproduce it:
Add a Link to the page with the problem:
Tell us about your browser and operating system:

Browser Name:
Browser Version:
Operating System:

If possible, add a screenshot here (you can drag and drop, png, jpg, gif, etc. in this box):

My code camp is not accepting my answers

https://learn.freecodecamp.org/responsive-web-design/basic-css/change-the-color-of-text/
My code camp is not accepting my answers

@JContreras210 Please use our forum when you experience problems with your code not passing tests of a challenge.  If members of the forum believe there is an actual problem with the curriculum and not your code, then feel free to request this issue be reopened.  Always start in the forum when you have trouble with a challenge.  You can also just click on the "Ask for Help" button located on each challenge to facilitate asking for help.
Thank you.

Thank you
i was able to figure it out. I was not on the correct web browser.
Sent from my iPhone

On Oct 10, 2019, at 10:08 PM, Randell Dawson<EMAIL_ADDRESS>wrote:
Closed #37196.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.


Add bordered inline alert to warn user about unresponsive systems

Description
Add bordered inline alert to warn user about unresponsive systems or systems without available updates
Fixes # (issue)
Type of change
What is it?

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Documentation update
[ ] Tests update

Checklist:

[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] I run npm run lint:js:fix to check that my code is properly formatted

Codecov Report
Base: 30.44% // Head: 30.44% // No change to project coverage :thumbsup:

Coverage data is based on head (f280d4a) compared to base (10c7d8a).
Patch coverage: 0.00% of modified lines in pull request are covered.

:exclamation: Current head f280d4a differs from pull request most recent head 6b7ed3d. Consider uploading reports for the commit 6b7ed3d to get more accurate results

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #566   +/-   ##
=======================================
  Coverage   30.44%   30.44%
=======================================
  Files         126      126
  Lines        2736     2736
  Branches      847      847
=======================================
  Hits          833      833
  Misses       1744     1744
  Partials      159      159

Impacted Files
Coverage Δ

src/Routes/Devices/UpdateDeviceModal.js
0.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


doc/vision.md

Being somewhat late to vello, having not played with it during the piet-gpu stages of development.
It reads to me like doc/vision.md is somewhat out of date with the current state of things,
In particular the section Why not wgpu doesn't seem relevant anymore since vello does use wgpu.
So that could perhaps use a pass over it to better reflect the current state of things.

See https://github.com/linebender/vello/issues/488


Allow to specify target versions of ECMAScript > 2015.

We introduce a new setting ESFeatures.esVersion to select specific ES versions to target for compliance. Values of ESVersion are defined from ES 5.1 up to ES 2020. The existing useECMAScript2015 is remapped to be the same as esVersion >= ESVersion.ES2015.
We leverage esVersion >= ESVersion.ES2017 in one place in the core JS lib. More tests like that could be introduced in the future.
We add esVersion in the generated BuildInfo for the test suite, and use it to better constrain some our tests of JS features.
The esVersion is also made available to user-space libraries through LinkingInfo.assumedESVersion, like assumeES6. This is not used yet in this commit, but will be used by the new regex support.
We enhance NodeJSEnvForcePolyfills and our Jenkins scripts to test with various target versions of ECMAScript.

Extracted from #4455 because I think this is generally useful, and it can be independently reviewed.

Hahaha, there is also a typo in the title EMCA vs ECMA

I think we should keep useECMAScript2015 as not deprecated, along with all its friends. IMO there is a qualitative difference between ES2015 and the following ones in Scala.js. Our semantics are specified in terms of ES2015, and when compiling to ES5.1, we do a best effort implementation of our semantics on top of that, but we lose some language semantics. That's why we've had this distinction for ages, even if we didn't have the following ECMAScript versions. Today we're adding them but mostly for library usage, not for language semantics. So I think the difference ES5.1 -> ES2015 still has significantly more weight than any of the following versions.
Does that make sense?

Does that make sense?

Yes it does. My concern here is two fold:

User level (ESLevel / LinkingInfo): I'd rather not give two ways to do the same thing without at least a recommendation which one to use.
Internally (BuildInfo): I'd rather not store data / have fields where some entirely derive of the others (ESFeatures already follows this).

Maybe we should remove storage and deprecate setters, but leave getters unchanged. (in any case, the qualitative differences of ES2015 should be made clear in the documentation on relevant fields on ESFeatures).
I think with this, we can argue that usage is less ambiguous:

setting: withESVersion shall be used.
reading: esVersion shall be used if a library feature is needed, useECMAScript2015 if a specific semantic only supported by ES6 is needed.

Nevertheless, ES6 keeps its special status, made clear by the documentation explaining when and why useECMAScript2015 shall be used over esVersion >= ESVersions.ES2015.
WDYT?

That sounds very good, yes. Thank you. I'll update that on Tuesday, after the Easter break.

Updated.

Looking once more into this, I figured we should go further than what we discussed previously. There was no reason to actually link together the ES version and the ES2015 semantics. So I added a separate useECMAScript2015Semantics configuration, and cleaned up everything on the way.
Now both are stored, because it is possible to use any esVersion with useECMAScript2015Semantics == false. The only forbidden combo is esVersion == ES5_1 with useECMAScript2015Semantics = true.
I added a few preliminary commits which should be used regardless, then added two commits with those changes. If that's OK with you, I'll squash the last 3 commits.

Now both are stored, because it is possible to use any esVersion with useECMAScript2015Semantics == false. The only forbidden combo is esVersion == ES5_1 with useECMAScript2015Semantics = true.

Two main questions I have:

Are we confident that we will not use semantics of a future ES standard for something existing in Scala.js? What do we do if that happens anyways? (just define useECMAScript20XXSemantics?).
What is a use-case for targeting a higher library level, with lower semantic level? Would such use cases be better met with allowing to switch individual semantics on/off? (TBH, I'm having trouble imagining a use case for this).

FWIW: While writing this, I realized that the answer to the first question is "No". If we decide to settle to use big ints for longs, we have exactly that situation.

Are we confident that we will not use semantics of a future ES standard for something existing in Scala.js? What do we do if that happens anyways? (just define useECMAScript20XXSemantics?).
FWIW: While writing this, I realized that the answer to the first question is "No". If we decide to settle to use big ints for longs, we have exactly that situation.

In general, this would happen if:

there are new features of ECMAScript that have new, unique semantics
and we decide that the most appropriate way to express those new ES semantics is using existing Scala concepts.

This is what happened to classes and non-native JS classes, or to => functions and js.FunctionNs.
The case of BigInts for Longs is a bit special. In that case, the ES feature choice is not so much dictated by semantics than by (supposed) performance gains. We would never do that for the semantics aspects, since a Long is in fact not adequate to represent the semantics of BigInts. Now that I think about it, I'm wondering whether we should box the BigInts in a $Long when it escapes to a JS context (or a generic context), in order to hide the semantics mismatch. But that's another story.

What is a use-case for targeting a higher library level, with lower semantic level? Would such use cases be better met with allowing to switch individual semantics on/off? (TBH, I'm having trouble imagining a use case for this).

IMO the typical use case would be: I have a codebase that works with the current semantics (say ES2015 semantics); it wouldn't work with the newer semantics (say ES2021), yet I'm targeting an environment know to support ES2021. I'd like to take advantage of all the performance and/or code size improvements I can get from ES2021, without compromising the semantics of my codebase today. Sure, it's niche; perhaps even contrived. But it seems legit enough to me.
Besides the use case, though, I came to this from the point of view of the linker code. It was clearly use useECMAScript2015 for two entirely different purposes. Actually decoupling them seems to produce cleaner code in the linker, IMO.

Actually decoupling them seems to produce cleaner code in the linker, IMO.

In that case (I'll review from this POV in detail), we should definitely do it internally.

Sure, it's niche; perhaps even contrived. But it seems legit enough to me.

I'm a bit concerned about the additional complexity that this introduces in the linker configuration. I'd like to avoid adding a configuration option that isn't actually needed by anyone (both from a user and a maintenance POV).

Sure, it's niche; perhaps even contrived. But it seems legit enough to me.

I'm a bit concerned about the additional complexity that this introduces in the linker configuration. I'd like to avoid adding a configuration option that isn't actually needed by anyone (both from a user and a maintenance POV).

That is true. It's also why I kept the last commit separate for now. If we don't want the additional complexity, but we want the internal cleanup of separating the two concepts, we can remove the last commit ("And now we can separately configure them").

After looking at the code in more detail, I think we should not make useECMAScript2015Semantics configurable for now but:

Definitely make the distinction internally.
Make it clear in the documentation that useECMAScript2015Semantics might become configurable in the future.

If that's OK with you, I'll fully review once you squashed the commits.

Updated:

Removed the last commit
Squashed
Addressed comments and revisited a few more Scaladoc comments
Adapted the commit message.

Updated to address the latest review.


feat: Show errored ports as an error state

Splitting this out from #7.

Thanks! This is really useful.

(Merged)


Support IEEE488.2 packed binary formats

I know you already have this support in your experimental branch, but this adds it to the currently working base version.  No tests I'm afraid, there were no existing tests for params/results to follow along from.

I have commited corrected experimental version with completely new parser. It is now capable to handle any format defined in SCPI-99 including "arbitrary program data" aka "packed binary format". Not all other types are now fully implemented but they are possible.

Now, it is implemented in version 2 as SCPI_ResultArbitraryBlock and SCPI_ParamArbitraryBlock so closing this.


overlay profiles: Fix a couple of issues with SLSA provenance stuff

CI: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/2543/cldsv

Update to slsa 1.0. This is only partially done, as we still need to provide a proper build type. Maybe we could reuse the Github Actions Workflow (https://slsa-framework.github.io/github-actions-buildtypes/workflow/v1).

Stop using portageq - its use in ebuilds is banned, so eventually it would stop working. Replace it with our hack.

 This setup is long gone, so a commit hash of toplevel scripts repo is enough.

Use zstd for compressing generated JSON files.

--

[x] Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
[x] Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

CI passed.

CI: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/3060/cldsv/

CI passed.


[WIP] Add margin to RM training

Fix #718

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@younesbelkada could you give me a quick hint if the interface is ok that way?
Especially, as I solely use the margin if it is provided, but implicitly assume it is zero, if it is not provided. Is that ok?
Also, the tests seem to be timed out. Is that something I can fix or unrelated?

Awesome! I will look into this tomorrow/eow. Should I also add a test?

Great! If possible yes, a simple test that checks computing the loss with margin works as expected would be really great ! 🙏

Done. Let me know what you think!
One thing: I noticed that the rewards are actually tensors with shape [1, 2] even if I process just one sample. Is that correct?
batch = [[dummy_dataset[0]]]
batch = trainer.data_collator(batch)
loss = trainer.compute_loss(trainer.model, batch, return_outputs=True)

# Output:
{
  'rewards_chosen': tensor([[ 0.2797, -0.1509]], grad_fn=<IndexBackward0>)
  'rewards_rejected': tensor([[0.0570, 0.0088]], grad_fn=<IndexBackward0>)
}


CP024 Default placeholders

Deprecate access::placeholder
Extend placeholder functionality to ignore template parameter

Still applies only to global and constant buffers

Updated handler::require
New accessor constructors

Default constructor
From a buffer, 0-dim
From a buffer, ranged

New access member functions

is_null
has_handler

Vector addition example

That looks good.
A minor point I see is that before by default an accessor constructor without a CGH was a host accessor. Now it is no longer the case, since it might be a default placeholder accessor to be used in a kernel instead.

A minor point I see is that before by default an accessor constructor without a CGH was a host accessor. Now it is no longer the case, since it might be a default placeholder accessor to be used in a kernel instead.

@keryell You're right that the default when constructing an accessor will now be a placeholder instead of a host accessor, but this is only really important for CTAD, otherwise you'd still need to specify target::host_buffer. Allowing CTAD to determine either a device accessor or a host accessor based on whether the handler was provided seems dangerous to us - it is already a common mistake to call get_access without a handler, which gives you a host accessor in a command group, which is UB. At least with a placeholder you still have the option of calling require (which is valid for placeholders and non-placeholders according to this proposal), which gets rid of the UB.
There is even some interest in making host accessors a separate type, as an extension of the work here and in https://github.com/codeplaysoftware/standards-proposals/pull/100.

Allowing CTAD to determine either a device accessor or a host accessor based on whether the handler was provided seems dangerous to us - it is already a common mistake to call get_access without a handler, which gives you a host accessor in a command group, which is UB.

The compiler should warn about it at least.
But this is what I do in all the example I show because it is shorter...

The compiler should warn about it at least.
But this is what I do in all the example I show because it is shorter...

I think that calling buf.get_access<mode>() is still fine, I'm just saying that sometimes this happens because the user forgets to pass the handler - they wanted a device accessor, but got a host one instead, so with new proposals I'm keen on making more of a distinction between host and device accessors. I'm not sure if the compiler has enough information to warn on this.