Page error (connect ECONNREFUSED

Hi there,
Apologies if I'm stumbling over something obvious which I haven’t figured out from my end but I keep encountering the following error when trying to run checDev with gulp

[21:58:19] Starting 'checkDev'...
Checked 1 page and 0 links, found 1 issue.
Summary of issues:
 http://localhost:3000
  Page error (connect ECONNREFUSED <IP_ADDRESS>:3000): http://localhost:3000 (24ms)

[21:58:20] 'checkDev' errored after 1.24 s
[21:58:20] Error: 1 issue.
    at /Users/fivedoor/Documents/development/fivedoor-email-frame/node_modules/check-pages/checkPages.js:426:13
    at next (/Users/fivedoor/Documents/development/fivedoor-email-frame/node_modules/check-pages/checkPages.js:435:5)
    at Request._callback (/Users/fivedoor/Documents/development/fivedoor-email-frame/node_modules/check-pages/checkPages.js:314:9)
    at self.callback (/Users/fivedoor/Documents/development/fivedoor-email-frame/node_modules/check-pages/node_modules/request/request.js:200:22)
    at emitOne (events.js:77:13)
    at Request.emit (events.js:169:7)
    at Request.onRequestError (/Users/fivedoor/Documents/development/fivedoor-email-frame/node_modules/check-pages/node_modules/request/request.js:831:8)
    at emitOne (events.js:77:13)
    at ClientRequest.emit (events.js:169:7)
    at Socket.socketErrorListener (_http_client.js:256:9)
    at emitOne (events.js:77:13)
    at Socket.emit (events.js:169:7)
    at emitErrorNT (net.js:1253:8)
    at nextTickCallbackWith2Args (node.js:442:9)
    at process._tickCallback (node.js:356:17)

I gather from a little googling that it may be a proxy issues but I’m not sure how to resolve it and if it’s a check-pages issue or something else my end (though not had any other issue using localhost: 3000 with any other tasks in my chain.
Any advice on what the issue is / what to try / a solution greatly appreciated
Running the following:
gulp": ^3.9.1
check-pages": ^0.10.0
MACOS 10.11.4
Using the same settings as the Usage example gulp task except with localhost:3000
Thanks

If you visit the following links with a web browser, do they both work?

http://localhost:3000/
http://<IP_ADDRESS>:3000/

If so, what about if you try with curl instead?

Hi David,
Thanks for your reply.
Yes both the following work (while I have browserSync initiated I see my index page on both):
http://localhost:3000/
http://<IP_ADDRESS>:3000/
By curl do you mean https://github.com/curl/curl?
This is new to me so will read up on the site, try to get an idea of what's required and report back.
thanks for your help

curl: https://en.wikipedia.org/wiki/CURL
It's a command-line way to make web requests. For example, curl http://localhost:3000 -v will make a request and show the outgoing/incoming HTTP headers.

Updating to report that the result of
curl http://localhost:3000
is
curl: (7) Failed to connect to localhost port 3000: Connection refused
Also curl http://localhost:3000 -v results in the more verbose:
* Rebuilt URL to: http://localhost:3000/
*   Trying ::1...
* connect to ::1 port 3000 failed: Connection refused
*   Trying <IP_ADDRESS>...
* connect to <IP_ADDRESS> port 3000 failed: Connection refused
* Failed to connect to localhost port 3000: Connection refused
* Closing connection 0
curl: (7) Failed to connect to localhost port 3000: Connection refused

Seems like something's weird, then. There may certainly be a bug in my code, but curl should definitely work here. :) You mentioned above that you use a proxy? curl lets you specify proxy info via the --proxy switch - if you use a proxy and add that, is it able to get to the site? (Though it would be unusual to need a proxy to reach localhost.)

I only really mentioned the proxy as from my digging around on line that seemed to be a common issue related to ECONNREFUSED errors.
To be honest I’m still a fairly novice user of gulp and all things command line so I wouldn’t confidently rule out something horribly obvious from my end. I will try a few more things and report back (in the morning (as it’s getting late here in the UK!))… hopefully with a bit more to go on. Thanks.

Logging a few more things tried without success today-
Checked  the ports availability with  netstat -a.  3000 is not by the looks of it already in use
Tried an clean npm cache with npm cache clean then retried checkDev with same results
curl <IP_ADDRESS>:3000
results in:
curl: (52) Empty reply from server

Hum. At this point, I'd suggest troubleshooting with curl - once that works, I'm optimistic that check-pages will as well. I wonder if BrowserSync is interfering here? Do things work okay without it? The fact that netstat says the port is not in use suggests that maybe there's some trickery in place that allows the browser to think it's visiting localhost:3000 but it's really doing something else, maybe? I don't have a Mac or BrowserSync, so I'm not familiar with any issues that might be specific to them. :(

Maybe this? https://github.com/BrowserSync/browser-sync/issues/799
(See linked issue as well)

Maybe also relevant? http://stackoverflow.com/questions/27755206/using-the-original-url-not-proxy-with-browser-sync

Thanks for the suggestions David
I didn’t have a lot of luck troubleshooting browserSync based ono the suggestions on the linked pages (though tbh the scope of what is discussed re the IPv4 vs IPv6  addresses not something I have any prior experience/knowledge of so there may be more milage to be gained there than I have managed as yet).
What I did try with some success was taking browserSync completely out of the equation, stripped down my gulp file and use gulp-connect to initiate a server instead.
Using gulp-connect I no longer came up against the ECONNREFUSED error so that narrows it down at least.
Even though gulp-connect will work I would much rather be using browserSync for its capabilities so I’m hoping I will still be able to solve the issue using BrowserSync… So perhaps worth posting a query with their issues forum?
(On a side note) I ran check-pages using the default checkDev task on a test page it was picking up a bad link that was an unresolved symbol:
Bad link (404): http://localhost:8080/_UNSUBLINK__ (17ms)
however my test link http://www.deadlink-test.com was not picked up as inaccessible content which seems odd.
Thanks

Thanks for opening that bug! I hope they're able to help. :)
Regarding the bad link issue you mentioned at the bottom, can you create a small page to reproduce the problem? Paste it into a new issue and I'll have a look!

Ok great - will do. thanks!

Closing as this appears to be due to an external component because curl has the same behavior. If you find a scenario where curl works and check-pages does not' please let me know the details! Thanks.


Negative and infinite large storage multiplication edge case

I found a multiplication bug similar to the one in (#3). It's for a similar reason, but when the storage is large this time. If both inputs have bodies filled with 0s and an appendix of 1, then the product should be the combined number of 0s followed by a single 1. This is the only case that does not fit in the combined body allocation.
Test().multiplication(InfiniInt<I8>(repeating: 1) << 16, InfiniInt<I8>(repeating: 1) << 16, Fallible(1 << 32, error: false)) // value: 0 ⚠
Test().multiplication(InfiniInt<U8>(repeating: 1) << 16, InfiniInt<U8>(repeating: 1) << 16, Fallible(1 << 32, error: true )) // value: 0 ⚠

I'll be busy for a few days but it should be easy to fix.
Also, I bet my life would be easier if I stored the appendix bit as the last bit in the body instead.

I wonder if I can salvage the current algorithm by looking at its overflow indicators. That would be nice.

I suppose I could also do the standard 2's complement shuffle before and after the unsigned multiplication, which is more or less trivial to do. I don't really want that, however, because then multiplication would no longer be a borrowing operation.

I've written a solution based on the zero-prefix-dropping approach. It works, but I think it's inelegant. I'm might opt for the compact storage alternative (#7) instead. It solves the problem by just not having it.

Yeah, I'll just use compact storage. It's the lesser the headache.

The commit message says "input combination" but I suppose it's more like "input shape". There's obviously many variants of (body: 0s, appendix: 1) times (body: 0s, appendix: 1), but all of them take the same multiplication path.

If anyone ever asks, the zero-dropping approach can be summarized as:

If we are going to count zeros to catch an edge case, let us count in a way that benefits the other cases too.

The (a << x) x (b << y) speedup is pretty nifty. I wonder if I'll get a chance to leverage it deliberately.


Laravel 9.2 Issue

Hi,
I'm using it in my Laravel 9 project with the following code:

the put method returns false and no file is created.
No exception thrown.
Any help?

Found the issue. =P
change "key_file" to "key_file_path" in filesystems.php
Sorry

Hey @andreladocruz how are you?
I'm having the exact same issue. That's how I declared the configs on my filesystems.php.

How did you fixed it?

I think you must use only one key.
key_file_path or key_file.

@andreladocruz Didn't work :(
Would it possible be a misconfiguration in my bucket?
When I try to debug my storage var, it keeps returning false.

I think $request->file is an object and has a method called "storePubliclyAs".

I'm working with @MatheusAparecido on the solution of this problem.
What we already tried:

Using the JSON file provided by the GCS (authenticating through service_account).
Using the key_file array, with all variables declared in the .env file.
Using the StorePubliclyAs() and Store() methods.

Using both methods returned "false" as response when we're debugging the $disk->put('folder/', $image) method. But when we try to retrieve the file from the blobstorage, we receive it sucessfully.
@andreladocruz @AlexVanderbist any possible issues that could be causing this? :(

Has the file is been written to GCS?
I really did not check the result of a successful write.

Well, after two days trying to solve this problem, we decided to port our project back to Laravel 8 (since we had the GCS implemented in other projects in this version).

for me the put method always return false while can I still make a directory \Illuminate\Support\Facades\Storage::makeDirectory()

I found that you can add throw to config so put will throw exception rather than return false
'gcs' => [
            'driver' => 'gcs',
            'throw' => true,
]

In my case the error is
"message": "Cannot insert legacy ACL for an object when uniform bucket-level access is enabled. Read more at https://cloud.google.com/storage/docs/uniform-bucket-level-access"


Do you have a course to learn what sagefy is?

Essentially can sagefy be used to help people learn what sagefy is?

This is a great idea, thank you :)

Live at https://sagefy.org/subjects/W9UmrCsnrmgPvwp8mQRuoc :)


Hash now uses an open addressing algorithm

This improves its performance, both in time (always) and memory (generally).
Fixes #4557
The algorithm
The algorithm is basically the one that Ruby uses.
How to review this
There's just a single commit. My recommendation is to first look at the final code, which is thoroughly documented (the important bits are at the top), then at the diff.
I tried to document this really well because otherwise in a month I won't remember any of this. It also enables anyone to understand how it works and possibly keep optimizing things.
Why?
The old Hash implementation uses closed addressing: buckets with linked lists. This apparently isn't great because of pointer chasing and not having good cache locality. Using open addressing removes pointer chasing and improves cache locality. It could just be a theory but empirically it performs much better.
Give me some benchmarks!
I used this code as a benchmark:

Code for benchmarking old Hash vs. new Hash
require "benchmark"
require "./old_hash"
require "random/secure"

def benchmark_create_empty
  Benchmark.ips do |x|
    x.report("old, create") do
      OldHash(Int32, Int32).new
    end
    x.report("new, create") do
      Hash(String, String).new
    end
  end
end

def benchmark_insert_strings(sizes)
  sizes.each do |size|
    values = Array.new(size) { Random::Secure.hex }

    Benchmark.ips do |x|
      x.report("old, insert (type: String, size: #{size})") do
        hash = OldHash(String, String).new
        values.each do |value|
          hash[value] = value
        end
      end
      x.report("new, insert (type: String, size: #{size})") do
        hash = Hash(String, String).new
        values.each do |value|
          hash[value] = value
        end
      end
    end
  end
end

def benchmark_insert_ints(sizes)
  sizes.each do |size|
    values = Array.new(size) { rand(Int32) }

    Benchmark.ips do |x|
      x.report("old, insert (type: Int32, size: #{size})") do
        hash = OldHash(Int32, Int32).new
        values.each do |value|
          hash[value] = value
        end
      end
      x.report("new, insert (type: Int32, size: #{size})") do
        hash = Hash(Int32, Int32).new
        values.each do |value|
          hash[value] = value
        end
      end
    end
  end
end

def benchmark_read_strings(sizes)
  sizes.each do |size|
    values = Array.new(size) { Random::Secure.hex }

    old_hash = OldHash(String, String).new
    new_hash = Hash(String, String).new

    values.each do |value|
      old_hash[value] = value
      new_hash[value] = value
    end

    Benchmark.ips do |x|
      x.report("old, read (type: String, size: #{size})") do
        values.each do |value|
          old_hash[value]
        end
      end
      x.report("new, read (type: String, size: #{size})") do
        values.each do |value|
          new_hash[value]
        end
      end
    end
  end
end

def benchmark_read_ints(sizes)
  sizes.each do |size|
    values = Array.new(size) { rand(1_00_000) }

    old_hash = OldHash(Int32, Int32).new
    new_hash = Hash(Int32, Int32).new

    values.each do |value|
      old_hash[value] = value
      new_hash[value] = value
    end

    Benchmark.ips do |x|
      x.report("old, read (type: Int32, size: #{size})") do
        values.each do |value|
          old_hash[value]
        end
      end
      x.report("new, read (type: Int32, size: #{size})") do
        values.each do |value|
          new_hash[value]
        end
      end
    end
  end
end

sizes = [5, 10, 15, 20, 30, 50, 100, 200, 500, 1_000, 10_000, 100_000]

benchmark_create_empty()
puts
benchmark_insert_strings(sizes)
puts
benchmark_insert_ints(sizes)
puts
benchmark_read_strings(sizes)
puts
benchmark_read_ints(sizes)

Results:
old, create  18.50M ( 54.06ns) (± 1.82%)   160B/op   2.34× slower
new, create  43.32M ( 23.08ns) (± 0.88%)  64.0B/op        fastest

old, insert (type: String, size: 5)   3.60M (277.58ns) (± 7.01%)  480B/op   1.38× slower
new, insert (type: String, size: 5)   4.97M (201.28ns) (± 4.77%)  384B/op        fastest
old, insert (type: String, size: 10)   1.95M (513.74ns) (± 1.64%)  801B/op   1.33× slower
new, insert (type: String, size: 10)   2.59M (386.02ns) (± 3.01%)  832B/op        fastest
old, insert (type: String, size: 15)   1.27M (786.17ns) (± 3.56%)  1.09kB/op   1.55× slower
new, insert (type: String, size: 15)   1.98M (505.71ns) (± 4.48%)    832B/op        fastest
old, insert (type: String, size: 20) 922.03k (  1.08µs) (± 0.99%)  1.41kB/op   1.37× slower
new, insert (type: String, size: 20)   1.27M (788.95ns) (± 2.86%)  1.67kB/op        fastest
old, insert (type: String, size: 30) 639.38k (  1.56µs) (± 5.63%)  2.03kB/op   1.58× slower
new, insert (type: String, size: 30)   1.01M (992.38ns) (± 6.01%)  1.67kB/op        fastest
old, insert (type: String, size: 50) 355.49k (  2.81µs) (± 5.77%)  3.28kB/op   1.49× slower
new, insert (type: String, size: 50) 530.85k (  1.88µs) (± 3.95%)  3.81kB/op        fastest
old, insert (type: String, size: 100) 154.93k (  6.45µs) (± 4.04%)  7.06kB/op   1.81× slower
new, insert (type: String, size: 100) 280.47k (  3.57µs) (± 3.92%)  7.09kB/op        fastest
old, insert (type: String, size: 200)  71.71k ( 13.94µs) (± 3.27%)  13.3kB/op   1.91× slower
new, insert (type: String, size: 200) 136.71k (  7.31µs) (± 2.21%)  14.5kB/op        fastest
old, insert (type: String, size: 500)  23.46k ( 42.62µs) (± 3.53%)  36.1kB/op   2.46× slower
new, insert (type: String, size: 500)  57.82k ( 17.30µs) (± 4.78%)  28.4kB/op        fastest
old, insert (type: String, size: 1000)  12.78k ( 78.27µs) (± 1.72%)  67.4kB/op   2.20× slower
new, insert (type: String, size: 1000)  28.12k ( 35.56µs) (± 4.20%)  56.4kB/op        fastest
old, insert (type: String, size: 10000)   1.00k (997.78µs) (± 6.41%)  662kB/op   2.12× slower
new, insert (type: String, size: 10000)   2.13k (469.91µs) (± 5.39%)  896kB/op        fastest

old, insert (type: Int32, size: 5)   4.83M (207.19ns) (± 0.80%)  400B/op   1.78× slower
new, insert (type: Int32, size: 5)   8.60M (116.23ns) (± 1.76%)  240B/op        fastest
old, insert (type: Int32, size: 10)   2.78M (359.30ns) (± 0.91%)  640B/op   1.72× slower
new, insert (type: Int32, size: 10)   4.79M (208.73ns) (± 1.11%)  448B/op        fastest
old, insert (type: Int32, size: 15)   1.90M (525.28ns) (± 2.96%)  880B/op   1.75× slower
new, insert (type: Int32, size: 15)   3.33M (300.60ns) (± 5.98%)  448B/op        fastest
old, insert (type: Int32, size: 20)   1.30M (769.02ns) (± 6.52%)  1.09kB/op   1.54× slower
new, insert (type: Int32, size: 20)   2.00M (500.54ns) (± 4.76%)    976B/op        fastest
old, insert (type: Int32, size: 30) 972.78k (  1.03µs) (± 4.94%)  1.56kB/op   1.72× slower
new, insert (type: Int32, size: 30)   1.67M (597.26ns) (± 4.29%)    976B/op        fastest
old, insert (type: Int32, size: 50) 582.06k (  1.72µs) (± 5.23%)   2.5kB/op   1.50× slower
new, insert (type: Int32, size: 50) 872.34k (  1.15µs) (± 7.97%)  1.88kB/op        fastest
old, insert (type: Int32, size: 100) 234.48k (  4.26µs) (± 7.09%)   5.5kB/op   2.08× slower
new, insert (type: Int32, size: 100) 488.20k (  2.05µs) (± 6.79%)  4.14kB/op        fastest
old, insert (type: Int32, size: 200) 126.98k (  7.88µs) (± 4.74%)  10.2kB/op   1.84× slower
new, insert (type: Int32, size: 200) 233.78k (  4.28µs) (± 7.45%)  8.47kB/op        fastest
old, insert (type: Int32, size: 500)  40.86k ( 24.47µs) (± 4.14%)  28.3kB/op   2.55× slower
new, insert (type: Int32, size: 500) 104.34k (  9.58µs) (± 7.55%)  16.5kB/op        fastest
old, insert (type: Int32, size: 1000)  17.87k ( 55.97µs) (± 3.24%)  51.8kB/op   2.57× slower
new, insert (type: Int32, size: 1000)  45.98k ( 21.75µs) (± 7.72%)  32.5kB/op        fastest
old, insert (type: Int32, size: 10000)   1.52k (658.91µs) (± 3.88%)  506kB/op   2.16× slower
new, insert (type: Int32, size: 10000)   3.28k (305.04µs) (± 4.09%)  513kB/op        fastest

old, read (type: String, size: 5)  11.64M ( 85.90ns) (± 6.51%)  0.0B/op   2.25× slower
new, read (type: String, size: 5)  26.20M ( 38.16ns) (± 2.98%)  0.0B/op        fastest
old, read (type: String, size: 10)   5.69M (175.80ns) (± 4.64%)  0.0B/op   1.39× slower
new, read (type: String, size: 10)   7.93M (126.07ns) (± 6.97%)  0.0B/op        fastest
old, read (type: String, size: 15)   4.15M (240.69ns) (± 2.01%)  0.0B/op   1.16× slower
new, read (type: String, size: 15)   4.82M (207.47ns) (± 3.30%)  0.0B/op        fastest
old, read (type: String, size: 20)   2.93M (341.71ns) (± 1.47%)  0.0B/op   1.52× slower
new, read (type: String, size: 20)   4.46M (224.21ns) (± 1.25%)  0.0B/op        fastest
old, read (type: String, size: 30)   1.87M (534.24ns) (± 2.90%)  0.0B/op   1.45× slower
new, read (type: String, size: 30)   2.71M (369.30ns) (± 1.90%)  0.0B/op        fastest
old, read (type: String, size: 50) 992.08k (  1.01µs) (± 4.51%)  0.0B/op   1.71× slower
new, read (type: String, size: 50)   1.70M (589.77ns) (± 1.65%)  0.0B/op        fastest
old, read (type: String, size: 100) 596.10k (  1.68µs) (± 1.60%)  0.0B/op   1.40× slower
new, read (type: String, size: 100) 832.91k (  1.20µs) (± 1.82%)  0.0B/op        fastest
old, read (type: String, size: 200) 260.30k (  3.84µs) (± 1.63%)  0.0B/op   1.53× slower
new, read (type: String, size: 200) 398.84k (  2.51µs) (± 1.84%)  0.0B/op        fastest
old, read (type: String, size: 500) 111.03k (  9.01µs) (± 2.89%)  0.0B/op   1.40× slower
new, read (type: String, size: 500) 155.71k (  6.42µs) (± 3.33%)  0.0B/op        fastest
old, read (type: String, size: 1000)  41.59k ( 24.04µs) (± 3.07%)  0.0B/op   1.71× slower
new, read (type: String, size: 1000)  71.27k ( 14.03µs) (± 3.81%)  0.0B/op        fastest
old, read (type: String, size: 10000)   2.66k (376.32µs) (± 4.55%)  0.0B/op   2.46× slower
new, read (type: String, size: 10000)   6.54k (152.95µs) (± 4.03%)  0.0B/op        fastest

old, read (type: Int32, size: 5)  23.67M ( 42.24ns) (± 1.64%)  0.0B/op   2.30× slower
new, read (type: Int32, size: 5)  54.35M ( 18.40ns) (± 3.14%)  0.0B/op        fastest
old, read (type: Int32, size: 10)  11.66M ( 85.77ns) (± 4.63%)  0.0B/op   1.85× slower
new, read (type: Int32, size: 10)  21.58M ( 46.33ns) (± 3.20%)  0.0B/op        fastest
old, read (type: Int32, size: 15)   7.81M (128.00ns) (± 4.15%)  0.0B/op   1.49× slower
new, read (type: Int32, size: 15)  11.63M ( 86.00ns) (± 3.99%)  0.0B/op        fastest
old, read (type: Int32, size: 20)   5.80M (172.48ns) (± 5.34%)  0.0B/op   1.73× slower
new, read (type: Int32, size: 20)  10.02M ( 99.77ns) (± 4.54%)  0.0B/op        fastest
old, read (type: Int32, size: 30)   3.67M (272.51ns) (± 2.99%)  0.0B/op   1.84× slower
new, read (type: Int32, size: 30)   6.74M (148.30ns) (± 1.97%)  0.0B/op        fastest
old, read (type: Int32, size: 50)   2.05M (488.04ns) (± 5.57%)  0.0B/op   1.89× slower
new, read (type: Int32, size: 50)   3.87M (258.41ns) (± 4.24%)  0.0B/op        fastest
old, read (type: Int32, size: 100)   1.09M (921.59ns) (± 8.72%)  0.0B/op   1.70× slower
new, read (type: Int32, size: 100)   1.84M (542.80ns) (± 5.52%)  0.0B/op        fastest
old, read (type: Int32, size: 200) 535.83k (  1.87µs) (± 5.84%)  0.0B/op   1.66× slower
new, read (type: Int32, size: 200) 891.31k (  1.12µs) (± 5.49%)  0.0B/op        fastest
old, read (type: Int32, size: 500) 236.68k (  4.23µs) (± 3.61%)  0.0B/op   1.52× slower
new, read (type: Int32, size: 500) 360.85k (  2.77µs) (± 4.31%)  0.0B/op        fastest
old, read (type: Int32, size: 1000) 106.13k (  9.42µs) (± 4.88%)  0.0B/op   1.66× slower
new, read (type: Int32, size: 1000) 175.92k (  5.68µs) (± 4.08%)  0.0B/op        fastest
old, read (type: Int32, size: 10000)   4.60k (217.62µs) (± 2.94%)  0.0B/op   3.00× slower
new, read (type: Int32, size: 10000)  13.80k ( 72.47µs) (± 1.67%)  0.0B/op        fastest

As you can see the new implementation is always faster than the old one. Sometimes more memory is used, sometimes less.
I also ran some @kostya benchmarks that used Hash in their implementation. Here are the results:
Havlak:
old: 12.49s, 375.1Mb
new: 7.58s, 215.7Mb

Havlak seems to be a benchmark measuring how well a language performs in general algorithmic tasks... the new results look good! 😊
Brainfuck:
old: 5.20s, 1.8Mb
new: 4.22s, 1.8Mb

JSON (when using JSON.parse):
old: 2.20s, 1137.0Mb
new: 2.07s, 961.3Mb

Knucleotide:
old: 1.63s, 26.5Mb
new: 1.01s, 32.4Mb

Then some more benchmarks...
There's HTTP::Request#from_io which I recently optimized:
old: from_io 498.47k (  2.01µs) (± 0.89%)  816B/op  fastest
new: from_io 549.22k (  1.82µs) (± 1.89%)  720B/op  fastest

(using wrk against the sample http server increases the requests/sec from 118355.73 to about 122000)
Also for curiosity I compared creating a Hash with 1_000_000 elements and seeing how it compares to Ruby and the old Hash.

Code for creating a Hash with 1_000_000 elements in Ruby and Crystal
size = 1_000_000

h = {0 => 0}

time = Time.now
size.times do |i|
  h[i] = i
end
puts "Insert: #{Time.now - time}"

time = Time.now
size.times do |i|
  h[i]
end
puts "Read: #{Time.now - time}"

Results:
Ruby 2.7-dev:
  Insert: 0.151813
  Read:   0.13749
Crystal old:
  Insert: 0.238662
  Read:   0.129462
Crystal new:
  Insert: 0.070804
  Read:   0.041008

Ruby was faster than Crystal! Ruby is simply amazing ❤. But now Crystal is even faster!
The compiler uses Hash all over the place!
So compile times now should go down! Right? ... Right?
Well, unfortunately no. I think the main reason is that the times are bound by the number of method instances, not by the performance of the many hashes used.
Memory consumption did seem to go down a bit.
When will I have this?
~We could push this to 0.31.0. Or... we could have it in 0.30.0 if we delay the release a bit more (note: I don't manage releases, but if the community doesn't mind waiting a bit to get a huge performance boost in their apps then I think we could relax the release date).~
In 0.31.0.
Final thoughts

Thank you Ruby! ❤ ❤ ❤
Thank you Vladimir Makarov! ❤ ❤ ❤
Algorithms are cool!
There's an optimization in the original Ruby algorithm which uses bitmasks instead of remainder or % to fit a number inside a range... I did that as almost the last thing because I didn't believe it would improve performance a lot... and it doubled the performance! 😮
Feel free to target this PR and benchmark your code and post any interesting speedups here!
I hope CI 32 bits passes! 😊

This gives a nice speedup on our internal app for log processing!
@bcardiff will this be considered for 0.30?
Time in seconds

test
0.29
0.30dev
speedup

forti
7.353
6.722
9.4%

forti_kv
5.462
4.867
12,2%

dns_grt
4.714
4.466
5,6%

Details
~/Work/crystal-1/bin/crystal version
Using compiled compiler at `.build/crystal'
Crystal 0.30.0-dev [ec2a26f9a] (2019-07-31)

LLVM: 8.0.0
Default target: x86_64-apple-macosx

~/Work/crystal-1/bin/crystal build --release -o bin/log_clean src/log_clean.cr
Using compiled compiler at `.build/crystal'
bin/log_clean -c test/forti/forti.yml >    /dev/null  6.83s user 1.17s system 119% cpu 6.722 total
bin/log_clean -c test/forti/forti_kv.yml > /dev/null  5.11s user 2.02s system 146% cpu 4.867 total
bin/log_clean -c config/dns_grt.yml >      /dev/null  4.50s user 0.74s system 117% cpu 4.466 total

crystal version
Crystal 0.29.0 (2019-06-06)

LLVM: 6.0.1
Default target: x86_64-apple-macosx

bin/log_clean -c test/forti/forti.yml >    /dev/null  7.49s user 1.26s system 119% cpu 7.353 total
bin/log_clean -c test/forti/forti_kv.yml > /dev/null  5.80s user 2.08s system 144% cpu 5.462 total
bin/log_clean -c config/dns_grt.yml >      /dev/null  4.78s user 0.79s system 118% cpu 4.714 total

I wish github had properly threaded review comments. This is a comment to my own review: I note that you get lets of repeated allocation warnings and that one CI fail due to out of memory.

The OOM failure on test_linux32 is pretty regular and unrelated to this PR.

@straight-shoota Are all the warnings about large allocations also old?

Rather clean code! I liked to read it.

@funny-falcon

Rather clean code! I liked to read it.

Thank you! 😊 And thank you for the thorough review, you found many good things to fix.

@yxhuvud

This is a comment to my own review comment: I note that you get lets of repeated allocation warnings and that one CI fail due to out of memory.

Yes, but I think the errors were there before. I think Hash might require a bit more memory because now it always doubles its size when a resize is needed. Also we keep adding tests to the spec suite and right now the compiler specs leak memory. I'll open another issue to track this problem. The solution for now is to just run specs on CI in smaller chunks instead of in two big chunks. This is a problem that only affects CI, regular machines usually have a lot of memory and don't need to compile the entire Crystal's spec suite.

Hash.shift is rather useful for:

simple LRU implementation
keyed FIFO queue
Given it is quite easy to add @first , it is better to add it than to not.

31 июля 2019 г., в 21:35, Ary Borenszweig<EMAIL_ADDRESS>написал(а):
@asterite commented on this pull request.
In src/hash.cr:

 end

end
private module BaseIterator

def initialize(@hash, @current)

def initialize(@hash)

 @index = 0

Ah, yes, I see what you mean. I think we should either remove Hash#shift or implement your suggestion. We copied Hash#shift from Ruby but I don't know what's a real use for that it.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

A quick benchmark from my simulation framework reveals a really nice speedup of ~1.37x!
Hash was already identified as a bottleneck in my case after profiling.
./devstone 10000 3 results:

Crystal version
Wall clock time (s)

Crystal 0.30.0-dev [008647480] (2019-07-31) LLVM: 8.0.0
7.797805118

Crystal 0.29.0 (2019-06-06) LLVM: 6.0.1
5.703113211

Given it is quite easy to add @FIRST , it is better to add it than to not.

Yes, but it adds 4 bytes for all hashes even if you never use shift. Same reason why we didn't do that for Array. If you need shift like that you can use Deque, which is in the standard library and it's more efficient (at the cost of a bit more memory).
Said another way, we can consider this as a separate issue.

@asterite: I think the warnings isn't due to requiring more memory but because libgc doesn't like big allocations. That is, it is perfectly happy to allocate a GB if you do it in 4KB chunks, but if you do it all at once you get a warning. So doing fewer tests wouldn't really help with the warnings as you would still get one once the amount of entries in a hash get big enough.

@yxhuvud Makes sense. I think I added a couple of tests that exercise some big-ish hashes.

@funny-falcon On second thought it seems we can put the 4 bytes of @first in a place where it doesn't increase the size of Hash, at least in 64 bits where everything is aligned to 8 bytes. And the same is true for Array! So I think I'll add those (I'll add @first for Hash in this PR, and for Array in a separate PR). It'll be cool to have O(1) shift. I'm thinking for Array I'll just increment @first but also clear the value behind that pointer. Though I'll probably wait until parallelism or coordinate with @bcardiff because I know he's been doing some changes to Array to make it a bit more safe in parallel environments.

Now with optimized shift, thanks to @funny-falcon 's suggestion.
Benchmark. Code:
old_hash = OldHash(Int32, Int32).new
new_hash = Hash(Int32, Int32).new

p instance_sizeof(typeof(old_hash))
p instance_sizeof(typeof(new_hash))

100_000.times do |i|
  old_hash[i] = i
  new_hash[i] = i
end

time = Time.now
while old_hash.shift?
end
puts "Old: #{Time.now - time}"

time = Time.now
while new_hash.shift?
end
puts "New: #{Time.now - time}"

Results:
56
56
Old: 00:00:02.272458000
New: 00:00:00.000244000

(meaning the bytesize of a Hash didn't change with this optimization)

This is absolutely amazing! I think robin hood hashing would be a neat future optimization

It is better to maintain @first in delete_entry_and_update_counts. It is single point of all deletions.
And it is better not simply @first+=1 but:
delete_entry(index)
@size-=1
@deleted_count+=1
return if index != @first
if @size == 0
  @first = @deleted_count
else
  @first += 1
  @first += 1 while<EMAIL_ADDRESS>end

And then first_entry could be optimized as well:
@size == 0 ? nil : @entries[@first]

And shift
if first_entry
  delete_entry_and_update_counts(@first)
  {first_entry.key, first_entry.value}
else

You may also consider the Robin Hound Hashing (site may not be responding) optimization. Basically:

when you probe for a position to insert a new element, if the probe length for the existing element is less than the current probe length for the element being inserted, swap the two elements and keep going.

Robin Hood Hashing should be your default Hash Table implementation

Plus the Learn Hash Table the Hard Way series that talks about probes (linear, quadratic, double hash) and Robin Hound Hashing:

Part 1: Probe Distributions
Part 2: Probe Distributions With Deletions
Part 3: Probe Distributions and Run Time Performance

@ysbaddaden Robin Hood hashing is useful for high load factor (close to 0.8-0.9). Proposed hash structure never exceeded load factor 0.5 by design (because indexes array always twice as large than entries array). Even already encoded linear search performs reasonably well with such low load factor. I believe, quadratic probing could be a bit better, but I'm not really sure.  I could be mistaken.

@funny-falcon the benefit is that we can increase the load factor, reducing the memory footprint.

@funny-falcon the benefit is that we can increase the load factor, reducing the memory footprint.

@RX14 increasing load factor along will make this code much more complex. Current it's beauty is from simplicity of sizes: always power of two, indices are twice larger than entries.
In my attempts for this issue I always tried to make load factor greater, and to untangle indices and entries size. But my code were much more complex.

increasing load factor along will make this code much more complex

I realise that, but Hash performance is extremely important for the speed of the language. The performance/complexity tradeoff here is complicated. That's why I suggest we discuss this later. The performance benefits of robin hood hashing probably need to be tried in practice to get an idea of whether it's wanted. The gains in this PR are very substantial, and should not be delayed for such an investigation.

I realise that, but Hash performance is extremely important for the speed of the language.

If hash performance is extremely important, then you don't want to increase load factor.
Rubin Hood has no performance benefits. If you read paper mentioned by @ysbaddaden carefully, then you noticed:

Actually, the mean of quadratic probing is slightly better than quadratic robin hood hashing (1.3 vs 1.45).

Robin Hood hashing doesn't give better "average" performance. It only gives guarantee of bounded "worst" performance. In other words, application will not be magically faster in average with Robin Hood hashing. But there will be more guarantee about time of each separate operation.
Rust used to have load factor 0.9 with Robin Hood hashing, but it gives measurable performance overhead. https://github.com/rust-lang/rust/issues/38003
Now they moved to other hash implementation that combines bulk SSE check of a bucket of entries with quadratic probing of buckets.
Yes, it has impressive 0.875 load factor.
But it is not easily mapped to "ordered" hash. Bits of hash values have to be embedded into index table to use same trick.
Note that main benefit from Robin Hood hashing is a CPU cache locality. But given we have to journey from index table to entries table to check value, and order of elements in index table does not correlate to order in entries table, we already have pure cache locality. Embedding bits of hash value into index table will help, but will increase memory consumption.
Other trick to shortage average collision chain in quadratic probing is to use "collision" bit: on insertion set "collision" on all collisioned index entries, and during lookup iteration should stop if current element has no "collision" bit set. It trades some CPU for shorter collision chain. I believe, it makes "quadratic probing" much closer to "robin hood" hashing in term of "collision chain length distribution".
Any way, all my words should be taken with grain of salt. I often make false assumtions.

@funny-falcon thank you! The above articles on RH hashing weren't entirely clear on this.
Memory overhead of Hash is important too, and I'd be interested in a graph of the allocation size of @indexes and @entries in bytes for varying Hash(Reference, Reference) sizes. I suspect that the load factor in @indexes is fairly insignificant compared to the size of @entries.

@funny-falcon Thank you! I implemented all your suggestions. Let me know if I got something wrong 😊
Although I think that many things can be still optimized (representation of things, the probing, the load factor, etc.) I think this is already a good improvement and we can continue refining things after this PR.

Could someone explain to me with some ascii art of graphics how Robin Hood hashing works?

When a collision occur, compare the two items’ probing count, the one with larger probing number stays and the other continue to probe. Repeat until the probing item finds an empty spot.

I can't understand this. Or... I can understand what it says but not what it means.
Let's suppose we start with empty buckets:
[_, _, _, _, _, _]

Now we want to insert 10, which hashes to bucket 1:
[_, 10, _, _, _, _]

Say we want to insert 20, which hashes to bucket 1 too... What do we do? The probe count for 10 is 0 (I guess?) and for 20 we don't know yet but it's definitely higher than 0 because it's occupied. So we swap them...?
[_, 20, 10, _, _, _]

Next comes 30, again with bucket 1. So 20 has probe count 0, 10 has probe count 1, and 30 will have probe count 2... so we span everything again?
[_, 30, 20, 10, _, _]

I can't see the benefit of this, but I'm sure there's something I'm not understanding.

In Robin Hood hashing, we compare each element
[_, _, _, _, _, _]
Now we want to insert 10, which hashes to bucket 1:
[_, 10, _, _, _, _]
  ,  0,  ,  ,  ,  ```

In Robin hood hashing, we compare old element probe count with current probe count of our element.
So in your example nothing will be shifted - hash table will be
[_, 10, 20, 30, _, _]
Now if we insert element 21 that hashes to 2, we will compare it to 20 (0 vs 1) and 30(1 vs 2) and still insert at 4:
[_, 10, 20, 30, 21, _]
But now if we insert 40 that hashes to 1:
it will skip 10, 20, 30 as they have same probe count, but will steal a place from 21 as it will have probe count 3 and 21 will have just 1:
[_, 10, 20, 30, 40, _]
and then we continue to search place for 21, finding it at next position:
[_, 10, 20, 30, 40, 21]

@konovod Thanks, I understand now!

Thank you everyone for the suggestions!
If you find optimizations please send PRs and benchmarks.

Some optimizations ideas:

Robin Hood hashing
Double hashing
Quadratic probing
Reduce the allocated size of @entries (it could be possible to allocate less than @indices_size / 2)
Other ideas...?

That's good points @asterite, better to create issue to track them. It can even be a RFC.

No. If someone finds an optimization please send a PR with benchmark. I don't plan to change anything else otherwise, and there's no need to track anything.

Gold words.
вс, 4 авг. 2019 г., 23:03 Ary Borenszweig<EMAIL_ADDRESS>
No. If someone finds an optimization please send a PR with benchmark. I
don't plan to change anything else otherwise, and there's no need to track
anything.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/crystal-lang/crystal/pull/8017?email_source=notifications&email_token=AAB44U37QCJJPFL6MJF4UR3QC4Y2HA5CNFSM4IIA36K2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3QIU3I#issuecomment-518031981,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAB44U25ZPY57CSTTDQIIFTQC4Y2HANCNFSM4IIA36KQ
.

I think it would be better to have proper RFCs about the design decisions, instead of PRs, just a thought.
Of course that's up to the core members to organize their project as they wish, and what they consider the best.


fix(ci): Fix variables references

More: https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs

:tada: This PR is included in version 1.1.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Rename diff symbol to partial

Description
@dccsillag, @Leedehai, and @Enivex expressed their desire/agreement on renaming diff to partial. It looks like latex also has a built-in partial instead of diff (just for the comparison). You can find the full discussion here: https://discord.com/channels/1054443721975922748/1196918099069390850
I will add that partial is the most agreed upon, but this issue is for further discussion of potentially other names for this symbol.
There was also a talk about dif and Dif. Someone wanted to rename dif to diff (it also defined in the wrong place in Rust file). This also perhaps should be discussed here, as the names are very similar.
Use Case
Better readability. Good examples are gradient and integral instead of grad and int.

The Unicode char is called "Partial Differential", so it can start from both "p" and "d", and "par" also will autocomplete the "partial".
https://symbl.cc/en/2202/

I would add that

The name is more semantic, readable, and an actual word
It's called \partial in LaTeX
Having both dif and diff separated by a single letter makes it look more like a typo than anything else. They're hard to distinguish at a glance.

(I would also like dif to be renamed diff, but that's a separate issue.)

The only thing I'd like to add here is that, much like how the name of the gradient symbol is nabla, the name of the partial derivative symbols is "del" (https://pt.wikipedia.org/wiki/Del). Just makes me wonder if we should do something similar to what we've done with gradient/nabla, by having the partial derivative symbol accessible both as partial and as del.
It's also worth mentioning that the same symbol can appear in other contexts, most notably perhaps in denoting the contour of a set.
Though I must say, if I had to choose one name then I'd certainly choose partial.

Please don't name it "del".. because the nabla symbol is also called "del".

Please don't name it "del".. because the nabla symbol is also called "del".

Totally agree with you.

Then partial it is :smile:

https://en.wikipedia.org/wiki/∂
The symbol is variously referred to as "partial", "curly d", "funky d", "rounded d", "curved d", "dabba", "number 6 mirrored",[6] or "Jacobi's delta",[5] or as "del"[7] (but this name is also used for the "nabla" symbol ∇). It may also be pronounced simply "dee",[8] "partial dee",[9][10] "doh",[11][12] or "die".[13]

The symbol is variously referred to as "partial", "curly d", "funky d", "rounded d", "curved d", "dabba", "number 6 mirrored",[6] or "Jacobi's delta",[5] or as "del"[7] (but this name is also used for the "nabla" symbol ∇). It may also be pronounced simply "dee",[8] "partial dee",[9][10] "doh",[11][12] or "die".[13]

Argh, how it pains me that "number 6 mirrorred" is listed as a more common name than "del"!

Argh, how it pains me that "number 6 mirrorred" is listed as a more common name than "del"!

I'm not sure Wikipedia is a great authority here :p


replace password with unix socket authentication for root user

this pull request implements two things:

load differences in installation through distribution dependent variables
replace password authentication for the database root user with authentication via unix socket

authentication via unix socket has been set as the default for the MariaDB user  'root'@'localhost' since MariaDB 10.4.3
Since then there is no need anymore to set a database root user password and it is actually better to not do so because it can have bad side affects, e.g. in Debian/Ubuntu some maintainer scripts rely on this function and also automysqlbackup. For the latter you would have to store again a password in a cleartext file.
For  the creation of the opencast user and database it is sufficient to use become for root permissions.
database_root_password is still left in as optional because some users may provision to an already installed MariaDB which has a database root user password already set.

Sorry, we seem to have overlooked this pull request. I've left a review and hope we can get it in soon.
Can you rebase this onto the current main branch, make sure that the tests pass and take a look at the comments?

Thank you for the review! I have incorporated your feedback now


Feat: add category for workflow step definition

Signed-off-by: FogDong<EMAIL_ADDRESS>Description of your changes

Add category for workflow step definition:

Application Delivery

apply-component
collect-service-endpoints
depends-on-app
deploy-cloud-resource
deploy
export-data
export-service
share-cloud-resource

Resource Management

apply-deployment
apply-object
clean-jobs
export2config
export2secret
read-object

Terraform

apply-terraform-config
apply-terraform-provider
generate-jdbc-connection

CI Integration

build-push-image

Config Management

create-config
delete-config
list-config
read-config

External Integration

notification
request
webhook

Process Control

print-message-in-status
step-group
suspend

Scripts & Commands

vela-cli

I have:

[x] Read and followed KubeVela's contribution process.
[ ] Related Docs updated properly. In a new feature or configuration option, an update to the documentation is necessary.
[ ] Run make reviewable to ensure this PR is ready for review.
[ ] Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

Special notes for your reviewer

cc @barnettZQG

shall we need to backport 1.7?

VelaUX will use the category for classification in the dashboard, we can move this to 1.8.


Bug/bridge 1754

Add IRB text back to consent htmls to fix issue with broken irb.html reference

Why is this broken? Should go through and replace the <import /> with the text in that document.


Full FE chapter and style changes

Thought I was missing a package or two in the DESCRIPTION file but it all built without a problem.
This is a first full draft. It needs some cleaning up, I'm sure.
It also includes the style changes we talked about on Slack. I think the actual CSS file could be cleaned up a bit. Mostly with renaming the variables. Like, it would be good to have a "main-color" and "main-accent" (which would be the dark green and the lighter green, probs). But I think it's good enough for now and we can do that later in another PR.

@brendanhcullen just tagging you here as an FYI that I'm going to go ahead and merge this in. You are free to do whatever style edits you want from here. Let me know if you want to chat about any of them.

Thanks @datalorax! Excited to read through it. I'll do my review through a separate PR. Maybe at our next team meeting we can narrow down a good Github workflow for reviews/feedback/code commenting. I'm new to that sort of thing but would love to learn best practices.


enhance readability of markdown contents

Based on the first results in #44 I slighty enhanced the readability and coverage of the regex patterns.
Please see https://github.com/AdrianDsg/poetry-plugin-up/pull/4 for the results before and after the changes.

Thank you for contributing! No you are good to go to merge it :)


fails to initialise if in iframe on Chrome 49 Android 6.0.1

https://github.com/borismus/webvr-polyfill/blob/master/src/viewer-selector.js#L33
if localStorage is unavailable then this.selectedKey = localStorage ... fails.
for our build I've added this.selectedKey = DEFAULT_VIEWER to the catch block until a better fix.

Are you in Incognito Mode?

i don't think so, @cvan .
49 on 6.0.1 is the only browser I tested to deny access to localStorage there.

Do you have "Block third-party cookies and site data" or something like that enabled?

Blocking third-party cookies would definitely do this. Or possibly an extension, maybe something like Privacy Badger? I've had third-party cookies blocked on my browser for years, and it's amazing how many sites forget to check for this. But it should be handled, since it's a well-defined, predictable, standard behavior built into the browsers. Same deal here.
Maybe it would have been better to have localStorage just return empty values rather than throw an error, but that ship has sailed.

Ping @gmarinov. More information needed. Are you using some kind of 3rd party blocker?

no

Can't repro without more info.

Whitelisting storageapis.google.com via "Settings -> Advanced -> Content Settings -> Cookies -> Manage Exceptions" fixed this issue for me on desktop Chrome. I was running into the same " Failed to read the 'localStorage' " error in the console.

Seeing as my answers above maybe did not make this clear enough.

no blockers
stock android and chrome from play store
no incognito mode
javascript is enabled
safe browsing enabled
cookies allowed except third-party cookies
do not track is on

Whitelisting storageapis.google.com by going to "Settings -> Advanced -> Content Settings -> Cookies -> Manage Exceptions", is something I cannot ask of the user. The code itself has no default fallback.

Is it possible to fix webvr-polyfill to allow incognito mode?


provide access to eXist fundocs app

In the eXist dashboard there is an app "XQuery Function Documentation" preinstalled. It allows for generating a documentation of all XQuery functions used based on the function annotations. IMHO it would be good to allow a user of DraCor understand what is going on under the hood of the API, e.g. how data is extracted from the TEI with the dutil: functions. A first step would be to generate the documentation upon deployment and make it (or selected modules) accessible like http://localhost:8080/exist/apps/fundocs/view.html?uri=http://dracor.org/ns/exist/util&location=/db/apps/dracor/modules/util.xqm
A second step would involve adding additional documentation to the function annotations, e.g. explaining, which elements are evaluated, where data is extracted and maybe reference the ODD from there.

I suggest not providing the whole fundocs app, but instead use its rendering functions to create the HTML documentation that can be embedded to the DraCor site.
why this? the fundocs app incorporates (off-scale outdated) libraries know for vulnerabilities and will be very hard to update.:
"bootstrap": "^3.4.1",
"jquery": "^1.12.4",
"marked": "^2.1.1",
"prismjs": "^1.23.0"

(from their package.json)
to publish this endpoint is like moving server infra to windows 95.
with great respect to the work we might take just what we need (citing the source, of course) and use the rendering feature as described above.
in addition providing the whole app will expose all other xqdocs and thus might rate down DraCor at search engines because of providing redundant copies of content.

OK, thanks for that input. I was just thinking of the easiest way to get the documentation, because of the work, that would already be involved in updating and enhancing the function annotations of the relevant functions. But maybe your suggestion would be a better approach. So where to start looking?
https://github.com/eXist-db/function-documentation/blob/master/src/main/xar-resources/modules/scan.xql
This  obviously seems to provide the browse functionality?
https://github.com/eXist-db/function-documentation/blob/master/src/main/xar-resources/modules/app.xql#L77-L81
This seems to render a single module:
https://github.com/eXist-db/function-documentation/blob/master/src/main/xar-resources/modules/app.xql#L114-L187
This a single function:
https://github.com/eXist-db/function-documentation/blob/master/src/main/xar-resources/modules/app.xql#L189-L276
It seems even to support markdown in the annotations, which would help in adding links to the ODD...

and this is the specification of the function annotations, right?
http://xqdoc.org/index.html

maybe we can omit the browse function.
in scan.xql a node tree according to xqdoc is created that can be passed for rendering either a module or a single function. this will result in HTML to be included in the website.

Before we start documenting XQuery functions that are not part of our public facing API I would suggest augmenting the documentation of our main contract, that is the DraCor REST API. The documentation of the functions that are not in the api.xqm module as it is right now is not very elucidating without also looking at the code, especially the XPath constructs therein. We would have to put significant effort into extending the XQDocs to turn them into a useful resource for understanding DraCor. I would prefer to put that effort into comprehensively describing the JSON (and CSV) output of the REST API. From there we could even link into the code on GitHub and achieve a very rich documentation that would help both consumers and developers of the API.
Also, even if the XQDocs were already useful enough to publish them, we would have to come up with some way to extract them, a workflow to integrate them into dracor-frontend, and some styling to fit them into the DraCor design. That's quite some work. Why don't we instead pick up things where we left them with openapi4restx? (@mathias-goebel has there been any development?) We have an open issue #59, there is already the /openapi endpoint. If we extend openapi4restxq so that it can extract all the information we need from the RESTXQ annotations in api.xqm we could continue to improve those annotations and have an always up-to-date automatically generated API documentation. We could stop maintaining api.yaml manually and switch to the /openapi endpoint for https://dracor.org/doc/api.
I'm not against improving the XQuery function documentation. But for now I think it is accessible enough for those who can benefit from it, i.e. developers of dracor-api who can always use the fundocs app in a local dracor instance. To improve the overall public facing documentation DraCor API documentation, I would argue there are better more efficient ways to do that.

The documentation of the functions that are not in the api.xqm module as it is right now is not very elucidating without also looking at the code, especially the XPath constructs therein.

agree. For the CLS INFRA report I am looking at the code and, true, the xPaths are key.
I think, we should communicate with the documentation, that there is 1) something encoded in the TEI, 2) this gets extracted by some means (XQuery) and 3) ends up in the API output. At the moment, I think, this is not transparent and just pointing to the code on github will not be enough, because, to be honest, XQuery isn't the most common of programming languages.
The best thing I could come up with for now, is a spreadsheet where I can, at least, collect the respective XQuery functions, the XPath, that extracts information and record the field in the (JSON) response:
https://docs.google.com/spreadsheets/d/1fgvexLfoJF-elYunQ_R0rvYNXZRKhgSZPrI6dOu9ekA/edit?usp=sharing
The idea is to come up with a list of all features about the central entities, i. e. corpus and play, probably character and relate these features to the API endpoints.
@cmil How would you augment the information in the spreadsheet with the OpenAPI Specification? Put it into the description field? And, yes, the schemata of the response objects. When using flask and apispec I could use marshmallow schemas to describe the response objects that are automatically included in the Open API Spec, but with XQuery...
So, you mean, they need to be maintained manually, right? Because where would you add something like schema to the XQuery function annotation?
In the very end I would also like to see a way to integrate OpenAPI and the ODD, but don't have a clue how to achieve that.
I think, the aim should be to have one source of documentation and generated the rest out of it. What I see at the moment, the closest thing to what we get out in the end on the side of the API is the XQuery code, so I thought, maybe start documenting there? And I thought, implementation-whise it would be relatively easy to achieve (because the thing to publish ist, fundocs, is already there). I understand that the real work is adding the annotations to the XQuery functions, but that would not involve any more development.


Support aliases to npm dependencies (used in Kotlin/JS) in versionCatalogs

Expected Behavior

Something like this should be supported:
val webrtcAdapter = version("webrtcAdapter", "^7.7.0")
alias("webrtcAdapter").to(npm("webrtc-adapter")).versionRef(webrtcAdapter)

Current Behavior

AliasBuilder.to supports only strings as arguments, so non-standard dependencies (including npm) aren't supported.
Context

For now, I have to continue specifying npm dependencies in build.gradle.kts:
val webrtcAdapterVersion: String by project

dependencies {
    implementation(npm("webrtc-adapter", webrtcAdapterVersion))
}

It would be more consistent and compact if I could move such declarations to settings.gradle.kts, leaving only implementation(libs.webrtc.adapter) in build.gradle.kts.

Thank you for the suggestion. This request makes sense. We have to think more about how exactly to approach the issue because currently the version catalog is tied to the GAV dependency notation. Most likely consumers will still have to indicate npm dependencies explicitly like implementation(npm(libs.webrtc.adapter)) and some sort of support will be required from the contributing plugin.

Still actual


Workflow Management extension name is changed to Workflow Module

Tutorials: https://developers.sap.com/tutorials/cp-starter-ibpm-employeeonboarding-2-configure.html
Hi Colleagues,
In the above mentioned tutorial, at Step 5: Download and import workflow sample application, and the sub-step 3, Click on the Dev Space to go into the workspace and from File menu choose Open Workspace.
Here, it is mentioned as " In the New Dev Space page, enter any dev space name, select SAP Fiori kind and Workflow Management extension. ". But "Workflow Management" is now changed to "Workflow Module". So please update the document with corresponding screenshot.
I have attached the screenshot for your reference.

Labels to be added:
Scenario label: SCPTest-eboard
Sprint Label: SCPTest-2301A
Landscape Label: trail2(Trial US East (VA) (AWS) ) & Trial3 (AP Region)
Priority: Medium
Many thanks & regards,
Baskar

@ArchanaShukla can you please respond to this user's feedback?


CDKN2A multiple transcripts issue

For this example:
https://www.genomenexus.org/variant/9:g.21994323C>T
The annotation_summary output picks to display CDKN2B-AS1.
The issue is that for CDKN2A there two canonical transcripts, but GN only supports one (ENST00000304494): https://github.com/genome-nexus/genome-nexus/issues/453#issue-718390410. It therefore fails to show the missense mutation for ENST00000361570.

We have a fix to manually add ENST00000361570 when pulling all canonical transcript ids (https://github.com/genome-nexus/genome-nexus/pull/763), for now only CDKN2A has two canonical transcripts (confirmed by checking mskcc isoform override table).
But this is not the ideal way to fix it, we should discuss how to architect this properly.


Feature request: Bluetooth device tracker

Anyone see any value in having a bluetooth device tracker? This would need to be tested to make sure it didn't break anything so I wouldn't implement this without some volunteers to test it.

Biggest downside I could see is that it requires sending a command to each echo device through Amazon so it might be too spammy to Amazon's API.

Willing to test. It could be limited to an action based POST, rather than on an interval (say a door opens and I'm currently home then wait a few minutes then scan etc.

If an Alexa bluetooth scan would return enough data to use it with monitor, it could be really awesome for presence detection. Monitor has previously been used with Google home until Google changed their API.

I'd love to test something like this if it would allow in house device tracking if we have Alexa devices in different rooms.

While I would hope that they wouldn't complain (and might not even notice), I'd hate for the device tracker feature to create so much traffic that Amazon changes some rules about things - or worse, deletes some features.

I would absolutely love to implement this, it’s the missing piece to true automations

I use room-assistant on a Pi Zero W for room-to-room presence. If AMP can do something like that, then I could remove the Zero W's in rooms that have Echo devices.  That would be nice

Another vote for this. I think a lot of us have more then one Alexa in our homes would be amazing if we can also use them just like Pi Zero W - Monitor scripts you can then have fairly accurate room presence detection. Not sure how that's possible but would be amazing :)

Can feature requests be left open?


Update analyzers, and add public API analyzer

This adds Roslyn.Diagnostics.Analyzers, which includes the public API analyzers and addresses all new warnings.
Also update Microsoft.CodeAnalysis.FxCopAnalyzers to 3.3.0 and address new warnings (in commits after 59136c65164895354d174d37944ee318da5e01c2).
One diagnostic per commit to make it easier to review/amend.

@drewnoakes This is awesome, thank you for doing the work here!
One quick question here.  What's the difference between "PublicAPI.Shipped.txt" and "PublicAPI.Unshipped.txt"?  I read what sparse documentation I could find, but I didn't really understand the split between these two files.  In my mind, the entire API surface at this point is "shipped" but maybe these mean something else?

Why use Roslyn.Diagnostics.Analyzers here?

This package contains rules that are very specific to the .NET Compiler Platform ("Roslyn") project, i.e. dotnet/roslyn repo. This analyzer package is not intended for general consumption outside the Roslyn repo. More info about rules in this package

What's the difference between "PublicAPI.Shipped.txt" and "PublicAPI.Unshipped.txt"?

As I understand, "unshipped" APIs refer to those which are present in the code base but haven't been released in NuGet packages.

As I understand, "unshipped" APIs refer to those which are present in the code base but haven't been released in NuGet packages.

I guess I don't understand the mechanism.  Do I manually move APIs from unshipped to shipped by cutting/pasting the text files?

What's the difference between "PublicAPI.Shipped.txt" and "PublicAPI.Unshipped.txt"?

The analyzer treats them both the same way, so whether an API is in one or the other is only important if you want it to be.
The distinction probably makes sense when you have long periods between shipping your API, in which time the unshipped file grows. When you ship you can document the API changes, then push everything into shipped and start fresh. It's my opinion that for many use cases this distinction isn't helpful, so I ignore it. The analyzer comes with a codefix which allows automatically updating the unshipped file, which is very convenient as editing the files manually is hard to get right. At some point I'd like to contribute a change to the analyzer that allows using only one file (eg PublicAPI.txt).

Why use Roslyn.Diagnostics.Analyzers here?

This package contains rules that are very specific to the .NET Compiler Platform ("Roslyn") project, i.e. dotnet/roslyn repo. This analyzer package is not intended for general consumption outside the Roslyn repo. More info about rules in this package

@NextTurn these analyzers are generally useful and used in many places outside of Roslyn.
Roslyn.Diagnostics.Analyzers brings in several other packages, and it's easier to just include Roslyn.Diagnostics.Analyzers directly so that the versions of those other packages are compatible.
Changes in response to Roslyn.Diagnostics.Analyzers:

RS0038 82f71031b6ea1f9dd593322310003d465ef9c89a
RS0042 e697de74c57ca0b9ae93cf931541cd61e2147c5a
RS0100 d9db26bfbbac020a64b9299ce67bfa4afcdfc7d7
RS0101 dadabd97e84722fe2506d540c9caebdbe025f6da

If there are any diagnostics that don't fit the project, it's easy to exclude them.

PublicApiGenerator also may be helpful.

@sungam3r thanks for the pointer. That tool looks identical in functionality to the analyzer. Am I missing something?

They solve approximately the same problem, but in different ways.


[META] PDF considersations

Before a PDF download feature goes into production, consider:

#76
What should be on the front page? Something based on the rights? (Mudd sample: http://findingaids.princeton.edu/AC044/c0095.pdf)
Should this always be available for open content? If not, how do we decide when? Case by case?
Should the PDF be degraded in some way (grayscale? watermarked? extra compression?

See also #25


fix: Table Widget property to specify if prev or next buttons were pressed

… (#10522)
Fixes #10522
We have added two new boolean meta properties, previousPageVisited and nextPageVisited, that are set whenever respective pagination button is clicked. This property is updated for server side pagination mode as well as non server side pagination mode.
By default, both properties are set to false.
When the user clicks on previous page button or tries to enter a page number which is less than current page number, previousPageVisited is set to true and nextPageVisited is set to false.
The users can use these property as follows in their code :
myFun1: () => {
		if (Table1.nextPageVisited == true){
			NextQuery.run()
		}
		else if (Table1.prevPageVisited == true){
			PrevQuery.run()
		}
		else {
			BaseQuery.run()
		}
	}

Checklist:
Dev activity

[x] My code follows the style guidelines of this project
[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] I have added tests that prove my fix is effective or that my feature works
[x] New and existing unit tests pass locally with my changes
[ ] PR is being merged under a feature flag

QA activity:

[ ] Test plan has been approved by relevant developers
[ ] Test plan has been peer reviewed by QA
[ ] Cypress test cases have been added and approved by either SDET or manual QA
[ ] Organized project review call with relevant stakeholders after Round 1/2 of QA
[ ] Added Test Plan Approved label after reveiwing all Cypress test

/ok-to-test sha=83aa75a

@rajatagrawal This adds an API for table widget, adding a new feature that did not exist earlier.
This means that we need to do two things:

Add this to the Table Widget PRD with the help of @dilippitchika and @sbalaji1192
Update the documentation of Table widget on docs.appsmith.com
Write a cypress test, testing this new API
We need a test plan for this which the @appsmithorg/qa can help create for your review and approval. Once created we can link to the test plan in the description.
Update the PR description answering the following questions (This will be necessary for our release notes)

What was the use-case? (I guess the original issue describes this, a TL;DR will be helpful)
What did we do to satisfy the use-case?
Was there anything else we considered but went with this approach?

/build-deploy-preview env=release

/ok-to-test sha=cc3b960

/ok-to-test sha=f5e752e5d7

/ok-to-test sha=b0a034a

@kamakshibhat-appsmith

/ok-to-test sha=a361226

/ok-to-test sha=73f3777

/build-deploy-preview env=release


French

Hello
Here is my small contribution for French translation
	"fr-FR": {
		monthNames: ["Janvier", "Février", "Mars", "Avril", "Mai", "Juin", "Juillet", "Août", "Septembre", "Octobre", "Novembre", "Décembre"],
		dayNames: ["Dimanche", "Lundi", "Mardi", "Mercredi", "Jeudi", "Vendredi", "Samedi"],
		msgEmptyContent: "Aucune information à afficher."
	},

Can be use for fr-FR, fr-CA, fr-BE, ... Perhaps should you also check on the three first positions ("en", "fr", ...) and not on the five...

the same for all locations ?
fr-BE	Belgium
fr-CA	Canada
fr-FR	France
fr-LU	Luxembourg
fr-MC	Monaco
fr-CH	Switzerland

For these strings, yes.
It's like your en-US : will also works for en-GB ;-)
So, perhaps, code something like "If I don't match pt-BR or pt-PT, check if there is something for just 'pt'"; just a suggestion ;-)

Check new version (2.0.0)
Override the default language.
The public variable "$.fn.albeTimeline.languages" allows you to specify multiple languages in the timeline.

$(document).ready(function () {
//New CutureInfo
$.fn.albeTimeline.languages = [{
    "fr-FR": {
        days: ["Dimanche", "Lundi", "Mardi", "Mercredi", "Jeudi", "Vendredi", "Samedi"],
        months: ["Janvier", "Février", "Mars", "Avril", "Mai", "Juin", "Juillet", "Août", "Septembre", "Octobre", "Novembre", "Décembre"],
        shortMonths: ["Jan","Fév","Mar","Avr","Mai","Juin", "Juil","Août","Sep","Oct","Nov","Déc"],
        msgEmptyContent: "Aucune information à afficher."
    }
}];

$("#myTimeline").albeTimeline(data, {
    language: "fr-FR",  //Default: pt-BR
});

});

It's working 👍
No need to hack your file, it's a good point.
I suggest you to put the english text immediatly in your script and other languages in a separater .intl.min.js file f.i. so, english speaking people will ony link one .js
At this time, I've just copy/paste your translations (fr and us) in my own js and it's working.
Note : If I type the following code, there is a javascript fatal error
$("#myTimeline").albeTimeline(data, {
    language: "xx-XX"
});

because that language is invalid.  The best deal would first to check the existence of that language in your languages array and, as fallback, use the international english text to avoid error.
(the language can be set by the user who don't know if the plugin supported that locale)

Fixed bug in v.2.0.1
Thanks for your contribution.


if overlay is omitted leave out dependency on mount unit

In my local setup with conair containers I don't use the overlay step. docker, etcd and fleet currently fail because of a hardcoded dependency on the mount unit in their templates.
so i passed through if the overlay is used to the templates and left out Requires,After if overlay isn't used.

ping @hectorj2f @denderello

I can feel the pain to pass the options down to the templates. For now it's the fastest way to get what you want I admit. Later we could think of generating the unit files rather than using static templates.
Codewise LGTM. If it works in your conair setup and tests pass let's go for it.


remove PhantomData from gates

in https://github.com/o1-labs/proof-systems/blob/master/kimchi/src/circuits/polynomials/complete_add.rs#L50
#[derive(Default)]
pub struct CompleteAdd<F>(PhantomData<F>);

should be able to compile to
pub struct CompleteAdd<F>;

unless I'm missing something
(same for all the other gates)

will take a look ;)

Note: It won't compile as it is, the compiles requires (from my understanding) some type (like PhantomData) to pretend it actually stores a value, even though it doesn't. Removing it would probably require moving a few things around, not sure if that's what is wanted

Ah interesting, maybe I was wrong in my assumption. I though it was here because we used to need to call default() on it. What was the error you got btw?

parameter `F` is never used
consider removing `F`, referring to it in a field, or using a marker such as `PhantomData`
if you intended `F` to be a const parameter, use `const F: usize` insteadrustc [E0392](https://doc.rust-lang.org/error-index.html#E0392)

I would suspect that removing PhantomData entirely would require some structural changes, but not sure - I could also just be missing something, tried a few things already

Yeah I guess the error makes sense! You can't use a generic parameter if it's not used, hence PhantomData. Well thanks a lot for investigating this! Sorry for recommending a task with no better outcome :D

No worries, thats also a part of the process!


Open a user's profile page when someone clicks on his name in my trips

In my trips, if someone clicks on any user's name, open that user's profile page similar to what we see in my profile currently

On clicking the + icon, the view should show/hide
Show the arrow icon on each friend item
On clicking anywhere on the list item, the user's profile page should open up.

Adding more enhancements to this issue:
Create the UI like this:

On clicking the + icon, the view should show/hide
Show the arrow icon on each friend item
On clicking anywhere on the list item, the user's profile page should open up.

I'm taking this up.

Sure @aayushinigam

@Swati4star Do you suggest using an ExpandableListView for implementing the first point?

Yes. Something like that only.


CLI flag to test given file or file/line instead of filtering by pattern

Clear and concise description of the problem
Coming from elixir I'm very used to running tests with mix test path/to/file_test.exs or mix test path/to/file_test.exs:20. That's very convenient to setup with an editor to have keyboard shortcuts to trigger either of them relative to where my cursor currently sits, which allows for a very rapid workflow between making changes and running tests.
vitest currently only offers test filtering by pattern, which doesn't really seem to support this usecase. Manually coming up and typing out the correct filter conditions is much more tedious.
Suggested solution
Have a CLI frag, which can receives a path. For a directory execute all test files in that directory, for a file path just that single file and potentially also be able to execute just the test at line 12 with path/to/file.test.js:12.
Alternative
No response
Additional context
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

This is a duplicate of https://github.com/vitest-dev/vitest/issues/5445


Enable regex's ^, $ and ^$ to be matched in file dialog

Fixes #278

Hi @Tyriar, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

In order for us to evaluate and accept your PR, we ask that you sign a contribution license agreement. It's all electronic and will take just minutes. I promise there's no faxing. https://cla.microsoft.com.

TTYL, MSBOT;

Find and replace /^/

Find and replace /^/

Find and replace /^/

@Tyriar, Thanks for signing the contribution license agreement so quickly! Actual humans will now validate the agreement and then evaluate the PR.
Thanks, MSBOT;

Looks awesome.

@Tyriar This is great! Thank you so much! :heart:

Guys i Want the oposite of that if i want to find   this pattern  }{     nad i want to make line break  like
I have this
{"index":{"_id":0}},{"index":{"_id":0}}
And i want this
{"index":{"_id":0}}
{"index":{"_id":0}}
is ant wat to filter for     },{   And add line break in midle
}
{

@atlabiz

search for },{
replace with },\n{
enable regex to have the \n in the replace field mapped to a newline and not to the literals \ and n.


Fix fabric3 for Python 3.10+

In Python 3.10+ the Mapping moved to collections.abc, this proposed change made my fabric3 install work

@urielka the upstream fabric project actually released an updated 1.x version last year which is working with Python 3.10.
So changing my requirements.txt as follows solved the issue for me:
-Fabric3==1.14.post1
+fabric==1.15.0

Thanks!
On Thu, Mar 23, 2023, 12:24 AM Michael Kurze @.***>
wrote:

@urielka https://github.com/urielka the upstream fabric project
https://www.fabfile.org/changelog-v1.html actually released an updated
1.x version last year which is working with Python 3.10.
So changing my requirements.txt as follows solved the issue for me:
-Fabric3==1.14.post1+fabric==1.15.0
—
Reply to this email directly, view it on GitHub
https://github.com/mathiasertl/fabric/pull/60#issuecomment-1480335647,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAHR3C257GYYDSVJMF2ECTW5N3YJANCNFSM6AAAAAAWC7CRDU
.
You are receiving this because you were mentioned.Message ID:
@.***>

Thanks for your contribution!
As @x1B  noted, please use the stock version of fabric.
The front readme mentions that the project is now unmaintained.


Add using_read_replica_pool

Adds the AutoReplica.using_read_replica_pool method, that allows passing one own's ConnectionPool: in this case, the connections won't be closed at the end of the block.

Closing since already merged upstream.


fix corejs warning for @babel/preset-env

see https://github.com/gatsbyjs/gatsby/pull/12781/files

ah it seems its only needed for a newer preset-env version...

Please check out the latest v7. Is this still needed?


Add host entries to coredns with typhoon k8s deploy

Hi,
Would like to know if and how i could add some host entries into 'coredns' deployment that is part of the typhoon k8s created cluster?
I'm doing some baremetal deployment tests. On those tests i'm deploying using ipxe/matchbox/dnsmasq, having a dns entry for each node and for the cluster name on the dnsmaq i'm running as part of the deployment process.
After the deployment is finished i would like to stop the dnsmasq container/service, and have the coredns service running on the cluster also answer for the node and cluster names. On the network where the cluster is, there is no other DNS server available, and for external names we use Cloudflare and Google DNS's as resolvers (forwarders)
Is this stupid thinking? i've read the documentation of typhoon and the only reference to dns that i can find is about creating the dns entries (A / AAAA) for the node and cluster name, so i'm not sure if what i'm trying to do is even possible.
Thanks for any tip how to address this.
Regards,

Every bare-metal node should be identified by a DNS record. Controller(s) should also have a resolvable DNS record. Nodes are configured to refer to the control plane and themselves by these fully qualified names (no hardcoded IPs) and they're used in TLS certificates. In-cluster CoreDNS is part of the control plane, so you cannot rely on it to provide node records (consider when etcd and CoreDNS are both offline). On every platform (not just bare-metal), the network provider must offer some stable means for node resolution. On bare-metal, that's up to you. See docs.
Lastly, please use the Issue Template in future. This is solidly in the advice / help category which is mentioned there.

Thank you!
And apologies for not having used the template.


Support for custom start & stop symbols

$interpolationProvider allows us to override the default start and stop symbols from {{ & }} to anything you'd want.
In my case, I use Nunjucks to render some templates, which also uses the same symbols for substitution. Can trNgGrid provide support for custom symbols? ng-grid just implemented this, but I love trNgGrid too much!

Will have a look. Might fit into some other things I have to work on.

Released in 3.1.6


Extend connectivity-check for HTTP policy validation via CUE

This PR introduces a CUE-based framework for defining connectivity checks to consolidate the definitions of the various connectivity checks and support simpler, more concise declarations of new connectivity checks.
Review commit-by-commit.
Why CUE?
CUE is a data constraint language built defined as a superset of JSON which aims to "simplify tasks involving defining and using data".  See the second commit for a more detailed discussion, talking points below as a summary:
In context of the connectivity check YAMLs, CUE is useful to allow us to
"evaporate" the boilerplate necessary to define Kubernetes YAMLs for
Deployments, Services and CiliumNetworkPolicies and allow developers to
specify various permutations for connectivity checks concisely.

Why should we use it?
* It's more concise. [...]
* Consolidate constant declaration. [...]
* More structured data declarations for k8s resources of checks [...]. Can build powerful tooilng around this.

What are the dangers (+mitigations)?
* It's relatively imature, v0.2.2. This PR pins CUE usage to this version. [...]
* Another language in the tree. [...] This however aligns with Go and k8s community exploration of the language.
* It's easy to hide all complexity, templating away all parameters in a way that makes the declarations implicit. It'll be up to us to figure out how explicitly to declare our check definitions. [...]
* Additive declarations. Evaluation of language statements are not ordered by line ordering. Can be counter-intuitive. [...]

Based on the above and the experience of converting these checks over to CUE, I believe this is a useful and powerful tool for declaring YAML variations, perfect for a use case like connectivity-checks. We may in future even be able to explore using CUE for defining manifests in the CI.
How do we know these changes are correct?
This PR was validated against the existing "hostport" connectivity check YAML from the tree using the make inspect target introduced in this PR. I've attached the diff for existing checks below for reference; the commit message for the relevant changes also highlights the specifics of what has generally changed:
 * All containers are converted to use readinessProbe and not
   livenessProbe.
 * All readiness probes now specify --connect-timeout of 5s.
 * Readiness probes access `/public` or `/private` per the underlying
   container HTTP server paths rather than just accessing `/`.
 * DNS allow policies are converted to consistently allow both TCP and
   UDP-based DNS.
 * Container names are derived from pod names.
 * The new YAMLs declare additional labels for all resourcess, such as
   'component' and 'topology'.

connectivity-check-hostport.yaml.diff.txt
Tested on a 2-node GKE cluster per the Cilium getting-started guide.
Extending the connectivity checks for HTTP policy validation
See the commit log for details:
These new checks configure various L7 proxy paths to validate
connectivity via L7 proxies, in the following dimensions:

- Apply policy on egress; ingress; or both (proxy-to-proxy)
- Intra-node / Multi-node
- Allow / Deny

Note that proxy-to-proxy always configures egress allow policy to ensure
that the traffic goes via the proxy and in the drop case the requests
are only rejected at the destination. This is because applying egress
deny at the source would prevent proxy-to-proxy connectivity, meaning
the test would be equivalent to the egress-only reject policy case. This
way, we ensure that the path via the egress proxy to the destination is
tested in the reject case.

These are implemented partially through a new 'echo-c' pod which always
has ingress policy applied to allow GET requests to '/public'. Depending
on whether ingress policy is needed to check the particular permutation
the new checks may connect to 'echo-a' or 'echo-c'.

These are implemented by adding pods for each permutation of policy
apply point and topology; then by adding allow / deny containers within
that pod to test the allow/deny cases.

The 'connectivity-check-proxy.yaml' includes all of the above.

Finally, the omissions: This commit does not attempt to address
variations in datapath configuration. This includes IPv4 vs. IPv6;
tunnel/direct-routing; endpoint config; kube proxy/free; encryption.
These are left up to the cluster operator configuring Cilium in specific
modes and subsequently deploying these YAMLs.

Tooling
One interesting aspect around CUE is the ability to write tailored CLIs against the data to filter the data you want and output in the format you're after. See the commit message for more details of the new tooling:
  $ cue cmd help
  List connectivity-check resources specified in this directory

  Usage:
    cue [-t component=<component>] [-t name=<name>] [-t topology=<topology>] <command>

  Available Commands:
    dump   Generate connectivity-check YAMLs from the cuelang scripts
    ls     List connectivity-check resources specified in this directory

List available connectivity-check components:

  $ cue cmd ls
  KIND                  COMPONENT        TOPOLOGY     NAME
  Service               network-check    any          echo-a
  Service               services-check   any          echo-b
  Service               services-check   any          echo-b-headless
  Service               services-check   any          echo-b-host-headless
  Deployment            network-check    any          echo-a
  Deployment            services-check   any          echo-b
  Deployment            services-check   any          echo-b-host
  Deployment            network-check    any          pod-to-a
  Deployment            network-check    any          pod-to-external-1111
  Deployment            policy-check     any          pod-to-a-allowed-cnp
  Deployment            policy-check     any          pod-to-a-denied-cnp
  Deployment            policy-check     any          pod-to-external-fqdn-allow-google-cnp
  Deployment            services-check   multi-node   pod-to-b-multi-node-clusterip
  Deployment            services-check   multi-node   pod-to-b-multi-node-headless
  Deployment            services-check   intra-node   pod-to-b-intra-node-clusterip
  Deployment            services-check   intra-node   pod-to-b-intra-node-headless
  Deployment            services-check   multi-node   host-to-b-multi-node-clusterip
  Deployment            services-check   multi-node   host-to-b-multi-node-headless
  CiliumNetworkPolicy   policy-check     any          pod-to-a-allowed-cnp
  CiliumNetworkPolicy   policy-check     any          pod-to-a-denied-cnp
  CiliumNetworkPolicy   policy-check     any          pod-to-external-fqdn-allow-google-cnp

These can be filtered by component, topology or name. For example:

  $ cue cmd -t component=network ls
  KIND         COMPONENT       TOPOLOGY    NAME
  Service      network-check   any         echo-a
  Deployment   network-check   any         echo-a
  Deployment   network-check   any         pod-to-a
  Deployment   network-check   any         pod-to-external-1111

Finally, to gather the (filtered) YAMLs for the specified resources:

  $ cue cmd dump | head -n 20
  metadata:
    name: echo-a
    labels:
      name: echo-a
      topology: any
      component: network-check
  spec:
    ports:
      - port: 80
    selector:
      name: echo-a
    type: ClusterIP
  apiVersion: v1
  kind: Service
  ---
  ...

Q: Do I need to install CUE?
No, the Makefile introduces usage of the cue docker image to run the YAML generation commands. For more advanced usage like the tooling described above however, you would need to either install CUE or run it via the docker container in a similar way to how the makefile does.
Outstanding questions

For now I have introduced the new checks (proxy-based and hostport+proxy which are quarantined) into separate YAMLs to allow them to be tested independently. They are not part of the simple YAML. This allows the regular connectivity-check YAML to remain small and contained to allow it to reliably pass in smaller environments. We should consider how to expand the testing of these YAMLs in CI.

Related: #8491

test-focus K8sConformance.*connectivity

This is because applying egress deny at the source would prevent proxy-to-proxy connectivity.

Well, it's also because then it would be exactly the same as the egress-only reject policy. I should update the commit message to reflect this.

Coverage decreased (-0.01%) to 37.014% when pulling f9f37572e5732f9e398531106b56def0e98c4e99 on joestringer:submit/proxy-connectivity-check into 9d6e9dda30116aa59749567660f0b4ae33ed808a on cilium:master.

The two tests that use this connectivity-check passed:

https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated-Focus/348/testReport/Suite-k8s-1/18/K8sConformance_Portmap_Chaining_Check_connectivity_check_compliance_with_portmap_chaining/
https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated-Focus/348/testReport/Suite-k8s-1/18/K8sConformance_Portmap_Chaining_Check_one_node_connectivity_check_compliance_with_portmap_chaining/

So from a CI perspective this should be good to go. Only question might be whether we want to add a dedicated CI test for this so we can more easily focus on just the proxy failures if a regression is introduced.

test-focus K8sConformance.*connectivity

There's one failure occurring in the hostport proxy case, which is consistent with what I'm observing in a test cluster. I believe that's actually a Cilium bug. Looking into adding quarantine labels to allow us to add such YAMLs but not automatically deploy them.

test-focus K8sConformance.*connectivity

This works in manual testing + CI so probably good enough for review. I'll follow up on the smoke tests next week.

I thought I'd found the underlying reason for the failure in smoke test (all tests that check for failure should have probe timeouts that are longer than the timeout of the probe command), but it appears that the smoke test is still failing. I've submitted https://github.com/cilium/cilium/pull/12689 to improve debuggability, so will look to merge that first then rebase & re-run the checks.

I thought I'd found the underlying reason for the failure in smoke test (all tests that check for failure should have probe timeouts that are longer than the timeout of the probe command), but it appears that the smoke test is still failing. I've submitted #12689 to improve debuggability, so will look to merge that first then rebase & re-run the checks.

Is the failure related to the change from livenessProbe to readinessProbe? I copied your additional test cases (including quarantine catagory as well) into another PR #12671, and need to pass --set global.hostServices.enabled=true flag.

List of developments
deployment.apps/echo-a condition met
deployment.apps/echo-b condition met
deployment.apps/echo-b-host condition met
deployment.apps/echo-c condition met
deployment.apps/host-to-b-multi-node-clusterip condition met
deployment.apps/host-to-b-multi-node-headless condition met
deployment.apps/pod-to-a condition met
deployment.apps/pod-to-a-allowed-cnp condition met
deployment.apps/pod-to-a-intra-node-proxy-egress-allow condition met
deployment.apps/pod-to-a-intra-node-proxy-egress-reject condition met
deployment.apps/pod-to-a-l3-denied-cnp condition met
deployment.apps/pod-to-a-multi-node-proxy-egress-allow condition met
deployment.apps/pod-to-a-multi-node-proxy-egress-reject condition met
deployment.apps/pod-to-b-intra-node condition met
deployment.apps/pod-to-b-intra-node-nodeport condition met
deployment.apps/pod-to-b-multi-node-clusterip condition met
deployment.apps/pod-to-b-multi-node-headless condition met
deployment.apps/pod-to-b-multi-node-nodeport condition met
deployment.apps/pod-to-c-intra-node-proxy-ingress-allow condition met
deployment.apps/pod-to-c-intra-node-proxy-ingress-reject condition met
deployment.apps/pod-to-c-intra-node-proxy-to-proxy-allow condition met
deployment.apps/pod-to-c-intra-node-proxy-to-proxy-reject condition met
deployment.apps/pod-to-c-multi-node-proxy-ingress-allow condition met
deployment.apps/pod-to-c-multi-node-proxy-ingress-reject condition met
deployment.apps/pod-to-c-multi-node-proxy-to-proxy-allow condition met
deployment.apps/pod-to-c-multi-node-proxy-to-proxy-reject condition met

Is the failure related to the change from livenessProbe to readinessProbe? I copied your additional test cases (including quarantine catagory as well) into another PR #12671, and need to pass --set global.hostServices.enabled=true flag.

Good question, it appears not related to this. I was able to reproduce locally using kind though. Still investigating.
It's odd that hostServices feature would be required; AFAICT kube-proxy should be present in the nodes and should provision the relevant service connectivity if Cilium does not.

It's odd that hostServices feature would be required; AFAICT kube-proxy should be present in the nodes and should provision the relevant service connectivity if Cilium does not.

I was checking on this point for past few days, but unable to crack it. Probably, I was having wrong assumption that it's IPv6 related issue. Keen to hear your inputs and findings on this matter.

I traced the smoke-test issue down to #12699 and added a new commit on top to mitigate.

test-focus K8sConformance.*connectivity

Go lint CI failure will be addressed by #12696. Smoke test is now passing. Running CI version of connectivity check to ensure it doesn't break there.

test-focus K8sConformance.*connectivity

test-focus K8sConformance.*connectivity

test-focus K8sConformance.*connectivity

All code owner signoffs, and all CI checks that rely on connectivity checks have passed:
https://github.com/cilium/cilium/pull/12599/checks?check_run_id=925168850
https://jenkins.cilium.io/job/Cilium-PR-Ginkgo-Tests-Validated-Focus/355/
Marking ready-to-merge.

@joestringer I've changed the release not from minor to misc since it's not user faced.


V4 btn-secondary-outline style fore-/back-ground contrast too low

Foreground: #ccc
Background: #fff (on demo page)
Contrast: 1.61:1  ❌ (minimum is 4.5:1)

Contrast this low makes it difficult to read button text which makes websites using this style less usable & less accessible to many people.
See Web Content Accessibility Guidelines for more info.

Duplicate of #17286


Off Nadir and GSD are optional values

think this issue is more related to the public docs rather than our code : https://satellitevu.slack.com/archives/C03MTLFFFLK/p1717078900968579?thread_ts=1717065595.767779&cid=C03MTLFFFLK

yeah, i want product's guidance here. we can of course just use defaults it no ONA or GSD is given (implicit approach) or require explicit use of either ONA or GSD. paging @mykolakozyr


Remove note that gitops is not supported for BareMetal

Issue #, if available:
This PR removes the note that GitOps is not suppported for BareMetal. It is support for all providers.
Description of changes:
Testing (if applicable):
Documentation added/planned (if applicable):
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

/cherry-pick release-0.18

/approve


ZEVA -ZEV Sales not appearing in Hyundai's MY report

Describe the Bug
Hyundai has pending and issued MY 2022 sales that are not showing up on the consumer ZEV sales page in 2022 MY report.  Similar situation to the Nissan issue that was resolved.
Expected Behaviour
These pending/issued sales should appear in the second tab of their MY report.
Implications
Hyundai will not see their MY2022 sales in their MY report.
Steps To Reproduce
Steps to reproduce the behaviour: The 2022 MY report has been submitted to the ZEVA system by the BCeID user.  Not sure if the MY report will have to be returned to the BCeID user to refresh the pages after the fix.
Development Checklist
(1) Copy the production database onto your local machine and investigate why this is happening using the version of the app that is currently in production.
(2) Fix the issue.

@shayjeff I took a look at this issue, and it looks like it's the same type of issue that happened with Nissan. There are a couple of options here:
(1) Send it back to Hyundai, after which I will make some data changes in the production database, then they should see their consumer sales, then they can resubmit.
or:
(2) I can make some code changes that address the underlying issue, then we can do a release to production. Note that this release will also contain everything that is "on dev", so this option will take longer because of testing, etc. Also, you will still have to send the report back to them for the sales to appear and for them to resubmit.
Let me know what you want to do!

@tim738745 option 1 works.  Should the report be returned today and approximately how long will the changes take before they can resubmit. The business would like to provide a timeline to Hyundai.
Thanks Tim!

@shayjeff Yes, you can return it today, then I can make the changes in a local copy of the database and confirm the numbers with you like last time (I'll send you a screenshot in teams), and then I can make the changes to the production database. This can probably all be done by the end of today; if not, then end of day Monday (I'm off tomorrow).

@tim738745 the report has been returned to Hyundai.

@shayjeff okay, sounds good! I sent you a screenshot with the sales numbers on teams.

The code fix for the underlying issue was also done as part of this card: https://github.com/bcgov/zeva/pull/2036


⚠ Bofrost.at has degraded performance

In d686566, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1100 ms

Resolved: Bofrost.at performance has improved in 57fd981 after 14 minutes.


ORC-859: Update maven-checkstyle-plugin to 3.1.2 and checkstyle to 8.44

What changes were proposed in this pull request?
This PR aims to update maven-checkstyle-plugin to 3.1.2 and checkstyle to 8.44.
Why are the changes needed?
This will bring the bug fixes of 8.30 ~ 8.44.

https://checkstyle.sourceforge.io/releasenotes.html

How was this patch tested?
Pass the GHA.

Thank you, @williamhyun . I'll merge this.


[FLINK-16351][flink-table-runtime-blink]Use LinkedHashMap for deterministic iterations

What is the purpose of the change
In this PR, we propose to fix [FLINK-16351] by making the test assertions more stable by considering any order of the result.
Brief change log
In flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/operators/bundle/AbstractMapBundleOperator.java, we use LinkedHashMap instead of HashMap in open.
Verifying this change
This change is already covered by existing tests, such as (please describe tests).
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): ( no)
The serializers: (don't know)
The runtime per-record code paths (performance sensitive): ( no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: ( no )
The S3 file system connector: ( no)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

Closing for the same reason as #11287.


Add ability for white team to add additional scoring

Add the ability for white team to add additional factors for scoring

Injects
Orange team checks (this will require a new login for orange team to be able to log in their scores)
IR/Malware Reports
CEO points

We will also need updated graphs on the main page:
Stacked graph for total (with all sections of scoring)
Clickable dropdown to change between different scoring sessions

http://www.chartjs.org/docs/latest/charts/radar.html for the total


Upgrade the envtest version

Signed-off-by: tenzen-y<EMAIL_ADDRESS>
What this PR does / why we need it:
We upgraded the K8s version in #1684, so I also upgraded the envtest and setup-env version.
/cc @johnugeorge
Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #
Checklist:

[ ] Docs included if any changes are user facing

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
9 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.2%) to 39.527%

Files with Coverage Reduction
New Missed Lines
%

pkg/controller.v1/pytorch/pytorchjob_controller.go
9
60.32%

Totals

Change from base Build<PHONE_NUMBER>:
-0.2%

Covered Lines:
2340

Relevant Lines:
5920

💛  - Coveralls


Null Authorization header field name and tool tip, when browser langunage is not en

Description:
The authorization header field name in API store becomes null when the browser language is not English.

Fixed with https://github.com/wso2/carbon-apimgt/pull/6027