MM-23665: Read channel right before leaving it

Summary
When switching channels, we need to mark the one we are leaving as read as soon as possible, to avoid race conditions when new posts are arriving.
The problem this PR solves is the following:
Users A and B are in channel X. User B sends a message and as soon as they sent it, user A clicks on channel Y to switch to it. The webapp receives the post in channel X, but it's not added into the posts list before user A leaves.
User A then sees channel X marked as unread, but a few milliseconds later, the unread mark is gone.  This happens because we mark channels X and Y as read at the same time, when channel Y has already been loaded and the new post has already been counted into the total messages of channel X. So we incorrectly mark the new post as read.
The solution is simple: mark channel X as read right before leaving it, not after having switched to Y.
Publishing as a draft PR because I am not sure of the implications of my changes, both performance-wise and about possible side-effects.
Ticket Link
https://mattermost.atlassian.net/browse/MM-23665
Related Pull Requests
None
Screenshots
None

This code looks good, although we might want to bump it to 5.24 since this is a pretty critical piece of functionality, and that would give it more time to soak on community-daily

we might want to bump it to 5.24

Fine for me. This issue has been around for a long long time (it reproduces in 5.16, at least) and is not critical, but any error in this PR may be.
I'll mark this as ready for review.

 Thank you!

/update-branch

Thank you @agarciamontoro
I am seeing few errors on the PR server on Chrome, not sure if they are related to this fix

There is also an issue with manually marking posts as unread.
I am expecting a channel with the posts I just marked as unread to remain Unread when I navigate away to a different channel. But on the test server the Unread status does not persist, in this case, channel is marked as unread as soon as I click on a different channel.

Thank you for testing this, @jgilliam17!
I'll review the errors you shared in the screenshot. I'd say that they are not related to these changes, but I'm handwaving here, I need to check.
Regarding the marking messages as unread, I see how this could be breaking that. Let me investigate this. Thank you for bringing this up!

The last commit should fix the issue with the manually marking the channel as unread.
Regarding the console errors: I was not able to reproduce them. Please tell me if you still see them and, in case they do, when they show up.
Thank you, @jgilliam17!

/update-branch

Fixed failing tests (forgot to add the new prop there!) and the failing builds (master was unstable). Ready to go, @jgilliam17 :rocket:

Thank you @agarciamontoro
The console error I am seeing shows up on Chrome only(not seeing this on Firefox), every time I switch channels, something about channel member count and timezones.

I am still seeing the issue with marking posts as unread on the test server - the setting does not persist when I switch channels.

@jgilliam17 Looks like the test server did not update correctly. It's on 26ae698c5fb87a20348760b62e32c977ce8d04cc Should we restart it?

Closing this for the moment. Reasons described in the ticket: https://mattermost.atlassian.net/browse/MM-23665?focusedCommentId=88206

/update-branch

Error trying to update the PR.
Please do it manually.

Reopening after @devinbinnie's work on this, that confirmed that this PR does not fix the issue completely but it at least mitigates it. See his comment on the ticket.

/update-branch

/cherry-pick release-5.24

Error trying doing the automated Cherry picking. Please do this manually
+++ Updating remotes...
Fetching upstream
Fetching origin
+++ Updating remotes done...
+++ Creating local branch automated-cherry-pick-of-#5385-upstream-release-5.24-1590510186
Branch 'automated-cherry-pick-of-#5385-upstream-release-5.24-1590510186' set up to track remote branch 'release-5.24' from 'upstream'.
+++ Downloading patch to /tmp/5385.patch (in case you need to do this again)

+++ About to attempt cherry pick of PR. To reattempt:
  $ git am -3 /tmp/5385.patch

Applying: MM-23665: Read channel right before leaving it
Applying: Fix spacing
Applying: Remove PostList's prevChannelId-related dead code
Using index info to reconstruct a base tree...
M	components/channel_view/channel_view.jsx
M	components/channel_view/channel_view.test.jsx
Falling back to patching base and 3-way merge...
Auto-merging components/channel_view/channel_view.test.jsx
CONFLICT (content): Merge conflict in components/channel_view/channel_view.test.jsx
Auto-merging components/channel_view/channel_view.jsx
Patch failed at 0003 Remove PostList's prevChannelId-related dead code
Use 'git am --show-current-patch' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

+++ Conflicts detected:

UU components/channel_view/channel_view.test.jsx

+++ Aborting in-progress git am.

+++ Returning you to the master branch and cleaning up.


Add JSON as optional format for sending activity events via MQTT

As of v6.1.1 user activity data (connection open/close, session start/stop), are sent as MQTT messages in a plain text string format.
This makes it harder than necessary for receiving systems to use the data coming from Butler SOS.
Add second option where data is sent as JSON instead.
Keep the old method for backwards compatibility.
Add config file setting to control which topic to use for the JSON data:
Butler-SOS:
  userEvents:
    sendToMQTT:
      topic: qliksense/userevent
      postTo:
        jsonTopic: true

This would post JSON data to qliksense/userevent/json

Duh. MQTT user activity messages are already in JSON...


Jvenstad/fix config model inconsitency

@hmusum please review. This should not have any functional changes.

@hakonhall FYI on the linting (last commit) :)


Add job service improvements to 2.7

This adds the below changes to branch-2.7:
https://github.com/Alluxio/alluxio/pull/14530
https://github.com/Alluxio/alluxio/pull/15095
https://github.com/Alluxio/alluxio/pull/15106
Many conflicts were manually resolved because these commits are not in branch-2.7:
I assume we don't want to touch the checkstyle version in branch-2.7 and therefore omitted
https://github.com/Alluxio/alluxio/commit/8f74e6584b6d2d71be3a3737057808b6f77c44ae
The below change was not included in branch-2.7, and we manually cherry-picked PR#14530. I take the hint that we don't want this fix to be in branch-2.7. And if anyone hits that issue, their only option is to upgrade.
https://github.com/Alluxio/alluxio/commit/209b7d1e0f417f4357a0688b974c93a010ceb7e7
This metrics complement was not deemed necessary and therefore omitted
https://github.com/Alluxio/alluxio/commit/696cb89bfc8b7dd41393e3003307947d2110e21f

The latest commit had to be done to avoid cherry-picking the missing commits from https://github.com/Alluxio/alluxio/commits/master/tests/src/test/java/alluxio/job/util/JobTestUtils.java
I tried bringing in the missing 2 commits in https://github.com/Alluxio/alluxio/commits/master/tests/src/test/java/alluxio/job/util/JobTestUtils.java but those just introduced more dependencies. This can go one forever and we just don't have the tools.

Codecov Report

Merging #15138 (c125372) into branch-2.7 (1a3b3a6) will decrease coverage by 24.83%.
The diff coverage is 24.23%.

@@                Coverage Diff                @@
##             branch-2.7   #15138       +/-   ##
=================================================
- Coverage         41.83%   16.99%   -24.84%
+ Complexity         9316     2707     -6609
=================================================
  Files              1493     1494        +1
  Lines             87480    87925      +445
  Branches          10427    10479       +52
=================================================
- Hits              36599    14945    -21654
- Misses            47900    71738    +23838
+ Partials           2981     1242     -1739

Impacted Files
Coverage Δ

...lluxio/client/file/cache/store/RocksPageStore.java
0.00% <0.00%> (-74.26%)
:arrow_down:

.../main/java/alluxio/exception/ExceptionMessage.java
0.00% <0.00%> (-99.41%)
:arrow_down:

...in/java/alluxio/underfs/ObjectUnderFileSystem.java
0.00% <0.00%> (ø)

...n/src/main/java/alluxio/underfs/UfsFileStatus.java
18.51% <ø> (-29.63%)
:arrow_down:

...luxio/master/journal/raft/JournalStateMachine.java
0.00% <0.00%> (-39.45%)
:arrow_down:

.../master/journal/raft/RaftJournalConfiguration.java
0.00% <0.00%> (-81.82%)
:arrow_down:

...alluxio/master/journal/raft/RaftJournalSystem.java
0.00% <0.00%> (-54.87%)
:arrow_down:

...master/journal/ufs/UfsJournalGarbageCollector.java
39.28% <0.00%> (-2.23%)
:arrow_down:

...alluxio/master/journalv0/ufs/UfsJournalReader.java
0.00% <0.00%> (ø)

...uxio/master/FaultTolerantAlluxioMasterProcess.java
0.00% <0.00%> (-50.49%)
:arrow_down:

... and 743 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5121af4...c125372. Read the comment docs.

Closed as the cherry-pick has been handled by another PR


log rotation feature crashes intermittently

I have no idea if it is related to the other crash issues I've seen documented, but I am seeing:
Error: ENOENT, open '/work/server/server.log'
Every once in a while. In the directory there is a server.log.gz that is 0 bytes along with other log files that have been successfully rotated.
The issue seems to happen when logging frequency and content is high.
I've handled the uncaught exception, but there's not really anything I can do about it since the logger blew chunks.
I'm doing:
winston.add(winston.transports.File, {
    filename: __dirname + '/server.log',
    json: false,
    colorize: true,
    timestamp: true,
    maxsize: 10000000,
    maxFiles: 50,
    showLevel: true,
    tailable: true,
    zippedArchive: true,
});
winston.remove(winston.transports.Console);

Moved here: https://github.com/winstonjs/winston-daily-rotate-file/issues/9

Why was this moved? I'm not using that transport. I'm using features in core winston.

Id love to get a reproducible case of this if you can take a stab at it @nikdunn

Copied from: https://github.com/winstonjs/winston/issues/800. One of the tickets should be cancelled.
The problem seems to be a result of the self._archive variable being set to '/work/server/server.log' when the code enters the compressFile() function. See the code here: https://github.com/winstonjs/winston/blob/master/lib/winston/transports/file.js#L468-L530
Usually, when the code executes:
self._stream = fs.createWriteStream(fullname, self.options);
fullname is  '/work/server/server.log' and self._archive is '/work/server/server1.log'
So winston opens a new file for writing new log data ('/work/server/server.log') and opens a read stream ('/work/server/server1.log') that gets piped through gzip and to a write stream ('/work/server/server1.log.gz').
I believe the crash occurs when '/work/server/server.log' sneaks through into the compressFile() function.
I think the sequence is:

self._stream = fs.createWriteStream(fullname, self.options). This does not wait for the 'open' event to be emitted before,
compressFile immediately tries to do fs.createReadStream(String(self._archive)) and in some circumstances the file does not exist yet.

That's my current theory anyway.
My question is: should winston ever gzip '/work/server/server.log'? Is this for the non-tailable log rotation case? Am I on the right track?
Okay fine that was three questions.

Follow up...
The problem is caused by two different maxsize tests conflicting with one another.
This line: https://github.com/winstonjs/winston/blob/master/lib/winston/transports/file.js#L371 tests maxsize against (I'm guessing) the number of bytes theoretically written to the file, but not necessarily written to the file. This causes _createStream() to be called.
This test: https://github.com/winstonjs/winston/blob/master/lib/winston/transports/file.js#L540 tests the actual file size of the log file.
I've seen the first test pass, and the second test fail. In that event, createAndFlush() gets called, but _incFile() does not. So createAndFlush attempts to create 'server.log' and reads from it to create 'server.log.gz'

I have this too:
Uncaught Error: ENOENT, open 'log/node.log'

Should've I created this file manually or what?

Maybe there's some racing condition when creating a file?

This is fixed. Here is the pull request. https://github.com/winstonjs/winston/pull/828

Closing this based on #828 but if problems persist in the latest versions of winston (3.1.0 or master), please let us know -- I am doing a bit of digging into 3.x log rotation today anyway.  Thanks!


Makes AcceptedConnectionsLimit configurable

bors merge

bors merge

closes https://github.com/input-output-hk/cardano-node/pull/3435


Deep FRI

Closes #159
Type of change

[x] New feature

Checklist

[x] Linked to Github Issue
[x] This change requires new documentation.

[x] Documentation has been added/updated.

Codecov Report

Merging #161 (11ff81a) into main (8b6a558) will increase coverage by 0.08%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #161      +/-   ##
==========================================
+ Coverage   96.08%   96.17%   +0.08%
==========================================
  Files          52       52
  Lines        6212     6319     +107
==========================================
+ Hits         5969     6077     +108
+ Misses        243      242       -1

Impacted Files
Coverage Δ

math/src/polynomial.rs
98.09% <100.00%> (+0.06%)
:arrow_up:

proving-system/stark/src/lib.rs
98.81% <100.00%> (+0.51%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


wasm on_http_request_body never end when decompressing gzipped request

Title: wasm on_http_request_body never end when decompression gzipped request
Description:

When GZip decompressor decompress the request before calling the wasm filter, on_http_request_body is never called with end_of_stream = true. Thus, if the wasm plugin need to buffer the body, it can never resume when the body is complete.

Repro steps:
With a Wasm plugins doing buffering on request body (see code fragment below), make a gzip encoded POST request. The request will never end.
fn on_http_request_body(&mut self, body_size: usize, end_of_stream: bool) -> Action {
        debug!("on_http_request_body: body_size({}) / end_of_stream: ({:?})", body_size, end_of_stream);

        if !end_of_stream {
            // Wait -- we'll be called again when the complete body is buffered
            // at the host side.
            return Action::Pause;
        }
        Action::Continue
    }

Config:
listeners filters configuration
        http_filters:
          - name: envoy.filters.http.decompressor
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.decompressor.v3.Decompressor
              decompressor_library:
                name: small
                typed_config:
                  "@type": "type.googleapis.com/envoy.extensions.compression.gzip.decompressor.v3.Gzip"
              response_direction_config:
                common_config:
                  enabled:
                    default_value: false
                    runtime_key: response_decompressor_enabled
              request_direction_config:
                common_config:
                  enabled:
                    default_value: true
                    runtime_key: request_decompressor_enabled
          - name: envoy.filters.http.wasm
            typed_config:
              "@type": type.googleapis.com/udpa.type.v1.TypedStruct
              type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
              value:
                config:
                  name: "prime_auth"
                  root_id: "prime_auth"
                  configuration:
                    "@type": "type.googleapis.com/google.protobuf.StringValue"
                    value: {}
                  vm_config:
                    runtime: "envoy.wasm.runtime.v8"
                    vm_id: "my_vm_id"
                    code:
                      local:
                        filename: "/etc/myenvoyfilter.wasm"
                    configuration: {}
          - name: envoy.filters.http.router
            typed_config: {}

Logs:
[2021-10-01 21:47:04.318][49][debug][conn_handler] [source/server/active_tcp_listener.cc:332] [C0] new connection from <IP_ADDRESS>:45082
proxy_1        | [2021-10-01 21:47:04.319][49][debug][http] [source/common/http/conn_manager_impl.cc:274] [C0] new stream
proxy_1        | [2021-10-01 21:47:04.319][49][debug][http] [source/common/http/conn_manager_impl.cc:858] [C0][S11407748546514101329] request headers complete (end_stream=false):
proxy_1        | ':authority', 'localhost:18000'
proxy_1        | ':path', '/anything'
proxy_1        | ':method', 'POST'
proxy_1        | 'content-length', '93'
proxy_1        | 'content-type', 'application/json'
proxy_1        | 'content-encoding', 'gzip'
proxy_1        |
proxy_1        | [2021-10-01 21:47:04.319][49][debug][filter] [source/extensions/filters/http/decompressor/decompressor_filter.cc:84] [C0][S11407748546514101329] DecompressorFilter::decodeHeaders: ':authority', 'localhost:18000'
proxy_1        | ':path', '/anything'
proxy_1        | ':method', 'POST'
proxy_1        | ':scheme', 'http'
proxy_1        | 'content-length', '93'
proxy_1        | 'content-type', 'application/json'
proxy_1        | 'content-encoding', 'gzip'
proxy_1        | 'x-forwarded-proto', 'http'
proxy_1        | 'x-request-id', 'ca44261b-733c-4a68-8299-b54b193fef12'
proxy_1        |
proxy_1        | [2021-10-01 21:47:04.319][49][debug][filter] [./source/extensions/filters/http/decompressor/decompressor_filter.h:178] [C0][S11407748546514101329] do decompress request: ':authority', 'localhost:18000'
proxy_1        | ':path', '/anything'
proxy_1        | ':method', 'POST'
proxy_1        | ':scheme', 'http'
proxy_1        | 'content-type', 'application/json'
proxy_1        | 'x-forwarded-proto', 'http'
proxy_1        | 'x-request-id', 'ca44261b-733c-4a68-8299-b54b193fef12'
proxy_1        |
proxy_1        | [2021-10-01 21:47:04.319][49][debug][router] [source/common/router/router.cc:444] [C0][S11407748546514101329] cluster 'web_service' match for URL '/anything'
proxy_1        | [2021-10-01 21:47:04.319][49][debug][router] [source/common/router/router.cc:630] [C0][S11407748546514101329] router decoding headers:
proxy_1        | ':authority', 'localhost:18000'
proxy_1        | ':path', '/anything'
proxy_1        | ':method', 'POST'
proxy_1        | ':scheme', 'http'
proxy_1        | 'content-type', 'application/json'
proxy_1        | 'x-forwarded-proto', 'http'
proxy_1        | 'x-request-id', 'ca44261b-733c-4a68-8299-b54b193fef12'
proxy_1        | 'x-envoy-expected-rq-timeout-ms', '15000'
proxy_1        |
proxy_1        | [2021-10-01 21:47:04.319][49][debug][pool] [source/common/http/conn_pool_base.cc:74] queueing stream due to no available connections
proxy_1        | [2021-10-01 21:47:04.319][49][debug][pool] [source/common/conn_pool/conn_pool_base.cc:245] trying to create new connection
proxy_1        | [2021-10-01 21:47:04.319][49][debug][pool] [source/common/conn_pool/conn_pool_base.cc:143] creating a new connection
proxy_1        | [2021-10-01 21:47:04.319][49][debug][client] [source/common/http/codec_client.cc:60] [C1] connecting
proxy_1        | [2021-10-01 21:47:04.319][49][debug][connection] [source/common/network/connection_impl.cc:864] [C1] connecting to <IP_ADDRESS>:80
proxy_1        | [2021-10-01 21:47:04.319][49][debug][connection] [source/common/network/connection_impl.cc:880] [C1] connection in progress
proxy_1        | [2021-10-01 21:47:04.319][49][debug][filter] [source/extensions/filters/http/decompressor/decompressor_filter.cc:157] [C0][S11407748546514101329] request data decompressed from 93 bytes to 178 bytes
proxy_1        | [2021-10-01 21:47:04.320][49][debug][wasm] [source/extensions/common/wasm/context.cc:1219] wasm log prime_auth prime_auth my_vm_id: on_http_request_body: body_size(178) / end_of_stream: (false)
proxy_1        | [2021-10-01 21:47:04.320][49][debug][http] [source/common/http/filter_manager.cc:825] [C0][S11407748546514101329] request end stream
proxy_1        | [2021-10-01 21:47:04.320][49][debug][filter] [source/extensions/filters/http/decompressor/decompressor_filter.cc:157] [C0][S11407748546514101329] request data decompressed from 0 bytes to 0 bytes
proxy_1        | [2021-10-01 21:47:04.320][49][debug][wasm] [source/extensions/common/wasm/context.cc:1219] wasm log prime_auth prime_auth my_vm_id: on_http_request_body: body_size(178) / end_of_stream: (false)
proxy_1        | [2021-10-01 21:47:04.320][49][debug][connection] [source/common/network/connection_impl.cc:669] [C1] connected
proxy_1        | [2021-10-01 21:47:04.320][49][debug][client] [source/common/http/codec_client.cc:88] [C1] connected
proxy_1        | [2021-10-01 21:47:04.320][49][debug][pool] [source/common/conn_pool/conn_pool_base.cc:289] [C1] attaching to next stream
proxy_1        | [2021-10-01 21:47:04.320][49][debug][pool] [source/common/conn_pool/conn_pool_base.cc:175] [C1] creating stream
proxy_1        | [2021-10-01 21:47:04.320][49][debug][router] [source/common/router/upstream_request.cc:386] [C0][S11407748546514101329] pool ready

on_http_request_body is called twice with same argument 178/ false. The second time should be 178/true.
The same request but not gzip encoded
[2021-10-01 21:53:24.570][59][debug][conn_handler] [source/server/active_tcp_listener.cc:332] [C0] new connection from <IP_ADDRESS>:47986
proxy_1        | [2021-10-01 21:53:24.570][59][debug][http] [source/common/http/conn_manager_impl.cc:274] [C0] new stream
proxy_1        | [2021-10-01 21:53:24.570][59][debug][http] [source/common/http/conn_manager_impl.cc:858] [C0][S14750111005009089871] request headers complete (end_stream=false):
proxy_1        | ':authority', 'localhost:18000'
proxy_1        | ':path', '/anything'
proxy_1        | ':method', 'POST'
proxy_1        | 'user-agent', 'Fiddler'
proxy_1        | 'content-length', '178'
proxy_1        | 'content-type', 'application/json'
proxy_1        |
proxy_1        | [2021-10-01 21:53:24.570][59][debug][filter] [source/extensions/filters/http/decompressor/decompressor_filter.cc:84] [C0][S14750111005009089871] DecompressorFilter::decodeHeaders: ':authority', 'localhost:18000'
proxy_1        | ':path', '/anything'
proxy_1        | ':method', 'POST'
proxy_1        | ':scheme', 'http'
proxy_1        | 'content-length', '178'
proxy_1        | 'content-type', 'application/json'
proxy_1        | 'x-forwarded-proto', 'http'
proxy_1        | 'x-request-id', '2be982fb-686a-4d40-8501-c3bb94383e63'
proxy_1        |
proxy_1        | [2021-10-01 21:53:24.570][59][debug][filter] [./source/extensions/filters/http/decompressor/decompressor_filter.h:182] [C0][S14750111005009089871] do not decompress request: ':authority', 'localhost:18000'
proxy_1        | ':path', '/anything'
proxy_1        | ':method', 'POST'
proxy_1        | ':scheme', 'http'
proxy_1        | 'content-length', '178'
proxy_1        | 'content-type', 'application/json'
proxy_1        | 'x-forwarded-proto', 'http'
proxy_1        | 'x-request-id', '2be982fb-686a-4d40-8501-c3bb94383e63'
proxy_1        |
proxy_1        | [2021-10-01 21:53:24.571][59][debug][router] [source/common/router/router.cc:444] [C0][S14750111005009089871] cluster 'web_service' match for URL '/anything'
proxy_1        | [2021-10-01 21:53:24.571][59][debug][router] [source/common/router/router.cc:630] [C0][S14750111005009089871] router decoding headers:
proxy_1        | ':authority', 'localhost:18000'
proxy_1        | ':path', '/anything'
proxy_1        | ':method', 'POST'
proxy_1        | ':scheme', 'http'
proxy_1        | 'user-agent', 'Fiddler'
proxy_1        | 'content-type', 'application/json'
proxy_1        | 'x-forwarded-proto', 'http'
proxy_1        | 'x-request-id', '2be982fb-686a-4d40-8501-c3bb94383e63'
proxy_1        | 'x-envoy-expected-rq-timeout-ms', '15000'
proxy_1        |
proxy_1        | [2021-10-01 21:53:24.571][59][debug][pool] [source/common/http/conn_pool_base.cc:74] queueing stream due to no available connections
proxy_1        | [2021-10-01 21:53:24.571][59][debug][pool] [source/common/conn_pool/conn_pool_base.cc:245] trying to create new connection
proxy_1        | [2021-10-01 21:53:24.571][59][debug][pool] [source/common/conn_pool/conn_pool_base.cc:143] creating a new connection
proxy_1        | [2021-10-01 21:53:24.571][59][debug][client] [source/common/http/codec_client.cc:60] [C1] connecting
proxy_1        | [2021-10-01 21:53:24.571][59][debug][connection] [source/common/network/connection_impl.cc:864] [C1] connecting to <IP_ADDRESS>:80
proxy_1        | [2021-10-01 21:53:24.571][59][debug][connection] [source/common/network/connection_impl.cc:880] [C1] connection in progress
proxy_1        | [2021-10-01 21:53:24.571][59][debug][wasm] [source/extensions/common/wasm/context.cc:1219] wasm log prime_auth prime_auth my_vm_id: on_http_request_body: body_size(178) / end_of_stream: (false)
proxy_1        | [2021-10-01 21:53:24.571][59][debug][http] [source/common/http/filter_manager.cc:825] [C0][S14750111005009089871] request end stream
proxy_1        | [2021-10-01 21:53:24.571][59][debug][wasm] [source/extensions/common/wasm/context.cc:1219] wasm log prime_auth prime_auth my_vm_id: on_http_request_body: body_size(178) / end_of_stream: (true)
proxy_1        | [2021-10-01 21:53:24.572][59][debug][connection] [source/common/network/connection_impl.cc:669] [C1] connected
proxy_1        | [2021-10-01 21:53:24.572][59][debug][client] [source/common/http/codec_client.cc:88] [C1] connected
proxy_1        | [2021-10-01 21:53:24.572][59][debug][pool] [source/common/conn_pool/conn_pool_base.cc:289] [C1] attaching to next stream
proxy_1        | [2021-10-01 21:53:24.572][59][debug][pool] [source/common/conn_pool/conn_pool_base.cc:175] [C1] creating stream
proxy_1        | [2021-10-01 21:53:24.572][59][debug][router] [source/common/router/upstream_request.cc:386] [C0][S14750111005009089871] pool ready
proxy_1        | [2021-10-01 21:53:24.575][59][debug][router] [source/common/router/router.cc:1230] [C0][S14750111005009089871] upstream headers complete: end_stream=false
proxy_1        | [2021-10-01 21:53:24.576][59][debug][filter] [source/extensions/filters/http/decompressor/decompressor_filter.cc:117] [C0][S14750111005009089871] DecompressorFilter::encodeHeaders: ':status', '200'
proxy_1        | 'server', 'gunicorn/19.9.0'
proxy_1        | 'date', 'Fri, 01 Oct 2021 21:53:24 GMT'
proxy_1        | 'connection', 'keep-alive'
proxy_1        | 'content-type', 'application/json'
proxy_1        | 'access-control-allow-origin', '*'
proxy_1        | 'access-control-allow-credentials', 'true'
proxy_1        | 'x-envoy-upstream-service-time', '3'
proxy_1        |
proxy_1        | [2021-10-01 21:53:24.576][59][debug][filter] [./source/extensions/filters/http/decompressor/decompressor_filter.h:182] [C0][S14750111005009089871] do not decompress response: ':status', '200'
proxy_1        | 'server', 'gunicorn/19.9.0'
proxy_1        | 'date', 'Fri, 01 Oct 2021 21:53:24 GMT'
proxy_1        | 'connection', 'keep-alive'
proxy_1        | 'content-type', 'application/json'
proxy_1        | 'access-control-allow-origin', '*'
proxy_1        | 'access-control-allow-credentials', 'true'
proxy_1        | 'x-envoy-upstream-service-time', '3'

This time, on_http_request_body is called twice two but the second time, end_of_stream is true.

cc @PiotrSikora

I do not have logs but a similar issue occured when the upstream returned gzip encoding response and a filter is set before wasm to uncompress the response.
While the wasm on_http_reponse_body do is called with end_of_stream flag, the content of the response body buffer is still the gzip one.

@PiotrSikora Did you had time to look at this ?


[AMBARI-25297] : Host names array to string conversion improvement (Backport to branch-2.7)

What changes were proposed in this pull request?
Instead of appending comma separated host names to StringBuilder and then doing toString, we can use String API for better readability and performance
How was this patch tested?
The patch was tested manually as well as with unit tests.
This is backport PR to branch-2.7 from here: https://github.com/apache/ambari/pull/3003

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/5280/
Test FAILed.
Test FAILured.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/Ambari-Github-PullRequest-Builder/5281/
Test PASSed.


[BUG] App Crashes on Xcode 16

Describe the bug
The application is crashing at launch when running with Xcode 16
Exception or Stack Trace
dyld[62802]: Symbol not found: __ZN5swift34swift50override_conformsToProtocolEPKNS_14TargetMetadataINS_9InProcessEEEPKNS_24TargetProtocolDescriptorIS1_EEPFPKNS_18TargetWitnessTableIS1_EES4_S8_E
Referenced from: <04AD06A6-6F99-3D10-B524-DF08994AC37F> /Users/yasheed/Library/Developer/CoreSimulator/Devices/66C55B41-9949-4F4B-8D4B-BA1489E973E5/data/Containers/Bundle/Application/43C88B12-E792-439B-97C7-20AC316BA7C0/Smart Office.app/Frameworks/AzureCommunicationCalling.framework/AzureCommunicationCalling
Expected in:     <29152051-C3D9-3F37-89DB-FEB770AEF05B> /Users/yasheed/Library/Developer/CoreSimulator/Devices/66C55B41-9949-4F4B-8D4B-BA1489E973E5/data/Containers/Bundle/Application/43C88B12-E792-439B-97C7-20AC316BA7C0/Smart Office.app/Frameworks/AzureCommunicationCommon.framework/AzureCommunicationCommon
To Reproduce
Open an existing project with Xcode 16 and run the project in any simulator or device.
Screenshots

Setup (please complete the following information):

OS: iOS 18
IDE : Xcode 16
SDK Version: 'AzureCommunicationCalling', '2.12.0'

Additional context
The same SDK is working fine with earlier version of Xcode

We already have a fix, just need to deliver the new patch version. We will update this thread once fix lands

Hi @lucianopa-msft, thank you for the swift response. When I can expect the fix? It's a complete development blocker for our team.

We already have the fix, just sorting out the build and release. Hoping to get it today or tomorrow, no later than Monday.

Hi @yasheedv, version 2.12.1 is out now with the fix. Can you please verify that it fixes the issue?
Note that it may take some time for it to be available on CocoaPods.

@lucianopa-msft the issue got resolved in version 2.12.1. Thank you for the support.

@lucianopa-msft Will there be a -beta version released too with the Xcode 16 fixes?

@JasonWeinzierl Next beta will have it, but I cannot give an ETA on that. I'll will update here as soon as we have it :)

@lucianopa-msft Will there be a -beta version released too with the Xcode 16 fixes?

@JasonWeinzierl new beta should be available with fixes https://github.com/Azure/Communication/releases/tag/v2.14.0-beta.1


Include instances as attributes of the relevant class

This is a proposal for how to include predefined metadata instances (e.g. controlled terms) in the Python library (cf #24). Usage example:
In [1]: from openminds.latest.controlled_terms import AgeCategory

In [2]: AgeCategory.adult
Out[2]: <openminds.latest.controlled_terms.age_category.AgeCategory at 0x104eb3970>

In [3]: AgeCategory.adult.id
Out[3]: 'https://openminds.ebrains.eu/instances/ageCategory/adult'

In [4]: AgeCategory.adult.definition
Out[4]: "'Adult' categorizes the life cycle stage of an animal or human that reached sexual maturity."

I don't quite understand from the jinja template whether all the instances will be embedded into the python class?
I had done something similar before in the openMINDS MATLAB package, but with suggestions from Lyuba ended up doing it differently.
This is the example of a schema definition for AgeCategory.
https://github.com/openMetadataInitiative/openMINDS_MATLAB/blob/main/code/schemas/latest/%2Bopenminds/%2Bcontrolledterms/AgeCategory.m#L1
As you can see, it has a hidden CONTROLLED_INSTANCES property, and this holds the name of all known instances of this schema. In the ControlledTerm superclass there is a case in the constructor to deserialise an json ld instance if the constructor is called with an input argument which matches one of the names in the CONTROLLED_INSTANCES property
Example:
adult = openminds.controlledterms.AgeCategory('adult')

adult =

  AgeCategory (https://openminds.ebrains.eu/controlledTerms/AgeCategory) with properties:

                     definition: "''Adult'' categorizes the life cycle stage of an animal or human that reached sexual maturity."
                    description: ""
             interlexIdentifier: "http://uri.interlex.org/base/ilx_0729043"
             knowledgeSpaceLink: ""
                           name: "adult"
    preferredOntologyIdentifier: "http://purl.obolibrary.org/obo/UBERON_0000113"
                        synonym: ["adult stage"    "post-juvenile adult"    "post-juvenile adult stage"]

  Required Properties: name

>> adult.id

ans =

    'https://openminds.ebrains.eu/instances/ageCategory/adult'

@ehennestad I let Andrew respond but it is not loading all instances into the class only the one of the stated type (as it is done for MATLAB as well, more or less)

Python name generated for 4-(2-Hydroxyethyl)-1-piperazineEthanesulfonicAcid is four_(2_hydroxyethyl)_1_piperazine_ethanesulfonic_acid which is not a valid python variable name.
6,7-Dinitro-1,4-dihydroquinoxaline-2,3-dione is converted to six,7_dinitro_1,4_dihydroquinoxaline_2,3_dione.

Just a comment. In MATLAB I ended up doing this:

Python name generated for 4-(2-Hydroxyethyl)-1-piperazineEthanesulfonicAcid is four_(2_hydroxyethyl)_1_piperazine_ethanesulfonic_acid which is not a valid python variable name.
6,7-Dinitro-1,4-dihydroquinoxaline-2,3-dione is converted to six,7_dinitro_1,4_dihydroquinoxaline_2,3_dione.

I also had this issue with a lot of instance names, as MATLAB is very strict about property names.
I ended up changing the constructor for controlled term classes to take the instance name as an input, for example:
openminds.controlledterms.MolecularEntity('4-(2-Hydroxyethyl)-1-piperazineEthanesulfonicAcid')
I think this is a better solution because you can keep using all the original names and don't have to worry about remapping them

@ehennestad and @Peyman-N I've fixed the Python name generation, but I've also added a by_name() class method to controlled-terms classes, similar to the solution suggested by @ehennestad. This method also uses synonyms for lookups, e.g.:
In [1]: from openminds.latest.controlled_terms import MolecularEntity

In [2]: MolecularEntity.by_name("HEPES") == MolecularEntity.by_name("4-(2-hydroxyethyl)-1-piperazine ethanesulfonic acid")
Out[2]: True


HomeGallery and folders

Hi!
i am exploring all available photo gallery open source tools out there for my self-hosting needs. So far i love how smooth is searching and moving trough photos with HomeGallery!
I have 20+ years of photos stored on the filesystem and sorted by folders, where each folder is an album (an event / trip / whatever). This sorting approach survived many photo organization tools so far and i find it quite effective.
While i have to say that HG approach with tags and smooth navitagion intrigues me a lot, i cannot find me stopping here because i still need to be able to browse my albums. Map view, timeline are cool but not enough.
Is there currently a way to see my folders/albums as they are on disk? Nothing fancy, but let's assume i want to view the photos from that trip i took to France in 2012? Specially, since memory fades, it would be useful to actually be able to see a list of "albums"(folders) like i can see a list of tags (which i love!).
Am i missing something and it's already possible (hope so!) or would it be possible in a near future?

Hi @gardiol
Appologize the late reply.
Thank you for reaching out and bringing up your possitive experience and pointing to your issues. I am happy that you are enjoying the smooth browsing experience like me.
Currently there is no mechanism to browse and drill down the folder structure. While I understand fully your needs and your requirement: Up to now there were more important things to complete than this feature according to my needs.
For me it is more important to have access somehow to all of my images without worring about the different media source folders or the folder structure. I also do not organize my picture in meaningful folder names, while it would be good for better search results.
Because the search honors folder names and the detail view in the single media view offers links to the single (sub) folders. So maybe it might be worth to solve your needs with the keyboard? Have a look to the search docs like path~france 2012 or with an explicit year path~france y:2012?
Other options are: Providing an PR, wait until this feature is implemented or try another gallery. For me this gallery is a spare time fun project and it depends on my free time/mood but also others inputs/work.
Maybe push me from time to time to implement it... I can give also guidance how and were to implement it by your own, if needed.
Does this answers helps you in any way?

Hi,
I have the very same usecase as @gardiol .
I think HG is great (good job @xemle 👍 ), but I can't use it due to the missing "browse my folders/albums" feature.
I highly appreciate your work on this and sharing it. Maybe you have some time to implement this @xemle . Would be nice to see this in an upcoming release... :)

Hi @Schnoogemetzger
thank you for sharing your demand.
Currently I am working on a plugin feature so that functionality can be added independently. This is a big thing and will need some time.
If you are able to provide some functionaltiy regarding your request of the folders navigation it would be really great. Pleas let me know if you need further information...
In the meanwhile you can subscribe to this ticket to receive any update and progress on it.

Hi @xemle,
thx for the reply.
Browsing the directory structure as on the disk would be sufficient. Let's say these are my folders:
/Pictures/Holiday/2005/USA
/Pictures/Holiday/2017/Iceland
/Pictures/Trips/2007/London
/Pictures/Birthdays/John_Doe
/Pictures/Screenshots
...

You would then select /Pictures as the root directory and you can then click through the albums and sub-albums. Just like you would do with a file explorer.

I think it can be much easier: create a tag with folder name at attach to all pictures in that folder, let browse/select by that tag.
No need to go the folder-tree approach, which is already present on basically any other similar tool and I think it's what set HomeGallery apart from the others, and why I like it.

@gardiol for my understanding: Is this a workaround for you at the moment or a suggestion how to solve it?

I like the discussion without me as maintainer ;-) Awesome
IMHO the feature of browsing the folder gives value for people thinking in folders to organize there memories. Since the folder is a very basic computer feature I assume that many people organize there media via that.
Also the implementation should not be too complex. All information are available on the browser. The tags view can be an inspiration for it and can be done within one weekend (for me).
Still, this project is a spare time project and there are also other topics to work on or to complete. So it drills down to the question of priority. Since I do not order my pictures by meaningful folder names this feature has a lower prio than other to me.
I am happy to provide any support if someone is willing to implement that. The dev docs could be a good starter.

I think it can be much easier: create a tag with folder name and attach it to all pictures in that folder, let then browse/select by that tag.
No need to go the folder-tree approach, which is already present on basically any other similar tool and I think it's what set HomeGallery apart from the others, and why I like it.

This solution would work for me too.
@xemle - your gallery is really the best for home use and I have tried many of them (piwigo, photoview, pigallery2, ...). As you can see the only thing that is missing is folder view (browse by folder) so please be so kind to somehow implement it, maybe by using those tags to auto-generate them from folder name structure.
Thank you in advance, you're doing a great work!

Thank you @cuks1 for your positive feedback and your vote to the folder feature. I feel proud that this one-man gallery gives value to people over other mature solutions. This is definitely a motivation to continue.
Since HomeGallery is a spare time open source project: is there anything you can offer if I would implement the folder feature? I would need support in social advertisements or improvements of the documentation? Any other help would be great, too.

Since I am not a programmer I cannot help you with the code but I can help you with the documentation and how-to's.
Regarding the social advertisements maybe I can help with some pc forums and reddit or something. Please contact me on PM so we will see how can I help. We'll figure out something 🙂

Hi @cuks1 thank you for your positive response. I can not find a public address to PM you. Would you mind to contact me via email for further support?

@cuks1 I've replied to your mail. Did you receive it? Maybe in the spam folder?

I was looking for the answer and you guess, it was in spam folder. I'll reply you via email.

@cuks1 Any progress an on the documentation so far?


Create clock.lu

Module: 00
Lab/Demo: 00
Fixes # .
Changes proposed in this pull request:

Hi @daaniakhalith, it looks like you're trying to commit the .lu export of your app from the end of lab 9. The lab directs learners to build their own app, learning how to do so along the way, and should all end up with an app that will look very similar. Could you elaborate on why you wanted this committed to the repo?


Test files are included in API files when using Kotlin Multiplatform

Example: https://github.com/chrisbanes/haze/blob/main/haze/api/api.txt (notice HazeScreenshotTest). The HazeScreenshotTest class is located in commonTest (here), so seems like something has gone awry in the compile classpath calculation.

I can confirm that also in an Android library, the test source set seems to be included in the API as well. So it looks like it's a general problem, not just when using Kotlin Multiplatform.

I think by default the source path is set to src. If a subdirectory is set by the Gradle plugin such as sourcePaths.setFrom("src/main"), then I believe anything from the test source set should be excluded.

Thanks for the quick response @tylerbwong! I found out myself that I could use sourcePaths = files(android.sourceSets.named("main").get().java.srcDirs) to only get that - but I like your workaround more :)

@tylerbwong Do you think it'll be complicated to include only the production source sets of modules by default - not the test source sets? I don't see how any user would want to leak any test sources into the API of a module - but I might be wrong :)

Added some upcoming changes in 54ce413.
Source sets should be detected automatically based on what type of module the plugin is being applied to (excluding test sources by default). The existing sourcePaths config option has been changed to additionalSourceSets for consumers to specify their own non-*main source sets.


Implement ability to define splits in metadata section of dataset card

Feature request
If you go here: https://huggingface.co/datasets/inria-soda/tabular-benchmark/tree/main you will see bunch of folders that has various CSV files. I’d like dataset viewer to show these files instead of only one dataset like it currently does. (and also people to be able to load them as splits instead of loading through data_files)
e.g GLUE has various splits on viewer but it’s too overkill to ask people to implement loading script, so it would be better to let them define these in the README file instead.
Also pinging @polinaeterna @lhoestq @adrinjalali

@merveenoyan Do you want different files to be splits or configurations?
From what you specified in Readme.md I hypothesize that you want to have 4 configs corresponding to directories: "clf_cat", "clf_num", "reg_cat", "reg_num". And inside each config you require to have as many splits as there are csv files
so if you run
load_dataset("inria-soda/tabular-benchmark", "clf_cat", split="compass")

you will generate the data only from compass.csv file.
In this case, running load_dataset("inria-soda/tabular-benchmark", "clf_cat") without split parameter will return DatasetDict object with "KDDCup09_upselling", "cat_compass", "cat_covertype", ... "road_safety" keys (which values are splits - Dataset objects)
or
do you want each file to be a separate config? Like:
load_dataset("inria-soda/tabular-benchmark", "clf_cat_compass")   # returns DatasetDict with a single "train" split

or
maybe smth completely different? :smile:
Anyway, now I have an impression that this is probably rather a matter of automatically inferring configs from repository structure rather than providing parameters in metadata yaml.

@polinaeterna I want the latter where you can think of every CSV file as a config, like MNLI from GLUE.

@merveenoyan @lhoestq I see two solutions to this case.

Parse configurations automatically from directories names. That is, if you have data structure like:

tabular-benchmark
  └─clf_cat_compass
      └─compass.csv
  └─clf_cat_cat_covertype
      └─covertype.csv
 ...
  └─reg_cat_house_sales
      └─house_sales.csv

you'll get  "clf_cat_compass", "clf_cat_cat_covertype", ... "reg_cat_house_sales" configurations that would contain only files from corresponding directories.
+ this is a requested change and needed in general and would solve other problems, see https://github.com/huggingface/datasets/issues/4578, would also help with https://github.com/huggingface/datasets/pull/5213 which I'm working on currently
+ would allow users to do just load_dataset(“inria-soda/tabular-benchmark”, “clf_cat_compass”), no data_files param required
- in this specific case it would require restructuring of the data - putting each file in a directory named as a config name (to me personally it doesn't seem to be a big deal)

More or less what we discussed before - add support for manually specifying parameters in the metadata. We can add new metadata yaml field (say, "custom_configs_info"), so that we can provide smth like:

---
...
dataset_info:
 ...
custom_configs_info:
- config_name: reg_cat_house_sales
 data_files:
 - reg_cat/house_sales.csv
- config_name: clf_cat_compass
 data_files:
 - clf_cat/compass.csv
...
---

+ Would be useful not only for tabular data and not only for data_files parameter - any packaged dataset’s viewer can be customized to use specific, non-default parameters. @merveenoyan do you maybe have any other examples/use cases in mind where you want to provide any specific parameters to the viewer?
- I'm not sure here but assume that it might require changes in interaction with the viewer on the hub side - to parse these configurations, as they not default configurations (not in BUILDER_CONFIGS list). cc @severo But probably this can be solved on the datasets side too.
Overall, I would start from implementing the first solution since it's related to what I'm doing now and is super useful for datasets in general. And then  if we agree that having more flexibility in providing parameters to the viewer is required, I can implement the second one. Let me know what you think :)

We can add new metadata yaml field (say, "custom_configs_info"), so that we can provide smth like:

Love it ! Some other ideas to name the "custom_configs_info" field: "configs", "parameters", "config_args", "configurations"

it might require changes in interaction with the viewer on the hub side - to parse these configurations, as they not default configurations (not in BUILDER_CONFIGS list)

If we update the get_dataset_config_names in inspect.py we should be fine - that's what the viewer is using

Overall, I would start from implementing the first solution since it's related to what I'm doing now and is super useful for datasets in general. And then if we agree that having more flexibility in providing parameters to the viewer is required, I can implement the second one. Let me know what you think :)

Actually I feel like the second solution includes the first use case you mentioned. If you implement the second solution, then users would just have to add a few lines of YAML and their directories would be considered configurations no ? Maybe there's no need to implement two different logics to do the same thing

is there any update on this? 🕵🏻

@merveenoyan I haven't started working on this yet, working on adding configs to packaged datasets instead: https://github.com/huggingface/datasets/pull/5213 because this both would allow you to solve your issue and is a frequently requested feature.
adding arbitrary parameters to yaml would be my next task i think!

@merveenoyan ignore my comment above, I'm switching to this task now :D

I want to be able to create folders in a model.

Addressed in #5331


Issue with custom stacking pipeline

HI,
I have created a pipeline by stacking bunch of models together. The pipeline looks like;
pipe_stacking = make_pipeline(min_max_scaler,pca,EnsembleClassifier(classifiers=[modelLogit,modelRF,modelXGB,                                                                                linear_classifier,dnnClassifier],                                                                     meta_classifier=gridGB_high))
The idea is to build a meta classifier on top of the probability scores of one of the class label from the base classifiers.
The ensemble classifier looks like;

from sklearn.base import BaseEstimator, TransformerMixin, ClassifierMixin
class EnsembleClassifier(BaseEstimator, ClassifierMixin):
    def __init__(self, classifiers=None,meta_classifier=None):
        self.classifiers=classifiers
        self.meta_classifier=meta_classifier

    def fit(self, X, y):
        return self

    def predict_proba(self, X):
        """
        Create a vector of probability score from 5 base classifiers .
        [clf1_proba, clf2_proba, clf3_proba, clf4_proba, clf5_proba]
        """
        self.prob_result = [self.classifiers[0].predict_proba(X)[0][0],
                            self.classifiers[1].predict_proba(X)[0][0],
                            self.classifiers[2].predict_proba(X)[0][0],
                            np.asarray(list( self.classifiers[3].predict_proba(X)))[0][0],
                            np.asarray(list( self.classifiers[4].predict_proba(X)))[0][0]]

        self.cols_df  = ['Logit_df','RF_df','XGB_df','TLinear_df','TDNN_df']
        self.vector = pd.DataFrame(data=[self.prob_result],columns=self.cols_df)

        # Retrieve the probability score from the meta classifier which is trained already
        prob = self.meta_classifier.predict_proba(self.vector)
        return prob

    def transform(self, X, **transform_params):
        return pd.DataFrame(self.meta_classifier.predict(X))

    def predict(self, x):
        return self.meta_classifier.fit_predict(x)

The pipeline call : pipe_stacking.predict_proba(test_data) works perfectly fine.
I am trying to use the LimeTabularExplainer on this stacking model;
explainer = `LimeTabularExplainer(data.as_matrix(),feature_names=features,class_names=class_names)

exp = explainer.explain_instance(test_data, pipe_stacking.predict_proba, num_features=153,num_samples=44)

I get this error log;
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1644-ff3b6d617477> in <module>()
----> 1 exp = explainer.explain_instance(test_pipe.iloc[32], pipe_stacking.predict_proba, num_features=153,num_samples=44)

/Users/raajaa/anaconda/lib/python2.7/site-packages/lime/lime_tabular.pyc in explain_instance(self, data_row, classifier_fn, labels, top_labels, num_features, num_samples, distance_metric, model_regressor)
    276                 scaled_data, yss, distances, label, num_features,
    277                 model_regressor=model_regressor,
--> 278                 feature_selection=self.feature_selection)
    279         return ret_exp
    280

/Users/raajaa/anaconda/lib/python2.7/site-packages/lime/lime_base.pyc in explain_instance_with_data(self, neighborhood_data, neighborhood_labels, distances, label, num_features, feature_selection, model_regressor)
    151                                                weights,
    152                                                num_features,
--> 153                                                feature_selection)
    154
    155         if model_regressor is None:

/Users/raajaa/anaconda/lib/python2.7/site-packages/lime/lime_base.pyc in feature_selection(self, data, labels, weights, num_features, method)
    100                 n_method = 'highest_weights'
    101             return self.feature_selection(data, labels, weights,
--> 102                                           num_features, n_method)
    103
    104     def explain_instance_with_data(self,

/Users/raajaa/anaconda/lib/python2.7/site-packages/lime/lime_base.pyc in feature_selection(self, data, labels, weights, num_features, method)
     73         elif method == 'highest_weights':
     74             clf = sklearn.linear_model.Ridge(alpha=0, fit_intercept=True)
---> 75             clf.fit(data, labels, sample_weight=weights)
     76             feature_weights = sorted(zip(range(data.shape[0]),
     77                                          clf.coef_ * data[0]),

/Users/raajaa/anaconda/lib/python2.7/site-packages/sklearn/linear_model/ridge.pyc in fit(self, X, y, sample_weight)
    640         self : returns an instance of self.
    641         """
--> 642         return super(Ridge, self).fit(X, y, sample_weight=sample_weight)
    643
    644

/Users/raajaa/anaconda/lib/python2.7/site-packages/sklearn/linear_model/ridge.pyc in fit(self, X, y, sample_weight)
    463     def fit(self, X, y, sample_weight=None):
    464         X, y = check_X_y(X, y, ['csr', 'csc', 'coo'], dtype=np.float64,
--> 465                          multi_output=True, y_numeric=True)
    466
    467         if ((sample_weight is not None) and

/Users/raajaa/anaconda/lib/python2.7/site-packages/sklearn/utils/validation.pyc in check_X_y(X, y, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, warn_on_dtype, estimator)
    529         y = y.astype(np.float64)
    530
--> 531     check_consistent_length(X, y)
    532
    533     return X, y

/Users/raajaa/anaconda/lib/python2.7/site-packages/sklearn/utils/validation.pyc in check_consistent_length(*arrays)
    179     if len(uniques) > 1:
    180         raise ValueError("Found input variables with inconsistent numbers of"
--> 181                          " samples: %r" % [int(l) for l in lengths])
    182
    183

ValueError: Found input variables with inconsistent numbers of samples: [44, 1]

It works only if num_samples = 1.
I am not sure what the issue is. Any direction will be greatly appreciated.
Regards,
Vijay Raajaa GS

Are you sure pipe_stacking.predict_proba(test_data) works perfectly fine? In this line:
       self.prob_result = [self.classifiers[0].predict_proba(X)[0][0],                             self.classifiers[1].predict_proba(X)[0][0],                             self.classifiers[2].predict_proba(X)[0][0],                             np.asarray(list( self.classifiers[3].predict_proba(X)))[0][0],                             np.asarray(list( self.classifiers[4].predict_proba(X)))[0][0]]
you seem to be taking the first row and first column ([0][0]) of the prediction probability for each classifier, regardless of the size of the input. That is, I think you'll always output one row in predict_proba, even if the input is 10 rows.
LimeTabular assumes the X in predict_proba can be a 2d array, not only a 1d array.

This is a use case specific implementation, wherein the pipeline is used only for prediction against a single input vector. Henceforth the size of the input is one here. X in predict_proba comes after PCA, which is a 2d array. I have used the same training input matrix for initialization & used the same input data point against lime explainer built on Random forest earlier, it did work fine. It doesnt seem to work for the custom stacking implementation.

can you share a notebook with the error?

Sure. I have shared the same to your inbox.

Thanks for the right direction. Your previous correction was perfectly right, the stacking predict_proba function was returning only the first output. I had changed it to work on an array of any given size. Since the sample in my implementation was initialized with a size 44, the ideal output from the predict_proba from my stacking implementation should be (44x2).


[changed] Don't restore scroll position on Forward

We can't reliably restore scroll position on Forward due to browser differences and incompatibilities.
Instead, we will only restore previous scroll position when user presses Back, and scroll to top on Forward.
Fixes #707.

Let's hold on with this until we have a solution for https://github.com/rackt/react-router/issues/707#issuecomment-70547686

Merged it myself so it gets to the next release.
The change is trivial, fixes buggy behavior and seems to work on my project so far..

Woohoo! Thanks @gaearon ! :+1:


chore: Remove old contract deployment flow

WIP

Benchmark results
Metrics with a significant change:

tx_sequencer_processing_time_ms (0): 9.74 (+37%)
note_trial_decrypting_time_in_ms (64): 130 (+136%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Values are compared against data from master at commit 07183200 and shown if the difference exceeds 1%.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
64 txs

l1_rollup_calldata_size_in_bytes
5,668 (-1%)
18,820
36,356

l1_rollup_calldata_gas
66,004 (-1%)
238,672
469,364

l1_rollup_execution_gas
666,484 (-1%)
953,381 (-3%)
1,336,520 (-4%)

l2_block_processing_time_in_ms
1,169 (-2%)
4,433 (-1%)
8,871 (-1%)

note_successful_decrypting_time_in_ms
198 (-1%)
541 (-3%)
998 (-3%)

note_trial_decrypting_time_in_ms
106
63.5 (-14%)
:warning: 130 (+136%)

l2_block_building_time_in_ms
11,585 (-5%)
46,360 (-6%)
92,585 (-3%)

l2_block_rollup_simulation_time_in_ms
7,033 (-5%)
28,495 (-6%)
56,890 (-3%)

l2_block_public_tx_process_time_in_ms
4,519 (-4%)
17,794 (-4%)
35,585 (-4%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
14,252 (+2%)
27,226 (+1%)

note_history_successful_decrypting_time_in_ms
1,296 (+5%)
2,491 (+4%)

note_history_trial_decrypting_time_in_ms
108 (+74%)
154 (+39%)

node_database_size_in_bytes
18,706,512 (-1%)
35,303,504 (-1%)

pxe_database_size_in_bytes
29,859
59,414

Circuits stats
Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit
circuit_simulation_time_in_ms
circuit_input_size_in_bytes
circuit_output_size_in_bytes

private-kernel-init
241 (-4%)
44,338 (-1%)
27,700 (-1%)

private-kernel-ordering
191 (-1%)
52,324 (-1%)
14,326 (-2%)

base-rollup
641 (-3%)
177,083
925 (-7%)

root-rollup
68.6 (-1%)
4,176 (-3%)
789 (-4%)

private-kernel-inner
313 (-2%)
73,197 (-1%)
27,700 (-1%)

public-kernel-app-logic
158 (-4%)
34,582 (-1%)
27,671 (-1%)

public-kernel-tail
81.4 (-1%)
34,206 (-1%)
27,671 (-1%)

merge-rollup
8.48 (-2%)
2,696 (-5%)
925 (-7%)

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
512 leaves
1024 leaves
2048 leaves
4096 leaves
32 leaves

batch_insert_into_append_only_tree_16_depth_ms
10.2 (+4%)
16.0 (-1%)
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.8 (-1%)
31.6
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.591 (+4%)
0.496 (-1%)
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
46.4 (+1%)
73.4 (+1%)
236 (+3%)
448 (+1%)
862
1,720 (-1%)
N/A

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
96.0
159
543
1,055
2,079
4,127
N/A

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.476 (+1%)
0.453 (+1%)
0.432 (+3%)
0.418
0.411
0.412 (-1%)
N/A

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
53.7 (-1%)
107
337
662
1,304 (-1%)
2,606
N/A

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
104
207
691
1,363
2,707
5,395
N/A

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.478 (-1%)
0.481
0.457
0.455
0.453
0.453
N/A

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
61.2

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
109

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.535

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes

tx_size_in_bytes
19,191 (-2%)

Transaction processing duration by data writes.

Metric
0 new note hashes
1 new note hashes

tx_pxe_processing_time_ms
2,585 (-5%)
1,340 (-3%)

Metric
0 public data writes
1 public data writes

tx_sequencer_processing_time_ms
:warning: 9.74 (+37%)
559 (-2%)

Docs Preview
Hey there! 👋 You can check your preview at https://65ebc1e2c5414f43a4d2ed35--aztec-docs-dev.netlify.app


fetch doesn't return embedded documents fields in the actual order

The order of fields in embedded documents is significant for equality matching to work. MongoDB docs say: "Equality matches on an embedded document require an exact match of the specified <value>, including the field order."
But the order of fields in embedded documents as returned by somnium.congomongo/fetch doesn't match the actual order:
(m/with-mongo conn (m/fetch :sessions :where {:embedded_test {:a "4" :b "5"}}))
;; => ({:embedded_test {:b "5", :a "4"}, :_id #object[org.bson.types.ObjectId 0x2057e5d3 "570e9b226588b6feb16d06bf"]})
(m/with-mongo conn (m/fetch :sessions :where {:embedded_test {:b "5" :a "4"}}))
;; => ()
(m/with-mongo conn (m/fetch :sessions :where {:embedded_test (array-map :a "4" :b "5")}))
;; => ({:embedded_test {:b "5", :a "4"}, :_id #object[org.bson.types.ObjectId 0x330428d4 "570e9b226588b6feb16d06bf"]})
(m/with-mongo conn (m/fetch :sessions :where {:embedded_test (array-map :b "5" :a "4")}))
;; => ()

I assume the actual order is  {:a "4" :b "5"} because the document is retrieved only when querying with the fields in that order, using both regular maps or array maps. Besides, the mongo shell also shows them in that order
db.sessions.find()
// => { "_id" : ObjectId("570e9b226588b6feb16d06bf"), "embedded_test" : { "a" : "4", "b" : "5" } }

You can query on {:embedded_test.a "4" :embedded_test.b "5"} and you can also construct a dbobject (in somnium.congomongo.coerce) which keeps fields in order.

Ok, but my point is...
(def s (m/with-mongo conn (m/fetch-one :sessions)))
s
; => {:embedded_test {:b "5", :a "4"}, :_id #object[org.bson.types.ObjectId 0x2e63034c "570e9b226588b6feb16d06bf"]}
(m/with-mongo conn (m/fetch :sessions :where s))
; => ()

I think this is unexpected at the least.
In the MongoDB shell, the equivalent example works as expected...
s = db.sessions.findOne()
// {
// 	"_id" : ObjectId("570e9b226588b6feb16d06bf"),
// 	"embedded_test" : {
// 		"a" : "4",
// 		"b" : "5"
// 	}
// }
db.sessions.find(s)
// { "_id" : ObjectId("570e9b226588b6feb16d06bf"), "embedded_test" : { "a" : "4", "b" : "5" } }

I think you're the first person to find it surprising: Clojure hash maps are unordered by design and you're getting a Clojure data structure back.

I was thinking that maybe Congomongo could return documents as array-maps with the keys ordered as in the database. This would get rid of the (for me at least!) gotcha and would be consistent with MongoDB shell behavior.
Anyway, I just wanted to raise a possible issue.
Thanks.

I'll take a look at the implications of that next week when I get back from Clojure/West.

It looks like any change to this will impact both directions in the MongoDB / Clojure coercions and have potential performance impacts. I don't have the bandwidth / time right now to set up benchmarks and do the level of analysis necessary to figure out exactly what would need to change and how to do it without impacting performance -- it's certainly a non-trivial change. Bottom line: not going to happen in the foreseeable future.


Update asciidoctorj-diagram to use asciidoctor-diagram 1.4.0

Getting the following exception in 1.3.1
asciidoctor-diagram: ERROR: Failed to generate image: PlantUML image generation failed: Bad Request java.io.IOException: Could not find GraphViz 'dot' tool
In the standard example project https://github.com/asciidoctor/asciidoctor-maven-examples/tree/master/asciidoctor-diagram-example
Even though this project makes the following statment

Asciidoctor Diagram bundles both the ditaa and PlantUML libraries and will use them to generate diagrams.

There seems to have been an issue addressed in 1.4.0 of asciidoctor-diagram for this problem
https://github.com/asciidoctor/asciidoctor-diagram/issues/71
But this is not included in 1.3.1 (latest) of asciidoctorj-diagram

Hi Hugh,
good point, we will surely also upgrade and release asciidoctorj-diagram with the next release of asciidoctorJ.
Nevertheless there are certain diagram types that require graphviz to be installed, so the new version will probably try not to locate graphviz over-eagerly, but depending on the type of diagram you will still need to install it.
Cheers
Robert

Thanks Robert,
It might be worth updating the example doc to be a bit clearer on that at the same time.
Any idea of the time frame?
Thanks for a fantastic project!
Regards,
Hugh

Hi
I also suffering from the same issue. Please release as soon as possible.

Released asciidoctorj-diagram 1.5.0

:+1:

:+1:


CircleCI 1.0 is being closed

I'm submitting a ...

[X] Bug report
[ ] Feature requestdd
[ ] Question / Support request

Have you seen https://circleci.com/sunset1-0/ ?
The CI will fail at the end of next month.

Yes, I see. I should migrate to 2.0.


Enable nvidia-optical-flow-sdk when building opencv4 with cuda

Motivation for this change
This adds and enables nvidia's optical flow sdk which causes cuda 4 to fail to build when cuda is enabled.
Things done
Patched CMakeLists.txt in opencv contrib to avoid trying to download opencv headers.

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@tbenst

Thanks for the pull-request! I got an error at the end of the build:
In file included from /build/source/modules/core/include/opencv2/core.hpp:54,
                 from /build/source/modules/core/include/opencv2/core/cuda.hpp:51,
                 from /build/source/opencv_contrib/cudaoptflow/include/opencv2/cudaoptflow.hpp:50,
                 from /build/source/opencv_contrib/cudaoptflow/src/precomp.hpp:48,
                 from /build/source/opencv_contrib/cudaoptflow/src/nvidiaOpticalFlow.cpp:7:
/build/source/modules/core/include/opencv2/core/base.hpp:320:40: error: expected constructor, destructor, or type conversion before '(' token
  320 | #define CV_Error( code, msg ) cv::error( code, msg, CV_Func, __FILE__, __LINE__ )
      |                                        ^
/build/source/opencv_contrib/cudaoptflow/src/nvidiaOpticalFlow.cpp:15:1: note: in expansion of macro 'CV_Error'
   15 | CV_Error(cv::Error::HeaderIsNull, "Nvidia Optical Flow headers not found. Make sure cmake downloads it properly");
      | ^~~~~~~~
make[2]: *** [modules/cudaoptflow/CMakeFiles/opencv_cudaoptflow.dir/build.make:110: modules/cudaoptflow/CMakeFiles/opencv_cudaoptflow.dir/src/nvidiaOpticalFlow.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:5679: modules/cudaoptflow/CMakeFiles/opencv_cudaoptflow.dir/all] Error 2
make: *** [Makefile:163: all] Error 2
builder for '/nix/store/7wqh2c7wfpgwknafi114kh382z5lsfw5-opencv-4.1.2.drv' failed with exit code 2
error: build of '/nix/store/7wqh2c7wfpgwknafi114kh382z5lsfw5-opencv-4.1.2.drv' failed

Full output here: https://gist.github.com/tbenst/7d53ace5b6533d7d1225429caddcc95c.
Also, could you please split into two commits? e.g. nvidia-optical-flow-sdk: init at 79c6cee80a2df9a196f20afd6b598a9810964c32 and opencv4: Enable nvidia-optical-flow-sdk when building with cuda

Build now succeeds. LGTM

@mdaiter @basvandijk would one of you be able to review? This fixes compiling opencv4 when cuda is enabled.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-ready-for-review-may-2019/3032/125

@veprbl I think I fixed everything you mentioned except what is in the comments. Please see my latest commit for the diff, after which I'll squash.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/122

Could someone pick this to 20.03? The build's broken there too, and this applies cleanly and works.

@averelld done. https://github.com/NixOS/nixpkgs/pull/83733


Update luxon format enum

Description
Add new LuxonDate formats so we can handle Datetime and timezones separately.
Motivation and Context
The <LocalizationProvider /> component does not support timezone tokens in the inputFormat string. So we need a way to display the Datetime and timezone tokens separately.
How Has This Been Tested?
Screenshots (if appropriate):
Types of changes

[x] Chore (a change that does not modify the application functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[ ] Documentation updates are needed, and have been made accordingly.
[ ] I have added and/or updated tests to cover my changes.
[x] All new and existing tests passed.
[ ] My changes include a database migration.

I have read the CLA Document and I hereby sign the CLA


More control for head tag

Hello. Do you plan to add more control for the head tag? I mean, that it will be nice to have some marker-placeholder in a decorator, so that a decorated page head content can be placed in exact place of it's decorator. Now as I see, a decorated page's content is placed only to the end of merged head tag.
Now I have faced the issue with applying google analytics scripts, which should be placed exactly before the closing head tag.

Done, I hope!  See #59.


[#855] feat(tez): Support Tez write OrderedPartitionedKVOutput

What changes were proposed in this pull request?
support tez write OrderedPartitionedKVOutput
Why are the changes needed?
Fix: #855
Does this PR introduce any user-facing change?
No.
How was this patch tested?
In a few days I will add test cases

Coud you add some uts for this pr?

Could you rebase  or merge the latest code of master? I add the pipeline of Tez in the latest code.

@jerqi  I closed this pr. And resubmitted this pr https://github.com/apache/incubator-uniffle/pull/930


leatherman: 1.12.3 -> 1.12.4

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/leatherman/versions.
meta.description for leatherman is: "A collection of C++ and CMake utility libraries"
meta.homepage for leatherman is: "https://github.com/puppetlabs/leatherman/"
meta.changelog for leatherman is: ""
Updates performed

Version update

To inspect upstream changes

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 1.12.4 with grep in /nix/store/dyfs0jjqc6m6sddylgsn19cxjf9w988d-leatherman-1.12.4

directory tree listing: https://gist.github.com/f02e24ab06f22d06e54120733c5e184f

du listing: https://gist.github.com/d83fb5e59667893c11e2e09b2f6efa08

Rebuild report (if merged into master) (click to expand)

15 total rebuild path(s)

4 package rebuild(s)

4 x86_64-linux rebuild(s)
4 i686-linux rebuild(s)
4 x86_64-darwin rebuild(s)
3 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
cpp-hocon
facter
leatherman
libwhereami

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/dyfs0jjqc6m6sddylgsn19cxjf9w988d-leatherman-1.12.4 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A leatherman https://github.com/r-ryantm/nixpkgs/archive/03e7d01b3f29c7a12bd4b318fa15c5f252229389.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/dyfs0jjqc6m6sddylgsn19cxjf9w988d-leatherman-1.12.4
ls -la /nix/store/dyfs0jjqc6m6sddylgsn19cxjf9w988d-leatherman-1.12.4/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

4 packages built:

cpp-hocon
facter
leatherman
libwhereami

Maintainer pings
cc @womfoo for testing.

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.
Result of nixpkgs-review pr 110003 run on x86_64-linux 1

4 packages built:

cpp-hocon
facter
leatherman
libwhereami


Submissions forwarded to<EMAIL_ADDRESS>

The email is<EMAIL_ADDRESS>

@graceatkins, would it be just as acceptable/help with judging if we had a page that showed results as they came in? Will share a screenshot shortly.

This sounds like it would be help, but I dont totally get what you mean, so
a screenshot would be helpful
On Sun, Aug 14, 2016 at 9:27 PM, Mateo Clarke<EMAIL_ADDRESS>wrote:

@graceatkins https://github.com/graceatkins, would it be just as
acceptable/help with judging if we had a page that showed results as they
came in? Will share a screenshot shortly.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/open-austin/buildabudget/issues/42#issuecomment-239717168,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATSl3QHhU2Dkv_ycCsGHVG8imRwABkkqks5qf86agaJpZM4JjsoY
.

--
Grace Atkins, Outreach Coordinator
Glasshouse Policy
https://www.glasshousepolicy.org/
T:<PHONE_NUMBER>
@OpenPolicyTX

So the idea is that we could project or display on a screen at the event everyone's results as they are submitted. Or if we don't want to project it, you could use it instead of the email dance.
Each row in the list would be a link. So now I'm working on a page layout to display the results of an individual in more detail.

Single Submission Review Page
departments listed alphabetically...

Just added comment boxes...

https://github.com/open-austin/buildabudget/commit/89eb53c7c9f3d01a9caf1bfb75af56b02848d028


Render includes in markdown exports

Describe the Bug
I am not sure if this is intended behavior, but using the API endpoint or the Export as Markdown function on a Page, will not include the content of an included page. It will come back as {{@123}}
Steps to Reproduce

Include a page with the {{@123}} functionality
Export the page as PDF to confirm the content is Included correctly
Export as Markdown to find the {{@123}} include tag

Expected Behaviour
I would have expected the Markdown to export the included Page, like pdf or plaintext do. Or at least to have the option to include it.
Screenshots or Additional Context
No response
Browser Details
No response
Exact BookStack Version
v24.10

Hi @JasonGehring,
Yeah, includes won't be handled with markdown, as no parts of the system consider includes when it comes to markdown content, so would need specific added handling.
I've updated this to an export feature request rather than a bug since it's more of a bug relative to expectations rather than a bug in existing functionality.


Break out bootstrap on installation page into separate guides

Update installation guide to point to bootstrap section
Update use cases to point to bootstrap page for Azure DevOps
Break out bootstrap guides on installation page into seprate pages

Signed-off-by: Alison Dowdney<EMAIL_ADDRESS>

Can we make the "Installation" into a section with bootstrap/upgrade/uninstall as sub-sections?

@stefanprodan
currently the end tree of all these pr's is going to look like this
/content/en/docs/guides
/install flux CLI
/bootstrap flux
/upgrade flux
/uninstall flux
Bootstrap, Upgrade and Uninstall flux all belong together because they revolve around having the CLI already installed and interacting with, or deploying Flux on a Cluster
Install the CLI feels like it belongs on it's own as a primer to getting into those topics
Maybe something like ?
/content/en/docs/guides
/install flux CLI
/Deploy and Manage Flux
/bootstrap flux
/upgrade flux
/uninstall flux
We can also look into the hierarchy after all the content is in too.
WDYT?

We moved the installation out of guides based on user feedback, I'm not keen on having all all these pages back to guides. Let's create a dedicated section please.


resume_to in Yielder discuss

I'm trying to build a scheduler based on corosensei and I've come up with a place that can be optimized, but needs corosensei support.
current logic:
main thread -> coroutine1 -> main thread -> coroutine2 -> main thread -> end
better logic:
main thread -> coroutine1 -> coroutine2 -> main thread -> end
The times of jump between coroutines has become half, but need Yielder support resume_to.
As for back_to_main, maybe I can directly get the first coroutine in scheduler and just call resume ?

The problem is that corosensei is designed as an asymmetric coroutine API. This means that a coroutine can only yield back to its caller. This is an intentional design choice, since it provides many benefits:

Backtraces can be linked to the parent context, for easier debugging.
Panics in a coroutine will continue unwinding out of the parent's context.
Support for trap recovery.

To switch to another corountine directly without a clear parent->child relationship, you need a symmetric coroutine API such as the one from Boost.context. This loses many of the benefits listed above.

In fact, I have already done so in open-coroutine-rs, but I have encountered some problems. Can you help me solve them?

I'm happy to help if you have any issues related to corosensei.

fine and thanks.


JS error on package when linting multiple open files concurrently

Steps to reproduce (preferable to be developing this package to see the codeclimate commands executed in the console):

Open multiple files to be linted (I open one css and one yaml, for example).
Close Atom.
Re-open Atom with atom -dto keep your previous context (the files of step 1 should be opened again).
Open dev tools and check the console. You should something like this:

linter-registry.js [sm]:144 [Linter] Error running Code Climate DOMException: Failed to execute 'measure' on 'Performance': The mark 'linter-codeclimate: Analysis-start' does not exist.
    at Error (native)
    at endMeasure (/opt/cgalvarez/linter-codeclimate/lib/index.js:34:17)
    at Object.<anonymous> (/opt/cgalvarez/linter-codeclimate/lib/index.js:244:9)
    at next (<anonymous>)
    at step (/opt/cgalvarez/linter-codeclimate/lib/index.js:7:273)(anonymous function) @ linter-registry.js [sm]:144

Because of long execution times of codeclimate (that's the reason behind PR #61), this should fail quite often (the affected code is not enclosed inside an if (atom.inDevMode()) { block, so it is run on every lint).
Running the affected code only in dev mode will still fire the error (although only in dev mode).
Besides, the message itself is not very detailed: linter-codeclimate: Analysis took:  6739.615000000001, so I think the easiest and most simple solution woud be using a string customized for the linted file, logging something like linter-codeclimate: Analysis for style.css took:  6739.615000000001.

I confirm that my proposed solution fixes the issue and works fine. I'll send another PR later.

I've discovered a new race condition on this same issue.
Although I fix the measure mark using the relative path to the file under linting, if you save that file multiple consecutive times (think of a typo that you quickly fix), it's possible to launch consecutive codeclimate calls for the same file, and the mark is the same for all of them.
If that mark is removed when the first result is received, then all the other calls will fail when reaching their endMeasure() call, because all of them try to clear a mark that has been already removed.
I think the most performant way to do this would be killing the previous and pending codeclimate executions by checking if the mark it's already in use, since we are only interested in the results from the last one. That would avoid executing codeclimate multiple times in parallel.


MKR regression tests: vat-move-diff

This adds a regression test supplied by MKR.

I checked for duplicates with lemmas.k. Versions in lemmas.k often have more rigorous side conditions, so there might be incompatibilities.

@denis-bogdanas I've addressed all your comments, please have another look.

@denis-bogdanas I've remove all lemmas which are not needed for this proof (but are needed for a later proof I'm working on), to make it clear which proofs the lemmas are needed for.
Since I've only deleted lemmas since your last review, I'm going to add automerge, but feel free to review again if you like.


build fio engine with meson

fio-ioengine/meson.build - included with subdir as part of the main
meson project. Builds a shared library for the io engine provided
'--with-fio=true' is set (default).
subprojects/fio.wrap -- meta-data providing meson with the means to
fetch the source code to the fio external dependency (a dependency
of the fio ioengine).
subprojects/packagefiles/fio/meson.build
provide a meson project wrapper for fio such that meson can build
it.

Signed-off-by: Jesper Wendel Devantier<EMAIL_ADDRESS>Testing
Add the following test script, e.g. fio_test.sh to the root of the source directory.
Note that this script assumbles the build directory is ./build, change accordingly.
#!/usr/bin/env bash
FLEXALLOC_SLAB_NLB="${FLEXALLOC_SLAB_NLB:-3276800}"
FLEXALLOC_NPOOLS="${FLEXALLOC_NPOOLS:-70}"
FLEXALLOC_DEV="${FLEXALLOC_DEV:-/dev/nvme0n1}"

echo "FLEXALLOC_SLAB_NLB:	${FLEXALLOC_SLAB_NLB}"
echo "FLEXALLOC_NPOOLS:		${FLEXALLOC_NPOOLS}"
echo "FLEXALLOC_DEV:		${FLEXALLOC_DEV}"

set -ex

meson build || meson --reconfigure build
meson compile -C build
build/mkfs.flexalloc --slba-nlb=${FLEXALLOC_SLAB_NLB} -p ${FLEXALLOC_NPOOLS} ${FLEXALLOC_DEV}
subprojects/fio/fio --ioengine=external:./build/fio-ioengine/libflexalloc-fio-engine.so test.fio

Save the following fio file as test.fio (see the script above)
[global]
ioengine=flexalloc
direct=1
dev_uri=/dev/nvme0n1
rw=randwrite
thread=1

[job1]
filesize=10M
nrfiles=100
poolname=testpool1

[job2]
filesize=100M
nrfiles=3
poolname=testpool2

Things to note:

flexalloc API and code is statically included into the fio engine
we can specify the engine as flexalloc in the fio job file and override it using  --ioengine=external:./build/fio-ioengine/libflexalloc-fio-engine.so which refers to the fio engine plugin we just built.
we are building fio itself because the plugin needs to be linked against a specific copy of fio - hence we are using subprojects/fio/fio instead of calling a system-wide fio binary.

Looks great! Here are some small nits:

consider using fio 3.28 instead of 3.29; might as well use the latest version
it should be possible to use the external ioengine with any fio as long as FIO_IOOPS_VERSION matches
consider updating the ioengine= line in fio-ioengine/flexalloc.fio

consider using fio 3.28 instead of 3.29; might as well use the latest version
@vincentkfu : did you mean use 3.29 instead of 3.28? Or are you making sure we use a "stable" version of fio?

consider using fio 3.28 instead of 3.29; might as well use the latest version
@vincentkfu : did you mean use 3.29 instead of 3.28? Or are you making sure we use a "stable" version of fio?

Yes that's what I meant. Sorry for the confusion.


JDK 23 support - UnsupportedOperationException: getSubject is supported only if a security manager is allowed

Driver version
12.8.1.jre11
SQL Server version
2019
Client Operating System
Rocky Linux
JAVA/JVM version
Eclipse Temurin 23
Problem description
Provide full details of the problem.
Expected behavior
Kerberos authentication should work out of the box with JDK 23 just as it did with JDK 22 and prior versions.
Actual behavior
Application fails to start.
Error message/stack trace
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: getSubject is supported only if a security manager is allowed
	at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:584)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:571)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:98)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:111)
	at com.infobip.spring.data.common.InfobipSpringDataCommonConfiguration.sqlTemplates(InfobipSpringDataCommonConfiguration.java:21)
	at com.infobip.spring.data.common.InfobipSpringDataCommonConfiguration$$SpringCGLIB$$0.CGLIB$sqlTemplates$0(<generated>)
	at com.infobip.spring.data.common.InfobipSpringDataCommonConfiguration$$SpringCGLIB$$FastClass$$1.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:348)
	at com.infobip.spring.data.common.InfobipSpringDataCommonConfiguration$$SpringCGLIB$$0.sqlTemplates(<generated>)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:146)
	... 124 common frames omitted
Caused by: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
	at java.base/javax.security.auth.Subject.getSubject(Subject.java:347)
	at com.microsoft.sqlserver.jdbc.KerbAuthentication.initAuthInit(KerbAuthentication.java:100)
	at com.microsoft.sqlserver.jdbc.KerbAuthentication.generateClientContext(KerbAuthentication.java:238)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.sendLogon(SQLServerConnection.java:6596)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.logon(SQLServerConnection.java:5411)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection$LogonCommand.doExecute(SQLServerConnection.java:5343)
	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7718)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:4393)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:3830)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:3355)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:3180)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:1963)
	at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:1259)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:137)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:360)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:202)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:461)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:550)
	... 135 common frames omitted

Any other details that can be helpful
https://jdk.java.net/23/release-notes#JDK-8296244
While suggested "fix" fixes this issue I assume KerbAuthentication class (and others) should move away from security manager. Especially since it's documented that this fix is going away in the future:

When the Security Manager is removed in a future release, the Subject.getSubject(AccessControlContext) method will be degraded further to throw UnsupportedOperationException unconditionally.

Quick follow up - it has been proposed that the workaround fix stops working already in the next version (jdk 24):
https://mail.openjdk.org/pipermail/jdk-dev/2024-September/009420.html
https://openjdk.org/jeps/486
A little bit weird to me that they break the code in a non lts version with a workaround flag and already in next non lts they remove the flag as well. It seems that between the LTS versions it'll look like a completely backwards incompatible change.

Hi, @lpandzic we are aware of the issue and we are working on it

workaround by adding jvm param: -Djava.security.manager=allow

pyspark failed due to this error as well:
Python 3.12.4 (v3.12.4:8e8a4baf65, Jun  6 2024, 17:33:18) [Clang 13.0.0 (clang-13<IP_ADDRESS>)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
24/10/25 19:16:36 WARN Utils: Your hostname, Pauls-Laptop.local resolves to a loopback address: <IP_ADDRESS>; using <IP_ADDRESS> instead (on interface en0)
24/10/25 19:16:36 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
24/10/25 19:16:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/10/25 19:16:36 WARN SparkContext: Another SparkContext is being constructed (or threw an exception in its constructor). This may indicate an error, since only one SparkContext should be running in this JVM (see SPARK-2243). The other SparkContext was created at:
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:501)
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:485)
py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
py4j.Gateway.invoke(Gateway.java:238)
py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
py4j.ClientServerConnection.run(ClientServerConnection.java:106)
java.base/java.lang.Thread.run(Thread.java:1575)
/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/python/pyspark/shell.py:74: UserWarning: Failed to initialize Spark session.
  warnings.warn("Failed to initialize Spark session.")
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/python/pyspark/shell.py", line 69, in <module>
    spark = SparkSession._create_shell_session()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/sql/session.py", line 1145, in _create_shell_session
    return SparkSession._getActiveSessionOrCreate()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/sql/session.py", line 1161, in _getActiveSessionOrCreate
    spark = builder.getOrCreate()
            ^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/sql/session.py", line 497, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/context.py", line 515, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/context.py", line 203, in __init__
    self._do_init(
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/context.py", line 296, in _do_init
    self._jsc = jsc or self._initialize_context(self._conf._jconf)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/context.py", line 421, in _initialize_context
    return self._jvm.JavaSparkContext(jconf)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/python/lib/py4j-<IP_ADDRESS>-src.zip/py4j/java_gateway.py", line 1587, in __call__
    return_value = get_return_value(
                   ^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/pyspark/python/lib/py4j-<IP_ADDRESS>-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.UnsupportedOperationException: getSubject is supported only if a security manager is allowed
	at java.base/javax.security.auth.Subject.getSubject(Subject.java:347)
	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:577)
	at org.apache.spark.util.Utils$.$anonfun$getCurrentUserName$1(Utils.scala:2416)
	at scala.Option.getOrElse(Option.scala:189)
	at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2416)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:329)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:501)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:485)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:238)
	at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
	at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
	at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
	at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
	at java.base/java.lang.Thread.run(Thread.java:1575)

@paullam328
This is happening because the SecurityManager is deprecated in the later versions of JDK. As mentioned please use the param -Djava.security.manager=allow

Not sure what to make of the comment

Hi, @lpandzic we are aware of the issue and we are working on it

combined with many later with the workaround that I linked before already. Is someone working on implementing an actual long-term fix for this issue?
But just to make sure everybody is aware of the other part of my original post since now it has been confirmed - the workaround WILL NOT WORK on jdk 24:
JEP proposed to target JDK 24: 486: Permanently Disable the Security Manager
https://mail.openjdk.org/pipermail/jdk-dev/2024-November/009601.html
Important section from the linked JEP in that mail:

There is a risk that frameworks which rely on the javax.security.auth API still use the deprecated methods in the Subject class, namely doAs and getSubject. We deprecated these methods in Java 17 and 18 because their signatures use deprecated classes of the Security Manager API. We introduced replacements for doAs and getSubject in Java 18. Since getSubject has thrown an UnsupportedOperationException since Java 23, we assume that frameworks have become aware of the deprecation and are working to adopt the replacements, e.g., HADOOP-19212.

Now to catch early the comments that 24 is a non LTS - there's another JEP coming to that release that has a pretty significant interest from wide community so this will likely be a much more sought after jdk release than 23 was:
JEP proposed to target JDK 24: 491: Synchronize Virtual Threads without Pinning
https://mail.openjdk.org/pipermail/jdk-dev/2024-November/009600.html
So it would be really great is this issue would be properly fixed before JDK 24 release which currently is planned for 18th of March, 2025.

Thank you for prompt fix!