Special case Duration display in datafusion-cli and sqllogictest

Is your feature request related to a problem or challenge?
In https://github.com/apache/arrow-datafusion/pull/6832 and https://github.com/apache/arrow-datafusion/issues/7068 , @tustvold proposes to use Duration consistently for timestamp arithmetic which makes the code simpler and consistent  across DataFusion
However, one implication of this change, at the time of this writigng, is shown https://github.com/apache/arrow-datafusion/pull/6832#discussion_r1267234606
Specifically, in DataFusion 28.0.0 the output of timestamp - timestamp is a interval and is displayed like this:

❯ select (now() - '2023-01-01'::timestamp);
+-----------------------------------------------------------+
| now() - Utf8("2023-01-01")                                |
+-----------------------------------------------------------+
| 0 years 0 mons 204 days 16 hours 16 mins 8.121077000 secs |
+-----------------------------------------------------------+
1 row in set. Query took 0.004 seconds.

However, without any additional changes after https://github.com/apache/arrow-datafusion/pull/6832 it is displayed in ISO 8601 duration format
❯ select (now() - '2023-01-01'::timestamp);
+----------------------------+
| now() - Utf8("2023-01-01") |
+----------------------------+
| P198DT72932.972880S        |
+----------------------------+
1 row in set. Query took 0.002 seconds.

I think this is problematic because the output:

Looks wildly different than 28.0.0
Looks wildly different than how intervals are expressed in queries (e.g. select now() + interval '1 year';)
Will likely cause significant confusion as the ISO duration format,  (e.g. P198DT72932.972880S), is not widely used for displaying intervals to humans
The casting logic to strings will be different (see below)

Casting logic difference will means that the following query in 28.0.0
❯ select (now() - '2023-01-01'::timestamp)::varchar || ' ago';
+----------------------------------------------------------------+
| now() - Utf8("2023-01-01") || Utf8(" ago")                     |
+----------------------------------------------------------------+
| 0 years 0 mons 204 days 16 hours 34 mins 15.407810000 secs ago |
+----------------------------------------------------------------+

Would produce this in 29.0.0:
+----------------------------------------------------------------+
| now() - Utf8("2023-01-01") || Utf8(" ago")                     |
+----------------------------------------------------------------+
| P198DT72932.972880S ago |
+----------------------------------------------------------------+

Describe the solution you'd like
I propose adding specific code for formatting DataFusion output, used in datafusion-cli, tests, and sqllogictests that formats Durations consistently with how Intervals are displayed in 28.0.0.
This change will unblock the arrow upgrade in https://github.com/apache/arrow-datafusion/pull/6832 as well as give us a single location to control formatting in DataFusion making it easier to improve the formatting of intervals / durations (or other types) in the future if we desire.
Anyone using DataFusion programmatically can either choose to use this new formatting routine, or can format Durations / Intervals to their specific need using arrow-rs or custom code (as we do in IOx, for example (source))
While this solution still suffers from the Duration --> String casting problem, I think it is ok because I don't think it is very common and there  is a workaround (to cast the duration to an interval):
❯ select (now() - '2023-01-01'::timestamp)::interval::varchar || ' ago';
+----------------------------------------------------------------+
| now() - Utf8("2023-01-01") || Utf8(" ago")                     |
+----------------------------------------------------------------+
| 0 years 0 mons 204 days 16 hours 34 mins 15.407810000 secs ago |
+----------------------------------------------------------------+

For those people who want even more control of duration printing, it would probably make sense to add a specific to_char or similar function: https://www.postgresql.org/docs/current/functions-formatting.html
Describe alternatives you've considered
Change the parsing / formattting of Durations in arrow-rs
The idea is to make PRs like https://github.com/apache/arrow-rs/pull/4557 that would change the formatting in arrow-rs
Some challenges with this approach:

It is not clear if there is an "ideal" canonical format is for durations
It would be a backwards compatible change that would likely impact other projects than DataFusion
Existing data that was serialized with Durations (e.g. JSON or CSV) may be impacted

Special case String casting / parsing
One way to avoid issues here would be to just special case the casting and parsing (Duration <==> String) in datafusion and instead of calling the arrow cast function we could call datafusion_cast which would have special handling for some type and then fall back to the arrow cast kernel
The challenges with this approach are:

It would be brittle (it would be hard to ensure that all new code called datafusion_cast rather than cast
It is not clear that it would be entirely consistent (e.g. if there is code in arrow itself that calls cast)

Additional context
No response

@waitingkuo  / @liukun4515  / @ozankabak  I wonder if you have any thoughts about this proposal (related to reducing the impact of https://github.com/apache/arrow-datafusion/issues/7068)

I think the way you suggested seems to be the most rational option. Since we do not use year and month units for durations, we can maybe ignore those parts while formatting.

I believe @tustvold  plans to handle this next week

Thinking about this a bit more, I think it would be cleanest if we added support for this to FormatOptions, this can then be plumbed through CastOptions.
https://github.com/apache/arrow-rs/pull/4581 adds the necessary upstream changes

Closed by #6832


Adding JWT secret with binary values

Our JWT secret contains binary data.  I've tried everything I can think of to send it to the Admin API as binary data.
I believe once I finally get it coming through properly, I'm getting this:
2015/09/30 18:18:15 [error] 57#0: *112 [lua] responses.lua:61: handler(): Cassandra error: Error during UNIQUE check: Cassandra error: Cassandra returned error (Invalid): String didn't validate., client: <IP_ADDRESS>, server: , request: "POST /consumers/d28eae06-3ba8-4763-c7a9-32b41694a844/jwt HTTP/1.1", host: "<IP_ADDRESS>:8001"

Our use-case is that we already have a JWT provider and would like to continue to use it.

Only text JWT secrets are supported. I don't believe Cassandra could support a select statement on a blob column anyways.

One solution would be to Base64 encode the secret.

It is currently not possible because we would need to base64 all the secrets if we do so.

That means that the JWT support won't work for any implementation where you do not have control over the secret.  For example auth0 which is a fairly large third party tool.

We could add a flag for a certain plugin enabled on an API and then Kong would systematically assume the secret is in base64 and decode it before validating the signature. Secrets would be added to consumers already base64 encoded, but there would be no way to enforce that, since credentials and plugins share no relation.
Btw, I wonder why auth0 is providing secrets with binary data? Why not stick to text?

I solved this issue because I was able to set my own secret in Auth0.  It wasn't particularly clear from their UI, but you can actually update the secret.
So you can close this if you like, as that workaround is fine.

@tecnobrat since I assume a lot of people will be finding this issue through Google, can you tell us a little bit more about the workaround and/or link to it? Thanks

I simply use the ruby JWT library to base64url_encode the secret and saved that in Auth0.
JWT.base64url_encode("secretfromkong")

Its worth mentioning that JWT.base64url_encode is NOT equivalent to the functions in the Base64 class.
tecnobrat@puma:~$ irb
irb(main):001:0> require 'jwt'
=> true
irb(main):002:0> require 'base64'
=> false
irb(main):003:0> JWT.base64url_encode("secretfromkong")
=> "c2VjcmV0ZnJvbWtvbmc"
irb(main):004:0> Base64.encode64("secretfromkong")
=> "c2VjcmV0ZnJvbWtvbmc=\n"
irb(main):005:0> Base64.urlsafe_encode64("secretfromkong")
=> "c2VjcmV0ZnJvbWtvbmc="

#838 adds support for base64 encoded secrets to next, it will be included in the next release.


missing-metadata workflow fails and posts a misleading comment if Cargo itself breaks

Example comment: https://github.com/bevyengine/bevy/pull/15344#issuecomment-2369401470
As of commit: 0ebd7fcdf4545c05dd6c1f0c4b6cf057d1f5f1a7
Example Cargo breakage:
diff --git a/crates/bevy_sprite/Cargo.toml b/crates/bevy_sprite/Cargo.toml
index 30602e07c..aed3e65b6 100644
--- a/crates/bevy_sprite/Cargo.toml
+++ b/crates/bevy_sprite/Cargo.toml
@@ -10,7 +10,7 @@ keywords = ["bevy"]

 [features]
 default = ["bevy_sprite_picking_backend"]
-bevy_sprite_picking_backend = ["bevy_picking", "bevy_window"]
+bevy_sprite_picking_backend = ["dep:bevy_picking", "dep:bevy_window"]
 serialize = ["dep:serde"]
 webgl = []
 webgpu = []

Output of breakage:
error: failed to select a version for `bevy_sprite`.
    ... required by package `bevy_internal v0.15.0-dev (/home/ltdk/code/fork/bevy/crates/bevy_internal)`
    ... which satisfies path dependency `bevy_internal` (locked to 0.15.0-dev) of package `bevy_dylib v0.15.0-dev (/home/ltdk/code/fork/bevy/crates/bevy_dylib)`
versions that meet the requirements `^0.15.0-dev` (locked to 0.15.0-dev) are: 0.15.0-dev

the package `bevy_internal` depends on `bevy_sprite`, with features: `bevy_picking` but `bevy_sprite` does not have these features.
 It has an optional dependency with that name, but that dependency uses the "dep:" syntax in the features table, so it does not have an implicit feature with that name.

failed to select a version for `bevy_sprite` which could resolve this conflict

This particular breakage was just a silly mistake of mine, thinking something was broken when it wasn't. But in general, the result is the same: if cargo breaks in general, it can fall back to displaying a comment to do some other random unrelated action.
A few recommendations:

Have a fail-fast test for dependency errors so things like this don't happen. cargo metadata --format-version 1 seems to be an option here, but you probably want --all-features as well just to verify that nothing is broken? Probably?
These comments should be updated to display the specific flow that failed, so that if there's any funny business, you have a quick link to the action logs to see the specific error.

having a metadata-check job would be a little useful to help the developer be in the right mindset while looking at what went wrong, and a quick green checkmark is a bit better than "see if it stays orange for a while"


Documentation: Add more Examples

This seems to be a really nice, comprehensive library, with lots of great features. As a non-Go programmer, it would have helped me to have had more examples to show off a lot of the varied support the library has. We all leave documentation to the last bit, but in a way, there's nothing worse than when someone's done a lot of work and a good job but not being able to see the final product clearly as a user programmer!
I also see that there are some in the Examples folder, but I didn't find this via the Go package documentation. It's an obvious place to put it, but that page is the first place I have been looking, and generally would be, so it would help to include a link on that page to the examples folder in the repo.

I agree. While this is a great library, there is nothing worse, even for an experienced programmer, to find one that has insufficient documentation. It wastes far too much developer time to scour through the API simply to infer how to use the library. For a product that intends to be production grade, the documentation is as important as the code itself.

Quite agree. please start with a low-hanging fruit -- how to use (sub) Commands.
I did find an example at
https://github.com/jessevdk/go-flags/blob/master/examples/add.go
But, take git for example, all of the add, commit, checkout, etc all have their own options, apart from the global options. How to do that with go-flags is still unclear to me.
Please complete the https://github.com/jessevdk/go-flags/blob/master/examples/add.go example with local options. Thx!


Add socket activation support (systemd) to bluealsa

This adds basic socket activation support to bluealsa. It allows bluealsa to be started on demand by bluealsa-aplay in systemd environment.
I did this to solve the racing problem between bluealsa and bluealsa-aplay during boot (socket not present yet when bluealsa-aplay is started).

Thanks for this PS, however, right now I'm in the middle of a BIG refactoring of BlueALSA internals, which finally should result in providing D-Bus interface instead of UNIX socket :)
The concept is simple:
Start bluealsa service during system boot and forget about it.

Ctl interface is now gone. Now BlueALSA uses D-Bus which should separate bluealsa server and bluealsa-aplay. You can use D-Bus signals for starting bluealsa-aplay.


Storybook custom name renders poorly when string is too long

The STORYBOOK in the upper left corner can be customized through addon-options. When the string overflows, it goes onto a new line and throws off the layout.
Perhaps it should just have ellipses?

I believe it was normally wrapping before, will check what happened

I believe it's #1769 that broke it


Check version of Apple clang correctly

The __clang_major__ macro provides a marketing version which is not required to be consistent between vendors[1]. And indeed, Apple uses a versioning scheme for its Clang releases that is unrelated to that of the llvm.org releases on which they are based. The correspondences between the versions can however be determined[2].
So, checking whether __apple_build_version__ is defined is necessary to determine whether an Apple version of Clang is being used, so that the correct version check can be done. Checking the value of that macro is simpler for the purposes of this code than checking (a different value of) __clang_major__, because distinguishing between the relevant versions of Apple Clang would also require checking __clang_minor__ and __clang_patchlevel__.
Closes: #2260

Thanks! I fix the format and rebased so that the AUTHORS check also works automatically...


Support sonata-project/media-bundle ^3.0

@ruscon Are the tests green locally?

➜  CoopTilleulsCKEditorSonataMediaBundle git:(master) phpunit
PHPUnit 4.3.4 by Sebastian Bergmann.

Configuration read from /Users/ruscon/projects/CoopTilleulsCKEditorSonataMediaBundle/phpunit.xml.dist

.

Time: 113 ms, Memory: 3.50Mb

OK (1 test, 7 assertions)

tag it plz after

We need to fix Travis first.

moved https://github.com/coopTilleuls/CoopTilleulsCKEditorSonataMediaBundle/pull/22


Fix bug in normalizeSortable to make sure key is either function or s…

The following commit "Support multiple state sorting" caused a regression on the column sort in normalizeSortable, the root cause is when sortable is object and 'key' property is a function, but function return is an array instead of string or function, it leads to normalizeOrderBy throwing exception later on, which has assumption the key must be string or function. Fix it by checking the function return, if it is an array, pick the first item and get its key property value.
The regression comes from this commit.
https://github.com/Microsoft/projection-grid/pull/54/commits/aca7ba52bcbdba883e3660df83fdfdcca053135f?diff=unified#diff-65444f7ad582109ef0b9b69757a875e6R131

Coverage decreased (-0.04%) to 90.272% when pulling 8e5997b707284d72b51e7a824dc979659b1b6357 on K05:0.1.0-dev into cc5cc0b6e2f5fe72f5f3b474e6bb93d92157f5a5 on Microsoft:0.1.0-dev.


Updates to the donation contract

Added a lot of stuff:

ERC20 token support
donation/distribution flow separation (that way donators don't have to pay splitting fees)
a way to remove contributors in the event their address is hacked
made it so that once we have 5+ contributors, only half need to sign proposals

I will discard due to the use of splits


Glorfindel: Blacklist test\Wx\W360

Glorfindel requests the blacklist of the following keyword(s):

test\Wx\W360 - MS search

test\Wx\W360 has been seen in 0 true positives, 0 false positives, and 0 NAAs.

7 TPs, actually


Rust Embedding/Binding

How difficult would it be to bind/embed this in Rust?

I haven't looked into it, so I can't give a good answer.  Probably some things would be easier to port than others.  Can we do static reflection on type structure in Rust?  That would allow us to avoid some of the macro meta-programming gymnastics that we have to do for binding (and structured storage in files/shm, serialization, compression, ...) in C++.
It's probably a worthwhile thing to do, since Rust covers use-cases similar to C++ and can similarly make use of type structure in the way that we do for our various uses for hobbes.

That's good to hear, we use a lot of Rust here at KloudTrader and Hobbes looks really interesting.

Thanks.  Your product looks interesting as well.
Although I can't comment on trading methods, I can say that we do have some large-scale production uses for hobbes.  You might find the structured logging feature useful, and the format we have for data structure persistence in files/shm (where "native" data structures can be persisted and viewed equally from C++, hobbes, python -- maybe a good start with Rust would be to add it to that list).  I haven't written much about it, but we have a custom statistical compression method in here (hobbes/include/cfregion.H) that builds on top of that idea to efficiently write very large structured datasets, which also could be ported to Rust (it's basically arithmetic coding + probabilities-as-types, or an extension of Curry-Howard to probabilities).

@kthielen on an unrelated note, I have been trying to compile it to WebAssembly. Very difficult (the LLVM bit especially).

@kthielen might want to watch this: https://github.com/kloudtrader-github/Susie

Thanks, sounds interesting but it looks like there’s no code there (yet?).
I made a workalike q interpreter once and Kx threatened to sue me personally, the company I worked for, and our clients.  You might want to be careful of those guys, they’re very spiteful and have an inflated sense of importance.


Update build workflow to use yarn

Description
Updated build workflow to use yarn over npm.
Related Issue
CEXT-2432
Motivation and Context
Adding common peer dependencies will introduce challenges managing peer dependencies in the long run. To better handle this we should be locking dependencies where appropriate. Internally the aio cli uses yarn for plugin installation. By leveraging yarn in our builds we can move towards more consistency locally, in CI, or via aio plugin:install. Plus we already have a yarn.lock in the project.
How Has This Been Tested?

yarn install
yarn lint
yarn test

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] I have signed the Adobe Open Source CLA.
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.

There is what appears to be an old Jenkinsfile in this project found when replacing npm > yarn. If we do not use Jenkins for CICD anymore this should get removed?
https://github.com/adobe/aio-cli-plugin-api-mesh/blob/49398bbee179ac3e7f3cbac99c6a59b88a818be7/Jenkinsfile#L36

There is what appears to be an old Jenkinsfile in this project found when replacing npm > yarn. If we do not use Jenkins for CICD anymore this should get removed?
https://github.com/adobe/aio-cli-plugin-api-mesh/blob/49398bbee179ac3e7f3cbac99c6a59b88a818be7/Jenkinsfile#L36

I think so. When we migrated this project from git.corp.adobe.com to github, the jenkinsfile stayed and we can remove it since gitHub actions doesn't use this

There is what appears to be an old Jenkinsfile in this project found when replacing npm > yarn. If we do not use Jenkins for CICD anymore this should get removed?
https://github.com/adobe/aio-cli-plugin-api-mesh/blob/49398bbee179ac3e7f3cbac99c6a59b88a818be7/Jenkinsfile#L36

I think so. When we migrated this project from git.corp.adobe.com to github, the jenkinsfile stayed and we can remove it since gitHub actions doesn't use this

Thank you. Done.


4D5307F9 - Project Gotham Racing 4

Marketplace
Tested on https://github.com/xenia-canary/xenia-canary/commit/a577a4ae8bd4955587202d27bf2d38ea0c2aa4f9
Issues:
Game runs without many issues, however you cannot progress past the title screen unless you already have a user profile with game data saved.  I had to use a build over a year out-of-date to create the save before I could test with an up-to-date one.  Once you see the message confirming you've signed in, the audio breaks entirely and is replaced with a constant buzzing tone.  The game is reasonably playable though slowdowns are frequent, seemingly dependent on what track you're on and how many opponents you're racing against.  Aside from this, the game can be reasonably played to completion without a hitch.
Log:
xenia.zip
Screenshot(s):

Labels:
state-gameplay

Loading and saving work completely fine, even restarted xenia and it works flawlessly now

New Xenia User
Xenia Canary 952526a78 (Sep 28 build), Xenia Manager v1.14.0
Ryzen 5 3600
Radeon 6700 (AMD Adrenaline 24.8.1)
16GB RAM
Xenia and game files installed on traditional HDD (Not SSD)
Win11

Can create new profile, navigate menus, start and complete races in Single Player.

No music during races, as mentioned by others. Other sounds including menu music seem fine to me as a first time player.

D3D12: has regular stutter (hitches) during pre-race countdown, racing and replays, It is playable in non-demanding races but becomes less playable as difficulty increases. Graphics driver overlay shows high frame rate and no CPU, GPU or RAM utilization issues.

D3D12: setting the Draw Resolution Scale >1 causes multiple issues including flicker (not playable).

Vulkan: menus, missing/flickering textures, lag in menu changes, noticeably slower than D3D12

Vulkan: causes texture issues in menu and races, a lot of the textures don't load or are overwritten, objects appear black or white, no stutter though!

Happy to supply as many xenia log files as required, just let me know which scenarios you want me to test.
Thanks!

Xenia Canary ea762e9f1 (Oct 13 build), Xenia Manager v1.15.1
Ryzen 5 3600
Radeon 6700 (AMD Adrenaline 24.9.1)
16GB RAM
Xenia and game files installed on traditional HDD (Not SSD)
Win11
Created new player profile via xenia canary
Performance is largely the same as previous version.

Stuttering in D3D12 may be slightly better in the 0.5 hr I've played so far with this version.
More lag in navigating menu's when Vulkan is used.

Happy to provide xenia.log files if you'd like a specific scenario tested.
Thanks devs

Build: canary_experimental@fbad66e5b on Nov  3 2024
AMD Ryzen 7 5800X3D 8-Core Processor
Nvidia 4070 Super (566.14)
32Gb RAM
Installed on M.2 ssd
Win 10 22H2
Game runs fine, even i have music but some cities are laking some kind of lighting. I have tried differnet Nvidia drivers, default settings, everytime same result. My guess after searching on the internet and several tests it is something with Nvidia RTX cards.
Example NYC:

But Macau looks fine:

Build c06b1f6_canary_experimental Nov 24 2024
Asus Flow X16, Windows 11 23H2, Intel i9-13900H, Nvidia 4070 mobile (Nvidia studio drivers 566.14), 64GB RAM, 4TB SSD
Seeing the lighting issues here too. Hard to describe, it's like watching a movie with an obvious "nighttime"dark filter used during daylight-- just a general dullness. Just started it up and made a profile and in the first race it was obvious something is off, like it's confused by an HDR setting or something. (my system supports HDR, but HDR is confirmed off).
Otherwise game runs well and pretty much a solid 30fps, some minor stuttering but likely shaders?

Another Nvidia RTX card. Old Nvidia cards does not seem to have this problem.
To attach files just copy and paste or drop in the message.

Ah, thanks, turns out my overzealous adblocking was keeping the upload script from actually working.

xenia pgr4 log 112524.zip

Tested on: https://github.com/xenia-canary/xenia-canary/commit/11f14e84888f8879efb841a4b90f7bf7940ba500
To avoid Nvidia issue use driver 528.49 or older. If old driver is not an option render_target_path_d3d12 = "rov" will fix this at the cost of reduced performance. Tested on 3070, afaik it applies to everything since Turing.
528.49 vs 531.18

ROV:

backwards-compatibility-unsupported | gpu-driver-issues-nvidia


Docx Attachment showing blank in gmail preview

Documents generated using this library can only be opened using Microsoft Word.
See README.md

Altchunks were not supported by Microsoft Word for Mac 2008 and are not supported by LibreOffice and Google Docs.


Need help using with Vue.js

Hi everyone,
I don't know if I allow to post question here. Anyone here using Vue.js? I'm trying to create a components with Media Element but I got this error "MediaElement is not a constructor". Here is my file
<template>
  <video width="320" height="240">
    <source type="video/mp4" src="video.mp4" />
  </video>
</template>

<script>
let MediaElementPlayer
if (process.browser) {
  MediaElementPlayer= require('mediaelement')
}

export default {
  mounted () {
    let player = new MediaElementPlayer('video', {
      pluginPath: 'https://cdnjs.com/libraries/mediaelement/',
      success: function (mediaElement, originalNode, instance) {

      }
    })
    player.on('play', () => {
      this.$emit('play')
    })
  }
}
</script>

any idea? By the way: I'm using Nuxt.js. I did try to require something like require('mediaelement/full')

Depends if which version you are using but as far as I know you don’t need to use the MediaElementPlayer = require(...) since that might be overriding the MediaElementPlayer object

@rafa8626 yes that is the problem. Thanks for your help 👍

You are welcome. Please star the project to support it

@rafa8626 yes that is the problem. Thanks for your help

How you fix this problem?

Could we get an example of how to implement this in vue please  :)

I switch to another player. Here is how I used it.
https://codesandbox.io/s/j3v30nw7pw

https://github.com/a526672351/vue-mediaelement


3 An error occurred while installing geocoder (1.8.0), and Bundler cannot continue

RUN bundle install
#0 403.3 NoMethodError: undefined method `[]' for nil:NilClass
#0 403.3 An error occurred while installing geocoder (1.8.0), and Bundler cannot
#0 403.3 continue.
#0 403.3
#0 403.3 In Gemfile:
#0 403.3   geocoder
failed to solve: executor failed running [/bin/sh -c bundle install]: exit code: 5
Environment info

Geocoder version: 1.8.0
Rails version: 5.1.4
Ruby 2.4.1

I'd need a little more context to troubleshoot this, but I suspect it's not a Geocoder issue. What's your full Gemfile?

Closing due to inactivity.


PDF file is displayed incorrectly

The file:
test.pdf
Configuration:

Google Chrome (latest)
Tested on Mac Os X High Sierra, Ubuntu Linux 17.10
PDF.js version: 1.10.88, but also reproduced if uploaded to pdf.js demo page (https://mozilla.github.io/pdf.js/web/viewer.html)

Steps to reproduce the problem:

upload attached file to the demo page
please see that all pages are grayed, and almost all pictures are gone (look at the first two pages, and disregard last two, they are actually incompleted)

To see how the file should be displayed, you can just open it in Adobe Reader or pure Google Chrome w/o pdf.js.
How it should be displayed:

How it actually looks:

Thanks!

I'm having issues with gray overlay on some documents (screenshot: http://prntscr.com/nbcl9x).
I understand this can be specific to that document file itself, because other documents are showing without problems. I assume that we have something in our PDF files that is not being parsed correctly by PDF.js
The document opens in Chrome normally when it's viewed with browser's native viewer (screenshot: http://prntscr.com/nbcm6n)
I don't have any errors in browser console log.


Find / copy support?

Hi, will these features be added at some point? Or am I missing something obvious to enable them?

Hi, could you explain better what you mean?

Hi, thanks for the response.
When I view the logs for a deployment in OpenLens, I can select some text from the log, but I can't copy it to the clipboard for whatever reason. The proprietary Lens log viewer (while not multi-pod) lets you select and copy, which is quite handy when I'm trying to send a backtrace to a coworker or capture it on a Jira ticket or whatever.
That's a bummer that OpenLens doesn't let you open a little Cmd+F modal inside the log viewer, but I agree that searching for something specific probably warrants dropping to the command line anyway.

Hi @jdolan-chwy,
for select and copy I don't understand what you mean, I can select and copy text (with Ctrl+C or Cmd+C) on terminal.
Maybe depend on which terminal you are using, check your Lens/OpenLens preferences.

Weird! I had to check, and then uncheck, the setting for "Copy on select and paste on right click" and now I can copy from the logs with Cmd-C as you'd expect. So strange!
Fwiw, I'm using OpenLens 6.5.2 on macOS with the zsh shell, and the default terminal emulation in Lens. 🤷♂
I'll go ahead and close this bug. Thank you for your replies!


Allow setting rate while buffering in ios

Currently trying to set rate right after calling play is a noop if track is being buffered and has not started playing yet, this pr fixes that, so code like that would work correctly:
await TrackPlayer.play()
TrackPlayer.setRate(someRate)

Thanks for this @biomancer :) I agree, that should be the expected functionality.


Observe Pebble Ready in LogForward

Issue

This PR fixes #358
Merge this after:

approving this one: https://github.com/canonical/cos-lib/pull/22
updating cos-lib

Integration test maybe are failing because of this issue: https://github.com/canonical/pebble/issues/385


School admin viewing student page

Feature
School admins are actually supposed to have all the view-based access that teachers have. As such, the previous restrictions around viewing students of administered teachers are being removed.
This requires additional work around access and permissions, which are expected to be accompanying this PR on the server side.
Implementation
Most changes are opening up access for the school admin, specifically around breadcrumbs used for the encompassing Vue page. Other changes are:

Allowing loading progress to be array based when multiple users are fetched
Adding back-button functionality across the pages that the school admin can view

Remaining clean up

[x] Consolidate the full template to leverage the partial template

Seems like a typo, supposed to be ..and not classroom without an Id?

@differentmatt You may be viewing an older commit in this PR? That was a typo that is now this:
https://github.com/codecombat/codecombat/pull/5212/files#diff-858eba9e7e6d0b085f452157ce56613cR100


How to use it with plain javascript application?

Hi, how can i use the module with an application that is built using plain vanilla application?

<EMAIL_ADDRESS>and follow example from codepen


Add Separator at position

Hello,
I read from the doc that it is possible to have a separator for each row:
table.render do |renderer|
  renderer.border.separator = :each_row
end

Is it possible to be able to specify where the separators apply.
For instance, being able to draw something like:
+---------------+----------+-----+
|white_name     |created_at|count|
+---------------+----------+-----+
|mommyville     |01/2016   |123  |
|mommyville     |02/2016   |17   |
|--------------------------------|
|alike          |01/2016   |38   |
|alike          |02/2016   |106  |
|--------------------------------|
|stephanefeugere|02/2016   |65   |
|--------------------------------|
|whysoserious   |02/2016   |126  |
|--------------------------------|
|hallcouture    |02/2016   |25   |
|--------------------------------|
|reinh          |02/2016   |3    |
|reinh          |03/2016   |1    |
+---------------+----------+-----+

This could be really useful also to display totals and/or subtotals rows for a table.
Cheers. TTY::Table is SO COOL.

Hi Olivier,
Cheers for using the library. It still needs a lot of improvements.
Regarding your issue, I agree it would be better to provide more flexibility for separating the records. Ideally, some way to provide conditional logic. Do you have any propositions?

Hello @piotrmurach ,
In my views there are two possible ways:

A simple one.
array = [["A", 1],["A", 2]]
array.push TTY::Table.separator
array.concat [["B", 1]]
TTY::Table.new ["letter", "number"], array

A Proc that can be passed to TTY::Table as group_by method.

     array = [["A", 1],["A", 2], ["B",1]]
     TTY::Table.new ["letter", "number"], array, :group_by => ->(entry){ entry.first }  # Group by letter

These two could also work together.  The first one is more useful when creating tables where you want to place separator "manually". For instance, draw a line before some kind of total at the bottom of a table. The second one is better when there's a simple rule to draw all separators.
That does not seem to add any weight to the TTY::Table interface. If you don't need it, don't use it (tm). :)
What do you think?

Thanks for your feedback, it's really helpful.
As a general observation on this library and the future development, I'm wanting to keep the data and actual rendering/representation parts separate. What that means is that I wouldn't necessairly add :separator directly to the table data but rather instruct the renderer on how to display it. Thus you would probably handle the data on matching condtion probabably akin to:
table.render separate_when: -> (row, col) {
  table[row][0] != table[row-1][0]
}

One of the things I want this library to support is streaming of data. In which case, knowing in advance how to insert separator is impossible, you can only do it on-the-fly kind of thing. You can easily imagine a stream of tweets pouring into terminal with a watch task each getting separated by a border as they come. Having solved that problem it would lend itself very well to solving fixed size tabular data.

I agree that having data separated from rendering is the best choice. Another thing in my opinion that would be real awesome is that the decision of adding a separator would not depend on the table data, but the original data the table was built with.
One good example of this is ActiveRecord records. While you may want to print only certain attributes of the record, the decision on how to group them would depend on attributes of the record not present in the table.
I don't know how complex the TTY::Table API will look if we add all these constraints and if people really need it. (Just like I don't personally know what could be the use for data streaming :)) but I'm looking forward to talking about it. As a matter of fact I've build pretty much all my app admin interface on TTY::Table and TTY::Prompt so I'm a big user. :)
To give you perspective, there are two reasons to that. One, it's faster than doing an admin web page. Two, I'm entertaining the idea of adding to my app a bot interface (a slack bot, irc bot, whatever bot) and I think TTY::Prompt/Table is a great proxy to the use case.

A big user - fantastic! I'm glad the tty components are helping you build terminal interface. I sense a future collaboration on this front if you are interested as I have some plans that may be exciting to yourself. I probably reserve some time for the idea to fully develop in my brain before I share it with yourself.
Regarding streaming of data, I suggest having a look at this issue where I have put my initial thoughts. I haven't had the need for this feature hence I have never implemented this functionality but I can totally see the value of it and plan to add it in future releases.

Hello @piotrmurach, do you have an example of usage for streaming tables?

I'm also very interested in this feature.

I'm also very interested in this feature.

I've been recently really focused on tty-prompt and providing good Windows support. There are still few features to implement such as autocomplete that people were asking for a long time. Unfortunately I do this in my spare time and would love to attend immediately to all issues but I simply cannot. This library needs more love and I will provide it but I doubt I will have time this month, just trying to be realistic. Having said that, if I receive PR for this feature I promise to review it quickly.

I'd love to contribute back to this project. If I have time, I'll submit a PR.

This is implemented in the latest v0.11.0 release. See separator section


luci-lib-httpclient: Comply with RFC 2616 standard for Host header

Maintainer: @jow-
Compile tested: ar71xx, GL-iNet AR750, OpenWrt master
Run tested: ar71xx, GL-iNet AR750, OpenWrt master
Description: According to the RFC standard a Host header should always contain the host and the port. Also an IPv6 address could specify an interface (like %eth0). This commit adds the missing port to the Host header and ensures that IPv6 addresses are stripped of their interface before turned into a Host header.
Signed-off-by: hwinwuzhere<EMAIL_ADDRESS>

@hwinwuzhere - thanks again for your contribution. After thinking about it for a while I decided to fix the issue in a more complete manner in commit https://github.com/openwrt/luci/commit/a54baf7be0d293b50c6b2754c11d61e6e2ad82af


fix: ensure dropdowns automatically close upon selection

Types of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING and other relevant documents.

Description

I fixed the bug where the menus don't automatically close when something is selected

Looks good, only noticed bug on /c-path menu where you can filter for latest and oldest.


python3Packages.pwntools: 4.3.1 -> 4.5.0

Signed-off-by: Pamplemousse<EMAIL_ADDRESS>Motivation for this change
Be up-to-date!
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 123055 at 796fba9b run on x86_64-linux 1

4 packages built successfully:

pwntools (python38Packages.pwntools)
python38Packages.colored-traceback
python39Packages.colored-traceback
python39Packages.pwntools

1 suggestion:

warning: python-include-tests
Consider adding a checkPhase for tests, or if not feasible, pythonImportsCheck.
Near pkgs/development/python-modules/pwntools/default.nix:87:0:
   |
87 |     description = "CTF framework and exploit development library";
   | ^


docs: Add log provider to README.md

Getting to know that this module can only be used to preview those metrics in development environment can involve people to use this great module.
I also added a small note about what this module does (related to #34)

Thank you @stafyniaksacha


php54-apache-imagemagick

ImageMagick is not visible in phpinfo ？

Hi @T-baby, I'm sorry but the Docker images we provide in this repo are not meant to be used by someone outside our organization. We don't have a BC promise (yet at least) and I can't promise you that we won't break everything in the near future.
I posted the issue on our Slack to find a solution.

https://github.com/Profideo/docker-images/blob/master/README.md


Include entity-less "scans.json" into an example of inheritance principle

Clarifies #789

I guess I misunderstood the previous discussion. I thought we did not need to single out scans.json?
This seems fine, though.

it is just an example to make it clearer - nothing is singled out IMHO.

This seems fine, though.

please bless it then with an approval ;-)

added some semantic linebreak

Added the exclude-from-changelog label because this doesn't affect the semantic meaning of the spec. For the sake of readable changelogs, I'm trying to be pretty stingy about what changes make it there.
Please let me know if you disagree.


starting REPL with a specific settings from project.cljs

Maybe I've missed it, but is there any way to set the repl to run an already set environment from project.cljs?
i have prod, test, and dev. once i run repl in proto-repl, I want to be able to tell it to run prod for example

You can do that in the Proto REPL settings. Try changing it to with-profile +test repl :headless The plus is needed so that the regular repl settings will still be available. If you need more information you can also check out the leiningen documentation.

Thanks
On Tue, Jul 12, 2016, 02:44 Jason Gilman<EMAIL_ADDRESS>wrote:

You can do that in the Proto REPL settings. Try changing it to with-profile
+test repl :headless The plus is needed so that the regular repl settings
will still be available. If you need more information you can also check
out the leiningen documentation.
[image: settings_ __users_jason_work_github_workspace_proto-repl-charts]
https://cloud.githubusercontent.com/assets/192983/16750281/afe235c4-479e-11e6-9225-b122f8c89e68.png
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/jasongilman/proto-repl/issues/119#issuecomment-231898275,
or mute the thread
https://github.com/notifications/unsubscribe/AA6MG58ll_RyRDiJMnFwecvwrVoJ38Xkks5qUtVCgaJpZM4JIxgb
.

I was referring more to a shortcut to a specific run
and not to replace the defaults


⚠ ClamAV has degraded performance

In 09ac882, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 215 ms

Resolved: ClamAV performance has improved in 7fa45f0.


Add sensors pack example for STM32

Is it possible to add one example of sensor (FXLS8974CF) pack for STM32 EVB. That way we demonstrate real benefit of having sensors interfaces and open-CMSIS pack defined that can work with MCUs (NXP and non-NXP) having Arduino UNO support. It would be a real good demonstration showcasing actual benefit of this effort.

Sure. We are working on a board layer for B-U585I-IOT02A Discovery Kit (STM32U5) with Arduino UNO support that should work with sensor shields too.

@RobertRostohar, i will be very interesting to see STM32U5 board layer and an working example with FXLS8974CF shield board. This will help us understand how Arduino UNO pin mappings and CMSIS GPIO handled in STM32.


inference

when I run the generate_beam for the caption,there are many space in the caption，Do you know why, thank you.

Have you cheacked your training data? Your training captions may contain extra spaces which leads to this.

there is not space in train data.And when inference, it's just need the clip_embedding tensor,not need tha caption.isn't it?

I find that not all of your training caption end with the '.', since the end token for beamsearch is '.', thus the model may not know when to end inference but keeps predcting the 'space' till reach the max inference length. Yes, there's no need to input the caption when inference, i mean the training data will influence your model's ability when inference. Maybe you can process the training caption to all end with '.' and re-training your model to have a try?

I find that not all of your training caption end with the '.', since the end token for beamsearch is '.', thus the model may not know when to end inference but keeps predcting the 'space' till reach the max inference length. Yes, there's no need to input the caption when inference, i mean the training data will influence your model's ability when inference. Maybe you can process the training caption to all end with '.' and re-training your model to have a try?

thank you. I will try it.

I got the same situation, may I ask that if this problem is solved through "process the training caption to all end with '.'" ?

I really want to know how you generate this caption, I am new to this direction, do not know how you run the result, I follow the readme run, it seems that I can not evaluate the model and predict the caption

I really want to know how you generate this caption, I am new to this direction, do not know how you run the result, I follow the readme run, it seems that I can not evaluate the model and predict the caption

https://github.com/thandal/CLIP_prefix_caption


[deploy] Checking Maven Central rules fails for submodules not producing jar archives

The current implementation of the AbstractMavenDeployer#checkMavenCentralRules seems not to be prepared for a multi-module Maven project containing modules that don't produce jar archives. For example, verification fails on a submodule with <packaging>pom</packaging> with a following message:
[nexus2] submodule-with-pom-packaging-0.0.1-sources.jar is missing
(...)
[ERROR] Failed to execute goal org.jreleaser:jreleaser-maven-plugin:1.3.1:full-release (default-cli) on project multi-module-project: Execution default-cli of goal org.jreleaser:jreleaser-maven-plugin:1.3.1:full-release failed: Rules for publishing to Maven Central were not met
(...)
Caused by: org.jreleaser.model.JReleaserException: Rules for publishing to Maven Central were not met
    at org.jreleaser.sdk.commons.AbstractMavenDeployer.collectDeployables (AbstractMavenDeployer.java:116)
    at org.jreleaser.sdk.nexus2.Nexus2MavenDeployer.deploy (Nexus2MavenDeployer.java:54)
    at org.jreleaser.engine.deploy.maven.ProjectMavenDeployer.deploy (ProjectMavenDeployer.java:55)
    at org.jreleaser.engine.deploy.maven.MavenDeployers.deploy (MavenDeployers.java:124)
    at org.jreleaser.engine.deploy.maven.MavenDeployers.deploy (MavenDeployers.java:111)

An example project structure causing such issues could be observed here.
Probably, the Deployable class  could be extended with a packaging type info, that could be then used to decide whether or not to expect sources and Javadoc jar archives.

Oof, absolutely. Staged artifacts with <packaging>pom</packaging> (such as BOMs and parent poms) will case rules to fail.

Deployable could add a packaging property containing the value of <packaging>. This means parsing XML will be a must. Recall that <packaging> may not be present, in which case it's default value is jar.
Simple XML parsing should be performed otherwise we'd have to resolve parent poms that may also be staged and that would complicate matters a lot.

What about a good-old javax.xml? A simple XPath query like this should do the trick:
// warning: just a POC
try {
    Document document = DocumentBuilderFactory.newInstance()
                       .newDocumentBuilder()
                       .parse(new File(path));
    String query = "/project/packaging";
    String packaging = (String) XPathFactory.newInstance()
                   .newXPath()
                   .compile(query)
                   .evaluate(document, XPathConstants.STRING);
     // value or an empty string (default packaging)
} catch (ParserConfigurationException | IOException | SAXException | XPathExpressionException e) {
    throw new RuntimeException(e); // TODO: do something meaningful instead
}

I'll be happy to pick this up unless you're very into XML parsing 😉

Thank you @mikemybytes, we welcome contributions! 😄

Released in v1.4.0 -> https://github.com/jreleaser/jreleaser/releases/tag/v1.4.0


[BUG] Grafana alerts are not working on deepflow datasource

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

DeepFlow Component
Grafana Plugin
What you expected to happen
I am trying to set up some alert on Grafana based on the deepflow datasource but Grafana will show a error.

How to reproduce
Just try to set up a alert on Grafana dashboard
DeepFlow version
v<IP_ADDRESS>
DeepFlow agent list
No response
Kubernetes CNI
No response
Operation-System/Kernel version
No response
Anything else
No response
Are you willing to submit a PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@twou12031  Take a look here.

Sorry, missed this one before.
We need multiple members working together to fix it.
If there is progress, will add comment.

Hi,
After our troubleshooting, we found the cause of the bug.
First of all, variables can be used in the dashboards of grafana, but not in the alert of grafana.
Then, when creating an alert based on a panel in dashboards that uses a variable, after the jump, the saved query condition contains the variable.
This is causing the bug, the query is not working properly.
It will take a lot of time to support the above usage scenarios.
If there is progress, will add comment.
Our current recommendation is to manually create grafana alerts on the alert page.

Tks fo getting back on this issue.

My workaround that was to use the deepflow promql interface. For the alerts I needed, it worked fine.


Create a mobile app

For the times we only have a phone to track time, a mobile app would be awesome!

Or maybe some Slack / Telegram bot.


Stop unrolling loops in the PathEnumerator

The current PathEnumerator is bugged because it is unable to detect loop blocks, and will therefore copy entire for loops including their condition (which is useless). This PR works around that issue by no longer unrolling loops, simply leaving the loops in the path.

Codecov Report

Merging #215 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##             master     #215      +/-   ##
============================================
- Coverage     79.57%   79.55%   -0.02%
+ Complexity      377      376       -1
============================================
  Files            46       46
  Lines           935      939       +4
  Branches        175      177       +2
============================================
+ Hits            744      747       +3
  Misses          114      114
- Partials         77       78       +1

Impacted Files
Coverage Δ
Complexity Δ

...tlin/org/cafejojo/schaapi/models/PathEnumerator.kt
96.42% <100%> (ø)
19 <9> (-1)
:arrow_down:

...rc/main/kotlin/org/cafejojo/schaapi/models/Node.kt
63.63% <0%> (+6.49%)
0% <0%> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 23086a5...9107759. Read the comment docs.


isomorphic with elastic-apm-node?

I use both elastic-apm-node and apm-agent-js-base in my nodejs + reactjs project with server side render, and i will import apm both the nodejs and browserify side,
if (typeof window !== 'undefined') {

    const { init: initApm } = require('elastic-apm-js-base');

    initApm({
      // Set required service name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
      serviceName: '',

      // Set custom APM Server URL (default: http://localhost:8200)
      serverUrl: 'http://localhost:8200',

      // Set service version (required for sourcemap feature)
      serviceVersion: '',
    });
}

import apmNode from 'elastic-apm-node';

const apm = apmNode.start({
  // Set required app name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
  serviceName: 'xxx',

  // Use if APM Server requires a token
  secretToken: '',

  // Set custom APM Server URL (default: http://localhost:8200)
  serverUrl: 'http://localhost:8200',
});

I think its better to do someting like axios and isomorphic-fetch to support both node and browserify side.
{
  "name": "isomorphic-fetch",
  "version": "0.0.0",
  "description": "Isomorphic WHATWG Fetch API, for Node & Browserify",
  "browser": "fetch-npm-browserify.js",
  "main": "fetch-npm-node.js",

Hi @justinyueh ,
Thanks for your suggestion.
Our Real User Monitoring (js-base) agent and our Node.js agent have different requirements and limitations in our APM solution therefore the code base will probably stay separate. However, it is a possibility to publish both agents under the same package for convenience.
I will make sure that we consider that.
Cheers

Hi @jahtalab
Thanks for your reply.
I published a npm package to handle this.
npm install isomorphic-apm

import apm from 'isomorphic-apm';

const myApm = apm.start({
  // Set required app name (allowed characters: a-z, A-Z, 0-9, -, _, and space)
  serviceName: 'xxx',

  // Use if APM Server requires a token
  secretToken: '',

  // Set custom APM Server URL (default: http://localhost:8200)
  serverUrl: '',

  // Setting it to 0 will disable stack trace collection. Any finite integer
  // value will be used as the maximum number of frames to collect. Setting it
  // to Infinity means that all frames will be collected.
  // stackTraceLimit: 50,

  // The agent maintains an in-memory queue to which recorded transactions are
  // added when they end. The queue is flushed with regular intervals
  // controlled by the flushInterval config option.
  // Use the maxQueueSize option to force a flush of the queue when it reaches
  // a certain size (number of ended transactions) - even if the flushInterval
  // time isn’t reached yet.
  // Set to -1 to disable, in which case only flushInterval counts.
  // maxQueueSize: 20,
  // flushInterval: 5,

  active: true,
});


[Fixes #1747] Enhanced word guessing game

PR Description 📜
Now when the user make a right guess , new question will be displayed after 2 seconds. Before, user had to click on 'Next button' to get the next question.
Fixes #1747

Mark the task you have completed ✅

[x] I follow CONTRIBUTING GUIDELINE & CODE OF CONDUCT of this project.
[x] I have performed a self-review of my own code or work.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generates no new warnings.
[x] I have followed proper naming convention showed in CONTRIBUTING GUIDELINE
[x] I have added screenshot for website preview in assets/images
[x] I have added entries for my game in main README.md
[x] I have added README.md in my folder
[x] I have added working video of the game in README.md (optional)
[x] I have specified the respective issue number for which I have requested the new game.

Add your screenshots(Optional) 📸
https://github.com/kunjgit/GameZone/assets/84488726/222687ec-1c3d-4c50-a291-13117c45dcc7

Thank you soo much for contributing to our repository 💗

Thank you @ritik48 ,for creating the PR and contributing to our GameZone 💗
Review team will review the PR and will reach out to you soon! 😇
Make sure that you have marked all the tasks that you are done with ✅.
Thank you for your patience! 😀

Thank you  @ritik48 , for your valuable time and contribution in our GameZone 💗.
It’s our GameZone, so Let’s build this GameZone altogether !!🤝
Hoping to see you soon with another PR again 😇
Wishing you all the best for your journey into Open Source🚀


ElectricalSeries: important chunking considerations

This is more of a discussion than an issue.
Buzsaki dataset from 2021 (YutaMouse33-150222)
ElectricalSeries data loading in neurosift is very slow especially for large numbers of channels. I looked into the chunking size and got the following:
https://dandiarchive.s3.amazonaws.com/blobs/c86/cdf/c86cdfba-e1af-45a7-8dfd-d243adc20ced
Opening file for lazy reading...
Getting data object
Shape of data:<PHONE_NUMBER>, 96)
Chunking: (298022, 1)
Getting first 10 rows of data
Shape of data chunk: (10, 96)
Elapsed time for reading data chunk: 21.24241590499878 sec

Compare that with a much more recent dataset (000463 draft)
https://dandiarchive.s3.amazonaws.com/blobs/082/8f8/0828f847-62e9-443f-8241-3960985ddab3
Opening file for lazy reading...
Getting data object
Shape of data: (31021056, 32)
Chunking: None
Getting first 10 rows of data
Shape of data chunk: (10, 32)
Elapsed time for reading data chunk: 0.000518798828125 sec

It appears that in the second case, chunking is turned off, and it makes a huge difference for remote reading of data (well, there must be some caching going on here).
You can also see a dramatic difference in loading time in neurosift:
The slow first one (with chunking):
https://flatironinstitute.github.io/neurosift/#/nwb?url=https://dandiarchive.s3.amazonaws.com/blobs/c86/cdf/c86cdfba-e1af-45a7-8dfd-d243adc20ced
Takes 10 seconds to load the initial view
The fast second one (with no chunking):
https://flatironinstitute.github.io/neurosift/#/nwb?url=https://dandiarchive.s3.amazonaws.com/blobs/082/8f8/0828f847-62e9-443f-8241-3960985ddab3
Takes less than 1 second to load the initial view
Even if chunking is needed for very large datasets, I think that the chunks should be much larger in both the channel and time dimensions.
@bendichter

I don't think it's just chunking that's at play here then. Chunking is a necessary step for enabling compression, which is likely what is actually causing the large slowdown compared to the uncompressed (the un-chunked) data
To confirm we would need to find or upload some example files that are chunked, but not compressed. I doubt any of those exist on the archive so I'll just make one and upload to staging
Note: it is, however, best practice to always use compression when storing data on the archive, and the NWB Inspector should warn people about it (though DANDI validation will not outright prevent upload or publishing). Otherwise the archive would be much larger than it is today

BTW out of curiosity, is there any parallelization going on right now w.r.t. data access calls? While decompression isn't super heavy computationally it might be beneficial to have multiple chunks requested and decompressed simultaneously, but not familiar enough with how Type/JavaScript interfaces with things like shared memory

@CodyCBakerPhD to confirm that the first is using compression and the second is not (I added a couple of fields in the Python output):
First (slow):
https://dandiarchive.s3.amazonaws.com/blobs/c86/cdf/c86cdfba-e1af-45a7-8dfd-d243adc20ced
Opening file for lazy reading...
Getting data object
Shape of data:<PHONE_NUMBER>, 96)
Chunking: (298022, 1)
Compression: gzip
Compression opts: 4
Getting first 10 rows of data
Shape of data chunk: (10, 96)
Elapsed time for reading data chunk: 9.983946084976196 sec

Second (fast):
https://dandiarchive.s3.amazonaws.com/blobs/082/8f8/0828f847-62e9-443f-8241-3960985ddab3
Opening file for lazy reading...
Getting data object
Shape of data: (31021056, 32)
Chunking: None
Compression: None
Compression opts: None
Getting first 10 rows of data
Shape of data chunk: (10, 32)
Elapsed time for reading data chunk: 0.0004906654357910156 sec

Do you think there's a way (in the Python code) to test how much time compression is taking, vs network requests? I suppose one way is to download the file and try it locally -- but the first file is very large so that's inconvenient.
For reference, this is the Python script I am using
import time
import numpy as np
import fsspec
import h5py
from fsspec.implementations.cached import CachingFileSystem

# nwb_object_id = 'c86cdfba-e1af-45a7-8dfd-d243adc20ced'
nwb_object_id = '0828f847-62e9-443f-8241-3960985ddab3'
s3_url = f'https://dandiarchive.s3.amazonaws.com/blobs/{nwb_object_id[:3]}/{nwb_object_id[3:6]}/{nwb_object_id}'
print(s3_url)

fs = CachingFileSystem(
    fs=fsspec.filesystem("http"),
    # cache_storage="nwb-cache",  # Local folder for the cache
)

print('Opening file for lazy reading...')
f = fs.open(s3_url, "rb")
file = h5py.File(f)
print('Getting data object')
x = file['acquisition']['ElectricalSeries']['data']
print(f'Shape of data: {x.shape}')
print(f'Chunking:', x.chunks)
print(f'Compression:', x.compression)
print(f'Compression opts:', x.compression_opts)
print('Getting first 10 rows of data')
timer = time.time()
data_chunk = x[:10, :]
print(f'Shape of data chunk: {data_chunk.shape}')
elapsed_sec = time.time() - timer
print(f'Elapsed time for reading data chunk: {elapsed_sec} sec')

Regarding parallelization in the browser, all the hdf5 access takes place in a worker thread... and that worker thread makes synchronous http requests and gunzip (and I believe that's a requirement for using h5wasm), so there is no concurrency there, within a single worker. However, I am using 2 worker threads, so that allow some concurrency at a higher level.
Is it the network latency or the decompression? - I'm just not sure.
Tagging @garrettmflynn

Rigorous performance testing is a very nuanced topic; this is something we want to really sink our teeth into pending a grant we've applied for related to cloud computing, but since we haven't started that yet I'll just keep things as general as I can to avoid making incorrect claims about specificities
In general we follow the recommendations of the HDF5 Group on optimizing I/O, but I'll admit that white paper was put together quite a while ago and was likely not tuned specifically for our S3 situation

Do you think there's a way (in the Python code) to test how much time compression is taking, vs network requests?

The most sophisticated way would be to do a full profile across the underlying h5py (decompression) / fsspec (streaming calls) callback trace but those reports are often quite hard to read and require extremely in depth knowledge about those packages - favorite package for this is cProfile
The easiest way would be what I'm preparing for you now, which is to use Python timing similar to how you're currently doing it (with one difference elaborated below) but applied to several copies of the same data. Across these copies you have the same exact data written
(i) chunked and compressed
(ii) chunked identically to ii but not compressed
(iii) not chunked or compressed at all
and you then test and compare timings of the same exact access patterns across each separate copy
The access pattern itself will also vary a lot between the 3 cases; the read pattern of (10, 96) you show above would of course work much better on any unchunked dataset because you're requesting random access of a 1.92 KB piece from what would otherwise subset from a 0.6 MB chunk - and while I'm not 100% positive that the entire chunk is returned when you access the data region from case i or ii, I suspect that may be the case -
So I'd also recommend testing against a number of read patterns that both subset the chunk shape, are equal to the chunk shape, and span multiple chunks to get fairer comparisons
Technically the type and/or options of compression can also make a big difference, which was the topic of a summer project by the LBNL team last summer, led by @rly. I'll set that aside for now since it gets even more complicated than this simpler discussion

For reference, this is the Python script I am using

Thanks for sharing the code - I'm going to point you towards this section of a blog about using timeit in Python, but feel free to check out the rest if interested
The short version is that timing things according to the time is not always accurate since it captures global system runtime, not the specific time it took the particular operation; timeit can also let you repeat the speed test some number of times so you can average it out in the end to reduce stochasticity (though remember to set unique cache locations each time to avoid subsequent runs pulling from cache instead of retesting the stream time)

Is it the network latency or the decompression? - I'm just not sure.

There's also potential I/O bottleneck if there's any caching to disk involved (which I see your example code is using a CachingFileSystem with fsspec) - that could especially be experienced if more CPUs were thrown at the requests (and the bandwidth was confirmed to not be the current bottleneck)

From @bendichter

Paging (aka sharding) may help with this but needs to be tested

Sure I can make a copy of each (i-iii) that have paging set (assuming I can get it to pass via H5DataIO)

Thanks for doing this @CodyCBakerPhD . I think it will be important to put the datasets in a cloud bucket to test the remote access. Also, they should be large enough to prevent artificially high level of cache hits when downloading chunks of the file (sorry if that is not expressed clearly)

Also, they should be large enough to prevent artificially high level of cache hits when downloading chunks of the file (sorry if that is not expressed clearly)

Yeah I'm copying the entire ElectricalSeries data from https://dandiarchive.s3.amazonaws.com/blobs/c86/cdf/c86cdfba-e1af-45a7-8dfd-d243adc20ced as a simpler TimeSeries, still in acquisition, with the different cases illustrated as the file names. Will let you know and send you links when they're ready

@magland The new 2 examples (chunking and uncompressed, no chunking and no compression) should be up by morning at https://gui-staging.dandiarchive.org/dandiset/200560/draft/files?location=, a folder called 'for_jeremy'
Two fs_strategy=page versions for of each of the 3 file types will be uploaded once I wake up tomorrow - one will have fs_page_size= 4 KiB (the default) and the other will have fs_page_size = 4 MiB just to see if that makes any difference. File names should indicate everything of importance
Where applicable, all chunking and compression parameters are equivalent to the source data

Thanks @CodyCBakerPhD !
Here are some timing results. But a couple notes first

It seems that the not_chunked_and_not_compressed actually does have chunking of (707, 96) (see output below)
I tried to use %timeit magic command in jupyter notebook, but it was not suitable since after the first run it was essentially instantaneous for loading the chunk. I think the traditional method of timing is okay here because it seems to be network-transfer limited.

import time
import numpy as np
import fsspec
import h5py
from fsspec.implementations.cached import CachingFileSystem

## see https://gui-staging.dandiarchive.org/dandiset/200560/draft/files?location=for_jeremy
s3_urls = {
    'chunked_but_not_compressed': 'https://dandi-api-staging-dandisets.s3.amazonaws.com/blobs/738/069/73806960-88ff-4d5e-9920-40d43186cf26',
    'not_chunked_and_not_compressed': 'https://dandi-api-staging-dandisets.s3.amazonaws.com/blobs/0c1/51d/0c151db4-27db-4c2f-8290-0626a66526c3'
}

fs = CachingFileSystem(
    fs=fsspec.filesystem("http")
)

timeseries_objects = {}

for k, s3_url in s3_urls.items():
    print('====================================')
    print(k)
    print('Opening file for lazy reading...')
    f = fs.open(s3_url, "rb")
    file = h5py.File(f)
    print('Getting data object')
    x = file['acquisition']['TimeSeries']['data']
    print(f'Shape of data: {x.shape}')
    print(f'Chunking:', x.chunks)
    print(f'Compression:', x.compression)
    print(f'Compression opts:', x.compression_opts)
    print('Getting first 10 rows of data')
    timeseries_objects[k] = x

# output
====================================
chunked_but_not_compressed
Opening file for lazy reading...
Getting data object
Shape of data:<PHONE_NUMBER>, 96)
Chunking: (298022, 1)
Compression: None
Compression opts: None
Getting first 10 rows of data
====================================
not_chunked_and_not_compressed
Opening file for lazy reading...
Getting data object
Shape of data:<PHONE_NUMBER>, 96)
Chunking: (707, 96) # <--------- still using chunking
Compression: None
Compression opts: None
Getting first 10 rows of data

for k, timeseries_object in timeseries_objects.items():
    print('====================================')
    print(f'Timing reading data for {k}')
    x = timeseries_object
    # timeit doesn't work here because the result is cached after the first run
    timer = time.time()
    data_chunk = x[:10, :]
    print(f'Shape of data chunk: {data_chunk.shape}')
    elapsed_sec = time.time() - timer
    print(f'Elapsed time for reading data chunk: {elapsed_sec} sec')

# output
====================================
Timing reading data for chunked_but_not_compressed
Shape of data chunk: (10, 96)
Elapsed time for reading data chunk: 28.612600326538086 sec
====================================
Timing reading data for not_chunked_and_not_compressed # <----- I think it actually is chunked (see above)
Shape of data chunk: (10, 96)
Elapsed time for reading data chunk: 7.263010501861572 sec

So there is a significant difference between the two schemes, even though both were not compressed.
Looking at my network monitor, it seems the difference is the amount of data transferred - and this will depend on the relationship between the chunking scheme of fsspec.filesystem and the chunking of the timeseries in the hdf5.
This difference is consistent with what I am seeing in neurosift in the browser
chunked_but_not_compressed
http://localhost:3000/neurosift/?p=/nwb&url=https://dandi-api-staging-dandisets.s3.amazonaws.com/blobs/738/069/73806960-88ff-4d5e-9920-40d43186cf26
not_chunked_and_not_compressed <------ I think it actually is chunked
http://localhost:3000/neurosift/?p=/nwb&url=https://dandi-api-staging-dandisets.s3.amazonaws.com/blobs/0c1/51d/0c151db4-27db-4c2f-8290-0626a66526c3

@magland The paginated versions of each combination are up (ignore the 'unchunked' ones of course, regenerating those now). Also includes a compressed + paginated version with same chunking as original data

It seems that the not_chunked_and_not_compressed actually does have chunking of (707, 96) (see output below)

Looks like I used the wrong approach to disable chunking - regenerating now

I tried to use %timeit magic command in jupyter notebook, but it was not suitable since after the first run it was essentially instantaneous for loading the chunk.

Yeah, as I mentioned above, would need to remember to set unique cache locations each time to avoid subsequent runs pulling from cache instead of retesting the stream time. Or not enable caching at all since we're really trying to test the bandwidth speed vs. other factors
I still strongly recommend the timeit approach, with a proper setup command to avoid conflation with other setup delays

So there is a significant difference between the two schemes, even though both were not compressed.

Yes, with respect to the read access pattern (10, 96). I'd recommend trying a wider range of access patterns representing real life situations, such as zooming in on several localized waveforms (smaller range of channels, maybe ~50 frame slice or so), monitoring global variations over longer time scales (num_seconds * sampling_rate number of frames, all channels, where num_seconds > 5 or so), and anything else you can think of
Despite my mistake, this is actually useful. What happened was the write procedure defaulted to using the h5py chunking scheme instead of entirely disabling chunking as intended. The default h5py schema, despite the HDF5 teams recommendations, actually generates much smaller chunk sizes (~0.13 MB seen here, opposed to the ~0.6 MB of the channel-wise chunked file)
So the difference in speed reading uncompressed data is in line with my hypothesis that data access requests for any sub-slice of a chunk streams the entire chunk worth of data. Not saying this is absolute confirmation but it could certainly explain the current observation

Thanks! I agree that we'll want to look at the various read access patterns. For sanity (this is getting pretty complicated), I suggest we focus on one single measure: how fast is the initial load of the traces in neurosift? For that question there is a very clear winner:
unchunked_and_uncompressed_* (which actually is the default chunking from h5py) beats all the others by a long shot. In fact, right now, neurosift is set to only load the first 15 channels... and the difference is: waiting for around 15 seconds, and 10MB data download on the one hand, and waiting for around 1 second and less than 1MB download on the other. If I adjust the app to try to load ALL the channels, the one goes for more than 100 seconds before I closed the tab, whereas the other is just a few seconds.
So, before getting into other use cases, there is an overwhelming winner here from the perspective of initial load of a small amount of data (which is highest priority for me). The question remains whether it would still be performant if we added compression to the default h5py chunking.
Lazy loading via Python is a different consideration, which is important, but I'm hoping to focus first on the neurosift performance.
Also, the different pagination schemes don't seem to make an obvious difference - at least when loading into neurosift.

Thanks! I agree that we'll want to look at the various read access patterns. For sanity (this is getting pretty complicated)

That's what I was trying to tell you lol 😆 this is a very deep rabbit hole and premature optimization can be detrimental to rapid dev momentum
Just constraining to rough neurosift app speeds for now is fine by me
I have now reuploaded (replaced) the filenames with properly unchunked versions, though I had to constrain it to the first ~100 seconds or so because any attempt to buffer the data write triggered chunking to be enabled - curious what you notice on the speeds when data is completely unchunked (though I notice I can't seem to control the second dimension of the TimeSeries view)
For the final round of fair use case tests I'll generate some with the default h5py chunking as well as modern NeuroConv chunking (differs from the old source data), both compressed and uncompressed (and name the files accordingly). Probably skip pagination for now on those

So, before getting into other use cases, there is an overwhelming winner here from the perspective of initial load of a small amount of data (which is highest priority for me).

The last thing I'll mention here is, if we accept my notion of the correlation between streaming speed and chunk size, then we are left with the following argument
a) published data assets have their dataset options, including chunking, frozen and cannot be changed to optimize streaming speeds
b) we can make changes to NeuroConv and related default behaviors, which will produce new dandisets with new files with more optimized streaming behavior, but that will not happen overnight
c) while the unchunked examples are interesting to play with, I really doubt the DANDI team would be OK with doubling (or more) the size of the archive - compression of large data is likely non-negotiable, which then requires chunking, which we can only work to optimize the shape and sharding thereof
d) we both want neurosift to be able to visualize contents as efficiently as the source allows (given non-optimal chunking, as well as compression)
Towards the aim of (d) given the constraints of (a-c) I'm driven to the following conclusion: neurosift should attempt to work as best as it can around the chunk shape instead of being tuned to specific target ranges, even if those targets are more 'ideal' in general
What I mean by this is, if streaming requests transfer an entire chunk worth of data even if you didn't intend to use all the data in that chunk, the way to improve performance is for default rendering to make the most of what's been returned.
That is, instead of always having a default view that targets say (5 seconds, 15 channels) - which would only work well when source chunks are more roughly (~moderate amount of time, ~small amount of channels) as opposed to poor performance seen with source chunks (~small amount of time, all channels) or (~large amount of time, 1 channel) - instead make the default view always equal in shape to the first chunk or first few chunks that add up to some very small data threshold (such as ~5 MB), thus ensuring the users bandwidth is being utilized as much as possible upon the first view of the data.
What would you think of that approach?
Granted, if the user then selects different channel ranges or scans/plays or zooms in/out over time, that's when we get sent to the broader discussion of access patterns that can wait for another day, but still not too much we can do right away given the constraints (a-c)

Thanks @CodyCBakerPhD, what you say makes sense. A lot of this is pretty mysterious to me. I'll follow up on Monday.

Hi @CodyCBakerPhD
There's a lot to discuss here - To start, jere are some Python timings based on the new data you uploaded, and your suggestion to look at reading other sizes of data.
I ran each timing twice to get at least some idea of the variability
For reading (10, 96) segment:
====================================
Timing reading data for chunked_but_not_compressed
Shape of data chunk: (10, 96)
Elapsed time for reading data chunk: 48.093024492263794 sec / 61.54763746261597 sec
====================================
Timing reading data for chunked_and_compressed_page_fs_strategy_4_KiB
Shape of data chunk: (10, 96)
Elapsed time for reading data chunk: 30.916049242019653 sec / 31.682336568832397 sec
====================================
Timing reading data for default_h5py_chunking_not_compressed <--- this is no longer in the folder you shared
Shape of data chunk: (10, 96)
Elapsed time for reading data chunk: 6.520519971847534 sec / 10.449405670166016 sec
====================================
Timing reading data for unchunked_and_uncompressed_page_fs_strategy_4_KiB
Shape of data chunk: (10, 96)
Elapsed time for reading data chunk: 0.00031256675720214844 sec / 0.0003273487091064453 sec

For reading (298022, 96) segment
====================================
Timing reading data for chunked_but_not_compressed
Shape of data chunk: (298022, 96)
Elapsed time for reading data chunk: 37.758856773376465 sec / 37.91836762428284 sec
====================================
Timing reading data for chunked_and_compressed_page_fs_strategy_4_KiB
Shape of data chunk: (298022, 96)
Elapsed time for reading data chunk: 19.3127863407135 sec / 25.656846284866333 sec
====================================
Timing reading data for default_h5py_chunking_not_compressed <--- this is no longer in the folder you shared
Shape of data chunk: (298022, 96)
Elapsed time for reading data chunk: 36.85222601890564 sec / 35.99728536605835 sec
====================================
Timing reading data for unchunked_and_uncompressed_page_fs_strategy_4_KiB
Shape of data chunk: (298022, 96)
Elapsed time for reading data chunk: 17.922171592712402 sec / 17.13180422782898 sec

I chose (298022, 96) so that the 298022 matches the chunking size for the first two examples.

There's a great youtube video on this: https://www.youtube.com/watch?v=rcS5vt-mKok We can try to implement some of these features

@magland

One issue is that all the meta information (attributes of groups and datasets, plus the data for small datasets) needs to be read very rapidly on page load. This is not efficient if that meta data is scattered randomly throughout a very large nwb file. To remedy this, I have been precomputing a meta-only version of the nwb file and storing it on a separate server. That file ends up being 2-10 mb in size (usually). And so for that I use a larger virtual file chunk size (4 mb), and it loads all that initial information very quickly.

It's fine to store intermediate files as a stop-gap but ultimately we'd like to have default file settings that are optimized for cloud access so these types of methods are unnecessary. We have been experimenting with packing h5 files so that all of this metadata is together and can be read one few requests. Can you point me to a file that is giving you trouble in this category? I'll use that to test this new approach.

@bendichter

It's fine to store intermediate files as a stop-gap but ultimately we'd like to have default file settings that are optimized for cloud access so these types of methods are unnecessary. We have been experimenting with packing h5 files so that all of this metadata is together and can be read one few requests. Can you point me to a file that is giving you trouble in this category? I'll use that to test this new approach.

There are certainly more extreme examples, but here's one where there is a noticeable difference in the initial load time:
Without using my meta nwb (slow):
http://flatironinstitute.github.io/neurosift/?p=/nwb&url=https://dandiarchive.s3.amazonaws.com/blobs/c86/cdf/c86cdfba-e1af-45a7-8dfd-d243adc20ced&no-meta=1
With using my meta nwb (fast):
http://flatironinstitute.github.io/neurosift/?p=/nwb&url=https://dandiarchive.s3.amazonaws.com/blobs/c86/cdf/c86cdfba-e1af-45a7-8dfd-d243adc20ced&no-meta=0
(note the no-meta=1 query parameter)
For me the difference is 6 seconds versus <1 second.

@bendichter @CodyCBakerPhD
I uploaded some ephys datasets to DANDI. Notice how fast this loads when I use no compression and no chunking.
https://flatironinstitute.github.io/neurosift/?p=/nwb&url=https://dandiarchive.s3.amazonaws.com/blobs/638/80a/63880a91-ca46-4ab8-ba07-bf4ab044145c&tab=neurodata-item:/acquisition/ElectricalSeries|ElectricalSeries
I would expect similar performance with large chunks (covering all channels) and no compression.
The next best option (IMO) would be large chunks (covering all channels) and compression.
The least cloud-optimal option is small chunks (or chunks covering a small number of channels) with or without compression.

The next best option (IMO) would be large chunks (covering all channels)

Even for NeuroPixels? By my calculations, a ~10MB chunk over all 384 channels would only cover about 1/3 of a second of equivalent real-time data.
I'm about to repackage some of the raw data from 409, which is multiprobe NP - what chunk shapes would you like me to try with that? I was thinking of trying at least
(54613, 96) (close to 2s)
(27307, 192) (closer to 1s)
(13653, 384)

The next best option (IMO) would be large chunks (covering all channels)

Even for NeuroPixels? By my calculations, a ~10MB chunk over all 384 channels would only cover about 1/3 of a second of equivalent real-time data.
I'm about to repackage some of the raw data from 409, which is multiprobe NP - what chunk shapes would you like me to try with that? I was thinking of trying at least
(54613, 96) (close to 2s) (27307, 192) (closer to 1s) (13653, 384)

Oh yeah, I wasn't thinking of the large channel counts (I mainly work with 32 ch). I guess this depends on whether people will usually want to access all the channels or a subset. This is really difficult - I don't know a good way here. :)

With 32 ch, how big of a time window would you ever want to look at?

With 32 ch, how big of a time window would you ever want to look at?

Usually just a few seconds, but perhaps up to 30 seconds.

Usually just a few seconds, but perhaps up to 30 seconds.

Thanks, that's helpful
I'll add
(81920, 64) (~2.7s)
(163840, 32) (~5.4s)
(327680, 16) (~11s)
to experiment with

@bendichter Here's a more dramatic example highlighting the need for the meta nwb file
https://flatironinstitute.github.io/neurosift/?p=/nwb&url=https://dandiarchive.s3.amazonaws.com/blobs/fd9/602/fd9602fd-77d5-42ad-8ea9-08fdc9f08226&no-meta=1
that takes around 20 seconds to load... compared with just a second for this one:
https://flatironinstitute.github.io/neurosift/?p=/nwb&url=https://dandiarchive.s3.amazonaws.com/blobs/fd9/602/fd9602fd-77d5-42ad-8ea9-08fdc9f08226&no-meta=0
It's because of the 188 acquisition items

that takes around 20 seconds to load

which file on DANDI is that? I can try repacking it really quick to see if paginated metadata solves the initial loading problem

that takes around 20 seconds to load

which file on DANDI is that? I can try repacking it really quick to see if paginated metadata solves the initial loading problem

It's this: https://dandiarchive.org/dandiset/000615/draft/files?location=sub-001 / sub-001_ses-20220411.nwb

@magland OK here is the example of that one paginated at 10 MiB: https://flatironinstitute.github.io/neurosift/?p=/nwb&url=https://dandi-api-staging-dandisets.s3.amazonaws.com/blobs/c61/76f/c6176f66-c606-44eb-8d68-85ef71c3808d
located in https://gui-staging.dandiarchive.org/dandiset/200560/draft/files?location=for_jeremy
Is that any faster for you?
One other thing is I do not know how or if h5wasm exposes page buffer size, which can also have a big impact on how these are read
The analog in h5py is the page_buf_size at this line: https://github.com/h5py/h5py/pull/1967/files#diff-6515aaa459980029aa5ae782298c82cacb68f8f6f3218a9f0419f7148bd80f30R357
and it must be set to a multiple of the page layout to work well; I'd recommend page_buf_size=20971520 here, if possible

@CodyCBakerPhD That makes a huge difference!
Your paginated version loads in just a couple seconds for me
https://flatironinstitute.github.io/neurosift/?p=/nwb&url=https://dandi-api-staging-dandisets.s3.amazonaws.com/blobs/c61/76f/c6176f66-c606-44eb-8d68-85ef71c3808d&no-meta=1
(notice no-meta=1)
compared with this slow one
https://flatironinstitute.github.io/neurosift/?p=/nwb&url=https://dandiarchive.s3.amazonaws.com/blobs/fd9/602/fd9602fd-77d5-42ad-8ea9-08fdc9f08226&no-meta=1
The meta nwb method still seems to be a bit faster than the paginated one, but both are fast enough.

That makes a huge difference!

Great to hear 👍 that alone is a pretty easy change to make aside from the chunking problem so we'll start incorporating that across the ecosystem

The new round of NeuroPixel examples are up for qualitative evaluation
The data stream that was repacked was the ElectricalSeriesAP from https://dandiarchive.org/dandiset/000409/draft/files?location=sub-CSH-ZAD-001%2F
The naming convention is -chunking-{num_frames}-{num_channels} as outlined above
Let me know which you prefer so we can update that as the default for that and similar formats

@magland I had to go into DANDI set 59 to fix some unrelated things, so while I was there I implemented all the practices suggested here
Try out any/all of the files here: https://dandiarchive.org/dandiset/000059/0.230907.2101/files?location=
The raw ones are not paginated as it inflated file size ~20%, pretty much undoing any gains from compression
The processed ones are paginated at 10 MiB
All chunk shapes for all data streams are around ~10 MiB, with the ElectricalSeries in particular chunked by 64 channels

Thanks @CodyCBakerPhD
I took a look at a few of them, and they do seem pretty responsive.

Can this be closed? These examples and greater discussion are making their way to https://github.com/NeurodataWithoutBorders/nwb_benchmarks
Also NeuroConv defaults have used 64 channel max up to 10 MB total size for a while now

closing.


huntr - Cross-site Request Forgery (CSRF) Fix

https://app.huntr.dev/users/anandtiwarics has fixed a security vulnerability (Cross-site Request Forgery (CSRF)) :hammer:. anandtiwarics has been awarded $25 for fixing the vulnerability through the huntr bug bounty program :dollar:. Think you could fix a vulnerability like this? Get involved at https://huntr.dev/!Q | AVersion Affected | ALLBug Fix | YESOriginal Pull Request | https://github.com/418sec/Django-CRM/pull/1

fix conflicts and open new pull request


Feeder

Hi, I have done some research and cannot find anything explaining exactly how the feeder works, my main question is "Is the feeder smart in a way that it will breed cows to progress? and not just keep breeding randoms cows?"
Thanks.

Feeder get all cows from radius.
Check all cows can mate with other cows.
If it can, check wheat count > 2.
And after start breeding two cows.
Its work same cows and with different cows which can breeding.


Add audio and video options to posts

Create 2 toggles in the settings page: 'Oral', 'Sign'. Default both to false.
Add a nullable file id reference to the versions table to store the audio/video file reference.
When users create a post, they need to choose between 'Text', 'Audio', and 'Video'. Show a selection modal first and then show the appropriate UI for a text post, audio post, or video post.
If the oral toggle is true, don't show text as a post option.
If the sign toggle is true, don't show text or audio as a post option and skip the selection screen.
Use Ionic microphone/webcam sdks to let the user record audio/video from their browser/app.
Use the file API to upload a file as a post's version.
Provide an option to let the user upload a file instead of record audio/video. Limit the file type to know audio files for audio, and known video files for video.
Update the post component to check for type and show the appropriate UI for if it is a post, audio, or video.
Bonus points if you can find a way to show an audio waveform when its an audio file.

I think the old site had some audio/video record functionality but I might be confusing projects. Take a look at crowd.Bible, there might be something there.


Add Versions to Blueprint

Once all blueprints are stable, let's add a version to the YAML, then update whenever changes are made, so we can tell when each of them changes, and how fast they are iterating.

I added a date version to each blueprint, which we should change to the date whenever the blueprint is modified.


chaining element() is not working with xpath (at least)

//HTML

1

                                                Successful additions

0

                                                Failed attempts

//Protractor code
function GetSummaryCounts(){

        var parentXpath = '//span[contains(text(),"Successful additions")]/..';
        var parentDiv = element(by.xpath(parentXpath));

        var h5XPath = '/h5[@class="description-header ng-binding"]';

        //var h5Element = parentDiv.element(by.xpath(h5XPath)); // does not work !!!

        var h5Element = element(by.xpath(parentXpath+h5XPath)); // does work

        var EC = protractor.ExpectedConditions;
        browser.wait(EC.presenceOf(h5Element), 5000);
        return h5Element.getText();

    };

What error message do you get?

Well get actually a java exception something to with casting hashmap to string I will paste it tomorrow once im back in the office

Chaining does not work with xpath, this is just how xpath selectors work in webdriver. This is documented at http://www.protractortest.org/#/api?view=webdriver.By.xpath. Closing this as working as intended.

Hi thanks for for point this out, you are right this is documented but to be honest, that documentation would be the last thing I would be looking at, this is because it's a special case and if this works with other elements selectors why would anyone think it a different case with xpath ?.
Im jus't saying it's not intuitive at all. If that was me I would probably change the API on protractor level and fix this under the hood. Otherwise this is extending a webdriver library together with their issues.  This should probably be fixed in the webdriver as well.


HOSTEDCP-1513: Support annotation scoping for hostedcluster resources

Allow the operator to manage a select group of hostedcluster resources that contain a specified annotation.  This feature can be useful for situations in which you want to test a new version of the operator without affecting an existing workload that is being managed by the current production-level operator.  In a dual operator setup like this, the "production" operator would be configured to ignore events for all scope annotated hostedclusters, while the "test" operator would ignore events for all hostedclusters that do not have the annotation.  Thus each operator would only process/reconcile hostedclusters that it manages.  The exact annotation to match in the test operator would be defined in the env var HOSTEDCLUSTERS_SCOPE_ANNOTATION, while the production operator wouldn't need to define anything.
The feature would NOT be enabled by default, as it requires the ENABLE_HOSTEDCLUSTERS_ANNOTATION_SCOPING to be set to go down that path.  Without that set, the operator would function as it does today.

/approve

/retest

@lucaspalm can you rebase you pr, that should fix the failing konflux test.

/lgtm

/cherry-pick release-4.15


[EASI-2267] - Prepare for clearance

EASI-2267
Changes and Description

Prepare for clearance task list item
Checklist form component
Review (readonly) component
Subscription lock/modal on review component
Clearance warning/modal if already cleared
Unit/cypress testing

Out of scope but made ticket much easier to develop

Context provider for getting model plan info from anywhere in app

How to test this change

Seed the database
Enter into Empty Plan and verify the info box on Prepare for clearance task list section
Enter into the Basics and populate the clearanceStarts with a date within 20 days
Return to task list and verify Prepare for clearance is now Ready
Enter into the task list item and check/uncheck checkboxes and update status
Verify task list status is now In progress
From the checklist, review all sections and try to edit a section that has already been cleared
Confirm modal when trying to access cleared section
GQL subscription lock modal will appear if another browser open and occupying task list section

PR Author Review Checklist

[ ] Met the ticket's acceptance criteria, or will meet them in a subsequent PR.
[ ] Added or updated tests for backend resolvers or other functions as needed.
[ ] Added or updated client tests for new components, parent components, functions, or e2e tests as necessary.
[ ] Updated the Postman Collection if necessary.

PR Reviewer Guidelines

It's best to pull the branch locally and test it, rather than just looking at the code online!
Check that all code is adequately covered by tests - if it isn't feel free to suggest the addition of tests.
Always make comments, even if it's minor, or if you don't understand why something was done.

@patrickseguraoddball This looks good! The seeded data update looks good too!
I didn't get the modal like the test script suggested when I updated a section that is marked ready for clearance, ( I tried specifically in Model Basics). When I click it, I'm brought directly to the first page to edit

I was also wondering if we should do anything we can do to make it clearer that you can only go back to in progress when you update a task list section from Ready for Clearance. Right now, I can get to the last page of the form with Ready for Clearance, and then mark that it is ready for review. Like we've previously talked about, the status goes directly to in progress instead of ready for review. Do you think this is worth bringing up with UX?

@patrickseguraoddball This looks good! The seeded data update looks good too!
I didn't get the modal like the test script suggested when I updated a section that is marked ready for clearance

The modal should only appear on the review page, bottom link to directly edit.

@patrickseguraoddball This looks good! The seeded data update looks good too!
I didn't get the modal like the test script suggested when I updated a section that is marked ready for clearance
Thanks Steven!

The modal should only appear on the review page, bottom link to directly edit.

Perfect! Thanks Patrick, that's working as expected then :)


refactor run_against_protocols to use loop

Currently run_against_protocols uses a map, but maybe this can be made nicer by using a loop. I'm hoping that then we can get rid of some of the cloning.