Integrating @pe-ro 's adjustments from #1353

Refs #1353

So I guess it is all about these two commits?
https://github.com/music-encoding/music-encoding/pull/1419/commits/cccf6986c23aa9df6624ce23654a473d24e32a2a

And it is going to @pe-ro's branch to 1) bring it in synch with develop, and 2) add the guidelines examples for it, right?

Yes, sadly i committed on a seperate branch. Noticed afterwarts.

OK. I think after the merge it will be a bit cleaner.


(feature/569) adding a getDestination() function for Endpoints.

[x] I have reviewed the Contributing Guidelines
[x] I agree to license my contributions under the same license as in this repository
[x] I have verified that the tests pass

Description
I'm adding a getter function for the targetDest property of the Endpoint class.

We probably should add a similar function to the C api.  And looking at that the C function should probably be helicsEndpointGetDefaultDestination, which makes me wonder if we should make the C++ function setDefaultDestination  and getDefaultDestination  to make everything consistent?  What do you think?  @nightlark  any opinions?

To be good we probably also need to add a statement to a test case somewhere along the line.

@afisher1  I added the functions to the C api and C++98 API and changed the names on the C++ api so it matches all the others.  I think I added everything take a look see what you think?  @nightlark  you will have to review this see if we are missing something.

@phlptp that looks correct to me.


Record video for WKC lecture

@scottdangelo I'm closing these recording issues, we are tracking in other repo.


Cannot read property 'length' of null

Windows 10
Anchor Wallet 1.0.4

I just installed the app, and clicked "Setup Cold Wallet". Then I click View -> Reload, I get the following error:

Also, the "Cancel" button on when creating a cold wallet does not work. The only way I was able to exit this page is to completely reset the wallet by clicking the bottom left corner, and clicking "Reset Application"

We should have a fix for this error in the next build, it looks like the node validation was trying to connect despite being setup as a cold wallet.

Both are available in 1.0.5 - thanks for reporting these!


try_private false

Hi there!
We've tried to use MQTTnet as a broker (server) with our old raspberry pi boards (where was installed mosquito as a client), but we couldn't get a proper communication until we didn't change clients settings try_private to false in mosquitto.conf file.
I understand that try_private settings related only to mosquito library, but I'm sure that it will be very great if current library would support that feature also.
try_private [ true | false ]
If try_private is set to true, the bridge will attempt to indicate to the remote broker that it is a bridge not an ordinary client. If successful, this means that loop detection will be more effective and that retained messages will be propagated correctly. Not all brokers support this feature so it may be necessary to set try_private to false if your bridge does not connect properly.
Thanks!
Which project is your question related to?

Server

Hi developer!
I'm also waiting for a decision))

Solved!
try_private true - default - add one bit fore protocol version (mast 0000 0011 = 3, but send 1000 0011 = 131)
if try_private false - send protocol version equals 3

You are the Rockstar, dude!


Button is impervious to attempts to style/align it...

... due to it having been embedded within a 'box' whose styles are untouchable by user code. e.g., try setting "padding-left: 5em" on a button and notice that the text inside is padded but the button itself occupies the entire padded space. Another example: try setting "align-self: flex-end" and notice that it doesn't have any effect at all.

Yep, I think this issue is the same as this: https://github.com/Day8/re-com/issues/76

Yes, looks the same to me as well.

As of 2.12.0 this should be solved by use of the new :parts arg and the :wrapper part keyword; see https://re-com.day8.com.au/#/button 'Parts' section.

As of 2.12.0 this should be solved by use of the new :parts arg and the :wrapper part keyword; see https://re-com.day8.com.au/#/button 'Parts' section.


Packet identifier collision between client and server

I have a client that sends 1000 Subscribe messages when it starts.  My server sends back a Publish message for every topic that has a stored message.
It turns out that the packet identifiers used by the client and server both start from 1, and monotonically increase.  Depending on scheduling and data behaviour in the server, it is possible that the client sends a Subscribe with packet identifier 90 (for example) at the the same time that the server sends out a Publish message with packet identifier 90.
The result is that the client receives a Publish packet with identifier 90 when it was expecting a SubAck with identifier 90.  The code matches the waiting subscriber thread with the Publish packet and then throws an error because it cannot coerce a MqttPublishPacket to a MqttSubAckPacket.
I know this sounds like a rare case.  You would be surprised how many times I encountered this before I figured out what is going on.
Can you suggest a good fix for this?  I was thinking that I could change MqttPacketIdentifierProvider to set _value to a random number to start, which would reduce the chance of this happening, but how can I eliminate this case entirely?

that is an mqtt issue in general the identifier is a uint, a complete fix would be to specify a new protcol version with 128 bit idents aka guids, but that is out of the scope of this implementation

Thank.  That's what I thought I was seeing in the spec.  Thinking about this, I'm going to hack the awaiter code to compare the packet by both type and identifier.  I cannot think of a case where both the identifier and the packet type could match while actually being the wrong message.

I am wondering because the original Dispatcher was comparing Type and ID as well. The Tuple contains both. To me only this makes sense.

I assume this is no longer relevant because the mentioned way is already implemented. Feel free to open it again if required.

Thanks for looking at this.  It was entirely my mistake.  I modified the code for synchronous operation and did not properly take into account the expected return packet type.  After fixing my problem it works as you describe.


Refactor key input and devise complete keyboard input model

Steps:

[x] implement input tree
[x] replace control structure in the Controller
[x] ~update pane model with input processing~ (inline interface suffices)
[x] move control from Controller to Root Pane
[x] implement switching focus between panes, e.g. events pane and tools on main view
[x] split up control across different panes
[x] realize modal input tree overlays concept
[x] generify input.Tree to some interface for input handling
[x] restructure wrapping panes to fit envisioned input hierarchy
[x] make Action an interface; implement basic versions; try to account for future undoables implementation
[x] come up with default keyboard input config
[x] implement dynamic help-content generation
[x] ~(make keyboard control configurable (config.yaml))~

misc

[x] fix NPE for event switching on empty day
[x] refactor event editor to use new input concepts to evaluate using the modal overlay idea more for realizing vim-like text navigation
[x] try to let child panes hold only the focus-querying interface (has focus? focusses?)

SUMMARY:
✓	S          closeSummary
✓ h          prev{Day,Week,Month} (depending on view)
✓ l          next{Day,Week,Month} (depending on view)

LOG:
	E          closeLog
	<c-u>      scrollUp
	<c-d>      scrollDown

IN EDTIOR:
✓	<esc>      abort
✓	<cr>       commit
✓	h          charLeft
✓	l          charRight
✓	0          nameBeginning
✓	$          nameEnd
✓	w          nextWordBeginning
✓	e          nextWordEnd
✓	b          prevWordBeginning
✓	x          deleteChar
✓	cc         clear,enterInsert
✓	dd         clear
	...
✓	i          enterInsert

IN EDITOR (--INSERT--):
✓	Each letter committed, escape as special case to remove mode
	overlay again.
	I think this can be done if we replace the concrete Tree type with
	an interface (process, active, ...?) and then have a text-input
	implementation for that interface that could get a single-rune
	callback to make with each change, so e.g. on event editor it
	could call a callback that knows the event-editor cursor pos and
	inserts the given letter at it.


ARROW-10588: [Rust] Safe and parallel bit operations for Arrow

Up to %95 improvements on many kernels.
Implements safe bit operations for Arrow
Implements typed_bits to get bits as Vec
Implements various bit operations for the use with arrow arrays
Adjusts parquet array reader to use Arrow bit operations

I squashed all my work into a single commit and closing PR: #8598 with comment: https://github.com/apache/arrow/pull/8598#issuecomment-727294219

Benchmarks

sum 2^20                time:   [160.84 us 161.83 us 163.10 us]
                        change: [-88.535% -88.450% -88.365%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild
  3 (3.00%) high severe

Benchmarking min 2^20: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.1s, enable flat sampling, or reduce sample count to 50.
min 2^20                time:   [1.4026 ms 1.4027 ms 1.4028 ms]
                        change: [-0.0516% -0.0286% -0.0098%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 12 outliers among 100 measurements (12.00%)
  1 (1.00%) low severe
  8 (8.00%) high mild
  3 (3.00%) high severe

sum nulls 2^20          time:   [783.74 us 797.75 us 812.02 us]
                        change: [-80.849% -80.392% -79.903%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild

min nulls 2^20          time:   [12.366 ms 12.375 ms 12.389 ms]
                        change: [+24.048% +24.202% +24.381%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high severe

min string 2^20         time:   [9.9124 ms 9.9145 ms 9.9174 ms]
                        change: [+1.9874% +2.0847% +2.1694%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 22 outliers among 100 measurements (22.00%)
  2 (2.00%) low severe
  5 (5.00%) low mild
  8 (8.00%) high mild
  7 (7.00%) high severe

min nulls string 2^20   time:   [19.348 ms 19.352 ms 19.357 ms]
                        change: [+3.4758% +3.6867% +3.8658%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) low mild
  3 (3.00%) high mild

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/arithmetic_kernels-ce51190c627182f2
add 2^20                time:   [971.06 us 972.57 us 974.19 us]
                        change: [-58.352% -58.310% -58.264%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) low mild
  1 (1.00%) high mild

subtract 2^20           time:   [970.69 us 972.03 us 973.30 us]
                        change: [-58.416% -58.371% -58.327%] (p = 0.00 < 0.05)
                        Performance has improved.

multiply 2^20           time:   [977.12 us 978.10 us 979.03 us]
                        change: [-58.449% -58.374% -58.308%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking divide 2^20: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 8.7s, enable flat sampling, or reduce sample count to 50.
divide 2^20             time:   [1.7273 ms 1.7275 ms 1.7278 ms]
                        change: [-53.618% -53.550% -53.507%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
  12 (12.00%) high mild
  2 (2.00%) high severe

limit 2^20, 512         time:   [114.62 ns 114.65 ns 114.69 ns]
                        change: [+4.5154% +4.6662% +4.7672%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low severe
  4 (4.00%) high mild
  1 (1.00%) high severe

add_nulls_2^20          time:   [980.77 us 982.49 us 984.18 us]
                        change: [-58.295% -58.224% -58.159%] (p = 0.00 < 0.05)
                        Performance has improved.

Benchmarking divide_nulls_2^20: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 8.7s, enable flat sampling, or reduce sample count to 50.
divide_nulls_2^20       time:   [1.7293 ms 1.7295 ms 1.7297 ms]
                        change: [-54.156% -54.132% -54.101%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/array_from_vec-45cab0a3ecf39249
array_from_vec 128      time:   [484.23 ns 484.68 ns 485.12 ns]
                        change: [+2.1162% +2.2409% +2.3687%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

array_from_vec 256      time:   [712.61 ns 713.50 ns 714.36 ns]
                        change: [+2.1834% +2.3743% +2.5626%] (p = 0.00 < 0.05)
                        Performance has regressed.

array_from_vec 512      time:   [1.1879 us 1.1892 us 1.1906 us]
                        change: [-4.2637% -4.1494% -4.0486%] (p = 0.00 < 0.05)
                        Performance has improved.

array_string_from_vec 128
                        time:   [2.2701 us 2.2709 us 2.2716 us]
                        change: [+6.6458% +6.7030% +6.7594%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high mild

array_string_from_vec 256
                        time:   [3.6563 us 3.6570 us 3.6577 us]
                        change: [+5.7356% +5.7737% +5.8114%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

array_string_from_vec 512
                        time:   [6.1225 us 6.1246 us 6.1268 us]
                        change: [+5.6870% +5.7640% +5.8419%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  6 (6.00%) high mild
  1 (1.00%) high severe

struct_array_from_vec 128
                        time:   [117.76 us 118.15 us 118.53 us]
                        change: [+3455.9% +3472.4% +3491.6%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  1 (1.00%) high severe

struct_array_from_vec 256
                        time:   [105.76 us 106.43 us 107.21 us]
                        change: [+2015.8% +2029.9% +2045.7%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
  4 (4.00%) low mild
  4 (4.00%) high mild
  3 (3.00%) high severe

struct_array_from_vec 512
                        time:   [9.0257 us 9.0293 us 9.0328 us]
                        change: [+23.421% +23.496% +23.570%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) low mild
  4 (4.00%) high mild
  1 (1.00%) high severe

struct_array_from_vec 1024
                        time:   [27.670 us 27.759 us 27.859 us]
                        change: [+134.39% +135.24% +136.13%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/array_slice-49615c84b19508ee
array_slice 128         time:   [163.62 ns 163.71 ns 163.80 ns]
                        change: [+28.932% +28.988% +29.037%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 17 outliers among 100 measurements (17.00%)
  5 (5.00%) low severe
  4 (4.00%) low mild
  6 (6.00%) high mild
  2 (2.00%) high severe

array_slice 512         time:   [265.55 ns 265.63 ns 265.71 ns]
                        change: [+98.289% +98.369% +98.439%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

array_slice 2048        time:   [649.40 ns 650.84 ns 652.87 ns]
                        change: [+319.75% +320.19% +320.76%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild
  5 (5.00%) high severe

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/boolean_kernels-6fc1ad233dfa3072
and                     time:   [34.437 us 34.447 us 34.457 us]
                        change: [+39.672% +39.729% +39.788%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low mild
  2 (2.00%) high mild

or                      time:   [34.540 us 34.550 us 34.560 us]
                        change: [+39.274% +39.429% +39.552%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

not                     time:   [17.377 us 17.386 us 17.393 us]
                        change: [+38.007% +38.289% +38.681%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/buffer_bit_ops-7f2013fd9e28372f
buffer_bit_ops and      time:   [28.631 us 28.749 us 28.893 us]
                        change: [+4064.1% +4101.4% +4148.8%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  2 (2.00%) high mild
  5 (5.00%) high severe

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/builder-b9aacab6b54f093d
Benchmarking bench_primitive: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.7s, enable flat sampling, or reduce sample count to 50.
bench_primitive         time:   [1.5190 ms 1.5193 ms 1.5195 ms]
                        thrpt:  [2.5707 GiB/s 2.5711 GiB/s 2.5715 GiB/s]
                 change:
                        time:   [+20.305% +20.339% +20.372%] (p = 0.00 < 0.05)
                        thrpt:  [-16.924% -16.902% -16.878%]
                        Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) low severe
  1 (1.00%) high mild
  2 (2.00%) high severe

bench_bool              time:   [3.7009 ms 3.7045 ms 3.7080 ms]
                        thrpt:  [134.84 MiB/s 134.97 MiB/s 135.10 MiB/s]
                 change:
                        time:   [+17.329% +17.443% +17.585%] (p = 0.00 < 0.05)
                        thrpt:  [-14.955% -14.853% -14.770%]
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) low mild
  1 (1.00%) high mild

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/cast_kernels-93bbf3677ef76b84
cast int32 to int32 512 time:   [29.420 ns 29.421 ns 29.423 ns]
                        change: [-0.5265% -0.4906% -0.4550%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
  3 (3.00%) high mild
  5 (5.00%) high severe

cast int32 to uint32 512
                        time:   [12.053 us 12.055 us 12.057 us]
                        change: [+126.57% +126.69% +126.87%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

cast int32 to float32 512
                        time:   [12.306 us 12.309 us 12.312 us]
                        change: [+115.72% +115.86% +115.97%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

cast int32 to float64 512
                        time:   [12.389 us 12.391 us 12.393 us]
                        change: [+133.82% +133.93% +134.04%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  4 (4.00%) high mild
  3 (3.00%) high severe

cast int32 to int64 512 time:   [12.402 us 12.405 us 12.407 us]
                        change: [+126.67% +126.81% +126.92%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

cast float32 to int32 512
                        time:   [12.765 us 12.767 us 12.768 us]
                        change: [+114.01% +114.11% +114.20%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low mild
  6 (6.00%) high mild

cast float64 to float32 512
                        time:   [12.522 us 12.524 us 12.526 us]
                        change: [+133.22% +133.35% +133.47%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

cast float64 to uint64 512
                        time:   [13.187 us 13.191 us 13.194 us]
                        change: [+100.11% +100.20% +100.30%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

cast int64 to int32 512 time:   [11.755 us 11.758 us 11.761 us]
                        change: [+122.67% +122.77% +122.87%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

cast date64 to date32 512
                        time:   [23.245 us 23.248 us 23.252 us]
                        change: [+37.888% +37.929% +37.969%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe

cast date32 to date64 512
                        time:   [22.948 us 22.951 us 22.954 us]
                        change: [+35.232% +35.391% +35.493%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

cast time32s to time32ms 512
                        time:   [1.1747 us 1.1750 us 1.1753 us]
                        change: [-34.441% -34.422% -34.402%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  1 (1.00%) high severe

cast time32s to time64us 512
                        time:   [13.960 us 13.961 us 13.962 us]
                        change: [+79.328% +79.399% +79.468%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

cast time64ns to time32s 512
                        time:   [25.761 us 25.765 us 25.768 us]
                        change: [+19.682% +19.723% +19.766%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe

cast timestamp_ns to timestamp_s 512
                        time:   [29.816 ns 29.825 ns 29.834 ns]
                        change: [+0.3917% +0.4310% +0.4744%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 8 outliers among 100 measurements (8.00%)
  6 (6.00%) high mild
  2 (2.00%) high severe

cast timestamp_ms to timestamp_ns 512
                        time:   [1.5295 us 1.5299 us 1.5304 us]
                        change: [-17.402% -17.337% -17.272%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

cast timestamp_ms to i64 512
                        time:   [176.67 ns 176.84 ns 177.19 ns]
                        change: [-1.7586% -1.6436% -1.4924%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  4 (4.00%) high mild
  2 (2.00%) high severe

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/comparison_kernels-f8939ec12975f45e
eq Float32              time:   [36.413 us 36.429 us 36.447 us]
                        change: [-95.357% -95.353% -95.349%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

eq scalar Float32       time:   [33.540 us 33.551 us 33.562 us]
                        change: [-94.698% -94.690% -94.684%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe

neq Float32             time:   [36.756 us 36.768 us 36.781 us]
                        change: [-94.049% -94.047% -94.045%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
  4 (4.00%) high mild
  2 (2.00%) high severe

neq scalar Float32      time:   [30.706 us 30.718 us 30.734 us]
                        change: [-95.092% -95.089% -95.086%] (p = 0.00 < 0.05)
                        Performance has improved.

lt Float32              time:   [36.489 us 36.498 us 36.509 us]
                        change: [-94.514% -94.504% -94.495%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe

lt scalar Float32       time:   [30.855 us 30.871 us 30.892 us]
                        change: [-94.996% -94.993% -94.990%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

lt_eq Float32           time:   [36.478 us 36.492 us 36.508 us]
                        change: [-94.700% -94.697% -94.695%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

lt_eq scalar Float32    time:   [32.641 us 32.653 us 32.668 us]
                        change: [-95.305% -95.299% -95.292%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

gt Float32              time:   [36.647 us 36.658 us 36.672 us]
                        change: [-94.121% -94.119% -94.116%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

gt scalar Float32       time:   [36.541 us 36.562 us 36.583 us]
                        change: [-94.288% -94.280% -94.273%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) low mild

gt_eq Float32           time:   [36.510 us 36.524 us 36.540 us]
                        change: [-95.402% -95.396% -95.390%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

gt_eq scalar Float32    time:   [32.121 us 32.141 us 32.163 us]
                        change: [-94.877% -94.875% -94.873%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
  12 (12.00%) low mild
  1 (1.00%) high mild

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/csv_writer-25ed34d22309b4ac
record_batches_to_csv   time:   [80.056 us 80.180 us 80.318 us]
                        change: [+0.3760% +0.6016% +0.8372%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) low mild
  5 (5.00%) high mild
  2 (2.00%) high severe

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/equal-df2ef777e50b709e
equal_512               time:   [42.140 ns 42.144 ns 42.150 ns]
                        change: [-0.2711% -0.2454% -0.2186%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
  3 (3.00%) high mild
  2 (2.00%) high severe

equal_nulls_512         time:   [5.1449 us 5.1573 us 5.1706 us]
                        change: [+44.552% +44.886% +45.227%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

equal_string_512        time:   [61.514 ns 61.519 ns 61.525 ns]
                        change: [+0.4486% +0.5006% +0.5374%] (p = 0.00 < 0.05)
                        Change within noise threshold.
Found 13 outliers among 100 measurements (13.00%)
  1 (1.00%) low severe
  5 (5.00%) high mild
  7 (7.00%) high severe

equal_string_nulls_512  time:   [5.6210 us 5.6224 us 5.6246 us]
                        change: [+21.881% +21.981% +22.074%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 12 outliers among 100 measurements (12.00%)
  1 (1.00%) low mild
  5 (5.00%) high mild
  6 (6.00%) high severe

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/filter_kernels-2401e6818fe1ed21
filter u8 low selectivity
                        time:   [166.39 us 166.42 us 166.45 us]
                        change: [+48.953% +49.302% +49.925%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe

filter u8 high selectivity
                        time:   [22.850 us 22.854 us 22.858 us]
                        change: [+456.68% +457.63% +459.36%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  4 (4.00%) high severe

filter u8 very low selectivity
                        time:   [29.303 us 29.309 us 29.316 us]
                        change: [+178.25% +178.35% +178.44%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild

filter context u8 low selectivity
                        time:   [146.14 us 146.35 us 146.58 us]
                        change: [+32.621% +32.779% +32.988%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 17 outliers among 100 measurements (17.00%)
  2 (2.00%) high mild
  15 (15.00%) high severe

filter context u8 high selectivity
                        time:   [2.8556 us 2.8600 us 2.8651 us]
                        change: [+9.7379% +9.9672% +10.234%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 25 outliers among 100 measurements (25.00%)
  4 (4.00%) low mild
  10 (10.00%) high mild
  11 (11.00%) high severe

filter context u8 very low selectivity
                        time:   [9.2290 us 9.2300 us 9.2312 us]
                        change: [+2.5952% +2.6384% +2.7096%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe

filter context u8 w NULLs low selectivity
                        time:   [188.52 us 188.57 us 188.62 us]
                        change: [+41.939% +41.998% +42.073%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) low severe
  1 (1.00%) high mild
  1 (1.00%) high severe

filter context u8 w NULLs high selectivity
                        time:   [3.2570 us 3.2574 us 3.2578 us]
                        change: [+9.3361% +9.3846% +9.4290%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  3 (3.00%) high mild
  3 (3.00%) high severe

filter context u8 w NULLs very low selectivity
                        time:   [177.11 us 177.14 us 177.17 us]
                        change: [+13.701% +13.737% +13.774%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  2 (2.00%) high severe

filter context f32 low selectivity
                        time:   [170.99 us 171.02 us 171.05 us]
                        change: [+17.562% +17.610% +17.667%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 12 outliers among 100 measurements (12.00%)
  2 (2.00%) low mild
  5 (5.00%) high mild
  5 (5.00%) high severe

filter context f32 high selectivity
                        time:   [3.1712 us 3.1720 us 3.1730 us]
                        change: [+6.1956% +6.3237% +6.4579%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) low severe
  2 (2.00%) low mild
  2 (2.00%) high mild
  2 (2.00%) high severe

filter context f32 very low selectivity
                        time:   [21.043 us 21.046 us 21.048 us]
                        change: [-1.2631% -1.2246% -1.1980%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/length_kernel-4e640ebc955ede9d
length                  time:   [34.337 us 34.348 us 34.359 us]
                        change: [+9.6250% +9.6772% +9.7281%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/sort_kernel-aeee7d9710cfd390
sort 2^10               time:   [424.17 us 424.21 us 424.25 us]
                        change: [+150.38% +150.44% +150.48%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 14 outliers among 100 measurements (14.00%)
  4 (4.00%) high mild
  10 (10.00%) high severe

sort 2^12               time:   [2.0103 ms 2.0105 ms 2.0108 ms]
                        change: [+134.93% +134.98% +135.03%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) high mild
  6 (6.00%) high severe

sort nulls 2^10         time:   [517.68 us 517.70 us 517.73 us]
                        change: [+234.77% +234.84% +234.90%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 12 outliers among 100 measurements (12.00%)
  1 (1.00%) low mild
  9 (9.00%) high mild
  2 (2.00%) high severe

sort nulls 2^12         time:   [2.4546 ms 2.4548 ms 2.4551 ms]
                        change: [+232.12% +232.39% +232.56%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 11 outliers among 100 measurements (11.00%)
  4 (4.00%) high mild
  7 (7.00%) high severe

     Running /home/vertexclique/projects/arrow/rust/target/release/deps/take_kernels-87f6d0017a97f9d0
take i32 512            time:   [9.8257 us 9.8271 us 9.8287 us]
                        change: [+302.80% +302.94% +303.11%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

take i32 1024           time:   [30.237 us 30.365 us 30.539 us]
                        change: [+583.96% +585.97% +588.58%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
  1 (1.00%) low mild
  4 (4.00%) high mild
  2 (2.00%) high severe

take i32 nulls 512      time:   [9.8708 us 9.8723 us 9.8738 us]
                        change: [+280.95% +281.07% +281.18%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) low mild
  2 (2.00%) high mild
  1 (1.00%) high severe

take i32 nulls 1024     time:   [30.473 us 30.540 us 30.616 us]
                        change: [+605.48% +606.87% +608.70%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 8 outliers among 100 measurements (8.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  4 (4.00%) high severe

take bool 512           time:   [11.558 us 11.561 us 11.563 us]
                        change: [+330.29% +332.21% +333.55%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 19 outliers among 100 measurements (19.00%)
  12 (12.00%) low mild
  4 (4.00%) high mild
  3 (3.00%) high severe

take bool 1024          time:   [34.171 us 34.219 us 34.275 us]
                        change: [+611.75% +613.60% +616.34%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
  1 (1.00%) high mild
  3 (3.00%) high severe

take bool nulls 512     time:   [12.292 us 12.295 us 12.298 us]
                        change: [+169.77% +170.18% +170.59%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
  7 (7.00%) low mild
  4 (4.00%) high mild
  2 (2.00%) high severe

take bool nulls 1024    time:   [32.251 us 32.353 us 32.466 us]
                        change: [+222.43% +223.62% +224.93%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
  1 (1.00%) low severe
  7 (7.00%) high mild
  5 (5.00%) high severe

take str 512            time:   [18.346 us 18.347 us 18.349 us]
                        change: [+380.69% +380.91% +381.13%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

take str 1024           time:   [48.017 us 48.128 us 48.283 us]
                        change: [+611.88% +613.44% +615.32%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe

take str nulls 512      time:   [20.669 us 20.673 us 20.678 us]
                        change: [+296.16% +296.32% +296.48%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  4 (4.00%) high mild
  1 (1.00%) high severe

take str nulls 1024     time:   [50.519 us 50.588 us 50.667 us]
                        change: [+369.23% +370.23% +371.36%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 10 outliers among 100 measurements (10.00%)
  4 (4.00%) high mild
  6 (6.00%) high severe

@vertexclique did you work on this on top of #8663 from @jhorstmann (or at least both PRs remove the popcnt table)?
The changes here look reasonable to me, so I can review after a rebase, as I've just merged #8663

This is using safe bitvec interface to manage bits. I worked on top of my existing PR. Since that didn't change all code to use safe operations, now this does. I squashed all my work into a single commit and closing PR: https://github.com/apache/arrow/pull/8598
Closed that with comment:  https://github.com/apache/arrow/pull/8598#issuecomment-727294219

Super cool. Great work @vertexclique ! 💯  I am all in in removing that unsafe code. 👍
Just curious, do you know why it has a 120% hit in performance on a filtering op?
filter u8 high selectivity
                        time:   [18.523 us 18.529 us 18.535 us]
                        change: [+122.09% +122.22% +122.33%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild

Finished, I have addressed all the comments. @jorgecarleitao I saw that it is because of the unaligned bit checks and boundary checks in byte slices. Yet another addition comes from as_buffer method. Since we are allocating a buffer for it now to make it well aligned this is creating an overhead. I have tried to overcome that problem by writing unowned buffer but one test failed. I am also unsure that test is actually testing correctly or not.

I ran the filtering selectivity benchmark on my laptop again and this time it shows a ~400% slowdown...
I double checked that I could reproduce the faster master numbers and I can

Weird, on my machine when I pushed the initial implementation of this PR I got the numbers above. Seems like it is regressed for me too.

So any idea how to proceed with this? Based on that I will close all related PRs. Though, seems like https://github.com/apache/arrow/pull/8688 can use these changes to build slice alignment on top.

If this is not stalling any other PRs, you can leave it open. I've started looking into what's causing the regression, but I'll only finish on the weekend (I'm mostly working on the Parquet writer with my evening time).
I'm also seeing big regressions of 400%+

@nevi-me @alamb @jorgecarleitao
The good news, I have found a solution for the performance related considerations. I have experimented on the sum and my roofline analysis brought some good results. Also, criterion benches are here:
Before:
sum 2^20                time:   [900.13 us 902.01 us 904.02 us]
Found 9 outliers among 100 measurements (9.00%)
  2 (2.00%) low mild
  3 (3.00%) high mild
  4 (4.00%) high severe

sum nulls 2^20          time:   [2.5859 ms 2.5909 ms 2.5967 ms]
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) high mild
  6 (6.00%) high severe

After:
sum 2^20                time:   [236.61 us 238.02 us 239.58 us]
                        change: [-73.888% -73.699% -73.493%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low mild
  5 (5.00%) high mild
  5 (5.00%) high severe

sum nulls 2^20          time:   [549.14 us 551.39 us 554.07 us]
                        change: [-78.784% -78.671% -78.548%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
  7 (7.00%) high mild
  5 (5.00%) high severe

Since it is a time-consuming task, I am not going to perform a full rewrite until we agree that this performance improvement is enough.

Thanks @vertexclique, that looks great. I'm here with the approach

Ok, so crucial operations are improved. I've updated the benchmarks. Since benchmarks with 512 elements fit most caches it creates unstable benchmarks. Kernels only can get better after this PR got merged and they are rewritten with parallel iterators. Feel free to benchmark this PR.
@nevi-me @jorgecarleitao @alamb

Really impressive improvement, @vertexclique .
@nevi-me , can I have 12h to review it?

@nevi-me , can I have 12h to review it?

Yup, I more meant that if someone else picks up things they'd like addressed, we could open JIRAs for them instead of trying to address them as part of this PR (unless they're clear blockers). In any case, we're still weeks or at least 2 months before the next release, so we still have time even for potential blockers.

Introducing thread parallelism at a compute kernel level

Conceptually this is not a small change. Personally I think parallelizing on the datafusion level and keeping kernels single-threaded is the better model.
The benchmark is now also testing a much larger array (2^20 elements) than what is usually used as a chunk size, so in reality the speedup due to parallelism would be much smaller.
I'm totally fine with a small slowdown if that leads to cleaner and safer code.

Conceptually this is not a small change. Personally I think parallelizing on the datafusion level and keeping kernels single-threaded is the better model.

I don't think personally, I am experimenting and working on that based on proofs. It is not a better model, even Intel says that. So there is still no conflict for further parallelizing in the DataFusion (which is already done via reactor). We are using arrow in the company. and yet still, we haven't parallelized chunks which is also wrong in many directions.

The benchmark is now also testing a much larger array (2^20 elements) than what is usually used as a chunk size, so in reality the speedup due to parallelism would be much smaller.

There are no predetermined chunk sizes and benchmarks shouldn't work on cache fittable data at all. That's against the benchmarking rule. Otherwise, you don't see the actual processing time, at all. Moreover, if this data format can't process large data on demand, there is no point in having this data format. (Users will think like that even I don't, which is also the most valid thing out there.) Also please don't alter my comments/words or use them against the pr.
Please also stop undermining others' work and be honest about what has been done. I have already told you to do this way but you preferred sharing raw pointers in an unsafe context, which won't and never ever going to be parallelized without a full rewrite. This is that full rewrite.
Finally, It is entirely unethical and dishonest to open PRs that I told/taught to you. I am not your rival, I don't see myself as a rival to anyone in any form, because there is no rivalry. This is an open-source environment. I don't see goodwill from your comments too. Also, some of your comments are giving false information (which I stopped giving feedback). I prefer instead of having counterproductive comments, productive comments from you. Even I told this plenty of times seems like that's not going to happen. So, please don't comment and review my PRs. Thanks.

Things to do after this pr:

Other kernels can be improved by different prs

I have already mentioned that here I think: https://github.com/apache/arrow/pull/8664#issuecomment-733276442

Conceptually this is not a small change. Personally I think parallelizing on the datafusion level and keeping kernels single-threaded is the better model.

I agree with @jhorstmann 's opinion here -- I think we should keep arrow single threaded (parallelizing their invocation across record batches can be done at a higher level).
One challenge of adding parallelism like rayon at a lower level such as the kernel is that the higher level libraries or applications lose control of the resources (for example, if some app wanted a particular background operation with only  a single core on a multi core, they couldn't do that easily with the code in this PR anymore)

Also, some of your comments are giving false information (which I stopped giving feedback). I prefer instead of having counterproductive comments, productive comments from you. Even I told this plenty of times seems like that's not going to happen. So, please don't comment and review my PRs. Thanks.

@vertexclique This really isn't acceptable behavior. It is against the project's code of conduct [1] to insult or harass other contributors. We don't do that here.
[1] https://www.apache.org/foundation/policies/conduct

Personally, I think violation against me also refers to the code of conduct even in two occurrences:

Be empathetic, welcoming, friendly, and patient. We work together to resolve conflict, assume good intentions, and do our best to act in an empathetic fashion. We may all experience some frustration from time to time, but we do not allow frustration to turn into a personal attack. A community where people feel uncomfortable or threatened is not a productive one. We should be respectful when dealing with other community members as well as with people outside our community.

I tried my best to assume good intentions. Even there is no problem in PRs I have received comments which are not meant to bring any productivity but changing the entire aim of the PR.
https://github.com/apache/arrow/pull/8609#discussion_r523458622
https://github.com/apache/arrow/pull/8665#issuecomment-727847232
But preventing PRs and keep raising debates(which are not related to the aim of the PR) in every PR is not what I expect as neither empathetic nor welcoming behavior.

Be collaborative. Our work will be used by other people, and in turn we will depend on the work of others. When we make something for the benefit of the project, we are willing to explain to others how it works, so that they can build on the work to make it even better. Any decision we make will affect users and colleagues, and we take those consequences seriously when making decisions.

Which I did:

explained the safety of the bit operations
the other features that previous pr (https://github.com/apache/arrow/pull/8598) and this pr (https://github.com/apache/arrow/pull/8664) brought in. They come intrinsically with the change.
And in turn, it turned into a benchmarking problem:
https://github.com/apache/arrow/pull/8598#issuecomment-723326224
and the actual point of all these has been missed. I have tried to explain it in various ways but some got me, some didn't.

I was thinking 3 months ago the project was going in a nice direction, I think it got prevented somehow and now it is not going very well from my point of view.
Moreover in general also Apache Foundation's values described here not quite held in my opinion. https://www.apache.org/foundation/how-it-works.html#meritocracy
So, finally, I would like to say that, I am totally ok with not committing to this project. Feel free to close this PR.

Hi @vertexclique.
All your contributions are very much appreciated.  You are one of the most advanced contributors to the project meaning that it's important for the other members of the community to be able to review and ask questions.  This may be frustrating for you but it is necessary.
In general, smaller more focused PR's will be easier to merge.  Larger PR's or ones that are low level in nature need to be reviewed in more depth and we should be glad to have @jhorstmann and others to provide constructive feedback.  Also, you may be asked to make changes to your PR's and you may not always agree.  This can always happen in a community run project.
In general, you need to not take feedback so personally.  I have re-read the interactions you mentioned and I'm sorry I can't see the issues you are referring to and I think you were wrong to call out @jhorstmann.  On this PR I agreed with @jhorstmann's perspective, as did @alamb.  I at least thought the question was a good one to ask.
On the other PR's you linked to it's clear that @jhorstmann is doing his best to be polite while providing feedback, for example:

Just an idea, ...
Nice performance improvement! I'm a bit surprised...

Hopefully, you can move past this and keep contributing to the project.

I think there are some very interesting things in this PR:

Usage of bitvec / new structure for null buffer. I think it makes sense to use this library here rather than reinvent it.
For the benchmarks it makes sense to have some bigger / more realistic ones as well. 2 ^ 20 maybe is a bit big. We also have some benchmarks in datafusion / benchmarks directory which can be extended to cover more realistic scenario's.
For parallelism,  I am also not convinced that it's a good idea to introduce rayon without being able to turn it off / control it. For big arrays it can be a good idea, but for smaller arrays, projects like datafusion and libraries, it can actually slow it down and/or use more resources overall. I think it would maybe be nice to revisit this sometime and see if we can make it an optional dependency (it's pretty big) and you could opt-in to use it for some kernels / ops?

Would love if this PR would be continued, maybe in a slimmed down form?

I agree with @Dandandan 's comments 👍

@vertexclique  --  Given the imminent Arrow 3.0 release, I am trying to clean up older Rust PRs and see if the authors have plans to move them forward.
Do you plan on working on this PR in the near future? If not, should we close this PR until there is time to make progress? Thanks again for your contributions so far.

@vertexclique  --  Given the imminent Arrow 3.0 release, I am trying to clean up older Rust PRs and see if the authors have plans to move them forward.
Do you plan on working on this PR in the near future? If not, should we close this PR until there is time to make progress? Thanks again for your contributions so far.

@alamb Thanks for reaching out! I don't have time to work on these PRs. Closing.

@alamb Thanks for reaching out! I don't have time to work on these PRs. Closing.


GM: Set new longitudinal scaling

Description
See https://github.com/commaai/opendbc/pull/951 for a description of the issue. See https://github.com/commaai/panda/pull/1905, https://github.com/commaai/opendbc/pull/1025 for submodule PRs.
Trivial changes: scaled all values according to the formula f(x) = x + (4096*5) - 22534 which is the new scaling defined in the DBC file.
Nontrivial change: using the above formula, f(ZERO_GAS) == -6. I figured that since this is quite close to 0, it is likely that our previous value of 2048 was a close but incorrect guess at the true zero value. I removed the reference to ZERO_GAS and used 0 instead.
Verification
Marking WIP to validate with the segments dataset as suggested by @sshane

I have validated these changes on the commaCarSegments dataset, see here. I was a bit unsure what exactly to be looking for, but in my tests nothing looks awry. I was unable to validate a few of the platforms since they have no segments with valid carParams.

dc7716b32bf25574/00000000--b86195eae6/0
dc7716b32bf25574/00000001--a72cadb253/0
dc7716b32bf25574/00000002--df117bd704/0

route with this branch and stock ACC
dc7716b32bf25574/00000004--b140f4714a/0

2023 Chevy Malibu in Dashcam Mode:
df51b2195de66a9b/00000000--45c2af4e69/0
2018 GMC Acadia:
df51b2195de66a9b/00000001--797163b3ba/0
df51b2195de66a9b/00000002--af4d93bcfb/0
df51b2195de66a9b/00000004--64c04ece5b/0

what else do we need to have this merged? POC version currently in FrogPilot with the corrected limit (which need this PR first) makes OP Long very usable

We've moved the car interfacing code to our opendbc repository, which is now the new home for car ports and fingerprints. Please re-open your pull request against opendbc at your convenience by using this command below. This will transform all changes under selfdrive/car/ to opendbc_repo/opendbc/car/. Make sure you have initialized submodules beforehand and have checked out the latest opendbc commit.
PR_NUMBER=33045
curl -L https://github.com/commaai/openpilot/pull/$PR_NUMBER.patch | sed -e 's/selfdrive\/car/opendbc_repo\/opendbc\/car/g' | git apply -v --reject

Simply replace the PR number with your own. Once done, add the files, fix any conflicts, and open a new PR. Alternatively, you may start a new PR from scratch if that is easier for you.


cupSODA: throw an error if model contains functional rate laws

cupSODA only supports mass-action kinetics so PySB should throw an error if a user tries to use it with a model that has functional rate laws, i.e., Expression objects used as rates in rules.

Implemented in 590e29d


chore(deps): update terraform terraform-ibm-modules/icd-postgresql/ibm to v3.17.2

This PR contains the following updates:

Package
Type
Update
Change

terraform-ibm-modules/icd-postgresql/ibm (source)
module
patch
3.17.1 -> 3.17.2

Release Notes

terraform-ibm-modules/terraform-ibm-icd-postgresql (terraform-ibm-modules/icd-postgresql/ibm)
v3.17.2
Compare Source
Bug Fixes

deps: update terraform terraform-ibm-modules/kms-all-inclusive/ibm to v4.15.9 (#483) (6ac4dd9)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 4.3.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Error when calling optional function prop with defined defaultProp

http://flow.org/try/#example
import React, { Component } from 'react';

type Props = {
  onFocus?: () => void,
};

class Input extends Component<Props> {
  static defaultProps = {
    onFocus: () => {},
  };

  handleFocus = () => {
    const { onFocus } = this.props;
    // do something here and then...
    onFocus();
  };

  render() {
    return <input type="text" onFocus={this.handleFocus} />
  }
}

    onFocus();
        ^ Cannot call `onFocus` because undefined [1] is not a function.

I'm fully expecting my response to be that the Flow isn't aware of defaultProps but thought I'd throw it out there. How do people tackle this? I was hoping not to have to wrap all my event props in a conditional before calling them 😕

You can solve this by making onFocus not optional in Props:
type Props = {
  onFocus: () => void,
};

// [rest of your code]

const examples = [
  // works (good)
  <Input />,
  // works (good)
  <Input onFocus={() => void console.log("Focused!")} />,
  // type error (good)
  <Input onFocus={3} />,
];

(Link to example.)
You can think of the Props type as specifying the interface for
this.props as used inside the component, not the exact props that
need to be provided to construct the component. It’s a bit
counterintuitive that this is the way that you’re supposed to do it,
but it is at least documented.

Thanks for getting back to me @wchargin. I discovered this issue after posting too https://github.com/facebook/flow/issues/1660
It seems bizarre to mark/document something as required when it's not. Not a fan of the approach but if that's the only option for now, will go with it.


Fix width for Checkbox/Radio

PR includes

[x] Bug Fix

Related issues
Resolve #859
Related issues https://github.com/taskany-inc/issues/issues/2160
QA Instructions, Screenshots, Recordings

/canary


Don't force incoming HTTP requests to be over SSL

I assume this was accidentally changed during the Rails upgrade (the
latest version bump seemed to want to touch quite a lot of config).
The traffic to the actual Rails app in non-development environments is
internal to our cluster, so it won't be over SSL.

I've deployed this to integration and it booted and passed the smoke tests


Use Biome instead of ESLint/Prettier

Description
Is it possible to add an option to use Biome instead of Prettier or ESLint for the format and lint options?
Since it offers superior performance and does not require complex configurations, I would like to use Biome in my project and avoid conflicts.

@7nohe funny you bring this up as I was just looking at Biome for the first time yesterday. Should be possible.

@7nohe because I also have interest in this (and it was trivial to implement). I have created a PR to implement this. In the next release you will be able to use:
format: 'biome',
lint: 'biome'

@jordanshatford It's awesome. I'm looking forward to the release!

@jordanshatford We might need a @next or @beta branch to allow people to use the latest features before they're released and avoid flooding everyone else with x weekly releases


chore: Reduce the 2x workflow save on Semaphore scenario

Checklist:

[ ] Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
[ ] The title of the PR is (a) conventional, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
[ ] I've signed the CLA.
[ ] I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
[ ] My builds are green. Try syncing with master if they are not.
[ ] My organization is added to USERS.md.

I just merged this into my PR, and I think you need to sync with master as you don't have the latest mutex code.


Graphical glitch : overhead status box of monster sometimes shown at the far-top of the screen

As said in the title. When hovering a monster, I sometimes get its status shown at the top of the screen, behind the portrait. This is rather annoying :@ .
To get a save where you can hover over the monsters (the scenario shown in the image) : http://www.filedropper.com/showdownload.php/saveoverheadglitch
My max accepted resolution is in 1368 x 766 but my in-game resolution is 1280x1024x32 (can't change it seems; it may be a thing that lasted from my co8 install; no idea).
Image of me hovering the top troll (the others have their usual overhead status box)

I've looked into it, the cause is a miscalculated obj_f_3d_render_height. I think it also manifests itself as the radius bug and the rare render corruption bug, and the root cause seems to be a wrong or unexecuted transformation of a model vertex.
I can create a manual check for the renderheight for this specific purpose but we have to solve this at the root.


Create WeeklyReportTemplate.md

Easier for github instead of the docx file
https://github.com/openmainframeproject-internship/resources/blob/master/forms/Weekly Project Status.docx

Signed-off-by: wlchurch<EMAIL_ADDRESS>


New icon in menu bar

Spinning icon is cute but it reminder me web page with BLINK tags :-1:
After a while I get hypnotized.
Few ideas.
icon as light bulb, down-arrow like in safari,
Some might need 2 colors (green-red or black-grey...) others just 2 shapes ⚫⚪  ⬛⬜
top of the top... user-defined emoji or symbol
👣🐫 🏃🏻🚀⛵💡🚥🐠
Later if some needs, in-out traffic icons... starts to look like littlesntich ;-)
Or with 2 small counters in the menu bar
4 Now | N | ● | ▼
6 Recent | R | ○ | ▽

For me, current spinning icon is better than suggested icons.
It's way simpler, cleaner, and fit to purpose.
There are many options for monitor network in detail.
But I need simple one, and that's why I use Loading.

not a fan of spinner,  ⚫⚪ ⬛⬜ is good enough to me. :+1:


[main] Upgrade to latest dependencies

Cron -knative-prow-robot
/cc knative/networking-wg-leads
/assign knative/networking-wg-leads
Produced by: knative-sandbox/knobots/actions/update-deps
Details:
bumping knative.dev/pkg 4d62e1d...ca82d2b:
  > ca82d2b Add `NewProxyAutoTLSTransport` and `DialTLSWithBackOff` to support TLS proxy (# 2479)
  > e2b4d74 Update community files (# 2486)
bumping knative.dev/hack 0e0784b...02c525c:
  > 02c525c Update community files (# 167)

/lgtm
/approve


Nuru/fix asset selection

Hi,
Attached contained some changes that would address some of our trade issues, for your perusal. The issues are as reported https://trello.com/c/WajP2doJ/189-getting-subscription-limit-message-too-often and https://trello.com/c/tPduKcu6/187-multiple-trade-windows-a-blank-window-is-shown-while-closing-them-one-by-one. There are quite few refactoring that should enhance and make our trade page better.

Changes Unknown when pulling 4966b61ce8904eda391096a67d0a097435120d4b on nuruddeensalihu:nuru/fix_asset_selection into ** on binary-com:master**.

Coverage decreased (-0.03%) to 31.54% when pulling 859bf43f74f40a4e194de3c5c3d8b0d5f6a3742b on nuruddeensalihu:nuru/fix_asset_selection into 1b322f5412783437bf2624573313a8f1f7b63ecd on binary-com:master.

@borisyankov the reason behind the travis test failing would be address by qinwei PR. I noticed we now have a single page trade component(i.e trade TradeLayouts) and and made some changes to accommodates it as well. I think we still need this PR because it comes with some benefit. It saves us some unnecessary contract subscriptions, reduce the contract limit issue reported by bijan. and seems to speed up the trade a little bit in the process. Unless otherwise , i think its necessary.

@nuruddeensalihu try merge to clear both conflict and test failure

Coverage decreased (-0.08%) to 30.9% when pulling a698ad5129850395ef965b8005392451b3e5955f on nuruddeensalihu:nuru/fix_asset_selection into 2ff70f7ce9a9cbe6cda990909330a107e1154bd1 on binary-com:master.

@nuruddeensalihu
this PR aim to solve repeated price proposal subscription, right ?
Does it still happen now, since we no longer have new trade button, I just test and find out we do not subscribe more than necessary when we change layout. But we have some other subscription issue though, I am checking.
The problem with this PR is the use of an seems to be random props, and it now couple the FullTradeCard component to some random component, for this case it's TradesLayout
It means now if someone want to use FullTradeCard, it needs to handle price proposal separately, which is pretty random ~


[VOTE] Adopt New Code Review Guideline

Dear Community:
This is a formal vote to adopt the proposed improvements to the code review guidelines. Please see the proposed improvements here  https://discuss.tvm.apache.org/t/improve-code-review-guideline/10898
This is a formal voting thread for the community to adopt the improvements.
Please vote
Vote:
+1
0 - I don't feel strongly about it, but don't object
-1 - because..
The VOTE will open for a week.
This thread is mirrored to dev@, please vote by replying to this thread

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

https://github.com/apache/tvm/issues/8997


Includes seems to be broken?

Minimal .gitlab-ci.yml illustrating the issue
---
include: 'common-lib.yaml'

stages:
  - plan
  - change-notifications

plan:
  stage: plan
  image: alpine
  script:
    - echo "hi"

notify-on-change:
  stage: change-notifications
  image: !reference [.common/util-image]
  script:
    - echo "bye"

common-lib.yaml
.common/util-image: "alpine"

Expected behavior
That gitlab-ci-local will execute successfully
Host information
Arch Linux
gitlab-ci-local 4.41.1
Additional context
This has to my knowledge been working fine previously.
Currently errors out with the following stderr:
❯ gitlab-ci-local --list
TypeError: Cannot read properties of undefined (reading 'includes')
    at Job.get imageName [as imageName] (/home/johan/.nvm/versions/node/v18.13.0/lib/node_modules/gitlab-ci-local/src/job.ts:267:26)
    at new Job (/home/johan/.nvm/versions/node/v18.13.0/lib/node_modules/gitlab-ci-local/src/job.ts:211:18)
    at /home/johan/.nvm/versions/node/v18.13.0/lib/node_modules/gitlab-ci-local/src/parser.ts:147:29
    at Function.forEachRealJob (/home/johan/.nvm/versions/node/v18.13.0/lib/node_modules/gitlab-ci-local/src/utils.ts:52:13)
    at Parser.init (/home/johan/.nvm/versions/node/v18.13.0/lib/node_modules/gitlab-ci-local/src/parser.ts:135:15)
    at Function.create (/home/johan/.nvm/versions/node/v18.13.0/lib/node_modules/gitlab-ci-local/src/parser.ts:57:9)
    at handler (/home/johan/.nvm/versions/node/v18.13.0/lib/node_modules/gitlab-ci-local/src/handler.ts:91:18)
    at Object.handler (/home/johan/.nvm/versions/node/v18.13.0/lib/node_modules/gitlab-ci-local/src/index.ts:25:34)

This was reproduced in a completely new git repo with the following dir contents:
❯ ls -la
total 12
drwxr-xr-x  3 johan johan  120 May 30 09:27 .
drwxrwxrwt 77 root  root  9100 May 30 09:28 ..
-rw-r--r--  1 johan johan   29 May 30 09:25 common-lib.yaml
drwxr-xr-x  8 johan johan  260 May 30 09:25 .git
-rw-r--r--  1 johan johan  250 May 30 09:21 .gitlab-ci.yml

If I comment out the change-notifications stage/job, gitlab-ci-local executes successfully

Hello 😊
Your !reference usage seems unconventional, it took me time to understand what you tried to achieve.
Is there official documentation on this ?
And why don't you use (documented) extends keyword for that usage ?
Mostly curious.

It doesn't matter if the minimal example is "weird" @bcouetil
This works on gitlab.com, so should we :smile: Thanks for the "weird" finding @fiskhest
I can reproduce myself, this is a bug :bug:

@bcouetil https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags there is nothing weird about this usage. This was just a very minimalistic example, we are doing some way more convoluted things with refs than this.

This works on gitlab.com, so should we

If it works but is not documented, it is subject to disappear without notice, but if you happily follows actually working use cases, no pb from me, it was a concern for your time 😊

This was just a very minimalistic example, we are doing some way more convoluted things with refs than this.

OK, I also do fairly complex pipelines, even dynamic mono-repo pipelines, and I have never had the need to replace an extends by an undocumented !reference usage.

@bcouetil  https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags

This is indeed the documentation I know, and it says Use the !reference custom YAML tag to select keyword configuration from OTHER JOB sections, and your reference is not a job.
And even from jobs, not all sections works (at least until recently), for example rules where created in gitlab v12.8 and rules/!reference is possible only since gitlab v14.3.
Not to mention the single argument !reference, which, AFAIK, is not documented 😊

This is indeed the documentation I know, and it says Use the !reference custom YAML tag to select keyword configuration from OTHER JOB sections, and your reference is not a job.
And even from jobs, not all sections works (at least until recently), for example rules where created in gitlab v12.8 and rules:!reference is possible only since gitlab v14.3.
For the record, it is not yet supported for artifacts:path : https://gitlab.com/gitlab-org/gitlab/-/issues/351169
Not to mention the single argument !reference, which, AFAIK, is not documented blush

Ah, well, we use it all over the place as cross-document yaml anchors pretty much, and gitlab seem to support that usage. Even gitlab-ci-local have supported that usage pattern at some previous version since I had this working for a repo previously that all of a sudden reported errors.
@firecow much appreciated, I owe you a drink!

This works on gitlab.com, so should we

If it works but is not documented, it is subject to disappear without notice, but if you happily follows actually working use cases, no pb from me, it was a concern for your time 😊

This was just a very minimalistic example, we are doing some way more convoluted things with refs than this.

OK, I also do fairly complex pipelines, even dynamic mono-repo pipelines, and I have never had the need to replace an extends by an undocumented !reference usage.

@bcouetil  https://docs.gitlab.com/ee/ci/yaml/yaml_optimization.html#reference-tags

This is indeed the documentation I know, and it says Use the !reference custom YAML tag to select keyword configuration from OTHER JOB sections, and your reference is not a job.
And even from jobs, not all sections works (at least until recently), for example rules where created in gitlab v12.8 and rules:!reference is possible only since gitlab v14.3.
For the record, it is not yet supported for artifacts:path : https://gitlab.com/gitlab-org/gitlab/-/issues/351169
Not to mention the single argument !reference, which, AFAIK, is not documented 😊

I always, just check these issues on our gitlab, never occurred to read the Gitlab docs 🤣


Renamed icons to fix conflicts with other libraries.

What kind of change does this PR introduce? (check one with "x")

[x] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Other... Please describe:

What is the current behavior? (You can also link to an open issue here)
icon classes begin with icon- and font-family is set to !important. This is causing it to override the icon font-family in other libraries.
What is the new behavior?
Renamed icon-* to datatable-icon-*
Does this PR introduce a breaking change? (check one with "x")

[x] Yes
[] No

If this PR contains a breaking change, please describe the impact and migration path for existing applications: ...
If someone has specified their icon classes using [cssClasses] and they reference the old names, they will no longer display.
Other information:

Nice, my only concern are these lines:

The icon should be datatable-icon right, to prevent issues?

It could likely just be removed as datatable-icon-filter is specific enough. I don't see that it was used before as the wildcard selectors has the - suffix on them.

Awesome! thanks for your contrib!


Investigate changing extensionlist/extension mechanism naming to more closely match Office use

History
Office team provides CT_ExtensionList and CT_Extension types in xsd definitions and uses a uri attribute in the new child elements under CT_Extension to authorize the relationship between the child and parent.
    <x:tableColumn id="1" xr3:uid="{00000000-0010-0000-0000-000001000000}" name="Id" dataDxfId="11">
      <x:extLst>
        <x:ext xmlns:xlmsforms="http://schemas.microsoft.com/office/spreadsheetml/2023/msForms" uri="{FCC71383-01E1-4257-9335-427F07BE8D7F}">
          <xlmsforms:question id="id"></xlmsforms:question>
        </x:ext>
      </x:extLst>
    </x:tableColumn>

In the backend of the SDK (not seen to the public), we have been using specialized CT_<feature>ExtensionList/CT_<feature>Extension complex types to generate C# classes which emit the same structure as above but provide this parent/child relationship authorization at compile time with strong typing instead of runtime via guids.
NOTE:  here refers to the parent type/class feature that uses the extension children being added.
By creating specialized  CT_<feature>ExtensionList/CT_<feature>Extension derived C# classes, we created mutual exclusivity between the CT_Extension* derived properties that exist before extending the parent type and the new specialized CT_<feature>Extenstion* properties that we build after Office extends the parent type.
The exclusivity, comes from the fact that both the  CT_Extension* and CT_<feature>Extenstion* represent the same element as derived from OpenXmlCompositeElement. Because of this exclusivity, when we release a new child extension, we replace the existing CT_Extension* property in the parent with the new CT_<feature>Extension* property.
This replacement sometimes represents a breaking change in the SDK framework API.  In order to maintain semantic versioning we would have to maintain both  CT_Extension* and CT_<feature>Extenstion*  properties in the extended parent type and create a shim or migration pathway from the old to the new (until a major version).
Problem Statement
Avoid introducing a breaking change.

...when we release a new child extension, we replace the existing CT_Extension* property in the parent with the new CT_<feature>Extension* property.

To qualify this, only when we release a new child extension to a type that has never had a child extension added by Office.
Approaches
One-time Refactor
One approach to this would be to replace all existing references to CT_ExtensionList and CT_Extension in the base Office xsds used by the SDK with CT_<feature>ExtensionList and CT_<feature>Extension appropriate to their use in the parent types. This would provide the extension list structure to allow new children extension elements to be added without breaking the SDK API going forward. This approach would require a one-time breaking change across the SDK but would avoid breaking changes going forward. If Office introduces a new parent type or adds a parent type with a CT_ExtensionList property, we could intercept that before it releases as an SDK class and change it to the CT_<feature>ExtensionList model.
Redesign To Use CT_Extension*
This involves redesigning the SDK validation, particle and element production and consumption processing code to use the Office method of Uri guids to determine parent/extension child relationships at runtime and move away from strong type enforcement.
Other <TBD>

@twsouthwick attempted code for a shim
Obsolete/Shim Temporary Approach
To avoid breaking the API and requiring a new major version, we were aiming to have something like the below. TableColumnExtensionList would return its own base ExtensionList.  ExtensionList would operate as before except it would have the ObsoleteAttribute, letting developers know that they will need to stop using it at the next major version.
namespace DocumentFormat.OpenXml.Spreadsheet
{
    public partial class TableColumn : OpenXmlCompositeElement
    {
        /// <summary>
        /// <para>Gets or sets the legacy ExtensionList property.</para>
        /// <para>Future Extensibility.</para>
        /// <para>Represents the following element tag in the schema: x:extLst.</para>
        /// </summary>
        /// <remark>
        /// xmlns:x = http://schemas.openxmlformats.org/spreadsheetml/2006/main
        /// </remark>
        [ObsoleteAttribute("This property is obsolete and will be removed in a future version. Use the TableColumnExtension property instead.", false)]
        public DocumentFormat.OpenXml.Spreadsheet.ExtensionList? ExtensionList
        {
            get => GetElement<DocumentFormat.OpenXml.Spreadsheet.ExtensionList>();
            set => SetElement((OpenXml.Spreadsheet.ExtensionList?)value);
        }

        /// <summary>Gets or sets the table column extension list.</summary>
        /// <value>The table column extension list.</value>
        public DocumentFormat.OpenXml.Spreadsheet.TableColumnExtensionList? TableColumnExtensionList
        {
            get => GetElement<DocumentFormat.OpenXml.Spreadsheet.ExtensionList>();
            set => SetElement((OpenXml.Spreadsheet.ExtensionList?)value);
        }
    }

    /// <summary>
    /// <para>Future Extensibility.</para>
    /// <para>This class is available in Office 2007 and above.</para>
    /// <para>When the object is serialized out as xml, it's qualified name is x:extLst.</para>
    /// </summary>
    /// <remark>
    /// <para>The following table lists the possible child types:</para>
    /// <list type="bullet">
    ///   <item><description><see cref="DocumentFormat.OpenXml.Spreadsheet.TableColumnExtension" /> <c><x:ext></c></description></item>
    /// </list>
    /// </remark>
    public partial class TableColumnExtensionList : ExtensionList
    {
        public TableColumnExtensionList()
            : base()
        {
        }

        public TableColumnExtensionList(IEnumerable<OpenXmlElement> childElements)
            : base(childElements)
        {
        }

        public TableColumnExtensionList(params OpenXmlElement[] childElements)
            : base(childElements)
        {
        }

        public TableColumnExtensionList(string outerXml)
            : base(outerXml)
        {
        }

        internal override void ConfigureMetadata(ElementMetadata.Builder builder)
        {
            // base.ConfigureMetadata(builder);

            builder.SetSchema("x:extLst");
            builder.AddChild<DocumentFormat.OpenXml.Spreadsheet.TableColumnExtension>();
            builder.Particle = new CompositeParticle.Builder(ParticleType.Sequence, 1, 1)
            {
                new ElementParticle(typeof(DocumentFormat.OpenXml.Spreadsheet.TableColumnExtension), 0, 0),
            };
        }

        public override OpenXmlElement CloneNode(bool deep) => CloneImp<TableColumnExtensionList>(deep);
    }

    /// <summary>
    /// <para>Defines the TableColumnExtension Class.</para>
    /// <para>This class is available in Office 2007 and above.</para>
    /// <para>When the object is serialized out as xml, it's qualified name is x:ext.</para>
    /// </summary>
    /// <remark>
    /// <para>The following table lists the possible child types:</para>
    /// <list type="bullet">
    ///   <item><description><see cref="DocumentFormat.OpenXml.Office.SpreadSheetML.Y2023.MsForms.Question" /> <c><xlmsforms:question></c></description></item>
    /// </list>
    /// </remark>
    public partial class TableColumnExtension : OpenXmlCompositeElement
    {
        public TableColumnExtension()
            : base()
        {
        }

        public TableColumnExtension(IEnumerable<OpenXmlElement> childElements)
            : base(childElements)
        {
        }

        public TableColumnExtension(params OpenXmlElement[] childElements)
            : base(childElements)
        {
        }

        public TableColumnExtension(string outerXml)
            : base(outerXml)
        {
        }

        public StringValue? Uri
        {
            get => GetAttribute<StringValue>();
            set => SetAttribute(value);
        }

        internal override void ConfigureMetadata(ElementMetadata.Builder builder)
        {
            base.ConfigureMetadata(builder);
            builder.SetSchema("x:ext");
            builder.AddChild<DocumentFormat.OpenXml.Office.SpreadSheetML.Y2023.MsForms.Question>();
            builder.AddElement<TableColumnExtension>()
                .AddAttribute("uri", a => a.Uri, aBuilder =>
                {
                    aBuilder.AddValidator(RequiredValidator.Instance);
                    aBuilder.AddValidator(new StringValidator() { IsToken = (true) });
                });
            builder.Particle = new CompositeParticle.Builder(ParticleType.Choice, 1, 1)
            {
                new ElementParticle(typeof(DocumentFormat.OpenXml.Office.SpreadSheetML.Y2023.MsForms.Question), 1, 1, version: FileFormatVersions.Microsoft365),
                new AnyParticle(0, 1),
            };
        }

        /// <inheritdoc/>
        public override OpenXmlElement CloneNode(bool deep) => CloneImp<TableColumnExtension>(deep);
    }
}

Currently, even if we were able to get a shim like this to compile, the effect with our current particle/element constraints processing will not allow us to have choice between reading in and <extLst> element of type TableColumnExtensionList and ExtensionList.

Can you add the existing pattern?

What about if you change the following:
public DocumentFormat.OpenXml.Spreadsheet.ExtensionList? ExtensionList { get => TableExtensionList; set => TableExtensionList = value }

Of course with an appropriate conversion for the setter

Here's the test code we current are using. Of e1 and e2, one of them will always get null depending on the order of the AddChild's in the TableColumnExtensionList (see the third excerpt below).
    WorksheetPart sheet = (WorksheetPart)spd.WorkbookPart.GetPartById("rId1");
    if (sheet == null)
    {
        throw new ArgumentException();
    }

    TableDefinitionPart tblprt = (TableDefinitionPart)sheet.GetPartById("rId2");
    if (tblprt == null)
    {
        throw new ArgumentException();
    }

    TableColumn tc = tblprt.Table.TableColumns.GetFirstChild<TableColumn>();
    if (tc == null)
    {
        throw new ArgumentException("No columns found!");
    }

    DocumentFormat.OpenXml.Spreadsheet.Extension e1 = tc.ExtensionList.GetFirstChild<DocumentFormat.OpenXml.Spreadsheet.Extension>();
    DocumentFormat.OpenXml.Spreadsheet.TableColumnExtension e2 = tc.TableColumnExtensionList.GetFirstChild<DocumentFormat.OpenXml.Spreadsheet.TableColumnExtension>();

        [ObsoleteAttribute("This property is obsolete and will be removed in a future version. Use the TableColumnExtension property instead.", false)]
        public ExtensionList? ExtensionList
        {
            get => TableColumnExtensionList;
            set => TableColumnExtensionList = (TableColumnExtensionList?)value;
        }

        public TableColumnExtensionList? TableColumnExtensionList
        {
            get => GetElement<TableColumnExtensionList>();
            set => SetElement((TableColumnExtensionList?)value);
        }

Here are the AddChild methods that we would like to allow us to return either Extension or TableColumnExtension.
(We think the GetFirstChild would need to be modified to return either of these types for the same XML element i.e. <x:ext>)
    public partial class TableColumnExtensionList : ExtensionList
    {

...

        internal override void ConfigureMetadata(ElementMetadata.Builder builder)
        {
            // base.ConfigureMetadata(builder);

            builder.SetSchema("x:extLst");
            builder.AddChild<DocumentFormat.OpenXml.Spreadsheet.TableColumnExtension>();
            builder.AddChild<DocumentFormat.OpenXml.Spreadsheet.Extension>();
            builder.Particle = new CompositeParticle.Builder(ParticleType.Sequence, 1, 1)
            {
                new ElementParticle(typeof(DocumentFormat.OpenXml.Spreadsheet.TableColumnExtension), 0, 0),
            };
        }

If we only have the specialized type then the child of the ExtensionList is a TableColumnExtension

Is that a problem? Given the details in the original message, it's the same type, just specialized now, right?
Do you have any unit tests to demonstrate the behaviors we want? That way we can play around with the implementation.
Have you explored making use of the URI field? We should be able to parse that and surface that in the ExtensionList to know if it's the right thing.

Is that a problem? Given the details in the original message, it's the same type, just specialized now, right?

Yes unfortunately, becuase if someone had used the original API they would have written GetFirstChild<DocumentFormat.OpenXml.Spreadsheet.Extension>();, because TableColumnExtension did not exist at the time.

Do you have any unit tests to demonstrate the behaviors we want? That way we can play around with the implementation. (and of course they'll be failing for now)

Not yet, but we know how we want the API to behave, so we should be able to add some.

Have you explored making use of the URI attribute? We should be able to parse that and surface that in the ExtensionList to know if it's the right thing.

The TableColumnExtension and Extension have the same URI, so this won't help distinguish between them.

Have you explored making use of the URI attribute? We should be able to parse that and surface that in the ExtensionList to know if it's the right thing.

We have not gone down this road in our implementation approaches yet, but definitely can. For this approach we wouldn't need the specialized extension list class i.e. TableColumnExtensionList. If we add this approach, then we could provide it as the sole solution for new extension XML elements like <x:tableColumn>xlmsforms:question. But for all the existing extension elements that already also have specialized *ExtensionList classes, we would have to leave them untouched or add an ExtensionList property back into the parent (extended) element class to sit alongside the specialized *ExtentionList property. Not sure what all the ramifications of that would be but sounds feasible.

@twsouthwick Added unit test for API usage: https://github.com/tomjebo/Open-XML-SDK/commit/51d8c744a1fa868dada22f5357b8fb7d55b40eb2

@twsouthwick We did some brainstorming this morning and something like this could be an approach:
Whereas in the current API usage, we would have to do these:
#pragma warning disable CS0618 // Type or member is obsolete
                Extension e1 = tc.ExtensionList.GetFirstChild<Extension>();
#pragma warning restore CS0618 // Type or member is obsolete
                TableColumnExtension e2 = tc.TableColumnExtensionList.GetFirstChild<TableColumnExtension>();

Instead, if we add this kind of method to the ExtensionList class:
                Question q1 = tc.ExtensionList.GetChildExtension<Question>();

We could just avoid providing the specialized *ExtensionList and *Extension for the new extension element (i.e. <xlmsforms:question>. The GetChildExtension would use the type to get the uri guid associated with question and validate that it belongs to a TableColumn.ExtensionList.
Of course, we would then need to provide the code in our document consuming paths to validate XML blocks containing <question> under <extLst> and <ext>. So there would be a bit more work than just providing GetChildExtension but that would be the pattern for that scenario.

@twsouthwick  TableColumnExtensionList is added as a shim and no longer a breaking change for the API with our test branch.
https://github.com/tomjebo/Open-XML-SDK/tree/obsolete_extensions
We have tests in TableColumnExtensionListTests.cs for this workaround.

I couldn't find it in your branch (it's not searchable it seems). Can you provide a deep link? Or better update the API suggestion above with what it looks like?

The two test files that we currently have which show how the shims for TableColumn.ExtensionList and DifferentialType.ExtensionList will be used are here:
https://github.com/tomjebo/Open-XML-SDK/blob/obsolete_extensions/test/DocumentFormat.OpenXml.Framework.Tests/Particles/TableColumnExtensionListTests.cs
https://github.com/tomjebo/Open-XML-SDK/blob/obsolete_extensions/test/DocumentFormat.OpenXml.Framework.Tests/Particles/DxfExtensionListTests2006main.cs
I should clarify, that these do not represent the "new design" for the API but rather the obsoleted shim that will allow us to provide the new extension children until the next major release.
We will still need to iterate on the design going forward but this gets us past the breaking change for minor releases with new extension children.

Using TableColumn and the new Question element, here is the "diff" for the current schema update minor release that avoids breaking API changes:

namespace DocumentFormat.OpenXml.Spreadsheet
{
    public partial class TableColumn : OpenXmlCompositeElement
    {
+        [ObsoleteAttribute("This property is obsolete and will be removed in a future version. Use the TableColumnExtension property instead.", false)]
         public DocumentFormat.OpenXml.Spreadsheet.ExtensionList? ExtensionList { get; set; }

+        public DocumentFormat.OpenXml.Spreadsheet.TableColumnExtensionList? TableColumnExtensionList { get; set; }
    }

With this change, here is the usage pattern for reading the XML:
...
            using (SpreadsheetDocument spd = SpreadsheetDocument.Open(stream, false))
            {
                WorksheetPart sheet = (WorksheetPart)spd.WorkbookPart.GetPartById("rId1");

                TableDefinitionPart tblprt = (TableDefinitionPart)sheet.GetPartById("rId2");

                TableColumn tc = tblprt.Table.TableColumns.GetFirstChild<TableColumn>();

+#pragma warning disable CS0618 // Type or member is obsolete
+              Extension e = tc.ExtensionList.Descendants<Extension>().Where(e => e.Uri == "{FCC71383-01E1-4257-9335-427F07BE8D7F}").First();

+#pragma warning restore CS0618 // Type or member is obsolete

+              TableColumnExtension e2 = tc.TableColumnExtensionList.GetFirstChild<TableColumnExtension>();
+
+              Question question = e2.GetFirstChild<Question>();
+
+              question = e.GetFirstChild<Question>();
            }

With this change, here is the usage pattern for writing the XML:
            using (SpreadsheetDocument spd = SpreadsheetDocument.Open(stream, false))
            {
                WorksheetPart sheet = (WorksheetPart)spd.WorkbookPart.GetPartById("rId1");

                TableDefinitionPart tblprt = (TableDefinitionPart)sheet.GetPartById("rId2");

                TableColumn tc = tblprt.Table.TableColumns.GetFirstChild<TableColumn>();

                ExtensionList eList = new ExtensionList(
                    new Extension() { Uri = "{FCC71383-01E1-4257-9335-427F07BE8D7F}" });

                tc.AddChild(eList);

                // Or

+#pragma warning disable CS0618 // Type or member is obsolete
                tc.ExtensionList = eList;
+#pragma warning restore CS0618 // Type or member is obsolete

+                TableColumnExtensionList teList = new TableColumnExtensionList(
+                   new TableColumnExtension(
+                       new Question() { Id = "r8a22544ad01d478e898ac5748745f765" })
+                    { Uri = "{FCC71383-01E1-4257-9335-427F07BE8D7F}" });
+
+                tc.AddChild(teList);
+
+                // Or
+
+                tc.TableColumnExtensionList = teList;
       }

Here is a proposed "diff" if for a new design that operates like Office, i.e. runtime comparison of Uri for an extension child.

namespace DocumentFormat.OpenXml.Spreadsheet
{
    public partial class TableColumn : OpenXmlCompositeElement
    {
         public DocumentFormat.OpenXml.Spreadsheet.ExtensionList? ExtensionList { get; set; }
    }

With this change, here is the usage pattern for reading the XML:
...
            using (SpreadsheetDocument spd = SpreadsheetDocument.Open(stream, false))
            {
                WorksheetPart sheet = (WorksheetPart)spd.WorkbookPart.GetPartById("rId1");

                TableDefinitionPart tblprt = (TableDefinitionPart)sheet.GetPartById("rId2");

                TableColumn tc = tblprt.Table.TableColumns.GetFirstChild<TableColumn>();

                Extension e = tc.ExtensionList.Descendants<Extension>().Where(e => e.Uri == "{FCC71383-01E1-4257-9335-427F07BE8D7F}").First();

+              Question question = e.GetFirstChild<Question>();
+
+              // Or alternatively, a new method that encapsulates everthing including tc.ExtensionList.Descendants and Question:
+
+              Question question = tc.GetExtensionChild<Question>();
            }

With this change, here is the usage pattern for writing the XML:
            using (SpreadsheetDocument spd = SpreadsheetDocument.Open(stream, false))
            {
                WorksheetPart sheet = (WorksheetPart)spd.WorkbookPart.GetPartById("rId1");

                TableDefinitionPart tblprt = (TableDefinitionPart)sheet.GetPartById("rId2");

                TableColumn tc = tblprt.Table.TableColumns.GetFirstChild<TableColumn>();

                ExtensionList eList = new ExtensionList(
                    new Extension(
+                       new Question() { Id = "r8a22544ad01d478e898ac5748745f765" })
                       { Uri = "{FCC71383-01E1-4257-9335-427F07BE8D7F}" });

                tc.AddChild(eList);

                // Or

                tc.ExtensionList = eList;
+
+              // Or alternatively, a new method that encapsulates everything including new ExtensionList, AddChild, etc...:
+
+              Question question = tc.AddExtensionChild<Question>();
       }

For classes which contained previously generated specialized extensionlist properties, the usage pattern would switch from using (for example) TableExtensionList to just ExtensionList. To update code to this would be a breaking change (i.e. removing TableExtensionList/TableExtention):
   public partial class Table : OpenXmlPartRootElement
   {
-         public DocumentFormat.OpenXml.Spreadsheet.TableExtensionList? TableExtensionList { get; set; }
+        public DocumentFormat.OpenXml.Spreadsheet.ExtensionList? ExtensionList  { get; set; }
   }

Obviously we could use a shim for minor version releases to migrate folks.

OK - sounds like we'll continue with the pattern we had, but just not remove the extension and obsolete it?

OK - sounds like we'll continue with the pattern we had, but just not remove the extension and obsolete it?

Yes, maybe. That is option 1 and what is currently in PR #1708. 2 & 3 are other options for the API.
IMO Option 2 seems like a lot of work for no or very little benefit.
Continuing with option 1 and obsoleting ExtensionList and Extension until we hit a major release to remove them would allow us to continue with the same pattern to create FeaturExtensions and restrict their children.
But this is not actually what the standard says. According to ISO 2016-1 18.2.7 ext (Extension), any valid xml element can be a child of Extension:

Upon encountering extensions, a processing consumer shall determine whether it knows how to process extensions using the value of the uri. If the consumer knows how to process such an extension, the markup contained within that extension is processed. Otherwise, the extension content shall be preserved so long as the structure that contains the extLst has not been removed.

and the schema for CT_Extension is
<xsd:complexType name="CT_Extension">
  <xsd:sequence>
    <xsd:any processContents="lax"/>
  </xsd:sequence>
  <xsd:attribute name="uri" type="xsd:token"/>
</xsd:complexType>

If we go with option 3, we lose the child validation, but we won't have to make a specialized extension manually for every use of ExtenstionList and Extension and if we want to mirror the standard this is actually correct. It should also allow us to clean up a lot of code and avoid introducing bugs by manually patching.


StoriesProgressview doesn't respect durations

After settings the durations with storiesProgressView.setStoriesCountWithDurations(durations), one of the progress should be around 30s but instead is around 3s. If I Log.d durations[counter] it  logs the correct duration. What could be the issue

FYI the view that is being shown during that duration is a videoplayer

I've used this library and setStoriesCountWithDurations also don't work with me up to now. The second story would just be showing for a split second and it will finish the activity. Have you fixed your error before?

Any update on this ?


Proto is not a valid IDL type

I'm trying to create Scala files from proto definitions but I'm getting this error:
Unknown IDL type 'proto'. Valid values: avro
My settings:
idlType := "proto",
srcGenSerializationType := "Protobuf",
srcGenTargetDir := (Compile / sourceManaged).value / "compiled_proto",
sourceGenerators in Compile += (srcGen in Compile).taskValue,

Sadly, frees-rpc currently doesn't support code generation from proto files.

Then the description is wrong
lazy val idlType: SettingKey[String] =
      settingKey[String]("The IDL type to work with, such as avro or proto")

Should be possible now with this https://github.com/AdrianRaFo/proton

This is currently being implemented in Skeuomorph, and will be integrated into the Mu source-gen sbt plugin.
Tracked at: https://github.com/higherkindness/skeuomorph/issues/30


Animation support?

Heya, I noted your repo was slightly more active than the original.
Would you be keem in adding animation group support? or is that too hard?
Cheers

Or you can also use 1.x vrm, which supports vrm animation，If it's anything like threejs, it might take a lot of refactoring


Publish module in NPM

because some developers doesn't want to use bower for the only one module :[

Can you put together a pull request for this?

https://github.com/ksylvest/jquery-gridly/pull/48