Standardize arrow navigation in OptionsLists

Details

Creates the ArrowKeyFocusManager component, which takes a few generic props and uses the arrow keys to manage a focusedIndex state.
Removes the keyboard event handling from the OptionsSelector component in preference of the ArrowKeyFocusManager
Implements ArrowKeyFocusManager in the IOUConfirmationList component
Implements scrollToIndex correctly for sections lists
Includes several performance improvements

Notable changes
The IOU currency page used to have an inconsistent UX compared to our other list pages. This PR DRYs that, gives it support for arrow navigation, and makes the UX consistent with other list pages. The behavior before was:

When you open the IOU currency page, the full list appears, and the current currency is selected in the list (appears with a checkbox next to it).
Selecting another currency deselects the current currency and selects another
There is a "confirm" button that changes the currency to whatever is selected.

The new behavior is:

When you open the IOU currency page, the full list appears
Selecting another currency changes the currency to whatever is selected

Furthermore, it now supports mouseless use via arrow navigation / enter to submit.
Known Limitations

When a single item is selected, arrow navigation does not always scroll to the right position (the selected item is sometimes partially or fully covered)
On Android, selecting the first item from recents section, scrolls the list and hide the selected item

Fixed Issues
$ https://github.com/Expensify/App/issues/7648
PR Reviewer Checklist

[ ] I verified the Author pulled main before submitting the PR
[ ] I verified the correct issue is linked in the ### Fixed Issues section above
[ ] I verified testing steps are clear and they cover the changes made in this PR

[ ] I verified the testing environment is mentioned in the test steps

[ ] I verified testing steps cover success & fail scenarios (if applicable)
[ ] I verified tests pass on all platforms & I tested again on all platforms
[ ] I checked that screenshots or videos are included for tests on all platforms
[ ] I verified there are no console errors related to changes in this PR
[ ] I verified proper code patterns were followed (see Reviewing the code)

[ ] I verified comments were added when the code was not self explanatory
[ ] I verified any copy / text shown in the product was added in all src/languages/* files (if applicable)
[ ] I verified proper naming convention for platform-specific files was followed (if applicable)
[ ] I verified style guidelines were followed

[ ] I verified that this PR follows the guidelines as stated in the Review Guidelines

Tests / QA Steps
Log in to New Expensify.
Search Page
All Platforms

Click/tap on the magnifying glass icon to open the search page.
Click/tap on the x icon to close the search page.
Click/tap on the magnifying glass to open the search page.
Ensure you can scroll up and down in the results.
Enter a search query – the list should update accordingly.
Click/tap on a result: the search page should close and that chat should open.

Web/Desktop

Press CMD+K to open the search page.
Press Esc to close the search page.
Press CMD+K to reopen the search page.
Type in a search query.
Use the arrow keys to go through the search results. Make sure that the list "wraps", so:

If you are focused on the last result and press ArrowDown you are taken to the first result
If you are focused on the first result and press ArrowUp you are taken to the last result

With the focus somewhere in the middle of the list, update the search input such that there are no results.
Press the arrow keys a few times – nothing should happen.
Undo the change from the previous step – the focus should be where you left it and you should be able to use the arrows to navigate focus (as before)
With a specific item focused, update the search term such that the position of that item moves up or down in the list.
Verify that that same item remains focused (even though its position in the list changed).
With a chat focused on the search page, press Enter. The search page should close and that chat should open.
Use the mouse to click on the magnifying glass icon and reopen the search page.
Press Esc to close the search page.
Click on the magnifying glass to reopen the search page.
Use the arrow keys and the enter key to select and open a different chat.

New Chat Page
All Platforms

Click/tap on the green + icon on the home screen
Click/tap on New Chat.
Ensure that you can scroll the list of results.
Enter a search query and the results should update accordingly.
click/tap on a search result
The New Chat Page should close and a chat with your selected result should open.

Web/Desktop

Click on the green + icon on the home screen
Click on New Chat
Press ESC – the New Chat Page should close.
Click on the green + icon on the home screen.
Click on New Chat
Use the arrow keys to traverse the list of results. Make sure that the list "wraps", so:

If you are focused on the last result and press ArrowDown you are taken to the first result
If you are focused on the first result and press ArrowUp you are taken to the last result

With the focus somewhere in the middle of the list, update the search input such that there are no results.
Press the arrow keys a few times – nothing should happen.
Undo the change from the previous step – the focus should be where you left it and you should be able to use the arrows to navigate focus (as before)
With a result highlighted, press Enter and that chat should open.
Reopen the New Chat page and use the arrow keys to highlight a different chat than the one you have open.
Press CMD+Enter and that chat should open.

New Group Page
All Platforms

Click/tap on the green + icon on the home screen
Click/tap on New Group.
Ensure that you can scroll the list of results.
Enter a search query and the results should update accordingly.
Click/tap on a result. That result should become checked and move to the top of the results.
Click/tap on another result. That result should become checked and move to the top of the results.
Click/tap on Create Group. The New Group page should close and a chat with all the selected users should open.

Web/Desktop

Click on the green + icon on the home screen.
Click New Group. The New Group Page should open.
Press ESC. The New Group page should close.
Press CMD+SHIFT+K. The New Group Page should open.
Press ESC. The New Group page should close.
Reopen the New Group Page.
If necessary, repeat the following until there are enough elements in the New Group page that the results are scrollable:

Close the New Group page.
Open the New Chat page.
Create a chat with a user that you've not interacted with before.
Reopen the New Group page and check if it's scrollable. If not, go back to the first step.

Use the arrow keys to traverse the list of results. Make sure that the list "wraps". There is a notable difference in expected behavior between this page and the New Chat Page in that the search bar is one of the focusable elements.

If you are focused on the last result and press ArrowDown you are taken to the search bar. None of the elements should be highlighted.
If you are focused on the first result and press ArrowUp you are taken to the search bar. None of the elements should be highlighted.
If you are focused on the search bar and press ArrowUp, you are taken to the last result.
If you are focused on the search bar and press ArrowDown, you are taken to the first result.

With one of the search results highlighted, press Enter. That result should get checked and move to the top of the list.
Use the ArrowDown key to highlight the first unselected result. It should be completely visible.
Use the arrow keys and Enter to select multiple results.
Use the arrow keys to highlight a selected result.
Press Enter to de-select that result.
With a result highlighted, press CMD+Enter, and a group should be opened with your selected results.
Reopen the New Group Page and select several results.
Use the arrow keys to go to the search bar (no results highlighted).
Press CMD+Enter, and a group should be opened with your selected results.
Reopen the New Group Page and select several results.
Use the arrow keys to go to the search bar (no results highlighted).
Press Enter (not CMD+Enter), and a group should be opened with your selected results.

IOU Participants (split bill)
All Platforms

Click/tap on the green + icon on the home screen.
Click/tap on Split Bill
Enter an amount.
Ensure that you can scroll the list of results.
Enter a search query and the results should update accordingly.
Make sure you can toggle / untoggle list items.
With some items selected and others not, press Next.
Verify that you the selected items are shown on the next page.

Web/Desktop

Click/tap on the green + icon on the home screen.
Click/tap on Split Bill
Enter an amount and press Next.
Enter a search query to get a reasonable-length list of options.
Use the arrow keys to traverse the list of results. Make sure that the list "wraps":

If you are focused on the last result and press ArrowDown you are taken to the search bar. None of the elements should be highlighted.
If you are focused on the first result and press ArrowUp you are taken to the search bar. None of the elements should be highlighted.
If you are focused on the search bar and press ArrowUp, you are taken to the last result.
If you are focused on the search bar and press ArrowDown, you are taken to the first result.

With one of the search results highlighted, press Enter. That result should get checked and move to the top of the list.
Use the arrow keys and Enter to select multiple results.
Use the arrow keys to highlight a selected result.
Press Enter to de-select that result.
With a result highlighted, press CMD+Enter, and you should move to the next page with your selected results.
Press the back arrow to go to the previous page.
With the search bar focused (no results highlighted), press CMD+Enter, and you should move to the next page with your selected results.
Press the back arrow to go to the previous page.
With the search bar focused (no results highlighted), press Enter (not CMD+Enter), and you should move to the next page with your selected results.

IOU Participants (1:1 request)
All Platforms

Click/tap on the green + icon on the home screen.
Click/tap on Request Money
Enter an amount and press Next
Ensure that you can scroll the list of results.
Enter a search query and the results should update accordingly.
Click/tap on a result, and there should be a new IOU request between you and your selected user.

Web/Desktop

Click/tap on the green + icon on the home screen.
Click/tap on Request Money
Enter an amount and press Next
Enter a search query to get a reasonable-length list of results.
Use the arrow keys to traverse the list of results. Make sure that the list "wraps":

If you are focused on the last result and press ArrowDown you are taken to the first result
If you are focused on the first result and press ArrowUp you are taken to the last result

With a result highlighted, press Enter. That should take you to the next page with your selected user.
Click on the back button.
Use the arrow keys to highlight different result.
Press CMD+Enter. That should take you to the next page with your new selected user.

IOU Confirmation List
All Platforms

Click/tap on the green + icon on the home screen.
Click/tap on Split Bill.
Enter an amount and press Next.
Select some participants and press Next.
Make sure that you can not select or deselect participants.
Hover the mouse over your participants – it should not change into a pointer cursor.
Press Split YOUR_AMOUNT
A new request should be created between you and each of your selected bill split participants.
Open a group chat.
Press the + in the composer and select Split Bill.
Make sure you can select and deselect participants,  and that their proportional amount adjusts accordingly.
Hover your mouse over the participants – it should change into a pointer cursor and clicking on participants should select/deselect them.
With some participants selected and others not, press Split YOUR_AMOUNT.
A new request should be created between you and each of your selected bill split participants.

Web/Desktop

Open a group chat.
Click/tap on the + in the composer.
Click/tap on Split Bill.
Enter an amount and press Next.
Use the arrow keys to highlight participants. Make sure that the list "wraps":

If you are focused on the last result and press ArrowDown you are taken to the text input. None of the elements should be highlighted.
If you are focused on the first result and press ArrowUp you are taken to the text input. None of the elements should be highlighted.
If you are focused on the text input and press ArrowUp, you are taken to the last result.
If you are focused on the text input and press ArrowDown, you are taken to the first result.

With a result highlighted, pressing Enter should select and deselect participants, and their proportional amount should adjust accordingly.
Deselect all participants.
Press CMD + Enter. Verify that nothing happens (no IOU should be created with just yourself)
Reselect one or more participants.
With no result highlighted, press Enter. A new request should be created between you and each of your selected participants.
Open a group chat.
Click/tap on the + in the composer.
Click/tap on Split Bill.
Enter an amount and press Next.
With a result highlighted, press CMD+Enter. A new request should be created between you and each of your selected participants.
Open a 1:1 DM with another user (not Chronos or Concierge).
Press the + button in the compose bar and click Request Money
Enter an amount and press Next
Press Enter. The IOU should be created.

Workspace invite page
All Platforms

Create a workspace if you do not already have one.
Navigate to Settings -> Your Workspace -> Manage Members -> Invite.
Verify that you can scroll + search the results on the invite page.
Select several users from the list.
Press Invite, and verify that you are taken to the previous page and the new invitees are shown as members.

Web/Desktop

Create a workspace if you do not already have one.
Navigate to Settings -> Your Workspace -> Manage Members -> Invite.
Use the arrow keys to traverse the list. Verify that the list "wraps":

If you are focused on the last result and press ArrowDown you are taken to the search bar. None of the elements should be highlighted.
If you are focused on the first result and press ArrowUp you are taken to the search bar. None of the elements should be highlighted.
If you are focused on the search bar and press ArrowUp, you are taken to the last result.
If you are focused on the search bar and press ArrowDown, you are taken to the first result.

With an option highlighted, press Enter. That option should be selected and moved to the top of the list.
Select a few more options.
Highlight a selected option and press Enter. That should deselect the option.
With a few options selected, move focus to the search bar (no items highlighted), and press Enter. That should take you to the previous page with the new invitees shown as members.
Press Invite again.
Use the arrow keys to select a few options.
With a search result highlighted, press CMD+Enter. That should take you to the previous page with the new invitees shown as members.

IOU currency selection
All Platforms

Press the big green + to open the global create menu.
Select "request money"
Tap on the currency symbol ($ in US)
Search for another currency, verify the results list updates.
Select another currency, verify that the currency selection modal closes and the IOU amount page shows the new symbol for the currency you selected.

Web/Desktop

Press the big green + to open the global create menu.
Select "request money"
Click on the currency symbol ($ in US)
Search for another currency, verify the results lists updates.
Use the arrow key to scroll through the list. Verify that it "wraps" such that if you are at the bottom and press ArrowDown, you are taken to the top, and vice-versa.
With an item highlighted, change the search term.
Verify that the list scrolls back to the top and that the first item is highlighted
With an item highlighted, press the enter key.
Verify that the currency selection modal close and the IOU amount page shows the new symbol for the currency you selected.

Applause (important): Ensure that all of the above steps are covered in our regression test suite in TestFlight
In particular, we may need to add regression tests to cover using arrows to traverse lists. That is the main focus of this PR. Feel free to create an issue if necessary to give yourself more time and make it easier to track. Let me know if you have any questions.

[x] Verify that no errors appear in the JS console

Tested On

[x] Web
[x] Mobile Web
[x] Desktop
[x] iOS
[x] Android

Screenshots
Web
https://user-images.githubusercontent.com/47436092/155441436-7bcd79b1-1c21-4203-8fd5-3732fa7500d2.mov
Mobile Web

Desktop
https://user-images.githubusercontent.com/47436092/155444001-e24ca7e3-ed27-4d19-ae5e-879e7c54bca9.mov
iOS

Android

Note to self: potential regression?

Also tagging in @puneetlath to review since you requested the feature in the linked issue.

also not crazy about a method named maybeToggleIndex but trying to figure out what it did broke my brain

Good call out @bondydaa 😆  That's helpful! ... I think I'll rename it to maybeToggleParticipant for now, which I think is slightly better. There are three things that function can do:

If the search bar is focused (no option highlighted), it does nothing.

Note: In this case, the Enter key event will bubble to the next handler and press the confirm button, creating the IOU.

If we are looking at a 1:1 request instead of a bill split, then we can't toggle the only participant. So again it does nothing.
If a participant is highlighted, it toggles that participant.

I can add a comment to explain this too 👍

Ooooh AND @bondydaa helped me realize I was missing an edge case in my testing, which I've fixed and added a test for 😄

Ooooh AND @bondydaa helped me realize I was missing an edge case in my testing, which I've fixed and added a test for 😄

:success: glad to have been of service 😂

@roryabraham There's one more edge case - Toggle participant shouldn't work when splitting bill from the FAB menu.
Source: https://github.com/Expensify/App/pull/5895

From slack 🧵
I think this is a good oppurtunity to add this bonus feature - once all users are selected, allow to start chat using CMD+Enter
This should help when we're in the middle of a list, and wanna press Next

If bill split flow initiated via FAB, we will Not Allow attendee deselection on bill split confirmation page.

Not sure I agree that we should impose this restriction 🤔

the current flows are confusing as users can remove users on multiple steps. The only reason we added removal deselection in the first place is that there is no way to deselect if splitting a bill from within a group chat.

Hmm, I'm leaning more towards the restriction because of the above reason.
cc: @Julesssss if you have some additional context on why https://github.com/Expensify/App/issues/5295 was created.

Not sure I agree that we should impose this restriction 🤔

@rushatgabhane linked the issue which introduced this change, but the short explanation is that bill splitting within a subset of the group participants is desired

Launched from the Global Create Menu: User has manually selected all participants at the participant step, therefore we do not allow deselection at the confirm step
Launched from within a group chat:  The participants list is generated from all group chat participants, it is not guaranteed that all members should take part in the split, therefore users must be able to deselect at the confirm step (there is no participants step in this flow).

We're allowing for the use case where you and your friends have a group chat that is used over many months/years. It sucks to have to create a new group just because one of the group participants is not included in the bill split, so we allow for a subset of the group to have splits.

@Julesssss thanks a lot, that's helpful!

@rushatgabhane fixed this with a hack, not sure the underlying cause of that issue. Feel free to suggest a better solution if you see one.

User has manually selected all participants at the participant step, therefore we do not allow deselection at the confirm step

This is the part I'm not sure I agree with. While it makes sense that the user 99% of the time wouldn't need or want to deselect any of their choices at this step, I'm not sure why we need to impose that restriction. Going to bring this up in Slack I guess.

@rushatgabhane Pending the result of this slack discussion, this is ready for another review 🙇

@roryabraham I don't think we should move forward with the hack. There are many cases where the participant is hidden, this is one of them.
Steps

New group
Make sure you have at least 5 recent participants
Select first 3 participants from recents
Navigate to the last participant in recents - It's partially hidden.
Navigate to first participant in contacts - It's completely hidden

https://user-images.githubusercontent.com/29673073/156625742-7802725b-40cf-438d-aa29-687b2c987a3f.mov

Going to temporarily put this on hold as I extracted most of the KeyboardShortcut-specific code to a separate PR: https://github.com/Expensify/App/pull/8059

@roryabraham https://github.com/Expensify/App/pull/8059#issuecomment-1068461819 hit production a week ago, can we take this off hold?

Taken off hold (just removed [HOLD] from title, based on this comment in the associated issue

Putting this back in WIP while I retest everything

Retested and ready for review.

@rushatgabhane updated per your suggestion!

@rushatgabhane updated!

Opening this back up to reviews from everyone. I retested all these flows and have been diligent about updating the testing steps to account for all of the bugs found during the previous reviews.

Hmmm, I was thinking that it was a feature to have your previous selection remain highlighted, but I can see how it appears janky.

Just to clarify, the alternative that you're suggesting is that the highlighted index should reset to the first item whenever the list changes?

If so, the change would just be:
- if (newOptions.length > 0 && this.focusManager) {
-     const prevFocusedIndex = this.focusManager.getFocusedIndex();
-     const prevFocusedItem = prevState.allOptions[prevFocusedIndex];
-     const newFocusedIndex = Math.max(_.findIndex(newOptions, option => _.has(option, 'login') && _.has(prevFocusedItem, 'login') && option.login === prevFocusedItem.login), 0);
-     this.focusManager.setFocusedIndex(newFocusedIndex);
-     this.scrollToIndex(newFocusedIndex);
- }
+ if (this.focusManager && newOptions.length !== prevState.allOptions.length) {
+     this.focusManager.setFocusedIndex(0);
+     this.scrollToIndex(0);
+ }

Going to get other opinions in Slack, I'm not sure which is preferable.

Ohh that's smart, but I think we can do better than the proposed solution.

Just to clarify, the alternative that you're suggesting is that the highlighted index should reset to the first item whenever the list changes?

@roryabraham Yes, I think that'll be simple, predictable and perfect!
Reason:

Whenever the search query is updated, the best result™ is always at the top. So shouldn't we just refocus the first item?
I can't think of any usecase for https://github.com/Expensify/App/pull/7702#issuecomment-1098353975,

Excited for your thoughts on this!

Caveat
The first item in New Group page (and split etc) should be equal to selectedOptions.length(), and not 0.

Whenever the search query is updated, the best result™ is always at the top. So shouldn't we just refocus the first item? This should set the customer for success!

Yeah, that's a good argument. I'll update this PR soon to just do that (and take into account the caveat for multiple selections)

Updated.

Woohoo! C+ approved and ready for internal reviews again. 🎉

@roryabraham I'm guessing this PR requires further C+ review.
If so, feel free to unassign me because I'm OOO until May 14.

I think this is ready for another review. imo @rushatgabhane already earned the C+ review bounty with his earlier testing, but now is ooo until the 15th.
@parasharrajat Want to jump in on a review of this for an additional $250? There are a lot of testing steps and I want to make sure our bases are covered / I didn't miss anything after the latest changes.

Found a style regression ... working on a fix.

@parasharrajat updated

I just saw the earlier comment to review. I will do a thorough review and test it.

When a single item is selected, arrow navigation does not scroll to the right position.

Agree – I looked into this a bit earlier and even had a hack in place that we subsequently removed. If it's okay I would like to treat this as a separate issue.

Enter some text in the search. Go to the last index of the list via arrow keys. Next arrow press removes the selection and the next one will select the first item.

Not able to reproduce this.

Search for somting like three chars raj. List will refresh and first item is hightligted. Now remove the last char from searched text. List scrolls and hightligted item is out of view.

This is a feature, not a bug. Whenever the search term is updated, the Best Result™ is the first one in the list, so that's the one that's highlighted. At one point we had it implemented such that the current highlighted result would stay visible in the window, but it caused some UI jank. @rushatgabhane and I agreed just keeping the first item highlighted is simpler / better.

Use arrowDown to navigate to some item towards to end of the page or in the scroll area out of the page area. Select a item. List scrolls to the first selected index. Now press arrow down once. List immidiately scrolls to the item next to previously selected item. As soon as the hightlighted item is selected the focus move to the previous item.

Yeah, the focus doesn't change when a new item is selected. I think in this case we probably want to stick with the same logic we established above where the first item (aka the Best Result™) is focused.
Edit: Made this change, so now the first unselected item it focused when you use the enter key to add a selected item

The focused item is not selected on pressing enter when Tab navigation is used. Could be a separate issue.

Yes, let's consider tab navigation issues as separate for now.

Should arrow navigation start from the highlighted item via mouse?

Good question, but I think probably not.

Split Bill page ... Pressing enter does not submit the form i.e. Does not spit the bill.

Not able to reproduce this.

Split Bill page ... On Android, selecting the first item from recents section, scrolls the list and hide the selected item

It sounds like this is the same or similar to the known limitation mentioned above ... When a single item is selected, arrow navigation does not scroll to the right position. Let's create a separate issue to fix this.

Request Money Page ... No Request button.

Whoops! Fixed that 😅

This is all cleaned up and ready for another review. Let's get this over the finish line!

From this slack thread, I updated this to implement getItemLayout and fix navigation in long lists where the height of each item is static.

Let's break these limitations into separate issues

This has been cleaned up a bit and I addressed the couple bugs @rushatgabhane pointed out.
@marcaaron I've gone down a few avenues trying to split it up, but almost all of these changes are interdependent or can't be properly tested in isolation of each other.

@marcaaron spent a few hours trying to come up with this potentially more readable solution for getItemLayout:
getItemLayout(data, flatDataArrayIndex) {
    const optionHeight = this.props.optionMode === CONST.OPTION_MODE.COMPACT ? variables.optionRowHeightCompact : variables.optionRowHeight;
    let offset = 0;

    // Start with just an empty list header
    const flatArray = [{length: 0, offset}];

    // Build the flat array
    for (let sectionIndex = 0; sectionIndex < data.length; sectionIndex++) {
        const section = data[sectionIndex];

        // Add the section header
        const sectionHeaderHeight = section.title && !this.props.hideSectionHeaders ? variables.optionsListSectionHeaderHeight : 0;
        flatArray.push({length: sectionHeaderHeight, offset});
        offset += sectionHeaderHeight;

        // Add section items
        for (let i = 0; i < section.data.length; i++) {
            flatArray.push({length: optionHeight, offset});
            offset += optionHeight;
        }

        // Add the section footer
        flatArray.push({length: 0, offset});
    }

    // Then add the list footer
    flatArray.push({length: 0, offset});
    const targetItem = flatArray[flatDataArrayIndex];
    return {
        length: targetItem.length,
        offset: targetItem.offset,
        index: flatDataArrayIndex,
    };
}

And at first I thought there was no notable performance difference, but after testing on a high-traffic account you can see that the original solution is substantially faster:

Original, faster
Simplified, slower

https://user-images.githubusercontent.com/47436092/175395229-3763cd9f-d4d1-49a3-b036-defc25d008e7.mov
https://user-images.githubusercontent.com/47436092/175395301-deb1715c-a630-40cd-ba23-7d8e82776ba8.mov

Okay, updated the "flat data array" solution to use memoization as you've suggested and it works as fast the original solution.

reviewing

Loving the latest changes. But while testing I ran into this issue...

Open "Search"
Enter some text into filter
Start deleting characters
💥

Does not happen on staging or main
https://user-images.githubusercontent.com/32969087/175614694-e553fdd1-c694-480f-b7aa-479e036ae08e.mp4

Fixed the crash that was happening if you deleted part of the search term

But I noticed another issue while testing. Also, not sure if it's a blocker.
Create a split IOU in a group DM
Tap arrow keys until end of list is reached
One additional tap
See that the focus is not returned to the top item and instead no item is focused

This is intentional and only happens on pages where you can select multiple options – you are focusing on the search bar and pressing Enter again would submit the form instead of toggling an option

Updated: fixed a bug and addressed some code feedback. Ready for another review

This is intentional and only happens on pages where you can select multiple options – you are focusing on the search bar and pressing Enter again would submit the form instead of toggling an option

That makes sense. But I find a couple of things strange about this.

The input doesn't lose focus when we navigate away to an option so it looks like a bug because we can't see it "refocus". It's not clear that pressing Enter will submit the form.
If the input is not focused at all when we are using the arrow keys to focus on options we will eventually reach the input again, but it won't focus so it's not clear that pressing Enter will submit the form.

https://user-images.githubusercontent.com/32969087/176011605-f23eb746-d09b-4622-990e-75ceb23a6067.mp4

🚀 Deployed to staging by @marcaaron in version: 1.1.79-0 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

@roryabraham In the step 11, Cmd+Enter does nothing, but only Enter works. Is this correct behavior?

Click/tap on the green + icon on the home screen.
Click/tap on Request Money
Enter an amount and press Next
Enter a search query to get a reasonable-length list of results.
Use the arrow keys to traverse the list of results. Make sure that the list "wraps":
If you are focused on the last result and press ArrowDown you are taken to the first result
If you are focused on the first result and press ArrowUp you are taken to the last result
With a result highlighted, press Enter. That should take you to the next page with your selected user.
Click on the back button.
Use the arrow keys to highlight different result.
Press CMD+Enter. That should take you to the next page with your new selected user.

@kavimuru We created a separate issue for this: https://github.com/Expensify/App/issues/9594. No need to block deploy on it

@roryabraham would you mind attempting to resolve the regression again today? Thanks

Found a potential bug with this one related to the loading spinner. To reproduce:

Be on mobile web
Open a new tab and navigate to /search
Note that the loading spinner never goes away.

@marcaaron interesting.. can you also repro it on staging?

nope just tried and it only happens on main

@marcaaron I'm not able to reproduce that on main, which makes me think it's probably related to some API or Auth change that I don't have locally.

Cool. I will update and try to see if that resolves the issue. Mainly mentioned it here because it broke on a line of code this PR introduced so it seemed like a possible regression.

🚀 Deployed to production by @roryabraham in version: 1.1.79-17 🚀

platform
result

🤖 android 🤖
failure ❌

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅

I believe that this PR introduced this really edge-case bug. We should have either disabled the focus on disabled options or updated the logic in ComponentDidUpdate to match the one used in state, like this.

ah sorry, edited with the correct link


Druid dialect

Adds DruidDialect
SQL lib methods now accept dialect as parameter instead of same dialect being used throughout once toolbox is created

I'm not seeing any test cases using the Druid dialect – am I missing something?
@begelundmuller
In sql/src/test/java/com/rilldata/CalciteTests.java, each of the test method now test for both Duckdb and Druid dialect. For testing operators here, expected operator output for both dialects are checked.


[Feature Request] Enable Kore to start Kodi on Android TV

Feature suggestion
Detailed Description
Enable Kore to control not only Kodi but also the platform Kodi is running on.
E.g. but using an integration mechanism for different platforms
Context
I just recently switched to an Android TV Box. Now I'm wondering wouldn't it be great to enable Kore to start Kodi on an AndroidTV.  To not make it simply Android TV specific, adding an integration mechanism where it would be possible to add Device integrations with e.g. registered events.
So when I open up Kore I could set it up to automatically start Kodi on Android TV
It would make the experience of Kodi on Android TV a lot more cohesive and depending on implementation could also make Kore a great remote control for Android TV (given the original one lacks lots of features...)
Possible Implementation
Option 1
Simply add an option to pair Kore with Android TV and allow control with the nav buttons
Option 2
Implement generic device integration where one can implement different event handlers to start platform specific actions. E.g.: start app on Linux (with respective service installed), start app on Atv,...

The way to do this on Android is install a 2nd service that does the jsonrpc listening only and forwards it to Kodi. That would be low memory and can be running in the background all the time and could start Kodi or bring it to the foreground.


Supported the behavior of .ts files

Unlike .mp4 files, .ts files can be read although the file is not completed.
So IkaWatcher executed IkaLog with incomplete .ts files.
This change makes IkaWatcher to wait the execution until another file is
created or timeout after 3 mins.
This change is only tested with AVT C875 on Mac.

Thanks!


Changing Environment type from "Sandbox" to "Production" creates a new environment_id

Changing Environment type from "Sandbox" to "Production" creates a new environment_id
Changing Environment type from "Sandbox" to "Production" creates a new environment_id and all the configuration on the state is broken because all the resources that use the environment_id now points to a new environment.
│ Error: Provider produced inconsistent result after apply │  │ When applying changes to │ module.test.module.power_platform.powerplatform_environment_settings.pp_environment_settings, │ provider "provider[\"registry.terraform.io/microsoft/power-platform\"]" │ produced an unexpected new value: .environment_id: was │ cty.StringVal("91b08444-52a6-e176-8ca4-b2db8599XXXX"), but now │ cty.StringVal("3484dea0-fa1e-e88a-b826-06567fd4XXXX"). │  │ This is a bug in the provider, which should be reported in the provider's │ own issue tracker.
To Reproduce
Steps to reproduce the behavior:

You create a new environment with environment type "Sandbox", with:  environment_type     = "Sandbox"
Deploy the environment
You change the environment type value from Sandbox to Production and deploy the change:  environment_type     = "Production"
Alternately you can add settings to see the error.
You will see that the environment_id is different.

Sample Terraform Code
If applicable, add terraform code to help explain your problem.
terraform {
  required_providers {
    powerplatform = {
      source = "microsoft/power-platform"
    }
  }
}

provider "powerplatform" {
  use_cli = true
}

resource "powerplatform_environment" "example_environment_settings" {
  display_name     = "example_environment_settings"
  location         = "europe"
  environment_type = "Sandbox"
  dataverse = {
    language_code     = "1033"
    currency_code     = "USD"
    security_group_id = "00000000-0000-0000-0000-000000000000"
  }
}

resource "powerplatform_environment_settings" "settings" {
  environment_id = powerplatform_environment.example_environment_settings.id

  audit_and_logs = {
    plugin_trace_log_setting = "Exception"
    audit_settings = {
      is_audit_enabled             = true
      is_user_access_audit_enabled = true
      is_read_audit_enabled        = true
    }
  }
  email = {
    email_settings = {
      max_upload_file_size_in_bytes = 123456
    }
  }
  product = {
    behavior_settings = {
      show_dashboard_cards_in_expanded_state = true
    }
    features = {
      power_apps_component_framework_for_canvas_apps = false
    }
  }
}

Expected behavior
If I change the environment type from Sandbox to production the environment_id should be the same and tied resources and configuration can conserve the same environment_id.
System Information

Provider Version: 2.7.0-preview
OS & Version: Linux.

Additional context
Add any other context about the problem here.
Contribution

@eduardodfmex  can you confirm that this is still an issue in 3.0.0

I will test with 3.0.0

@mattdot I confirm yes, is still an issue in 3.0.0.
Is destroying the Environment and creating a new one:
Plan: 1 to add, 1 to change, 1 to destroy.
Changes to Outputs:
~ environment_id   = "a1deee2c-f6aa-ee7b-b0c7-c36fec629443" -> (known after apply)
~ environment_type = "Sandbox" -> "Production"
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
powerplatform_environment.example_environment_settings: Destroying... [id=a1deee2c-f6aa-ee7b-b0c7-c36fec629443]
powerplatform_environment.example_environment_settings: Still destroying... [id=a1deee2c-f6aa-ee7b-b0c7-c36fec629443, 10s elapsed]
powerplatform_environment.example_environment_settings: Still destroying... [id=a1deee2c-f6aa-ee7b-b0c7-c36fec629443, 20s elapsed]
powerplatform_environment.example_environment_settings: Still destroying... [id=a1deee2c-f6aa-ee7b-b0c7-c36fec629443, 30s elapsed]
powerplatform_environment.example_environment_settings: Destruction complete after 3m7s
powerplatform_environment.example_environment_settings: Creating...


child svn dependencies not getting pulled down

If an svn dependency contains child svn dependancy then it will not get pulled.
Here is the scenario
package.json
svnDependancy: 'svn://parent'
parent package.json
svnDependancy: 'svn://child'

Interesting - does your svn-dependency list svn-npm-crutch as one of its own dependencies?

No svn-npm-crutch goes in to dependencies array while svn dependencies only contain the custom libs that need to be pulled from svn.
"dependencies": { 		"svn-npm-crutch": "~0.5.4" }, "svnDependencies": { 	"libA": "https://.../svn/trunk/libA"		 }


Make Conjure log window size and direction configurable

Related to #7, I wanted to see how looks like if Conjure's window log opens in an horizontal split, and wrote this code.
It adds 3 new variables, g:conjure_log_direction, g:conjure_log_size_small and g:conjure_log_size_large to define the direction and size of the Conjure log window.
Also modifies the size meaning. Before the size was the number of columns to use, now it is the percentage of the window to use.
To me, makes more sense to make it a percentage, since sometimes I work on a small monitor (laptop) and other times on a bigger one.

Oh this is fantastic! I was going to address your points in #7 eventually but you beat me to it :tada: I'll give it a review now, I've found myself wanting a horizontal split regularly too.
I'm not 100% (hah) sure about percentage sizes but I'll give it a think. Maybe a choice makes sense between a number and a percentage. I like knowing my split will be exactly wide enough to fit all of the formatted output inside it.
I suppose 25% and 50% would be more than enough on a 1080p monitor though :thinking: I was thinking of bumping the large size to 100 columns today, coincidentally.

To follow on from that slightly, if we defined the initial config in plugin/conjure.vim like this:
let g:conjure_log_direction = get(g:, 'conjure_log_direction', "vertical")
let g:conjure_log_size_small = get(g:, 'conjure_log_size_small', 25)
let g:conjure_log_size_large = get(g:, 'conjure_log_size_large', 50)

It would allow us to default the values while also letting users override the config before or after Conjure has been loaded. It's a neat trick I think :slightly_smiling_face:

Thanks for your comments. I understand better now how things are working.
Keep in mind that I have no Lua experience, just let me know if the code is unidiomatic or looks bad, feedback is always welcome :)
About the percentage sizes, maybe we can make the size options accept a string, like "50%". E.g.: if the value is 50 (a number) or "50" (a string) the size means number of lines/columns, while if the value is "50%" it means a percentage. This way we can choose between a number and a percentage without introducing a new option. What do you think?
About moving the functionality from Clojure to Lua, it makes a lot of sense, I didn't think about performance (and I also prefer to write Clojure ;) ) . But something is not completely clear to me, I understand that calling Neovim from Clojure blocks the process. But what happens when calling Neovim from Lua (e.g.: vim.api.nvim_win_set_width)? Is it blocking too?
You mention that you have wider thoughts about configuration. I can address your feedback and update my PR, but if you have other ideas about how to implement it, it's totally fine if you close this PR and you do it in a different way. Just let me know how to prefer to proceed.

I'm learning lua as I go along with this too, so that's fine. I prefer to write Clojure too of course, that's why the "find matching pairs" code is in Clojure because it's quite complex. So I minimised the amount of calls it made (two I think?) then built all of the logic in Clojure, I've had to be super careful with the performance of it though, batching and reusing as much as possible.
I may well translate that to lua some day if benchmarks (when I write them) prove that read-form is one of the slowest parts of Conjure, which it may well be! I don't look forward to trying to write that in lua though. Worth noting that I think the lua in Neovim runs a lot faster than the VimL, so I would favour lua over VimL for perf.
Using the lua does prevent porting to regular vim though which might be a "fun" modification to make later...
So I think calling Neovim from lua ends up being a direct call in memory, luajit is built into the neovim binary which is cool. There's no RPC / msgpack encode and decode which is what eats up time when Conjure's JVM and Neovim communicate. There appears to be almost no cost to that.
I think the way you designed the config looks good to me and how I was going to approach it, I'd just set defaults using the get method I mentioned in one of my replies so users can override before or after they're set but we can rely on there always being a value in those g: variables.
With regards to percentages, I think just stick with numbers for now and it'll always be a percent. Then maybe we add a toggle at some point that disables the percentage part. I think it's a better way to do it so we'll swap to percentages for now.
Sorry about writing so much in response to these, just want to provide context and hopefully help you decide more things / understand why I'm thinking one way or another :slightly_smiling_face:

It definitely helps me understand, I appreciate long responses with more context, thanks :)
I will update my PR. FYI, I'm not sure if I'll have time for it this week, if not I'll do it next week

No problem, take your time. Calm development is good development. I'll be working on Conjure most of this weekend I think trying to get sustainable dependency injection working for Compliment and future dependencies like Orchard.
If you want me to take a look over the weekend I'll be happy to merge your changes and tweak them as we discussed. Happy to leave it until you're done though so you can claim the lines :smile:

For me, if you want to work on Conjure during the weekend, and you'd like to finish this feature, go for it.
Don't merge my changes if they are not needed, fine with me, no claims ;)
On the other hand, if you prefer to focus on other features, I'm happy to work on this PR.

I just updated the PR. I think I addressed all your feedback, but let me know if I forgot something.
There is no ternary operator in Lua, but I found that the same can be accomplish with $condition and ... or ...

All merged, let's test this on master for a bit. I think maybe I'm going to need a develop branch soon...


Config: Refactor SCSS build process in frontend

This pull request refactors the SCSS build process in the frontend by removing the unnecessary "yarn scss" command from the build process. Vite and Storybook have built-in support for importing and handling sass.
Resolves #686

LGTM. A question: while this also watch for scss changes during development?

It does! 😄 So no more manual yarn scss:watch while developing


Add Escalator/conveying preset

https://wiki.openstreetmap.org/wiki/Key:conveying
highway=steps + conveying=yes
highway=footway + conveying=yes

My implementation suggestions:
Conveying Field

Path: conveying
Name: Movement Direction
Type: typeCombo
Key: conveying
Strings: yes (perhaps labeled "Any"), forward, backward, reversible

Escalator Preset

Path: highway/steps/conveying
Name: Escalator
Geometry: line
Fields: incline_steps, conveying, access_simple, lit, perhaps others from the steps preset
Tags: "highway"="steps", "conveying"="*"
Reference: "key": "conveying"
Icon: I'd swear there's an icon of a person riding an escalator but I can't seem to find it.

Moving Walkway Preset

Path: footway/conveying
Name: Moving Walkway
Geometry: line
Fields: conveying, access_simple, lit, perhaps others from the footway preset
Tags: "highway"="footway", "conveying"="*"
Reference: "key": "conveying"
Terms: moving sidewalk, autowalk, skywalk, travolator, travelator, travellator, conveyor
Icon: Walking person

I'd like to work on this.

@mchlp Go for it! I just updated my suggestions with the icon names you might want to use.


Suggestions

Time should not be like d.m.y, h-m-s, timezone are different in every country
Make it like
Player have been muted for 1d 2h 5m ( 1day 2 hours 5minutes)

I'll implement a way to change this in BetterMute's next version.


Add WISE datasets

Adds

allsky
cryo-3band
postcryo
neowiser

Updates

allwise

To Do

[x] add dataset and resource descriptions
[x] add DOIs
[ ] get reviews from Vandana and Brigitta
[ ] submit PR to awslabs/open-data-registry
[ ] submit an application for each dataset at https://application.opendata.aws/ (for dataset tracking purposes only)
[ ] send logo to<EMAIL_ADDRESS>

@bsipocz will you give these a sanity check? Same idea as the recent spitzer-seip one. Vandana is in the process of reviewing them as well. This is just in my fork for now; I'll submit to the upstream repo after we work out the kinks.

@stargaser
Will you review this? Here’s some additional info, if it’s helpful:

These yaml files will be used to display a webpage for each dataset. Here is the current AllWISE page for reference.
The bucket/folder containing the data described in these yamls: nasa-irsa-wise/wise.
Yaml field descriptions are in the upstream repo readme.

superseded by https://github.com/awslabs/open-data-registry/pull/2026


How to keep the Twisted server running forever to improve performance?

Rasa version: Current
Python version:  3.6.5
Operating system (windows, osx, ...): Windows/Ubuntu/ AWS Linux
Issue: The twisted server seems to time out very often and not give a stable performance. When reconnected it takes time to process the request and that's a considerable cold start for the app to run!
Content of configuration file (config.yml):
- name: tokenizer_whitespace
- name: ner_crf
- name: intent_featurizer_count_vectors
  OOV_token: oov
  token_pattern: (?u)\b\w+\b
- name: intent_classifier_tensorflow_embedding
  epochs: 50
- name: ner_duckling_http
  url: http://localhost:8000
  locale: "NL_Nothing"
  dimensions: ["time"]
- name: ner_synonyms
  language: en

Content of domain file (domain.yml) (if used & relevant):

@axchanda Which version of rasa are you running and what command exactly does it time out on?
We have moved to a unified Core + NLU server which runs Sanic on master branch. Can you try to switching that and see if the issue still occurs?


tools/kmodules: Update sched_tp kernel module

Add sched_cpu_capacity event
update_time field to sched_pelt_cfs
Remove sched_update_nr_running

This is just the verbatim content of the module shipped in the integration branch that is used with LISA in practice for now, feel free to open a new PR with only the updates to the events themselves

Okay I see the source of the confusion now. It seems the integration branch had extra changes that I wasn't aware of. The current version of the module comes from here for what it's worth

Superseded by https://github.com/ARM-software/lisa/pull/1729


Refused to execute script from '' because its MIME type ('text/html') is not executable

Hi Matt,
Thank you for great article I have the following error after pressing login button:
Refused to execute script from '' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.
callback:1 Refused to execute script from 'http://localhost:4200/implicit/inline.bundle.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.

If you close me this project and configure it for your Okta app, do you see the same error?

On Apr 15, 2018, at 14:50, oktay-rectasolutions<EMAIL_ADDRESS>wrote:
Hi Matt,
Thank you for great article I have the following error after pressing login button:
Refused to execute script from '' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.
callback:1 Refused to execute script from 'http://localhost:4200/implicit/inline.bundle.js' because its MIME type ('text/html') is not executable, and strict MIME type checking is enabled.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

If I clone your project, I have just done this I don't see any errors , but may be some library causing this for me, there is the request going to Okta server, it is probably this request Mime Type is failing is there any mimetype settings?

If you've done the tutorial, and you're seeing this issue, maybe you missed a step or versions don't match? You can use a tool like SmartSynchronize to compare the source from this example with yours. Beyond Compare works well too (if you're on Windows).

I copied content of your package.json to mine so there is no version mismatch but I will give it a try thank you, I am on Linux will try with Intellij plugin. Thank you for your time


[tool_crash] ShaderCompilerException

Command
flutter run

Steps to Reproduce

...
...
...

Logs
ShaderCompilerException
#0      ShaderCompiler.compileShader (package:flutter_tools/src/build_system/targets/shader_compiler.dart:77:7)
<asynchronous suspension>
#1      writeBundle.<anonymous closure> (package:flutter_tools/src/bundle_builder.dart:184:25)
<asynchronous suspension>
#2      Future.wait.<anonymous closure> (dart:async/future.dart:522:21)
<asynchronous suspension>
#3      writeBundle (package:flutter_tools/src/bundle_builder.dart:162:3)
<asynchronous suspension>
#4      WebDevFS.update (package:flutter_tools/src/isolated/devfs_web.dart:843:9)
<asynchronous suspension>
#5      ResidentWebRunner._updateDevFS (package:flutter_tools/src/isolated/resident_web_runner.dart:488:35)
<asynchronous suspension>
#6      ResidentWebRunner.run.<anonymous closure> (package:flutter_tools/src/isolated/resident_web_runner.dart:283:41)
<asynchronous suspension>
#7      asyncGuard.<anonymous closure> (package:flutter_tools/src/base/async_guard.dart:111:24)
<asynchronous suspension>

[✓] Flutter (Channel stable, 3.3.0, on Microsoft Windows [Version 10.0.18363.327], locale fa-IR)
    • Flutter version 3.3.0 on channel stable at C:\flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision ffccd96b62 (3 months ago), 2022-08-29 17:28:57 -0700
    • Engine revision 5e9e0e0aa8
    • Dart version 2.18.0
    • DevTools version 2.15.0

[✗] Android toolchain - develop for Android devices
    ✗ Unable to locate Android SDK.
      Install Android Studio from: https://developer.android.com/studio/index.html
      On first launch it will assist you in installing the Android SDK components.
      (or visit https://flutter.dev/docs/get-started/install/windows#android-setup for detailed instructions).
      If the Android SDK has been installed to a custom location, please use
      `flutter config --android-sdk` to update to that location.

[✓] Chrome - develop for the web
    • Chrome at C:\Program Files (x86)\Google\Chrome\Application\chrome.exe

[✗] Visual Studio - develop for Windows
    ✗ Visual Studio not installed; this is necessary for Windows development.
      Download at https://visualstudio.microsoft.com/downloads/.
      Please install the "Desktop development with C++" workload, including all of its default components

[!] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.dev/docs/get-started/install/windows#android-setup for detailed instructions).

[✓] Connected device (2 available)
    • Windows (desktop) • windows • windows-x64    • Microsoft Windows [Version 10.0.18363.327]
    • Chrome (web)      • chrome  • web-javascript • Google Chrome 107.0.5304.123

[✓] HTTP Host Availability
    • All required HTTP hosts are available

! Doctor found issues in 3 categories.

Flutter Application Metadata
Type: app
Version: 1.0.0+1
Material: true
Android X: false
Module: false
Plugin: false
Android package: null
iOS bundle identifier: null
Creation channel: stable
Creation framework version: ffccd96b62ee8cec7740dab303538c5fc26ac543

Closing this as a duplicate of https://github.com/flutter/flutter/issues/115433.
Kindly follow up in the issue linked above for updates.
Thank you


x/tools/cmd/oracle: fails when test defines method on non-test receiver

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?
1.6

What operating system and processor architecture are you using (go env)?
darwin

What did you do?

go get github.com/tschottdorf/goplay && oracle \
  -pos=$GOPATH/src/github.com/tschottdorf/goplay/oracle_test_boom/main.go:#40 \
  callers github.com/tschottdorf/goplay/oracle_test_boom/

What did you expect to see?
The list of callers of (*X).Bar() (i.e., an empty list).

What did you see instead?

/Users/tschottdorf/go/src/github.com/tschottdorf/goplay/oracle_test_boom/main_test.go:10:2: invalid operation: (&(main.X literal)) (value of type *github.com/tschottdorf/goplay/oracle_test_boom.X) has no field or method Foo
oracle: couldn't load packages due to errors: github.com/tschottdorf/goplay/oracle_test_boom/_test

In fact, this completely renders oracle unusable in source trees which have this phenomenon.
(cc: @mrtracy)

@tschottdorf, oracle is no longer developed. It has been renamed to guru (golang.org/x/tools/cmd/guru) recently.
This problem exists in guru as well. The following command line can be used to reproduce:
guru -scope github.com/tschottdorf/goplay/oracle_test_boom/ callers $GOPATH/src/github.com/tschottdorf/goplay/oracle_test_boom/main.go:#40

Thanks for the bug report.
I notice that the trailing slash in the -scope argument seems to confuse the tool:
$ guru -scope github.com/tschottdorf/goplay/oracle_test_boom/ callers $GOPATH/src/github.com/tschottdorf/goplay/oracle_test_boom/main.go:#40
/usr/local/google/home/adonovan/got3/src/github.com/tschottdorf/goplay/oracle_test_boom/main_test.go:10:2: invalid operation: (&(main.X literal)) (value of type *github.com/tschottdorf/goplay/oracle_test_boom.X) has no field or method Foo
guru: couldn't load packages due to errors: github.com/tschottdorf/goplay/oracle_test_boom/_test

Removing the slash seems to be an effective workaround:
$ guru -scope github.com/tschottdorf/goplay/oracle_test_boom callers $GOPATH/src/github.com/tschottdorf/goplay/oracle_test_boom/main.go:#40
/usr/local/google/home/adonovan/got3/src/github.com/tschottdorf/goplay/oracle_test_boom/main.go:5:10: (github.com/tschottdorf/goplay/oracle_test_boom.X).Bar is called from these 0 sites:


Why does SortedSet use reference equality when it removes its element?

I've been using a tuple(Array in JS) as a key with SortedSet.remove.
I found that SortedSet.remove doesn't remove its element when it's not a primitive type.
So I investigated the implementation and found that removeInternal checks the reference equality before actually removing the element.
https://github.com/rimbu-org/rimbu/blob/main/packages/sorted/src/set-custom/implementation/immutable.mts#L435
Even in the case that my custom compare function returns 0, the element won't be deleted.
Just commenting that line out made remove operate as I expected.
Is this intended behavior? Or am I missing something else?

I'm pretty sure you found a genuine bug there! It should not be using reference equality, but the context.comp instance.


Translating the interface labels

How can I translate the cockpit interface?
In the code i saw @lang('KEY') calls which seem to support the translation.

It seems to be initialized on the getUser() call:
    if ($translationspath = $app->path("#config:cockpit/i18n/{$locale}.php")) {
        $app('i18n')->locale = $locale;
        $app('i18n')->load($translationspath, $locale);
    }

so it requires having in config/cockpit/i18n/<code>.php (e.g. config/cockpit/i18n/pt.php)
<?php

/**
 * @file
 * Language file for PT.
 */

return [
  'Save' => 'Guardar',
  'Close' => 'Fechar',
  // Other entries...
];

But above only works, if the user language is defined to pt.
Believe that would be a good idea if Cockpit provide a simple language selector for the interface, so user can switch the language at any time.


Separated reference model from quaternion_pd_controller.cpp

Added header file, cpp file of reference model. Removed it from original quaternion_pd_controller.cpp

Changed in new pull request

Changed in new pull request


ci: update ArmoniK.Extensions.Csharp and ArmoniK.Core

Update ArmoniK.Extensions.Csharp from 0.12.2 to 0.12.4
Update ArmoniK.Core from 0.18.0 to 0.19.2 (api 3.14)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


[gbc] Fix ambiguous ocurrence of some

Do not merge, it is incomplete

seems like it is fixed now https://hackage.haskell.org/package/bond-<IP_ADDRESS>/reports/


replicate select_columns on the index

select_columns can be adapted to index selection as well; not sure about the name though  ---> select_index/select_rows?
df.select_index()
df.select_rows()

Some of the great work you've done, @samukweku, like incorporating slice()s, would be usable here too! Out of curiosity, what prompted the idea here?

Thanks @ericmjl . I have come to love the flexibility/abstraction of select_columns hence the idea of extending it to rows.

Got it!
select_rows() would be a great addition, definitely.
I can also see us building on top of select_columns() and select_rows() to provide a select(rows=..., columns=...) API that then dispatches down to select_columns() and select_rows(). It would be a nicely chained API that could be more flexible than the built-in .loc[] calls.

Yea, that would be great. Not sure of the select word though, it seems pandas has plans for that too : https://github.com/pandas-dev/pandas/issues/40322


[Breadcrumb] - Your Bug Name

the play music button directs to a broken link on soundcloud.

Woah looks like it just broke. Must be the service i'm using to do the popup

I just fixed the music player.

Hi! Looks like it's broken again. It opens the Stratus 2 popup alright, but hit play on a track and nothing happens.


Revise AbstractConsumerSeekAware.onPartitionsAssigned() for concurreny

Looks like different partitions are registered for same ConsumerSeekCallback when we have a multi-@KafkaListener scenario.
See AbstractConsumerSeekAwareTests and its TODO.
when we have different groups and concurrency, different listener container instances are going to deal with different partitions.
But looks like we add all of them to the same map entry:
this.callbackToTopics.computeIfAbsent(threadCallback, key -> new LinkedList<>()).add(tp);

This way when we call seek later from the logic, may lead to error like:
2024-07-15 20:30:17,431 ERROR [org.springframework.kafka.KafkaListenerEndpointContainer#0-1-C-1] [org.springframework.kafka.listener.KafkaMessageListenerContainer] - Exception while seeking TopicPartitionOffset{topicPartition=Seek-0, offset=null, relativeToCurrent=false, position=BEGINNING}
java.lang.IllegalStateException: No current assignment for partition Seek-0
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:367) ~[kafka-clients-3.7.1.jar:?]
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.lambda$requestOffsetReset$3(SubscriptionState.java:650) ~[kafka-clients-3.7.1.jar:?]
	at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4966) ~[?:?]
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.requestOffsetReset(SubscriptionState.java:648) ~[kafka-clients-3.7.1.jar:?]
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.seekToBeginning(LegacyKafkaConsumer.java:804) ~[kafka-clients-3.7.1.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.seekToBeginning(KafkaConsumer.java:1178) ~[kafka-clients-3.7.1.jar:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.processSeeks(KafkaMessageListenerContainer.java:3040) [main/:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1397) [main/:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1298) [main/:?]
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
2024-07-15 20:30:17,431 ERROR [org.springframework.kafka.KafkaListenerEndpointContainer#1-0-C-1] [org.springframework.kafka.listener.KafkaMessageListenerContainer] - Exception while seeking TopicPartitionOffset{topicPartition=Seek-2, offset=null, relativeToCurrent=false, position=BEGINNING}
java.lang.IllegalStateException: No current assignment for partition Seek-2
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:367) ~[kafka-clients-3.7.1.jar:?]
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.lambda$requestOffsetReset$3(SubscriptionState.java:650) ~[kafka-clients-3.7.1.jar:?]
	at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4966) ~[?:?]
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.requestOffsetReset(SubscriptionState.java:648) ~[kafka-clients-3.7.1.jar:?]
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.seekToBeginning(LegacyKafkaConsumer.java:804) ~[kafka-clients-3.7.1.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.seekToBeginning(KafkaConsumer.java:1178) ~[kafka-clients-3.7.1.jar:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.processSeeks(KafkaMessageListenerContainer.java:3040) [main/:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1397) [main/:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1298) [main/:?]
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
2024-07-15 20:30:17,432 ERROR [org.springframework.kafka.KafkaListenerEndpointContainer#0-1-C-1] [org.springframework.kafka.listener.KafkaMessageListenerContainer] - Exception while seeking TopicPartitionOffset{topicPartition=Seek-1, offset=null, relativeToCurrent=false, position=BEGINNING}
java.lang.IllegalStateException: No current assignment for partition Seek-1
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:367) ~[kafka-clients-3.7.1.jar:?]
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.lambda$requestOffsetReset$3(SubscriptionState.java:650) ~[kafka-clients-3.7.1.jar:?]
	at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4966) ~[?:?]
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.requestOffsetReset(SubscriptionState.java:648) ~[kafka-clients-3.7.1.jar:?]
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.seekToBeginning(LegacyKafkaConsumer.java:804) ~[kafka-clients-3.7.1.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.seekToBeginning(KafkaConsumer.java:1178) ~[kafka-clients-3.7.1.jar:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.processSeeks(KafkaMessageListenerContainer.java:3040) [main/:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1397) [main/:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1298) [main/:?]
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]

@artembilan
Thanks for reporting!
Could you please point me to the code that reproduces?
I've used different group IDs in multiple listeners, but I don't think I've figured out the problem yet.

See the AbstractConsumerSeekAwareTests you have introduced and that my TODO on concurrency attribute.
I have an impression that the same callback is used for different groups (or different container instances in case of concurrency) and those partitions are distributed between containers. However that add() to the list of the map would lead to the action when that callback will try to seek that partition which does not belong to the current container.
Does not happen consistently . Probably it is OK when both groups have the same partition distributions.

Thanks!
I'll have to look into the issue further, but for personal reasons I won't be able to dive into it until next week. I'll try to get to it as soon as I can.

No worries! The first milestone is out today. The next one in a month, so we have plenty of time 😅

@artembilan
Hi, I've looked into the above issue a bit more.
To start with the results, the issue is reproducible in v3.2, which is before we enhanced AbstractConsumerSeekAware, and I'm guessing it's reproducible in earlier versions as well.
The reproducible code is in the branch here.

Example description

Error message: No current assignment for partition Seek-6 ..
Error thread: org.springframework.kafka.KafkaListenerEndpointContainer#1-2-C-1 (Hereafter, we'll refer to it as KafkaListenerEndpointContainer#1-2 for convenience)
Causes of the error

This issue can be broken down into before and after the call to multiGroupListener.seekToBeginning() in the test code.
Before calling that method, the thread consumer was assigned partitions 6, 7, and 8.
Then it calls the seekToBeginning() method, and the ListenerConsumer.seeks queue adds partitions 6, 7, and 8 to the consumer.
However, after the seekToBeginning() method exited, the ListenerConsumerRebalanceListener was activated and the partition was reassigned to 7.
Partitions 6, 7, and 8 still exist in the ListenerConsumer.seeks queue, but the consumer no longer subscribes to 6, hence the error.

Workaround

To use the partition in seeks, there should be a condition that the partition is not reallocated due to rebalancing, etc.

-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#1-2, assigned: [6, 7, 8]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#1-2, assigned: [6, 7, 8]
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#1-2, assigned: [6, 7, 8], seeks: []

================= Where `ListenerConsumer.seeks` queue is appended. =====================================================
## [multiGroupListener.seekToBeginning()] start
---------------- [ListenerConsumer.seekToBeginning] id: org.springframework.kafka.KafkaListenerEndpointContainer#1-2, seeks: [6, 7, 8]
## [multiGroupListener.seekToBeginning()] end
==========================================================================================================================================

==========================================================================================================================================
=========== Before `ListenerConsumer.processSeeks()`, sometimes partitions are newly assigned, BUT `seeks` doesn't change.
=========== For this reason, the partition within seeks may not be newly assigned to the consumer, and you may see an error such as `No current assignment for partition Seek-N`.
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#1-2, assigned: [7]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#1-2, assigned: [7]
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#1-2, assigned: [7], seeks: [6, 7, 8]
2024-08-02 22:26:11,273 ERROR [org.springframework.kafka.KafkaListenerEndpointContainer#1-2-C-1] [org.springframework.kafka.listener.KafkaMessageListenerContainer] - Exception while seeking TopicPartitionOffset{topicPartition=Seek-6, offset=null, relativeToCurrent=false, position=BEGINNING}
java.lang.IllegalStateException: No current assignment for partition Seek-6
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:367) ~[kafka-clients-3.7.0.jar:?]
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.lambda$requestOffsetReset$3(SubscriptionState.java:650) ~[kafka-clients-3.7.0.jar:?]
	at java.base/java.util.Collections$SingletonList.forEach(Collections.java:5186) ~[?:?]
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.requestOffsetReset(SubscriptionState.java:648) ~[kafka-clients-3.7.0.jar:?]
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.seekToBeginning(LegacyKafkaConsumer.java:804) ~[kafka-clients-3.7.0.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.seekToBeginning(KafkaConsumer.java:1178) ~[kafka-clients-3.7.0.jar:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.processSeeks(KafkaMessageListenerContainer.java:3044) ~[main/:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1401) ~[main/:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1302) ~[main/:?]
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

So, is the idea for fix to filter out only actual partition  to seek and ignore those which are not assigned to us at the moment?

Isn’t that partition assigned / revoked API about? Shouldn’t we adjust our maps respectively ?

We should also look at the maps used by the assigned/revoked partition API, but I think we may think more about how we can change the ListenerConsumer.seeks queue. I thought that if I change the map but not the seeks, the result would be the same, but I'll check this out a bit more

This issue seems to occur frequently when concurrency grows. It happens with multi-group listeners, but it also happens with single listeners with high concurrency.
I'm wondering if the high concurrency is causing latency somewhere, and the rebalancing is happening when it exceeds that latency. I'd love to know where, but it's not easy to find.
Below is an error from a partition = 9, concurrency = 7 single-threaded consumer.
The KafkaListenerEndpointContainer#0-4 consumer was initially assigned partition 8, which added 8 to the seeks queue, but later the partition was reassigned to 6 and the offset of partition 8 became unchangeable.
## ConsumerStartedEvent [source=KafkaMessageListenerContainer [id=org.springframework.kafka.KafkaListenerEndpointContainer#0-0, clientIndex=-0, topicPartitions=[]]]
## ConsumerStartedEvent [source=KafkaMessageListenerContainer [id=org.springframework.kafka.KafkaListenerEndpointContainer#0-1, clientIndex=-1, topicPartitions=[]]]
## ConsumerStartedEvent [source=KafkaMessageListenerContainer [id=org.springframework.kafka.KafkaListenerEndpointContainer#0-2, clientIndex=-2, topicPartitions=[]]]
## ConsumerStartedEvent [source=KafkaMessageListenerContainer [id=org.springframework.kafka.KafkaListenerEndpointContainer#0-3, clientIndex=-3, topicPartitions=[]]]
## ConsumerStartedEvent [source=KafkaMessageListenerContainer [id=org.springframework.kafka.KafkaListenerEndpointContainer#0-4, clientIndex=-4, topicPartitions=[]]]
## ConsumerStartedEvent [source=KafkaMessageListenerContainer [id=org.springframework.kafka.KafkaListenerEndpointContainer#0-5, clientIndex=-5, topicPartitions=[]]]
## ConsumerStartedEvent [source=KafkaMessageListenerContainer [id=org.springframework.kafka.KafkaListenerEndpointContainer#0-6, clientIndex=-6, topicPartitions=[]]]
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-0, assigned: [0, 1]
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-2, assigned: [4, 5]
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-1, assigned: [2, 3]
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-4, assigned: [8]
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-3, assigned: [6, 7]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-4, assigned: [8]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-1, assigned: [2, 3]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-2, assigned: [4, 5]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-3, assigned: [6, 7]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-0, assigned: [0, 1]
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-0, assigned: [0, 1], seeks: []
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-1, assigned: [2, 3], seeks: []
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-3, assigned: [6, 7], seeks: []
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-4, assigned: [8], seeks: []
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-2, assigned: [4, 5], seeks: []

================= Where `ListenerConsumer.seeks` queue is appended. =====================================================
## [multiGroupListener.seekToBeginning()] start
---------------- [ListenerConsumer.seekToBeginning] id: org.springframework.kafka.KafkaListenerEndpointContainer#0-3, seeks: [6, 7]
---------------- [ListenerConsumer.seekToBeginning] id: org.springframework.kafka.KafkaListenerEndpointContainer#0-0, seeks: [0, 1]
---------------- [ListenerConsumer.seekToBeginning] id: org.springframework.kafka.KafkaListenerEndpointContainer#0-2, seeks: [4, 5]
---------------- [ListenerConsumer.seekToBeginning] id: org.springframework.kafka.KafkaListenerEndpointContainer#0-1, seeks: [2, 3]
---------------- [ListenerConsumer.seekToBeginning] id: org.springframework.kafka.KafkaListenerEndpointContainer#0-4, seeks: [8]
## [multiGroupListener.seekToBeginning()] end
==========================================================================================================================================

==========================================================================================================================================
=========== Before `ListenerConsumer.processSeeks()`, sometimes partitions are newly assigned, BUT `seeks` doesn't change.
=========== For this reason, the partition within seeks may not be newly assigned to the consumer, and you may see an error such as `No current assignment for partition Seek-N`.
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-2, assigned: [4]
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-1, assigned: [2, 3]
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-4, assigned: [6]
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-3, assigned: [5]
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-6, assigned: [8]
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-5, assigned: [7]
-- [ListenerConsumerRebalanceListener.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-0, assigned: [0, 1]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-4, assigned: [6]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-5, assigned: [7]
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-5, assigned: [7], seeks: []
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-6, assigned: [8]
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-6, assigned: [8], seeks: []
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-2, assigned: [4]
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-2, assigned: [4], seeks: [4, 5]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-3, assigned: [5]
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-3, assigned: [5], seeks: [6, 7]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-1, assigned: [2, 3]
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-4, assigned: [6], seeks: [8]
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-1, assigned: [2, 3], seeks: [2, 3]
-------- [AbstractConsumerSeekAware.onPartitionsAssigned] thread: KafkaListenerEndpointContainer#0-0, assigned: [0, 1]
------------ [ListenerConsumer.seekPartitions] thread: KafkaListenerEndpointContainer#0-0, assigned: [0, 1], seeks: [0, 1]
2024-08-03 13:19:07,945 ERROR [org.springframework.kafka.KafkaListenerEndpointContainer#0-4-C-1] [org.springframework.kafka.listener.KafkaMessageListenerContainer] - Exception while seeking TopicPartitionOffset{topicPartition=Seek-8, offset=null, relativeToCurrent=false, position=BEGINNING}
java.lang.IllegalStateException: No current assignment for partition Seek-8
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:367) ~[kafka-clients-3.7.0.jar:?]
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.lambda$requestOffsetReset$3(SubscriptionState.java:650) ~[kafka-clients-3.7.0.jar:?]
	at java.base/java.util.Collections$SingletonList.forEach(Collections.java:5186) ~[?:?]
	at org.apache.kafka.clients.consumer.internals.SubscriptionState.requestOffsetReset(SubscriptionState.java:648) ~[kafka-clients-3.7.0.jar:?]
	at org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.seekToBeginning(LegacyKafkaConsumer.java:804) ~[kafka-clients-3.7.0.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.seekToBeginning(KafkaConsumer.java:1178) ~[kafka-clients-3.7.0.jar:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.processSeeks(KafkaMessageListenerContainer.java:3047) ~[main/:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1404) ~[main/:?]
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1305) ~[main/:?]
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
2024-08-03 13:19:07,945 ERROR [org.springframework.kafka.KafkaListenerEndpointContainer#0-2-C-1] [org.springframework.kafka.listener.KafkaMessageListenerContainer] - Exception while seeking TopicPartitionOffset{topicPartition=Seek-5, offset=null, relativeToCurrent=false, position=BEGINNING}

@artembilan
IMHO, at runtime, the moment you try to reset the offset, rebalancing may occur for some reasons (for example, due to an increase in the number of consumers or partitions).
In this case, the partitions will be reassigned and you may not be able to adjust the offset as you want, but Spring Kafka will tell you in the error log which partition failed.
Therefore, users should be aware of the possibility of this error in advance and have an alternative, such as re-requesting to seek the offset for the failed partition.
However, the official documentation doesn't talk about this error possibility, so I think it would be nice to leave a cautionary note in the documentation. What do you think about this?

Thank you, @bky373 , for investigation!
So, according to your findings it  looks like the problem is even reproducible with Spring for Apache Kafka, but really with plain Apache Kafka client and respective concurrency and seeking.
Which, probably, just fully left up to target project implementation.
So, how about instead of throwing that exception up to the listener container thread, just emit a WARN into logs and ignore respectively?
That is really volatile illegal state, but apparently it is out of our client control: the broker may decide to evict this consumer from a group at any moment.
Another mitigation could be done via onPartitionsRevoked/onPartitionsLost callbacks to clean those partitions up from our seeks to avoid possible WARN on the next poll.
According to the ConsumerRebalanceListener, it is called from the Consumer.poll(), so that should be safe to modify our seeks queue since it is going to be used only on the next poll.
It might be turned into LinkedHashMap for easier handling.
However it might be also guarded by a dedicated Lock since all those seek...() method might be called from different threads.
Another way is just to ignore onPartitionsRevoked/onPartitionsLost and filter out seeks loop according to actual assignment.
I don't see a reason in warning end-user that partition we would like to seek is not under our control any more.
The race condition with the broker is really good to notify, at least as a WARN into logs.
We may think about some special PartitionNotAssignedEvent, but that might be done later, on demand.
WDYT?

@artembilan
Thanks for your comment!
Hmm... I guess I'm still not convinced. I'm still concerned about whether WARN is the right log level.
End-users will expect and use AbstractConsumerSeekAware for the benefit of being able to move the offset in a non-disruptive way at runtime, and the current ERROR log seems good enough to let them know that the seeking failed.
Considering that resetting the offset is usually to (re)process the message, if it fails, we should probably take immediate action, such as a re-request; in this respect, I think the ERROR is explicit..., But perhaps there is a point where it seems unreasonable. Can you tell me a little bit more about the point where it seems unreasonable?
I would greatly appreciate your response, as it would help me understand better! Thank you, as always.

Well, when it is an ERROR that means something is wrong with an application.
In our case the partitions assignment on the broker is out of our control.
At least in this race condition when we have already sent seek command, but didn't receive onPartitionsRevoked/onPartitionsLost yet.
I don't know Kafka algorithm in details, but we can receive onPartitionsRevoked/onPartitionsLost only when we call poll.
At the same time we don't want to call poll because we want to seek before that.
So, that is that kind of race condition which we are facing.
(Don't get me wrong, but that's my impression what we got here).
So, since this is not what has been caused by the application, therefore ERROR would be misleading.
The WARN is totally OK to notify that the state has been changed and we cannot seek partitions we don't owe any more.
Therefore, something else has to be done in the application to mitigate a situation.
But, no, that is indeed not an ERROR and application logic is OK.
And that's why I suggested that PartitionNotAssignedEvent to extend an application logic to deal with this race condition around revoked partition during seek.
Does that make sense?

Thanks! That helped me understand.
I guess it would be beneficial to test further when onPartitionsRevoked/onPartitionsLost events occur, as my previous output focused solely on partition assignment. I'll proceed with this and leave another comment afterward!
Due to time zone differences, it might take up to a day before I can get back to you.

@artembilan
After some more testing, the scenario for partition reassignment became a bit clearer.
Assume the following arbitrary conditions.

partition = 9
concurrency = 7

1. Consumers started

You can tell by the ConsumerStartedEvent being published.

2. Leading consumers join the group and monopolize the partitions

Some fast-moving consumers are assigned all partitions.
The broker property has group.initial.rebalance.delay.ms (default: 3000). Only consumers that send joinGroup requests within this time will be assigned partitions, while others will wait until the next rebalance occurs.

3. Store partition offsets for seek

This is done when you run the seek...() method.
Store the offset to seeks queue relative to the partition assigned in step 2.

4. Rebalancing consumers

Before the next pollAndInvoke(), the slow consumers join the group late.
Partition revocation/reassignment takes place at this time.

5. Errors during seeking

The rebalancing in step 4 changes the previously assigned partitions of the leading consumers to new partitions.
This causes the newly assigned partition to be different from the 'seeks' partition at the next polling, resulting in the error.

Now that I've found the cause of the error, I'm going to write a PR that includes two things.

Setting the group.initial.rebalance.delay.ms value generously so that the existing test code always succeeds without error with high concurrency. sample
And use WARN logs instead of ERROR when an error occurs in step 6. sample

logs

2024-08-09 01:20:41,787 WARN [org.springframework.kafka.KafkaListenerEndpointContainer#1-2-C-1] - No current assignment for partition Seek-5 due to partition reassignment prior to seeking. ```

And I didn't use the PartitionNotAssignedEvent because I wasn't sure how to use it well.

I think I'm fine with the plan.
Let's see some PR from you!
The WARN is enough for now: we can come back to the PartitionNotAssignedEvent when someone requests.


Connector crashes on request timeouts

Actual behavior
Blockchain connector microservice crashes when the requests from the Lisk node times out.
/Users/sameer/Documents/Lisk/github/lisk-service/services/blockchain-connector/shared/sdk/tokens.js:57
			throw new TimeoutException('Request timed out when calling \'getEscrowedAmounts\'.');
			      ^

TimeoutException: Request timed out when calling 'getEscrowedAmounts'.
    at getEscrowedAmounts (/Users/sameer/Documents/Lisk/github/lisk-service/services/blockchain-connector/shared/sdk/tokens.js:57:10)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:538:9)
    at process.processTimers (node:internal/timers:512:7)
    at async EventEmitter.<anonymous> (/Users/sameer/Documents/Lisk/github/lisk-service/services/blockchain-connector/shared/sdk/events.js:52:6)

Expected behavior
The application should not crash on timeouts.
Steps to reproduce

Start Lisk Service against Lisk Core v4.0.0-beta.1
Make sure the node is busy

Start Lisk Core v4.0.0-beta.1 on betanet with an old snapshot

Which version(s) does this affect? (Environment, OS, etc...)
v0.7.0-beta.3

Duplicate of  #1758.


sockets: Implementing poll(POLLOUT) support

Currently, zsock_poll(POLLOUT) implemented as unconditionally returning this flag. This means that if there're no free packet/data buffers, then user application will be effectively in busy-polling loop: POLLOUT will be returned, and app will try to send() to a socket and will get EAGAIN, will call poll() and immediately get POLLOUT, repeated.
There can be different solutions to this, of different level of overhead, "precision" (i.e. how big there's a chance still that POLLOUT will be returned from poll(), yet EAGAIN from send()), and changes required to Zephyr kernel and/or IP stack code. Btw, the fact that overhead and/or changes to code beyond BSD Sockets layer would be required, was the reason why such support wasn't implemented initially and postponed until later.
Two solutions on roughly opposite sides of "precision" spectrum:

The first idea is to limit number of outgoing inflight packets for a socket. If socket is currently sending out pkt, it's not POLLOUT'able. Implementation of that would require adding a semaphore to each socket (aka net_context, because socket is nothing but a net_context). send() would take semaphore, "sent" callback would give it. That alone doesn't guarantee high precision though, as a socket might not have an inflight pkt, and yet pkt pool might not have a free pkt either. To solve precision "problem", would need to associate a static outgoing pkt buffer with each socket. That's another field in socket structure, and (much) higher pressure on pkt pool. This still isn't going to resolve too much though, due to 2-level net allocation structure of Zephyr: to send out a pkt, it's required to allocate net_pkt, then allocate one of more net_buf, and attach to net_pkt. To guarantee something, would require to keep static net_pkt and net_buf per each socket.

Another idea is to poll pkt pool for free pkt slots. By definition, this method is not precise, because several sockets may compete for a single pkt buffer. And it's even less precise due to 2-level alloc structure, described above.

This is still the case, right?
At least in 2.7.
I tried using this to tell webserver running on Zephyr to slow down, but both select and poll return immediately.

Right, this has not been resolved yet.

Implemented in https://github.com/zephyrproject-rtos/zephyr/pull/45626


The tool for generating valid_c_idx.tensor, x.tensor, y.tensor

Could you please share your tool which you used to generate out the indices data valid_c_idx.tensor, x.tensor, y.tensor ?
Because of the file-missing issue https://github.com/Sense-GVT/Fast-BEV/issues/77, we cannot generate such data by the script/code provided in FastBEV.
Thank you

Resolved


WP: Flash/torch does not turn on

Hi! Thanks for a great lib!
I have a problem though, the flash won't turn on.
Trying this with Xamarin Forms, has no effect on either platform. (iOS, Droid, Wp).
(Lib works and we can scan codes - but would really like the flash be on by default).
iOS has a flash button, and that turns on the flash.
But doing:
scanner.Torch(true);
or
scanner.ToggleTorch();
before starting to scan has no effect.
(On Droid, scanner.IsTorchOn=true after the call, but on Wp = false)
Am I using it wrong?
Thanks

did you solve the issue yet? but I think you should be using the toggleTorch() method..
that's what i am using with the ZxingScannerView...

Thanks, I actually forked it and modded so the flash is turned on after the camera is initialized. The problem with creating a scanner, activating the torch then running the modal Scan-method, is that it's trying to activate the torch before the camera is initialized doesn't work.
Will push a pull request when the pressure drops in this project...

So apparently Flash doesn't work with the PhotoCamera class I'm using unless you are actually capturing an image (not using the preview frame).  I'm going to look into using the AudioVideoCaptureDevice class instead (which gives some additional control for other features) which should allow this.  For now, this will be an enhancement, sorry (I never had a device with flash to test on until recently)

Please follow #183 for this issue/enhancement

I'm using 2.1.0-beta1 and I had the same issue still.
Workaround was a Delay like so. Has the issue been adressed yet?
var scanTask = scanner.Scan();
await Task.Delay(100);
scanner.Torch(true);
var result = await scanTask;
scanner.Torch(false);

@wezzix I still can't set torch on before scanning using the default overlay.   And there is no torch button on the default overlay and no information on the documentation about the restrictions of using the torch like, "you can only use it with a custom overlay".  So many people probably spend countless hours trying to get it to work with the default overlay when I'm guessing it isn't possible.  But it would help if someone would clarify.


SECOAUTH-205: add ClientTrustStrategy

I think this is superceded by the UserApprovalHandler interface change (see comments in log on SECOAUTH-205).


Provide linux/ppc64le binary

It would be awesome if the team could produce binary for linux ppc64le.
It could be done with cross-compilation OR if needed I could provide you the access to my ppc64le linux box

Will be included in v0.7.3.


test: SVPI-621: rewrite and reenable spi get-file-content test

Description

Rewrite and reenable spi get-file-content test.

Issue ticket number and link
SVPI-621
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
 ./bin/e2e-appstudio --ginkgo.focus="spi-suite"
Checklist:

[x] I have performed a self-review of my code
[x] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] I have added meaningful description with JIRA/GitHub issue key(if applicable), for example HASSuiteDescribe("STONE-123456789 devfile source")
[ ] I have updated labels (if needed)

/retest

/retest

/retest

/retest

/retest

/retest

/retest

/retest

/retest


import playlist from youtube

Hi Ian, I would like to give this feature a try.

go ahead @zjjiang2

will give this issue a try :)