python38Packages.azure-mgmt-marketplaceordering: 1.0.0 -> 1.1.0

Automatic update generated by nixpkgs-update tools. This update was made based on information from passthru.updateScript.
meta.description for python38Packages.azure-mgmt-marketplaceordering is: "This is the Microsoft Azure Market Place Ordering Client Library"
meta.homepage for python38Packages.azure-mgmt-marketplaceordering is: "https://github.com/Azure/azure-sdk-for-python"
meta.changelog for python38Packages.azure-mgmt-marketplaceordering is: ""
Updates performed

Ran passthru.UpdateScript

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

Warning: a test defined in passthru.tests did not pass

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 1.1.0 with grep in /nix/store/rx8g0y779qvm959z8swx5d4b1lcsk3hm-python3.8-azure-mgmt-marketplaceordering-1.1.0

directory tree listing: https://gist.github.com/8257850ef63813a9bd75387f9d013ba8

du listing: https://gist.github.com/093511fe0f5da9b61a9ec9093eb69b35

Rebuild report (if merged into master) (click to expand)

12 total rebuild path(s)

3 package rebuild(s)

3 x86_64-linux rebuild(s)
3 i686-linux rebuild(s)
3 x86_64-darwin rebuild(s)
3 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
azure-cli
python38Packages.azure-mgmt-marketplaceordering
python39Packages.azure-mgmt-marketplaceordering

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/rx8g0y779qvm959z8swx5d4b1lcsk3hm-python3.8-azure-mgmt-marketplaceordering-1.1.0 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A python38Packages.azure-mgmt-marketplaceordering https://github.com/r-ryantm/nixpkgs/archive/749d90615b9e6964566f2bed63c1cc6a5efa9270.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/rx8g0y779qvm959z8swx5d4b1lcsk3hm-python3.8-azure-mgmt-marketplaceordering-1.1.0
ls -la /nix/store/rx8g0y779qvm959z8swx5d4b1lcsk3hm-python3.8-azure-mgmt-marketplaceordering-1.1.0/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

3 packages built:

azure-cli
python38Packages.azure-mgmt-marketplaceordering
python39Packages.azure-mgmt-marketplaceordering

Maintainer pings
cc @mwilsoncoding for testing.

Result of nixpkgs-review pr 117090 at 749d9061 run on aarch64-linux 1

3 packages built successfully:

azure-cli
python38Packages.azure-mgmt-marketplaceordering
python39Packages.azure-mgmt-marketplaceordering


Implement log file location parameter on docker run

We have several logging options, but we can't change the log file location.
Using the json-file driver we can find the file at /var/lib/docker/<container_id>/<container_id>-json.log (default), but it would be useful to set the log file name and path as we can set the max-size and max-file on json-file.
https://docs.docker.com/engine/admin/logging/overview/#/json-file-options
e.g.: docker run -d --log-driver=json-file --log-opt max-file=5 --log-opt file-path=/var/log/my-container.log my-image

The file written by the json-file logging driver are not intended for consumption by external software, and should be regarded the "internal storage mechanism" for the JSON logging driver. For that reason, the location is not configurable.
If you want to have the logs written to a different location, consider using (e.g.) the syslog driver, journald, or one of the drivers that allow sending the logs to a central log aggregator.

@thaJeztah Please, what is then the proposed and desired way of consuming the json-file logging driver produced logs? Since the file itself is just "internal storage mechanism" (i.e. we shouldn't access and read them directly), what other ways of obtaining the produced logs can we use in this case (other than docker logs CONTAINER)? Thank you.

docker logs <container>, or using the Docker API is the way

Log plugin support is now merged and should be part of 17.05.
I do not think it makes sense to implement this in docker core, however can be implemented via plugins.
Here is a reference plugin that just wraps jsonfilelog: https://github.com/cpuguy83/docker-log-driver-test
If anyone wants to discuss this feel free to ping me on Slack, or come see me @ DockerCon!

@thaJeztah  It's very strange, because Kubernetes consumes Docker's LogPath.
I want to move all Docker logs into tmpfs due to reduce disk I/O, but... I can't. Very painful.

because Kubernetes consumes Docker's LogPath.

Yes, the LogPath should (IMO) never have been part of the information returned by the API, or at least have been documented as an internal implementation details (not for external consumption). It's quite unfortunate that Kubernetes chose to consume those files directly instead of using the API; it's one of the consequences of Docker ever having been designed as a runtime for Kubernetes.
Generally, as a runtime for Kubernetes, running Kubernetes directly on containerd (which supports the  container runtime interface (CRI)) https://github.com/containerd/containerd. Containerd was designed from the ground up to be used as a runtime for Kubernetes (removing parts that are not needed for that purpose); in that setup, dockerd (through  docker-shim) is taken out as intermediate daemon between kubernetes and containerd.
From that perspective, I don't think we want to implement this feature

@thaJeztah
Thank you for the detailed explanation!
I opened fatrace tool: Now I see that only Docker daemon is accessing json log file (now I have Kubernetes 1.16). Maybe it was a problem in later k8s releases.
Now I'd like to examine docker's json driver code to understand how it interracts with write() system calls and ring buffer.
Trying to find workaround: don't want to lose logs in a non-blocking mode.

If you don't need the JSON files, I would also recommend having a look at the local logging driver https://docs.docker.com/config/containers/logging/local/, which should be more performant than the json-file driver, and defaults to use log-rotation.


Help with modcharts

Describe your problem here.
So I was trying to recreate the Ectospasm modchart in Lua, but I'm unable to recreate the last part where the entire HUD sways about twice as fast.
Here's the code for it, I don't know what I'm missing or doing wrong.
if swayfast then
    doTweenAngle('swayfast', 'camHUD', 6 * math.sin(currentBeat / 2), 0.01, 'linear')
end

Note: It's not that it won't sway, but rather it sways slowly.
For reference, in the same modchart, there are parts where it sways slowly, with the code for that below.
if sway then
    doTweenAngle('sway', 'camHUD', 5 * math.sin(currentBeat / 2), 0.1, 'linear')
end

I'm essentially trying to make the last part sway about twice as fast, but it always just sways just the same.
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
No response

Alright, apparently this one is my fault that I didn't read the modchart correctly. I'ma close this issue, if the moderators see this, pls delete this issue if you can.


Fix incorrect redirection when creating a PR fails

This is only a quick fix to make it easier to backport.
After this PR gets merged, I will propose a new PR to fix the FIXME.

I removed all these {{if .Flash.WarningMsg}} code blocks. Could you check whether it works as your expectation? @wolfogre


Add export type documentation (+ small updates)

Summary
Adds documentation for the exportType parameter, adds it to an example, and adds other small changes to the OperationDefinition.
Testing Guidance

To run locally, navigate to the BulkSubmitData directory and run ./_genonce.sh. You may need to run the ./_updatePublisher.sh script first if the publisher cannot be found.
Open output/index.html to see the IG.
Check that all parameters are accounted for, all examples are accurate, etc. Check the bulk import pnp proposal to see if anything included in that proposal should also be reflected in this operation definition

Question: Is it worth adding anything to the OperationDefinition about how the supplied parameters may include additional FHIR Bulk Data Export kickoff parameters (e.g. _since)?

I'm a little concerned that http://hl7.org/fhir/R4/async.html requires the prefer HTTP header, so doing the current (synchronous) $submit-data would now need prefer: ??? (not respond-async).  The current $submit-data does not require the prefer header as far as I know.
I don't think the OperationDefinition has a computable way of saying to use the Async pattern; it is just stated in the text or description.
The markdown (.md) talks about the required header, but it should probably be stated in the FSH somewhere also.

I'm a little concerned that http://hl7.org/fhir/R4/async.html requires the prefer HTTP header, so doing the current (synchronous) $submit-data would now need prefer: ??? (not respond-async). The current $submit-data does not require the prefer header as far as I know.
I don't think the OperationDefinition has a computable way of saying to use the Async pattern; it is just stated in the text or description.
The markdown (.md) talks about the required header, but it should probably be stated in the FSH somewhere also.

I believe omission of the prefer header would trigger the synchronous submit data, it doesn't need to be set to a different value. I don't know of a way to specify a required header within the OperationDefinition itself (only required parameters)


Don't use interpolation if it's not required (ui/api)

Purpose or Intent
Interpolation syntax can be error prone so don't use it when you can just convert to string.
Prefer variable.to_s over "#{variable}"
Also, some places we use Array#join which returns a String already:
"#{rate_detail.errors.full_messages.join(', ')}"
Becomes:
rate_detail.errors.full_messages.join(', ')
rubocop -a --only "Lint/StringConversionInInterpolation,Style/UnneededInterpolation" app/controllers app/presenters app/helpers spec/controllers spec/requests

I'm not fixing the other cops.  Some of this code really needs some ❤  but I'm holding back and only removing the unneeded interpolations.  @chessbyte I think I got all of your comments resolved.  Please review.

@jrafanie Uh-oh, found a problem...
-        page.replace("form_role_visibility", :partial => "layouts/role_visibility", :locals => {:rec_id =><EMAIL_ADDRESS>|| "new"}", :action => "widget_form_field_changed"})
 +        page.replace("form_role_visibility", :partial => "layouts/role_visibility", :locals => {:rec_id => (@widget.id.to_s || "new"), :action => "widget_form_field_changed"})

This is not an identical change, when @widget.id = nil, it would previously return "new" because (nil || "new").to_s, but now, it returns "" because (nil.to_s || "new") is ("" || "new")...
We may want to fix all of those...

Good find @himdel


Hcb checksinterval

I have worked on the feature to extend check periods and grace periods to a maximum of 5 years. Please review my chnages

I have made the requested changes, please review


[FEATURE]: C++ Style Interface

Is there an existing issue for this?

[X] I have searched the existing issues

New feature
State DB currently has a C style interface. It would be cleaner to have a proper C++ style interface both for interacting with nodes as well as managing locking of State DB.
Anything else?
No response

This issue is stale because it has been open for 30 days with no activity.


Event handling ?

Hello,
So in you package, you fire some events when the model is updated, deleted or created.
Example :
A random model is updated. The repository fire an event : RepositoryEntityUpdated.
But what if the listener should have a different response depending on the action ?
Maybe, I want to send an email when the "status" attribute is modified and a different one, when the "type" attribute is modified.
Maybe, I want to fire an event depending on the route.
But the listener always catch the same event right ? (RepositoryEntityUpdated)
So is there any clean solution here ? Or do we have to fire our own events ?
Thanks,

+1. Faced the same problem and what I did as a workaround is to create an Interface to be implemented by the model. On the listener, I then check if the model is an instance of an interface then perform the necessary tasks. I'm also interested to know a better approach? Thanks!

I've been very busy in recent months with my work projects, which eventually pulling away of my personal projects, so it could not provide the necessary support to you, I am more relaxed now and I'm checking all the issues in order to work on fixes and new implementations, apologize to everyone.

@yajra did you found a better approuch?
I see that I can get the class name without creating interface like:

    public function handle(RepositoryEntityCreated $event)
    {
        Log::info('action: ' . $event->getAction() . ' - model: ' . get_class($event->getModel()));
    }


Back button not visible when in Tablet mode.

1: Put Windows in Tablet mode.
2: Launch Minimal
3: Enter text on Main page and click submit
4: You are brought to the details page, but no back button is visible.
(the above occurs regardless of the type of back button used (shell drawn vs in page header)
If you turn off tablet mode, the back button re-appears.

Further..
if in tablet mode and you use the back button on the Details page that is beside the Windows button in the taskbar, the Minimal app exits.

In tablet mode, use the software back button, it is located on the taskbar next to the Start button

Well that goes against what the "News" app does.
If launched in tablet mode, News does present the back button in the page
header.
Also, if I do use the back button in the taskbar in Minimal on the Details
page, the app exits instead of returning to the Main page.
On Fri, Aug 28, 2015 at 10:45 AM, Dmitry<EMAIL_ADDRESS>wrote:

In tablet mode, use the software back button, it is located on the taskbar
next to the Start button
—
Reply to this email directly or view it on GitHub
https://github.com/Windows-XAML/Template10/issues/53#issuecomment-135842597
.

Did you look in the TaskBar?

No. I mean page header.
It is removed when in tablet mode.
News / Money app show back button in page header when in tablet mode.
Should T10 do the same (?)

The T10 on-canvas back button will automatically hide itself in tablet mode only if the shell-drawn back button is visible in the taskbar. News does not do this. It has two back buttons. I think that is an error, actually. Two back buttons? Is that what you are asking for?

No, two back buttons doesn't sound correct.
Close issue.


Column names with space is not supported

Having some data in Elasticsearch, I have some fields like "TOTAL AMOUNT" which contains space
{{TOTAL AMOUNT}} results with
Couldn't build text from template - Missing helper TOTAL
I tried to use Grafana Transform / Organize Fields and RENAME to rename fields with spaces
"TOTAL AMOUNT" --> total
But seems that transforms is not respected by plugin
Also someone mentioned using handlebar syntax for fields with spaces
{{{[TOTAL AMOUNT]}}}
But no data is shown
Is there some additional trick so one can use columns with space?
In elastic I use
Metric Max: TOTAL
Group BY: Terms SOURCE
Then By Date Histogram datetime, interval 10m
So here I don't know hot to rename particular SOURCEs (multiple totals with names containing spaces) to something which can be used here.

@igorantolic It works as expected with quotes and brackets using Static Data Source and the latest version of Dynamic Text:

The latest version also supports Transformations.

Let me know if I am missing something.


[FEAT]: Implement API Endpoint for User Password Reset - Backend

Description
Develop an endpoint to handle requests to reset password for registered users. If the password reset is successfully, it will be returned to the client with a '200' status. If an error occurs, an appropriate error status will be returned.
Acceptance Criteria

Validate that all required fields (email, newPassword, resetToken) are present in the request body.

Ensure the newPassword meets the defined strength criteria (e.g., minimum length, complexity)

Verify that the reset token is valid and not expired. Check that the reset token matches the user’s email.

Returns a  200  status code and the correct response body when password is successfully updated and hashed into the database.

Returns a  404  status code and the correct response body when user is not found.

Returns an appropriate error message when an error occurs.

Send an email notification to the user upon successful password update

Maintain audit logs of password reset requests for forensic purposes.

Purpose
Implement endpoints to facilitate secure password updates for users who have forgotten their passwords or suspect their credentials have been compromised.
Requirement

[ ]  Verify the user's identity before allowing a password reset.

[ ]  Use HTTPS to encrypt all password reset-related communications.

[ ]  Create a unique, time-limited token for each password reset request

[ ]  Associate tokens with the correct user account

[ ]  Enforce strong password policies (e.g., minimum length, complexity)

[ ]  Hash the new password using a strong, slow hashing algorithm (e.g., bcrypt, Argon2)

Expected Outcome

Users are able to update their password successfully.

STEP 1
Endpoints
[POST] /api/v1/request-password-request
Description: Initiates the password reset process by verifying the user's email and sending a reset token.
Request
Header Content-Type: application/json
Body

[ ]  json

{
 "email": "string"
}

Response
Success (200 OK)
{
  "message": "If a user with that email exists, a password reset link has been sent."
}

Error Response If user cant be found (400 Bad Request)

[ ]  json

{
  "status": "error",
  "message": "Invalid input data",
  "errors": [
  {
  "field": "email",
  "message": "Invalid email"
  }
  ]
}

Server-side error(500 Internal Server Error)

[ ]  json

{
  "status": "error",
  "message": "Internal server error",
  "errorCode": "SERVER_ERROR_001"
}

STEP 2
Endpoints
[POST] /api/v1/reset-password
Description: Resets the user's password using the token received via email.
Request
Headers
Content-Type: application/json
X-Reset-Token: <reset_token

Body

[ ]  json

{
  "new_password": "string"
}

Response
Successful Password Update Status: 200 OK

[ ]   json

{
 "message": "Password updated successfully.",
 "status_code": 200
}

Invalid Input or Token Status: 400 Bad Request
{
 "message": "Invalid input or token",
 "status_code": 400
}

Server-side error Status: 500 Internal Server Error

[ ]  json

{
  "message": "Server error during processing.",
 "status_code": 500,
 }

Tests
Validation and Input Checks:

[ ]  Verify that all required fields (ewPassword, resetToken) are present in the request body.

[ ]  Ensure that the newPassword meets the defined strength criteria (e.g., minimum length, complexity requirements).

Token Validation:

[ ]  Verify that the reset token is valid and not expired.

[ ]  Check that the reset token matches the user’s email.

Response Scenarios:
Successful Password Update:

[ ]  Send a request with a valid reset token and a strong new password.

[ ]  Expect a 200 OK status code.

[ ]  Verify that the response body contains:

{
  "message": "Password updated successfully.",
  "status_code": 200
}

Invalid Input or Token:

[ ]  Send a request with missing fields or an invalid reset token.

[ ]  Expect a 400 Bad Request status code.

[ ]  Verify that the response body contains:

{
  "message": "Invalid input or token",
  "status_code": 400
}

User Not Found:

[ ]  Send a request with a valid reset token but an email not associated with any user.

[ ]  Expect a 404 Not Found status code.

[ ]  Verify that the response body contains:

{
  "message": "User not found",
  "status_code": 404
}

Server Error:

[ ]  Simulate a server error during processing (e.g., database failure).

[ ]  Expect a 500 Internal Server Error status code.

[ ]  Verify that the response body contains:

{
  "message": "Server error during processing.",
  "status_code": 500
}

Security Measures:

[ ]  Ensure HTTPS is used for all password reset-related communications.

[ ]  Create unique, time-limited tokens for each password reset request.

[ ]  Enforce strong password policies (length, complexity).

[ ]  Hash the new password using a strong hashing algorithm.

@Laban254 please could you assign the task me.


Support Expr::Case in aggregate module

Thanks to @24seconds

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

22 of 22 (100.0%)  changed or added relevant lines in 2 files are covered.
4 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.02%) to 97.865%

Files with Coverage Reduction
New Missed Lines
%

core/src/executor/context/filter_context.rs
4
93.44%

Totals

Change from base Build<PHONE_NUMBER>:
0.02%

Covered Lines:
38047

Relevant Lines:
38877

💛  - Coveralls

Added missing implementation of Expr::Case to check function.


Found this sphere near me!

As credit, attach my social media link: https://twitter.com/FrostKiwiOne

Here's a mirrorball I found at the IAH airport in Houston about a month after your first video. I will be returning to the airport next week, so I may be able to get a picture with a better camera.
these cropping values look decent:
top: 800
bot: 1426
left: 721
right: 572
fov: 294

Here's a mirrorball I found at the IAH airport in Houston about a month after your first video. I will be returning to the airport next week, so I may be able to get a picture with a better camera. these cropping values look decent: top: 800 bot: 1426 left: 721 right: 572 fov: 294

Many Many thanks! <3
The first submission, from the other side of the world. To me this is truly special.
It's online in the "User Submission" tab now. If there is any other way you might wanna be credit, pls mention.


add icon on each diff header

Add delete/add icon on each diff header.
Because it is difficult to be discriminated binary file has been added or deleted.

↓

@nazoking Good pull request, thanks. We want to add diff indicator like GitHub actually :-)


upgrade AA to 0.6.0

Issue Type

[ ] bug report
[ x] feature request

Current Behavior
https://github.com/eth-infinitism/bundler/releases/tag/v0.5.0
Expected Behavior

https://github.com/eth-infinitism/bundler/releases/tag/v0.6.0
update the documentation

Steps to Reproduce

Suggested Fix

the changes for this should already be in with https://github.com/bobanetwork/boba/pull/757. Will double-check this 😄


Storage performance improvements

Closing for now as the gain from this is minimal so we should probably stick to only part of those fixes.


test PR

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: Jeffwan
To complete the pull request process, please assign
You can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

The following users are mentioned in OWNERS file(s) but are untrusted for the following reasons. One way to make the user trusted is to add them as members of the Jeffwan org. You can then trigger verification by writing /verify-owners in a comment.

jinchihe

User is not a member of the org. User is not a collaborator. Satisfy at least one of these conditions to make the user trusted.

@Jeffwan: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

jeffwan-pytorch-operator-presubmit
b76cf2060a4564e2e6bf5fed75f71a3e9f04b9b5
link
/test jeffwan-pytorch-operator-presubmit

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

The following users are mentioned in OWNERS file(s) but are untrusted for the following reasons. One way to make the user trusted is to add them as members of the Jeffwan org. You can then trigger verification by writing /verify-owners in a comment.

jinchihe

User is not a member of the org. User is not a collaborator. Satisfy at least one of these conditions to make the user trusted.


Make launch error:   no member named 'PostPhysicsComponentTick'

The following error appears when I make lauch, hope someone can help me.
tecnalia@U110593:~/carla$ make launch
Setup.sh: llvm-7.0 already installed.
Setup.sh: boost-1.69.0-c7 already installed.
Setup.sh: rpclib-v2.2.1_c2-c7 already installed.
Setup.sh: gtest-1.8.1-c7 already installed.
Setup.sh: recast-c7 already installed.
Setup.sh: CARLA version 0.9.6-53-g3e15ed3a.
Setup.sh: Generating CMake configuration files.
Setup.sh: Success!
BuildLibCarla.sh: Building LibCarla "Server.Release" configuration.
ninja: no work to do.
[0/1] Install the project...
-- Install configuration: "Server"
BuildLibCarla.sh: Success!
BuildCarlaUE4.sh: Using Unreal Engine at '/home/tecnalia/Escritorio/UnrealEngine/UnrealEngine-release'
BuildCarlaUE4.sh: Build CarlaUE4 project.
make[1]: Entering directory '/home/tecnalia/carla/Unreal/CarlaUE4'
bash "/home/tecnalia/Escritorio/UnrealEngine/UnrealEngine-release/Engine/Build/BatchFiles/Linux/RunMono.sh" "/home/tecnalia/Escritorio/UnrealEngine/UnrealEngine-release/Engine/Binaries/DotNET/UnrealBuildTool.exe" CarlaUE4Editor Linux Development  -project="/home/tecnalia/carla/Unreal/CarlaUE4/CarlaUE4.uproject"
Running Mono...
Fixing inconsistent case in filenames.
Setting up Mono
~/Escritorio/UnrealEngine/UnrealEngine-release/Engine ~/carla/Unreal/CarlaUE4
Using 'git status' to determine working set for adaptive non-unity build (/home/tecnalia/Escritorio/UnrealEngine/UnrealEngine-release).
Performing 4 actions (8 in parallel)
[1/4] Compile Module.Carla.cpp
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Intermediate/Build/Linux/B4D820EA/UE4Editor/Development/Carla/Module.Carla.cpp:2:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Actor/ActorBlueprintFunctionLibrary.cpp:8:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Actor/ActorBlueprintFunctionLibrary.h:14:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/GnssSensor.h:9:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/Sensor.h:9:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Game/CarlaEpisode.h:11:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/WorldObserver.h:9:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/DataStream.h:9:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/AsyncDataStream.h:13:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/CarlaDependencies/include/carla/streaming/Stream.h:9:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/CarlaDependencies/include/carla/streaming/detail/MultiStreamState.h:9:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/CarlaDependencies/include/carla/streaming/detail/StreamStateBase.h:10:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/CarlaDependencies/include/carla/streaming/detail/Session.h:9:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/CarlaDependencies/include/carla/streaming/detail/tcp/ServerSession.h:18:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/CarlaDependencies/include/boost/asio/ip/tcp.hpp:25:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/CarlaDependencies/include/boost/asio/ip/basic_resolver.hpp:28:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/CarlaDependencies/include/boost/asio/ip/basic_resolver_query.hpp:21:
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/CarlaDependencies/include/boost/asio/ip/resolver_query_base.hpp:19:
/home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/CarlaDependencies/include/boost/asio/ip/resolver_base.hpp:69:5: warning: declaration shadows a variable in namespace 'boost::asio::ip' [-Wshadow]
v4_mapped = BOOST_ASIO_OS_DEF(AI_V4MAPPED),
^
/home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/CarlaDependencies/include/boost/asio/ip/address_v6.hpp:291:20: note: previous declaration is here
enum v4_mapped_t { v4_mapped };
^
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Intermediate/Build/Linux/B4D820EA/UE4Editor/Development/Carla/Module.Carla.cpp:52:
/home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/PixelReader.cpp:57:3: warning: 'FTextureRHIParamRef' is deprecated: FTextureRHIParamRef typedef is deprecated; please use FRHITexture* directly instead. Please update your code to the new API before upgrading to the next release, otherwise your project will no longer compile. [-Wdeprecated-declarations]
FTextureRHIParamRef Texture = RenderResource->GetRenderTargetTexture();
^
/home/tecnalia/Escritorio/UnrealEngine/UnrealEngine-release/Engine/Source/Runtime/RHI/Public/RHIResources.h:1165:1: note: 'FTextureRHIParamRef' has been explicitly marked deprecated here
UE_DEPRECATED(4.23, "FTextureRHIParamRef typedef is deprecated; please use FRHITexture* directly instead.")
^
/home/tecnalia/Escritorio/UnrealEngine/UnrealEngine-release/Engine/Source/Runtime/Core/Public/Misc/CoreMiscDefines.h:227:43: note: expanded from macro 'UE_DEPRECATED'
#define UE_DEPRECATED(Version, Message) [[deprecated(Message " Please update your code to the new API before upgrading to the next release, otherwise your project will no longer compile.")]]
^
In file included from /home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Intermediate/Build/Linux/B4D820EA/UE4Editor/Development/Carla/Module.Carla.cpp:57:
/home/tecnalia/carla/Unreal/CarlaUE4/Plugins/Carla/Source/Carla/Sensor/Sensor.cpp:20:9: error: no member named 'PostPhysicsComponentTick' in 'UStaticMeshComponent'
Mesh->PostPhysicsComponentTick.bCanEverTick = false;
2 warnings and 1 error generated.
Makefile:210: recipe for target 'CarlaUE4Editor' failed
make[1]: *** [CarlaUE4Editor] Error 5
make[1]: Leaving directory '/home/tecnalia/carla/Unreal/CarlaUE4'
Util/BuildTools/Linux.mk:7: recipe for target 'launch' failed
make: *** [launch] Error 2

same issue I have!

You're running a newer version of Unreal Engine, Carla requires 4.22 at the moment.

Thank you nsubiron
I don't know if it was an Unreal Engine version problem. But the thing is that I downloaded Unreal directly from their webpage, and even its name was unreal engine 4.22 could have something different.
To sum up, I restart all the installing process from the Carla's documentation, Build in Linux, and it works.
Thank you again.
Jon.

Based upon a brief examination of the CARLA 0.9.6 codebase, that once deprecated, now removed API call isn't used elsewhere in the code (unless I'm missing something).  The attached patch file simply comments out the problematic portion of the code and allows compilation to proceed with UE 4.23.  Based on the other deprecation warnings generated, 4.24 will likely be more problematic.  Obviously, proceed with caution since upstream only supports UE 4.22 with CARLA 0.9.6.
ue4233_PostPhysicsComponentTick_fix.txt


How to use dash cash (kebab case) file names?

I've inherited an existing project that has gone through a few developers and has no rhyme or reason to its file structure (mixed camelCase.js, TitleCase.js, snake_case.js, etc).
I'm wrote a script to switch everything to dash-cash.js (the linux and node standard) and update the require()s, etc, but then the objection models stopped loading.
How do I tell objection to expect dash case file names for the models?

Objection doesn't care about the file names.

Objection doesn't care about the file names.

Ah, but it does: the field modelClass seems to be a misnomer - referring to a file in the same folder and not a class name per se.
So what looks like something related to code:
modelClass: 'Account'

It's actually the file name:
modelClass: './Account.js'

Perhaps modelClass has some other meaning and the file feature is for backwards compatibility, but that's what it does in the codebase I inherited.
Example
Existing code looks like this:
'use strict';

const { Model } = require('objection');
const BaseModelUser = require('./BaseModelUser');

module.exports = class User extends BaseModelUser {
  static tableName = 'user';

  static relationMappings = {
    accounts: {
      relation: Model.ManyToManyRelation,
      modelClass: 'Account',
      join: {
        from: 'user.id',
        through: {
          modelClass: 'AccountUser',
          from: 'account_user.user_id',
          extra: ['role_id', 'is_current'],
          to: 'account_user.account_id',
        },
        to: 'account.id',
      },
    },
    confirmations: {
      relation: Model.HasManyRelation,
      modelClass: 'ConfirmationCode',
      join: {
        from: 'user.id',
        to: 'confirmation_code.user_id',
      },
    },
  };
};

Updated code needs to look like this:
'use strict';

const { Model } = require('objection');
const BaseModelUser = require('./BaseModelUser');

module.exports = class User extends BaseModelUser {
  static tableName = 'user';

  static relationMappings = {
    accounts: {
      relation: Model.ManyToManyRelation,
      modelClass: 'Account',
      join: {
        from: 'user.id',
        through: {
          modelClass: './account-user.js',
          from: 'account_user.user_id',
          extra: ['role_id', 'is_current'],
          to: 'account_user.account_id',
        },
        to: 'account.id',
      },
    },
    confirmations: {
      relation: Model.HasManyRelation,
      modelClass: './confirmation-code.js',
      join: {
        from: 'user.id',
        to: 'confirmation_code.user_id',
      },
    },
  };
};


gossip.h lacks newline at the end of file

…generated telehash.h does not compile as a result

Oops, thanks!


[Aio] Support for context

Acceptance Criteria

A mechanism for propagating context from server to client within the same process has been implemented.
The mechanism supports at least the following:

timeout
cancellation
middleware-definable distributed tracing information

We still want this

We still want this


Connection and channel handshake fails with non-batching transactions

Summary of Bug
With single message per Tx (i.e. max_msg_num = 1 in config) connection and channel creation fails
$ hermes create connection ibc-0 ibc-1
...
2022-03-16T15:43:14.842710Z ERROR ThreadId(35) send_tx_commit{id=ConnectionOpenTry}:send_tx{id=ibc-1}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: InvalidArgument, message: "failed to execute message; message index: 0: connection handshake open try failed: failed connection state verification for client (07-tendermint-2): client state height < proof height ({0 799} < {0 801}), please ensure the client has been updated: invalid height: invalid request", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
2022-03-16T15:43:14.842869Z ERROR ThreadId(01) failed ConnTry ConnectionSide { chain: BaseChainHandle { chain_id: ChainId { id: "ibc-1", version: 1 }, runtime_sender: Sender { .. } }, client_id: ClientId("07-tendermint-2"), connection_id: None }: failed during a transaction submission step to chain id ibc-1: gRPC call failed with status: status: InvalidArgument, message: "failed to execute message; message index: 0: connection handshake open try failed: failed connection state verification for client (07-tendermint-2): client state height < proof height ({0 799} < {0 801}), please ensure the client has been updated: invalid height: invalid request", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }

Packet relaying still works as we check for this particular failure during simulation.

Version
master and all recent ones

Steps to Reproduce

Acceptance Criteria

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate milestone (priority) applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

I can reproduce this error with #1980.
I have also tried checking if it is CachingChainHandle that caused the issue, but it does not appear to be so. Therefore the error might be from elsewhere.

Should we simply make max_msg_num a domain type and enforce that it has a value > 1 ?

Should we simply make max_msg_num a domain type and enforce that it has a value > 1 ?

Why can't we fix it in the same way we do for packet case? I am asking because i sometime use max_msg_num for testing purposes.

Why can't we fix it in the same way we do for packet case?

What do you mean by that?

Why can't we fix it in the same way we do for packet case?

What do you mean by that?

For packet relaying we see the same error if using gaia v6.0.x. I think it's not there for gaia v7.0.x.
We used to handle the packet case by checking the simulation error, see: https://github.com/informalsystems/ibc-rs/blob/9bad6e2dedd462bd8e35538c9f0540e7f443e36f/relayer/src/chain/cosmos/estimate.rs#L148
For that specific client failure we would use the default. So my comment was to use the same approach. But...
I just checked master and this is not the case anymore so if a set of packet messages is split in two Tx-es, the second will fail. Packets will never be cleared. I also noticed that the connection setup also hits the same check above. But for some reason this doesn't catch the error anymore. I think if we fix this we solve both the connection/channel and packet problems. And we don't have to restrict the max_msg_num as in https://github.com/informalsystems/ibc-rs/pull/2163

These diffs seem to fix the issues:
diff --git a/relayer/src/error.rs b/relayer/src/error.rs
index 9fa4b160..2234ad26 100644
--- a/relayer/src/error.rs
+++ b/relayer/src/error.rs
@@ -513,7 +513,7 @@ impl GrpcStatusSubdetail {
     /// - status: InvalidArgument
     /// - message: verification failed: ... failed packet acknowledgement verification for client: client state height < proof height ...
     pub fn is_client_state_height_too_low(&self) -> bool {
-        if self.status.code() != tonic::Code::InvalidArgument {
+        if self.status.code() != tonic::Code::Unknown {
             return false;
         }

Still need to figure out if this would work for gaia versions before v6.0.x.

@ancazamfir Neat! Can you go into a bit more detail as to why this solves the problem?

These diffs seem to fix the issues:

While testing #2167 I'm encountering a similar problem that has this same fix. It seems gaiad v6.0.1 uses tonic::Code::InvalidArgument whereas v6.0.4 uses tonic::Code::Unknown.
More concretely, with v6.0.4 we get:

2022-05-04T09:51:57.482661Z ERROR ThreadId(11) send_tx_commit{id=nWl0l6enWK}:send_tx_with_account_sequence_retry{id=ibc-0}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 0: receive packet verification failed: couldn't verify counterparty packet commitment: failed packet commitment verification for client (07-tendermint-0): client state height < proof height ({1 12} < {1 19}), please ensure the client has been updated: invalid height", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "20"} }
Error: link error: failed with underlying error: [skipped]

With v6.0.1 we get:

2022-05-04T09:46:19.064742Z  WARN ThreadId(11) send_tx_commit{id=h5m5kQx361}:send_tx_with_account_sequence_retry{id=ibc-0}:estimate_gas: failed to simulate tx, falling back on default gas because the error is potentially recoverable: gRPC call failed with status: status: InvalidArgument, message: "failed to execute message; message index: 0: receive packet verification failed: couldn't verify counterparty packet commitment: failed packet commitment verification for client (07-tendermint-0): client state height < proof height ({1 15} < {1 34}), please ensure the client has been updated: invalid height: invalid request", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
2022-05-04T09:46:19.065096Z DEBUG ThreadId(11) send_tx_commit{id=h5m5kQx361}:send_tx_with_account_sequence_retry{id=ibc-0}: send_tx: using 10000000 gas, fee Fee { amount: "100001stake", gas_limit: 10000000 } id=ibc-0

I will do these changes Anca highlighted in the diff above directly in PR #2167.
Gaia v7.0.1 uses the same as 6.0.1.
More details about upstream changes in Gaia/SDK
Gaiad v6.0.1 uses sdk 0.44.5
Gaiad v6.0.4 uses sdk 0.44.6
that's a patch version change. I had a quick look through the changelog but didn't find anything.

Maybe in the fix we should relax the self.status.code() check so we cover multiple gaia versions?


Added Transactional annotation.

Added to ensure connection obtained from db has autocommit false.
Fixes Issue
fixes #386
Changes proposed
Add Transactional annotation to method.
Check List (Check all the applicable boxes)

[x] My code follows the code style of this project.
[x] All existing tests passed.

Rebased, and merged - and back-ported.
Thanks for your contribution !


Error on second task

Hey there, I got this from role from ansible-galaxy, but I'm encountering an issue when the role runs. The error is:
TASK: [groover.maven | Install Maven download directory (local)] **************
failed: [default] => {"failed": true, "item": "", "parsed": false}
invalid output was: [sudo via ansible, key=yxlbvosgslyqorwbhexokkxeyczgcprc] password:

Do you have any thoughts on what this might be?
My playbook is:
-
  hosts: all
  remote_user: vagrant
  sudo: yes
  vars:
    app_base_path: "/vagrant/"
    default_user: "vagrant"
  roles:
    - briancoca.oracle_java7
    - groover.maven

I ran into the same issue, and I am using this role in my playbook via Vagrant 1.7.2 and Ansible 1.8.4

Hi,
I'm currently working on a general purpose fix for local download and upload of redistributables.

Fixed with current release.


Test against Python 3.8 on CI

Not sure if GitHub actions support Python 3.8 beta

Done in 962921a


Handle group name was failed to be written to ZK gracefully

Motivation
Currently KoP writes group name and the associated client's host and id to ZK so that consumer stats could be updated while FETCH request contains no group info. However, there is something wrong with ZooKeeperUtils.
First, when the groupIdStoredPath already existed, the write to ZK may fail with NodeExists because of race condition.
Second, if the data was failed to be written to ZK, every time a FETCH request comes, a ZK read operation will be performed. It's because ZooKeeperUtils#getData returns null if the node doesn't exist and a null value cannot be put into a Map. In this case, ConcurrentHashMap#computeIfAbsent will always execute the second argument to create a value.
Modifications

Create groupIdZooKeeperPath only when KoP starts, and only log the error message if ZK write failed.
When ZooKeeperUtils#getData failed, returns an empty string instead of null so that if the group name was failed to be written to ZK, the empty returned value would be cached.

Here're two logs screenshot from my test environment.

LGTM

@hangc0276 PTAL again


[SPARK-28367][SS] Use new KafkaConsumer.poll API in Kafka connector

What changes were proposed in this pull request?
Spark uses an old and deprecated API named KafkaConsumer.poll(long) which never returns and stays in live lock if metadata is not updated (for instance when broker disappears at consumer creation). Please see Kafka documentation and standalone test application for further details.
In this PR I've applied the new KafkaConsumer.poll(Duration) API.
How was this patch tested?
Existing unit tests + started Kafka to Kafka query on cluster.

Test build #107595 has started for PR 25135 at commit ad1863a.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Test build #107595 has finished for PR 25135 at commit ad1863a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107595/
Test PASSed.

cc @HeartSaVioR

Btw, just wondering, if you managed to test with situation delaying updating metadata (finally timed-out), then what would be occurred? Query fails? Or reattempt of relevant tasks happen?

Presume you're interested in what is the new behavior. If exception arrives withRetriesWithoutInterrupt catches all NonFatal and takes care of the reattempt.

Test build #107681 has started for PR 25135 at commit f92c758.

Test build #107681 has finished for PR 25135 at commit f92c758.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107681/
Test PASSed.

Ah OK there's already a retry loop. Thanks for the info.

LGTM again.

Test build #107731 has started for PR 25135 at commit 2968c2a.

Test build #107731 has finished for PR 25135 at commit 2968c2a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107731/
Test FAILed.

retest this please

Test build #107732 has started for PR 25135 at commit 2968c2a.

Test build #107732 has finished for PR 25135 at commit 2968c2a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107732/
Test FAILed.

Created a quick fix https://github.com/apache/spark/pull/25171

retest this please

Test build #107735 has started for PR 25135 at commit 2968c2a.

Test build #107735 has finished for PR 25135 at commit 2968c2a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107735/
Test FAILed.

Checking the issue...

Test build #107739 has started for PR 25135 at commit 2ed24d3.

Test build #107739 has finished for PR 25135 at commit 2ed24d3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/107739/
Test PASSed.

Little bit explanation to the last change:

While I was debugging the issue I've realized requirement failed message is not really talkative so added a meaningful message
When consumer.poll(Duration.ZERO) with Thread.sleep(100) used then under some circumstances even 2 minutes was not enough to get the assignment. To test this I've put the standalone application into a 1k execution loop. The end result is that with non-zero timeout (including Duration.ofMillis(1)) everything works fine.

So the end judgement is that the actual implementation works but as a side track I'm going to analyze the consumer.poll(Duration.ZERO) behavior with the Kafka guys.

@gaborgsomogyi yeah, I think it's better to ask Kafka guys to clarify how to use this new poll API. It seems it cannot be used directly without retrying.

@zsxwing the main approach is created in agreement with the Kafka guys. The only thing which bothers me is why producing the following approaches different results (maybe an edge case hit with 0?!).
Works:
...
    consumer.poll(jt.Duration.ZERO)
    var partitions = consumer.assignment()
    while (partitions.isEmpty && System.currentTimeMillis() - startTimeMs < pollTimeoutMs) {
      consumer.poll(jt.Duration.ofMillis(100))
      partitions = consumer.assignment()
    }
...

Fails consistently and no flakyness (worth to mention only couple of tests, so maybe tests are wrongly implemented):
...
    var partitions = Set.empty[TopicPartition].asJava
    while (partitions.isEmpty && System.currentTimeMillis() - startTimeMs < pollTimeoutMs) {
      consumer.poll(jt.Duration.ZERO)
      partitions = consumer.assignment()
      if (partitions.isEmpty) {
        Thread.sleep(100)
      }
    }
...

It feels me like consumer.poll(jt.Duration.ZERO) just doesn't work.

If that would be true several users would protest with blocker bugs on Kafka side but haven't seen such situation (polling without assignment is not possible). I've seen several heavy users who is doing the polling (and nothing other magic) with consumer.poll(Duration.ZERO). The use-case what I've seen is something like this:
wile (!interrupted) {
  val data = consumer.poll(Duration.ZERO)
  process(data)
}

Anyway, I'm going to have 2 weeks vacation but after that I'm going to sit together with the Kafka guys and having a deeper look at this (again, only couple of Spark tests are failing consistently and it may happen we're using the API not properly).

Here's a part of test code Kafka has been doing with new poll.
https://github.com/apache/kafka/blob/f98e176746d663fadedbcd3c18312a7f476a20c8/core/src/test/scala/integration/kafka/api/PlaintextConsumerTest.scala#L1748-L1752
  private def awaitAssignment(consumer: Consumer[_, _], expectedAssignment: Set[TopicPartition]): Unit = {
    TestUtils.pollUntilTrue(consumer, () => consumer.assignment() == expectedAssignment.asJava,
      s"Timed out while awaiting expected assignment $expectedAssignment. " +
        s"The current assignment is ${consumer.assignment()}")
  }

https://github.com/apache/kafka/blob/f98e176746d663fadedbcd3c18312a7f476a20c8/core/src/test/scala/unit/kafka/utils/TestUtils.scala#L767-L775
  def pollUntilTrue(consumer: Consumer[_, _],
                    action: () => Boolean,
                    msg: => String,
                    waitTimeMs: Long = JTestUtils.DEFAULT_MAX_WAIT_MS): Unit = {
    waitUntilTrue(() => {
      consumer.poll(Duration.ofMillis(50))
      action()
    }, msg = msg, pause = 0L, waitTimeMs = waitTimeMs)
  }

Kafka has still some parts of test code relying on deprecated poll(0) (so co-usage on both poll(Duration) and poll(long)). It might not be technical reason to do so, but they're still relying on old favor, which might mean they indicate the needs of usage on poll(0).
Sometimes Kafka calls updateAssignmentMetadataIfNeeded directly which deals with metadata update in poll() with max long timer, effectively blocking. The method is for testing: defined as package private.
consumer.updateAssignmentMetadataIfNeeded(time.timer(Long.MAX_VALUE));

In many cases of calling poll(Duration.ZERO) in test code, updateAssignmentMetadataIfNeeded is called prior to the call. In other cases the verification codes just seem to confirm calling poll doing nothing or returning already fetched records.
I guess in our case we need to either leverage updateAssignmentMetadataIfNeeded to only deal with metadata (it may require some hack and they clarified it's for testing so unsafe one), or poll with small timeout (50ms) with tolerating the case where record to pull is not available (incorporated in latency regardless of availability of metadata).
Btw, I'm seeing KIP-288 to propose new public API waitForAssignment similar to updateAssignmentMetadataIfNeeded but KIP-288 was discarded since KIP-266 superseded KIP-288, and KIP-266 didn't finally add it. Not sure it is declined or just missed it.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-288%3A+[DISCARDED]+Consumer.poll()+timeout+semantic+change+and+new+waitForAssignment+method

First of all thanks @HeartSaVioR for the deep look, it helped!
@zsxwing I've had another round with the Kafka guys on this and here are the conclusions:

The approach to call poll(Duration... in a loop is the suggested solution
The second approach failed because of the following: Originally the partition assignment was synchronous in the old API (and may hang infinitely) but with the new implementation proper timeout applied. In the mentioned tests "kafkaConsumer.pollTimeoutMs" -> "1000" is set. Since the partition assignment happens on my machine (and seems like on Jenkins as well) around 1000 ms it was mainly race luck whether the test passed or not. The first approach was more on the success side but the second was more on the failing side.

Here is the client log:
19/08/05 06:55:27.650 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE NetworkClient: [Consumer clientId=consumer-522, groupId=spark-kafka-source-42d70d54-e5ce-453a-a84c-40cd8126a3ff--1337036482-driver-2] Sending JOIN_GROUP {group_id=spark-kafka-source-42d70d54-e5ce-45$
19/08/05 06:55:28.758 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE KafkaConsumer: [Consumer clientId=consumer-522, groupId=spark-kafka-source-42d70d54-e5ce-453a-a84c-40cd8126a3ff--1337036482-driver-2] Closing the Kafka consumer
19/08/05 06:55:28.758 kafka-coordinator-heartbeat-thread | spark-kafka-source-42d70d54-e5ce-453a-a84c-40cd8126a3ff--1337036482-driver-2 DEBUG AbstractCoordinator: [Consumer clientId=consumer-522, groupId=spark-kafka-source-42d70d54-e5ce-453a-a84c-40cd8126a3ff--1337036482-driver-2] Heartbeat thread has closed
19/08/05 06:55:28.758 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE NetworkClient: [Consumer clientId=consumer-522, groupId=spark-kafka-source-42d70d54-e5ce-453a-a84c-40cd8126a3ff--1337036482-driver-2] Completed receive from node 0 for METADATA with correlation id 6$
19/08/05 06:55:28.759 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metadata: [Consumer clientId=consumer-522, groupId=spark-kafka-source-42d70d54-e5ce-453a-a84c-40cd8126a3ff--1337036482-driver-2] Determining if we should replace existing epoch 0 with new epoch 0
19/08/05 06:55:28.759 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metadata: [Consumer clientId=consumer-522, groupId=spark-kafka-source-42d70d54-e5ce-453a-a84c-40cd8126a3ff--1337036482-driver-2] Updating last seen epoch from 0 to 0 for partition failOnDataLoss-0-0
19/08/05 06:55:28.759 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metadata: [Consumer clientId=consumer-522, groupId=spark-kafka-source-42d70d54-e5ce-453a-a84c-40cd8126a3ff--1337036482-driver-2] Updated cluster metadata updateVersion 3 to MetadataCache{cluster=Clu$
19/08/05 06:55:28.760 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE NetworkClient: [Consumer clientId=consumer-522, groupId=spark-kafka-source-42d70d54-e5ce-453a-a84c-40cd8126a3ff--1337036482-driver-2] Completed receive from node<PHONE_NUMBER> for JOIN_GROUP with corre$
19/08/05 06:55:28.760 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=connection-count, group=consumer-metrics, description=The current number of active connections., tags={client-id=consumer-522}]
19/08/05 06:55:28.760 data-plane-kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-1 DEBUG Selector: [SocketServer brokerId=0] Connection with /<IP_ADDRESS> disconnected
java.io.EOFException
        at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:96)
        at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
        at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
        at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
        at kafka.network.Processor.poll(SocketServer.scala:863)
        at kafka.network.Processor.run(SocketServer.scala:762)
        at java.lang.Thread.run(Thread.java:748)
19/08/05 06:55:28.760 data-plane-kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-0 DEBUG Selector: [SocketServer brokerId=0] Connection with /<IP_ADDRESS> disconnected
java.io.EOFException
        at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:96)
        at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
        at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
        at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
        at kafka.network.Processor.poll(SocketServer.scala:863)
        at kafka.network.Processor.run(SocketServer.scala:762)
        at java.lang.Thread.run(Thread.java:748)
19/08/05 06:55:28.760 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=connection-close-total, group=consumer-metrics, description=The total number of connections closed, tags={client-id=consumer-522}]
19/08/05 06:55:28.760 data-plane-kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-2 DEBUG Selector: [SocketServer brokerId=0] Connection with /<IP_ADDRESS> disconnected
java.io.EOFException
        at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:96)
        at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
        at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
        at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
        at kafka.network.Processor.poll(SocketServer.scala:863)
        at kafka.network.Processor.run(SocketServer.scala:762)
        at java.lang.Thread.run(Thread.java:748)
19/08/05 06:55:28.760 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=connection-close-rate, group=consumer-metrics, description=The number of connections closed per second, tags={client-id=consumer-522}]
19/08/05 06:55:28.760 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name connections-closed:
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=connection-creation-total, group=consumer-metrics, description=The total number of new connections established, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=connection-creation-rate, group=consumer-metrics, description=The number of new connections established per second, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name connections-created:
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=successful-authentication-total, group=consumer-metrics, description=The total number of connections with successful authentication, tags={client-id=co$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=successful-authentication-rate, group=consumer-metrics, description=The number of connections with successful authentication per second, tags={client-i$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name successful-authentication:
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=successful-reauthentication-total, group=consumer-metrics, description=The total number of successful re-authentication of connections, tags={client-id$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=successful-reauthentication-rate, group=consumer-metrics, description=The number of successful re-authentication of connections per second, tags={clien$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name successful-reauthentication:
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=successful-authentication-no-reauth-total, group=consumer-metrics, description=The total number of connections with successful authentication where the$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name successful-authentication-no-reauth:
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=failed-authentication-total, group=consumer-metrics, description=The total number of connections with failed authentication, tags={client-id=consumer-5$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=failed-authentication-rate, group=consumer-metrics, description=The number of connections with failed authentication per second, tags={client-id=consum$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name failed-authentication:
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=failed-reauthentication-total, group=consumer-metrics, description=The total number of failed re-authentication of connections, tags={client-id=consume$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=failed-reauthentication-rate, group=consumer-metrics, description=The number of failed re-authentication of connections per second, tags={client-id=con$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name failed-reauthentication:
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=reauthentication-latency-max, group=consumer-metrics, description=The max latency observed due to re-authentication, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=reauthentication-latency-avg, group=consumer-metrics, description=The average latency observed due to re-authentication, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name reauthentication-latency:
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=network-io-total, group=consumer-metrics, description=The total number of network operations (reads or writes) on all connections, tags={client-id=cons$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=network-io-rate, group=consumer-metrics, description=The number of network operations (reads or writes) on all connections per second, tags={client-id=$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name bytes-sent-received:
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=outgoing-byte-total, group=consumer-metrics, description=The total number of outgoing bytes sent to all servers, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=outgoing-byte-rate, group=consumer-metrics, description=The number of outgoing bytes sent to all servers per second, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-total, group=consumer-metrics, description=The total number of requests sent, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-rate, group=consumer-metrics, description=The number of requests sent per second, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-size-avg, group=consumer-metrics, description=The average size of requests sent., tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-size-max, group=consumer-metrics, description=The maximum size of any request sent., tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name bytes-sent:
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=incoming-byte-total, group=consumer-metrics, description=The total number of bytes read off all sockets, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=incoming-byte-rate, group=consumer-metrics, description=The number of bytes read off all sockets per second, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=response-total, group=consumer-metrics, description=The total number of responses received, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=response-rate, group=consumer-metrics, description=The number of responses received per second, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name bytes-received:
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=select-total, group=consumer-metrics, description=The total number of times the I/O layer checked for new I/O to perform, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=select-rate, group=consumer-metrics, description=The number of times the I/O layer checked for new I/O to perform per second, tags={client-id=consumer-$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=io-wait-time-ns-avg, group=consumer-metrics, description=The average length of time the I/O thread spent waiting for a socket ready for reads or writes$
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=io-waittime-total, group=consumer-metrics, description=The total time the I/O thread spent waiting, tags={client-id=consumer-522}]
19/08/05 06:55:28.761 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=io-wait-ratio, group=consumer-metrics, description=The fraction of time the I/O thread spent waiting, tags={client-id=consumer-522}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name select-time:
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=io-time-ns-avg, group=consumer-metrics, description=The average length of time for I/O per select call in nanoseconds., tags={client-id=consumer-522}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=iotime-total, group=consumer-metrics, description=The total time the I/O thread spent doing I/O, tags={client-id=consumer-522}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=io-ratio, group=consumer-metrics, description=The fraction of time the I/O thread spent doing I/O, tags={client-id=consumer-522}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name io-time:
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=outgoing-byte-total, group=consumer-node-metrics, description=The total number of outgoing bytes, tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=outgoing-byte-rate, group=consumer-node-metrics, description=The number of outgoing bytes per second, tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-total, group=consumer-node-metrics, description=The total number of requests sent, tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-rate, group=consumer-node-metrics, description=The number of requests sent per second, tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-size-avg, group=consumer-node-metrics, description=The average size of requests sent., tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-size-max, group=consumer-node-metrics, description=The maximum size of any request sent., tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name node--1.bytes-sent
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=incoming-byte-total, group=consumer-node-metrics, description=The total number of incoming bytes, tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=incoming-byte-rate, group=consumer-node-metrics, description=The number of incoming bytes per second, tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=response-total, group=consumer-node-metrics, description=The total number of responses received, tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=response-rate, group=consumer-node-metrics, description=The number of responses received per second, tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name node--1.bytes-received
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-latency-avg, group=consumer-node-metrics, description=, tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-latency-max, group=consumer-node-metrics, description=, tags={client-id=consumer-522, node-id=node--1}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name node--1.latency
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=outgoing-byte-total, group=consumer-node-metrics, description=The total number of outgoing bytes, tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=outgoing-byte-rate, group=consumer-node-metrics, description=The number of outgoing bytes per second, tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-total, group=consumer-node-metrics, description=The total number of requests sent, tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-rate, group=consumer-node-metrics, description=The number of requests sent per second, tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-size-avg, group=consumer-node-metrics, description=The average size of requests sent., tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-size-max, group=consumer-node-metrics, description=The maximum size of any request sent., tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name node-0.bytes-sent
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=incoming-byte-total, group=consumer-node-metrics, description=The total number of incoming bytes, tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=incoming-byte-rate, group=consumer-node-metrics, description=The number of incoming bytes per second, tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=response-total, group=consumer-node-metrics, description=The total number of responses received, tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=response-rate, group=consumer-node-metrics, description=The number of responses received per second, tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name node-0.bytes-received
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-latency-avg, group=consumer-node-metrics, description=, tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-latency-max, group=consumer-node-metrics, description=, tags={client-id=consumer-522, node-id=node-0}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name node-0.latency
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=outgoing-byte-total, group=consumer-node-metrics, description=The total number of outgoing bytes, tags={client-id=consumer-522, node-id=node-2147483647$
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=outgoing-byte-rate, group=consumer-node-metrics, description=The number of outgoing bytes per second, tags={client-id=consumer-522, node-id=node-214748$
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-total, group=consumer-node-metrics, description=The total number of requests sent, tags={client-id=consumer-522, node-id=node-2147483647}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-rate, group=consumer-node-metrics, description=The number of requests sent per second, tags={client-id=consumer-522, node-id=node-2147483647}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-size-avg, group=consumer-node-metrics, description=The average size of requests sent., tags={client-id=consumer-522, node-id=node-2147483647}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-size-max, group=consumer-node-metrics, description=The maximum size of any request sent., tags={client-id=consumer-522, node-id=node-2147483647$
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name node-2147483647.bytes-sent
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=incoming-byte-total, group=consumer-node-metrics, description=The total number of incoming bytes, tags={client-id=consumer-522, node-id=node-2147483647$
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=incoming-byte-rate, group=consumer-node-metrics, description=The number of incoming bytes per second, tags={client-id=consumer-522, node-id=node-214748$
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=response-total, group=consumer-node-metrics, description=The total number of responses received, tags={client-id=consumer-522, node-id=node-2147483647}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=response-rate, group=consumer-node-metrics, description=The number of responses received per second, tags={client-id=consumer-522, node-id=node-2147483$
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name node-2147483647.bytes-received
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-latency-avg, group=consumer-node-metrics, description=, tags={client-id=consumer-522, node-id=node-2147483647}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=request-latency-max, group=consumer-node-metrics, description=, tags={client-id=consumer-522, node-id=node-2147483647}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG Metrics: Removed sensor with name node-2147483647.latency
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=version, group=app-info, description=Metric indicating version, tags={client-id=consumer-522}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=commit-id, group=app-info, description=Metric indicating commit-id, tags={client-id=consumer-522}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] TRACE Metrics: Removed metric named MetricName [name=start-time-ms, group=app-info, description=Metric indicating start-time-ms, tags={client-id=consumer-522}]
19/08/05 06:55:28.762 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] DEBUG KafkaConsumer: [Consumer clientId=consumer-522, groupId=spark-kafka-source-42d70d54-e5ce-453a-a84c-40cd8126a3ff--1337036482-driver-2] Kafka consumer has been closed
19/08/05 06:55:28.769 stream execution thread for DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] ERROR MicroBatchExecution: Query DontFailOnDataLoss [id = 9396b4b5-bc30-4e1e-8c6a-1d3a9d890ccf, runId = 5b1ee39c-c174-4c42-b0d5-1c4b59061ab3] terminated with error
java.lang.IllegalArgumentException: requirement failed
        at scala.Predef$.require(Predef.scala:268)
        at org.apache.spark.sql.kafka010.KafkaOffsetReader.getPartitions(KafkaOffsetReader.scala:431)
        at org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$fetchSpecificOffsets$2(KafkaOffsetReader.scala:172)
        at org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$withRetriesWithoutInterrupt$1(KafkaOffsetReader.scala:383)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at org.apache.spark.util.UninterruptibleThread.runUninterruptibly(UninterruptibleThread.scala:77)
        at org.apache.spark.sql.kafka010.KafkaOffsetReader.withRetriesWithoutInterrupt(KafkaOffsetReader.scala:382)
        at org.apache.spark.sql.kafka010.KafkaOffsetReader.$anonfun$fetchSpecificOffsets$1(KafkaOffsetReader.scala:171)
        at org.apache.spark.sql.kafka010.KafkaOffsetReader.runUninterruptibly(KafkaOffsetReader.scala:350)
        at org.apache.spark.sql.kafka010.KafkaOffsetReader.fetchSpecificOffsets(KafkaOffsetReader.scala:171)
        at org.apache.spark.sql.kafka010.KafkaSource.$anonfun$initialPartitionOffsets$1(KafkaSource.scala:136)
        at scala.Option.getOrElse(Option.scala:138)
        at org.apache.spark.sql.kafka010.KafkaSource.initialPartitionOffsets$lzycompute(KafkaSource.scala:132)
        at org.apache.spark.sql.kafka010.KafkaSource.initialPartitionOffsets(KafkaSource.scala:100)
        at org.apache.spark.sql.kafka010.KafkaSource.getOffset(KafkaSource.scala:151)
        at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$constructNextBatch$3(MicroBatchExecution.scala:361)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:328)
        at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:326)
        at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:68)

To resolve this situation I've considered mainly 2 things:

Increase kafkaConsumer.pollTimeoutMs
Introduce an additional timeout for partition assignment

The second approach is more like an overkill from my point of view. Such configuration can be added later if we see the need.

Test build #108719 has started for PR 25135 at commit 69c7ca5.

Test build #108720 has started for PR 25135 at commit 82f1520.

Test build #108719 has finished for PR 25135 at commit 69c7ca5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108719/
Test PASSed.

Test build #108720 has finished for PR 25135 at commit 82f1520.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/108720/
Test PASSed.

To make a small extract about the discussion in the Kafka dev channel the proposal where tends to be agreement is the following:

Call AdminClient.listTopics to get possible topic for subscribe pattern (in case of assign and subscribe such step not needed). Here there are 2 possibilities.

Do the filtering in Spark code
Kafka adds regex to the mentioned API

Worth to mention such filtering is happening on the client side in Consumer at the moment so this would not cause any unwanted extra memory consumption.

KIP-396 would add AdminClient.listOffsets API from where offsets can be obtained without data polling.

@gaborgsomogyi thanks for coordinating with the Kafka community. I think we can close this PR for now and revisit it when they have a release with the new API.

@zsxwing yep, not much to do for now. Thanks for your help finding this out.
@HeartSaVioR thanks for initiating + pushing the discussion forward.
I'm going to open a new PR when the API is available...

@HeartSaVioR gave me the pointer of this PR.
Since Apache Kafka 2.4.0 is released, it would be great to have this in 3.0. Thanks.

Sorry to make confusion. Proposal of KIP-396 was accepted in time, but the code seems to be merged after code freeze of Kafka 2.4. (Though it's just 4 days later... I thought it was merged in time.)
We may need to wait on Kafka 2.5, or exclude poll(0) and go ahead, and revisit poll(0) in Kafka 2.5.

Thank you for updates, @HeartSaVioR .

@Tagar
Please refer the comment in JIRA issue:
https://issues.apache.org/jira/browse/SPARK-28367?focusedCommentId=17088514&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17088514
In short, it's OK to use the new API in executors (we may need to set longer timeout then), but still not be ideal to use it in driver. It requires Spark to implement the subscription modes by itself with AdminClient API, which is not impossible but brings complexity to match the behavior same as we expect from Kafka client.

That said, it can be unblocked if we concern about infinite timeout seriously (see the behavior fairly often) and consider as critical one tolerating additional complexity brought in driver side.
I'm even OK to go with partial address, fix executors usage first, and leave an issue for driver. This would greatly reduce the chance of infinite timeout - only Kafka consumer in driver can hang.
@zsxwing @gaborgsomogyi WDYT?

@HeartSaVioR I think it makes sense to split up the problem so created sub-tasks and executor side PR is on the way. The driver side requires further attention and discussion w/ Kafka. Re-implementing the subscription in Spark is an overkill even if we would like to avoid infinite wait. We need another way...

@gaborgsomogyi and @HeartSaVioR thanks. saw the first PR for SPARK-32033 - that's great!