Fix Bug and refactoring

I have fixed a bug (index out of range)  in file merge.py  and do some beautification and improvements in the files:

check_input.py
calculator.py

And I add my python lib called primelib. This python library contains some useful functions to deal with
prime numbers and whole numbers.

Thank you for you contribution to the repo and the community

Thanks for merging.


Fix out of memory

Process in VM runs out of memory on ray.yorksj.ac.uk.

Tested with ray.yorksj.ac.uk. Found that the process is killed always at the same link. Monitoring memory during the first half our of execution showed that normal memory consumption is at about 15%, so not too high. It also doesn't change much during execution - no spikes or longer term increases. Will isolate the script on the PDF processed when error occurs.


spelling errors

@DuncanMcLaggan, would be great to have all the spelling and copy changes that you and others have notices?
thank you!

New text updated in Google docs - sofia updating


Disables navbar links while on registration page.

This (and other) PRs would benefit from terse explanations in the PR description.
If I understand your intent, you're trying to avoid people clicking out of the registration page while still filling the form out. There's a more conventional mechanism for this:
https://stackoverflow.com/questions/7317273/warn-user-before-leaving-web-page-with-unsaved-changes

It goes farther than not letting them click out of the page while filling out the form. We want it to be so that the user is redirected to the registration page when it is their first time logging in (so we don't have a User object for their email yet) and then make it so that they are not allowed to go to any other page on the site until they have registered. If they type in another page's url, they should be reidrected back to the registration page.


[testsuite] smoke_test::test_basic_fault_tolerance is flaky and can run indefinitely

Today we identified a number of builds both on PRs as well as master that were running for longer than 20 minutes (up to multiple hours on some builds). From my initial investigation it seems like the cause of this is the e2e test smoke_test::test_basic_fault_tolerance as you can see on this canceled ci run which never terminates.

Closed by #1092


chore/add @nix1g as reviewer

Realitio cross-chain proxy Review @ 43c204b
Minor issue

A change of proxy can break some ongoing arbitrations (e.g. the Foreign Proxy never receives Realitio's acknowledgement, the Home Proxy never receives the arbitration answer...).
Thus their addresses and chain ID should be immutable as is the case for realitio, arbitrator and amb.

Proxy Interfaces | Documentation | Typo

L8: "deems to be incorrect"
L26: "arbitrat**i**or"

Home Proxy
Code improvements | Gas

L206 At this point, only the request.status is set, simply resetting the status to Status.None is enough.

Documentation
Since the flow was updated, some more comments need to be updated as well

L49-L51: "and becomes the same answer as the one from requester."

Foreign Proxy
Minor issue
NUMBER_OF_CHOICES_FOR_ARBITRATOR is set to 2 ** 256 - 2 choices, plus the "refuse to vote" option. This is one option less than what Realitio supports. I guess the reason is that KlerosLiquid's getVoteCounter() would overflow with 2 ** 256 - 1 choices. However, this function cannot be executed in this case anyway since it would run out of gas (and Solidity enforces a maximum size of 2 ** 64 - 1 on arrays).
This constant can thus be set to uint256(-1).
Code improvement

L226-L228: It is OK to send 0 ETH (cf. Kleros' CONTRIBUTING.md).

Documentation

L215: "greater than or equal to the arbitration cost"

A change of proxy can break some ongoing arbitrations (e.g. the Foreign Proxy never receives Realitio's acknowledgement, the Home Proxy never receives the arbitration answer...).
Thus their addresses and chain ID should be immutable as is the case for realitio, arbitrator and amb.

They can't be immutable because those variables can only be set in the constructor, as stated in solidity docs.
The problem here is that we have a circular reference between the Home Proxy and the Foreign Proxy, so we can't now the address of both by the time we are creating the contract.
The issue is valid though, which means we should add a mechanism to not allow changing them more than once.

L206 At this point, only the request.status is set, simply resetting the status to Status.None is enough.

Good catch!

Fixed all typos/bad wording.

Minor issue
NUMBER_OF_CHOICES_FOR_ARBITRATOR is set to 2 ** 256 - 2 choices, plus the "refuse to vote" option. This is one option less than what Realitio supports. I guess the reason is that KlerosLiquid's getVoteCounter() would overflow with 2 ** 256 - 1 choices. However, this function cannot be executed in this case anyway since it would run out of gas (and Solidity enforces a maximum size of 2 ** 64 - 1 on arrays).
This constant can thus be set to uint256(-1).

I got this constant from the current version of the Realitio Proxy contract. To be honest, what you're saying makes sense, but I don't have enough context to know why this was chosen to be this value because I didn't take a part in the original contract writing.

L226-L228: It is OK to send 0 ETH (cf. Kleros' CONTRIBUTING.md).

Originally I coded it this way, but this was a suggestion from the external auditor review to save some gas. Almost all the time the remainder is going to be 0. I wrote a small PoC to understand what is the gas usage when there is a condition and where there is not:
contract Sender {
    function sendVal() public payable {
        msg.sender.send(msg.value);
    }
}

contract ConditionalSender {
    function sendVal() public payable {
        if (msg.value > 0) {
            msg.sender.send(msg.value);
        }
    }
}

Below are the values for both calls when msg.value is 0:
Sender:

transaction cost:	21389 gas
execution cost:	117 gas

ConditionalSender:

transaction cost:	22131 gas
execution cost:	859 gas

So yeah, the "optimized" version is more expensive than the original one.


first errors found

I did test, I did see the following errors:
&& became & &
|| became | |
!= became ! =
%hour became % hour

After fixings this (in text editor) there is still an error but not completely clear, the logging says:
ERROR: Expected a parenthesis block, function, number or variable

Commit 7f72707 should have fixed the errors for &&, ||, !=, and %hour. I'm not sure yet about the expected parenthesis block, could you send me the ruleset that generates that error so I can test it?

maybe stupid question, but what is the easiest way to update?

I've updated manually the .py file, all above errors are fixed; great! I do get the same error. Attached my current ruleset.
Heishamon_rules20240210.txt

I just released an update with some more fixes. Your input seems to be working on v3.2.3 now. On the alpha release of the new rules library the HeishaMon resets most of the time when trying to upload the full ruleset, but sometimes it does upload correctly. I'm not sure yet why that is. But I don't think that has anything to do with the minifier, as the same happens when I use the unminified ruleset as you provided.

Great, I’ll test later as I’m not at home next days. I don’t expect the ruleset to work with @curlymoo latest version, the size it too big.

I've been able to test via VPN with my current ruleset; it works, great!!


⚠ Bofrost.at has degraded performance

In db46c80, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 2596 ms

Resolved: Bofrost.at performance has improved in 4768500.


Integration help

I am trying to install the Popup picker from Picmojs but I am limited by the following:

I can't modify any file as I am integrating the code in a co-code builder (no access to index.js,..) but can host files
I can't use NPM nor Yarn as I am working in the browser

Here's what the documentation says

Use ESM from CDN You can also import the ESM version of PicMo
directly. You will first to create an ES module that imports PicMo:
index.js import { createPicker } from<EMAIL_ADDRESS>createPicker(...);
Then you can import the local module from a script tag:

I indeed tried in a Fiddle (even without the script that as I didn't understand this part) and it seems to be working all fine: Link to Fiddle
However, this relates to the createPicker function, but I'm interested in the Popup picker and therefore need the createPopup function.
According to their documentation:

A popup picker is not displayed until it is triggered by clicking on a
popup trigger, usually a button.
To use a popup picker, you must first install the @picmo/popup-picker
package. This package contains the createPopup function.
createPopup(pickerOptions: PickerOptions, popupOptions: PopupOptions):PopupPickerController

-> I don't know how to "first install the @picmo/popup-picker package".
Here's what I tried based on the previous working example:
import { createPopup } from 'https://unpkg.com/<EMAIL_ADDRESS>
Link to Fiddle
But I always get the same error: "43:10 Uncaught SyntaxError: The requested module 'https://unpkg.com/<EMAIL_ADDRESS>does not provide an export named 'createPopup'"
Any hint for me? I'm really stuck on this part.

You're using the UMD version of the module with an import statement. This won't work; only ES modules work with imports.  UMD modules are loaded with a script tag.
But then you are mixing an ESM import (your import of the main picmo package) with a UMD, which won't work either.
Two options:
(1) Use the UMD version of the base picmo package as well, both loaded with script tags.
(2) Use the ESM distribution of @picmo/popup-picker from unpkg. Note that you will need to add ?module to the URL, otherwise your browser will likely give an error about an invalid relative path. If you do it this way you don't need the picmo import since the popup module imports it.
So for the ESM route, you need just a single import:
import { createPopup } from 'https://unpkg.com/@picmo/popup-picker@latest/dist/index.js?module';

Hope this helps. In the future please use the discussions for asking general questions like this, thanks!

Wahoo, that's a really complete answer, I can't thank you enough for that!
I managed to make it work using your solution, but it only works in Fiddle for some reason and throws the SyntaxError: Cannot use import statement outside a module error when used on a regular web page:

Note: I am aware this is a very frequently asked question but I am using pure Javascript and therefore I can't manage to fix it using the Node.js solutions on StackOverflow

I am trying to install the Popup picker from Picmojs but it was really tricky as it's not a usual library to import as a script tag.
For some reason, my code is working in Fiddle, but as soon as implemented on my web page, it throws the SyntaxError: Cannot use import statement outside a module error.
Here's my working fiddle code:

import { createPopup } from 'https://unpkg.com/@picmo/popup-picker@latest/dist/index.js?module';

const triggerButton = document.getElementById('button');

// Create the picker
const picker = createPopup({

	className: "",
  hideOnClickOutside: true,
  hideOnEmojiSelect: true,
  hideOnEscape: true,
  showCloseButton: true,
  position: 'bottom-start', //https://floating-ui.com/docs/computeposition#placement
  //position: 'auto';

}, {
  // The element that triggers the popup
  triggerElement: triggerButton,

  // The element to position the picker relative to - often this is also the trigger element,
  referenceElement: triggerButton,

});

// The picker emits an event when an emoji is selected. Do with it as you will!
picker.addEventListener('emoji:select', event => {
  console.log('Emoji selected:', event.emoji);
});

triggerButton.addEventListener('click', event => {
  picker.toggle()
});

<input id="button" class="pickerContainer" type="button" value="Input Button">

However, when used on an HTML element of Bubble (a no-code editor), I get errors.

I also get the same result when the code is used in a script tag on Fiddle (Example)

//import { createPopup } from 'https://unpkg.com/@picmo/popup-picker@latest/dist/index.js?module';

const triggerButton = document.getElementById('button');

// Create the picker
const picker = createPopup({

	className: "",
  hideOnClickOutside: true,
  hideOnEmojiSelect: true,
  hideOnEscape: true,
  showCloseButton: true,
  position: 'bottom-start', //https://floating-ui.com/docs/computeposition#placement
  //position: 'auto';

}, {
  // The element that triggers the popup
  triggerElement: triggerButton,

  // The element to position the picker relative to - often this is also the trigger element,
  referenceElement: triggerButton,

});

// The picker emits an event when an emoji is selected. Do with it as you will!
picker.addEventListener('emoji:select', event => {
  console.log('Emoji selected:', event.emoji);
});

triggerButton.addEventListener('click', event => {
  picker.toggle()
});

<input id="button" class="pickerContainer" type="button" value="Input Button">

Any hint on why this is not working? Thanks!


Message Threads

Hey! I have a bug... When I send a Message to ONE User. all Users can see and read this thread, why? Can anyone help me?

Hey! You must get the user Thread like this:
Getting all user Threads:
$thread = Thread::forUser(Auth::user()->id)->latest('updated_at')->get();
Then, check participation:
$participant= Participant::where('thread_id',$thread->id)->where('user_id',Auth::user()->id)->first();

I add in the MessagesController.php show_message right? Must I edit the routes MessagesController@show in MessagesController@show_message?

Oh, well, sorry. I was using my own methods.
Use show() or show_message(), doesn't matter.

I've did it with the explicit Model binding and just check the participation of the current user:
routes.php
Route::group(['prefix' => 'chat'], function () {
        Route::get('/', 'ChatController@getIndex');
        Route::get('/create', 'ChatController@getCreate');
        Route::post('/create', 'ChatController@postCreate');
        Route::get('/{chat_thread}', 'ChatController@getShow');
        Route::post('/{chat_thread}', 'ChatController@postReply');
});

RouteServiceProvider
$router->model('chat_thread', ChatThread::class);

MessageController
public function getShow(Thread $thread)
{
    if(!$thread->hasParticipant(\Auth::id())) {
        abort(404);
    }

    $thread->markAsRead(\Auth::id());

    return view('app.chat.index')->with([
        'display_thread' => $thread,
    ]);
}

@JoanVt: Okay. Now I have this:
[code]
public function show($id)
{
try {
$thread = Thread::findOrFail($id);
} catch (ModelNotFoundException $e) {
Session::flash('error_message', 'The thread with ID: ' . $id . ' was not found.');
        return redirect('user/messages');
    }

    // show current user in list if not a current participant
    // $users = User::whereNotIn('id', $thread->participantsUserIds())->get();

    // don't show the current user in list
    $userId = Auth::user()->id;
    $users = User::whereNotIn('id', $thread->participantsUserIds($userId))->get();

    $thread->markAsRead($userId);

    return view('messenger.show', compact('thread', 'users'));
}

[/code]
Where I can add
$participant= Participant::where('thread_id',$thread->id)->where('user_id',Auth::user()->id)->first();
Sorry but just I´m very confused..... XD

Oh no!!!! I wanted not close it :(

I'm also ... what is your problem and what are you trying to get working?
Do you want to list all Threads the current User is participating? Or do you want to display a single Thread, but only if the current user is a participant of this Thread, or what is your goal?

When I go to yourwebsite.com/messages/ID than I can see the Messages, but I'm not participant from this thread....

Ok, than thsi will solve your problem:
if(!$thread->hasParticipant(\Auth::id())) {
    abort(404);
}

just add it in your show Method of your controller. You can do everything you want inside the if what you want to do if anyone want to see a message thread where he isn't a participant.

My routes:
Route::get('mis-mensajes', ['as' => 'messages', 'uses' => 'ProfileController@my_messages']);
			Route::get('nuevo-mensaje', ['as' => 'messages.create', 'uses' => 'ProfileController@new_message']);
			Route::post('enviar-message', ['as' => 'messages.store', 'uses' => 'ProfileController@send_message']);
			Route::get('mensaje/{id}', ['as' => 'messages.show', 'uses' => 'ProfileController@show_message']);
			Route::put('mensaje/{id}', ['as' => 'messages.update', 'uses' => 'ProfileController@update_message']);

My Controller:

public function my_messages(){
        // All threads, ignore deleted/archived participants
		$data_header['title'] = 'Tus mensajes '.Auth::user()->user_config->name1;
		$data_header['user'] = Auth::user();
        $data_header['threads'] = Thread::forUser(Auth::user()->id)->latest('updated_at')->get();
        // All threads that user is participating in
        // $threads = Thread::forUser($currentUserId)->latest('updated_at')->get();
        // All threads that user is participating in, with new messages
        // $threads = Thread::forUserWithNewMessages($currentUserId)->latest('updated_at')->get();
        return view('statics/user_my_messages',$data_header);
	}

	public function new_message(){
		$data_header['title'] = 'Este es tu perfil '.Auth::user()->user_config->name1;
		$data_header['user'] = Auth::user();
		$data_header['users'] = User::where('id', '!=', Auth::id())->get();
		return view('statics/user_create_message',$data_header);
	}

public function show_message($id){
		$data_header = [];
		try {
            $thread = Thread::findOrFail($id);
        } catch (ModelNotFoundException $e) {
            Session::flash('error_message', 'No existe esta conversación.');
            return redirect('perfil/mis-mensajes');
        }

		$yo = Participant::where('thread_id',$thread->id)->where('user_id',Auth::user()->id)->first();

		if($yo){
			// show current user in list if not a current participant
			// $users = User::whereNotIn('id', $thread->participantsUserIds())->get();
			// don't show the current user in list
			$userId = Auth::user()->id;

			$participantes = $thread->participantsUserIds();
			if(count($participantes) != 2){
				redirect('/');
			}
			foreach($participantes as $v){
				if($v != Auth::user()->id){
					$to_find = $v;
				}
			}

			$user = User::find($to_find);

			if (isset($user->user_config->nickname)){ $name_user = $user->user_config->nickname; }else { $name_user = $user->user_config->name1; }

			$data_header['title'] = 'Mensajes intercambiados con '.$name_user;

			$thread->markAsRead($userId);

			Session::flash('to_send',$id);

			return view('statics/user_show_messages', compact('thread', 'user'))->with($data_header);
		}else{
			return redirect('perfil/mis-mensajes/');
		}
	}

public function update_message($id)
    {
        try {
            $thread = Thread::findOrFail($id);
        } catch (ModelNotFoundException $e) {
            Session::flash('error_message', 'The thread with ID: ' . $id . ' was not found.');
            return redirect('messages');
        }
        $thread->activateAllParticipants();

        // Message
        Message::create(
            [
                'thread_id' => $thread->id,
                'user_id'   => Auth::id(),
                'body'      => Input::get('message'),
            ]
        );
        // Add replier as a participant
        $participant = Participant::firstOrCreate(
            [
                'thread_id' => $thread->id,
                'user_id'   => Auth::user()->id
            ]
        );
        $participant->last_read = new Carbon;
        $participant->save();
        // Recipients
        if (Input::has('recipients')) {
            $thread->addParticipants(Input::get('recipients'));
        }
        return redirect('perfil/mensaje/' . $id);
    }

You can use \Auth::id() instead of \Auth::User()->id just to say. ;)

@Gummibeer Your answer is ok! Don't know why i'm not using this method. Updating and posting in secs...

@Gummibeer THANKS!!!!!!!!!! <3 <3 <3...
But now a last question. In will the messages in thread sort by ID DESC. Your answer in the another issues not helpful for me...
[php]$messages = $thread->messages()->orderBy('id', 'desc')->get()[/php]
In my Controller show() or who?

where you want to get the sorted collection of the messages. In the Controller, View or where ever you need it.
You get a collection of, by ID, ordered messages for the given thread.

Ohh now I have understand it! Thanks for you help.


Ubuntu version number is hardwired into Nexus configuration

Describe the bug
in templates/workspace_services/guacamole/user_resources/guacamole-azure-linuxvm/terraform/apt_sources_config.yml, the version number 18.04 is hardwired. This needs to be dependent on the actual version the VM boots with.

Configure it to be the latest stable version for now

Version hard coded to 22.04 atm


chore: pkg imported more than once

Reason for Change:

pkg imported more than once
Issue Fixed:

Requirements:

[ ] uses conventional commit messages

[ ] includes documentation
[ ] adds unit tests
[ ] relevant PR labels added

Notes:

@huntergregory can you triage this?


Error: Direct database rename not implemented MySQL

I tried to rename the database and received the error:

Direct database rename is not yet implemented in MySQL. You should use export/import functions for that.

I tried MySQL and MariaDB drivers, both has the same error. There is no way to rename the database from import/export. Please help.

Database (or schema) rename is not supported by MySQL/MariaDB. Check this: https://stackoverflow.com/questions/12190000/rename-mysql-database
In DBeaver you can:

make database dump (right click on database, Tools->Dump database)
create new database
restore it from the dump (Tools->Restore database)

@serge-rider can it be automated? I used tool HeidiSQL, it provides database renaming and I never knew mysql does not support renaming because HeidiSQL make it transparent for user. It creates database, transfer all tables and drops the old database.

Just a suggestion: https://stackoverflow.com/a/13944924


TFLite InterpreterBuilder Segmentation fault 11 on MacOS 11.6

I use C++ to load tflite model. Same code works fine on Linux & Android. But when running on MacOS it gives Segmentation fault 11  error.
I am using tensorflow 2.8.0.
#include <tensorflow/lite/kernels/register.h>

std::unique_ptr<tflite::FlatBufferModel> model;
std::unique_ptr<tflite::Interpreter> interpreter;

int main() {
  model = tflite::FlatBufferModel::BuildFromFile"model.tflite");
  tflite::ops::builtin::BuiltinOpResolver resolver;
  tflite::InterpreterBuilder(*model, resolver)(&interpreter); // -> Segmentation fault 11 here
  return 0;
}

Hi @HedgeHao ! Can you check this code and let us know the output?
// Create model from file. Note that the model instance must outlive the
// interpreter instance.
auto model = [tflite::FlatBufferModel::BuildFromFile](...);
if (model == nullptr) {
  // Return error.
}
// Create an [Interpreter]
std::unique_ptr interpreter;
tflite::ops::builtin::BuiltinOpResolver resolver;
if (InterpreterBuilder(*model, resolver)(&interpreter) != kTfLiteOk) {
  // Return failure.
}
if (interpreter->AllocateTensors() != kTfLiteOk) {
  // Return failure.
}

@mohantym Not working, either. Still Segmentation fault: 11 in InterpreterBuilder.

Hi @HedgeHao ! Could you check this code snippet and let us know the output? Thanks!
#include <tensorflow/lite/kernels/register.h>
using namespace tflite;
std::unique_ptr<tflite::FlatBufferModel> model;
std::unique_ptr<tflite::Interpreter> interpreter;

int main() {
  auto model = tflite::FlatBufferModel::BuildFromFile("model.tflite"); // saw a missing open bracket in original code
  if (model == nullptr) {
  // Return error if model is empty
}
tflite::ops::builtin::BuiltinOpResolver resolver;
if (InterpreterBuilder(*model, resolver)(&interpreter) != kTfLiteOk) {
  // Return failure.
}
if (interpreter->AllocateTensors() != kTfLiteOk) {
  // Return failure.
}
  return 0;
}

Anyone?

Hi, Did you try it in the nightly version and did you find the same behavior.
Could you also let us know your Mac's processor. Thanks!

I built tensorflowlite shared library following this [tutorial] (https://www.tensorflow.org/lite/guide/build_cmake) with version 2.8.0. I use Mac Air 2014 , Big Sur 11.6, Intel Core i5.

Problem is not resolve

@terryheo I use debug build but the debugger stuck at InterpreterBuilder(). I can't step into tflite function to debug.
Can anyone provide the sample code and prebuilt dylib that can work. I've been stuck on this very beginning sample for a month..

Hi @HedgeHao,
Can you provide the "model.tflite" file so that we can attempt to replicate this?
Thanks!


Kong 2.4 Auth Bugs OAuth2.0 , JWT.

Summary
Seeing some unexplainable behavior out of my Kong Dev nodes post upgrade to 2.4, Sometime attempting to generate an OAuth2.0 token fails with an error in a given datacenter, while the other DC succeeds to generate a token(and each DC may be able to flip flop after redeployments.. and sometimes I think I have seen moments where both DC deployments can work at the same time too, its very odd and not fully predictable). I also may have seen a similar issue with JWT Auth plugin telling me the error (No credentials found for given 'iss') found for given JWT token the gateway received on a proxy request but figured we take this one at a time and see if we can see what may cause the OAuth2.0 behavior, might fix the common core component that also made JWT auth seemingly fail... Nothing in the SRC code of OAuth2.0 in the version bump changes make sense for how they could be the culprit for this behavior, + I remember testing the changes in the OAuth2.0 plugin on earlier versions of Kong to improve the edge cases and OAuth2.0 worked fine then so I think its something indirectly related that has done this...
Example:
Fail case:
POST https://gateway-dev-network-zone-dc2.company.com/auth/oauth2/token

{
 "client_id": "6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV",
 "client_secret": "C1nnQ0wmwgT2PkUunUhs8NfkMQLNM7CA",
 "grant_type": "client_credentials"
}

Response:
Status: 400

{
    "error": "invalid_client",
    "error_description": "Invalid client authentication"
}

And Success Case:
POST https://gateway-dev-network-zone-dc1.company.com/auth/oauth2/token

{
 "client_id": "6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV",
 "client_secret": "C1nnQ0wmwgT2PkUunUhs8NfkMQLNM7CA",
 "grant_type": "client_credentials"
}

Response:
Status: 200

{
    "expires_in": 3600,
    "access_token": "PcH8nuM7MSI1SHAHVv4QrK8psIq75c8X",
    "token_type": "bearer"
}

OAuth2.0 Plugin configured on the Gateway endpoint Kong 2.4:
{
  "name": "oauth2",
  "route": {
    "id": "ab469dbf-3806-4b22-8a32-246d7b158360"
  },
  "created_at":<PHONE_NUMBER>,
  "tags": null,
  "id": "50a3fcfa-a660-425b-b181-698e8d1a196d",
  "config": {
    "enable_client_credentials": true,
    "enable_password_grant": false,
    "enable_authorization_code": false,
    "auth_header_name": "authorization",
    "reuse_refresh_token": false,
    "enable_implicit_grant": false,
    "token_expiration": 3600,
    "accept_http_if_already_terminated": false,
    "hide_credentials": false,
    "anonymous": null,
    "scopes": null,
    "provision_key": "function",
    "refresh_token_ttl": 3600,
    "global_credentials": true,
    "pkce": "lax",
    "mandatory_scope": false
  },
  "service": null,
  "consumer": null,
  "enabled": true,
  "protocols": [
    "http",
    "https"
  ]
}

Data of our user above from the DB for further evidence from C*, first their credential pair as seen above:
kongdba@cqlsh:kong_dev> select * from oauth2_credentials WHERE client_secret='C1nnQ0wmwgT2PkUunUhs8NfkMQLNM7CA';

 id                                   | client_id                                                             | client_secret                    | client_type  | consumer_id                          | created_at                      | hash_secret | name   | redirect_uris         | tags | ws_id
--------------------------------------+-----------------------------------------------------------------------+----------------------------------+--------------+--------------------------------------+---------------------------------+-------------+--------+-----------------------+------+--------------------------------------
 16093f3c-c51d-4523-ab10-1fa91464d443 | 7e175ef7-c50d-49d1-a648-0091156e2b4c:6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV | C1nnQ0wmwgT2PkUunUhs8NfkMQLNM7CA | confidential | 092798fd-5055-4b14-a699-4f915d874695 | 2019-02-22 18:59:48.000000+0000 |       False | oauth2 | {'https://optum.com'} | null | 7e175ef7-c50d-49d1-a648-0091156e2b4c

(1 rows)

And their consumer table data:
kongdba@cqlsh:kong_dev> select * from consumers WHERE id=092798fd-5055-4b14-a699-4f915d874695;

 id                                   | created_at                      | custom_id | tags | username                                                | ws_id
--------------------------------------+---------------------------------+-----------+------+---------------------------------------------------------+--------------------------------------
 092798fd-5055-4b14-a699-4f915d874695 | 2019-02-22 18:59:48.000000+0000 |      null | null | 7e175ef7-c50d-49d1-a648-0091156e2b4c:stargate.testsuite | 7e175ef7-c50d-49d1-a648-0091156e2b4c

(1 rows)

Workspace table just for reference, thought that could be the culprit but nah just 1 entry that is in sync w the above tables:
kongdba@cqlsh:kong_dev> select * from workspaces;

 id                                   | comment | config | created_at                      | meta | name
--------------------------------------+---------+--------+---------------------------------+------+---------
 7e175ef7-c50d-49d1-a648-0091156e2b4c |    null |   null | 2020-07-21 20:21:51.000000+0000 | null | default

Note just for sanity I even was querying the local 3 node cluster in the DC thats currently throwing the oauth2 token gen error just to prove the actual raw data seems to be there and in place.. Why Kong seemingly does not resolve the db data consistently to recognize our user here and their token gen request creds I am not sure.
I am assuming since this is the client credential grant flow we are failing right here(but I will try to confirm where in code its doing it later today):
https://github.com/Kong/kong/blob/master/kong/plugins/oauth2/access.lua#L680
Worth noting Kong logs do no good here, no DB connectivity or error reports. I will likely start hacking some debug log print lines to figure out which of the error blocks we fell into.
Our functional test libs that hit the gateway regularly consistently flakey fails due to poor token generation capabilities in dev now as well calling the 2.4 gateways while still succeeding every time in stage/prod against our Kong 2.1.4 nodes.
Some ideas are some kinda poisoned cache or lookup issue. Bug is very persistent once the issue presents itself, basically the working datacenter will continue to work proper it seems and the dysfunctional datacenter calls will continue to throw that 400 error like that users creds didn't exist.
Definitely gonna be a blocker for deploying Kong 2.4 into customer facing environments for now until we get to the bottom of this. Worth noting I was testing Kong 2.3.3 in dev at one point(after upgrading from 2.1.4) and this kinda behavior was not presenting itself. So likely if there is some component of Kong not directly related to the plugin functional code itself that changed its likely somewhere between 2.3.3 to 2.4 caused it. I believe there were not even any db migration schema changes from 2.3 to 2.4 looking at core schema_meta so likely not a db schema/data related issue either hmm.
Lemme know if you have any debug code snippets you want me to drop into the plugins or whatever, would like to get this fixed.
Steps To Reproduce

Stand up multi DC Kong(2) w 6 node C* cluster, 3 per DC
Test OAuth2.0 endpoint above with your produced creds by a consumer.
See if you see flakey behavior out of the global token gen endpoint too?

Additional Details & Logs

Kong version 2.4

ENV:
        - env:
            - name: KONG_ADMIN_SSL_ENABLED
              value: 'off'
            - name: KONG_CASSANDRA_DATA_CENTERS
              value: 'DC1:3,DC2:3'
            - name: KONG_CASSANDRA_REPL_STRATEGY
              value: NetworkTopologyStrategy
            - name: KONG_CASSANDRA_SCHEMA_CONSENSUS_TIMEOUT
              value: '30000'
            - name: KONG_UPSTREAM_KEEPALIVE_IDLE_TIMEOUT
              value: '30'
            - name: KONG_UPSTREAM_KEEPALIVE_MAX_REQUESTS
              value: '50000'
            - name: KONG_UPSTREAM_KEEPALIVE_POOL_SIZE
              value: '400'
            - name: KONG_WORKER_CONSISTENCY
              value: eventual
            - name: KONG_WORKER_STATE_UPDATE_FREQUENCY
              value: '5'
            - name: KONG_NGINX_PROXY_REAL_IP_HEADER
              value: X-Forwarded-For
            - name: KONG_NGINX_PROXY_REAL_IP_RECURSIVE
              value: 'on'
            - name: KONG_SSL_CIPHER_SUITE
              value: intermediate
            - name: KONG_NGINX_MAIN_WORKER_PROCESSES
              value: '6'
            - name: KONG_BOUNCE_REASON
              value: 'Automated Bounce 20200511-06:58:13.262'
            - name: KONG_HEADERS
              value: latency_tokens
            - name: KONG_DNS_ORDER
              value: 'LAST,SRV,A,CNAME'
            - name: KONG_CASSANDRA_CONTACT_POINTS
              value: 'server8171,server8176,server8180,server8172,server8173,server8175'
            - name: KONG_LOG_LEVEL
              value: notice
            - name: KONG_PROXY_ACCESS_LOG
              value: 'off'
            - name: KONG_ADMIN_ACCESS_LOG
              value: 'off'
            - name: KONG_PROXY_ERROR_LOG
              value: /dev/stderr
            - name: KONG_ADMIN_ERROR_LOG
              value: /dev/stderr
            - name: KONG_ANONYMOUS_REPORTS
              value: 'off'
            - name: KONG_PROXY_LISTEN
              value: '<IP_ADDRESS>:8000, <IP_ADDRESS>:8443 ssl http2 deferred reuseport'
            - name: KONG_ADMIN_LISTEN
              value: '<IP_ADDRESS>:8001 deferred reuseport'
            - name: KONG_MEM_CACHE_SIZE
              value: 1024m
            - name: KONG_SSL_CERT
              value: /usr/local/kong/ssl/kongcert.crt
            - name: KONG_SSL_CERT_KEY
              value: /usr/local/kong/ssl/kongprivatekey.key
            - name: KONG_SSL_CERT_DER
              value: /usr/local/kong/ssl/kongcertder.der
            - name: KONG_CLIENT_SSL
              value: 'off'
            - name: KONG_TRUSTED_IPS
              value: '<IP_ADDRESS>/0,::/0'
            - name: KONG_CLIENT_MAX_BODY_SIZE
              value: 50m
            - name: KONG_CLIENT_BODY_BUFFER_SIZE
              value: 50m
            - name: KONG_ERROR_DEFAULT_TYPE
              value: text/plain
            - name: KONG_DATABASE
              value: cassandra
            - name: KONG_PG_SSL
              value: 'off'
            - name: KONG_CASSANDRA_PORT
              value: '9042'
            - name: KONG_CASSANDRA_KEYSPACE
              value: kong_dev
            - name: LAST_AUTOMATED_RESTART
              value: '20210426_0533'
            - name: KONG_CASSANDRA_TIMEOUT
              value: '8000'
            - name: KONG_PLUGINS
              value: >-
                kong-siteminder-auth,kong-kafka-log,stargate-waf-error-log,mtls,stargate-oidc-token-revoke,kong-tx-debugger,kong-plugin-oauth,zipkin,kong-error-log,kong-oidc-implicit-token,kong-response-size-limiting,request-transformer,kong-service-virtualization,kong-cluster-drain,kong-upstream-jwt,kong-splunk-log,kong-spec-expose,kong-path-based-routing,kong-oidc-multi-idp,correlation-id,oauth2,statsd,jwt,rate-limiting,acl,request-size-limiting,request-termination,cors
            - name: KONG_CASSANDRA_SSL
              value: 'on'
            - name: KONG_CASSANDRA_SSL_VERIFY
              value: 'on'
            - name: KONG_CASSANDRA_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: cassandra-secret
            - name: KONG_CASSANDRA_USERNAME
              valueFrom:
                secretKeyRef:
                  key: username
                  name: cassandra-secret
            - name: KONG_CASSANDRA_CONSISTENCY
              value: LOCAL_QUORUM
            - name: KONG_CASSANDRA_LB_POLICY
              value: RequestDCAwareRoundRobin
            - name: KONG_CASSANDRA_LOCAL_DATACENTER
              value: CTC
            - name: KONG_DB_UPDATE_FREQUENCY
              value: '5'
            - name: KONG_DB_UPDATE_PROPAGATION
              value: '5'
            - name: KONG_DB_CACHE_TTL
              value: '0'
            - name: KONG_DNS_HOSTSFILE
              value: /etc/hosts
            - name: KONG_DNS_STALE_TTL
              value: '4'
            - name: KONG_DNS_NOT_FOUND_TTL
              value: '30'
            - name: KONG_DNS_ERROR_TTL
              value: '1'
            - name: KONG_DNS_NO_SYNC
              value: 'off'
            - name: KONG_LUA_SSL_TRUSTED_CERTIFICATE
              value: /usr/local/kong/ssl/kongcert.pem
            - name: KONG_LUA_SSL_VERIFY_DEPTH
              value: '3'
            - name: KONG_LUA_SOCKET_POOL_SIZE
              value: '30'
            - name: SPLUNK_HOST
              value: gateway-dev-dmz-dc1.company.com
            - name: KONG_DB_CACHE_WARMUP_ENTITIES
              value: 'services,plugins,consumers'
            - name: LUA_PATH
              value: >-
                /usr/local/kong/luarocks/share/lua/5.1/?.lua;;/usr/local/kong/luarocks/share/lua/5.1/?/init.lua;
            - name: KONG_NGINX_HTTP_SSL_PROTOCOLS
              value: TLSv1.2 TLSv1.3
            - name: KONG_CASSANDRA_REFRESH_FREQUENCY
              value: '0'

Can you check with this:
https://github.com/Kong/kong/pull/7102

Will try that fix when I wake up! About to catch some zzz, typing from cellphone in bed. Thanks for prompt response, sounds like you may have potentisllybfound the shared component causing the issue.

For a second I thought it was stable because I could not get it to error again, but after a few deploys even with that connection:closed plugin I was able to reproduce the issues I am seeing by the 5th or 6th redeploy... Hmm.

Output trying to curl the endpoint localhost in the pod that was throwing the Kong auth error in a given DC... weird. I had the connection close header response transformer plugin on:
/ $ curl -k -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials&client_id=6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV&client_secret=C1nnQ0wmwgT2PkUunUhs8NfkMQL
NM7CA" https://<IP_ADDRESS>:8443/auth/oauth2/token -vv
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying <IP_ADDRESS>:8443...
* TCP_NODELAY set
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: REDACTED
*  start date: Mar 19 17:05:40 2021 GMT
*  expire date: Mar 19 17:05:40 2022 GMT
*  issuer: REDACTED
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f1a4ec86e0)
> POST /auth/oauth2/token HTTP/2
> Host: <IP_ADDRESS>:8443
> user-agent: curl/7.67.0
> accept: */*
> content-type: application/x-www-form-urlencoded
> content-length: 119
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* We are completely uploaded and fine
* http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [connection], value: [close]
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host <IP_ADDRESS> left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

Okay without the Response transformer connection closed plugin we are getting the expected error over here:
curl -k -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials&client_id=6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV&client_secret=C1nnQ0wmwgT2PkUunUhs8NfkMQL
NM7CA" https://<IP_ADDRESS>:8443/auth/oauth2/token -vv
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying <IP_ADDRESS>:8443...
* TCP_NODELAY set
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: REDACTED
*  start date: Mar 19 17:05:40 2021 GMT
*  expire date: Mar 19 17:05:40 2022 GMT
*  issuer: REDACTED
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x555b3e1a86e0)
> POST /auth/oauth2/token HTTP/2
> Host: <IP_ADDRESS>:8443
> user-agent: curl/7.67.0
> accept: */*
> content-type: application/x-www-form-urlencoded
> content-length: 119
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 400
< date: Fri, 30 Apr 2021 22:37:55 GMT
< content-type: application/json; charset=utf-8
< cache-control: no-store
< pragma: no-cache
< content-length: 78
< optum-cid-ext: f42cd45b-c2ec-468f-a68b-d5248956a9f1#3
< x-response-latency: 2
<
* Connection #0 to host <IP_ADDRESS> left intact
{"error":"invalid_client","error_description":"Invalid client authentication"}/ $

My configs for listener ENV var has always been this:
KONG_PROXY_LISTEN <IP_ADDRESS>:8000, <IP_ADDRESS>:8443 ssl http2 deferred reuseport

So I have http2 support enabled but thats never done anything before for me or caused problems...
Try working datacenter side for same call too to see success in the pod:
/ $ curl -k -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials&client_id=6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV&client_secret=C1nnQ0wmwgT2PkUunUhs8NfkMQL
NM7CA" https://<IP_ADDRESS>:8443/auth/oauth2/token -vv
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying <IP_ADDRESS>:8443...
* TCP_NODELAY set
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: REDACTED
*  start date: Mar 19 17:07:50 2021 GMT
*  expire date: Mar 19 17:07:50 2022 GMT
*  issuer: REDACTED
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55aa31e486e0)
> POST /auth/oauth2/token HTTP/2
> Host: <IP_ADDRESS>:8443
> user-agent: curl/7.67.0
> accept: */*
> content-type: application/x-www-form-urlencoded
> content-length: 119
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Fri, 30 Apr 2021 22:48:32 GMT
< content-type: application/json; charset=utf-8
< cache-control: no-store
< pragma: no-cache
< content-length: 91
< optum-cid-ext: 1ce74e10-e589-4e25-ad9d-cf4ab2aa0e3e#911
< x-response-latency: 26
<
* Connection #0 to host <IP_ADDRESS> left intact
{"access_token":"dZkOOythef6xvjrDsvwwmGPtDLskTrpi","token_type":"bearer","expires_in":3600}/ $

Lets try in Stage Kong: 2.1.3, same curl call:
$ curl -k -X POST -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=client_credentials&client_id=6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV&client_secret=C1nnQ0wmwgT2PkUunUhs8NfkMQL
NM7CA" https://<IP_ADDRESS>:8443/auth/oauth2/token -vv
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying <IP_ADDRESS>:8443...
* TCP_NODELAY set
* Connected to <IP_ADDRESS> (<IP_ADDRESS>) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: REDACTED
*  start date: Aug 11 23:02:46 2020 GMT
*  expire date: Aug 11 23:02:46 2021 GMT
*  issuer: REDACTED
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55802c1286a0)
> POST /auth/oauth2/token HTTP/2
> Host: <IP_ADDRESS>:8443
> user-agent: curl/7.67.0
> accept: */*
> content-type: application/x-www-form-urlencoded
> content-length: 119
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* We are completely uploaded and fine
< HTTP/2 200
< date: Fri, 30 Apr 2021 22:42:51 GMT
< content-type: application/json; charset=utf-8
< cache-control: no-store
< pragma: no-cache
< content-length: 91
< optum-cid-ext: c4922c7b-37cb-4fcd-a911-a5135c7ba01a#42596
< x-response-latency: 30
<
* Connection #0 to host <IP_ADDRESS> left intact
{"token_type":"bearer","access_token":"Dy1rxZjetMo48Y3kNMcW2MvvxuUhWp0D","expires_in":3600}/

I wonder if some kind http request body context issue is happening(persisting or saving that incorrectly between calls sometimes), maybe I will log what request body the plugin is seeing when I get an error in a given datacenter... that might help will add that print statement in the access phase..

Welp applied the https://github.com/Optum/kong-tx-debugger , to the token gen endpoint... at least by the log phase the request body is claimed to be correct as seen in this debug log:
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:33 [kong-tx-debugger] -------- START OF LOGGED TX -------- while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #1 x-forwarded-proto : https while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #2 content-length : 143 while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #3 host : gateway-dev-network-zone-dc2.optum.com while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #4 postman-token : e65754a9-a863-490f-abc0-65cb011aa731 while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #5 accept-encoding : gzip, deflate, br while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #6 user-agent : PostmanRuntime/7.25.0 while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #7 traceparent : 00-6e9da59b541469bcf6c81f794980d132-fb969533aff2283b-00 while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #8 x-forwarded-for : <IP_ADDRESS> while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #9 testheader : Test1 while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #10 content-type : application/json while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #11 accept : */* while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #12 cookie : Cookie_2=value while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:38 [kong-tx-debugger] Request Header #13 connection : keep-alive while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:62 [kong-tx-debugger]
Request Body:
{
 "client_id": "6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV",
 "client_secret": "C1nnQ0wmwgT2PkUunUhs8NfkMQLNM7CA",
 "grant_type": "client_credentials"
} while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:69 [kong-tx-debugger] Response Header #1 content-length : 78 while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:69 [kong-tx-debugger] Response Header #2 connection : close while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:69 [kong-tx-debugger] Response Header #3 cache-control : no-store while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:69 [kong-tx-debugger] Response Header #4 pragma : no-cache while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:69 [kong-tx-debugger] Response Header #5 content-type : application/json; charset=utf-8 while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:69 [kong-tx-debugger] Response Header #6 x-response-latency : 1 while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:69 [kong-tx-debugger] Response Header #7 optum-cid-ext : 2f04c9ff-ca36-4ede-8735-fb98c0bf60c9#13 while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:85 [kong-tx-debugger]

Response Body:
{"error":"invalid_client","error_description":"Invalid client authentication"}

while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"
2021/04/30 23:06:08 [notice] 86#0: *83223 [kong] handler.lua:88 [kong-tx-debugger] -------- END OF LOGGED TX -------- while logging request, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-network-zone-dc2.optum.com"

Darn was hoping that would give a lead... but programmatically the OAuth2.0 plugin should not return a 400 for such a valid token request body with good creds... Arggg, yeah I can't feel 100% about that result... gonna drop in code in the oauth2.0 access.lua file too to capture what it sees on these 400 invalid client auth errors too more directly..

Confirmed via log print statements we are indeed hitting here when the error happens:
https://github.com/Kong/kong/blob/master/kong/plugins/oauth2/access.lua#L680
so client is nil, which gets set here:
https://github.com/Kong/kong/blob/master/kong/plugins/oauth2/access.lua#L540
Digging deeper, will print the client_id/client_secret and that client var as a table as string dump(my guess is its nil on the failures at this step though).

Okay... I thought I was onto something predictable in the behavior at this point and It seems I am after 4 test runs.. the FIRST datacenter I go through to get to a kong pod over there the oauth2 token gen will always fail with the client as nil and continue to do so.... the SECOND call when I send to the other datacenter will succeed and will continue to do so, notes from the test. After each test I would rolling redeploy both Kong pods in each respective datacenter:
1st run:
1st hit failed (DC1)
2nd hit success (DC2)
2nd run:
1st hit failed (DC2)
2nd hit success (DC1)
3nd run:
1st hit failed (DC1)
2nd hit success (DC2)
4th run:
1st hit failed (DC2)
2nd hit success (DC1)
So timing of which datacenter kong pod oddly got that oauth2 token gen traffic first fails, and whichever DC gets the next call will succeed. Very strange. What under the covers could cause that...

Some deeper print statements from the first DC that starts failing on the calls:
Based on this code adjustment to OAuth2.0 plugin:
local function load_oauth2_credential_by_client_id(client_id)
  local credential, err = kong.db.oauth2_credentials:select_by_client_id(client_id)
  kong.log("load_oauth2_credential_by_client_id credential: " .. dump(credential))
  kong.log("load_oauth2_credential_by_client_id err: " .. dump(err))
  if err then
    return nil, err
  end

  return credential
end

local function get_redirect_uris(client_id)
  local client, err
  if type(client_id) == "string" and client_id ~= "" then
    local credential_cache_key = kong.db.oauth2_credentials:cache_key(client_id)

    kong.log("get_redirect_uris credential_cache_key: " .. credential_cache_key)

    client, err = kong.cache:get(credential_cache_key, nil,
                                 load_oauth2_credential_by_client_id,
                                 client_id)

    kong.log("get_redirect_uris err: " .. dump(err))

    if err then
      return error(err)
    end
  end

  return client and client.redirect_uris or nil, client
end

2021/05/01 08:20:43 [notice] 83#0: *2244 [kong] access.lua:559 [oauth2] Client id: 6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
2021/05/01 08:20:43 [notice] 83#0: *2244 [kong] access.lua:560 [oauth2] Client secret: C1nnQ0wmwgT2PkUunUhs8NfkMQLNM7CA, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
2021/05/01 08:20:43 [notice] 83#0: *2244 [kong] access.lua:183 [oauth2] get_redirect_uris credential_cache_key: oauth2_credentials:6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV:::::, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
2021/05/01 08:20:43 [notice] 83#0: *2244 [kong] access.lua:168 [oauth2] load_oauth2_credential_by_client_id credential: nil, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
2021/05/01 08:20:43 [notice] 83#0: *2244 [kong] access.lua:169 [oauth2] load_oauth2_credential_by_client_id err: nil, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
2021/05/01 08:20:43 [notice] 83#0: *2244 [kong] access.lua:189 [oauth2] get_redirect_uris err: nil, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
Some logs from the second Datacenter where the call will succeed:
2021/05/01 08:25:35 [notice] 86#0: *4735 [kong] access.lua:559 [oauth2] Client id: 6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
2021/05/01 08:25:35 [notice] 86#0: *4735 [kong] access.lua:560 [oauth2] Client secret: C1nnQ0wmwgT2PkUunUhs8NfkMQLNM7CA, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
2021/05/01 08:25:35 [notice] 86#0: *4735 [kong] access.lua:183 [oauth2] get_redirect_uris credential_cache_key: oauth2_credentials:6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV:::::374dc773-9bf3-482e-921f-9cd6b5ffe15d, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
2021/05/01 08:25:35 [notice] 86#0: *4735 [kong] access.lua:168 [oauth2] load_oauth2_credential_by_client_id credential: { ["created_at"] =<PHONE_NUMBER>,["id"] = f6779d52-f615-4e98-960a-23220e046b62,["name"] = OAuth2,["client_type"] = confidential,["client_id"] = 6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV,["hash_secret"] = false,["redirect_uris"] = { [1] = https://optum.com ,} ,["consumer"] = { ["id"] = 092798fd-5055-4b14-a699-4f915d874695,} ,["client_secret"] = C1nnQ0wmwgT2PkUunUhs8NfkMQLNM7CA,} , client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
2021/05/01 08:25:35 [notice] 86#0: *4735 [kong] access.lua:169 [oauth2] load_oauth2_credential_by_client_id err: nil, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
2021/05/01 08:25:35 [notice] 86#0: *4735 [kong] access.lua:189 [oauth2] get_redirect_uris err: nil, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1",
Looks like the difference that leads to all the failures in the second DC is the get_redirect_uris credential_cache_key variable  , from the given first called DC its returned as:
get_redirect_uris credential_cache_key: oauth2_credentials:6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV:::::

Second working DC call will look like:
get_redirect_uris credential_cache_key: oauth2_credentials:6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV:::::374dc773-9bf3-482e-921f-9cd6b5ffe15d

Which looks to have included an extra ws uuid at the end of the key on lookup, the correct and only "default" workspace in this environment. Why would the same codebase fail to do the same in the other datacenter, and all cassandra nodes in the cluster report this is indeed the workspace id across both datacenters... oauth2_credentials table has that ws id set properly in the table in the client_id col. and the ws_id col.
Welp thats about as deep as I can go on the problem. Let me know if you have any other code snippits you want me to try that would help or if you want me to demo the issues on a zoom call or w/e. Enough weekend debugging for me :P .

@jeremyjpj0916, thanks for all the new info. I was first thinking that could it be negative caching, but your last comment in this thread seems to point to a missing workspace id. I wish we could have just hard coded default workspace id, but there were reasons that we could not, and thus it varies between clusters. All cluster nodes should be able to get the default workspace id though. I will try to look if there is something we might have missed.

Yeah I was thinking if I could deeper dive and see where that id comes from on the:
[oauth2] get_redirect_uris credential_cache_key

lookup call. If its just reading it from the db to build that cache key then thats really strange how same code, same db clusters showing the data would behave like that, if its cached earlier in lua land or something and appended in then maybe somewhere there I can just do a quick print statement to tell me what my Kong sees as my workspace id may give some insight to whats going on, I assume it would just be nil or empty in the second datacenter that would be called.. but I would think all this gets cached at startup so idk why it matters which dc kong node I call first to get the fails leads to the second dc kong node succeeding and alternating their successes and failures when I rotate them, super odd bug haha.

@jeremyjpj0916, I tried to give eye, but I could not find anything specific, but I prepared one PR in case if it helps any. Could you try changes I made here: https://github.com/Kong/kong/pull/7118
It is just my try to make it more robust to never have nil workspace set. It is still hard for me to understand why different nodes connecting to different DC have different state, when as you said, cql and friends see exactly the same data. But if you can try it out and report back it would help a bit.

Will give those patch adjustments a try later today. Thanks!

Patches from PR 7118 dropped in, still seeing first call to the first DC kong node throwing the error and the cache key lookup yeilding:
 2021/05/06 07:25:02 [notice] 89#0: *3697 [kong] access.lua:183 [oauth2] get_redirect_uris credential_cache_key: oauth2_credentials:6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV:::::,

And the second token gen call to the other dc kong node yielding:
2021/05/06 07:25:32 [notice] 86#0: *3570 [kong] access.lua:183 [oauth2] get_redirect_uris credential_cache_key: oauth2_credentials:6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV:::::374dc773-9bf3-482e-921f-9cd6b5ffe15d,

Also did another cql query tonight on all 6 C* nodes involved in this cluster:
kongdba@cqlsh:kong_dev> select * from workspaces;

 id                                   | comment | config | created_at                      | meta | name
--------------------------------------+---------+--------+---------------------------------+------+---------
 374dc773-9bf3-482e-921f-9cd6b5ffe15d |    null |   null | 2020-08-14 05:28:21.000000+0000 | null | default

Thats the reply on all of them, same ole workspace id. Any kinda logging statements we could add that would reveal why the ws id isn't loading into the one kong nodes cache keys? Or a log print that would give better visibility to the problem you can think up?

@jeremyjpj0916,
Umm, perhaps it is an issue with schema.workspaceable, so I made changes to #7118, but I accidentally amended the commit there, but if you can try that (it is all in one commit) - I also added log statements there so we can better see what is going on?
https://github.com/Kong/kong/pull/7118/files

Added the files! Will rebuild and test this evening and see what I can see with it. Will include the logs you added to the git issue here from both datacenter kong nodes when I test it.

All right, got 2.4 on older openresty/nginx:
/ $ kong version --vv
2021/05/12 05:10:53 [verbose] Kong: 2.4.0
2021/05/12 05:10:53 [debug] ngx_lua: 10017
2021/05/12 05:10:53 [debug] nginx: 1017008
2021/05/12 05:10:53 [debug] Lua: LuaJIT 2.1.0-beta3
2.4.0
/ $

##############################################################
##### BEGIN Jeremy Jeremy 2.4 revert openresty patch #####

COPY patches/2.4fix/kong/openrestyrevert/kong/meta.lua /usr/local/kong/luarocks/share/lua/5.1/kong/meta.lua
RUN chmod 777 /usr/local/kong/luarocks/share/lua/5.1/kong/meta.lua

COPY patches/2.4fix/kong/openrestyrevert/kong/resty/ctx.lua /usr/local/kong/luarocks/share/lua/5.1/kong/resty/ctx.lua
RUN chmod 777 /usr/local/kong/luarocks/share/lua/5.1/kong/resty/ctx.lua

##### END Jeremy 2.4 revert openresty patch #####
############################################################

Testing functionality now.

So far seemingly working?
2021/05/12 05:19:32 [notice] 88#0: *3345 [kong] access.lua:560 [oauth2] Client secret: C1nnQ0wmwgT2PkUunUhs8NfkMQLNM7CA, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"
2021/05/12 05:19:32 [error] 88#0: *3345 [lua] init.lua:1461: cache_key(): DAO:CACHE_KEY -> KEY: "6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV", client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"
2021/05/12 05:19:32 [error] 88#0: *3345 [lua] init.lua:1462: cache_key(): DAO:CACHE_KEY -> WS_ID: nil, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"
2021/05/12 05:19:32 [error] 88#0: *3345 [lua] init.lua:1463: cache_key(): DAO:CACHE_KEY -> WORKSPACEABLE: true, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"
2021/05/12 05:19:32 [error] 88#0: *3345 [lua] init.lua:58: get_workspace_id(): GET WORKSPACE ID (WITH REQUEST): "374dc773-9bf3-482e-921f-9cd6b5ffe15d", client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"
2021/05/12 05:19:32 [error] 88#0: *3345 [lua] init.lua:59: get_workspace_id(): GET WORKSPACE ID DEFAULT (WITH REQUEST): "374dc773-9bf3-482e-921f-9cd6b5ffe15d", client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"
2021/05/12 05:19:32 [error] 88#0: *3345 [lua] init.lua:1464: cache_key(): DAO:CACHE_KEY -> GET_WORKSPACE_ID: "374dc773-9bf3-482e-921f-9cd6b5ffe15d", client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"
2021/05/12 05:19:32 [error] 88#0: *3345 [lua] init.lua:1465: cache_key(): DAO:CACHE_KEY -> DEFAULT_WORKSPACE: "374dc773-9bf3-482e-921f-9cd6b5ffe15d", client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"
2021/05/12 05:19:32 [error] 88#0: *3345 [lua] init.lua:58: get_workspace_id(): GET WORKSPACE ID (WITH REQUEST): "374dc773-9bf3-482e-921f-9cd6b5ffe15d", client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"
2021/05/12 05:19:32 [error] 88#0: *3345 [lua] init.lua:59: get_workspace_id(): GET WORKSPACE ID DEFAULT (WITH REQUEST): "374dc773-9bf3-482e-921f-9cd6b5ffe15d", client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"
2021/05/12 05:19:32 [notice] 88#0: *3345 [kong] access.lua:183 [oauth2] get_redirect_uris credential_cache_key: oauth2_credentials:6RE9IXHWgBbfybOXCqxZkUZnCQPKEuIV:::::374dc773-9bf3-482e-921f-9cd6b5ffe15d, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"
2021/05/12 05:19:32 [notice] 88#0: *3345 [kong] access.lua:189 [oauth2] get_redirect_uris err: nil, client: <IP_ADDRESS>, server: kong, request: "POST /auth/oauth2/token HTTP/1.1", host: "gateway-dev-networkzone-dc1.optum.com"

Okay, I have redeployed in both DCs each 8-9 times and its worked every time. No issues now. What does that mean?
Speculating here:
Either the changes in:
kong/resty/ctx.lua are problematic with newer openresty
OR
The behavior/changes of newer OpenResty cause issues with Kong 2.4 logic somewhere.
My guess is the second one but I keep both options open because I am not super familiar with the resty/ctx.lua other than its origins when I mentioned it for how 3scale gets around context droppage and Tibo integrated the logic into Kong long ago.

Yes, I am too leaning towards LuaJIT changes that 1.19.3.x brought. But sure, let's keep both on a table until we know better.

@jeremyjpj0916,
Could you for example add:
ngx.log(ngx.ERR, "LOADING PLUGIN: ", plugin)

After this:
https://github.com/Kong/kong/blob/master/kong/db/dao/plugins.lua#L272
I am interested that when the bug happens, is the order different compared to when it does not happen? Do you have any custom plugins that reference entities in core plugins?

@bungle
Will give that a test and report back, will convert back to using the newer openresty for that test to reproduce issues.
Hmm as for custom plugins referencing entities in core plugins. We have basically a clone of the kong oauth2 plugin under a different plugin name that acts very similarly but its behavior is to cache the token kong level and if people request a new token sooner than its exp it will return the older token with its declining expiration up to 1 min of its life left (to prevent a new db call and record injection, faster and less stress on C*). So like 90% of the code is the same besides a little bit added that uses the cache response to clients that are not using oauth2 correctly and reusing their token like responsible clients would.

Does your OAuth2 clone have same name for entities, or different? Is the plugin name different or same? Also do you have daos file with that?

Note when I am reporting errors with OAuth2.0 above I have not been using the OAuth2.0 "perf" endpoint we have which is the cloned code with minor tweaks. I was using all the defacto kong oauth2.0 logic. As for the perf plugin/code that lives+runs  on a separate kong route repo looks like this:

We drop it in a folder:
##### New OAuth2 perf from source #####

RUN git clone -b dev https://github.company.com/APIGateway/kong-plugin-oauth.git /usr/local/kong/luarocks/share/lua/5.1/kong/plugins/kong-plugin-oauth

##### END New OAuth2 perf from source #####
############################################################

using its own SHM:
local access = require "kong.plugins.kong-plugin-oauth.access"
local mlcache = require "resty.mlcache"
local kong = kong
local KongOAuthHandler = {}

KongOAuthHandler.PRIORITY = 1004
KongOAuthHandler.VERSION = "2.0.0"

function KongOAuthHandler:init_worker()
  local oauthcache, err = mlcache.new("kong_oauth_cache", "kong_oauth_cache", {
    shm_miss         = "kong_oauth_cache_miss",
    shm_locks        = "kong_oauth_cache_locks",
    shm_set_retries  = 3,
    lru_size         = 1000,  -- size of the L1 (Lua VM) cache
    ttl              = 0,     --No expire on init per Korver suggestion
    neg_ttl          = 5,     --5 second ttl for misses
    resty_lock_opts  = {exptime = 10, timeout = 5,},
  })

  if not oauthcache then
    ngx.log(ngx.ERR, "failed to instantiate oauth mlcache: " .. err)
    return
  end

  kong.oauth_cache = oauthcache
end

function KongOAuthHandler:access(conf)
  access.execute(conf)
end

return KongOAuthHandler

Ooo but we may be getting somewhere here too, so the schema.lua file (which dictates plugin name is:
local typedefs = require "kong.db.schema.typedefs"

local function validate_flows(config)
  if config.enable_authorization_code
  or config.enable_implicit_grant
  or config.enable_client_credentials
  or config.enable_password_grant
  then
    return true
  end

  return nil, "at least one of these fields must be true: enable_authorization_code, enable_implicit_grant, enable_client_credentials, enable_password_grant"
end

return {
  name = "oauth2",
  fields = {
    { consumer = typedefs.no_consumer },
    { protocols = typedefs.protocols_http },
    { config = {
        type = "record",
        fields = {
          { scopes = { type = "array", elements = { type = "string" }, }, },
          { mandatory_scope = { type = "boolean", default = false, required = true }, },
          { provision_key = { type = "string", unique = true, auto = true, required = true }, },
          { token_expiration = { type = "number", default = 7200, required = true }, },
          { enable_authorization_code = { type = "boolean", default = false, required = true }, },
          { enable_implicit_grant = { type = "boolean", default = false, required = true }, },
          { enable_client_credentials = { type = "boolean", default = false, required = true }, },
          { enable_password_grant = { type = "boolean", default = false, required = true }, },
          { hide_credentials = { type = "boolean", default = false, required = true }, },
          { accept_http_if_already_terminated = { type = "boolean", default = false }, },
          { anonymous = { type = "string" }, },
          { global_credentials = { type = "boolean", default = false }, },
          { auth_header_name = { type = "string", default = "authorization" }, },
          { refresh_token_ttl = { type = "number", default = 1209600, required = true }, },
        },
        custom_validator = validate_flows,
        entity_checks = {
          { conditional = {
              if_field = "mandatory_scope",
              if_match = { eq = true },
              then_field = "scopes",
              then_match = { required = true },
          }, },
        },
      },
    },
  },
}

So its called oauth here which matches the Kong oauth plugin too... But admin api and konga UI still display it as its own named thing, and I had to add kong-plugin-oauth long ago to plugins list to make it work...

And I do see here in your DAOS where you mention workspaceable true:
https://github.com/Kong/kong/blob/master/kong/plugins/oauth2/daos.lua , the custom plugin does not do so in its daos.lua file.
Could this all potentially be related then as well? And I wonder if it is, why does the OpenResty bump seem to bring out the issue that otherwise would not present itself... hmmm. Let me try updating the dao on the custom plugin and also rename it in the schema.lua in dev and see if it all plays any role here.

Did find the custom plugin in the db plugins table too, has the name kong-plugin-oauth at least, hence why konga and the plugins list env variable need it like so I guess, and maybe because this plugin was written prior to the plugin name being in the schema and I used to have it in the handler.lua maybe and forgot to further update it later when kong transitioned?:

Yes, it is possible that the names conflict even when the plugins are in their own folders.

Testing confirmed, still had your patch fixes in for that ssl context issue causing plugins to run but retesting after refactoring that plugins schema.lua file to have its own name and editing the daos.lua to match what the core kong oauth2 plugin had resolved the behavior of workspaceable sometimes being nil, which means yeah the custom plugin due to same name in schema somehow got loaded in sometimes instead of the kong core oauth2 daos logic is my thoughts...
will delete the api.lua and see if that custom plugin can still be added / removed via the admin api, maybe it can maybe it can't and will add it back but the plugin guide still mentiones api.lua helps expose admin api functionality:

Still does not explain the JWT auth issue as we have no custom stuff or custom plugin going on there similarly to oauth2 but I will be on the lookout for it, haven't been able to reproduce the issue there personally myself yet... I would say drop in the fixes so far for 4.0.1 from the other git issue and ship it,
And this one we figured out its due to new openresty version behavior and kong 2.4 exposing a clash in plugins that never exposed itself before. So basically any Kong user who has named another plugin same in schema as a core plugin and the daos are not in sync could face this flake loading between the two stuff. Rare case, but interesting stuff. Sorry we dug through it so deep to get here but glad to know what could cause it if it pops up again, I am probably the only one doing a custom thing like this though.

Closing now, think we got to bottom of it and I see 4.0.1 is released, will test it out.

2.4.1 with no patches dropped in seems to be executing fine w my additional edits to the custom oauth2 plugin.

@jeremyjpj0916, good to know. I will collect some ideas from this thread, and create a bunch of tasks for us. I don't for example like that in your case, the different startups could result a different state (randomly). Also we could log warnings if things with same name are seen. And yes, please open another ticket for possible JWT issue, if you can still see it or it happens.
Thank you!

Good points, random state is never a great thing, but I think Kong still maintains plugins that share the same execution priority have undetermined priority as well between each other 😆, so not the only time a little mystery is still around, keeps it interesting. I am always for good logs and warnings. More visibility folks get to things that seem incorrect/invalid the better.


allow disabling crud actors generation

[development_crud_disable]
for packages in tfgrid it's not right to allow crud actors to manipulate data in there without proper permissions or without an option at least to disable the generation

@zaibon @despiegk

Added disable option for crud actors to disable generation of actors. This option can be used by adding to  disable_crud = true in the package's package.toml file.
https://github.com/threefoldtech/jumpscaleX_core/commit/52ce2a6a29698dbdf4d5dc2b281f1ea661d8d399

do we have some doc somewhere?

lets close today

do we have some doc somewhere?

No, ill add some documentation for how to use it

Verified
Steps followed:

Find/create a package with models directory, with  a model in a toml file

you can use  this package /sandbox/code/github/threefoldtech/jumpscaleX_threebot/ThreeBotPackages/zerobot/webplatform

include in the package.toml source section disable_crud=true

load the package from within the shell

When trying to access the actors, there will be no actors generated for the crud models only the normal package actors

To test the other way around on a clean environment, remove the disable option, and try the steps again but this time the actors will be generated for that package


Uppercase character 'W' is not rendered correctly in the terminal with 'Cascadia' series font

Does this issue occur when all extensions are disabled?: Yes

版本: 1.88.0-insider (user setup)
提交: be210b3a60c7d60030c1d3d92da00d008edf6ab9
日期: 2024-03-15T05:48:19.790Z
Electron: 28.2.6
ElectronBuildId: 27476517
Chromium: 120.0.6099.291
Node.js: 18.18.2
V8: 12.0.267.19-electron.0
OS: Windows_NT x64 10.0.22631
The uppercase character 'W' is not rendered correctly in the terminal with 'Cascadia' series font. It will render properly in editor, terminal tab with color, in Windows Terminal application, and with other font series. Also it's seems that uppercase 'W' is the only character that occurs this problem. I have no idea why this happened.

('W' wasn't rendered correctly)

(but normally in editor)

(also in Windows Terminal)
'Cascadia Code' and 'Cascadia Mono' have the same problem.

This has been fixed in latest commit.
版本: 1.88.0-insider (user setup)
提交: 409da0445d397e37a790ce5d2499688955c3fcfa
日期: 2024-03-22T05:50:17.322Z
Electron: 28.2.6
ElectronBuildId: 27476517
Chromium: 120.0.6099.291
Node.js: 18.18.2
V8: 12.0.267.19-electron.0
OS: Windows_NT x64 10.0.22631


Failed to set the pex root directory

I tried to use pex with PySpark and run into the permission denied issue with the default pex root directory. I tried to set a different writable pex root directory using a few approaches but all failed.
--conf spark.executorEnv.PEX_ROOT
It seems to me that pex doesn't honor the environment variable set by this way. I can verify from PySpark (not with pex) that environment variables set this is readable by Python but when used with pex, it has no effect.  I know nothing about spark, but just now searched through the docs, and given your config (yarn), it looks like you need --conf spark.yarn.appMasterEnv.PEX_ROOT=/some/writable/directory instead of --conf spark.executorEnv.PEX_ROOT=/some/writable/directory. The docs I read are here: https://spark.apache.org/docs/2.3.0/running-on-yarn.html#configuration, but snipped:

Add the environment variable specified by EnvironmentVariableName to the Application Master process launched on YARN. The user can specify multiple of these and to set multiple environment variables. In cluster mode this controls the environment of the Spark driver and in client mode it only controls the environment of the executor launcher.

Let me know if this works for you.

@jsirois Thank you for your inspirations!  This means that I need of the following configurations.
--conf spark.executorEnv.PEX_ROOT=/some/writable/directory
--conf spark.yarn.appMasterEnv.PEX_ROOT=/some/writable/directory


asyncio: memory leak when registering more than one socket to poller

#968 fixed memory leak for one socket but polling multiple sockets still leak memory
import zmq
import zmq.asyncio
import asyncio

sock1 = zmq.asyncio.Context.instance().socket(zmq.REP)
sock1.connect("tcp://localhost:55570")
sock2 = zmq.asyncio.Context.instance().socket(zmq.REP)
sock2.connect("tcp://localhost:55560")

poller = zmq.asyncio.Poller()
poller.register(sock1, zmq.POLLIN)
poller.register(sock2, zmq.POLLIN)

async def main():
    while True:
        try:
            await poller.poll(timeout=1)
            pass
        except KeyboardInterrupt:
            break
asyncio.run(main())

It seems that _recv_futures is still expanding indefinitely. The watcher is removed from _recv_futures on one socket but remains in others sockets' list.

closed by #1362


hls multibitrate support

I couldn't have a success for live hls multi-bitrate setup.
My setup as below;

created single application with hls support on with name hls.
push 3 level multi src stream to the hls application by using, adobe live encoder.
I have the latest jwplayer with hls support.
There is no problem on single level hls test.
The problem is, rtmp module creates 3 independent .m3u8 files.

I couldn't find any information for this subject.  I wondered if multi-bitrate hls supported on your modul or not?

Sorry to use this issue ticket for a different question:
@tanero I am trying to do the same: ingest 3 streams with afmle but only the last stream (highest number) gets segemented in the hls app. Could you share your config so I could figure out what I missed?


time_index only in seconds?

Hi guys, i'm new in FIWARE.
I'm wondering why if i send to the broker messages with a time interval < 1 second i get this behaviour (more record in same second):

I want to use milliseconds in my project.
Any advice?
Thanks in advance

Hi there!
QL processes time_index with precision of milliseconds so that shouldn't be a problem.
You need to make sure the Orion subscription has no throttling to actually get more than 1 per second. But that what you show, 4 times the same instant, looks weird. I've seen sometimes a duplicate of two, specially with the first notification Orion sends, but not more than that.
What does your subscription look like? which Orion and QL versions do you have?.

Orion version: 2.2.0
QL version: 0.6
I made subscription using QL API:
POST url = 'http://host8668/v2/subscribe'
With this data {
"orionUrl": 'http://orion:1026/v2',  "quantumleapUrl":'http://quantumleap:8668/v2',
"entityType": entityType
}

@AngeloIapichino most probably the problem is what orion sends to ql. please check that dateModified does not include milliseconds. also, refer to the time-index discussion to see how to use alternative columns as time_index

Orion does not support milliseconds. See for example this issue.

Thanks ohylli! We get the problem know...

If you insert directly to QuantumLeap via its notify API I think it will then keep the milliseconds.

AFAIK, it supports it for but not for default dateCreatetion
dateModification

i suppose we can close the issue, if not please, let us know, we will re-open it


bower version is not the latest, doesn't include dibari:refresh-ellipsis

I tried using this functionality from the bower version and it seems like it's not there yet, when I clone the repository from here it works, but only in the not-minified version

👍  Can someone tag and publish the latest please? @dibari or @MauMaGau?

+1

Updated to 0.1.7.

It's up to @dibari to publish to npm.

@dibari @MauMaGau
The latest<EMAIL_ADDRESS>version too doesn't have dibari:refresh-ellipsis change.
In the source code downloaded from tag/0.1.7  the raw version has the dibari:refresh-ellipsis change but not the min version.
Any plans to push the latest changes to npm ?

@dibari @MauMaGau
The latest<EMAIL_ADDRESS>version too doesn't have dibari:refresh-ellipsis change.
In the source code downloaded from tag/0.1.7  the raw version has the dibari:refresh-ellipsis change but not the min version.
Any plans to push the latest changes to npm ?

I'm afraid I no longer maintain this package. Unless @dibari says otherwise I would consider this package stale.
If there is an actively maintained fork then I'd be happy to link to that in the readme

I'm afraid I no longer maintain this package. Unless @dibari says otherwise I would consider this package stale.
If there is an actively maintained fork then I'd be happy to link to that in the readme

Hi all, unfortunately I no longer maintain this package and I would consider it stale.  I'm happy to transfer it another maintainer if anyone is interested?

Hi all, unfortunately I no longer maintain this package and I would consider it stale.  I'm happy to transfer it another maintainer if anyone is interested?


{CosmosDB} Fix for local variable 'throughput_resource' referenced before assign…

…ment
Considering the --max-throughput and --throughput are optional parameters you will get the below exception.
More Info: https://docs.microsoft.com/en-us/cli/azure/cosmosdb/mongodb/collection/throughput?view=azure-cli-latest#az-cosmosdb-mongodb-collection-throughput-update
Exception: local variable 'throughput_resource' referenced before assignment
This PR will address this exception.
Steps to reproduce:
az cosmosdb mongodb collection throughput update --account-name {} --database-name { } --name { } --resource-group { }
Exception:
The command failed with an unexpected error. Here is the traceback:
local variable 'throughput_resource' referenced before assignment
Traceback (most recent call last):
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 231, in invoke
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 658, in execute
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 721, in _run_jobs_serially
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 692, in _run_job
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 328, in call
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 121, in handler
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/cosmosdb/custom.py", line 1280, in cli_cosmosdb_mongodb_collection_throughput_update
File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/cosmosdb/custom.py", line 1445, in _get_throughput_settings_update_parameters
UnboundLocalError: local variable 'throughput_resource' referenced before assignment
Description

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

This checklist is used to make sure that common guidelines for a pull request are followed.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[x] I adhere to the Error Handling Guidelines.

CosmosDB


Possible mistake in the Java SDK composite index example

In the composite index example for the Java SDK, the composite paths are set to /name/* and /age/*, while in the corresponding examples for the other SDKs, they are set to /name and /age. I think the syntax ending with /* isn't even valid for composite paths, so this seems like a typo.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f1c7d0e0-b932-085e-6816-0b2219a1f37b
Version Independent ID: b463dc0b-313c-d90a-b8aa-c2b64004f812
Content: Manage indexing policies in Azure Cosmos DB
Content Source: articles/cosmos-db/how-to-manage-indexing-policy.md
Service: cosmos-db
GitHub Login: @ThomasWeiss
Microsoft Alias: thweiss

Hi @fabiobm, million thanks for spotting that! It is a typo indeed and we're in the process of updating this page accordingly. We really appreciate your help here.
#please-close


Define default sort key

Choose default sort keys to easily spot problems. (Rx-drop/s?)

rx-dps & rx-Bps have been chosen as most meaningful in the cloud service provider where rim  has born.

Closed with 0c6d689bdd70c64f5e40abc15da84dc89c8d420e


Fixes dev-deploy script

Which issue(s) this PR fixes:
dev-deploy script was no longer working since the last commit was merged.
Review Checklist:

[ ] Follows the developer guidelines
[ ] Relevant tests are added or updated
[ ] Relevant docs in this repo added or updated
[ ] Relevant carvel.dev docs added or updated in a separate PR and there's
a link to that PR
[ ] Code is at least as readable and maintainable as it was before this
change

i checked out this branch locally and it seems to work


Implement EvalResultsService/ListEvaluationResults()

Summary
This PR implements the handler for minder.v1.EvalResultsService/ListEvaluationResults() to query the evaluation statuses of profiles and entities. It adds a field to include the ruletype guidance to the sql query behind ListRuleEvaluationsByProfileId.
/cc @rdimitrov @kantord
Fixes #2525
Change Type
Mark the type of change your PR introduces:

[ ] Bug fix (resolves an issue without affecting existing features)
[x] Feature (adds new functionality without breaking changes)
[ ] Breaking change (may impact existing functionalities or require documentation updates)
[ ] Documentation (updates or additions to documentation)
[ ] Refactoring or test improvements (no bug fixes or new functionality)

Testing
Outline how the changes were tested, including steps to reproduce and any relevant configurations.
Attach screenshots if helpful.
Review Checklist:

[x] Reviewed my own code for quality and clarity.
[x] Added comments to complex or tricky code sections.
[ ] Updated any affected documentation.
[ ] Included tests that validate the fix or feature.
[ ] Checked that related changes are merged.

Hi @puerco, I noticed that the remediation status is an arbitrary string in the JSON schema:
        "remediationStatus": {
          "type": "string",
          "title": "remediation_status is the status of the remediation"
        },

this makes it difficult to correctly display this value in the FE, because this value is not actually displayed as a free-form string. For example each value is displayed with a different icon and color.
It would be really useful to have the list of all possible values restricted in the JSON schema using enum or sth similar.
If that is not possible then at the very least I will need the list of possible values in order to make sure that all values are displayed properly :pray:

coverage: 46.596% (-0.005%) from 46.601%
when pulling 85f7faeb635b1d26af2f53e9e3f2116e625b53b2 on puerco:profile-status
into 8a0bc3ff37459d634286dd35bdacfd24af0aba61 on stacklok:main.

@kantord here are the eval statuses currently supported:
https://github.com/stacklok/minder/blob/main/internal%2Fengine%2Ferrors%2Ferrors.go#L111
Since this is a new API call I think we can safely merge it soon if the approach is right, and then follow up with any changes, I can check how to have an enum.if the strings.

@kantord here are the eval statuses currently supported:
https://github.com/stacklok/minder/blob/main/internal%2Fengine%2Ferrors%2Ferrors.go#L111
Since this is a new API call I think we can safely merge it soon if the approach is right, and then follow up with any changes, I can check how to have an enum.if the strings.

sure, I agree, it's not a blocker for testing so let's merge